- developers - lists.mariadb.org

[Maria-developers] -Wnon-virtual-dtor and scripts in
by Alexander Barkov 08 Dec '17

08 Dec '17

Hello Vlad, Are we still supposed to use these scripts in the BUILD directory: compile-pentium64-debug compile-pentium64-debug-all compile-pentium64-debug-max compile-pentium64-gcov compile-pentium64-gprof compile-pentium64-max compile-pentium64-valgrind-max I stopped using them long time ago. But these scripts set some extra warnings for gcc, so now it's easy to overlook some warnings. Monty is still using these scripts and gets those extra warnings. In particular, -Wnon-virtual-dtor is often a subject of warnings. Is there a way to enable the -Wnon-virtual-dtor for gcc in the default build, i.e. when just using "cmake" instead of BUILD scripts? Thanks!

2 2

Re: [Maria-developers] 78a1779: mysqldump fix for invisible column
by Sergei Golubchik 07 Dec '17

07 Dec '17

Hi, Sachin! Looks fine. Two comments: 1. optimization idea, see below 2. add a test case that actually restores from the dump (and checks that it was restored correctly). mysqldump.test has examples, search for $MYSQL. After that - please push! On Dec 07, sachin wrote: > revision-id: 78a1779f29ed8d2a96658583065270de8bdd14b4 (mariadb-10.3.2-61-g78a1779) > parent(s): e8c8300a1c7e7e2b99e98c5c6986df69943cb661 > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2017-12-07 16:09:16 +0530 > message: > > mysqldump fix for invisible column > > Actually there are 2 issues in the case of invisible columns > > 1st `select fields from t1` will have more fields then `select * from t1`. > So instead of `select * from t1` we are using `select a,b,invisible from t1` > these fields are supplied from `select fields from t1`. > > 2nd We are using --complete-insert when we detect that this table is using > invisible columns. > > diff --git a/client/mysqldump.c b/client/mysqldump.c > index a260065..818bc07 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -2971,6 +2979,12 @@ static uint get_table_structure(char *table, char *db, char *table_type, > DBUG_RETURN(0); > } > > + while ((row= mysql_fetch_row(result))) > + { > + if (strlen(row[SHOW_EXTRA]) && strstr(row[SHOW_EXTRA],"INVISIBLE")) > + complete_insert= 1; > + } > + mysql_data_seek(result, 0); > /* > If write_data is true, then we build up insert statements for > the table's data. Note: in subsequent lines of code, this test > @@ -3006,10 +3020,16 @@ static uint get_table_structure(char *table, char *db, char *table_type, > { > dynstr_append_checked(&insert_pat, ", "); > } > - init=1; > dynstr_append_checked(&insert_pat, > quote_name(row[SHOW_FIELDNAME], name_buff, 0)); > } > + if (init) > + { > + dynstr_append_checked(&select_field_names, ", "); > + } > + dynstr_append_checked(&select_field_names, > + quote_name(row[SHOW_FIELDNAME], name_buff, 0)); > + init=1; I think, you can do it with just one loop: while ((row= mysql_fetch_row(result))) { if (strlen(row[SHOW_EXTRA]) && strstr(row[SHOW_EXTRA],"INVISIBLE")) complete_insert= 1; if (init) dynstr_append_checked(&select_field_names, ", "); dynstr_append_checked(&select_field_names, quote_name(row[SHOW_FIELDNAME], name_buff, 0)); init=1; } if (complete_insert) dynstr_append_checked(&insert_pat, select_field_names->str); > } > num_fields= mysql_num_rows(result); > mysql_free_result(result); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 339dd3d: Invisible Column mysqldump bug fix
by Sergei Golubchik 07 Dec '17

07 Dec '17

Hi, Sachin! Summary: 1. Add a test case, it should include a mix of tables with and without invisible columns, and columns with strange names (reserved words, spaces). 2. Always use full SELECT syntax, even if the table has no invisible columns. On Dec 05, sachin wrote: > revision-id: 339dd3d7508a1e5c7c014b5a59ed99ff0ad96542 (mariadb-10.3.2-59-g339dd3d) > parent(s): 6ed3374c8a67ae64448783ab107d1a11cb87f40c > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2017-12-05 05:35:28 +0530 > message: > > Invisible Column mysqldump bug fix > > --- > client/mysqldump.c | 46 +++++++++++++++++++++++++++++++++++++++++----- > 1 file changed, 41 insertions(+), 5 deletions(-) > > diff --git a/client/mysqldump.c b/client/mysqldump.c > index a260065..e59b203 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -115,7 +115,7 @@ static my_bool verbose= 0, opt_no_create_info= 0, opt_no_data= 0, opt_no_data_m > opt_events= 0, opt_comments_used= 0, > opt_alltspcs=0, opt_notspcs= 0, opt_logging, > opt_drop_trigger= 0 ; > -static my_bool insert_pat_inited= 0, debug_info_flag= 0, debug_check_flag= 0; > +static my_bool insert_pat_inited= 0, debug_info_flag= 0, debug_check_flag= 0, invisble_table= 0; typo, *invisible* > static ulong opt_max_allowed_packet, opt_net_buffer_length; > static MYSQL mysql_connection,*mysql=0; > static DYNAMIC_STRING insert_pat; > @@ -2726,7 +2726,7 @@ static uint get_table_structure(char *table, char *db, char *table_type, > complete_insert= 0; > if ((write_data= !(*ignore_flag & IGNORE_DATA))) > { > - complete_insert= opt_complete_insert; > + invisble_table= complete_insert= opt_complete_insert; > if (!insert_pat_inited) > { > insert_pat_inited= 1; > @@ -2971,6 +2971,12 @@ static uint get_table_structure(char *table, char *db, char *table_type, > DBUG_RETURN(0); > } > > + while ((row= mysql_fetch_row(result))) > + { > + if (strlen(row[SHOW_EXTRA]) && strstr(row[SHOW_EXTRA],"INVISIBLE")) > + invisble_table= complete_insert= opt_complete_insert= 1; I think, you need to set invisible_table flag per table, just as you do now, but you should not change complete_insert or opt_complete_insert. That is, if a database has many tables, some with invisible columns and some have only normal columns - tables with invisible columns should use complete_insert, while other tables should not. Basically, INSERT should contain column names if (invisible_table || opt_complete_insert) > + } > + mysql_data_seek(result, 0); > /* > If write_data is true, then we build up insert statements for > the table's data. Note: in subsequent lines of code, this test > @@ -3617,7 +3623,7 @@ static void dump_table(char *table, char *db) > { > char ignore_flag; > char buf[200], table_buff[NAME_LEN+3]; > - DYNAMIC_STRING query_string; > + DYNAMIC_STRING query_string, select_field_names; > char table_type[NAME_LEN]; > char *result_table, table_buff2[NAME_LEN*2+3], *opt_quoted_table; > int error= 0; > @@ -3687,7 +3693,25 @@ static void dump_table(char *table, char *db) > verbose_msg("-- Sending SELECT query...\n"); > > init_dynamic_string_checked(&query_string, "", 1024, 1024); > + init_dynamic_string_checked(&select_field_names, "", 1024, 1024); > > + if (invisble_table) > + { > + /* > + Copy field name from insert_pat into select_field_names because > + select * from table will leave hidden columns. > + We will search for "(" in insert_pat and will copy the whole data > + into select_field_names , and then we will search for ")" in > + select_field_names and decrese the lenght accordingly. > + */ > + char *ptr= insert_pat.str; > + while ( *ptr++ != '(' ){} > + dynstr_append_checked(&select_field_names, ptr); > + ptr= &select_field_names.str[select_field_names.length - 1]; > + while ( *ptr-- != ')') > + select_field_names.length--; > + select_field_names.str[select_field_names.length- 1]= 0; please, check how INSERT code is doing it, you need to quote column names. > + } > if (path) > { > char filename[FN_REFLEN], tmp_path[FN_REFLEN]; > @@ -3708,7 +3732,13 @@ static void dump_table(char *table, char *db) > > /* now build the query string */ > > - dynstr_append_checked(&query_string, "SELECT /*!40001 SQL_NO_CACHE */ * INTO OUTFILE '"); > + dynstr_append_checked(&query_string, "SELECT /*!40001 SQL_NO_CACHE */ "); > + if (invisble_table) > + dynstr_append_checked(&query_string, select_field_names.str); > + else > + dynstr_append_checked(&query_string, "*"); I don't think you need to bother, you can always have a complete column list here. It should not break anything. > + dynstr_append_checked(&query_string, " INTO OUTFILE '"); > + dynstr_free(&select_field_names); > dynstr_append_checked(&query_string, filename); > dynstr_append_checked(&query_string, "'"); > > @@ -3757,7 +3787,13 @@ static void dump_table(char *table, char *db) > "\n--\n-- Dumping data for table %s\n--\n", > fix_for_comment(result_table)); > > - dynstr_append_checked(&query_string, "SELECT /*!40001 SQL_NO_CACHE */ * FROM "); > + dynstr_append_checked(&query_string, "SELECT /*!40001 SQL_NO_CACHE */ "); > + if (invisble_table) > + dynstr_append_checked(&query_string, select_field_names.str); > + else > + dynstr_append_checked(&query_string, "*"); same > + dynstr_free(&select_field_names); > + dynstr_append_checked(&query_string, " FROM "); > dynstr_append_checked(&query_string, result_table); > > if (where) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 0f47171: MDEV-14005 Remove need for Partition Key to be part of Primary Key.
by Sergei Golubchik 06 Dec '17

06 Dec '17

Hi, Alexey! On Dec 04, Alexey Botchkov wrote: > revision-id: 0f4717118e135c178fc1b12065e33ac8a5e21d0b (mariadb-10.3.2-82-g0f47171) > parent(s): ddac2d7a1ef725361c7e83b819cb0bec72db024a > committer: Alexey Botchkov > timestamp: 2017-12-04 17:45:13 +0400 > message: > > MDEV-14005 Remove need for Partition Key to be part of Primary Key. > > Now we check all partitions for the unique keys if necessary > in ha_partition::write_row and ha_partition::update_row. > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 6bebf5d..12a6287 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -372,6 +372,7 @@ void ha_partition::init_handler_variables() > part_share= NULL; > m_new_partitions_share_refs.empty(); > m_part_ids_sorted_by_num_of_records= NULL; > + m_cu_errkey= (uint) -1; > > #ifdef DONT_HAVE_TO_BE_INITALIZED > m_start_key.flag= 0; > @@ -4088,6 +4089,79 @@ void ha_partition::try_semi_consistent_read(bool yes) > } > > > +int ha_partition::check_files_for_key(uchar *key, int n_key, > + int partition_to_skip, int *res) > +{ > +// uint num_subparts= part_info->num_subparts; > + for (uint n=0; n < m_part_info->num_parts; n++) > + { > + handler *f= m_file[n]; > + int ires; > + > + if ((int) n == partition_to_skip) > + continue; > + > + f->active_index= n_key; > + if ((*res= f->extra(HA_EXTRA_KEYREAD)) || > + (*res= f->ha_index_init(n_key, FALSE))) > + return 1; > + > + *res= f->index_read_map(m_part_info->table->record[0], > + key, HA_WHOLE_KEY, HA_READ_KEY_EXACT); use index_read_idx_map here > + ires= f->ha_index_end(); > + > + if (*res == HA_ERR_KEY_NOT_FOUND) > + continue; > + > + if (*res == 0) > + *res= ires ? ires : HA_ERR_FOUND_DUPP_KEY; > + > + m_last_part= n; > + m_cu_errkey= n_key; > + return 1; > + } > + > + *res= 0; > + return 0; > +} > + > + > +int ha_partition::check_uniques_insert(uchar *buf, > + int partition_to_skip, int *res) > +{ > + uint n_key; > + > + for (n_key=0; n_key < m_part_info->n_uniques_to_check; n_key++) > + { can you use key_copy() here? > + uchar *cbuf= m_part_info->unique_key_buf[0]; > + KEY *ckey= m_part_info->uniques_to_check[n_key]; > + uint n; > + > + for (n=0; n < ckey->user_defined_key_parts; n++) > + { > + const KEY_PART_INFO *cpart= ckey->key_part + n; > + uint maybe_null= MY_TEST(cpart->null_bit); > + Field *f= cpart->field; > + my_ptrdiff_t ofs= buf-table->record[0]; > + > + f->move_field_offset(ofs); > + if (maybe_null) > + cbuf[0]= f->is_null(); > + f->get_key_image(cbuf+maybe_null, cpart->length, Field::itRAW); > + cbuf+= cpart->store_length; > + f->move_field_offset(-ofs); > + } > + > + if (check_files_for_key(m_part_info->unique_key_buf[0], n_key, > + partition_to_skip, res)) > + return 1; > + } > + > + *res= 0; > + return 0; > +} > + > + > /**************************************************************************** > MODULE change record > ****************************************************************************/ > @@ -4199,6 +4273,11 @@ int ha_partition::write_row(uchar * buf) > error= HA_ERR_NOT_IN_LOCK_PARTITIONS; > goto exit; > } > + > + if (table->part_info->n_uniques_to_check && > + check_uniques_insert(buf, part_id, &error)) > + goto exit; This would _probably_ introduce the race condition I've mentioned in a comment to MDEV-14005. Say, you have a table with two partitions and a unique key. Now, two threads try to insert rows, the first thread inserts 1 into the first partition (there are more columns in a table and a partitioning key on those columns, but I omit these details for simplicity). And a second thread inserts 1 into the second partition. So, the first thread checks for 1 in the second partition, finds nothing. Second thread checks for 1 in the first partition, finds nothing. Then the first thread inserts 1 into the first partition, second thread inserts 1 into the second partition. Monty's idea was to insert optimistically - first you insert, then check for duplicates. Not ideal, in the above example you can have both threads getting a duplicate key error. There could be other, e.g. locking, solutions too. > m_last_part= part_id; > DBUG_PRINT("info", ("Insert in partition %d", part_id)); > start_part_bulk_insert(thd, part_id); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-13995 MAX(timestamp) returns a wrong result near DST change
by Alexander Barkov 04 Dec '17

04 Dec '17

Hi Sergei, Please review a patch for MDEV-13995. Thanks!

1 0

[Maria-developers] Review for MDEV-8320 Allow index usage for DATE(datetime_column) = const
by Alexander Barkov 30 Nov '17

30 Nov '17

Hello Sergey, I generally like your patch for MDEV-8320: http://lists.askmonty.org/pipermail/commits/2017-November/011680.html However, there is one problem with it. We're changing the code to have data type plugins. Every data type plugin will provide its own list of SQL functions. Those data type specific functions should be able to use the same optimization approach without having to change the server code. Therefore, everything related to YEAR(field) transformation should reside inside Item_func_year, everything related to DATE(field) transformation should reside inside Item_date_typecast. I reorganized the code to implement this idea. Note, also I changed the logic behind DATE(indexed_date_field) transformation slightly. The DATE() just gets removed. DATE(indexed_date_field) = const is changed to: indexed_date_field = const instead of: indexed_date_field BETWEEN ... AND ... Please have a look: sargable_date_cond.result is slightly different in my version. Greetings.

1 1

[Maria-developers] Delete vs multi_delete performance issue
by jerome brauge 29 Nov '17

29 Nov '17

Hi All, I've a question on kind of delete on whose we have performance issues. Consider two tables: CREATE TABLE `gapfl` ( `pgegapfl` varchar(150) COLLATE utf8_bin NOT NULL, `blkgapfl` varchar(30) COLLATE utf8_bin NOT NULL, `fldgapfl` varchar(30) COLLATE utf8_bin NOT NULL, `ordgapfl` int(11) NOT NULL, ... `stlgapfl` varchar(30) COLLATE utf8_bin DEFAULT NULL, UNIQUE KEY `gapfl_fld` (`pgegapfl`,`blkgapfl`,`fldgapfl`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin and CREATE TABLE `gapge` ( `nuugapge` varchar(150) COLLATE utf8_bin NOT NULL, `nuigapge` int(11) NOT NULL, `frmgapge` varchar(30) COLLATE utf8_bin NOT NULL, `prsgapge` varchar(60) COLLATE utf8_bin NOT NULL, `typgapge` varchar(2) COLLATE utf8_bin NOT NULL, `catgapge` varchar(1) COLLATE utf8_bin NOT NULL, `numgapge` int(11) NOT NULL, `aplgapge` varchar(8) COLLATE utf8_bin DEFAULT NULL, `tragapge` varchar(8) COLLATE utf8_bin DEFAULT NULL, ... `mafgapge` varchar(1) COLLATE utf8_bin NOT NULL, UNIQUE KEY `gapge_nuu` (`nuugapge`), UNIQUE KEY `gapge_frm` (`frmgapge`,`prsgapge`,`typgapge`,`numgapge`,`tragapge`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin Table "gapfl " contains 364740 rows Table "gapge " contains 25423 rows The following delete tooks more than 38 seconds to delete 0 rows . DELETE FROM gapfl WHERE pgegapfl IN ( SELECT nuugapge FROM gapge WHERE frmgapge = 'AMCEVA' AND prsgapge = 'QUALIAC.*.*') It explain plan is : +------+--------------------+-------+-----------------+---------------------+-----------+---------+------+--------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+--------------------+-------+-----------------+---------------------+-----------+---------+------+--------+-------------+ | 1 | PRIMARY | gapfl | ALL | NULL | NULL | NULL | NULL | 356417 | Using where | | 2 | DEPENDENT SUBQUERY | gapge | unique_subquery | gapge_nuu,gapge_frm | gapge_nuu | 452 | func | 1 | Using where | +------+--------------------+-------+-----------------+---------------------+-----------+---------+------+--------+-------------+ It's not really good and I don't understand why the subselect is consider as a "DEPENDENT SUBQUERY" (there is no correlation columns). Now, I convert this delete into a multi_delete : DELETE gapfl FROM gapfl WHERE pgegapfl IN ( SELECT nuugapge FROM gapge WHERE frmgapge = 'AMCEVA' AND prsgapge = 'QUALIAC.*.*') It tooks 0.00 second and it explain plan is : +------+-------------+-------+------+---------------------+-----------+---------+--------------------+------+--------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------------+-----------+---------+--------------------+------+--------------------------+ | 1 | PRIMARY | gapge | ref | gapge_nuu,gapge_frm | gapge_frm | 274 | const,const | 1 | Using where; Using index | | 1 | PRIMARY | gapfl | ref | gapfl_fld | gapfl_fld | 452 | dvt.gapge.nuugapge | 7 | | +------+-------------+-------+------+---------------------+-----------+---------+--------------------+------+--------------------------+ as expected. Is there any way to have the right plan with single delete ? Or do we must convert all our complex update to multi_update ? Sorry if it's not the right place to post this query. Regards.

1 0

[Maria-developers] Error 1422 - Explicit or implicit commit is not allowed in stored function or trigger
by jerome brauge 29 Nov '17

29 Nov '17

Hello Alexander, I'm facing to a new issue with my stored procedures. Some of them are called directly or from trigger. When these procedures are called directly, they manage the transaction and when they are called from trigger, they doesn't manage the transaction. This works fine on Oracle, SQLServer , ... but fail on Mariadb because the check is done in is_not_allowed_in_function() before the call. use test; set sql_mode=oracle; create or replace table t1 (c1 int); insert into t1 values (1); delimiter / create or replace procedure p1(manage_transactionnal int) is begin if manage_transactionnal = 1 then commit; end if; end; / create or replace trigger tg1 after delete on t1 for each row begin call p1(0); end; / delimiter ; start transaction; delete from t1; rollback; What do you think about this ? Regards, Jérôme.

2 2

[Maria-developers] MariaDB 10.2 now available on CentOS 6 &7 via SCL !!
by Michal Schorm 29 Nov '17

29 Nov '17

Hello everybody, In case you missed it, I just want to share an announce of availability of MariaDB 10.2 on CentOS 6 & 7 as a Software collections. Here are the announce emails from CentOS mailing list, which contains most basic howto: https://lists.centos.org/pipermail/centos-announce/2017-November/022640.html https://lists.centos.org/pipermail/centos-announce/2017-November/022642.html -- This made MariaDB 10.2 a big step closer to a new mass of users, because the standard MDB version in CentOS & RHEL is 5.5. It took quite an effort to fine-tune and perk-up the MDB 10.2 since its first attemps in Fedora, however it is always pleasure to cooperate with developers like you :) -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

1 0

[Maria-developers] Interaction between rpl_slave_state and rpl_binlog_state
by Sachin Setiya 28 Nov '17

28 Nov '17

Hi Kristian, Andrei I have some question related to rpl_slave_state. Suppose A circular async replication between A < -- > B (gtid_ignore_duplicates on) Now, we set some temp server_id on server A , lets say `X`. Now the problem is each event group which originates from A is executed 2 times. For example we insert into table t1 and gtid is 0-X-2. The event goes to slave B B applies it, And send it back to A, Since its server_is different from A global server_id , this event is not filtered in queue_event(). Now event goes to handle_sql thread , And check_duplicate_gtid function is called. Since rpl_slave_state::hash does not have a element whose sequence_no is >= 2, So this event is applied again. Andrei suggested a solution of checking rpl_binlog_state in check_duplicate_gtid, This solution solves some problem but creates some problem. (multi_source.gtid_ignore_duplicates fails in master_sync, test for mdev_10715 also fails ). Reason for this is sync_with_master calls master_gtid_wait , which internally calls gtid_waiting::wait_for_gtid , and this checks rpl_salve_state::hash. Although this method can also be patched for to look for rpl_gtid_global_binlog_state. But I am not sure whether this will solve all problem or create new problem. Another solution might be somehow update rpl_slave_state::hash when we write gtid_event in log. But this does not make sense. rpl_slave_state should be used for slave replication usage. I think we need a more better solution for this. -- Regards sachin

3 4

[Maria-developers] Add hidden level 2/3 (PSEDUO_HIDDEN_COLUMN / COMPLETELY_HIDDEN_COLUMN) to mariadb
by Sachin Setiya 27 Nov '17

27 Nov '17

Hello Everyone! Currently I am implementing 10177. But I have one doubt I am not sure how to implement a function which will create hidden level 2/3 field for server. Suppose we create a function (say add_hidden_field(.....)) which will add the field to table T. But the issue is where should we call this function in mysql_prepare_create_table / mysql_prepare_alter_table or init_from_binary_frm_image Doing it from init_from_binary_frm_image make sense because this way server and slave frm of table T will be same (We have decided that we wont log hidden level 2/3 field to binlog) , But the issue is optimizer wont be ready to decide whether we require a hidden level 3 field at that time. We can also do it from mysql_prepare_create_table , but I dont think this wont be good idea because at this time optimizer wont be ready plus this will complicate binlog handling. Or may be we should have a function which can modify table object any time. But I am not sure if it is possible/good idea ? What you guys think ? -- Regards Sachin Setiya Software Engineer at MariaDB

2 4

Re: [Maria-developers] [Commits] b2571e5: Removed ut_crc32_byte_by_byte: never used
by Marko Mäkelä 23 Nov '17

23 Nov '17

Indeed, the functions and function pointers were never used. On Thu, Nov 23, 2017 at 12:36 PM, Sergey Vojtovich <svoj(a)mariadb.org> wrote: > revision-id: b2571e534f9eed873cc9b1ce5f48b109c1ee0fcc > (mariadb-10.3.2-65-gb2571e5) > parent(s): 0bfe5e349210c01404d20dfc8660d5224bd9c658 > committer: Sergey Vojtovich > timestamp: 2017-11-23 14:35:50 +0400 > message: > > Removed ut_crc32_byte_by_byte: never used > > --- > storage/innobase/include/ut0crc32.h | 4 ---- > storage/innobase/ut/ut0crc32.cc | 48 ------------------------------ > ------- > 2 files changed, 52 deletions(-) > > diff --git a/storage/innobase/include/ut0crc32.h > b/storage/innobase/include/ut0crc32.h > index 36b389b..509ac09 100644 > --- a/storage/innobase/include/ut0crc32.h > +++ b/storage/innobase/include/ut0crc32.h > @@ -51,10 +51,6 @@ extern ut_crc32_func_t ut_crc32; > when converting byte strings to integers internally. */ > extern ut_crc32_func_t ut_crc32_legacy_big_endian; > > -/** Pointer to CRC32-byte-by-byte calculation function (byte order > agnostic, > -but very slow). */ > -extern ut_crc32_func_t ut_crc32_byte_by_byte; > - > extern const char* ut_crc32_implementation; > > #endif /* ut0crc32_h */ > diff --git a/storage/innobase/ut/ut0crc32.cc b/storage/innobase/ut/ > ut0crc32.cc > index 386d32f..6383e6d 100644 > --- a/storage/innobase/ut/ut0crc32.cc > +++ b/storage/innobase/ut/ut0crc32.cc > @@ -93,10 +93,6 @@ ut_crc32_func_t ut_crc32; > when converting byte strings to integers internally. */ > ut_crc32_func_t ut_crc32_legacy_big_endian; > > -/** Pointer to CRC32-byte-by-byte calculation function (byte order > agnostic, > -but very slow). */ > -ut_crc32_func_t ut_crc32_byte_by_byte; > - > /** Text description of CRC32 implementation */ > const char* ut_crc32_implementation; > > @@ -409,26 +405,6 @@ ut_crc32_legacy_big_endian_hw( > > return(~crc); > } > - > -/** Calculates CRC32 using hardware/CPU instructions. > -This function processes one byte at a time (very slow) and thus it does > -not depend on the byte order of the machine. > -@param[in] buf data over which to calculate CRC32 > -@param[in] len data length > -@return CRC-32C (polynomial 0x11EDC6F41) */ > -uint32_t > -ut_crc32_byte_by_byte_hw( > - const byte* buf, > - ulint len) > -{ > - uint32_t crc = 0xFFFFFFFFU; > - > - while (len > 0) { > - ut_crc32_8_hw(&crc, &buf, &len); > - } > - > - return(~crc); > -} > #endif /* defined(__GNUC__) && defined(__x86_64__) */ > > /* CRC32 software implementation. */ > @@ -669,28 +645,6 @@ ut_crc32_legacy_big_endian_sw( > return(~crc); > } > > -/** Calculates CRC32 in software, without using CPU instructions. > -This function processes one byte at a time (very slow) and thus it does > -not depend on the byte order of the machine. > -@param[in] buf data over which to calculate CRC32 > -@param[in] len data length > -@return CRC-32C (polynomial 0x11EDC6F41) */ > -uint32_t > -ut_crc32_byte_by_byte_sw( > - const byte* buf, > - ulint len) > -{ > - uint32_t crc = 0xFFFFFFFFU; > - > - ut_a(ut_crc32_slice8_table_initialized); > - > - while (len > 0) { > - ut_crc32_8_sw(&crc, &buf, &len); > - } > - > - return(~crc); > -} > - > /********************************************************************//** > Initializes the data structures used by ut_crc32*(). Does not do any > allocations, would not hurt if called twice, but would be pointless. */ > @@ -701,7 +655,6 @@ ut_crc32_init() > ut_crc32_slice8_table_init(); > ut_crc32 = ut_crc32_sw; > ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_sw; > - ut_crc32_byte_by_byte = ut_crc32_byte_by_byte_sw; > ut_crc32_implementation = "Using generic crc32 instructions"; > > #if defined(__GNUC__) && defined(__x86_64__) > @@ -735,7 +688,6 @@ ut_crc32_init() > if (features_ecx & 1 << 20) { > ut_crc32 = ut_crc32_hw; > ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_hw; > - ut_crc32_byte_by_byte = ut_crc32_byte_by_byte_hw; > ut_crc32_implementation = "Using SSE2 crc32 instructions"; > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/

1 0

Re: [Maria-developers] [Commits] f28af02: Removed HW acceleration for big endian checksum
by Marko Mäkelä 23 Nov '17

23 Nov '17

Looks OK to me. This can only slightly punish the very few people who are upgrading from a buggy MySQL 5.6 or MariaDB 10.0 or 10.1 installation on a big-endian machine (such as SPARC) into a newer version. And those users could invoke innochecksum to rewrite the page checksums. On Thu, Nov 23, 2017 at 12:57 PM, Sergey Vojtovich <svoj(a)mariadb.org> wrote: > revision-id: f28af027a0439d760c6d26eb67a932e97eb665d7 > (mariadb-10.3.2-66-gf28af02) > parent(s): b2571e534f9eed873cc9b1ce5f48b109c1ee0fcc > committer: Sergey Vojtovich > timestamp: 2017-11-23 14:53:16 +0400 > message: > > Removed HW acceleration for big endian checksum > > Big endian checksum is needed to workaround 10+ years old bug, where > checksum > was calculated incorrectly on big endian hardware. We can still checksum > such > tablespaces using software implementation of CRC32. > > --- > storage/innobase/include/ut0crc32.h | 4 +- > storage/innobase/ut/ut0crc32.cc | 89 +----------------------------- > ------- > 2 files changed, 3 insertions(+), 90 deletions(-) > > diff --git a/storage/innobase/include/ut0crc32.h > b/storage/innobase/include/ut0crc32.h > index 509ac09..527a3e81 100644 > --- a/storage/innobase/include/ut0crc32.h > +++ b/storage/innobase/include/ut0crc32.h > @@ -47,9 +47,9 @@ typedef uint32_t (*ut_crc32_func_t)(const byte* > ptr, ulint len); > /** Pointer to CRC32 calculation function. */ > extern ut_crc32_func_t ut_crc32; > > -/** Pointer to CRC32 calculation function, which uses big-endian byte > order > +/** CRC32 calculation function, which uses big-endian byte order > when converting byte strings to integers internally. */ > -extern ut_crc32_func_t ut_crc32_legacy_big_endian; > +extern uint32_t ut_crc32_legacy_big_endian(const byte* buf, ulint len); > > extern const char* ut_crc32_implementation; > > diff --git a/storage/innobase/ut/ut0crc32.cc b/storage/innobase/ut/ > ut0crc32.cc > index 6383e6d..36c52b7 100644 > --- a/storage/innobase/ut/ut0crc32.cc > +++ b/storage/innobase/ut/ut0crc32.cc > @@ -89,10 +89,6 @@ mysys/my_perf.c, contributed by Facebook under the > following license. > /** Pointer to CRC32 calculation function. */ > ut_crc32_func_t ut_crc32; > > -/** Pointer to CRC32 calculation function, which uses big-endian byte > order > -when converting byte strings to integers internally. */ > -ut_crc32_func_t ut_crc32_legacy_big_endian; > - > /** Text description of CRC32 implementation */ > const char* ut_crc32_implementation; > > @@ -238,37 +234,6 @@ ut_crc32_64_hw( > *len -= 8; > } > > -/** Calculate CRC32 over 64-bit byte string using a hardware/CPU > instruction. > -The byte string is converted to a 64-bit integer using big endian byte > order. > -@param[in,out] crc crc32 checksum so far when this function is called, > -when the function ends it will contain the new checksum > -@param[in,out] data data to be checksummed, the pointer will be > advanced > -with 8 bytes > -@param[in,out] len remaining bytes, it will be decremented with 8 */ > -inline > -void > -ut_crc32_64_legacy_big_endian_hw( > - uint32_t* crc, > - const byte** data, > - ulint* len) > -{ > - uint64_t data_int = *reinterpret_cast<const > uint64_t*>(*data); > - > -#ifndef WORDS_BIGENDIAN > - data_int = ut_crc32_swap_byteorder(data_int); > -#else > - /* Currently we only support x86_64 (little endian) CPUs. In case > - some big endian CPU supports a CRC32 instruction, then maybe we > will > - NOT need a byte order swap here. */ > -#error Dont know how to handle big endian CPUs > -#endif /* WORDS_BIGENDIAN */ > - > - *crc = ut_crc32_64_low_hw(*crc, data_int); > - > - *data += 8; > - *len -= 8; > -} > - > /** Calculates CRC32 using hardware/CPU instructions. > @param[in] buf data over which to calculate CRC32 > @param[in] len data length > @@ -355,56 +320,6 @@ ut_crc32_hw( > > return(~crc); > } > - > -/** Calculates CRC32 using hardware/CPU instructions. > -This function uses big endian byte ordering when converting byte sequence > to > -integers. > -@param[in] buf data over which to calculate CRC32 > -@param[in] len data length > -@return CRC-32C (polynomial 0x11EDC6F41) */ > -uint32_t > -ut_crc32_legacy_big_endian_hw( > - const byte* buf, > - ulint len) > -{ > - uint32_t crc = 0xFFFFFFFFU; > - > - /* Calculate byte-by-byte up to an 8-byte aligned address. After > - this consume the input 8-bytes at a time. */ > - while (len > 0 && (reinterpret_cast<uintptr_t>(buf) & 7) != 0) { > - ut_crc32_8_hw(&crc, &buf, &len); > - } > - > - while (len >= 128) { > - /* This call is repeated 16 times. 16 * 8 = 128. */ > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - } > - > - while (len >= 8) { > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - } > - > - while (len > 0) { > - ut_crc32_8_hw(&crc, &buf, &len); > - } > - > - return(~crc); > -} > #endif /* defined(__GNUC__) && defined(__x86_64__) */ > > /* CRC32 software implementation. */ > @@ -600,7 +515,7 @@ integers. > @param[in] len data length > @return CRC-32C (polynomial 0x11EDC6F41) */ > uint32_t > -ut_crc32_legacy_big_endian_sw( > +ut_crc32_legacy_big_endian( > const byte* buf, > ulint len) > { > @@ -654,7 +569,6 @@ ut_crc32_init() > { > ut_crc32_slice8_table_init(); > ut_crc32 = ut_crc32_sw; > - ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_sw; > ut_crc32_implementation = "Using generic crc32 instructions"; > > #if defined(__GNUC__) && defined(__x86_64__) > @@ -687,7 +601,6 @@ ut_crc32_init() > > if (features_ecx & 1 << 20) { > ut_crc32 = ut_crc32_hw; > - ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_hw; > ut_crc32_implementation = "Using SSE2 crc32 instructions"; > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/

1 0

Re: [Maria-developers] [Commits] 14bd406: Less code to depend on ut_crc32_init()
by Marko Mäkelä 23 Nov '17

23 Nov '17

Thanks, this looks OK to me. I guess the function ut_crc32_init() will be made optional separately. On Thu, Nov 23, 2017 at 1:22 PM, Sergey Vojtovich <svoj(a)mariadb.org> wrote: > revision-id: 14bd406970c9bb18c744a452359aa4505f325b2d > (mariadb-10.3.2-67-g14bd406) > parent(s): f28af027a0439d760c6d26eb67a932e97eb665d7 > committer: Sergey Vojtovich > timestamp: 2017-11-23 15:22:10 +0400 > message: > > Less code to depend on ut_crc32_init() > > --- > storage/innobase/include/ut0crc32.h | 1 + > storage/innobase/ut/ut0crc32.cc | 20 +++++++------------- > 2 files changed, 8 insertions(+), 13 deletions(-) > > diff --git a/storage/innobase/include/ut0crc32.h > b/storage/innobase/include/ut0crc32.h > index 527a3e81..32ad066 100644 > --- a/storage/innobase/include/ut0crc32.h > +++ b/storage/innobase/include/ut0crc32.h > @@ -51,6 +51,7 @@ extern ut_crc32_func_t ut_crc32; > when converting byte strings to integers internally. */ > extern uint32_t ut_crc32_legacy_big_endian(const byte* buf, ulint len); > > +/** Text description of CRC32 implementation */ > extern const char* ut_crc32_implementation; > > #endif /* ut0crc32_h */ > diff --git a/storage/innobase/ut/ut0crc32.cc b/storage/innobase/ut/ > ut0crc32.cc > index 36c52b7..48f6b82 100644 > --- a/storage/innobase/ut/ut0crc32.cc > +++ b/storage/innobase/ut/ut0crc32.cc > @@ -86,12 +86,6 @@ mysys/my_perf.c, contributed by Facebook under the > following license. > #include "univ.i" > #include "ut0crc32.h" > > -/** Pointer to CRC32 calculation function. */ > -ut_crc32_func_t ut_crc32; > - > -/** Text description of CRC32 implementation */ > -const char* ut_crc32_implementation; > - > /** Swap the byte order of an 8 byte integer. > @param[in] i 8-byte integer > @return 8-byte integer */ > @@ -125,6 +119,13 @@ ut_crc32_power8( > { > return crc32c_vpmsum(0, buf, len); > } > + > +ut_crc32_func_t ut_crc32 = ut_crc32_power8; > +const char* ut_crc32_implementation = "Using POWER8 crc32 > instructions"; > +#else > +uint32_t ut_crc32_sw(const byte* buf, ulint len); > +ut_crc32_func_t ut_crc32 = ut_crc32_sw; > +const char* ut_crc32_implementation = "Using generic crc32 > instructions"; > #endif > > #if defined(__GNUC__) && defined(__x86_64__) > @@ -568,8 +569,6 @@ ut_crc32_init() > /*===========*/ > { > ut_crc32_slice8_table_init(); > - ut_crc32 = ut_crc32_sw; > - ut_crc32_implementation = "Using generic crc32 instructions"; > > #if defined(__GNUC__) && defined(__x86_64__) > uint32_t vend[3]; > @@ -603,10 +602,5 @@ ut_crc32_init() > ut_crc32 = ut_crc32_hw; > ut_crc32_implementation = "Using SSE2 crc32 instructions"; > } > - > -#elif defined(HAVE_CRC32_VPMSUM) > - ut_crc32 = ut_crc32_power8; > - ut_crc32_implementation = "Using POWER8 crc32 instructions"; > #endif > - > } > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/

1 0

[Maria-developers] IB: trx_t:: start_time and start_time_micro
by Aleksey Midenkov 21 Nov '17

21 Nov '17

Hello! Different semantics of these 2 members doesn't look good: time_t start_time; /*!< time the state last time became TRX_STATE_ACTIVE */ ib_uint64_t start_time_micro; /*!< start time of transaction in microseconds */ Either `start_time_micro` should be drawn to the base of `start_time` or it should be explicitly commented that it has nothing to do with `start_time`. I.e. there is nothing wrong on algorithmic level, but such data members may (and certainly would) lead developers to misuse of `start_time_micro`. -- All the best, Aleksey Midenkov @midenok

2 2

[Maria-developers] Please review MDEV-11292 Item_func_sp::fix_length_and_dec() does not set collation derivation properly
by Alexander Barkov 20 Nov '17

20 Nov '17

Hello Alexey, Please review a patch for MDEV-11292 (for 10.3). Thanks.

2 2

[Maria-developers] MariaDB GUI: BSL 1.1 - Debate
by Juan Telleria 19 Nov '17

19 Nov '17

Dear MariaDb Developers, I am writing because I would like to open a debate whether if MariaDB, now that has gained investment from Microsoft, Alibaba, and the European Union, shall create its own SQL Graphical User Interface, as good as (or better than) MySQL Workbench. I agree that MariaDB includes by default HeidiSQL in its Windows builds, but in my opinion, if MariaDB really wants to look Oracle's MySQL face to face, HeidiSQL is not nearly as good as MySQL Workbench. If this wants to be done: 1st) The brand new GUI shall have a Business Source Licence v1.1, which only companies which invest in MariaDB would be able to use. 2nd) In 3 years now, MariaDB would be better positioned and have a fully Open Source GUI which would allow MariaDB to look face to face to Oracle. 3rd) And in 4/5 years now, MariaDB Corporation shall be brave enough to to continue Developing this GUI as a truly open source product, without the BSL, and adopting a purely GPLv2 with incremental improvements, distributing it through the MariaDB Foundation Webpage, as individuals or other companies apart from MariaDB Corporation, will not support the product if it is not totally Open Source. So BSL v1.1 might prove useful to launch the project, but with time, it shall have a GPLv2 for gaining investment from third companies. ¿What do you thing on the topic? Thank you all, Juan Telleria

1 0

[Maria-developers] Please review MDEV-13788 Server crash when issuing bad SQL partition syntax
by Alexander Barkov 16 Nov '17

16 Nov '17

Hello Sergei, Can you please review a patch for MDEV-13788, for 5.5? Thanks!

2 3

Re: [Maria-developers] MDEV-10173.
by Sergei Golubchik 15 Nov '17

15 Nov '17

Hi, Alexey! On Nov 14, Alexey Botchkov wrote: > Hi, Serg. > > I'd like your opinion about this issue. > Georg complains about weird column metadata > https://jira.mariadb.org/browse/MDEV-10173 > > Personally i see slightly different result, but basically same thing. > -------------------------------- > MariaDB [(none)]> select from_unixtime("0"); > Field 1: `from_unixtime("0")` > Catalog: `def` > Database: `` > Table: `` > Org_table: `` > Type: DATETIME > Collation: binary (63) > Length: 26 > Max_length: 19 > Decimals: 39 > Flags: BINARY Right, I get used to 31 being NOT_FIXED_DEC. But recently Monty changed it to 39. > +---------------------+ > | from_unixtime("0") | > +---------------------+ > | 1970-01-01 04:00:00 | > +---------------------+ > 1 row in set (0.00 sec) > -------------------------------- > > From what i see it is not a bug. The 'Decimals: 39' is not a real > precision here, but just a hint to be flexible converting the data to > the string. The 'Length: 26' takes into account the possible > 'second_part' of 6 digits. Agree. > Georg said on this: "Well, the client "trusts" the metadata. If there > is decimal 31, it appends 31 decimal places and returned truncation > error the metadata (MYSQL_FIELD) describes the data which are sent > from server to client. If there are no decimal places, it should be > zero. " Yes, the cient trusts the metadata, but if decimals is NOT_FIXED_DEC it means "unknown", not 31 (or 39). If the number of decimals for a timestamp is unknown, the max length is 26: "YYYY-MM-DD hh:mm:ss.uuuuuu" > As an improvement i can see using '7' instead of '39' as an > 'NOT_FIXED_DEC' marker for the Item_temporal_func. Why? One wouldn't be able simply to compare with NOT_FIXED_DEC, it'll be type dependent. > Or we can just document that '39' value so clients interpret it right. Isn't it documented? > Serg, maybe you can suggest something? Close as "Not a bug" :) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 730a22c: Fixed bug mdev-13453 Executing a query via CTE requires more permissions
by Oleksandr Byelkin 14 Nov '17

14 Nov '17

Am 14.11.2017 um 01:06 schrieb IgorBabaev: > revision-id: 730a22c1ce1010e853e7a3e807d1d07409ec4dfc (mariadb-10.2.10-28-g730a22c) > parent(s): 1e2d4f677e08294691a7d218acb3b9f78727ae18 > author: Igor Babaev > committer: Igor Babaev > timestamp: 2017-11-13 16:06:04 -0800 > message: > > Fixed bug mdev-13453 Executing a query via CTE requires more permissions > than the query itself > > ACL checks were not properly supported for tables used in CTE > specifications. This patch fixes the problem. > > --- > mysql-test/r/cte_nonrecursive.result | 58 ++++++++++++++++++++++++++++++++++++ > mysql-test/t/cte_nonrecursive.test | 51 +++++++++++++++++++++++++++++++ > sql/sql_acl.cc | 4 +++ > sql/sql_cte.cc | 3 +- > sql/sql_parse.cc | 4 +++ > 5 files changed, 119 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/cte_nonrecursive.result b/mysql-test/r/cte_nonrecursive.result > index ebe1aae..92f9151 100644 > --- a/mysql-test/r/cte_nonrecursive.result > +++ b/mysql-test/r/cte_nonrecursive.result > @@ -1147,3 +1147,61 @@ SELECT * FROM cte_test; > a > 1 > DROP VIEW cte_test; > +# > +# MDEV-13453: > +# [skip] > --- a/mysql-test/t/cte_nonrecursive.test > +++ b/mysql-test/t/cte_nonrecursive.test > @@ -790,3 +790,54 @@ SHOW CREATE VIEW cte_test; > SELECT * FROM cte_test; > > DROP VIEW cte_test; > + > +--echo # > +--echo # MDEV-13453: privileges checking for CTE > +--echo # > + This is for sure do not match results above, so please fix it and make sure that full test suite passed. [skip] > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -7557,6 +7557,10 @@ bool check_grant(THD *thd, ulong want_access, TABLE_LIST *tables, > tl->correspondent_table ? tl->correspondent_table : tl; > sctx= t_ref->security_ctx ? t_ref->security_ctx : thd->security_ctx; > > + if (tl->with || > + (tl->with= tl->select_lex->find_table_def_in_with_clauses(tl))) > + continue; > + > const ACL_internal_table_access *access= > get_cached_table_access(&t_ref->grant.m_internal, > t_ref->get_db_name(), > diff --git a/sql/sql_cte.cc b/sql/sql_cte.cc > index 6fe08e3..e1bd455 100644 > --- a/sql/sql_cte.cc > +++ b/sql/sql_cte.cc > @@ -823,9 +823,10 @@ st_select_lex_unit *With_element::clone_parsed_spec(THD *thd, > tbl; > tbl= tbl->next_global) > { > - tbl->grant.privilege= with_table->grant.privilege; > spec_tables_tail= tbl; > } > + if (check_table_access(thd, SELECT_ACL, spec_tables, FALSE, UINT_MAX, FALSE)) > + goto err; > if (spec_tables) > { > if (with_table->next_global) > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index bf5144b..39a4da9 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -3443,6 +3443,10 @@ mysql_execute_command(THD *thd) > ulong privileges_requested= lex->exchange ? SELECT_ACL | FILE_ACL : > SELECT_ACL; > > + res= check_dependencies_in_with_clauses(thd->lex->with_clauses_list); > + if (res) > + break; Above works only for SELECT, do we support CTE in subselects which can be put everywhere? > + > if (all_tables) > res= check_table_access(thd, > privileges_requested, > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] Handle failures from malloc
by Sergey Petrunia 14 Nov '17

14 Nov '17

Hi Monty, Please find my comments below. Anything not commented on seems to be ok to me. > commit 7d9c24f6790c66d2f457837661852bf5e870b605 > Author: Michael Widenius <monty(a)mariadb.org> > Date: Tue Nov 14 07:47:58 2017 +0200 > > Handle failures from malloc > > Most "new" failures fixed in the following files: > - sql_select.cc > - item.cc > - opt_subselect.cc > > diff --git a/sql/item.cc b/sql/item.cc > index 2285ae28041..baae2a9b6fb 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -1261,7 +1261,7 @@ Item *Item::safe_charset_converter(THD *thd, CHARSET_INFO *tocs) > if (!needs_charset_converter(tocs)) > return this; > Item_func_conv_charset *conv= new (thd->mem_root) Item_func_conv_charset(thd, this, tocs, 1); > - return conv->safe ? conv : NULL; > + return conv && conv->safe ? conv : NULL; > } > I am not fully convinced that returning NULL from this function is called everywhere. It is of course better to return NULL and hope for the best than to crash right away. > @@ -3296,6 +3301,8 @@ void Item_field::fix_after_pullout(st_select_lex *new_parent, Item **ref, > } > > Name_resolution_context *ctx= new Name_resolution_context(); > + if (ctx) > + return; // Fatal error set > if (context->select_lex == new_parent) > { > /* I assume this is a typo it should be "if (!ctx)" > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index abbf2616537..7e42890143b 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -713,7 +713,7 @@ JOIN::prepare(TABLE_LIST *tables_init, > union_part= unit_arg->is_unit_op(); > > if (select_lex->handle_derived(thd->lex, DT_PREPARE)) > - DBUG_RETURN(1); > + DBUG_RETURN(-1); > > thd->lex->current_select->context_analysis_place= NO_MATTER; > thd->lex->current_select->is_item_list_lookup= 1; I assume this is just for the sake of consistency with other DBUG_RETURN statements in this function? > @@ -14422,7 +14446,7 @@ static COND* substitute_for_best_equal_field(THD *thd, JOIN_TAB *context_tab, > This works OK with PS/SP re-execution as changes are made to > the arguments of AND/OR items only > */ > - if (new_item != item) > + if (new_item && new_item != item) > li.replace(new_item); > } new_item was returned from a substitute_for_best_equal_field() call. That one doesn't ever return NULL. It will return without substitution if it encounters an error. So this change is not needed. > @@ -25044,8 +25103,11 @@ int JOIN::save_explain_data_intern(Explain_query *output, > > if (message) > { > - explain= new (output->mem_root) Explain_select(output->mem_root, > - thd->lex->analyze_stmt); > + if (!(explain= new (output->mem_root) > + Explain_select(output->mem_root, > + thd->lex->analyze_stmt))) > + DBUG_RETURN(1); > + > join->select_lex->set_explain_type(true); > > explain->select_id= join->select_lex->select_number; The return value of this function is not checked. I assume the above statement is there so that we dont try to continue to execute the function if we've got an OOM error? In this case, probably JOIN::save_explain_data_intern() should also have a similar check for its tab->save_explain_data(eta, used_tables, distinct_arg, first_top_tab); call? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] STRICT mode and Cast
by jerome brauge 14 Nov '17

14 Nov '17

Hello Alexander, To have a behavior the nearest possible to Oracle, I use STRICT_TRANS_TABLES sql_mode. But then I have a problem with cast to varchar that failed with "truncated incorrect value" . Ex: set sql_mode='ORACLE,STRICT_TRANS_TABLES'; delimiter / CREATE or replace procedure p3(b1 integer) AS strres VARCHAR(4); BEGIN strres:=cast('123456' as char(4)); END / call p3(-1) / -- Failed with : ERROR 1292 (22007): Truncated incorrect CHAR(4) value: '123456' But select cast('11111' as varchar(2)) from dual; just issue a warning, not an error. On Oracle, Sybase,SQLServer and DB2, cast a string to a shorter string never issue warnings nor error. What do you think about ? Regards, Jérôme.

2 4

[Maria-developers] Patch for MDEV-10574
by jerome brauge 13 Nov '17

13 Nov '17

Hello Alexander, I've begun to work on new sql_mode EMPTY_STRING_IS_NULL_FUNCTIONS. Can you take a glance on this patch to known if I'm on the right direction ? I've also found a different behavior between Oracle and Mariadb on cast of string in a numeric format. Mariadb send a warning or a note and return 0 when conversion failed while Oracle send an error. MariaDB [test]> select cast(' ' as integer) from dual; +----------------------+ | cast(' ' as integer) | +----------------------+ | 0 | +----------------------+ DVTORA> select cast(' ' as integer) from dual; select cast(' ' as integer) from dual * ERROR at line 1: ORA-01722: invalid number Are "cast" and aggregate functions to be handled by this mdev? Regards, Jérôme. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : samedi 7 octobre 2017 11:44 > À : jerome brauge > Objet : Re: Patch for MDEV-10574 > > Hello Jerome, > > > On 10/06/2017 01:41 PM, jerome brauge wrote: > > Hello Alexander, > > > >> -----Message d'origine----- > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : jeudi 5 > >> octobre 2017 21:26 À : jerome brauge Objet : Re: Patch for MDEV-10574 > >> > >> Hello Jerome, > >> > >> > >> On 10/05/2017 12:45 PM, jerome brauge wrote: > >>> Hello Alexander, > >>> Have you gone ahead on your reflection about nulls and empty strings ? > >> > >> Yes, I think so. > >> > >> One addition: Monty suggested to rename flags to: > >> - EMPTY_STRING_IS_NULL for literals > >> - EMPTY_STRING_IS_NULL_RESULTS for functions. > >> > >> I'm fine with EMPTY_STRING_IS_NULL for literals, but think that > >> EMPTY_STRING_IS_NULL_FUNCTIONS should be more clear than > >> EMPTY_STRING_IS_NULL_RESULTS. > >> Which name do you prefer for functions? > > > > I think like you. > > For me, EMPTY_STRING_IS_NULL_RESULTS sounds like including > RESULTSET > > of select (which is perhaps included in point 4) > > Okey, let's go with _FUNCTIONS then. > > > > >>> > >>> Can I > >>> - redo my pull request for substring_oracle function > >>> - make a patch for first new sql_mode > >> (MODE_EMPTY_AS_NULL_LITERALS) > >> > >> > >> Yes, please. Let's move forward. > >> > >> Let's push the patch for substr_oracle first. > >> Let's keep it return empty strings for the cases when the > >> substring_length parameters is less than 1. > >> It will start automatically returning NULL when we add the new > >> sql_mode for functions. > > > > Pull request is done. > > I have merged it. Thanks! > > > > >> Also, let's go ahead with the part for literals. > > > > Ok, I will make a separate patch. > > Many thanks. > > > >> > >> > >> I have created MDEVs. Please use them in the commit comment and in > >> the > >> tests: > >> > >> MDEV-14012 sql_mode=Oracle: substr(): treat position 0 as position 1 > >> MDEV-14013 sql_mode=EMPTY_STRING_IS_NULL > >> > >> > >> Thanks! > >> > >>> > >>> Regards, > >>> Jérôme. > >>> > >>> > >>>> -----Message d'origine----- > >>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mercredi 27 > >>>> septembre 2017 08:50 À : jerome brauge Objet : Re: Patch for > >>>> MDEV-10574 > >>>> > >>>> Hello Jerome, > >>>> > >>>> I told to Monty yesterday. > >>>> > >>>> We were going to implement the following transformations: > >>>> > >>>>> We are considering to implement both of the following options > >>>>> (with a > >>>>> switch) > >>>>> > >>>>> Transform Insert > >>>>> - insert values ("") -> insert values (null) > >>>>> > >>>>> Transform Select > >>>>> > >>>>> where v=x => (v <> "" and V=X) > >>>>> where v is null => (v="" or v is null) > >>>>> > >>>> > >>>> See MDEV-10574 for details. > >>>> > >>>> Monty did not fully understand the idea of translating literals and > >> functions > >>>> from empty strings to null. He asked if you can explain use cases > >>>> for these transformations. > >>>> > >>>> But we agreed that: > >>>> > >>>> 1. It's OK to have multiple NULL handling flags in sql_mode. > >>>> > >>>> 2. New flags will not be enabled in 10.3, neither in > >>>> sql_mode=DEFAULT, > >> nor in > >>>> sql_mode=ORACLE. So one will have to specify them explicitly: > >>>> set > >>>> > >> > sql_mode='ORACLE,MODE_null_transfrormation_flag1,MODE_null_transfor > >>>> mation_flag2'; > >>>> > >>>> > >>>> > >>>> So I propose we introduce the following flags: > >>>> > >>>> > >>>> 1. MODE_EMPTY_AS_NULL_LITERALS - for literals and PS parameters, > as > >> in > >>>> your patch. > >>>> > >>>> 2. MODE_EMPTY_AS_NULL_FUNCTIONS - for functions. > >>>> This should be done as a separate patch. > >>>> Note, your patch currently implements empty-to-null translation > >>>> only for Item_str_func descendants that use make_empty_result(). > >>>> This patch should be extended to cover Item_str_func descendants > >>>> which do not use make_empty_result(), as well as > >>>> all other Item_func_or_sum descendants > >>>> (which are not Item_str_func descendants). > >>>> > >>>> > >>>> 3. MODE_EMPTY_AS_NULL_UPDATES > >>>> > >>>> For INSERT and UPDATE transformation, as in MDEV: > >>>> > >>>> insert values ("") -> insert values (null) > >>>> update set a='' -> update set a=null; > >>>> > >>>> > >>>> 4. MODE_EMPTY_AS_NULL_PREDICATES > >>>> > >>>> For predicate transformation, as in MDEV: > >>>> > >>>> where v=x => (v <> "" and V=X) > >>>> where v is null => (v="" or v is null) > >>>> > >>>> > >>>> What do you think about this proposal with multiple flags? > >>>> Note, we can do N1 and N2 now, and add N3 and N4 and later. > >>>> > >>>> Also, can you please write a description why you need N1 and N2. > >>>> Monty thinks that N3 and N4 should cover most use cases. > >>>> So we need a good rationale for Monty and Sergei explaining why we > >> adding > >>>> these flags N1 and N2. > >>>> > >>>> > >>>> Thanks! > >>>> > >>>> > >>>> On 09/26/2017 04:33 PM, jerome brauge wrote: > >>>>> Hi Alexander, > >>>>> I'm disappointed. > >>>>> Can we at least keep this part in sql_prepare : > >>>>> > >>>>> /* > >>>>> set_param_str_oracle : convert empty string to null value > >>>>> This allow to bind empty string in JDBC as a null value. > >>>>> Ex : > >>>>> String sel="select coalesce(?,'null param') from dual"; > >>>>> PreparedStatement ps_sel = m_cnx.prepareStatement(sel); > >>>>> ps_sel.setString(1, ""); > >>>>> ResultSet rs = ps_sel.executeQuery(); > >>>>> while (rs.next()) > >>>>> { > >>>>> System.out.println(rs.getString(1)); > >>>>> } > >>>>> Returns : 'null param' > >>>>> */ > >>>>> static void set_param_str_oracle(Item_param *param, uchar **pos, > >> ulong > >>>>> len) { > >>>>> ulong length= get_param_length(pos, len); > >>>>> if (length == 0) > >>>>> param->set_null(); > >>>>> else > >>>>> { > >>>>> if (length > len) > >>>>> length= len; > >>>>> param->set_str((const char *) *pos, length); > >>>>> *pos+= length; > >>>>> } > >>>>> } > >>>>> > >>>>> Regards, > >>>>> Jérôme. > >>>>> > >>>>> > >>>>>> -----Message d'origine----- > >>>>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mardi 26 > >>>>>> septembre 2017 13:34 À : jerome brauge Objet : Re: Patch for > >>>>>> MDEV-10574 > >>>>>> > >>>>>> Hello Jérôme, > >>>>>> > >>>>>> I'm afraid we cannot accept this change at this point of time. > >>>>>> > >>>>>> The problem is that all around the code we assume that NULL is > >>>>>> not equal to ''. > >>>>>> > >>>>>> Changing '' to NULL only in the visible part of the iceberg (such > >>>>>> as in the parser and in Item_str_func::make_empty_result()) > >>>>>> will likely introduce various anomalies in fundamental things > >>>>>> like joins execution, optimizer, equality propagation, etc. > >>>>>> > >>>>>> Sorry, we don't have resources to dive into this topic deeper now. > >>>>>> Moreover, Oracle's documentation suggests not to reply on the > >>>>>> fact > >> that '' > >>>>>> and NULL are treated as same, as this can change in the future. > >>>>>> > >>>>>> > >>>>>> > >>>>>> On 09/20/2017 03:52 PM, jerome brauge wrote: > >>>>>>> Hello Alexander, > >>>>>>> Here is a new version of this patch (minor optimizations) > >>>>>>> > >>>>>>> Regards, > >>>>>>> Jérôme. > >>>>>>> > >>>>>>> > >>>>>>>> -----Message d'origine----- > >>>>>>>> De : Maria-developers [mailto:maria-developers- > >>>>>>>> bounces+j.brauge=qualiac.com(a)lists.launchpad.net] De la part > de > >>>>>>>> bounces+jerome > >>>>>>>> brauge > >>>>>>>> Envoyé : mardi 19 septembre 2017 16:09 À : 'Alexander Barkov' > >>>>>>>> Cc : MariaDB Developers (maria-developers(a)lists.launchpad.net) > >>>>>>>> Objet : [Maria-developers] Patch for MDEV-10574 > >>>>>>>> > >>>>>>>> Alexander, > >>>>>>>> Here is the patch I was talking about. > >>>>>>>> It not address all aspects of MDEV-10574, especially, empty > >>>>>>>> strings in table column are not view as null. > >>>>>>>> But as in oracle mode, no empty string in columns should be > >>>>>>>> created, an fully oracle application will never fall in this case. > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> Jérôme. > >>>>>>>> > >>>>>>>> > >>>>>>>>> -----Message d'origine----- > >>>>>>>>> De : jerome brauge > >>>>>>>>> Envoyé : mardi 19 septembre 2017 14:13 À : 'Alexander Barkov' > >>>>>>>>> Objet : RE: Oracle strings functions compatibility : substr > >>>>>>>>> > >>>>>>>>> Hello Alexander, > >>>>>>>>> Yes I'm aware. > >>>>>>>>> This point will be treated by my next patch which will also > >>>>>>>>> affect the others string functions (a part of MDEV-10574). > >>>>>>>>> I will refresh this new patch with the current development > >>>>>>>>> branch, and I submit to you. > >>>>>>>>> Regards. > >>>>>>>>> Jérôme. > >>>>>>>>> > >>>>>>>>>> -----Message d'origine----- > >>>>>>>>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : > mardi > >> 19 > >>>>>>>>>> septembre 2017 13:16 À : jerome brauge Objet : Re: Oracle > >>>>>>>>>> strings functions compatibility : substr > >>>>>>>>>> > >>>>>>>>>> Hi Jerome, > >>>>>>>>>> > >>>>>>>>>> I noticed one more difference in SUBSTR: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> SELECT SUBSTR('aaa',1,-1) FROM DUAL; > >>>>>>>>>> > >>>>>>>>>> MariaDB returns an empty string. > >>>>>>>>>> Oracle returns NULL. > >>>>>>>>>> > >>>>>>>>>> Here's Oracle's documentation: > >>>>>>>>>> > >>>>>>>>>> > >>>>>> > >> https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions162. > >>>>>>>>>> ht > >>>>>>>>>> m > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> I think we should do this in the same patch, to avoid > >>>>>>>>>> behavior change in the future. > >>>>>>>>>> > >>>>>>>>>> Would you like to try doing this additional change? > >>>>>>>>>> Or would you like me to make an incremental patch on top of > >> yours? > >>>>>>>>>> > >>>>>>>>>> Thanks! > >>>>>>>>>> > >>>>>>>>>> On 09/19/2017 10:33 AM, jerome brauge wrote: > >>>>>>>>>>> Hello Alexander, > >>>>>>>>>>> It's done. > >>>>>>>>>>> > >>>>>>>>>>> Do you have news from Sergei for MDEV-12874, MDEV-13417 > >> and > >>>>>>>> MDEV- > >>>>>>>>>> 13418 ? > >>>>>>>>>> > >>>>>>>>>> I asked Sergei, awaiting for his answer. I'll let you know. > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Thank you very much. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> -----Message d'origine----- De : Alexander Barkov > >>>>>>>>>>>> [mailto:bar@mariadb.org] Envoyé : lundi > >>>>>>>>>>>> 18 septembre 2017 18:12 À : jerome brauge Objet : Re: > >>>>>>>>>>>> Oracle strings functions compatibility : substr > >>>>>>>>>>>> > >>>>>>>>>>>> Hello Jerome, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On 09/18/2017 05:14 PM, jerome brauge wrote: > >>>>>>>>>>>>> Hello Alexander, > >>>>>>>>>>>>> I don't understand how I've missed such an obvious > solution ! > >>>>>>>>>>>>> Here is the new patch. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks. Now it looks simpler. > >>>>>>>>>>>> > >>>>>>>>>>>> Can you please do one small thing: > >>>>>>>>>>>> move get_position() from "public:" to "protected:". > >>>>>>>>>>>> > >>>>>>>>>>>> Ok to make a pull request after this change. > >>>>>>>>>>>> > >>>>>>>>>>>> Thank you very much! > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Regards. > >>>>>>>>>>>>> Jérôme. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> -----Message d'origine----- De : Alexander Barkov > >>>>>>>>>>>>>> [mailto:bar@mariadb.org] Envoyé : > >> lundi > >>>>>>>>>>>>>> 18 septembre 2017 11:30 À : jerome brauge Objet : Re: > >> Oracle > >>>>>>>>>>>>>> strings functions compatibility : substr > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hello Jerome, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 08/17/2017 04:43 PM, jerome brauge wrote: > >>>>>>>>>>>>>>> Hello Alexander, > >>>>>>>>>>>>>>> Here is a patch for function SUBSTR in sql_mode=oracle > >>>>>>>>>>>>>>> (If position is 0, > >>>>>>>>>>>>>> then it is treated as 1). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I'm thinking of how to get a smaller patch. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Wouldn't it be possible to introduce: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> virtual longlong get_position(); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> longlong Item_func_substr::get_position() { > >>>>>>>>>>>>>> return args[1]->val_int(); } > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> longlong Item_func_substr_oracle::get_position() > >>>>>>>>>>>>>> { > >>>>>>>>>>>>>> longlong pos= args[1]->val_int(); > >>>>>>>>>>>>>> return pos == 0 ? 1 : pos; } > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> and just replace all calls for args[1]->val_int() to > >> get_position() > >>>> ? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>> Jérôme. > >>>>>>>>>>>>>>>

2 3

Re: [Maria-developers] [Commits] f87dddaaa95: MDEV-7865: Server crashes in Item_equal_iterator<List_iterator_fast, Item>::get_curr_field on query
by Sergey Petrunia 09 Nov '17

09 Nov '17

On Sat, Oct 28, 2017 at 12:47:03AM +0530, Varun wrote: > revision-id: f87dddaaa95f9e7fc597c8839b539cc6f816c750 (mariadb-5.5.56-91-gf87dddaaa95) > parent(s): fb5fe497e526f0677a7e2f86f34237e7efd4b806 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-10-28 00:46:43 +0530 > message: > > MDEV-7865: Server crashes in Item_equal_iterator<List_iterator_fast, Item>::get_curr_field on query > with impossible condition and OR/AND expressions > > When multiple-equalites are merged into one in the function Item_equal::merge_with_check, then the fields that are merged from one > equality to the other still point to the previous mulitple_equality while the should be pointing to the new equality > after the merge is done .. > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index ebe088e5092..8c79359ce88 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -5832,7 +5832,18 @@ bool Item_equal::merge_with_check(Item_equal *item, bool save_merged) > if (intersected) > { > if (!save_merged) > + { > + Item *item_it; > + fi.rewind(); > + while ((item_it= fi++)) > + { Wrong indentation > + if (!contains(fi.get_curr_field())) > + { > + item_it->set_item_equal(this); I am not sure if this can actually make a difference, but is there any reason we don't just call set_item_equal unconditionally here? My concern is that f.get_curr_field() checks for Field* objects, while set_item_equal() call applies to Items (either Item_field or Item_direct_view_ref) What if there are two different Item objects that refer to the same field? (this looks like it should not happen but I am not 100% certain about this). > + } > + } > merge(item); > + } > else > { > Item *c= item->get_const(); > @@ -5843,9 +5854,12 @@ bool Item_equal::merge_with_check(Item_equal *item, bool save_merged) > Item *item; > fi.rewind(); > while ((item= fi++)) > - { > + { > if (!contains(fi.get_curr_field())) > + { > add(item); > + item->set_item_equal(this); > + } here, we have save_merged=true, that is, the old Item_equal will still be used. Is this ok that we call set_item_equal($new_equality) for its members? If an object is a participant in two Item_equal, which one should it point to? the "most specific" one? I'll try to ask Igor about this today. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

[Maria-developers] MDEV-13418 Compatibility: The order of evaluation of SELECT..INTO assignments
by jerome brauge 09 Nov '17

09 Nov '17

Hello Alexander. I've begun to implement your proposal but now I'm not sure that it's a better solution than mine. Let me explain . - first : number of temporary variables can be significant because we don't know when they are really needed and their scope are local to the statement. declare b1 INTEGER; declare res INTEGER; ... if b1 = 0 then select 1,b1+1 into b1, res from dual; end if; if b1 = 1 then select 2,b1+2 into b1, res from dual; end if; will be transform in : declare b1 INTEGER; declare res INTEGER; ... if b1 = 0 then declare _b1 INTEGER default res; declare _res INTEGER default res; select 1,b1+1 into _b1, _res from dual; set b1=_b1; set res=_res; -- _res is not needed, but we don't know because the select statement is not parsed end if; if b1 = 1 then declare _b1 INTEGER default res; declare _res INTEGER default res; select 2,b1+2 into b1, res from dual; set b1=_b1; set res=_res; -- same thing here, and we have declare two variables for each target variables end if; Perhaps we could - declare these temporary variables only one time in the first frame of the stored procedure (may be tricky) - parse columns of each select to know what variables are really assigned and reused (heavy cost in cpu and time) - second : if we can't determine variables which must have a temporary variable, number of sp_instr_set and sp_instr_set_var will be too high and their cpu cost is not acceptable. My first solution has a fixed memory impact (and memory is not an issue nowadays), and especially a very light cpu cost. What do you think about ? Regard, Jérôme. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : lundi 30 octobre 2017 10:02 > À : jerome brauge > Objet : Re: MDEV-14139 Anchored data types for variables > > > > On 10/28/2017 07:29 PM, Alexander Barkov wrote: > > > > On 10/27/2017 10:27 PM, Alexander Barkov wrote: > >> Hello Jerome, > >> > >> I have implemented "MDEV-14139 Anchored data types for variables". > >> and pushed it to bb-10.2-ext: > >> > >> > https://github.com/MariaDB/server/commit/5dd5253f7e50c21fa758e2eb58f > 3 > >> aa9c9754e733 > >> > >> So now it should be easier to implement consistent SET by creating > >> backup variables. > >> > >> > >> LEX::sp_variable_declarations_vartype_finalize() implements the logic > >> which copies data type from another variable. > >> > >> > >> The idea is that for all variables, which are assignment targets in a > >> SET or a SELECT INTO statement, we create a backup variable. > >> > >> It will involve these calls for every such variable: > >> > >> LEX::sp_variable_declarations_init(thd, 1); > >> sp_pcontext::add_variable(thd, backup_variable_name); > >> LEX::sp_variable_declarations_vartype_finalize(thd, 1, > >> orig_variable_name, def); > >> > >> where "def" is Item_splocal created for the original variable. > > > > Just an idea: instead of creating sp_instr_set, it's easier to > > introduce a new class sp_instr_set_var, to copy the value from one > > variable to another variable. > > > > This operation will not need neither Item, nor sp_lex_keeper. It will > > only need two offsets: > > - the source variable offset and > > - the target variable offset. > > > > Using these offsets, we can access to > > spcont->m_var_table->field[source] and > > spcont->spcont->m_var_table->field[target] > > and copy the value between them using Field::store_field(). > > > > This won't however for the ROW variables at the moment, because ROW > > fields are stored in the Item_spvar_args::m_table member of > > Item_field_row. > > > > It seems we need a new class Field_row and move Virtual_tmp_table from > > Item_field_row to Field_row. > > > > I will try to do it. > > > I have implemented "MDEV-14212 Add Field_row for SP ROW variables" and > pushed to bb-10.2-ext. > > Also, added a comment: > > MDEV-13418 Compatibility: The order of evaluation of SELECT..INTO > assignments > > > Now, when MDEV-14212 is done, these declarations: > > DECLARE a_tmp TYPE OF a DEFAULT a; > DECLARE b_tmp TYPE OF b DEFAULT b; > > and these assignments: > > SET a=a_tmp; > SET b=b_tmp; > > can use a new sp_instr_setvar instead of sp_inst_set. > > The new command sp_instr_setvar can do direct copying between two > spcont->m_var_table->field[XXX], without a need to create Item and LEX. > > > > >> > >> Greetings. > >>

2 9

[Maria-developers] Summary of the InnoDB changes in the AliSQL 5.6.32 branch
by Marko Mäkelä 08 Nov '17

08 Nov '17

In preparation for the 2017-2 MariaDB Developer Unconference in Shenzhen <https://mariadb.org/2017-2-developers-unconference-and-related-events-shenz…> next week, I cloned the https://github.com/alibaba/AliSQL repository and reviewed the changes to storage/innobase. The repository is based on commit c3a8ca1ed0063d4bca761ac9b57718796242ccc1 (tag: mysql-5.6.32) in the https://github.com/mysql/mysql-server/ repository. I read the 13,587 lines of diff generated by the following command: git diff c3a8ca1ed0063d4bca761ac9b57718796242ccc1 storage/innobase Features that already exist in MariaDB in some form: * Split the btr_search_latch (already present in MySQL 5.7/MariaDB 10.2.2) * Buffer pool "hazard pointer" (5.7/10.2.2) * Faster read view creation (done differently in WL#6578 MySQL 5.7/MariaDB 10.2.2) * Persistent auto_increment (done differently in MDEV-6076 in MariaDB 10.2.4) * Instant ADD COLUMN (done differently in MDEV-11369 in MariaDB 10.3.2) * Sequence engine (done differently in MDEV-10139 in MariaDB 10.3.1) New features: * Counters for buffer pool LRU eviction: MONITOR_LRU_BATCH are split into MONITOR_LRU_BATCH_FLUSH and MONITOR_LRU_BATCH_EVICT (Sounds like a good idea to me; maybe MariaDB should export these via status variables.) * Issue#24 SPLIT LOG BUFFER TO ROTATE LOG WRITE (I’d go further and change the LSN to count mtr_t::commit() instead of bytes, and to use a global buffer for mtr_t::log. This would require a new redo log format.) * Issue#31 OPTIMIZE CHECK/GRANT OF INNODB TABLE LOCK (I’d rather replace InnoDB table locks and dict_operation_lock with MDL.) * Issue#29: ADD INFORMATION_SCHEMA.INNODB_RSEG TABLE (I’d rather split the undo log into tablename.ibu files, also to ease import/export/backup.) * Issue#9: PROVIDE ADAPTIVE ALGORITHM FOR INNODB CONCURRENCY TICKETS (I think that the whole ticket business should be moved to the thread pool plugin.) * Issue#34: Support force drop table (I’d rather remove the InnoDB dictionary and have transactional .frm files, instead of writing work-arounds.) Best regards, Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation

1 0

Re: [Maria-developers] d78273d: MDEV-14288 Add MTR option to run tests with real server defaults
by Sergei Golubchik 05 Nov '17

05 Nov '17

Hi, Elena! On Nov 05, elenst(a)montyprogram.com wrote: > revision-id: d78273d1e53308c8fe78b9e067a88c43bd05e540 > parent(s): 737a4da94ae9b25a7ebe828c5b35a1ce4acf03d6 > committer: Elena Stepanova > branch nick: bb-10.0-elenst > timestamp: 2017-11-05 01:30:40 +0200 > message: > > MDEV-14288 Add MTR option to run tests with real server defaults > > diff --git a/mysql-test/lib/My/ConfigFactory.pm b/mysql-test/lib/My/ConfigFactory.pm > index b8ce038..dcdb534 100644 > --- a/mysql-test/lib/My/ConfigFactory.pm > +++ b/mysql-test/lib/My/ConfigFactory.pm > @@ -37,7 +37,9 @@ sub add_opt_values { > > # add auto-options > $config->insert('OPT', 'port' => sub { fix_port($self, $config) }); > - $config->insert('mysqld', "loose-skip-plugin-$_" => undef) for (@::optional_plugins); > + if (!$::opt_default_server_options) { > + $config->insert('mysqld', "loose-skip-plugin-$_" => undef) for (@::optional_plugins); > + } Dunno. I'd prefer to remove magic from mtr, not to add it. Generating different configs feels like magic. I wonder whether this could work: - mtr always disables plugins, but in a separate .cnf file. And default_mysqld.cnf will !include it, but yours won't. or - all this disabling will go where, say, @DISABLE_ALL_PLUGINS line is. it'll be in default_mysqld.cnf but not in yours. > } > > my @pre_rules= > diff --git a/mysql-test/lib/mtr_cases.pm b/mysql-test/lib/mtr_cases.pm > index 38c52b7..26a4edd 100644 > --- a/mysql-test/lib/mtr_cases.pm > +++ b/mysql-test/lib/mtr_cases.pm > @@ -876,7 +876,10 @@ sub collect_one_test_case { > if (not $config) > { > # Suite has no config, autodetect which one to use > - if ($tinfo->{rpl_test}) { > + if ($::opt_default_server_options) { > + $config= "include/hard_default_my.cnf"; > + } > + elsif ($tinfo->{rpl_test}) { yuck. I never liked that rpl suite magically gets a different cnf file. may be it's time to fix it? e.g. by moving it into suite/rpl/my.cnf or into include/rpl_init.cnf ? > $config= "suite/rpl/my.cnf"; > } else { > $config= "include/default_my.cnf"; > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index 33d9c7b..997fb82 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -131,6 +131,7 @@ our $plugindir; > my $path_vardir_trace; # unix formatted opt_vardir for trace files > my $opt_tmpdir; # Path to use for tmp/ dir > my $opt_tmpdir_pid; > +our $opt_default_server_options; > > my $opt_start; > my $opt_start_dirty; > @@ -1106,6 +1107,9 @@ sub command_line_setup { > # Extra config file to append to all generated configs > 'defaults-extra-file=s' => \&collect_option, > > + # Use hard defaults instead of heavily modified MTR configuration > + 'default-server-options' => \$opt_default_server_options, with the changes above, you could use --defaults-file=hard_default_my or something like that (even better with a short cnf file name) You could even commit a bunch of templates, like vanilla.cnf, rpm.cnf, debian.cnf and use them like ./mtr --defaults-file=debian > + > # Control what test suites or cases to run > 'force+' => \$opt_force, > 'with-ndbcluster-only' => \&collect_option, > @@ -3509,12 +3517,14 @@ sub mysql_install_db { > mtr_add_arg($args, "--basedir=%s", $install_basedir); > mtr_add_arg($args, "--datadir=%s", $install_datadir); > mtr_add_arg($args, "--plugin-dir=%s", $plugindir); > - mtr_add_arg($args, "--default-storage-engine=myisam"); > - mtr_add_arg($args, "--loose-skip-plugin-$_") for @optional_plugins; > - # starting from 10.0 bootstrap scripts require InnoDB > - mtr_add_arg($args, "--loose-innodb"); > - mtr_add_arg($args, "--loose-innodb-log-file-size=5M"); > - mtr_add_arg($args, "--disable-sync-frm"); > + if (!$opt_default_server_options) { > + mtr_add_arg($args, "--default-storage-engine=myisam"); > + mtr_add_arg($args, "--loose-skip-plugin-$_") for @optional_plugins; > + # starting from 10.0 bootstrap scripts require InnoDB > + mtr_add_arg($args, "--loose-innodb"); > + mtr_add_arg($args, "--loose-innodb-log-file-size=5M"); > + mtr_add_arg($args, "--disable-sync-frm"); > + } why is that? it's just mysql_install_db > mtr_add_arg($args, "--tmpdir=%s", "$opt_vardir/tmp/"); > mtr_add_arg($args, "--core-file"); > mtr_add_arg($args, "--console"); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 58c9df453f5: Remove usage of unknown parameter.
by Marko Mäkelä 02 Nov '17

02 Nov '17

This file should not have been re-added in the first place. I removed this badly working file for a good reason. Please try to read the git history before re-adding broken tests. We now have many useless and outright broken tests in --suite=innodb_gis that need some cleanup. I would have liked to review the test changes. Good tests are as valuable as good code. commit 86e0a73eaa166f752d62b31e96925e29c4fe0c8c Author: Marko Mäkelä <marko.makela(a)mariadb.com> Date: Mon Aug 7 23:38:27 2017 +0300 Remove wait_innodb_all_purged.inc The file wait_innodb_all_purged.inc waited for InnoDB purge in a way that only worked in debug builds. The file wait_all_purged.inc provides a better mechanism. The wait_all_purged.inc was added almost 6 months ago: file commit 0bfa3dff8b95ed8b8bd133e434b47189857121ac Author: Marko Mäkelä <marko.makela(a)mariadb.com> Date: Tue May 16 12:07:26 2017 +0300 MDEV-12698 innodb.innodb_stats_del_mark test failure On Tue, Oct 24, 2017 at 12:24 PM, jan <jan.lindstrom(a)mariadb.com> wrote: > revision-id: 58c9df453f5d8408dd189fb2e5ae4445446b6701 (mariadb-10.2.9-80-g58c9df453f5) > parent(s): 39f86dd27a4c772f0b0a65fbf9b125be9e2dc8ac > author: Jan Lindström > committer: Jan Lindström > timestamp: 2017-10-24 12:23:57 +0300 > message: > > Remove usage of unknown parameter. > > --- > mysql-test/include/wait_innodb_all_purged.inc | 1 - > 1 file changed, 1 deletion(-) > > diff --git a/mysql-test/include/wait_innodb_all_purged.inc b/mysql-test/include/wait_innodb_all_purged.inc > index 5bf816fd72f..c5edc5937da 100644 > --- a/mysql-test/include/wait_innodb_all_purged.inc > +++ b/mysql-test/include/wait_innodb_all_purged.inc > @@ -48,7 +48,6 @@ WHERE VARIABLE_NAME = 'INNODB_PURGE_TRX_ID_AGE';`; > } > if (!$success) > { > - set global innodb_purge_run_now=ON; > real_sleep 0.1; > dec $wait_counter; > } > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation

1 0

[Maria-developers] 10.2.10 compiler warnings Fedora 26 / GCC7
by Reindl Harald 01 Nov '17

01 Nov '17

maybe worth to take a look [ 8%] Building C object libmariadb/mariadb_config/CMakeFiles/mariadb_config.dir/mariadb_config.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/mariadb_config/mariadb_config.c:9:49: warning: backslash and newline separated by space #define LIBS "-L/usr/lib64/mysql/ -lmariadb "\ [ 20%] Building C object libmariadb/libmariadb/CMakeFiles/mariadb_obj.dir/ma_dtoa.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c: In function 'dtoa': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c:1465:14: warning: this statement may fall through [-Wimplicit-fallthrough=] leftright= 0; ~~~~~~~~~^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c:1467:3: note: here case 4: ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c:1473:14: warning: this statement may fall through [-Wimplicit-fallthrough=] leftright= 0; ~~~~~~~~~^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c:1475:3: note: here case 5: ^~~~ At top level: /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c:1161:21: warning: 'tinytens' defined but not used [-Wunused-const-variable=] static const double tinytens[]= ^~~~~~~~ [ 24%] Building C object strings/CMakeFiles/strings.dir/dtoa.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/mariadb_dyncol.c: In function 'mariadb_dyncol_val_double': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/mariadb_dyncol.c:4049:15: warning: argument 1 null where non-null expected [-Wnonnull] *dbl= strtod(str, &end); ^~~~~~~~~~~~~~~~~ In file included from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/include/ma_global.h:168:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/mariadb_dyncol.c:64: /usr/include/stdlib.h:126:15: note: in a call to function 'strtod' declared here extern double strtod (const char *__restrict __nptr, [ 33%] Building C object mysys/CMakeFiles/mysys.dir/my_malloc.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/mysys/my_lib.c: In function 'my_dir': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/mysys/my_lib.c:144:3: warning: 'readdir_r' is deprecated [-Wdeprecated-declarations] while (!(READDIR(dirp,(struct dirent*) dirent_tmp,dp))) ^~~~~ In file included from /usr/include/features.h:410:0, from /usr/include/pthread.h:21, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_global.h:278, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/mysys/mysys_priv.h:19, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/mysys/my_lib.c:19: /usr/include/dirent.h:189:12: note: declared here extern int __REDIRECT (readdir_r, [ 68%] Building C object storage/maria/CMakeFiles/aria.dir/ma_check.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/myisamchk.c: In function 'myisamchk': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/myisamchk.c:1411:22: warning: '%ld' directive writing between 1 and 10 bytes into a region of size 8 [-Wformat-overflow=] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/myisamchk.c:1411:21: note: directive argument in the range [1, 4294967296] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/myisam.h:26, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/myisamdef.h:20, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/fulltext.h:21, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/myisamchk.c:18: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 2 and 11 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [ 68%] Building C object storage/maria/CMakeFiles/aria.dir/ma_preload.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/mi_test3.c: In function 'test_write': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/mi_test3.c:367:30: warning: '%7ld' directive writing between 7 and 11 bytes into a region of size 8 [-Wformat-overflow=] sprintf((char*) record.id,"%7ld",(long) getpid()); ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/mi_test3.c:367:29: note: directive argument in the range [-2147483648, 2147483647] sprintf((char*) record.id,"%7ld",(long) getpid()); ^~~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/myisam.h:26, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/mi_test3.c:20: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 8 and 12 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [ 71%] Building C object storage/maria/CMakeFiles/aria_chk.dir/maria_chk.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_chk.c: In function 'maria_chk': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_chk.c:1703:22: warning: '%ld' directive writing between 1 and 10 bytes into a region of size 8 [-Wformat-overflow=] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_chk.c:1703:21: note: directive argument in the range [1, 4294967296] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/maria.h:22, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_def.h:18, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/ma_fulltext.h:20, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_chk.c:18: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 2 and 11 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [ 73%] Building C object storage/maria/CMakeFiles/ma_test3.dir/ma_test3.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/ma_test3.c: In function 'test_write': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/ma_test3.c:365:30: warning: '%7ld' directive writing between 7 and 11 bytes into a region of size 8 [-Wformat-overflow=] sprintf((char*) record.id,"%7ld", (long) getpid()); ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/ma_test3.c:365:29: note: directive argument in the range [-2147483648, 2147483647] sprintf((char*) record.id,"%7ld", (long) getpid()); ^~~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/maria.h:22, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_def.h:18, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/ma_test3.c:20: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 8 and 12 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1 0

Re: [Maria-developers] [Commits] f88fcf2: MDEV-12542 Add bind_address system variable.
by Sergei Golubchik 30 Oct '17

30 Oct '17

Hi, Alexey! On Oct 30, Alexey Botchkov wrote: > revision-id: f88fcf2514c64df8408e9df41dc129ab6f655c18 (mariadb-10.3.2-17-gf88fcf2) > parent(s): ecee3c71e1e74039e75089f1d76c0b6eea9e43ba > committer: Alexey Botchkov > timestamp: 2017-10-30 07:37:25 +0400 > message: > > MDEV-12542 Add bind_address system variable. > > bind_address variable added. > > --- > .../suite/sys_vars/r/bind_address_basic.result | 57 +++++++++++++ > .../suite/sys_vars/t/bind_address_basic.test | 95 ++++++++++++++++++++++ There's no need to add these files anymore (since 10.1) They were supposed to test basic properties like num/string/readonly, etc. But now sysvars_server_notembedded.result has that for all variables anyway. So these *_basic tests are not needed. > sql/mysqld.cc | 3 - > sql/mysqld.h | 1 + > sql/sys_vars.cc | 7 ++ > 6 files changed, 174 insertions(+), 3 deletions(-) > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-13997 Change Item_bool_rowready_func2 to cache const items at fix time rather than evaluation time
by Alexander Barkov 29 Oct '17

29 Oct '17

Hello Sergei, Please review a patch for MDEV-13997. It's needed for: MDEV-13995 MAX(timestamp) returns a wrong result near DST change and for: MDEV-4912 Add a plugin to field types (column types) Thanks!

2 3

[Maria-developers] Review Custom Aggregate Functions
by Vicențiu Ciorbaru 28 Oct '17

28 Oct '17

Hi Varun! Here's a final review for Custom Aggregate Functions * You haven't run the full test suite, there are a few failing tests. Please update those. rpl.rpl_sp funcs_1.is_columns_mysql funcs_1.storedproc * Push your code to a BB branch and rebase it on top of current 10.3. * Fix commits messages while rebasing. Use the reword flag to clean those up. The rule is this: 1. Have one line that doesn't exceed 80 Characters as a Title. 2. Leave one empty line 3. Add the rest of the commit message. Don't exceed 80 Characters per line. > diff --git a/sql/sp_head.h b/sql/sp_head.h > index bb516187a57..d447807bb3b 100644 > --- a/sql/sp_head.h > +++ b/sql/sp_head.h > @@ -334,6 +334,11 @@ class sp_head :private Query_arena, > execute_function(THD *thd, Item **args, uint argcount, Field *return_fld); > > bool Rename this to execute_function please and delete the now dead-code from the current execute_function. Update comments for this function appropriately. Please run all tests again after this and make sure you update results for all the failing ones. > + execute_aggregate_function(THD *thd, Item **args, uint argcount, > + Field *return_fld, sp_rcontext **nctx, > + MEM_ROOT *caller_mem_root); > + > + bool > execute_procedure(THD *thd, List<Item> *args); > > static void > @@ -813,6 +818,9 @@ class sp_head :private Query_arena, > bool > execute(THD *thd, bool merge_da_on_success); > This is no longer used please remove! > + bool > + execute_agg(THD *thd, bool merge_da_on_success); > + > /** > Perform a forward flow analysis in the generated code. > Mark reachable instructions, for the optimizer. > +++ b/mysql-test/t/sp_agg.test A lot of empty lines at the end of this test. Please delete those. Ping me once you have a green buildbot before pushing. Vicentiu

1 0

[Maria-developers] Review of Percentile Window Functions
by Vicențiu Ciorbaru 28 Oct '17

28 Oct '17

Hi Varun, Here's the final review for Percentile Functions. Just 2 minor coding style comments. Rebase on top of current 10.3 and push to a bb-* branch to double check nothing breaks. Make sure that no merge commit happens. Regards, Vicențiu > diff --git a/sql/item_windowfunc.h b/sql/item_windowfunc.h > index 77cbd556e60..589f7cf5d96 100644 > --- a/sql/item_windowfunc.h > +++ b/sql/item_windowfunc.h > @@ -678,6 +687,277 @@ class Item_sum_ntile : public Item_sum_window_with_row_count > ulong current_row_count_; > }; > > +class Item_sum_percentile_disc : public Item_sum_cume_dist, > + public Type_handler_hybrid_field_type > +{ > +public: > + Item_sum_percentile_disc(THD *thd, Item* arg) : Item_sum_cume_dist(thd, arg), > + Type_handler_hybrid_field_type(&type_handler_longlong), > + value(NULL), val_calculated(FALSE), first_call(TRUE), > + prev_value(0), order_item(NULL){} > + > + double val_real() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_real(); > + } > + > + longlong val_int() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_int(); > + } > + > + my_decimal* val_decimal(my_decimal* dec) > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_decimal(dec); > + } > + > + String* val_str(String *str) > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_str(str); > + } > + > + bool add() > + { > + Item *arg= get_arg(0); > + if (arg->is_null()) > + return false; > + > + if (first_call) > + { > + prev_value= arg->val_real(); > + if (prev_value > 1 || prev_value < 0) > + { > + my_error(ER_ARGUMENT_OUT_OF_RANGE, MYF(0), func_name()); > + return true; > + } > + first_call= false; > + } > + > + double arg_val= arg->val_real(); > + > + if (prev_value != arg_val) > + { > + my_error(ER_ARGUMENT_NOT_CONSTANT, MYF(0), func_name()); > + return true; > + } > + > + if (val_calculated) > + return false; > + > + value->store(order_item); > + value->cache_value(); > + if (value->null_value) > + return false; > + > + Item_sum_cume_dist::add(); > + double val= Item_sum_cume_dist::val_real(); > + > + if (val >= prev_value && !val_calculated) > + val_calculated= true; > + return false; > + } > + > + enum Sumfunctype sum_func() const > + { > + return PERCENTILE_DISC_FUNC; > + } > + > + void clear() > + { > + val_calculated= false; > + first_call= true; > + value->clear(); > + Item_sum_cume_dist::clear(); > + } > + > + const char*func_name() const > + { > + return "percentile_disc"; > + } > + > + void update_field() {} > + void set_type_handler(Window_spec *window_spec); > + const Type_handler *type_handler() const > + {return Type_handler_hybrid_field_type::type_handler();} > + > + void fix_length_and_dec() > + { > + decimals = 10; // TODO-cvicentiu find out how many decimals the standard > + // requires. > + } > + > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_sum_percentile_disc>(thd, mem_root, this); } > + void setup_window_func(THD *thd, Window_spec *window_spec); > + void setup_hybrid(THD *thd, Item *item); > + bool fix_fields(THD *thd, Item **ref); > + > +private: > + Item_cache *value; > + bool val_calculated; > + bool first_call; > + double prev_value; > + Item *order_item; > +}; > + > +class Item_sum_percentile_cont : public Item_sum_cume_dist, > + public Type_handler_hybrid_field_type > +{ > +public: > + Item_sum_percentile_cont(THD *thd, Item* arg) : Item_sum_cume_dist(thd, arg), > + Type_handler_hybrid_field_type(&type_handler_double), > + floor_value(NULL), ceil_value(NULL), first_call(TRUE),prev_value(0), > + ceil_val_calculated(FALSE), floor_val_calculated(FALSE), order_item(NULL){} > + > + double val_real() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + double val= 1 + prev_value * (get_row_count()-1); > + > + /* > + Applying the formula to get the value > + If (CRN = FRN = RN) then the result is (value of expression from row at RN) > + Otherwise the result is > + (CRN - RN) * (value of expression for row at FRN) + > + (RN - FRN) * (value of expression for row at CRN) > + */ > + > + if(ceil(val) == floor(val)) > + return floor_value->val_real(); > + > + double ret_val= ((val - floor(val)) * ceil_value->val_real()) + > + ((ceil(val) - val) * floor_value->val_real()); > + > + return ret_val; > + } > + > + bool add() > + { > + Item *arg= get_arg(0); > + if (arg->is_null()) > + return false; > + > + if (first_call) > + { > + first_call= false; > + prev_value= arg->val_real(); > + if (prev_value > 1 || prev_value < 0) > + { > + my_error(ER_ARGUMENT_OUT_OF_RANGE, MYF(0), func_name()); > + return true; > + } > + } > + > + double arg_val= arg->val_real(); > + if (prev_value != arg_val) > + { > + my_error(ER_ARGUMENT_NOT_CONSTANT, MYF(0), func_name()); > + return true; > + } > + > + if (!floor_val_calculated) > + { > + floor_value->store(order_item); > + floor_value->cache_value(); > + if (floor_value->null_value) > + return false; Add one extra space before return > + } > + if (floor_val_calculated && !ceil_val_calculated) > + { > + ceil_value->store(order_item); > + ceil_value->cache_value(); > + if (ceil_value->null_value) > + return false; Add one extra space before return > + } > + > + Item_sum_cume_dist::add(); > + double val= 1 + prev_value * (get_row_count()-1); > + > + if (!floor_val_calculated && get_row_number() == floor(val)) > + floor_val_calculated= true; too much indentation here. Delete 2 spaces > + > + if (!ceil_val_calculated && get_row_number() == ceil(val)) > + ceil_val_calculated= true; too much indentation here. Delete 2 spaces > + return false; > + } > + > + enum Sumfunctype sum_func() const > + { > + return PERCENTILE_CONT_FUNC; > + } > + > + void clear() > + { > + first_call= true; > + floor_value->clear(); > + ceil_value->clear(); > + floor_val_calculated= false; > + ceil_val_calculated= false; > + Item_sum_cume_dist::clear(); > + } > + > + const char*func_name() const > + { > + return "percentile_cont"; > + } > + void update_field() {} > + void set_type_handler(Window_spec *window_spec); > + const Type_handler *type_handler() const > + {return Type_handler_hybrid_field_type::type_handler();} > + > + void fix_length_and_dec() > + { > + decimals = 10; // TODO-cvicentiu find out how many decimals the standard > + // requires. > + } > + > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_sum_percentile_cont>(thd, mem_root, this); } > + void setup_window_func(THD *thd, Window_spec *window_spec); > + void setup_hybrid(THD *thd, Item *item); > + bool fix_fields(THD *thd, Item **ref); > + > +private: > + Item_cache *floor_value; > + Item_cache *ceil_value; > + bool first_call; > + double prev_value; > + bool ceil_val_calculated; > + bool floor_val_calculated; > + Item *order_item; > +}; > + > + > + > > class Item_window_func : public Item_func_or_sum > { > diff --git a/sql/lex.h b/sql/lex.h > index 17dd45f9c0b..fcb44a0f836 100644 > --- a/sql/lex.h > +++ b/sql/lex.h > @@ -735,6 +736,7 @@ static SYMBOL sql_functions[] = { > { "LAG", SYM(LAG_SYM)}, > { "LEAD", SYM(LEAD_SYM)}, > { "MAX", SYM(MAX_SYM)}, > + { "MEDIAN", SYM(MEDIAN_SYM)}, Fix indentation here > { "MID", SYM(SUBSTRING)}, /* unireg function */ > { "MIN", SYM(MIN_SYM)}, > { "NOW", SYM(NOW_SYM)}, > \ No newline at end of file

1 0

Re: [Maria-developers] 9145b7d: MDEV-11153 - Introduce status variables for table cache monitoring and tuning
by Sergei Golubchik 27 Oct '17

27 Oct '17

Hi, Sergey! Looks ok, thanks. Just one question: On Aug 10, Sergey Vojtovich wrote: > revision-id: 9145b7dd11a0e95016304e53acf306daac323580 (mariadb-10.3.0-49-g9145b7d) > parent(s): 63ad4fe5bbe77fe17ab441fcce0106bdee9b618d > committer: Sergey Vojtovich > timestamp: 2017-08-10 15:45:03 +0400 > message: > > MDEV-11153 - Introduce status variables for table cache monitoring and tuning > > Status variables added: Table_open_cache_hits, Table_open_cache_misses, > Table_open_cache_overflows. > > System variables added: table_open_cache_active_instances (read-only). > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index e89dd47..92721b6 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -3219,6 +3219,12 @@ static Sys_var_ulong Sys_table_cache_size( > BLOCK_SIZE(1), NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0), > ON_UPDATE(fix_table_open_cache)); > > +static Sys_var_uint Sys_table_cache_active_instances( > + "table_open_cache_active_instances", > + "Number of active table cache instances", > + READ_ONLY GLOBAL_VAR(tc_active_instances), NO_CMD_LINE, > + VALID_RANGE(1, 64), DEFAULT(1), BLOCK_SIZE(1)); Why did you do it as a sysvar, not as a status variable? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] Code syntax: questions on pointers, etc.
by Aleksey Midenkov 27 Oct '17

27 Oct '17

Recent refactorings of replacing C strings with LEX_CSTRING which is no doubt a good thing raise some questions: 1. Is it still guaranteed that Field::field_name.str is NULL-terminated? 2. It is still passed as a pointer to functions. Why is that? The main feature of C++ references is that it cannot be NULL, so we get segfault on top of the stack (closer to a cause), not the bottom of it. I see that pointers are now widely used and mainly assumed to be always non-NULL (i.e. dereferenced without assertion). But placing such implicit contract on data is not evident and bug-prone. IMHO it's much better to use references whenever it is possible (and when there is no need in cosy NULL semantic). What do you think? But for such lightweight structs like LEX_CSTRING it is even better to pass by value, so we could have the conventience of type cast. 3. LEX_CSTRING and LEX_STRING are now non-convertible. Why not to make: template <typename char_t> struct st_mysql_lex_string { char_t *str; size_t length; }; typedef st_mysql_lex_string<char *> LEX_STRING; typedef st_mysql_lex_string<const char *> LEX_CSTRING; typedef st_mysql_lex_string<const unsigned char *> LEX_CUSTRING; ? 4. There are some duplicate types: MYSQL_LEX_STRING, MYSQL_CONST_LEX_STRING. Why? -- All the best, Aleksey Midenkov @midenok

2 3

[Maria-developers] Obsolete GTID domain delete on master (MDEV-12012, MDEV-11969)
by andrei.elkin＠pp.inet.fi 24 Oct '17

24 Oct '17

Hello. Let me propose methods to clean master off unused gtid domains. The issue is quite practical, as a couple of references on the subject line tell. Either of them narrates a scenario of two default domain-id masters serving to one slave initially by "legacy" non-gtid protocol. When later the masters have changed their common domain to different private ones *and* the slave turns gtid replication ON, it can't connect to a master due to the gtid protocol. And the reason is the prior-to-gtid-connect gained slave's gtid position consisting of gtids generated by the other master obviously (the two masters never replicated each other) does not fit to the current master binlog state. The past default domain-id is actually permanent past from the user perspective in these cases. Its events have been already replicated and none new will be generated and replicated. Therefore such domain conceptually may be cleaned away from either the masters and slave states. Once it's done, the gtid-enabled slave will successfully connect to any master. The slave state purge is simple SET @@global.gtid_slave_pos to a sequence free of the purged domain. The master side binlog state one requires a new "command" unless the user is happy with RESET MASTER. While setting the new gtid binlog state to be old-domain-free we would like for the new "command" to preserve the existing binlog files. This could be accomplished as Kristian suggests in MDEV-12012 (I could not find any earlier references to the idea) as a new option to FLUSH BINARY LOGS DELETE DOMAIN d1, d2 KN> This command would check that the current binlog state is equal to KN> the GTID_LIST_LOG_EVENT at the start of the first binary log file, KN> within the specified domains. If not, it is an error. But if so, KN> the new binary log file created by FLUSH will be written with the KN> specified domains omitted from GTID_LIST_LOG_EVENT (and the current KN> binlog state updated accordingly). The idea looks quite sane, I only could not grasp why presence of being deleted domains in the very first binlog's GTID_LIST_LOG_EVENT list is warrant for throwing error. Maybe we should leave it out to the user, Kristian? That is to decide what domain is garbage regardless of the binlog state history. While the FLUSH way looks sufficient and robust I could not help to think over an alternative. Consider a scenario when a domain's sequence number got run out of range. While deems unrealistic in practice we can simulate it with SET @@SESSION.gtid_domain_id=11; SET @@SESSION.server_id=1; SET @@SESSION.gtid_seq_no=18446744073709551615; /* Exec some dummy loggable query, e.g */ CREATE TABLE IF NOT EXISTS `table_dummy`; SHOW LOCAL VARIABLES LIKE '%gtid_binlog_pos%'; 11-1-18446744073709551615 SET @@SESSION.gtid_seq_no=0 DROP TABLE `table_dummy`; SHOW LOCAL VARIABLES LIKE '%gtid_binlog_pos%'; 11-1-0 I've used two gtids to show domain overflow because I also liked to read the zero of the last gtid as ... there's *nothing* in this domain. So it's actually a new "namesake" one, replacing the old that is wrapped around. The 1st group of events created in the new domain - 11-1-1 - could shadow the old domain's 11-1-1 as well as all the rest of the old domain from gtid replication protocol. And that means the old domain is actually deleted. So if my reading of zero is correct the binlog status would be empty instead. That's how we also can approach the master side old gtid domain purging: 1. Leave wrapping around an old domain to the user via running the queries like above; 2. The binary logger would be made to react on the fact of wrap-around with binary log rotation ("internal" FLUSH BINARY LOG). And the new binlog file won't contain the wrapped "away" domain (because there are no new event group in it of yet). I would be glad to hear your opinions, dear colleagues. Cheers, Andrei

4 27

[Maria-developers] Bitmap indexing with Mariadb
by chanaka rathnayaka 16 Oct '17

16 Oct '17

Hi, I am Chanaka who is an undergraduate engineering student. I am doing a research to check, how bitmap indexing improves the query performance and I hope to use MariaDB as database engine. So I need to know whether MariaDB has bitmap indexing already or not ? Thank you, Regards, Chanaka <http://www.justforlearning.info> Chanaka Rathnayaka / undergraduate, Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka bmcrathnayaka(a)gmail.com / +94776608457 [image: Twitter] <https://htmlsig.com/t/0000001BX4MCF> [image: Facebook] <https://htmlsig.com/t/0000001BR3W18> [image: LinkedIn] <https://htmlsig.com/t/0000001BT4W90> [image: Github] <https://htmlsig.com/t/0000001BYB6WK>

2 1

[Maria-developers] MDEV-13550 is test case required?
by Aleksey Midenkov 14 Oct '17

14 Oct '17

Hello! Do we have to write test cases for such coding errors? Because this is developer factor: either he does such errors or not. What are conditions of getting PR merged?

2 2

Re: [Maria-developers] MDEV-14013
by Alexander Barkov 14 Oct '17

14 Oct '17

Hello Jerome, Thank you very much for you contribution! On 10/11/2017 02:32 PM, jerome brauge wrote: > Hello Alexander, > Here is the patch for MDEV-14013, can you review it ? The patch looks very good. I only found small things: 1. LEX::make_text_literal_concat() thd->variables.character_set_client was used instread of thd->variables.collation_connection 2. LEX::make_text_literal_concat() was somewhat hard to read. I propose to change the logic to do: if (item->type() == Item::NULL_ITEM) { } instead of if (item->type() != Item::NULL_ITEM || !(thd->variables.sql_mode & MODE_EMPTY_STRING_IS_NULL)) I think it looks clearer this way. Also, I added some DBUG_ASSERTs to make more sure that a non-Item_string item is not erroneously cast to Item_string. 3. Also, I found that it's easier to place the new methods into THD instead of LEX. Please find a patch with changes 1,2,3 attached. Do you agree with these changes? Also, can you please add some tests: - SHOW CREATE VIEW into "Test in a view" in empty_string_literal.result.diff - EXPLAIN EXTENDED for the affected grammar: EXPLAIN EXTENDED SELECT ''; EXPLAIN EXTENDED SELECT _latin1''; EXPLAIN EXTENDED SELECT N''; EXPLAIN EXTENDED SELECT '' ''; Thanks! > > Thanks. > > >> -----Message d'origine----- >> De : Alexander Barkov [mailto:bar@mariadb.org] >> Envoyé : mercredi 11 octobre 2017 04:15 >> À : jerome brauge >> Objet : Re: Question about MDEV-10574 >> >> Hello Jerome, >> >> >> On 10/10/2017 04:23 PM, jerome brauge wrote: >>> Hello Alexander, >>> For sql_mode EMPTY_STRING_IS_NULL, do you think that it must cover call >> of statement with use of user variable initialized with an empty string ? >>> >>> Ex : >>> SET sql_mode=default; >>> SET @param=''; >>> >>> SET sql_mode=EMPTY_STRING_IS_NULL; >>> PREPARE test FROM 'select 1,?'; >>> EXECUTE test USING @param; >>> >>> Or >>> >>> delimiter / >>> CREATE OR REPLACE PROCEDURE p1(IN a VARCHAR(50)) BEGIN >>> IF a is null THEN >>> SELECT 'IS NULL'; >>> ELSE >>> select 'IS NOT NULL'; >>> END IF; >>> END; >>> / >>> delimiter ; >>> SET sql_mode=default; >>> SET @a=''; >>> >>> SET sql_mode=EMPTY_STRING_IS_NULL; >>> EXECUTE IMMEDIATE 'CALL p1(?)' USING @a; CALL p1(@a); >>> >>> For me, it's a study case and we can forget it. >> >> I agree. Oracle doesn't have user variables. >> >>> >>> Regards, >>> Jérôme. >>> >>>> -----Message d'origine----- >>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : samedi 7 >>>> octobre 2017 11:44 À : jerome brauge Objet : Re: Patch for MDEV-10574 >>>> >>>> Hello Jerome, >>>> >>>> >>>> On 10/06/2017 01:41 PM, jerome brauge wrote: >>>>> Hello Alexander, >>>>> >>>>>> -----Message d'origine----- >>>>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : jeudi 5 >>>>>> octobre 2017 21:26 À : jerome brauge Objet : Re: Patch for >>>>>> MDEV-10574 >>>>>> >>>>>> Hello Jerome, >>>>>> >>>>>> >>>>>> On 10/05/2017 12:45 PM, jerome brauge wrote: >>>>>>> Hello Alexander, >>>>>>> Have you gone ahead on your reflection about nulls and empty strings >> ? >>>>>> >>>>>> Yes, I think so. >>>>>> >>>>>> One addition: Monty suggested to rename flags to: >>>>>> - EMPTY_STRING_IS_NULL for literals >>>>>> - EMPTY_STRING_IS_NULL_RESULTS for functions. >>>>>> >>>>>> I'm fine with EMPTY_STRING_IS_NULL for literals, but think that >>>>>> EMPTY_STRING_IS_NULL_FUNCTIONS should be more clear than >>>>>> EMPTY_STRING_IS_NULL_RESULTS. >>>>>> Which name do you prefer for functions? >>>>> >>>>> I think like you. >>>>> For me, EMPTY_STRING_IS_NULL_RESULTS sounds like including >>>> RESULTSET >>>>> of select (which is perhaps included in point 4) >>>> >>>> Okey, let's go with _FUNCTIONS then. >>>> >>>>> >>>>>>> >>>>>>> Can I >>>>>>> - redo my pull request for substring_oracle function >>>>>>> - make a patch for first new sql_mode >>>>>> (MODE_EMPTY_AS_NULL_LITERALS) >>>>>> >>>>>> >>>>>> Yes, please. Let's move forward. >>>>>> >>>>>> Let's push the patch for substr_oracle first. >>>>>> Let's keep it return empty strings for the cases when the >>>>>> substring_length parameters is less than 1. >>>>>> It will start automatically returning NULL when we add the new >>>>>> sql_mode for functions. >>>>> >>>>> Pull request is done. >>>> >>>> I have merged it. Thanks! >>>> >>>>> >>>>>> Also, let's go ahead with the part for literals. >>>>> >>>>> Ok, I will make a separate patch. >>>>> Many thanks. >>>>> >>>>>> >>>>>> >>>>>> I have created MDEVs. Please use them in the commit comment and in >>>>>> the >>>>>> tests: >>>>>> >>>>>> MDEV-14012 sql_mode=Oracle: substr(): treat position 0 as position >>>>>> 1 >>>>>> MDEV-14013 sql_mode=EMPTY_STRING_IS_NULL >>>>>> >>>>>> >>>>>> Thanks! >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Jérôme. >>>>>>> >>>>>>> >>>>>>>> -----Message d'origine----- >>>>>>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mercredi >>>>>>>> 27 septembre 2017 08:50 À : jerome brauge Objet : Re: Patch for >>>>>>>> MDEV-10574 >>>>>>>> >>>>>>>> Hello Jerome, >>>>>>>> >>>>>>>> I told to Monty yesterday. >>>>>>>> >>>>>>>> We were going to implement the following transformations: >>>>>>>> >>>>>>>>> We are considering to implement both of the following options >>>>>>>>> (with a >>>>>>>>> switch) >>>>>>>>> >>>>>>>>> Transform Insert >>>>>>>>> - insert values ("") -> insert values (null) >>>>>>>>> >>>>>>>>> Transform Select >>>>>>>>> >>>>>>>>> where v=x => (v <> "" and V=X) >>>>>>>>> where v is null => (v="" or v is null) >>>>>>>>> >>>>>>>> >>>>>>>> See MDEV-10574 for details. >>>>>>>> >>>>>>>> Monty did not fully understand the idea of translating literals >>>>>>>> and >>>>>> functions >>>>>>>> from empty strings to null. He asked if you can explain use cases >>>>>>>> for these transformations. >>>>>>>> >>>>>>>> But we agreed that: >>>>>>>> >>>>>>>> 1. It's OK to have multiple NULL handling flags in sql_mode. >>>>>>>> >>>>>>>> 2. New flags will not be enabled in 10.3, neither in >>>>>>>> sql_mode=DEFAULT, >>>>>> nor in >>>>>>>> sql_mode=ORACLE. So one will have to specify them explicitly: >>>>>>>> set >>>>>>>> >>>>>> >>>> >> sql_mode='ORACLE,MODE_null_transfrormation_flag1,MODE_null_transfor >>>>>>>> mation_flag2'; >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> So I propose we introduce the following flags: >>>>>>>> >>>>>>>> >>>>>>>> 1. MODE_EMPTY_AS_NULL_LITERALS - for literals and PS >> parameters, >>>> as >>>>>> in >>>>>>>> your patch. >>>>>>>> >>>>>>>> 2. MODE_EMPTY_AS_NULL_FUNCTIONS - for functions. >>>>>>>> This should be done as a separate patch. >>>>>>>> Note, your patch currently implements empty-to-null translation >>>>>>>> only for Item_str_func descendants that use >> make_empty_result(). >>>>>>>> This patch should be extended to cover Item_str_func descendants >>>>>>>> which do not use make_empty_result(), as well as >>>>>>>> all other Item_func_or_sum descendants >>>>>>>> (which are not Item_str_func descendants). >>>>>>>> >>>>>>>> >>>>>>>> 3. MODE_EMPTY_AS_NULL_UPDATES >>>>>>>> >>>>>>>> For INSERT and UPDATE transformation, as in MDEV: >>>>>>>> >>>>>>>> insert values ("") -> insert values (null) >>>>>>>> update set a='' -> update set a=null; >>>>>>>> >>>>>>>> >>>>>>>> 4. MODE_EMPTY_AS_NULL_PREDICATES >>>>>>>> >>>>>>>> For predicate transformation, as in MDEV: >>>>>>>> >>>>>>>> where v=x => (v <> "" and V=X) >>>>>>>> where v is null => (v="" or v is null) >>>>>>>> >>>>>>>> >>>>>>>> What do you think about this proposal with multiple flags? >>>>>>>> Note, we can do N1 and N2 now, and add N3 and N4 and later. >>>>>>>> >>>>>>>> Also, can you please write a description why you need N1 and N2. >>>>>>>> Monty thinks that N3 and N4 should cover most use cases. >>>>>>>> So we need a good rationale for Monty and Sergei explaining why >>>>>>>> we >>>>>> adding >>>>>>>> these flags N1 and N2. >>>>>>>> >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> >>>>>>>> On 09/26/2017 04:33 PM, jerome brauge wrote: >>>>>>>>> Hi Alexander, >>>>>>>>> I'm disappointed. >>>>>>>>> Can we at least keep this part in sql_prepare : >>>>>>>>> >>>>>>>>> /* >>>>>>>>> set_param_str_oracle : convert empty string to null value >>>>>>>>> This allow to bind empty string in JDBC as a null value. >>>>>>>>> Ex : >>>>>>>>> String sel="select coalesce(?,'null param') from dual"; >>>>>>>>> PreparedStatement ps_sel = m_cnx.prepareStatement(sel); >>>>>>>>> ps_sel.setString(1, ""); >>>>>>>>> ResultSet rs = ps_sel.executeQuery(); >>>>>>>>> while (rs.next()) >>>>>>>>> { >>>>>>>>> System.out.println(rs.getString(1)); >>>>>>>>> } >>>>>>>>> Returns : 'null param' >>>>>>>>> */ >>>>>>>>> static void set_param_str_oracle(Item_param *param, uchar >> **pos, >>>>>> ulong >>>>>>>>> len) { >>>>>>>>> ulong length= get_param_length(pos, len); >>>>>>>>> if (length == 0) >>>>>>>>> param->set_null(); >>>>>>>>> else >>>>>>>>> { >>>>>>>>> if (length > len) >>>>>>>>> length= len; >>>>>>>>> param->set_str((const char *) *pos, length); >>>>>>>>> *pos+= length; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Jérôme. >>>>>>>>> >>>>>>>>> >>>>>>>>>> -----Message d'origine----- >>>>>>>>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mardi >>>>>>>>>> 26 septembre 2017 13:34 À : jerome brauge Objet : Re: Patch for >>>>>>>>>> MDEV-10574 >>>>>>>>>> >>>>>>>>>> Hello Jérôme, >>>>>>>>>> >>>>>>>>>> I'm afraid we cannot accept this change at this point of time. >>>>>>>>>> >>>>>>>>>> The problem is that all around the code we assume that NULL is >>>>>>>>>> not equal to ''. >>>>>>>>>> >>>>>>>>>> Changing '' to NULL only in the visible part of the iceberg >>>>>>>>>> (such as in the parser and in >>>>>>>>>> Item_str_func::make_empty_result()) >>>>>>>>>> will likely introduce various anomalies in fundamental things >>>>>>>>>> like joins execution, optimizer, equality propagation, etc. >>>>>>>>>> >>>>>>>>>> Sorry, we don't have resources to dive into this topic deeper now. >>>>>>>>>> Moreover, Oracle's documentation suggests not to reply on the >>>>>>>>>> fact >>>>>> that '' >>>>>>>>>> and NULL are treated as same, as this can change in the future. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 09/20/2017 03:52 PM, jerome brauge wrote: >>>>>>>>>>> Hello Alexander, >>>>>>>>>>> Here is a new version of this patch (minor optimizations) >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Jérôme. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> -----Message d'origine----- >>>>>>>>>>>> De : Maria-developers [mailto:maria-developers- >>>>>>>>>>>> bounces+j.brauge=qualiac.com(a)lists.launchpad.net] De la part >>>> de >>>>>>>>>>>> bounces+jerome >>>>>>>>>>>> brauge >>>>>>>>>>>> Envoyé : mardi 19 septembre 2017 16:09 À : 'Alexander Barkov' >>>>>>>>>>>> Cc : MariaDB Developers >>>>>>>>>>>> (maria-developers(a)lists.launchpad.net) >>>>>>>>>>>> Objet : [Maria-developers] Patch for MDEV-10574 >>>>>>>>>>>> >>>>>>>>>>>> Alexander, >>>>>>>>>>>> Here is the patch I was talking about. >>>>>>>>>>>> It not address all aspects of MDEV-10574, especially, empty >>>>>>>>>>>> strings in table column are not view as null. >>>>>>>>>>>> But as in oracle mode, no empty string in columns should be >>>>>>>>>>>> created, an fully oracle application will never fall in this case. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Jérôme. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> -----Message d'origine----- >>>>>>>>>>>>> De : jerome brauge >>>>>>>>>>>>> Envoyé : mardi 19 septembre 2017 14:13 À : 'Alexander >> Barkov' >>>>>>>>>>>>> Objet : RE: Oracle strings functions compatibility : substr >>>>>>>>>>>>> >>>>>>>>>>>>> Hello Alexander, >>>>>>>>>>>>> Yes I'm aware. >>>>>>>>>>>>> This point will be treated by my next patch which will also >>>>>>>>>>>>> affect the others string functions (a part of MDEV-10574). >>>>>>>>>>>>> I will refresh this new patch with the current development >>>>>>>>>>>>> branch, and I submit to you. >>>>>>>>>>>>> Regards. >>>>>>>>>>>>> Jérôme. >>>>>>>>>>>>> >>>>>>>>>>>>>> -----Message d'origine----- De : Alexander Barkov >>>>>>>>>>>>>> [mailto:bar@mariadb.org] Envoyé : >>>> mardi >>>>>> 19 >>>>>>>>>>>>>> septembre 2017 13:16 À : jerome brauge Objet : Re: Oracle >>>>>>>>>>>>>> strings functions compatibility : substr >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Jerome, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I noticed one more difference in SUBSTR: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> SELECT SUBSTR('aaa',1,-1) FROM DUAL; >>>>>>>>>>>>>> >>>>>>>>>>>>>> MariaDB returns an empty string. >>>>>>>>>>>>>> Oracle returns NULL. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here's Oracle's documentation: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>> >> https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions162. >>>>>>>>>>>>>> ht >>>>>>>>>>>>>> m >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think we should do this in the same patch, to avoid >>>>>>>>>>>>>> behavior change in the future. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Would you like to try doing this additional change? >>>>>>>>>>>>>> Or would you like me to make an incremental patch on top >> of >>>>>> yours? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 09/19/2017 10:33 AM, jerome brauge wrote: >>>>>>>>>>>>>>> Hello Alexander, >>>>>>>>>>>>>>> It's done. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you have news from Sergei for MDEV-12874, MDEV- >> 13417 >>>>>> and >>>>>>>>>>>> MDEV- >>>>>>>>>>>>>> 13418 ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I asked Sergei, awaiting for his answer. I'll let you know. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you very much. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -----Message d'origine----- De : Alexander Barkov >>>>>>>>>>>>>>>> [mailto:bar@mariadb.org] Envoyé : lundi >>>>>>>>>>>>>>>> 18 septembre 2017 18:12 À : jerome brauge Objet : Re: >>>>>>>>>>>>>>>> Oracle strings functions compatibility : substr >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hello Jerome, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 09/18/2017 05:14 PM, jerome brauge wrote: >>>>>>>>>>>>>>>>> Hello Alexander, >>>>>>>>>>>>>>>>> I don't understand how I've missed such an obvious >>>> solution ! >>>>>>>>>>>>>>>>> Here is the new patch. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks. Now it looks simpler. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can you please do one small thing: >>>>>>>>>>>>>>>> move get_position() from "public:" to "protected:". >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ok to make a pull request after this change. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you very much! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards. >>>>>>>>>>>>>>>>> Jérôme. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -----Message d'origine----- De : Alexander Barkov >>>>>>>>>>>>>>>>>> [mailto:bar@mariadb.org] Envoyé : >>>>>> lundi >>>>>>>>>>>>>>>>>> 18 septembre 2017 11:30 À : jerome brauge Objet : Re: >>>>>> Oracle >>>>>>>>>>>>>>>>>> strings functions compatibility : substr >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hello Jerome, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 08/17/2017 04:43 PM, jerome brauge wrote: >>>>>>>>>>>>>>>>>>> Hello Alexander, >>>>>>>>>>>>>>>>>>> Here is a patch for function SUBSTR in >> sql_mode=oracle >>>>>>>>>>>>>>>>>>> (If position is 0, >>>>>>>>>>>>>>>>>> then it is treated as 1). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm thinking of how to get a smaller patch. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Wouldn't it be possible to introduce: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> virtual longlong get_position(); >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> longlong Item_func_substr::get_position() { >>>>>>>>>>>>>>>>>> return args[1]->val_int(); } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> longlong Item_func_substr_oracle::get_position() >>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>> longlong pos= args[1]->val_int(); >>>>>>>>>>>>>>>>>> return pos == 0 ? 1 : pos; } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> and just replace all calls for args[1]->val_int() to >>>>>> get_position() >>>>>>>> ? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> Jérôme. >>>>>>>>>>>>>>>>>>>

2 2

Re: [Maria-developers] [Commits] e0a1e63: MDEV-9619: Assertion `null_ref_table' failed in virtual table_map Item_direct_view_ref::used_tables() const on 2nd execution of PS
by Sergei Golubchik 13 Oct '17

13 Oct '17

Hi, Oleksandr! On Feb 18, Oleksandr Byelkin wrote: > revision-id: e0a1e632e4e6084cd989f217943618f4e9ac9179 (mariadb-5.5.54-25-ge0a1e63) > parent(s): 29d78dbb44ee9890b6bc28873344f20fc9157928 > committer: Oleksandr Byelkin > timestamp: 2017-02-18 17:47:31 +0100 > message: > > MDEV-9619: Assertion `null_ref_table' failed in virtual table_map > Item_direct_view_ref::used_tables() const on 2nd execution of PS > > Refer left expression indirectly in case it changes from execution to > execution. Looks reasonable. Ok to push with these changes: 1. replace - DBUG_ASSERT(null_ref_table); + DBUG_ASSERT(fixed); as discussed. 2. Remove bool Item_direct_view_ref::fix_fields(THD *thd, Item **reference) { - DBUG_ASSERT(1); these two changes should, probably, be done in a separate cleanup commit. 3. > diff --git a/sql/item.h b/sql/item.h > index 89155ac..dff7bc4 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1543,6 +1543,8 @@ class Item { > virtual void mark_as_condition_AND_part(TABLE_LIST *embedding) {}; > }; > > +typedef Item (*Item_ptr); > + remove this, your patch compiles just fine without it. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Please review MDEV-10802 TIMESTAMP NOT NULL field with explicit_defaults_for_timestamp and NO_ZERO_DATE shouldn't throw error
by Alexander Barkov 13 Oct '17

13 Oct '17

Hi Sergei, Please review a patch for MDEV-10802. Thanks!

2 2

Re: [Maria-developers] [Commits] 06bb7ee: MDEV-10892 - rpl.rpl_semi_sync_uninstall_plugin fails with Assertion `0' failure
by Sergei Golubchik 13 Oct '17

13 Oct '17

Hi, Sergey! On Oct 12, Sergey Vojtovich wrote: > revision-id: 06bb7eef9882536fc1d99b4462a65b53ec9f137e (mariadb-5.5.57-14-g06bb7ee) > parent(s): 93aadda513d8b5c2b49001514e235c4fdd73e08a > committer: Sergey Vojtovich > timestamp: 2017-10-12 15:37:45 +0400 > message: > > MDEV-10892 - rpl.rpl_semi_sync_uninstall_plugin fails with Assertion `0' failure > in buildbot > > Removed plugin_array_version: it is being checked without mutex protection and > thus is prone to data race and race conditions. In effect plugins are not > protected from concurrent destruction. > > Removed state_mask inversion: doesn't seem to make any sense. > > When collecting local plugins list, only add plugins that match state_mask. > > Use plugin ref counting to protect against concurrent plugin destruction. > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 6e04564..512548b 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -262,7 +262,7 @@ int fill_plugins(THD *thd, TABLE_LIST *tables, COND *cond) > TABLE *table= tables->table; > > if (plugin_foreach_with_mask(thd, show_plugins, MYSQL_ANY_PLUGIN, > - ~PLUGIN_IS_FREED, table)) > + ~(PLUGIN_IS_FREED | PLUGIN_IS_DYING), table)) > DBUG_RETURN(1); > > DBUG_RETURN(0); > @@ -5294,7 +5294,8 @@ int fill_schema_engines(THD *thd, TABLE_LIST *tables, COND *cond) > DBUG_ENTER("fill_schema_engines"); > if (plugin_foreach_with_mask(thd, iter_schema_engines, > MYSQL_STORAGE_ENGINE_PLUGIN, > - ~PLUGIN_IS_FREED, tables->table)) > + ~(PLUGIN_IS_FREED | PLUGIN_IS_DYING), > + tables->table)) > DBUG_RETURN(1); > DBUG_RETURN(0); > } agree, ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 96e982b: MDEV-7140 - Sporadic buildbot failures in rpl.rpl_semi_sync_uninstall_plugin
by Sergei Golubchik 13 Oct '17

13 Oct '17

Hi, Sergey! On Oct 12, Sergey Vojtovich wrote: > revision-id: 96e982bb1a72db47a41b97bb632b595b347d2956 (mariadb-5.5.57-14-g96e982b) > parent(s): 93aadda513d8b5c2b49001514e235c4fdd73e08a > committer: Sergey Vojtovich > timestamp: 2017-10-12 14:42:17 +0400 > message: > > MDEV-7140 - Sporadic buildbot failures in rpl.rpl_semi_sync_uninstall_plugin > > Removed plugin_array_version: it is being checked without mutex protection and > thus is prone to data race and race conditions. In effect plugins are not > protected from concurrent destruction. > > Removed state_mask inversion: doesn't seem to make any sense. > > When collecting local plugins list, only add plugins that match state_mask. > > Use plugin ref counting to protect against concurrent plugin destruction. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-13530 VARBINARY doesn't convert to to BLOB for sizes 65533, 65534 and 65535
by Alexander Barkov 12 Oct '17

12 Oct '17

Hi Sergei, Please review a patch for MDEV-13530. Thanks!

1 0

Re: [Maria-developers] 46b93fc: MDEV-13676: Field "create Procedure" is NULL, even if the the user has role which is the definer. (SHOW CREATE PROCEDURE)
by Sergei Golubchik 11 Oct '17

11 Oct '17

Hi, Vicentiu! Looks good! A couple of comments, see below. Ok to push after fixing, the second review is not needed. On Oct 09, Vicentiu Ciorbaru wrote: > revision-id: 46b93fc2a2fa198d721813d9b93bbe09d2e36eb3 (mariadb-10.0.32-50-g46b93fc) > parent(s): dbeffabc83ed01112e09d7e782d44f044cfcb691 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2017-10-09 13:32:40 +0300 > message: > > MDEV-13676: Field "create Procedure" is NULL, even if the the user has role which is the definer. (SHOW CREATE PROCEDURE) > > During show create procedure we ommited to check the current role, if it > is the actual definer of the procedure. In addition, we should support > indirectly granted roles to the current role. Implemented a recursive > lookup to search the tree of grants if the rolename is present. We both had to check the SQL standard to see what the behavior should be. Please add here (in the commit comment) a reference to the exact part of the standard that specifies this behavior. Like SQL Standard 2016, Part ... Section ... View I_S.ROUTINES selects ROUTINE_BODY and its WHERE clause says that the GRANTEE must be either PUBLIC, or CURRENT_USER or in the ENABLED_ROLES. > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > index ea9e1c1..3dd1a65 100644 > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -2588,10 +2588,18 @@ bool check_show_routine_access(THD *thd, sp_head *sp, bool *full_access) > *full_access= ((!check_table_access(thd, SELECT_ACL, &tables, FALSE, > 1, TRUE) && > (tables.grant.privilege & SELECT_ACL) != 0) || > + /* Check if user owns the routine. */ > (!strcmp(sp->m_definer_user.str, > thd->security_ctx->priv_user) && > !strcmp(sp->m_definer_host.str, > - thd->security_ctx->priv_host))); > + thd->security_ctx->priv_host)) || > + /* Check if current role or any of the sub-granted roles > + own the routine. */ > + (sp->m_definer_host.length == 0 && > + (!strcmp(sp->m_definer_user.str, > + thd->security_ctx->priv_role) || > + check_role_is_granted(thd->security_ctx->priv_role, NULL, > + sp->m_definer_user.str)))); Perhaps you could simplify the condition above a little bit, by replacing it with /* Check if current role or any of the sub-granted roles own the routine. */ (sp->m_definer_host.length == 0 && check_role_is_granted(thd->security_ctx->priv_user, thd->security_ctx->priv_host, sp->m_definer_user.str))); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] MDEV-14038 ALTER TABLE does not exit on error with InnoDB + bad default function
by Alexander Barkov 11 Oct '17

11 Oct '17

Hello Marko, please review a patch for MDEV-14038. Thanks!

2 1

[Maria-developers] MDEV-13972 crash in Item_func_sec_to_time::get_date
by Alexander Barkov 09 Oct '17

09 Oct '17

Hello Sergei, Please review a patch for MDEV-13972. The patch is for 5.5, as we agreed in slack. Thanks!

2 3

Re: [Maria-developers] [Commits] 0240b866abc: MDEV-13838: Wrong result after altering a partitioned table
by Marko Mäkelä 09 Oct '17

09 Oct '17

On Mon, Oct 9, 2017 at 3:00 PM, jan <jan.lindstrom(a)mariadb.com> wrote: > revision-id: 109463fdb86d (mariadb-10.0.32-54-g0240b866abc) > parent(s): 172cc70bf8c0aea3d2d0c73bcf94f36c172b769a > author: Jan Lindström > committer: Jan Lindström > timestamp: 2017-10-09 14:52:51 +0300 > message: > > MDEV-13838: Wrong result after altering a partitioned table > > Reverted incorrect changes done on MDEV-7367 and MDEV-9469. Fixes properly > also related bugs: > > MDEV-13668: InnoDB unnecessarily rebuilds table when renaming a column and > adding index > MDEV-9469: 'Incorrect key file' on ALTER TABLE > MDEV-9548: Alter table (renaming and adding index) fails with "Incorrect > key file for table" > MDEV-10535: ALTER TABLE causes standalone/wsrep cluster crash > MDEV-13640: ALTER TABLE CHANGE and ADD INDEX on auto_increment column > fails with "Incorrect key file for table..." > This is a very welcome change that looks correct to me. I only have a few comments on the formatting and tests. The patch does not cleanly apply to 10.0 due to some difference in the comments of removed functions. Later, you informed me that this is https://github.com/MariaDB/server/commit/a8db085c51ccfd6df737d9261abac9d2b0… on bb-10.0-MDEV-13838 <https://github.com/MariaDB/server/compare/bb-10.0-MDEV-13838>. +Warnings: > +Note 1831 Duplicate index `c4_2`. This is deprecated and will be > disallowed in a future release. > The intention of the test was not to create duplicated indexes. The "rename column" and "add index" operations were split into two ALTER TABLE in order to work around the limitation that was introduced in a previous incorrect commit. You should have removed the added ALTER TABLE statements instead of adding SHOW CREATE TABLE statements. Please refer to https://github.com/MariaDB/server/commit/f56bd70f51020c67a30415ae381ae93b90… and revert the relevant parts of the test changes. The other tests seem to be adjusted fine. > @@ -171,3 +172,39 @@ ALTER TABLE ticket > SHOW CREATE TABLE ticket; > > DROP TABLE ticket; > + > +# > +# MDEV-13838: Wrong result after altering a partitioned table > +# > + > +CREATE TABLE `t` ( > +`id` bigint(20) unsigned NOT NULL auto_increment, > +`d` date NOT NULL, > +`a` bigint(20) unsigned NOT NULL, > +`b` smallint(5) unsigned DEFAULT NULL, > +PRIMARY KEY (`id`,`d`) > +) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_cs > STATS_SAMPLE_PAGES=200 > +/*!50500 PARTITION BY RANGE COLUMNS(d) > +(PARTITION p20170913 VALUES LESS THAN ('2017-09-14') ENGINE = InnoDB, > +PARTITION p20170914 VALUES LESS THAN ('2017-09-15') ENGINE = InnoDB, > +PARTITION p20170915 VALUES LESS THAN ('2017-09-16') ENGINE = InnoDB, > +PARTITION p20170916 VALUES LESS THAN ('2017-09-17') ENGINE = InnoDB, > +PARTITION p20170917 VALUES LESS THAN ('2017-09-18') ENGINE = InnoDB, > +PARTITION p99991231 VALUES LESS THAN (MAXVALUE) ENGINE = InnoDB) */; > Please clean up the CREATE TABLE statement. Remove trailing space and unnecessary quotes and comments. > +insert into t(d,a,b) values ('2017-09-15',rand()*10000,rand()*10); > +insert into t(d,a,b) values ('2017-09-15',rand()*10000,rand()*10); > + > +replace into t(d,a,b) select '2017-09-15',rand()*10000,rand()*10 from t > t1, t t2, t t3, t t4, t t5, t t6, t t7, t t8, t t9, t t10, t t11, t t12, t > t13, t t14; > Does the table really have to be this big for the bug to repeat? Can you please reduce the test? Shouldn’t it work with 1 or 2 partitions and a few records in each? > + > +select count(*) from t where d ='2017-09-15'; > + > +ALTER TABLE t CHANGE b c smallint(5) unsigned , ADD KEY idx_d_a (d, a); > +SHOW CREATE TABLE t; > +analyze table t; > + > +select count(*) from t where d ='2017-09-15'; > +select count(*) from t force index(primary) where d ='2017-09-15'; > + > +DROP TABLE t; > + > Please do not end files in empty lines. > @@ -1625,13 +1606,9 @@ innobase_create_index_def( > index->ind_type |= DICT_FTS; > } > > - if (!new_clustered) { > - altered_table = NULL; > - } > - > for (i = 0; i < n_fields; i++) { > - innobase_create_index_field_def( > - altered_table, &key->key_part[i], > &index->fields[i], fields); > + innobase_create_index_field_def(new_clustered, > + altered_table, &key->key_part[i], > &index->fields[i]); > } > > DBUG_VOID_RETURN; > Do not add the first parameter to the same line with the function name. Instead, add it on a line on its own. In this way, it follows the indentation rules set by (defun innodb-cc-mode (&optional map) "InnoDB adjustments to cc-mode. In MAP, binds C-c C-c to 'compile." (if (and (buffer-file-name) (string-match "/\$innobase\\|xtradb\$.*/.*[ihc]\\'" (buffer-file-name))) (progn (setq mode-name (concat "Ib" mode-name) c-basic-offset 8 c-label-minimum-indentation 0) (add-to-list 'c-offsets-alist '(c . 0)) (add-to-list 'c-offsets-alist '(arglist-intro . +)) (add-to-list 'c-offsets-alist '(label . [0])))) (if map (define-key map "\C-c\C-c" 'compile))) > @@ -2121,7 +2097,7 @@ struct ha_innobase_inplace_ctx : public > inplace_alter_handler_ctx > /** mapping of old column numbers to new ones, or NULL */ > const ulint* col_map; > /** new column names, or NULL if nothing was renamed */ > - const char** col_names; > + const char** col_names; > /** added AUTO_INCREMENT column position, or ULINT_UNDEFINED */ > const ulint add_autoinc; > /** default values of ADD COLUMN, or NULL */ > Please do not change TAB to spaces. > @@ -3067,7 +3043,7 @@ prepare_inplace_alter_table_dict( > > ctx->add_index[a] = row_merge_create_index( > ctx->trx, ctx->new_table, > - &index_defs[a], ctx->col_names); > + &index_defs[a]); > > add_key_nums[a] = index_defs[a].key_number; > Consider joining the line after new_table. Best regards, Marko

1 0

[Maria-developers] MDEV-11819 NO_ZERO_IN_DATE: Incorrect generated column value
by Alexander Barkov 06 Oct '17

06 Oct '17

Hello Sergei, Please review a patch for MDEV-11819. Thanks.

2 1

[Maria-developers] Add RMariaDB CRAN Link to MariaDB Foundation Download Webpage
by Juan Telleria 06 Oct '17

06 Oct '17

Dear R Developers, A new package has been released for the "R Statistical Programming Language" in which the R community has put a great effort into it: RMariaDB. It enables a robust data transfer between R and MariaDB. The link on CRAN (R package repository) can be accessed from: https://cran.r-project.org/web/packages/RMariaDB/index.html And can be installed in R by using the following statement: install.packages("RMariaDB") library(RMariaDB) So, just as PHP community developed connector was distributed in the past within MySQL webpage, I do think that this connector could benefit from being distributed (or having a link to CRAN) from the MariaDB Foundation Download Webpage. I believe that many people has putted a great effort on this connector, and it would benefit from greater visibility for better support and maintenance. Thank you, Juan

2 1

Re: [Maria-developers] [Commits] 3f7e619647f: MDEV-13928: Missing symbols building RocksDB on macOS 10.12.6
by Sergei Golubchik 05 Oct '17

05 Oct '17

Hi, Varun! Why did it fail only on MacOS X? On Oct 06, Varun wrote: > revision-id: 3f7e619647f7bc1ec31a6118b403da805c48ae51 (mariadb-10.2.5-653-g3f7e619647f) > parent(s): 08c493c62afae0e81f9da6bb9b86efe99a9a763d > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-10-06 00:20:08 +0530 > message: > > MDEV-13928: Missing symbols building RocksDB on macOS 10.12.6 > > rdb_sst_info.cc must be compiled with RTTI > > --- > storage/rocksdb/CMakeLists.txt | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/storage/rocksdb/CMakeLists.txt b/storage/rocksdb/CMakeLists.txt > index 6db82119b45..80a6614b757 100644 > --- a/storage/rocksdb/CMakeLists.txt > +++ b/storage/rocksdb/CMakeLists.txt > @@ -170,6 +170,7 @@ IF(CMAKE_CXX_COMPILER_ID MATCHES "GNU" OR CMAKE_CXX_COMPILER_ID MATCHES "Clang") > # includes requires it. So, now everything in MariaRocks is compiled with > # -frtti) > set_source_files_properties(event_listener.cc rdb_cf_options.cc > + rdb_sst_info.cc > PROPERTIES COMPILE_FLAGS -frtti) > ENDIF() Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-13417 / MDEV - 13418 The order of evaluation of SELECT..INTO assignments
by jerome brauge 03 Oct '17

03 Oct '17

Hello Sergei, Here is a patch for MDEV-13417 and MDEV-13418. Until you choose the right name for this new sql_mode, I temporarily chosen "SET_CONSISTENCY". It is set only for sql_mode=oracle for now. Can you review it ? Thank you. Regard.

2 6

[Maria-developers] MDEV-10596
by jerome brauge 27 Sep '17

27 Sep '17

Hi Alexander, In sql_mode=oracle, when a stored procedure varchar parameter is defined without size, you have chosen to translate it by VARCHAR(4000) (or CHAR(2000) for CHAR parameter). Oracle does not work like this. Size is inherited from the size of argument at runtime. Example: CREATE OR REPLACE PROCEDURE p1(p OUT VARCHAR) AS BEGIN p:='0123456789'; END; / declare w varchar(10); begin p1(w); end; / --> work fine declare w varchar(8); begin p1(w); end; / --> failed with : ERROR at line 1: ORA-06502: PL/SQL: numeric or value error: character string buffer too small ORA-06512: at "OPS$DBO.P1", line 4 ORA-06512: at line 3 Furthermore, since Oracle 9, VARCHAR datatype in PL/SQL is not limited to 4000 char but to 32k. It's the size of varchar column in a table that is limited to 4000 (until Oracle 12C which allow 32k when MAX_STRING_SIZE=EXTENDED). I've done the attached patch to resize these parameters at runtime. What do you think about it ? Another question : Oracle does not allow to change the value of an IN parameter. The following procedure cannot be compile : CREATE OR REPLACE PROCEDURE p1(p IN VARCHAR) AS BEGIN p:='0123456789'; END; / Warning: Procedure created with compilation errors. //sun10:1521/CS> select * from user_errors; NAME TYPE SEQUENCE LINE POSITION ------------------------------ ------------ ---------- ---------- ---------- TEXT ------------------------------------------------------------------------------------------------------------------------------------ ATTRIBUTE MESSAGE_NUMBER --------- -------------- P1 PROCEDURE 1 4 3 PLS-00363: expression 'P' cannot be used as an assignment target ERROR 363 Mariadb allow this. It's fine for us but do you have plan to change this behavior ? (or add a new STRICT_xxx mode ?) Regard, Jérôme.

2 3

[Maria-developers] GCC7 warnings
by Reindl Harald 24 Sep '17

24 Sep '17

hard to say how serious they are but maybe a good idea to take a look /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/mariadb_config/mariadb_config.c:9:49: warning: backslash and newline separated by space #define LIBS "-L/usr/lib64/mysql/ -lmariadb "\ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_lib.c: In function 'mariadb_get_infov': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_lib.c:3872:30: warning: this statement may fall through [-Wimplicit-fallthrough=] *((unsigned long *)arg)= mysql->client_flag; ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_lib.c:3875:3: note: here default: ^~~~~~~ In file included from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/plugin/handler_socket/libhsclient/hstcpcli.cpp:13:0: /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/plugin/handler_socket/libhsclient/hstcpcli.hpp:33:14: warning: 'template<class> class std::auto_ptr' is deprecated [-Wdeprecated-declarations] typedef std::auto_ptr<hstcpcli_i> hstcpcli_ptr; ^~~~~~~~ In file included from /usr/include/c++/7/memory:80:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/plugin/handler_socket/libhsclient/hstcpcli.hpp:15, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/plugin/handler_socket/libhsclient/hstcpcli.cpp:13: /usr/include/c++/7/bits/unique_ptr.h:51:28: note: declared here template<typename> class auto_ptr; /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c: In function 'dtoa': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c:1465:14: warning: this statement may fall through [-Wimplicit-fallthrough=] leftright= 0; ~~~~~~~~~^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c:1467:3: note: here case 4: ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c:1473:14: warning: this statement may fall through [-Wimplicit-fallthrough=] leftright= 0; ~~~~~~~~~^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c:1475:3: note: here case 5: ^~~~ At top level: /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c:1161:21: warning: 'tinytens' defined but not used [-Wunused-const-variable=] static const double tinytens[]= ^~~~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_dyncol.c: In function 'mariadb_dyncol_val_double': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_dyncol.c:4044:7: warning: this statement may fall through [-Wimplicit-fallthrough=] { ^ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_dyncol.c:4060:5: note: here case DYN_COL_DATETIME: ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_dyncol.c:4050:15: warning: argument 1 null where non-null expected [-Wnonnull] *dbl= strtod(str, &end); ^~~~~~~~~~~~~~~~~ In file included from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/include/ma_global.h:168:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_dyncol.c:64: /usr/include/stdlib.h:126:15: note: in a call to function 'strtod' declared here extern double strtod (const char *__restrict __nptr, ^~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/unittest/libmariadb/t_aurora.c: In function 'test_reconnect': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/unittest/libmariadb/t_aurora.c:100:3: warning: ignoring return value of 'system', declared with attribute warn_unused_result [-Wunused-result] system("/usr/local/aws/bin/aws rds failover-db-cluster --db-cluster-identifier instance-1-cluster"); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/mysys/my_lib.c: In function 'my_dir': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/mysys/my_lib.c:144:3: warning: 'readdir_r' is deprecated [-Wdeprecated-declarations] while (!(READDIR(dirp,(struct dirent*) dirent_tmp,dp))) ^~~~~ In file included from /usr/include/features.h:410:0, from /usr/include/pthread.h:21, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_global.h:278, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/mysys/mysys_priv.h:19, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/mysys/my_lib.c:19: /usr/include/dirent.h:189:12: note: declared here extern int __REDIRECT (readdir_r, /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/myisamchk.c: In function 'myisamchk': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/myisamchk.c:1411:22: warning: '%ld' directive writing between 1 and 10 bytes into a region of size 8 [-Wformat-overflow=] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/myisamchk.c:1411:21: note: directive argument in the range [1, 4294967296] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/myisam.h:26, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/myisamdef.h:20, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/fulltext.h:21, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/myisamchk.c:18: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 2 and 11 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/mi_test3.c: In function 'test_write': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/mi_test3.c:367:30: warning: '%7ld' directive writing between 7 and 11 bytes into a region of size 8 [-Wformat-overflow=] sprintf((char*) record.id,"%7ld",(long) getpid()); ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/mi_test3.c:367:29: note: directive argument in the range [-2147483648, 2147483647] sprintf((char*) record.id,"%7ld",(long) getpid()); ^~~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/myisam.h:26, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/mi_test3.c:20: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 8 and 12 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_chk.c: In function 'maria_chk': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_chk.c:1703:22: warning: '%ld' directive writing between 1 and 10 bytes into a region of size 8 [-Wformat-overflow=] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_chk.c:1703:21: note: directive argument in the range [1, 4294967296] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/maria.h:22, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_def.h:18, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/ma_fulltext.h:20, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_chk.c:18: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 2 and 11 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/ma_test3.c: In function 'test_write': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/ma_test3.c:365:30: warning: '%7ld' directive writing between 7 and 11 bytes into a region of size 8 [-Wformat-overflow=] sprintf((char*) record.id,"%7ld", (long) getpid()); ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/ma_test3.c:365:29: note: directive argument in the range [-2147483648, 2147483647] sprintf((char*) record.id,"%7ld", (long) getpid()); ^~~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/maria.h:22, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_def.h:18, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/ma_test3.c:20: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 8 and 12 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1 0

Re: [Maria-developers] [Commits] 2846574452e: MDEV-11553 Can't restore a PERSISTENT column that uses DATE_FORMAT()
by Alexander Barkov 23 Sep '17

23 Sep '17

Hello Sergei, Ok to push. One small suggestion. Instead of having a standalone function, please move get_locale as method in Item. Something like this: const MY_LOCALE *Item::get_locale() { StringBuffer<MAX_FIELD_WIDTH> tmp; String *locale_name= val_str_ascii(&tmp); const MY_LOCALE *lc; if (!locale_name || !(lc= my_locale_by_name(locale_name->c_ptr_safe()))) { THD *thd= current_thd; push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, ER_UNKNOWN_LOCALE, ER_THD(thd, ER_UNKNOWN_LOCALE), locale_name ? locale_name->c_ptr_safe() : "NULL"); lc= &my_locale_en_US; } return lc; } Thanks! On 09/22/2017 10:30 PM, serg(a)mariadb.org wrote: > revision-id: 2846574452e82b9f391ca5d41d5021ac94730dec (base-0.6-627-g2846574452e) > parent(s): 5a8b6e38967f1e4aa7fe4885e54ee8ea44097e8e > author: Sergei Golubchik > committer: Sergei Golubchik > timestamp: 2017-09-22 20:24:53 +0200 > message: > > MDEV-11553 Can't restore a PERSISTENT column that uses DATE_FORMAT() > > 3-argument form of DATE_FORMAT > > --- > mysql-test/r/locale.result | 71 +++++++++++++++++++++- > .../suite/vcol/r/vcol_blocked_sql_funcs.result | 2 + > .../suite/vcol/r/vcol_supported_sql_funcs.result | 15 +++++ > .../suite/vcol/t/vcol_blocked_sql_funcs_main.inc | 3 + > .../suite/vcol/t/vcol_supported_sql_funcs_main.inc | 6 ++ > mysql-test/t/locale.test | 28 +++++++++ > sql/item_create.cc | 27 +------- > sql/item_strfunc.cc | 10 +-- > sql/item_strfunc.h | 4 +- > sql/item_timefunc.cc | 25 ++++++-- > sql/item_timefunc.h | 27 +++++--- > sql/lex.h | 1 + > sql/sql_yacc.yy | 16 ++++- > sql/sql_yacc_ora.yy | 16 ++++- > 14 files changed, 202 insertions(+), 49 deletions(-) > > diff --git a/mysql-test/r/locale.result b/mysql-test/r/locale.result > index a02e80ed21e..b750d038205 100644 > --- a/mysql-test/r/locale.result > +++ b/mysql-test/r/locale.result > @@ -229,15 +229,80 @@ SET lc_time_names=@old_50915_lc_time_names; > # End of 10.1 tests > # > create view v1 as select > +date_format('2001-10-02', '%c %b %M') as a, > +date_format('2001-10-02', '%c %b %M', 'ru_RU') as a1, > format(123456789,2) as b, > format(123456789,2,'rm_CH') as b1; > select * from v1; > -b b1 > -123,456,789.00 123'456'789,00 > +a a1 b b1 > +10 Oct October 10 Окт Октября 123,456,789.00 123'456'789,00 > show create view v1; > View Create View character_set_client collation_connection > -v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select format(123456789,2) AS `b`,format(123456789,2,'rm_CH') AS `b1` utf8 utf8_general_ci > +v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select date_format('2001-10-02','%c %b %M') AS `a`,date_format('2001-10-02','%c %b %M','ru_RU') AS `a1`,format(123456789,2) AS `b`,format(123456789,2,'rm_CH') AS `b1` utf8 utf8_general_ci > drop view v1; > # > # End of 10.2 tests > # > +select date_format('2001-01-01', '%w %a %W', 'ro_RO'); > +date_format('2001-01-01', '%w %a %W', 'ro_RO') > +1 Lu Luni > +select date_format('2001-01-03', '%w %a %W', 'ro_RO'); > +date_format('2001-01-03', '%w %a %W', 'ro_RO') > +3 Mi Miercuri > +select date_format('2001-01-05', '%w %a %W', 'ro_RO'); > +date_format('2001-01-05', '%w %a %W', 'ro_RO') > +5 Vi Vineri > +select date_format('2001-01-07', '%w %a %W', 'ro_RO'); > +date_format('2001-01-07', '%w %a %W', 'ro_RO') > +0 Du Duminică > +select date_format('2001-01-01', '%w %a %W', 'de_AT'); > +date_format('2001-01-01', '%w %a %W', 'de_AT') > +1 Mon Montag > +select date_format('2001-02-01', '%w %a %W', 'de_AT'); > +date_format('2001-02-01', '%w %a %W', 'de_AT') > +4 Don Donnerstag > +select date_format('2001-03-01', '%w %a %W', 'de_AT'); > +date_format('2001-03-01', '%w %a %W', 'de_AT') > +4 Don Donnerstag > +select date_format('2001-01-01', '%w %a %W', 'en_US'); > +date_format('2001-01-01', '%w %a %W', 'en_US') > +1 Mon Monday > +select date_format('2001-03-01', '%c %b %M', 'en_US'); > +date_format('2001-03-01', '%c %b %M', 'en_US') > +3 Mar March > +select date_format('2001-01-01', '%w %a %W', 'rm_CH'); > +date_format('2001-01-01', '%w %a %W', 'rm_CH') > +1 gli glindesdi > +select date_format('2001-01-03', '%w %a %W', 'rm_CH'); > +date_format('2001-01-03', '%w %a %W', 'rm_CH') > +3 me mesemna > +select date_format('2001-01-05', '%w %a %W', 'rm_CH'); > +date_format('2001-01-05', '%w %a %W', 'rm_CH') > +5 ve venderdi > +select date_format('2001-01-07', '%w %a %W', 'rm_CH'); > +date_format('2001-01-07', '%w %a %W', 'rm_CH') > +0 du dumengia > +select date_format('2001-02-01', '%c %b %M', 'rm_CH'); > +date_format('2001-02-01', '%c %b %M', 'rm_CH') > +2 favr favrer > +select date_format('2001-04-01', '%c %b %M', 'rm_CH'); > +date_format('2001-04-01', '%c %b %M', 'rm_CH') > +4 avr avrigl > +select date_format('2001-06-01', '%c %b %M', 'rm_CH'); > +date_format('2001-06-01', '%c %b %M', 'rm_CH') > +6 zercl zercladur > +select date_format('2001-08-01', '%c %b %M', 'rm_CH'); > +date_format('2001-08-01', '%c %b %M', 'rm_CH') > +8 avust avust > +select date_format('2001-10-01', '%c %b %M', 'rm_CH'); > +date_format('2001-10-01', '%c %b %M', 'rm_CH') > +10 oct october > +select date_format('2001-12-01', '%c %b %M', 'rm_CH'); > +date_format('2001-12-01', '%c %b %M', 'rm_CH') > +12 dec december > +select date_format('2001-01-06', '%w %a %W', 'de_CH'); > +date_format('2001-01-06', '%w %a %W', 'de_CH') > +6 Sa Samstag > +select date_format('2001-09-01', '%c %b %M', 'de_CH'); > +date_format('2001-09-01', '%c %b %M', 'de_CH') > +9 Sep September > diff --git a/mysql-test/suite/vcol/r/vcol_blocked_sql_funcs.result b/mysql-test/suite/vcol/r/vcol_blocked_sql_funcs.result > index 6605f7c99b4..2d5bcfe553d 100644 > --- a/mysql-test/suite/vcol/r/vcol_blocked_sql_funcs.result > +++ b/mysql-test/suite/vcol/r/vcol_blocked_sql_funcs.result > @@ -247,3 +247,5 @@ ERROR HY000: Expression in the GENERATED ALWAYS AS clause is too big > # Constant expression > create or replace table t1 (a int as (PI()) PERSISTENT); > drop table if exists t1; > +create table t1 (a timestamp, b varchar(255) as (date_format(a, '%w %a %m %b')) stored); > +ERROR HY000: Function or expression 'date_format()' cannot be used in the GENERATED ALWAYS AS clause of `b` > diff --git a/mysql-test/suite/vcol/r/vcol_supported_sql_funcs.result b/mysql-test/suite/vcol/r/vcol_supported_sql_funcs.result > index 3fa4f6e1431..10ab32f9486 100644 > --- a/mysql-test/suite/vcol/r/vcol_supported_sql_funcs.result > +++ b/mysql-test/suite/vcol/r/vcol_supported_sql_funcs.result > @@ -2894,6 +2894,21 @@ a b > 2012-12-12 Wednesday Wed December Dec > drop table t1; > set sql_warnings = 0; > +# DATE_FORMAT() STORED > +set sql_warnings = 1; > +create table t1 (a date, b varchar(100) as (date_format(a, '%W %a %M %b', 'de_DE')) STORED); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` date DEFAULT NULL, > + `b` varchar(100) GENERATED ALWAYS AS (date_format(`a`,'%W %a %M %b','de_DE')) STORED > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values ('2012-12-12',default); > +select * from t1; > +a b > +2012-12-12 Mittwoch Mi Dezember Dez > +drop table t1; > +set sql_warnings = 0; > # CURRENT_USER() > set sql_warnings = 1; > create table t1 (a char, b varchar(32) as (current_user())); > diff --git a/mysql-test/suite/vcol/t/vcol_blocked_sql_funcs_main.inc b/mysql-test/suite/vcol/t/vcol_blocked_sql_funcs_main.inc > index 766d0c7410c..37de02a2b29 100644 > --- a/mysql-test/suite/vcol/t/vcol_blocked_sql_funcs_main.inc > +++ b/mysql-test/suite/vcol/t/vcol_blocked_sql_funcs_main.inc > @@ -354,3 +354,6 @@ eval create or replace table t1 (a int, b varchar(16384) as (concat(a,'$tmp_long > create or replace table t1 (a int as (PI()) PERSISTENT); > > drop table if exists t1; > + > +--error ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED > +create table t1 (a timestamp, b varchar(255) as (date_format(a, '%w %a %m %b')) stored); > diff --git a/mysql-test/suite/vcol/t/vcol_supported_sql_funcs_main.inc b/mysql-test/suite/vcol/t/vcol_supported_sql_funcs_main.inc > index dafc42098dd..fa2162e5981 100644 > --- a/mysql-test/suite/vcol/t/vcol_supported_sql_funcs_main.inc > +++ b/mysql-test/suite/vcol/t/vcol_supported_sql_funcs_main.inc > @@ -1197,6 +1197,12 @@ let $values1 = '2012-12-12',default; > let $rows = 1; > --source suite/vcol/inc/vcol_supported_sql_funcs.inc > > +--echo # DATE_FORMAT() STORED > +let $cols = a date, b varchar(100) as (date_format(a, '%W %a %M %b', 'de_DE')) STORED; > +let $values1 = '2012-12-12',default; > +let $rows = 1; > +--source suite/vcol/inc/vcol_supported_sql_funcs.inc > + > --echo # CURRENT_USER() > let $cols = a char, b varchar(32) as (current_user()); > let $values1 = 'a', default; > diff --git a/mysql-test/t/locale.test b/mysql-test/t/locale.test > index 4570b968ecf..a9a507bc387 100644 > --- a/mysql-test/t/locale.test > +++ b/mysql-test/t/locale.test > @@ -144,6 +144,8 @@ SET lc_time_names=@old_50915_lc_time_names; > > # Item::print > create view v1 as select > + date_format('2001-10-02', '%c %b %M') as a, > + date_format('2001-10-02', '%c %b %M', 'ru_RU') as a1, > format(123456789,2) as b, > format(123456789,2,'rm_CH') as b1; > select * from v1; > @@ -153,3 +155,29 @@ drop view v1; > --echo # > --echo # End of 10.2 tests > --echo # > + > +# > +# MDEV-11553 Can't restore a PERSISTENT column that uses DATE_FORMAT() > +# > +# 3-argument syntax for DATE_FORMAT() > +select date_format('2001-01-01', '%w %a %W', 'ro_RO'); > +select date_format('2001-01-03', '%w %a %W', 'ro_RO'); > +select date_format('2001-01-05', '%w %a %W', 'ro_RO'); > +select date_format('2001-01-07', '%w %a %W', 'ro_RO'); > +select date_format('2001-01-01', '%w %a %W', 'de_AT'); > +select date_format('2001-02-01', '%w %a %W', 'de_AT'); > +select date_format('2001-03-01', '%w %a %W', 'de_AT'); > +select date_format('2001-01-01', '%w %a %W', 'en_US'); > +select date_format('2001-03-01', '%c %b %M', 'en_US'); > +select date_format('2001-01-01', '%w %a %W', 'rm_CH'); > +select date_format('2001-01-03', '%w %a %W', 'rm_CH'); > +select date_format('2001-01-05', '%w %a %W', 'rm_CH'); > +select date_format('2001-01-07', '%w %a %W', 'rm_CH'); > +select date_format('2001-02-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-04-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-06-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-08-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-10-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-12-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-01-06', '%w %a %W', 'de_CH'); > +select date_format('2001-09-01', '%c %b %M', 'de_CH'); > diff --git a/sql/item_create.cc b/sql/item_create.cc > index e0bfe7a4402..88c1ac1f64f 100644 > --- a/sql/item_create.cc > +++ b/sql/item_create.cc > @@ -770,19 +770,6 @@ class Create_func_crosses : public Create_func_arg2 > #endif > > > -class Create_func_date_format : public Create_func_arg2 > -{ > -public: > - virtual Item *create_2_arg(THD *thd, Item *arg1, Item *arg2); > - > - static Create_func_date_format s_singleton; > - > -protected: > - Create_func_date_format() {} > - virtual ~Create_func_date_format() {} > -}; > - > - > class Create_func_datediff : public Create_func_arg2 > { > public: > @@ -4023,15 +4010,6 @@ Create_func_crosses::create_2_arg(THD *thd, Item *arg1, Item *arg2) > #endif > > > -Create_func_date_format Create_func_date_format::s_singleton; > - > -Item* > -Create_func_date_format::create_2_arg(THD *thd, Item *arg1, Item *arg2) > -{ > - return new (thd->mem_root) Item_func_date_format(thd, arg1, arg2, 0); > -} > - > - > Create_func_datediff Create_func_datediff::s_singleton; > > Item* > @@ -4530,7 +4508,7 @@ Create_func_from_unixtime::create_native(THD *thd, LEX_CSTRING *name, > Item *param_1= item_list->pop(); > Item *param_2= item_list->pop(); > Item *ut= new (thd->mem_root) Item_func_from_unixtime(thd, param_1); > - func= new (thd->mem_root) Item_func_date_format(thd, ut, param_2, 0); > + func= new (thd->mem_root) Item_func_date_format(thd, ut, param_2); > break; > } > default: > @@ -6483,7 +6461,7 @@ Create_func_time_format Create_func_time_format::s_singleton; > Item* > Create_func_time_format::create_2_arg(THD *thd, Item *arg1, Item *arg2) > { > - return new (thd->mem_root) Item_func_date_format(thd, arg1, arg2, 1); > + return new (thd->mem_root) Item_func_time_format(thd, arg1, arg2); > } > > > @@ -6844,7 +6822,6 @@ static Native_func_registry func_array[] = > { { C_STRING_WITH_LEN("CRC32") }, BUILDER(Create_func_crc32)}, > { { C_STRING_WITH_LEN("CROSSES") }, GEOM_BUILDER(Create_func_crosses)}, > { { C_STRING_WITH_LEN("DATEDIFF") }, BUILDER(Create_func_datediff)}, > - { { C_STRING_WITH_LEN("DATE_FORMAT") }, BUILDER(Create_func_date_format)}, > { { C_STRING_WITH_LEN("DAYNAME") }, BUILDER(Create_func_dayname)}, > { { C_STRING_WITH_LEN("DAYOFMONTH") }, BUILDER(Create_func_dayofmonth)}, > { { C_STRING_WITH_LEN("DAYOFWEEK") }, BUILDER(Create_func_dayofweek)}, > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 83432f95612..3e12a545c97 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -2668,11 +2668,11 @@ String *Item_func_soundex::val_str(String *str) > const int FORMAT_MAX_DECIMALS= 30; > > > -MY_LOCALE *Item_func_format::get_locale(Item *item) > +const MY_LOCALE *get_locale(Item *item) > { > - DBUG_ASSERT(arg_count == 3); > - String tmp, *locale_name= args[2]->val_str_ascii(&tmp); > - MY_LOCALE *lc; > + StringBuffer<MAX_FIELD_WIDTH> tmp; > + String *locale_name= item->val_str_ascii(&tmp); > + const MY_LOCALE *lc; > if (!locale_name || > !(lc= my_locale_by_name(locale_name->c_ptr_safe()))) > { > @@ -2712,7 +2712,7 @@ String *Item_func_format::val_str_ascii(String *str) > int dec; > /* Number of characters used to represent the decimals, including '.' */ > uint32 dec_length; > - MY_LOCALE *lc; > + const MY_LOCALE *lc; > DBUG_ASSERT(fixed == 1); > > dec= (int) args[1]->val_int(); > diff --git a/sql/item_strfunc.h b/sql/item_strfunc.h > index 685a64816be..542af13df5c 100644 > --- a/sql/item_strfunc.h > +++ b/sql/item_strfunc.h > @@ -28,6 +28,7 @@ > extern size_t username_char_length; > > class MY_LOCALE; > +const MY_LOCALE *get_locale(Item *item); > > class Item_str_func :public Item_func > { > @@ -932,14 +933,13 @@ class Item_func_make_set :public Item_str_func > > class Item_func_format :public Item_str_ascii_func > { > - MY_LOCALE *locale; > + const MY_LOCALE *locale; > public: > Item_func_format(THD *thd, Item *org, Item *dec): > Item_str_ascii_func(thd, org, dec) {} > Item_func_format(THD *thd, Item *org, Item *dec, Item *lang): > Item_str_ascii_func(thd, org, dec, lang) {} > > - MY_LOCALE *get_locale(Item *item); > String *val_str_ascii(String *); > void fix_length_and_dec(); > const char *func_name() const { return "format"; } > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > index 86bde00cced..5e660c58583 100644 > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -455,7 +455,8 @@ static bool extract_date_time(DATE_TIME_FORMAT *format, > */ > > static bool make_date_time(DATE_TIME_FORMAT *format, MYSQL_TIME *l_time, > - timestamp_type type, MY_LOCALE *locale, String *str) > + timestamp_type type, const MY_LOCALE *locale, > + String *str) > { > char intbuff[15]; > uint hours_i; > @@ -1833,7 +1834,14 @@ bool Item_func_sec_to_time::get_date(MYSQL_TIME *ltime, ulonglong fuzzy_date) > void Item_func_date_format::fix_length_and_dec() > { > THD* thd= current_thd; > - locale= thd->variables.lc_time_names; > + if (!is_time_format) > + { > + if (arg_count < 3) > + locale= thd->variables.lc_time_names; > + else > + if (args[2]->basic_const_item()) > + locale= get_locale(args[2]); > + } > > /* > Must use this_item() in case it's a local SP variable > @@ -1875,6 +1883,8 @@ bool Item_func_date_format::eq(const Item *item, bool binary_cmp) const > if (this == item) > return 1; > item_func= (Item_func_date_format*) item; > + if (arg_count != item_func->arg_count) > + return 0; > if (!args[0]->eq(item_func->args[0], binary_cmp)) > return 0; > /* > @@ -1884,6 +1894,8 @@ bool Item_func_date_format::eq(const Item *item, bool binary_cmp) const > */ > if (!args[1]->eq(item_func->args[1], 1)) > return 0; > + if (arg_count > 2 && !args[2]->eq(item_func->args[2], 1)) > + return 0; > return 1; > } > > @@ -1967,15 +1979,18 @@ String *Item_func_date_format::val_str(String *str) > String *format; > MYSQL_TIME l_time; > uint size; > - int is_time_flag = is_time_format ? TIME_TIME_ONLY : 0; > + const MY_LOCALE *lc= 0; > DBUG_ASSERT(fixed == 1); > > - if (get_arg0_date(&l_time, is_time_flag)) > + if (get_arg0_date(&l_time, is_time_format ? TIME_TIME_ONLY : 0)) > return 0; > > if (!(format = args[1]->val_str(str)) || !format->length()) > goto null_date; > > + if (!is_time_format && !(lc= locale) && !(lc= get_locale(args[2]))) > + goto null_date; // invalid locale > + > if (fixed_length) > size=max_length; > else > @@ -1998,7 +2013,7 @@ String *Item_func_date_format::val_str(String *str) > if (!make_date_time(&date_time_format, &l_time, > is_time_format ? MYSQL_TIMESTAMP_TIME : > MYSQL_TIMESTAMP_DATE, > - locale, str)) > + lc, str)) > return str; > > null_date: > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index 7d2fe46b644..20ecb4774b3 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -797,22 +797,24 @@ class Item_func_from_days :public Item_datefunc > > class Item_func_date_format :public Item_str_func > { > - MY_LOCALE *locale; > + const MY_LOCALE *locale; > int fixed_length; > - const bool is_time_format; > String value; > +protected: > + bool is_time_format; > public: > - Item_func_date_format(THD *thd, Item *a, Item *b, bool is_time_format_arg): > - Item_str_func(thd, a, b), is_time_format(is_time_format_arg) {} > + Item_func_date_format(THD *thd, Item *a, Item *b): > + Item_str_func(thd, a, b), locale(0), is_time_format(false) {} > + Item_func_date_format(THD *thd, Item *a, Item *b, Item *c): > + Item_str_func(thd, a, b, c), locale(0), is_time_format(false) {} > String *val_str(String *str); > - const char *func_name() const > - { return is_time_format ? "time_format" : "date_format"; } > + const char *func_name() const { return "date_format"; } > void fix_length_and_dec(); > uint format_length(const String *format); > bool eq(const Item *item, bool binary_cmp) const; > bool check_vcol_func_processor(void *arg) > { > - if (is_time_format) > + if (arg_count > 2) > return false; > return mark_unsupported_function(func_name(), "()", arg, VCOL_SESSION_FUNC); > } > @@ -820,6 +822,17 @@ class Item_func_date_format :public Item_str_func > { return get_item_copy<Item_func_date_format>(thd, mem_root, this); } > }; > > +class Item_func_time_format: public Item_func_date_format > +{ > +public: > + Item_func_time_format(THD *thd, Item *a, Item *b): > + Item_func_date_format(thd, a, b) { is_time_format= true; } > + const char *func_name() const { return "time_format"; } > + bool check_vcol_func_processor(void *arg) { return false; } > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_func_time_format>(thd, mem_root, this); } > +}; > + > > class Item_func_from_unixtime :public Item_datetimefunc > { > diff --git a/sql/lex.h b/sql/lex.h > index f2dac95fce5..67c3bc8620d 100644 > --- a/sql/lex.h > +++ b/sql/lex.h > @@ -726,6 +726,7 @@ static SYMBOL sql_functions[] = { > { "CURTIME", SYM(CURTIME)}, > { "DATE_ADD", SYM(DATE_ADD_INTERVAL)}, > { "DATE_SUB", SYM(DATE_SUB_INTERVAL)}, > + { "DATE_FORMAT", SYM(DATE_FORMAT_SYM)}, > { "DECODE", SYM(DECODE_SYM)}, > { "DENSE_RANK", SYM(DENSE_RANK_SYM)}, > { "EXTRACT", SYM(EXTRACT_SYM)}, > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 7e89ad91a17..17d6f016b2c 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -875,7 +875,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, ulong *yystacksize); > Comments for TOKENS. > For each token, please include in the same line a comment that contains > the following tags: > - SQL-2011-N : Non Reserved keywird as per SQL-2011 > + SQL-2011-N : Non Reserved keyword as per SQL-2011 > SQL-2003-R : Reserved keyword as per SQL-2003 > SQL-2003-N : Non Reserved keyword as per SQL-2003 > SQL-1999-R : Reserved keyword as per SQL-1999 > @@ -1010,6 +1010,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, ulong *yystacksize); > %token DATA_SYM /* SQL-2003-N */ > %token DATETIME > %token DATE_ADD_INTERVAL /* MYSQL-FUNC */ > +%token DATE_FORMAT_SYM /* MYSQL-FUNC */ > %token DATE_SUB_INTERVAL /* MYSQL-FUNC */ > %token DATE_SYM /* SQL-2003-R */ > %token DAY_HOUR_SYM > @@ -9806,6 +9807,18 @@ function_call_nonkeyword: > if ($$ == NULL) > MYSQL_YYABORT; > } > + | DATE_FORMAT_SYM '(' expr ',' expr ')' > + { > + $$= new (thd->mem_root) Item_func_date_format(thd, $3, $5); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + | DATE_FORMAT_SYM '(' expr ',' expr ',' expr ')' > + { > + $$= new (thd->mem_root) Item_func_date_format(thd, $3, $5, $7); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > | DECODE_SYM '(' expr ',' expr ')' > { > $$= new (thd->mem_root) Item_func_decode(thd, $3, $5); > @@ -14719,6 +14732,7 @@ keyword_sp_not_data_type: > | CYCLE_SYM {} > | DATA_SYM {} > | DATAFILE_SYM {} > + | DATE_FORMAT_SYM {} > | DAY_SYM {} > | DECODE_SYM {} > | DEFINER_SYM {} > diff --git a/sql/sql_yacc_ora.yy b/sql/sql_yacc_ora.yy > index a475914d369..54ddd3ae922 100644 > --- a/sql/sql_yacc_ora.yy > +++ b/sql/sql_yacc_ora.yy > @@ -284,7 +284,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, ulong *yystacksize); > Comments for TOKENS. > For each token, please include in the same line a comment that contains > the following tags: > - SQL-2011-N : Non Reserved keywird as per SQL-2011 > + SQL-2011-N : Non Reserved keyword as per SQL-2011 > SQL-2003-R : Reserved keyword as per SQL-2003 > SQL-2003-N : Non Reserved keyword as per SQL-2003 > SQL-1999-R : Reserved keyword as per SQL-1999 > @@ -419,6 +419,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, ulong *yystacksize); > %token DATA_SYM /* SQL-2003-N */ > %token DATETIME > %token DATE_ADD_INTERVAL /* MYSQL-FUNC */ > +%token DATE_FORMAT_SYM /* MYSQL-FUNC */ > %token DATE_SUB_INTERVAL /* MYSQL-FUNC */ > %token DATE_SYM /* SQL-2003-R */ > %token DAY_HOUR_SYM > @@ -9462,6 +9463,18 @@ column_default_non_parenthesized_expr: > if ($$ == NULL) > MYSQL_YYABORT; > } > + | DATE_FORMAT_SYM '(' expr ',' expr ')' > + { > + $$= new (thd->mem_root) Item_func_date_format(thd, $3, $5); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + | DATE_FORMAT_SYM '(' expr ',' expr ',' expr ')' > + { > + $$= new (thd->mem_root) Item_func_date_format(thd, $3, $5, $7); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > | DECODE_SYM '(' expr ',' decode_when_list ')' > { > $5->push_front($3, thd->mem_root); > @@ -14893,6 +14906,7 @@ keyword_sp_not_data_type: > | CYCLE_SYM {} > | DATA_SYM {} > | DATAFILE_SYM {} > + | DATE_FORMAT_SYM {} > | DAY_SYM {} > | DECODE_SYM {} > | DEFINER_SYM {} > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

1 0

Re: [Maria-developers] [Commits] c132349bc79: MDEV-11846: ERROR 1114 (HY000) table full when performing GROUP BY
by Sergey Petrunia 21 Sep '17

21 Sep '17

Hi Varun, Ok to push. On Fri, Sep 15, 2017 at 01:28:53PM +0530, Varun wrote: > revision-id: c132349bc79d987947c8659e71601914cfec71e8 (mariadb-10.1.23-156-gc132349bc79) > parent(s): fa2701c6f7b028782cf231565f578b2fc0f10d51 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-09-15 13:27:11 +0530 > message: > > MDEV-11846: ERROR 1114 (HY000) table full when performing GROUP BY > > The problem is there is an overflow for the key_file_length. > Added the maximum limit for the key_file_length > > --- > storage/maria/ma_create.c | 41 +++++++++++++++++++++++++++++++++-------- > 1 file changed, 33 insertions(+), 8 deletions(-) > > diff --git a/storage/maria/ma_create.c b/storage/maria/ma_create.c > index 0ddd8b226e2..79e60a76abc 100644 > --- a/storage/maria/ma_create.c > +++ b/storage/maria/ma_create.c > @@ -660,11 +660,24 @@ int maria_create(const char *name, enum data_file_type datafile_type, > > if (length > max_key_length) > max_key_length= length; > - tot_length+= ((max_rows/(ulong) (((uint) maria_block_size - > + > + if (tot_length == ULLONG_MAX) > + continue; > + > + ulonglong tot_length_part= (max_rows/(ulong) (((uint) maria_block_size - > MAX_KEYPAGE_HEADER_SIZE - > KEYPAGE_CHECKSUM_SIZE)/ > - (length*2))) * > - maria_block_size); > + (length*2))); > + if (tot_length_part >= (ULLONG_MAX / maria_block_size + > + ULLONG_MAX % maria_block_size)) > + tot_length= ULLONG_MAX; > + else > + { > + if (tot_length > ULLONG_MAX - tot_length_part * maria_block_size) > + tot_length= ULLONG_MAX; > + else > + tot_length+= tot_length_part * maria_block_size; > + } > } > > unique_key_parts=0; > @@ -673,11 +686,24 @@ int maria_create(const char *name, enum data_file_type datafile_type, > uniquedef->key=keys+i; > unique_key_parts+=uniquedef->keysegs; > share.state.key_root[keys+i]= HA_OFFSET_ERROR; > - tot_length+= (max_rows/(ulong) (((uint) maria_block_size - > + > + if (tot_length == ULLONG_MAX) > + continue; > + ulonglong tot_length_part= (max_rows/(ulong) (((uint) maria_block_size - > MAX_KEYPAGE_HEADER_SIZE - > KEYPAGE_CHECKSUM_SIZE) / > - ((MARIA_UNIQUE_HASH_LENGTH + pointer)*2)))* > - (ulong) maria_block_size; > + ((MARIA_UNIQUE_HASH_LENGTH + pointer)*2))); > + > + if (tot_length_part >= (ULLONG_MAX / maria_block_size + > + ULLONG_MAX % maria_block_size)) > + tot_length= ULLONG_MAX; > + else > + { > + if (tot_length > ULLONG_MAX - tot_length_part * maria_block_size) > + tot_length= ULLONG_MAX; > + else > + tot_length+= tot_length_part * maria_block_size; > + } > } > keys+=uniques; /* Each unique has 1 key */ > key_segs+=uniques; /* Each unique has 1 key seg */ > @@ -746,8 +772,7 @@ int maria_create(const char *name, enum data_file_type datafile_type, > Get estimate for index file length (this may be wrong for FT keys) > This is used for pointers to other key pages. > */ > - tmp= (tot_length + maria_block_size * keys * > - MARIA_INDEX_BLOCK_MARGIN) / maria_block_size; > + tmp= (tot_length / maria_block_size + keys * MARIA_INDEX_BLOCK_MARGIN); > > /* > use maximum of key_file_length we calculated and key_file_length value we > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 0319ee58fbb: MDEV-13290: Assertion Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' or `! is_set()' failed
by Alexander Barkov 19 Sep '17

19 Sep '17

Hi Sanja, The patch looks OK for me. Please find one suggestion below: On 09/19/2017 05:56 PM, Oleksandr Byelkin wrote: > revision-id: 0319ee58fbb6372ccc8533613e0c55666abc948a (mariadb-10.0.32-32-g0319ee58fbb) > parent(s): 389f7cdf3ccd11a6b4fb9b6346436790254e615c > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2017-09-19 15:55:59 +0200 > message: > > MDEV-13290: Assertion Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' or `! is_set()' failed > > Check error status which can be set by conversion procedures. > > --- > mysql-test/r/insert.result | 22 ++++++++++++++++++++++ > mysql-test/t/insert.test | 26 ++++++++++++++++++++++++++ > sql/table.cc | 3 +++ > 3 files changed, 51 insertions(+) > > diff --git a/mysql-test/r/insert.result b/mysql-test/r/insert.result > index 82f3977e231..0efee36e22d 100644 > --- a/mysql-test/r/insert.result > +++ b/mysql-test/r/insert.result > @@ -717,3 +717,25 @@ insert ignore into t1 values (1,12); > Warnings: > Warning 1062 Duplicate entry '1' for key 'f1' > DROP TABLE t1; > +# > +# MDEV-13290 Assertion Assertion `!is_set() || (m_status == DA_OK_BULK > +# && is_bulk_op())' or `! is_set()' failed > +# > +SET @save_mode= @@sql_mode; > +SET sql_mode= 'STRICT_ALL_TABLES'; > +CREATE TABLE t1 (f1 INT DEFAULT 0, f2 INT); > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = 'x' WITH CHECK OPTION; > +REPLACE INTO v1 SET f2 = 1; > +ERROR 22007: Truncated incorrect DOUBLE value: 'x' > +drop view v1; > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = cast('' as decimal) WITH CHECK OPTION; > +REPLACE INTO v1 SET f2 = 1; > +ERROR 22007: Truncated incorrect DECIMAL value: '' > +drop view v1; > +SELECT 0,0 INTO OUTFILE 't1.txt'; > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = 'x' WITH CHECK OPTION; > +LOAD DATA INFILE 't1.txt' INTO TABLE v1; > +ERROR 22007: Truncated incorrect DOUBLE value: 'x' > +drop view v1; > +drop table t1; > +SET @@sql_mode= @save_mode; > diff --git a/mysql-test/t/insert.test b/mysql-test/t/insert.test > index ff8396fd7fd..d84c0cee90d 100644 > --- a/mysql-test/t/insert.test > +++ b/mysql-test/t/insert.test > @@ -573,3 +573,29 @@ insert ignore into t1 values (1,12) on duplicate key update f2=13; > set @@old_mode=""; > insert ignore into t1 values (1,12); > DROP TABLE t1; > + > +--echo # > +--echo # MDEV-13290 Assertion Assertion `!is_set() || (m_status == DA_OK_BULK > +--echo # && is_bulk_op())' or `! is_set()' failed > +--echo # > + > +SET @save_mode= @@sql_mode; > +SET sql_mode= 'STRICT_ALL_TABLES'; > +CREATE TABLE t1 (f1 INT DEFAULT 0, f2 INT); > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = 'x' WITH CHECK OPTION; > +--error ER_TRUNCATED_WRONG_VALUE > +REPLACE INTO v1 SET f2 = 1; Can you please add "SELECT * FROM t1" here and some other places (see below), to make sure that REPLACE did not actually insert new records. Please see my comments in MDEV. > +drop view v1; > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = cast('' as decimal) WITH CHECK OPTION; > +--error ER_TRUNCATED_WRONG_VALUE > +REPLACE INTO v1 SET f2 = 1; Please add "SELECT * FROM t1". > +drop view v1; > +SELECT 0,0 INTO OUTFILE 't1.txt'; > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = 'x' WITH CHECK OPTION; > +--error ER_TRUNCATED_WRONG_VALUE > +LOAD DATA INFILE 't1.txt' INTO TABLE v1; Please add "SELECT * FROM t1". > +let $MYSQLD_DATADIR= `select @@datadir`; > +remove_file $MYSQLD_DATADIR/test/t1.txt; > +drop view v1; > +drop table t1; > +SET @@sql_mode= @save_mode; > diff --git a/sql/table.cc b/sql/table.cc > index 975d9d53882..ff9c4217b7d 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -4642,6 +4642,9 @@ int TABLE_LIST::view_check_option(THD *thd, bool ignore_failure) > main_view->view_name.str); > return(VIEW_CHECK_ERROR); > } > + /* We check thd->error() because it can be set by conversion problem. */ > + if (thd->is_error()) > + return(VIEW_CHECK_ERROR); > return(VIEW_CHECK_OK); > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

1 0

[Maria-developers] Patch for MDEV-10574
by jerome brauge 19 Sep '17

19 Sep '17

Alexander, Here is the patch I was talking about. It not address all aspects of MDEV-10574, especially, empty strings in table column are not view as null. But as in oracle mode, no empty string in columns should be created, an fully oracle application will never fall in this case. Regards, Jérôme. > -----Message d'origine----- > De : jerome brauge > Envoyé : mardi 19 septembre 2017 14:13 > À : 'Alexander Barkov' > Objet : RE: Oracle strings functions compatibility : substr > > Hello Alexander, > Yes I'm aware. > This point will be treated by my next patch which will also affect the others > string functions (a part of MDEV-10574). > I will refresh this new patch with the current development branch, and I > submit to you. > Regards. > Jérôme. > > > -----Message d'origine----- > > De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mardi 19 > > septembre 2017 13:16 À : jerome brauge Objet : Re: Oracle strings > > functions compatibility : substr > > > > Hi Jerome, > > > > I noticed one more difference in SUBSTR: > > > > > > SELECT SUBSTR('aaa',1,-1) FROM DUAL; > > > > MariaDB returns an empty string. > > Oracle returns NULL. > > > > Here's Oracle's documentation: > > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions162.ht > > m > > > > > > I think we should do this in the same patch, to avoid behavior change > > in the future. > > > > Would you like to try doing this additional change? > > Or would you like me to make an incremental patch on top of yours? > > > > Thanks! > > > > On 09/19/2017 10:33 AM, jerome brauge wrote: > > > Hello Alexander, > > > It's done. > > > > > > Do you have news from Sergei for MDEV-12874, MDEV-13417 and MDEV- > > 13418 ? > > > > I asked Sergei, awaiting for his answer. I'll let you know. > > > > > > > > Thank you very much. > > > > > > > > > > > >> -----Message d'origine----- > > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 18 > > >> septembre 2017 18:12 À : jerome brauge Objet : Re: Oracle strings > > >> functions compatibility : substr > > >> > > >> Hello Jerome, > > >> > > >> > > >> On 09/18/2017 05:14 PM, jerome brauge wrote: > > >>> Hello Alexander, > > >>> I don't understand how I've missed such an obvious solution ! > > >>> Here is the new patch. > > >> > > >> Thanks. Now it looks simpler. > > >> > > >> Can you please do one small thing: > > >> move get_position() from "public:" to "protected:". > > >> > > >> Ok to make a pull request after this change. > > >> > > >> Thank you very much! > > >> > > >> > > >>> > > >>> Regards. > > >>> Jérôme. > > >>> > > >>> > > >>>> -----Message d'origine----- > > >>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 18 > > >>>> septembre 2017 11:30 À : jerome brauge Objet : Re: Oracle strings > > >>>> functions compatibility : substr > > >>>> > > >>>> Hello Jerome, > > >>>> > > >>>> > > >>>> On 08/17/2017 04:43 PM, jerome brauge wrote: > > >>>>> Hello Alexander, > > >>>>> Here is a patch for function SUBSTR in sql_mode=oracle (If > > >>>>> position is 0, > > >>>> then it is treated as 1). > > >>>> > > >>>> I'm thinking of how to get a smaller patch. > > >>>> > > >>>> Wouldn't it be possible to introduce: > > >>>> > > >>>> virtual longlong get_position(); > > >>>> > > >>>> > > >>>> > > >>>> longlong Item_func_substr::get_position() { > > >>>> return args[1]->val_int(); > > >>>> } > > >>>> > > >>>> longlong Item_func_substr_oracle::get_position() > > >>>> { > > >>>> longlong pos= args[1]->val_int(); > > >>>> return pos == 0 ? 1 : pos; > > >>>> } > > >>>> > > >>>> > > >>>> and just replace all calls for args[1]->val_int() to get_position() ? > > >>>> > > >>>> Thanks! > > >>>> > > >>>> > > >>>>> > > >>>>> Regards, > > >>>>> Jérôme. > > >>>>>

1 0

Re: [Maria-developers] NUMA Support in MariaDB
by serg＠mariadb.org 19 Sep '17

19 Sep '17

Hi, Sumit! On Sep 16, Sumit Lakra wrote: > The project 'NUMA Support in MariaDB' is still incomplete and like I > mentioned in my mails to my mentors, I am still willing to work on it > and see it through to the end. Great, thanks! > Most of the minor tasks along with some important tasks are already > complete. The important part that remains is to work on the task-queue > interface between SQL and InnoDB. I was unable to implement this for > the NUMA case and I won't be able to complete it without some help. > So, do you think you can connect me with someone from the > MariaDB/InnoDB/MySQL team who is an expert of this interface? I will > need someone who has a very good understanding of the present > structure to help me come up with a way to implement this for the NUMA > case. If you do find someone who can be of help, please see to it that > since GSoC is over (and otherwise as well), it shouldn't be a > Student-Mentor kind of thing, but more like two or more developers > working together trying to come up with a solution to this problem. Why wouldn't you ask the question on the maria-developers@ mailing list? I'll try to make sure that it'll be answered. > However, if you are unable to connect me with someone who has > expertise in the handler interface, I am afraid I won't be able to > complete this project on my own. In this case, would you like me to > create a pull request of the work done so far ( > https://github.com/theGodlessLakra/server/tree/NUMA_Support) Although > NUMA Support is incomplete, merging this code to the MariaDB code base > will allow other developers to continue working on this. Is it sufficiently complete to be useful as is? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 6a72d15: MDEV-10767 /tmp/wsrep_recovery.${RANDOM} file created in unallowed SELinux context
by Sergei Golubchik 13 Sep '17

13 Sep '17

Hi, Sachin! On Aug 31, sachin wrote: > revision-id: 6a72d154d6ec231eeff861496e4d07101ac41553 (mariadb-10.1.26-16-g6a72d15) > parent(s): dda40b930498b70bb5546f857b27744039a5649d > author: sachin > committer: sachin > timestamp: 2017-08-31 12:17:10 +0530 > message: > > MDEV-10767 /tmp/wsrep_recovery.${RANDOM} file created in unallowed SELinux context > > Solution:- Allowed mysqld_t to open file with context initrc_tmp_t. Why does the server need it? I've only found scripts/galera_recovery.sh, that does: log_file=$(mktemp /tmp/wsrep_recovery.XXXXXX) /usr/sbin/mysqld $cmdline_args --user=$user --wsrep_recover --log-error="$log_file" For that use case, I wouldn't bother with a new selinux policy, it'd be safer to do log_file=$(mktemp /tmp/wsrep_recovery.XXXXXX) /usr/sbin/mysqld $cmdline_args --user=$user --wsrep_recover --disable-log-error 2>"$log_file" Note, that the similar piece of code is used in mysqld_safe.sh, only there it uses DATADIR/wsrep_recovery.XXXXXX, apparently somebody's attempt to work around selinux. Better to make it identical to galera_recovery.sh, I'd say. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] [External] Obsolete GTID domain delete on master (MDEV-12012, MDEV-11969)
by andrei.elkin＠pp.inet.fi 13 Sep '17

13 Sep '17

Hello, Simon, Kristian. (The mail was meant to be sent out yesterday, but it got stuck in my outgoing box). > Simon Mudd <simon.mudd(a)booking.com> writes: > >> ids. Obviously once all appropriate bin logs have been purged >> (naturally by other means) then no special processing will be needed. > > Right. Hence my original idea (which was unfortunately never implemented so > far). If at some point a domain has been unused for so long that all GTIDs > in that domain are gone, it is relatively safe to pretend that the domain > never existed. > > I would like to understand if you can think of significant use cases where > the DBA needs to have active binlog files in the master containing some > domain, while simultaneously pretending that this domain never existed. > > Or if it is more of a general concern, and the inconvenience for users to > have to save old binlogs somewhere else than the master's data directory and > binlog index (SHOW BINARY LOGS). > >> removing old binary logs should _not_ IMO be done as a way of >> forgetting the past obsolete domains. >> BINLOGS are important so throwing them away is an issue. I think >> that somehow the code needs >> to be aware of the cut-off point and when the “stale domain ids” are removed.) Simon, initially I thought of masking out the problematic domain so that the most recent binlog file would not have it in its Gtid_list header. Yet I've given up that idea to have agreed the strict setup on master weighs much more. > > I understand the desire to not delete binlog files. And the mdev-12012 use case might even not require to conduct this purging/flush-delete-domain procedure if IGNORE_DOMAIN_IDS (for the zero id domain in question) would do. There seems to be MDEV-9108 in the way though, but conceptually the DBA may have a way to stay with the binlog files even having a problematic domain. > > The problem is: If you want to have GTIDs with some domain in your active > binlog files, _and_ you also want to pretend that this domain never existed, > what does it mean? What is the semantics? It creates a lot of complexities > for defining the semantics, for documenting it, for the users to understand > it, and for the code to implement it correctly. > > So basically, I do not understand what is the intended meaning of FLUSH > BINARY LOGS DELETE DOMAIN d _and_ at the same time keeping GTIDs with domain > d around in active binlog files? In what respects is the domain deleted, and > in what respects not? > > For the master, the binlog files are mainly used to stream to connecting > slaves. Deleting a domain means replacing the conceptual binlog history with > one in which that domain never existed. So that domain will be ignored in a > connecting slaves position, assuming it is served by another multi-source > master. If a new GTID in that domain appears later, it will be considered > the very first GTID ever in that domain. > > So consider what happens if there is anyway GTIDs in that domain deeper in > the binlog: > > 1. An already connected slave may be happily replicating those GTIDs. If > that slave reconnects (temporary network error for example), it will instead > fail with unknown GTID, or perhaps just start silently ignoring all further > GTIDs in that domain. This kind of unpredictable behaviour seems bad. > > 2. Suppose a slave connects with a position without the deleted domain. The > master starts reading the binlog from some point. What happens if a GTID is > encountered that contains the deleted domain? The slave will start > replicating that domain from some arbitrary point that depends on where it > happened to be in other domains at the last disconnect. This also seems > undesirable. > > There may be other scenarios that I did not think about. > >> DBAs do not like to remove bin logs “early" as unless you keep a copy >> somewhere you may lose valuable information, >> for recovery, for backups etc. Not everyone will be making automatic >> copies (as MySQL does not provide an automatic way to do this) > > Understood. Maybe what is needed is a PURGE BINARY LOGS that removes the > entries from the binlog index (SHOW BINARY LOGS), but leaves the files in > the file system for the convenience of the sysadmin? (Well, you can just > hand-edit binlog.index, but that requires master restart I think). Like I said above, a filtering solution could be helpful. > >> The other comment I see mentioned here was “make sure all slaves are >> up to date”. That’s going to be hard. The master can only be >> aware of “connected slaves” and if you have intermediate masters, or a > > Indeed, the master cannot ensure this. The idea is that the DBA, who decides > to delete a domain, must understand that this should not be done if any > slave still needs GTIDs from that domain. This is similar to configuring > normal binlog purge, where the DBA needs to ensure that binlogs are kept > long enough for the needs of the slowest slave. > >> FWIW expiring old domains is good to do. There’s a similar FR for > >> completely different the problem space is the same. Coming up with a >> solution which is simple to use and understand and also >> avoids where that’s possible making mistakes which may break >> replication is good. So thanks for looking at this. > > Indeed. And the input from people like you with strong operational > experience is very valuable to end up with a good solution, hence my request > for additional input. > > - Kristian. I hope clarifications given by Kristian back up the new feature idea. On the other hand it would be good to have a way for DBA to cope with the likes of mdev-12012 (there are few reports on this matter) without purging. I rely on a solution to MDEV-9108 in that regard. Cheers, Andrei

1 0

[Maria-developers] A problem with implementing Group Commit with Binlog with MyRocks
by Sergey Petrunia 12 Sep '17

12 Sep '17

Hello, This is about https://jira.mariadb.org/browse/MDEV-11934. I've encountered an insteresting issue here, so I thought I would consult on both MyRocks and MariaDB lists. == Some background == "group commit with binlog" feature needs to accomplish two goals: 1. Keep the binlog and the storage engine in sync. This is done by employing XA between the binlog and the storage engine. It works by making these calls: /* Make the transaction's changes to be ready to be committed (no conflicts with other transactions, etc) but do not commit them yet. The effects of the prepare operation must be synced to disk, as the storage engine needs to be able to recover (i.e. commit) the prepared transaction after a crash */ storage_engine->prepare(sync=true); /* After this call, the transaction is considered committed. In case of a crash, the recovery process will use the contents of the binlog to determine which of the prepared transactions are to be committed and which are to be rolled back. */ binlog->write(sync=true); /* Commit the transaction in the storage engine. This makes its changes visible to other transactions (and also releases its locks and so forth) Note that most of the time(*) we don't need to sync there. In case of a crash we will be able to recover using the binlog. */ storage_engine->commit(); 2. The second goal is to make operation performant. We need two coordinated disk flushes per transaction, the idea is to do "Group Commit" where multiple transactions share disk flushes. So, we need to do group commit and keep the storage engine and the binlog in sync while doing that. == Group Commit with Binlog in MySQL == MySQL (and fb/mysql-5.6 in particular) does in the following phases: Phase #1: Call storage_engine->prepare() for all transactions in the group. The call itself is not persistent. Phase #2: Call storage->engine->flush_logs(). This makes the effect of all Prepare operations from Phase#1 persistent. Phase #3: Write and sync the binary log. Phase #4: Call storage_engine->commit(). This does not need to be persistent. MyRocks implements them. == Group Commit with Binlog in MariaDB == MariaDB does not have these phases described above: > Phase #1: > Call storage_engine->prepare() for all transactions in the group. > The call itself is not persistent. > > Phase #2: Call storage->engine->flush_logs(). > This makes the effect of all Prepare operations from Phase#1 persistent. A quote from Kristian's description at https://lists.launchpad.net/maria-developers/msg10832.html >> So the idea is to do group prepare with the same group of transactions that >> will later group commit to the binlog. In MariaDB, this concept does not >> exist. Storage engine prepares are allowed to run in parallel and in any >> order compared to binlog commit. Initially this looked like it could work for MyRocks. MyRocks has a group commit implementation, both Prepare() and Commit() operations participate in groups. However when I implemented a group commit implementation I found its performance to be close to what one would expect if there was no commit grouping, and commit() call flushed to disk https://jira.mariadb.org/browse/MDEV-11934 has the details. == The issue == (I'm 95% certain about this. It's not 100% yet but it is very likely) RocksDB's Group Write (see rocksdb/rocksdb/db/db_impl_write.cc, DBImpl::WriteImpl function) handles both Prepare() and Commit() commands and does the following: 1. Controls writing the commited data into the MemTable 2. Writes transactions to WAL 3. Syncs the WAL. All three steps are done for the whole group. This has a consequence: a Commit() operation that does not need to sync the WAL will still be delayed if another operation in the group needs the WAL to be synced. This delay has a disastrous effect, because SQL layer tries to have the same order of transactions in the storage engine and in the binlog. In order to do that, it calls rocksdb_commit_ordered() for each transaction sequentially. Delaying one transaction causes a delay of the entire SQL-level commit group. == Possible solutions == I am not sure what to do. - Make the SQL layer's Group Commit implementation invoke hton->flush_logs() explicitly, like MySQL does? - Modify RocksDB so that Transaction::Commit(sync=false) do not use Group Write? I am not sure if this is possible: Group Write is not about only performance, it's about preventing concurrent MemTable writes. AFAIU one cannot just tell a certain DBImpl::WriteImpl() call to not participate in write groups and work as if there was no other activity. - Modify RocksDB so that Transaction::Commit(sync=false) does not wait until its write group finishes WAL sync? This could be doable but is potentially complex. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 5

[Maria-developers] ha_innobase::info_low() n_rows hack
by Aleksey Midenkov 11 Sep '17

11 Sep '17

In ha_innobase::info_low() there is following dirty hack: /* The MySQL optimizer seems to assume in a left join that n_rows is an accurate estimate if it is zero. Of course, it is not, since we do not have any locks on the rows yet at this phase. Since SHOW TABLE STATUS seems to call this function with the HA_STATUS_TIME flag set, while the left join optimizer does not set that flag, we add one to a zero value if the flag is not set. That way SHOW TABLE STATUS will show the best estimate, while the optimizer never sees the table empty. */ if (n_rows == 0 && !(flag & HA_STATUS_TIME)) { n_rows++; } It is very old (from 5.0 or earlier) and bug-prone. Because in ha_innobase::open(): info(HA_STATUS_NO_LOCK | HA_STATUS_VARIABLE | HA_STATUS_CONST); every opened empty table will be non-empty! I don't know what is the problem with join optimizer, but having storage engine to handle it seems not the right thing to do. Moreover, relying on HA_STATUS_TIME in this is definitely wrong. We can make join optimizer to ignore "0 rows case" for all storage engines. Is it big win from "1 row case" anyway? Or we can make new flag HA_JOIN_STAT and use it in make_join_statistics().

2 3

Re: [Maria-developers] A problem with implementing Group Commit with Binlog with MyRocks
by Sergey Petrunia 11 Sep '17

11 Sep '17

Hi Kristian, On Mon, Sep 11, 2017 at 01:02:31PM +0200, Kristian Nielsen wrote: > Sergey Petrunia <sergey(a)mariadb.com> writes: > > > This is about https://jira.mariadb.org/browse/MDEV-11934. I've encountered > > an insteresting issue here, so I thought I would consult on both MyRocks and > > MariaDB lists. > > Is your current code available somewhere? > No, it wasn't. Just pushed the current code into bb-10.2-mdev11934 branch: https://github.com/MariaDB/server/tree/bb-10.2-mdev11934 All changes are in the last commit. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] 2nd 2017 Developers Unconference
by Ian Gilfillan 11 Sep '17

11 Sep '17

The dates for the Developers Unconference in Shenzhen, China, are coming up fast. This is our first official event in Asia, and will be held from 13 November - 17 November. 13 November - new contributor day 14-15 November - traditional developers unconference 16-17 November - patch days As always, it's free to attend, and open to anyone interested in MariaDB development. Details are at https://mariadb.org/2017-2-developers-unconference-and-related-events-shenz… and signup on the Meetup page: https://www.meetup.com/MariaDB-Developers/events/241873269/

1 0

[Maria-developers] GSoC NUMA Evaluation Student failed Mail 2
by Sumit Lakra 06 Sep '17

06 Sep '17

Hello, I am not aware of what information regarding this project has been exchanged between my mentor Daniel Black and others including serg, so I am sharing this here for everyone to see. My last mail contains the conversations I had with my mentors. Some quotes from the final evaluation - "Your questions in the last week have been exceptionally keen even though these seem to be the same questions from 1-2 months ago" Remind me when you (Daniel Black) answered them. "Sumit, overall we are happy with the quality of work. Unfortunately the quantity of work showed a lack of putting in hours and limited test cases fell short of the project plan and organisational expectations" What limited test cases ? I wrote all the test cases you (Daniel Black) and JAN Lindstrom asked me to write. Which test case did I miss ? I even ran a bench-marking test with all possible values and sent you the results. However, testing the actual performance difference is something that can only be done in a real NUMA Machine. Sorry, I don't have that. And have you ever even tested my code in a real NUMA Machine ? I too deserve to know if my work has made any real difference on the machine it is supposed to work on. And I have asked you to run it on a real machine more than once over the past months. You are happy with the quality of the work. Good. You are unhappy with the quantity. How do you expect me to increase the quantity when I already completed the tasks I promised in my GSoC proposal except for those that you removed from the list. What project plan are you referring to ? My GSoC proposal - https://docs.google.com/document/d/1UbOQysgOOzCM7z5FPC7gmuCpyxPmB7RGg7y9Ogq… Tasks spreadsheet - https://docs.google.com/spreadsheets/d/1nE-qFXhwwhF0hpOX3wUlczPjlPZTK_UzBu_… The tasks can be categorised as follows. [1] The tasks mentioned in my GSoC proposal which were not removed / changed later. - I completed all these. [2] The tasks mentioned in my GSoC proposal which were crossed out / (under consideration to be dropped / unnecessary) / or were implemented differently. - I either did not complete these for obvious reasons or implemented some of them differently. [3] The tasks you added (in the last month) to the spreadsheet originally under the title 'Out of Time' in the last month. - I completed some of these too. Then when these were done you moved a task (the queue per node task) from the 'Out of Time' to the third month group. We both agreed it was an important part for the completion of the project and I tried my best to implement it. I spent days trying to figure it out, asked you for help, didn't get any, tried again and yet failed because I am no expert of either InnoDB or SQl side of MariaDB. Which organisational expectations did you ask me to fulfil ? When ? I don't remember ever being told about any organisational expectations. Perhaps these were your personal expectations of having a complete NUMA Support in MariaDB by the end of GSoC. If so, then I am really sorry I wasn't able to fulfill those expectations because I got stuck at an important part of the project which also was a seemingly impossible task to do and *YOU FAILED TO HELP ME* with it even though I repeatedly asked (begged actually) you for help. You just kept reminding me that to do the task was my job. That's some real good mentoring right there. Another quote from the final evaluation "Passing to get your continued work isn't honoring the rules of the project or being fair to Google" Yeah, How about showing some good mentor ship by helping me to complete the project while we had the time to do it within Google's timelines in stead. Oh! wait a sec... you never went through the code.. until 3-4 days ago. I was determined to complete it if you had helped me. I have been mentioning it in all mails since the mail I wrote to you and Jan on : 22nd Aug - "Last but not least, if you think it can still be done and have a idea, I will be more than willing to attempt it. After all, GSoC has only got me started to contribute to open-source and this evaluation won't be the end of it." 25th Aug - "I am sorry I am unable to create an implementation. It's not like I don't want to. I just can't completely understand how it has been implemented currently. I have tried to follow the code before, and I did again the whole day today, but I can't come up with an idea at all." 25th Aug - "Like I said, I am not unwilling to complete this task. But I can't even think of where to start. Since, the two of you have greater experience with InnoDB, maybe you should give it a shot. You will definitely understand the present structure better than I do. And if you can come up with even a verbal solution that sounds like it could work, I will implement it asap." 25th Aug - "I spent all day trying to think of any way to implement a queue-per-node thing, and I confess I failed to come up with one. So, in stead of spending more time on that, I will work on migrating these commits to a new branch today. And I also urge you to give this task a try yourselves." 25th Aug - " I haven't been able to implement the queue per node structure for background threads, and I am afraid I won't be able to do it without your help. It is beyond me. I gave it a few attempts and I have failed. It's not the implementation part but the 'coming up with a way to implement it' part that I haven't been able to figure out yet. I need you to know that I am trying my best here." 25th Aug - "But anyway I kindly request you two to try it out yourselves now. You don't have to do the work. I just urge you to take some time out this weekend and go through the code. Let me know how you think it may be possible to implement it. I will do it. I have 4 days 2 hrs before the deadline to submit the evaluation, ends. If you can come up with an implementation plan within the next 2 or 2.5 days, I assure you I will code the implementation within a day of that and probably commit it before submitting the final evaluation as well. I am willing to take this risk." On Aug 28th you wrote a mail and asked me "Knowing what plans you had and how they failed would have been good to know." So, on the same day I wrote you a mail with a detailed explanation of the attempts I had made in trying to complete the task, asking you again to try it yourself and help me if you could come up with an idea. You never did. 31st Aug http://marialog.archivist.info/2017-08-31.txt Me - "did you have an attempt at the task ? come up with any ideas yet ?" dragonheart - been a bit busy with my own work so far. I know I'm going to have to look at a lot of stuff to finish the final GSoC evaluation 4th Sept http://marialog.archivist.info/2017-09-04.txt Me - "You must have seen my mails. Did you have an attempt at the queue per node thing ? Any ideas ?" dragonheart - well, implementing it was your task. but no, haven't found the hours to look though the code again. As per our conversation on IRC clearly Daniel Black hadn't even gone through the code by 4th Sept or attempted to solve the problem. How did he hope to complete the final evaluation. Even if he went through the code a day before submitting his evaluation, isn't this too late for a mentor to go through his/her student's code. Here is a list of some questions I would like MariaDB staff to ponder upon especially serg. [1] Was the task doable at all ? [A] Not unless you are an expert of both the SQL side and the InnoDB side, expert enough to re-write a major part of the interfaace between them. Jan himself acknowledged he wouldn't be able to help me with this, because he has no experience with the interface. I really appreciate that. [2] Even if the task was doable by a team of SQL and InnoDB experts, was it doable by an individual student working on MariaDB for the past four months ? [A] Nope. [3] Why didn't Daniel Black help me with the task even when I clearly mentioned I would need help with the task weeks ago ? [A] ??? [4] Did my mentor Daniel Black even try to help me with this particular task ? Ever ? [A] Nope. [5] Did Daniel Black evaluate me on the basis of the number of commits or the tasks accomplished by those commits ? [A] Based on the evaluation, he clearly has been unhappy with the decreased number of commits, not the content, as all the tasks mentioned in the original proposal have been achieved along with a few more. [6] Can Daniel Black himself complete this task, and thus the project ? [A] I bet not. No doubt the project is incomplete because it involves doing a task that is nearly impossible to do, at least for me, especially without any help. But does GSoC means that the student has to necessarily complete the entire project. I have gone through the code submitting guidelines plenty of times. It clearly states that we must state which tasks were done, and which tasks remain. Obviously not all projects are complete, some only complete the project partially, but mention everything and continue working on them. I did the same, and mentioned in my evaluation that almost all tasks were complete, in stead of all, and I have always been willing to complete this project but man, I can't do something impossible, especially when the people I am supposed to ask for help themselves fail to help their students. Thanks, Sumit

1 0

[Maria-developers] GSoC NUMA Evaluation Student failed Mail 1
by Sumit Lakra 06 Sep '17

06 Sep '17

Hello Serg, I am sorry to bug you with this mail but I am not sure what discussions you and my mentors have had before I was failed in GSoC. I completed all the tasks I had mentioned in my proposal but the project is incomplete. No doubts about that, but it is so because I got stuck with a major task (added in the last month) of the project for which I received no help from my mentors. These are the conversations I had with my mentors recently regarding the same task. Thanks, Sumit ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Tue, Aug 22, 2017 at 1:04 AM Subject: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com>, Jan Lindström < jan.lindstrom(a)mariadb.com> Hello, I went through the code that the InnoDB background threads use to pick tasks and execute them, in detail. This is how they seem to work. The reader/writer threads are passed an integer which acts as a segment. They then call fil_aio_wait(segment) which calls os_aio_handler(segment, &node, &message, &type). The control then goes to the os_aio_simulated_handler(segment, m1, m2, request) where the code gets more complicated with AIO arrays and slots. It gets harder to understand how they choose their tasks. It is definitely not a simple queue structure from which they pick their tasks. Also, which buffer pool the task is related to can only be figured out quite later.. based on the value of m2, which stores the address of a bpage. A simple queue could have been easily replaced with multiple queues, i.e. a queue per numa-node like we had once discussed on IRC. Lastly, all these procedures are common for log threads as well. Another thing, you mentioned more than once you wanted the reader threads to look for a bpage in their local nodes before looking them up in other nodes, but they use the bpage structure itself like I mentioned. Obviously neither of us had a proper understanding of how InnoDB worked in these aspects when we started the project. These threads seem to operate on bpages mostly rather than buf_pools, which makes numa mapping even harder (buf_pools to numa nodes would comparatively be easier), but is definitely more efficient than buf_pools and hence shouldn't be changed. Then again there were cases when the tasks assigned to the background threads were done by other server threads as well, especially in case of flushing, but I successfully restricted them to their nodes. If we were to create queues per node, which queues would these other threads work on. In other words the way in which InnoDB was initially written and expanded has made it significantly difficult to be adapted for explicit NUMA support. It wouldn't just require restructuring and changing the way these threads use parameters, pick tasks etc, but may also require re-ordering the order in which they are executed. For example, the background threads are created before the user threads, and trying to use the NUMA node from user THD later on would mean more system calls and probably thread migrations. When you added the task in the spreadsheet, you were right to anticipate that it could require large cleanup of the InnoDB structure, but I am beginning to think it will be way more complicated. Also, as you once mentioned that most architectures have 2 to 4 NUMA nodes. No doubt, if we could implement a support for NUMA architectuure such that InnoDB would make the best use out of it, there would be a performance difference but it would still be quite negligible in a very fast computer, and I really don't think making such big changes to InnoDB would be worth the effort and risk. Trying to bring a major change to a working and verified code (written by someone else) is more error-prone than adding some modular code and functions to support a new feature. I hope you agree. Last but not least, if you think it can still be done and have a idea, I will be more than willing to attempt it. After all, GSoC has only got me started to contribute to open-source and this evaluation won't be the end of it. And since there's no other database software out there which supports NUMA (let's ignore numa-interleave) and since I was the first to start working on this, I will be really proud to see it through to the end. Some Evaluation stuff : I see only one text field in the form for a link to the work done, and it takes a single url. Reserving this for the github link, where do I add a link to the spreadsheet or documentation ? Assuming you have already verified the work done so far, I will be creating a pull request shortly, so I can add whether it was merged or not, in the evaluation. Thanking You, Sumit ---------- Forwarded message ---------- From: Daniel Black <daniel.black(a)au1.ibm.com> Date: Thu, Aug 24, 2017 at 12:11 PM Subject: Re: GSoC Final Evaluation To: Sumit Lakra <sumitlakradev(a)gmail.com>, Jan Lindström < jan.lindstrom(a)mariadb.com> On 22/08/17 05:34, Sumit Lakra wrote: > Hello, > > I went through the code that the InnoDB background threads use to pick > tasks and execute them, in detail. This is how they seem to work. > > The reader/writer threads are passed an integer which acts as a segment. > They then call fil_aio_wait(segment) which calls os_aio_handler(segment, > &node, &message, &type). The control then goes to the > os_aio_simulated_handler(segment, m1, m2, request) where the code gets > more complicated with AIO arrays and slots. It gets harder to understand > how they choose their tasks. It is definitely not a simple queue > structure from which they pick their tasks. Also, which buffer pool the > task is related to can only be figured out quite later.. based on the > value of m2, which stores the address of a bpage. > A simple queue could > have been easily replaced with multiple queues, i.e. a queue per > numa-node like we had once discussed on IRC. Yes, you know where requests are created and where the job message ends up so you should be able to create an implementation. Even if it doesn't preserver the existing behaviour it should be enough to test it. > Lastly, all these > procedures are common for log threads as well. relevance? > Another thing, you mentioned more than once you wanted the reader SQL threads? Or is this the same thing? > threads to look for a bpage in their local nodes before looking them up > in other nodes, but they use the bpage structure itself like I > mentioned > Obviously neither of us had a proper understanding of how > InnoDB worked in these aspects when we started the project. No project knows all the details. > These > threads seem to operate on bpages mostly rather than buf_pools, which > makes numa mapping even harder You could add a node_id to the class buf_page_t? > (buf_pools to numa nodes would > comparatively be easier), but is definitely more efficient than > buf_pools and hence shouldn't be changed. > Then again there were cases when the tasks assigned to the background > threads were done by other server threads as well, especially in case of > flushing, but I successfully restricted them to their nodes. If we were > to create queues per node, which queues would these other threads work on. Yes, which is why this issue was raised with you months ago. > In other words the way in which InnoDB was initially written and > expanded has made it significantly difficult to be adapted for explicit > NUMA support. The split of buffer pool instances was in appreciation that there there are hot locks causes by a large number of threads. It just needed more thought than implementing a few thread bindings. I'm sorry you though it was that easy. It wouldn't just require restructuring and changing the > way these threads use parameters, pick tasks etc, You've changed very few parameters so far. https://github.com/MariaDB/server/compare/10.2...theGodlessLakra:numa_one but may also require > re-ordering the order in which they are executed. I can't think of one example of this. > For example, the > background threads are created before the user threads, and trying to > use the NUMA node from user THD later on would mean more system calls > and probably thread migrations. Huh, what? "may..require..(bad design)". Which is why a mapping of THD to known background threads per node was desired, to avoid syscall thread migrations. > When you added the task in the spreadsheet, you were right to anticipate > that it could require large cleanup of the InnoDB structure, but I am > beginning to think it will be way more complicated. I mentioned in on 2017-07-10 too (http://marialog.archivist.info/2017-07-10.txt) too and you said you'd give it a try. > Also, as you once > mentioned that most architectures have 2 to 4 NUMA nodes. No doubt, if > we could implement a support for NUMA architectuure such that InnoDB > would make the best use out of it, there would be a performance > difference but it would still be quite negligible in a very fast > computer, Cross node cache-coherency takes significant clock cycles compared to local access. What you'd end up with is a system that has idle time that can't be used. By finishing this task this could be measured. local/remote perf access can be measure (perhaps even with fake_numa). https://joemario.github.io/blog/2016/09/01/c2c-blog/ > and I really don't think making such big changes to InnoDB > would be worth the effort and risk. There is no gain without risk. Small changes with careful review, running full tests between them is they way large changes are made low risk. > Trying to bring a major change to a > working and verified code (written by someone else) is more error-prone > than adding some modular code and functions to support a new feature. I > hope you agree. When developing you are always developing for current and future use. As such architectural changes need to be commiserate to the functional changes. Jan and I have placed emphasis on writing test cases. These ensure that changes. > Last but not least, if you think it can still be done and have a idea, I > will be more than willing to attempt it. After all, GSoC has only got me > started to contribute to open-source and this evaluation won't be the > end of it. > And since there's no other database software out there which > supports NUMA (let's ignore numa-interleave) and since I was the first > to start working on this There is, found https://github.com/aerospike/aerospike-server/blob/master/cf/src/hardware.c the other day. Wouldn't have helped much as it has a different architecture and getting on top of one was hard enough. >, I will be really proud to see it through to > the end. This week looks like you've done one commit that is largely a revert that Jan asked for. You have slowed down considerably. > Some Evaluation stuff : > I see only one text field in the form for a link to the work done, and > it takes a single url. Reserving this for the github link, where do I > add a link to the spreadsheet or documentation ? > > Assuming you have already verified the work done so far, I will be > creating a pull request shortly, so I can add whether it was merged or > not, in the evaluation. Looks like the fake_numa stuff is incomplete. Pinning of background threads looks odd as they aren't big allocators of memory and aren't associated with SQL threads/buffer pool instance nodes. I have looked though it. As a pull request 60 commits with 26(?) of them fixing prior commits will look messy for anyone trying to look at the changes. If you are up for trying to clean it up please give it a go. Doing it in a copied repository is a good idea as you've seen before it can get a bit hard to keep track of. > Thanking You, > Sumit > ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Fri, Aug 25, 2017 at 3:03 AM Subject: Re: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> On Thu, Aug 24, 2017 at 12:11 PM, Daniel Black <daniel.black(a)au1.ibm.com> wrote: > > > On 22/08/17 05:34, Sumit Lakra wrote: > > Hello, > > > > I went through the code that the InnoDB background threads use to pick > > tasks and execute them, in detail. This is how they seem to work. > > > > The reader/writer threads are passed an integer which acts as a segment. > > They then call fil_aio_wait(segment) which calls os_aio_handler(segment, > > &node, &message, &type). The control then goes to the > > os_aio_simulated_handler(segment, m1, m2, request) where the code gets > > more complicated with AIO arrays and slots. It gets harder to understand > > how they choose their tasks. It is definitely not a simple queue > > structure from which they pick their tasks. Also, which buffer pool the > > task is related to can only be figured out quite later.. based on the > > value of m2, which stores the address of a bpage. > > > A simple queue could > > have been easily replaced with multiple queues, i.e. a queue per > > numa-node like we had once discussed on IRC. > > Yes, you know where requests are created and where the job message ends > up so you should be able to create an implementation. Even if it doesn't > preserver the existing behaviour it should be enough to test it. > I am sorry I am unable to create an implementation. It's not like I don't want to. I just can't completely understand how it has been implemented currently. I have tried to follow the code before, and I did again the whole day today, but I can't come up with an idea at all. > > > Lastly, all these > > procedures are common for log threads as well. > > relevance? The reader/writer/log threads all start by executing fil_aio_wait(segment), in srv0start.cc > > > Another thing, you mentioned more than once you wanted the reader > > SQL threads? Or is this the same thing? > > > threads to look for a bpage in their local nodes before looking them up > > in other nodes, but they use the bpage structure itself like I > > mentioned > > Obviously neither of us had a proper understanding of how > > InnoDB worked in these aspects when we started the project. > > No project knows all the details. > > > These > > threads seem to operate on bpages mostly rather than buf_pools, which > > makes numa mapping even harder > > You could add a node_id to the class buf_page_t? > Okay, but how to use it ? where ? In which functions ? > > > (buf_pools to numa nodes would > > comparatively be easier), but is definitely more efficient than > > buf_pools and hence shouldn't be changed. > > > > Then again there were cases when the tasks assigned to the background > > threads were done by other server threads as well, especially in case of > > flushing, but I successfully restricted them to their nodes. If we were > > to create queues per node, which queues would these other threads work > on. > > Yes, which is why this issue was raised with you months ago. > > > In other words the way in which InnoDB was initially written and > > expanded has made it significantly difficult to be adapted for explicit > > NUMA support. > > The split of buffer pool instances was in appreciation that there there > are hot locks causes by a large number of threads. > > It just needed more thought than implementing a few thread bindings. I'm > sorry you though it was that easy. > The split of buffer pool is not problematic to NUMA implementation. Its the task queues for the background threads that's wrecking my brain. > > It wouldn't just require restructuring and changing the > > way these threads use parameters, pick tasks etc, > > > You've changed very few parameters so far. > > https://github.com/MariaDB/server/compare/10.2...theGodlessLakra:numa_one > > but may also require > > re-ordering the order in which they are executed. > > I can't think of one example of this. > > For example, the > > background threads are created before the user threads, and trying to > > use the NUMA node from user THD later on would mean more system calls > > and probably thread migrations. > > Huh, what? "may..require..(bad design)". > One way (a really bad design), would be to create background threads when a request is made by a SQL thread, and then bind it to the node depending on the request i.e. the buffer pool instance in which the associated bpage belongs, which would mean more system calls and migrations. Obviously a very bad design. > > Which is why a mapping of THD to known background threads per node was > desired, to avoid syscall thread migrations. > The SQL threads were expected to be bound to numa nodes, which I did, but I can't think of how to map them to background threads. Can you think of a way to do this ? Let me know how you think this can be done. I will implement it asap. > When you added the task in the spreadsheet, you were right to anticipate > > that it could require large cleanup of the InnoDB structure, but I am > > beginning to think it will be way more complicated. > > I mentioned in on 2017-07-10 too > (http://marialog.archivist.info/2017-07-10.txt) too and you said you'd > give it a try. > > > Also, as you once > > mentioned that most architectures have 2 to 4 NUMA nodes. No doubt, if > > we could implement a support for NUMA architectuure such that InnoDB > > would make the best use out of it, there would be a performance > > difference but it would still be quite negligible in a very fast > > computer, > > Cross node cache-coherency takes significant clock cycles compared to > local access. What you'd end up with is a system that has idle time that > can't be used. > > By finishing this task this could be measured. local/remote perf access > can be measure (perhaps even with fake_numa). > https://joemario.github.io/blog/2016/09/01/c2c-blog/ > > > and I really don't think making such big changes to InnoDB > > would be worth the effort and risk. > > There is no gain without risk. Small changes with careful review, > running full tests between them is they way large changes are made low > risk. > Agreed, but I honestly can't make out the heads and tails of the changes that will be required to replace the present task queue with a queue-per-node structure :( > > > Trying to bring a major change to a > > working and verified code (written by someone else) is more error-prone > > than adding some modular code and functions to support a new feature. I > > hope you agree. > > When developing you are always developing for current and future use. As > such architectural changes need to be commiserate to the functional > changes. Jan and I have placed emphasis on writing test cases. These > ensure that changes. > Like I said, I am not unwilling to complete this task. But I can't even think of where to start. Since, the two of you have greater experience with InnoDB, maybe you should give it a shot. You will definitely understand the present structure better than I do. And if you can come up with even a verbal solution that sounds like it could work, I will implement it asap. > > > Last but not least, if you think it can still be done and have a idea, I > > will be more than willing to attempt it. After all, GSoC has only got me > > started to contribute to open-source and this evaluation won't be the > > end of it. > > > > And since there's no other database software out there which > > supports NUMA (let's ignore numa-interleave) and since I was the first > > to start working on this > > There is, found > https://github.com/aerospike/aerospike-server/blob/master/cf > /src/hardware.c > the other day. Wouldn't have helped much as it has a different > architecture and getting on top of one was hard enough. > > >, I will be really proud to see it through to > > the end. > > This week looks like you've done one commit that is largely a revert > that Jan asked for. You have slowed down considerably. > You have a good reason to be upset at me and I am not complaining. But although it doesn't look from Github like much work due to no changes, I spent a good deal of time brainstorming on the implementation of the first task from the spreadsheet under the "should be done" title, unfortunately to no success. > > > Some Evaluation stuff : > > I see only one text field in the form for a link to the work done, and > > it takes a single url. Reserving this for the github link, where do I > > add a link to the spreadsheet or documentation ? > > > > Assuming you have already verified the work done so far, I will be > > creating a pull request shortly, so I can add whether it was merged or > > not, in the evaluation. > > Looks like the fake_numa stuff is incomplete. Pinning of background > threads looks odd as they aren't big allocators of memory and aren't > associated with SQL threads/buffer pool instance nodes. > Fixed the fake numa test which was failing in non-debug builds. Most of the memory allocation is done during the buffer pool creation, and these have been managed properly while creating buffer pools per numa node. What other allocators of memory do you wish to be managed ? and how ? (Non-pinned threads should probably be allowed to allocate memory from any node) > > I have looked though it. As a pull request 60 commits with 26(?) of them > fixing prior commits will look messy for anyone trying to look at the > changes. If you are up for trying to clean it up please give it a go. > Doing it in a copied repository is a good idea as you've seen before it > can get a bit hard to keep track of. > I spent all day trying to think of any way to implement a queue-per-node thing, and I confess I failed to come up with one. So, in stead of spending more time on that, I will work on migrating these commits to a new branch today. And I also urge you to give this task a try yourselves. Thanks, Sumit ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Fri, Aug 25, 2017 at 6:52 PM Subject: Re: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> Hello, I am currently working on copying the changes from numa_one branch to a different branch in a more orderly fashion. I haven't been able to implement the queue per node structure for background threads, and I am afraid I won't be able to do it without your help. It is beyond me. I gave it a few attempts and I have failed. It's not the implementation part but the 'coming up with a way to implement it' part that I haven't been able to figure out yet. I need you to know that I am trying my best here. Back in http://marialog.archivist.info/2017-07-10.txt, you also said "warning the whole io read/write thread segements and scheduling is really messy and could do with a significant rework", and you also mentioned that you and Jan would have an attempt at it yourselves, if I am unable to do it and I move on. Well, I regret moving on to SQL threads back then without informing you that I wasn't successfull. A 'queue-per-numa-node' is easier said than done. But anyway I kindly request you two to try it out yourselves now. You don't have to do the work. I just urge you to take some time out this weekend and go through the code. Let me know how you think it may be possible to implement it. I will do it. I have 4 days 2 hrs before the deadline to submit the evaluation, ends. If you can come up with an implementation plan within the next 2 or 2.5 days, I assure you I will code the implementation within a day of that and probably commit it before submitting the final evaluation as well. I am willing to take this risk. However, if you are unable to come up with an implementation plan, we don't really have to hurry and stress ourselves. We can work together (I can't do this on my own) on this particular task later on and come up with something, assuming I am not failed in the final evaluation. Happy Weekend, Sumit ---------- Forwarded message ---------- From: Daniel Black <daniel.black(a)au1.ibm.com> Date: Mon, Aug 28, 2017 at 7:56 AM Subject: Re: GSoC Final Evaluation To: Sumit Lakra <sumitlakradev(a)gmail.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> On 25/08/17 23:22, Sumit Lakra wrote: > Hello, > > I am currently working on copying the changes from numa_one branch to a > different branch in a more orderly fashion. This is looking good: https://github.com/theGodlessLakra/server/tree/NUMA_Support Don't be afraid to be very descriptive in the commit message especially about the problem being solved and what functionality it implements/changes. Don't re-say what the code is doing, just cover its meaning at a high level. > I haven't been able to > implement the queue per node structure for background threads, and I am > afraid I won't be able to do it without your help. It is beyond me. > I > gave it a few attempts and I have failed. It's not the implementation > part but the 'coming up with a way to implement it' part that I haven't > been able to figure out yet. I need you to know that I am trying my best > here. Knowing what plans you had and how they failed would have been good to know. > Back in http://marialog.archivist.info/2017-07-10.txt, > you also said "warning the whole io read/write thread segements and > scheduling is really messy and could do with a significant rework", To do it properly yes. > and > you also mentioned that you and Jan would have an attempt at it > yourselves, if I am unable to do it and I move on. Well, I regret moving > on to SQL threads back then without informing you that I wasn't > successfull. I probably should of harassed you a bit too. Acknowledging failures is an important part to learning, even if you just acknowledge to yourself. How did SQL threads end up? > A 'queue-per-numa-node' is easier said than done. But > anyway I kindly request you two to try it out yourselves now. You don't > have to do the work. I just urge you to take some time out this weekend > and go through the code. Let me know how you think it may be possible to > implement it. I will do it. Having only just read this, this is a bit late notice. The aspect of understanding the interoperation between parts is also a fundamental aspect of getting a plan to implement it. > I have 4 days 2 hrs before the deadline to submit the evaluation, ends. > If you can come up with an implementation plan within the next 2 or 2.5 > days, I assure you I will code the implementation within a day of that > and probably commit it before submitting the final evaluation as well. I > am willing to take this risk. I thought your submission of a URL left it open to still committing on it. Please don't risk missing it. Making sure the existing code/comments/documentation is as clean as possible is better than tackling new work now. > However, if you are unable to come up with an implementation plan, we > don't really have to hurry and stress ourselves. We can work together (I > can't do this on my own) on this particular task later on and come up > with something, assuming I am not failed in the final evaluation. > > Happy Weekend, > Sumit ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Mon, Aug 28, 2017 at 9:50 PM Subject: Re: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> Hello, The following is a detailed explanation of what I did, how and why, while trying to come up with a solution to the task queue problem. However, before you go through this I would still request you to have an attempt at it yourselves, so your perspectives won't be affected by my ideas and you may come up with something different. I had once tried this before in https://github.com/MariaDB/ server/commit/4b8436d3f7c573668a0d37b54dab2d05e8e102be. I even pointed this commit out to once on IRC, (can't access logs of this time). Anyway I didn't put much focus on this as I was onto some other task. Before trying to replace the present queue structure with a queue-per-node, I first tried to let all the threads access this queue as before but only pick tasks dealing with their local nodes. Neither this queue nor these tasks seem to be easily distinguishable in code. The threads start by running fil_aio_wait(segment). This function calls os_aio_handler(segment, &node, &message, &type). This calls os_aio_simulated_handler(segment, m1, m2, request). This calls AIO::get_array_and_local_segment(&array, global_segment), and also creates SimulatedAIOHandler handler(array, segment), followed by calling handler.check_pending(global_segment, event), handler.init(n_slots), and slot = handler.check_completed(&n_reserved). Now, in order to skip a task to be done by other thread, what a thread must do is pick the task, check whether the task is related to the same node or not. If the task is concerned with the same node execute it or else leave it in the queue. This is where the problem occurs. In order to figure out whether a task is concerned with the same node or not, it must know which buffer pool instance the task is associated to. This can only be figured out after the task picks up the bpage it has to deal with. The bpage already has this information by now. Anyway, the bpage a thread has to deal with can only be found out by buf_page_t* bpage = static_cast<buf_page_t*>(*m2). Also, the task is not picked up at any particular place, but is picked up in parts in different functions. All these function calls seem to make some or other kind of changes to the segment/slot/array they are picking up the task from. So, when a thread finally gets a task and finds it concerned with a different node, then in order to safely return the task back to the queue, it must also undo the changes made by all these procedures so far. This is the difficult part. Compare this with pc_flush_slot() where I successfully implemented this method of skipping tasks if not related to local pool. In the above commit you will find I have a few lines like 'ib::info() << thread_name << " accessing buf_page on node : " << node_of_pool << " in fil_aio_wait()";' If you compile and run this code you can see in real time which threads are accessing which pools. I have given proper names to them. Also, when I tried to skip tasks which were not of the same queue by returning from the function os_aio_simulated_handler(), which partly worked but on trying to return from the funtion fil_aio_wait(), there seems to be either an infinite loop or a deadlock somewhere, as the server stops midway (doesn't abort) while starting up. Hence I have commented out those lines on lines 5463 and 5464 in fil0fil.cc in the above commit. Another thing is the role of segments here. Are these segments related to buffer pool instances somehow ? Because all these threads including the log threads start there work by executing fil_aio_wait(segment), but segment here seems to be just an incrementing integer. At one point assuming these segments could contain tasks related with different pools/nodes, I added a loop such that both the read and write threads loop around fil_aio_wait(segment) calling it with all valid segments. Didn't work out. These were attempted by me when I pushed this commit weeks ago in the temp branch. And even though I pointed it out to you, I think I should have given much more importance to this task. Over the past few days, when I tried to give this task another try, I couldn't come up with any idea either to replace this queue with a queue-per-node thing or even to complete this skip-if-diff-node thing.

1 0

Re: [Maria-developers] [Commits] dccebdf862d: Rdb_io_watchdog doesn't support osx so disabling it
by Vicențiu Ciorbaru 30 Aug '17

30 Aug '17

Hi Varun! Please change the CMakeLists.txt to be like so: diff --git a/storage/rocksdb/CMakeLists.txt b/storage/rocksdb/CMakeLists.txt index 5db6d88..6db8211 100644 --- a/storage/rocksdb/CMakeLists.txt +++ b/storage/rocksdb/CMakeLists.txt @@ -146,7 +146,10 @@ ADD_CONVENIENCE_LIBRARY(rocksdb_aux_lib ADD_DEPENDENCIES(rocksdb_aux_lib GenError) # MARIAROCKS-TODO: how to properly depend on -lrt ? -TARGET_LINK_LIBRARIES(rocksdb_aux_lib rocksdblib ${ZLIB_LIBRARY} -lrt) +TARGET_LINK_LIBRARIES(rocksdb_aux_lib rocksdblib ${ZLIB_LIBRARY}) +if (UNIX AND NOT APPLE) + TARGET_LINK_LIBRARIES(rocksdb_aux_lib -lrt) +endif() TARGET_LINK_LIBRARIES(rocksdb rocksdb_aux_lib) @@ -182,7 +185,9 @@ ENDIF() # ADD_SUBDIRECTORY(unittest) #ENDIF() -SET(rocksdb_static_libs ${rocksdb_static_libs} "-lrt") +if (UNIX AND NOT APPLE) + SET(rocksdb_static_libs ${rocksdb_static_libs} "-lrt") +endif() ADD_LIBRARY(rocksdb_tools STATIC rocksdb/tools/ldb_tool.cc Also, change how you define the __APPLE__ #ifdef like so: - #ifndef _WIN32 + #if not defined(_WIN32) && not defined(__APPLE__) This way, you don't have to add an extra #endif. The point of #ifndef is to avoid typing, but it doesn't work well with multiple conditions. We can make use #if then. Vicențiu On Wed, 30 Aug 2017 at 15:37 Varun <varunraiko1803(a)gmail.com> wrote: > revision-id: dccebdf862d66e65528e18562c4503f40b695eca > (mariadb-10.3.0-107-gdccebdf862d) > parent(s): e1051590b61b45532284071ac78c50e7a2f24c63 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-08-30 18:05:52 +0530 > message: > > Rdb_io_watchdog doesn't support osx so disabling it > > --- > storage/rocksdb/CMakeLists.txt | 4 ++++ > storage/rocksdb/ha_rocksdb.cc | 14 +++++++++++++- > storage/rocksdb/rdb_io_watchdog.cc | 2 ++ > storage/rocksdb/rdb_io_watchdog.h | 6 ++++-- > 4 files changed, 23 insertions(+), 3 deletions(-) > > diff --git a/storage/rocksdb/CMakeLists.txt > b/storage/rocksdb/CMakeLists.txt > index 5db6d888bb6..435fe784d6c 100644 > --- a/storage/rocksdb/CMakeLists.txt > +++ b/storage/rocksdb/CMakeLists.txt > @@ -146,7 +146,11 @@ ADD_CONVENIENCE_LIBRARY(rocksdb_aux_lib > ADD_DEPENDENCIES(rocksdb_aux_lib GenError) > > # MARIAROCKS-TODO: how to properly depend on -lrt ? > +if (APPLE) > +TARGET_LINK_LIBRARIES(rocksdb_aux_lib rocksdblib ${ZLIB_LIBRARY}) > +else() > TARGET_LINK_LIBRARIES(rocksdb_aux_lib rocksdblib ${ZLIB_LIBRARY} -lrt) > +endif() > > TARGET_LINK_LIBRARIES(rocksdb rocksdb_aux_lib) > > diff --git a/storage/rocksdb/ha_rocksdb.cc b/storage/rocksdb/ha_rocksdb.cc > index e4d9ce02f10..4ffe6caf70b 100644 > --- a/storage/rocksdb/ha_rocksdb.cc > +++ b/storage/rocksdb/ha_rocksdb.cc > @@ -173,9 +173,11 @@ Rdb_ddl_manager ddl_manager; > const char *m_mysql_gtid; > Rdb_binlog_manager binlog_manager; > > -#ifndef _WIN32 > +#ifndef _WIN32 > +#ifndef __APPLE__ > Rdb_io_watchdog *io_watchdog = nullptr; > #endif > +#endif > /** > MyRocks background thread control > N.B. This is besides RocksDB's own background threads > @@ -555,7 +557,9 @@ static void rocksdb_set_io_write_timeout( > DBUG_ASSERT(save != nullptr); > DBUG_ASSERT(rdb != nullptr); > #ifndef _WIN32 > +#ifndef __APPLE__ > DBUG_ASSERT(io_watchdog != nullptr); > +#endif > #endif > > RDB_MUTEX_LOCK_CHECK(rdb_sysvars_mutex); > @@ -564,7 +568,9 @@ static void rocksdb_set_io_write_timeout( > > rocksdb_io_write_timeout_secs = new_val; > #ifndef _WIN32 > +#ifndef __APPLE__ > io_watchdog->reset_timeout(rocksdb_io_write_timeout_secs); > +#endif > #endif > RDB_MUTEX_UNLOCK_CHECK(rdb_sysvars_mutex); > } > @@ -3746,7 +3752,9 @@ static int rocksdb_init_func(void *const p) { > */ > if (status.IsIOError() > #ifndef _WIN32 > +#ifndef __APPLE__ > && errno == ENOENT > +#endif > #endif > ) { > sql_print_information("RocksDB: Got ENOENT when listing column > families"); > @@ -3985,8 +3993,10 @@ static int rocksdb_init_func(void *const p) { > } > > #ifndef _WIN32 > +#ifndef __APPLE__ > io_watchdog = new Rdb_io_watchdog(directories); > io_watchdog->reset_timeout(rocksdb_io_write_timeout_secs); > +#endif > #endif > > // NO_LINT_DEBUG > @@ -4077,9 +4087,11 @@ static int rocksdb_done_func(void *const p) { > commit_latency_stats = nullptr; > > #ifndef _WIN32 > +#ifndef __APPLE__ > delete io_watchdog; > io_watchdog = nullptr; > #endif > +#endif > > // Disown the cache data since we're shutting down. > // This results in memory leaks but it improved the shutdown time. > diff --git a/storage/rocksdb/rdb_io_watchdog.cc > b/storage/rocksdb/rdb_io_watchdog.cc > index a599ba58aec..a9e79283382 100644 > --- a/storage/rocksdb/rdb_io_watchdog.cc > +++ b/storage/rocksdb/rdb_io_watchdog.cc > @@ -23,6 +23,7 @@ > > /* Rdb_io_watchdog doesn't work on Windows [yet] */ > #ifndef _WIN32 > +#ifndef __APPLE__ > > namespace myrocks { > > @@ -236,4 +237,5 @@ int Rdb_io_watchdog::reset_timeout(const uint32_t > &write_timeout) { > } // namespace myrocks > > #endif > +#endif > > diff --git a/storage/rocksdb/rdb_io_watchdog.h > b/storage/rocksdb/rdb_io_watchdog.h > index de8c1b9500e..12aec516f26 100644 > --- a/storage/rocksdb/rdb_io_watchdog.h > +++ b/storage/rocksdb/rdb_io_watchdog.h > @@ -35,7 +35,8 @@ > namespace myrocks { > > // Rdb_io_watchdog does not support Windows ATM. > -#ifndef _WIN32 > +#ifndef _WIN32 > +#ifndef __APPLE__ > > class Rdb_io_watchdog { > const int RDB_IO_WRITE_BUFFER_SIZE = 4096; > @@ -113,5 +114,6 @@ class Rdb_io_watchdog { > Rdb_io_watchdog &operator=(const Rdb_io_watchdog &) = delete; > }; > > -#endif > +#endif > +#endif > } // namespace myrocks > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Review: MDEV-12985: Support percentile and media window functions
by Vicențiu Ciorbaru 30 Aug '17

30 Aug '17

Hi Varun! Here is the in-detail review for the percentile functions. Please add a grammar definition for median too and test it. All in all, not many changes, but please address my comments. If you disagree with some of them, let's discuss them. > diff --git a/mysql-test/r/win_percentile_cont.result b/mysql-test/r/win_percentile_cont.result > new file mode 100644 > index 00000000000..61f70892887 > --- /dev/null > +++ b/mysql-test/r/win_percentile_cont.result > diff --git a/mysql-test/t/win_percentile_cont.test b/mysql-test/t/win_percentile_cont.test > new file mode 100644 > index 00000000000..75fde963b2a > --- /dev/null > +++ b/mysql-test/t/win_percentile_cont.test > @@ -0,0 +1,55 @@ Please put more tests here, as we discussed. Also, to make test cases easier to follow, please add cume_dist as a function in the select to all these use cases. > +CREATE TABLE student (name CHAR(10), test double, score DECIMAL(19,4)); > +INSERT INTO student VALUES > +('Chun', 0, 3), ('Chun', 0, 7), > +('Kaolin', 0.5, 3), ('Kaolin', 0.6, 7), > +('Kaolin', 0.5, 4), > +('Tatiana', 0.8, 4), ('Tata', 0.8, 4); > + Make this --echo # No partition clause > +#no partition clause > +select name, percentile_disc(0.5) within group(order by score) over () from student; > +select name, percentile_cont(0.5) within group(order by score) over () from student; > + Same as before, use --echo # so we can see it in the result set too. > +# argument set to null > +select name, percentile_cont(null) within group(order by score) over (partition by name) from student; > +select name, percentile_disc(null) within group(order by score) over (partition by name) from student; > + Use --echo # > +# complete query with partition column > +select name, percentile_cont(0.5) within group(order by score) over (partition by name) as c from student; > +select name, percentile_disc(0.5) within group(order by score) over (partition by name) as c from student; > + Use --echo # > +#subqueries having percentile functions > + > +select * from ( select name , percentile_cont(0.5) within group ( order by score) over (partition by name ) from student ) as t; > +select * from ( select name , percentile_disc(0.5) within group ( order by score) over (partition by name ) from student ) as t; > +select name from student a where (select percentile_disc(0.5) within group (order by score) over (partition by name) from student b limit 1) >= 0.5; > + USe --echo > +# WITH STORED PROCEDURES > + > + USe --echo > +#DISALLOWED FIELDS IN ORDER BY CLAUSE > +--error ER_WRONG_TYPE_FOR_PERCENTILE_CONT > +select score, percentile_cont(0.5) within group(order by name) over (partition by score) from student; > +select score, percentile_disc(0.5) within group(order by name) over (partition by score) from student; > USe --echo > + > +# error with 2 order by elements > + > +--error ER_NOT_SINGLE_ELEMENT_ORDER_LIST > +select percentile_disc(0.5) within group(order by score,test) over (partition by name) from student; > +--error ER_NOT_SINGLE_ELEMENT_ORDER_LIST > +select percentile_cont(0.5) within group(order by score,test) over (partition by name) from student; > + Use --echo and fix that comment. We thought of a way for this right? > +#parameter value should be in the range of 0 to 1( NEED TO THINK A WAY FOR THIS) > +--error ER_ARGUMENT_OUT_OF_RANGE > +select percentile_disc(1.5) within group(order by score) over (partition by name) from student; > +--error ER_ARGUMENT_OUT_OF_RANGE > +select percentile_cont(1.5) within group(order by score) over (partition by name) from student; > + > +--error ER_ARGUMENT_NOT_CONSTANT > +select name,percentile_cont(test) within group(order by score) over (partition by name) from student; > +--error ER_ARGUMENT_NOT_CONSTANT > +select name, percentile_disc(test) within group(order by score) over (partition by name) from student; > + This is commented out. Not good, fix it please. And use --echo # > +#CHECK TYPE OF THE ARGUMENT, SHOULD BE ONLY NUMERICAL > +#select name, percentile_cont(name) within group(order by score) over (partition by name) from student; > + > > +drop table student; > diff --git a/sql/item.h b/sql/item.h > index 76ce4aa935f..108b0838dfc 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -5245,6 +5246,7 @@ class Cached_item_int :public Cached_item_item > Cached_item_int(Item *item_par) :Cached_item_item(item_par),value(0) {} > bool cmp(void); > int cmp_read_only(); Delete this, it's not used. > + longlong get_value(){ return value;} > }; > > > @@ -5255,6 +5257,7 @@ class Cached_item_decimal :public Cached_item_item > Cached_item_decimal(Item *item_par); > bool cmp(void); > int cmp_read_only(); Delete this, it's not used. > + my_decimal *get_value(){ return &value;}; > }; > > class Cached_item_field :public Cached_item > diff --git a/sql/item_windowfunc.cc b/sql/item_windowfunc.cc > index 6ab903a81bb..916f4113682 100644 > --- a/sql/item_windowfunc.cc > +++ b/sql/item_windowfunc.cc > diff --git a/sql/item_windowfunc.h b/sql/item_windowfunc.h > index 77cbd556e60..ca2ed7136ec 100644 > --- a/sql/item_windowfunc.h > +++ b/sql/item_windowfunc.h > @@ -678,6 +687,275 @@ class Item_sum_ntile : public Item_sum_window_with_row_count > ulong current_row_count_; > }; > > +class Item_sum_percentile_disc : public Item_sum_cume_dist, > + public Type_handler_hybrid_field_type > +{ > +public: > + Item_sum_percentile_disc(THD *thd, Item* arg) : Item_sum_cume_dist(thd, arg), > + Type_handler_hybrid_field_type(&type_handler_longlong), > + value(NULL), val_calculated(FALSE), first_call(TRUE), > + prev_value(0), order_item(NULL){} > + > + double val_real() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_real(); > + } > + > + longlong val_int() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_int(); > + } > + > + my_decimal* val_decimal(my_decimal* dec) > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_decimal(dec); > + } > + > + String* val_str(String *str) > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_str(str); > + } > + > + bool add() > + { An extra space here before = > + Item *arg = get_arg(0); > + if (arg->is_null()) > + return false; > + > + if (first_call) > + { > + prev_value= arg->val_real(); We are missing a space after >. > + if (prev_value >1 || prev_value < 0) > + { > + my_error(ER_ARGUMENT_OUT_OF_RANGE, MYF(0)); > + return true; > + } > + first_call= false; > + } > + > + double arg_val= arg->val_real(); > + There is one extra space here. > + if (prev_value != arg_val) > + { > + my_error(ER_ARGUMENT_NOT_CONSTANT, MYF(0)); > + return true; > + } > + > + if (val_calculated) > + return false; > + > + value->store(order_item); > + value->cache_value(); > + if (value->null_value) > + return false; > + > + Item_sum_cume_dist::add(); > + double val= Item_sum_cume_dist::val_real(); > + > + if (val >= prev_value && !val_calculated) > + val_calculated= true; > + return false; > + } > + > + enum Sumfunctype sum_func() const > + { > + return PERCENTILE_DISC_FUNC; > + } > + > + void clear() > + { > + val_calculated= false; > + first_call= true; > + value->clear(); > + Item_sum_cume_dist::clear(); > + } > + > + const char*func_name() const > + { > + return "percentile_disc"; > + } > + > + void update_field() {} > + void set_type_handler(Window_spec *window_spec); > + const Type_handler *type_handler() const > + {return Type_handler_hybrid_field_type::type_handler();} > + > + void fix_length_and_dec() > + { > + decimals = 10; // TODO-cvicentiu find out how many decimals the standard > + // requires. > + } > + > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_sum_percentile_disc>(thd, mem_root, this); } > + void setup_window_func(THD *thd, Window_spec *window_spec); > + void setup_hybrid(THD *thd, Item *item); > + > +private: > + Item_cache *value; > + bool val_calculated; > + bool first_call; > + double prev_value; > + Item *order_item; > +}; > + > +class Item_sum_percentile_cont : public Item_sum_cume_dist, > + public Type_handler_hybrid_field_type > +{ > +public: > + Item_sum_percentile_cont(THD *thd, Item* arg) : Item_sum_cume_dist(thd, arg), > + Type_handler_hybrid_field_type(&type_handler_double), > + floor_value(NULL), ceil_value(NULL), first_call(TRUE),prev_value(0), > + ceil_val_calculated(FALSE), floor_val_calculated(FALSE), order_item(NULL){} > + > + double val_real() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + double val= 1 + prev_value * (get_row_count()-1); > + > + /* > + Applying the formula to get the value > + If (CRN = FRN = RN) then the result is (value of expression from row at RN) > + Otherwise the result is > + (CRN - RN) * (value of expression for row at FRN) + > + (RN - FRN) * (value of expression for row at CRN) > + */ > + > + if(ceil(val) == floor(val)) > + return floor_value->val_real(); > + > + double ret_val= ((val - floor(val)) * ceil_value->val_real()) + > + ((ceil(val) - val) * floor_value->val_real()); > + > + return ret_val; > + } > + > + bool add() > + { > + Item *arg = get_arg(0); > + if (arg->is_null()) > + return false; > + > + if (first_call) > + { > + first_call= false; > + prev_value= arg->val_real(); > + if (prev_value >1 || prev_value < 0) > + { > + my_error(ER_ARGUMENT_OUT_OF_RANGE, MYF(0)); > + return true; > + } > + } > + > + double arg_val= arg->val_real(); > + if (prev_value != arg_val) > + { > + my_error(ER_ARGUMENT_NOT_CONSTANT, MYF(0)); > + return true; > + } > + > + if (!floor_val_calculated) > + { > + floor_value->store(order_item); > + floor_value->cache_value(); > + if (floor_value->null_value) > + return false; > + } > + if (floor_val_calculated && !ceil_val_calculated) > + { > + ceil_value->store(order_item); > + ceil_value->cache_value(); > + if (ceil_value->null_value) > + return false; > + } > + > + Item_sum_cume_dist::add(); > + double val= 1 + prev_value * (get_row_count()-1); > + > + if (!floor_val_calculated && get_row_number() == floor(val)) > + floor_val_calculated= true; > + > + if (!ceil_val_calculated && get_row_number() == ceil(val)) > + ceil_val_calculated= true; > + return false; > + } > + > + enum Sumfunctype sum_func() const > + { > + return PERCENTILE_CONT_FUNC; > + } > + > + void clear() > + { > + first_call= true; > + floor_value->clear(); > + ceil_value->clear(); > + floor_val_calculated= false; > + ceil_val_calculated= false; > + Item_sum_cume_dist::clear(); > + } > + > + const char*func_name() const > + { > + return "percentile_cont"; > + } > + void update_field() {} > + void set_type_handler(Window_spec *window_spec); > + const Type_handler *type_handler() const > + {return Type_handler_hybrid_field_type::type_handler();} > + > + void fix_length_and_dec() > + { > + decimals = 10; // TODO-cvicentiu find out how many decimals the standard > + // requires. > + } > + > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_sum_percentile_cont>(thd, mem_root, this); } > + void setup_window_func(THD *thd, Window_spec *window_spec); > + void setup_hybrid(THD *thd, Item *item); > + > +private: > + Item_cache *floor_value; > + Item_cache *ceil_value; > + bool first_call; > + double prev_value; > + bool ceil_val_calculated; > + bool floor_val_calculated; > + Item *order_item; > +}; > + > + > + > > class Item_window_func : public Item_func_or_sum > { > @@ -781,12 +1063,40 @@ class Item_window_func : public Item_func_or_sum > case Item_sum::DENSE_RANK_FUNC: > case Item_sum::PERCENT_RANK_FUNC: > case Item_sum::CUME_DIST_FUNC: > + case Item_sum::PERCENTILE_CONT_FUNC: > + case Item_sum::PERCENTILE_DISC_FUNC: > return true; > default: > return false; > } > } Let's make the grammar enforce this. This is silly. Is there a big limitation why we can't define a special rule for these functions? There already is order_by_single_element_list in sql_yacc.yy, it just isn't defined properly. We can get rid of these checks altogether then. > + bool only_single_element_order_list() const > + { > + switch (window_func()->sum_func()){ > + case Item_sum::PERCENTILE_CONT_FUNC: > + case Item_sum::PERCENTILE_DISC_FUNC: > + return true; > + default: > + return false; > + } > + } > + This is not needed. Put this in Item_sum_percentile_disc's fix_fields. > + void setting_handler_for_percentile_functions(Item_result rtype) const > + { > + switch (window_func()->sum_func()){ > + case Item_sum::PERCENTILE_DISC_FUNC: > + ((Item_sum_percentile_disc* ) window_func())->set_handler_by_cmp_type(rtype); > + break; > + default: > + return; > + } > + } > + Get rid of this function here. Put the logic of checking this result type in fix_fields specific to Item_sum_percentile_cont / disc. > + bool check_result_type_of_order_item(); > + > + > + > /* > Computation functions. > TODO: consoder merging these with class Group_bound_tracker. > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 1f282e6aee5..d8d6975d92c 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7490,3 +7490,11 @@ ER_WRONG_INSERT_INTO_SEQUENCE > eng "Wrong INSERT into a SEQUENCE. One can only do single table INSERT into a squence object (like with mysqldump). If you want to change the SEQUENCE, use ALTER SEQUENCE instead." > ER_SP_STACK_TRACE > eng "At line %u in %s" This one will go away when we enforce the limitation in the grammar, make sure to remove it. > +ER_NOT_SINGLE_ELEMENT_ORDER_LIST > + eng "Incorrect number of elements in the order list for '%s'" > +ER_WRONG_TYPE_FOR_PERCENTILE_CONT > + eng "Numeric datatype is required for Percentile_CONT function" As discussed have these two errors print the function name to which the error is for, just like ER_NOT_SINGLE_ELEMENT_ORDER_LIST uses now. > +ER_ARGUMENT_NOT_CONSTANT > + eng "Argument to the percentile functions is not a constant" > +ER_ARGUMENT_OUT_OF_RANGE > + eng "Argument to the percentile functions does not belong to the range [0,1]" > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index 74e1ad25183..127d6d73ee6 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -304,6 +304,12 @@ setup_windows(THD *thd, Ref_ptr_array ref_pointer_array, TABLE_LIST *tables, > win_func_item->update_used_tables(); > } > 2 problems with this code. 1. We can probably move this logic in Item_sum_percentile_[cont|disc]::fix_fields. See if that works. 2. There already is an identical while loop before this. USe that one, don't have an extra loop here for no reason. > + li.rewind(); > + while ((win_func_item= li++)) > + { > + if (win_func_item->check_result_type_of_order_item()) > + DBUG_RETURN(1); > + } > DBUG_RETURN(0); > } > > @@ -1658,6 +1666,49 @@ class Frame_unbounded_following_set_count : public Frame_unbounded_following > } > }; > This cursor counts all rows in the partition that don't have the order_item set to null. Let's rewrite this like so: Extract this logic: """ List_iterator_fast<Item_sum> it(sum_functions); Item_sum* item; while ((item= it++)) { Item_sum_window_with_row_count* item_with_row_count = static_cast<Item_sum_window_with_row_count *>(item); item_with_row_count->set_row_count(num_rows_in_partition); } """ 1. Nake it a protected method for Frame_unbounded_following_set_count. Lets call it set_win_funcs_row_count. 2. Rewrite Frame_unbounded_following_set_count to use that function. 3. Make a subclass Frame_unbounded_following_set_count_no_nulls 4. Make use of the protected method in the Frame_unbounded_following_set_count_no_nulls > > +class Frame_unbounded_following_set_count_special: public Frame_unbounded_following_set_count > +{ > + > +public: > + Frame_unbounded_following_set_count_special(THD *thd, > + SQL_I_List<ORDER> *partition_list, > + SQL_I_List<ORDER> *order_list, Item* arg) : > + Frame_unbounded_following_set_count(thd,partition_list, order_list) > + { > + order_item= order_list->first->item[0]; > + } > + void next_partition(ha_rows rownum) > + { > + ha_rows num_rows_in_partition= 0; > + if (cursor.fetch()) > + return; > + > + /* Walk to the end of the partition, find how many rows there are. */ > + do > + { > + if (!order_item->is_null()) > + num_rows_in_partition++; You are missing a space after } > + }while (!cursor.next()); > + > + List_iterator_fast<Item_sum> it(sum_functions); > + Item_sum* item; > + while ((item= it++)) > + { > + Item_sum_window_with_row_count* item_with_row_count = > + static_cast<Item_sum_window_with_row_count *>(item); > + item_with_row_count->set_row_count(num_rows_in_partition); > + } > + } > + > + ha_rows get_curr_rownum() const > + { > + return cursor.get_rownum(); > + } > + > +private: > + Item* order_item; > +}; > + > ///////////////////////////////////////////////////////////////////////////// > // ROWS-type frame bounds > ///////////////////////////////////////////////////////////////////////////// > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index a0bbf39b138..b3b34fce3a4 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -10210,6 +10215,11 @@ geometry_function: > Geometry::wkb_polygon, > Geometry::wkb_linestring)); > } I think this is something that should not be here right? What is this? > + | WITHIN '(' expr ',' expr ')' > + { > + $$= GEOM_NEW(thd, Item_func_spatial_precise_rel(thd, $3, $5, > + Item_func::SP_WITHIN_FUNC)); > + } > ; > > /* > @@ -10671,6 +10681,47 @@ simple_window_func: > } > ; > > +inverse_distribution_function: > + inverse_distribution_function_def WITHIN GROUP_SYM > + '(' > + { Select->prepare_add_window_spec(thd); } > + order_by_single_element_list ')' OVER_SYM > + '(' opt_window_ref opt_window_partition_clause ')' > + { > + LEX *lex= Lex; > + if (Select->add_window_spec(thd, lex->win_ref, > + Select->group_list, > + Select->order_list, > + NULL)) > + MYSQL_YYABORT; > + $$= new (thd->mem_root) Item_window_func(thd, (Item_sum *) $1, > + thd->lex->win_spec); > + if ($$ == NULL) > + MYSQL_YYABORT; > + if (Select->add_window_func((Item_window_func *) $$)) > + MYSQL_YYABORT; > + } > + ; > + > +inverse_distribution_function_def: > + PERCENTILE_CONT_SYM '(' expr ')' > + { > + $$= new (thd->mem_root) Item_sum_percentile_cont(thd, $3); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + | PERCENTILE_DISC_SYM '(' expr ')' > + { > + $$= new (thd->mem_root) Item_sum_percentile_disc(thd, $3); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + ; > + > See if you can make this accept only one order element. > +order_by_single_element_list: > + ORDER_SYM BY order_list > + ; > + > window_name: > ident > {

1 0

[Maria-developers] Patch for MDEV-12874 - UPDATE statements with the same source and target
by jerome brauge 29 Aug '17

29 Aug '17

Hello, Here is a patch for MDEV-12874 (on branch 10.2-ext). Can anyone review it ? Alexander, during my work on this MDEV, I've found a new issue with update statement and Oracle compatibility. create table t1 (c1 integer, c2 integer, c3 integer); insert into t1(c1,c2,c3) values (1,1,1); update t1 set c1 = c1+1, c2 = c1+1, c3 = c2+1; select * from t1; Mariadb : +------+------+------+ | c1 | c2 | c3 | +------+------+------+ | 2 | 3 | 4 | +------+------+------+ 1 row in set (0.00 sec) Oracle : DVTORA> C1 C2 C3 ---------- ---------- ---------- 2 2 2 Can you open a new MDEV for this ? Best regards, Jérôme.

3 8

[Maria-developers] Oracle strings functions compatibility : substr
by jerome brauge 17 Aug '17

17 Aug '17

Hello Alexander, Here is a patch for function SUBSTR in sql_mode=oracle (If position is 0, then it is treated as 1). Regards, Jérôme.

1 0

[Maria-developers] sys_vars.sysvars_server_embedded test failure
by Alexander Barkov 15 Aug '17

15 Aug '17

Hello Igor, sys_vars.sysvars_server_embedded is failing. It looks like it's related to your recent optimizer change. -SESSION_VALUE index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on -GLOBAL_VALUE index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on +SESSION_VALUE index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_grouping_derived=on +GLOBAL_VALUE index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown= Can you please fix this? Thanks!

1 0

[Maria-developers] Valgrind reports uninitialized value on "if (!with_two_phase_optimization..."
by Alexander Barkov 15 Aug '17

15 Aug '17

Hello Igor, Please check the output of "mtr --valgrind main.sp" I'm getting these warnings: line ==6490== Thread 6: ==6490== Conditional jump or move depends on uninitialised value(s) ==6490== at 0x7762A1: JOIN::optimize() (sql_select.cc:1123) ==6490== by 0x77F2A7: mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:3721) ==6490== by 0x773994: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:376) ==6490== by 0x73F055: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:6431) ==6490== by 0x7354C9: mysql_execute_command(THD*) (sql_parse.cc:3631) ==6490== by 0x742A48: mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) (sql_parse.cc:7886) ==6490== by 0x730684: dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) (sql_parse.cc:1814) ==6490== by 0x72EFF7: do_command(THD*) (sql_parse.cc:1372) ==6490== by 0x883380: do_handle_one_connection(CONNECT*) (sql_connect.cc:1354) ==6490== by 0x883100: handle_one_connection (sql_connect.cc:1260) ==6490== by 0x10D6329: pfs_spawn_thread (pfs.cc:1862) ==6490== by 0x4E3C619: start_thread (in /usr/lib64/libpthread-2.22.so) ==6490== by 0x6DB75FC: clone (in /usr/lib64/libc-2.22.so) ^ Found warnings in /home/bar/maria-git/server.10.2-ext/mysql-test/var/6/log/mysqld.1.err Thanks!

1 0

[Maria-developers] history of lf_hash
by MARK CALLAGHAN 14 Aug '17

14 Aug '17

Comments for lf_hash are sparse. Does it implement a well-known algorithm? https://github.com/mysql/mysql-server/blob/8.0/mysys/lf_hash.cc https://github.com/mysql/mysql-server/blob/8.0/include/lf.h -- Mark Callaghan mdcallag(a)gmail.com

2 1

[Maria-developers] On Recursive CTE and CONNECT BY
by Sergey Petrunia 12 Aug '17

12 Aug '17

Hi Vicentiu, This is a followup to the Wednesday discussion. Background: see attached file for two queries. One uses a Recursive-CTE, another uses CONNECT-BY syntax. They produce different results for some reason. Initial guess about the cause was depth-first vs breadth-first traversal. Short answer: No it's not. The reason is that Recursive-CTE query is not equivalent to the CONNECT-BY query. Results from the recursive-CTE query: ID N P CYCLE PATH 1 A 0 /1A 2 B A 0 /1A/2B 3 C A 0 /1A/3C 4 D B 0 /1A/2B/4D 5 D C 0 /1A/3C/5D 6 E D 0 /1A/3C/5D/6E 6 E D 0 /1A/2B/4D/6E 7 F E 0 /1A/2B/4D/6E/7F 7 F E 0 /1A/3C/5D/6E/7F 8 C F 0 /1A/3C/5D/6E/7F/8C 8 C F 0 /1A/2B/4D/6E/7F/8C 5 D C 0 /1A/2B/4D/6E/7F/8C/5D 12 rows selected. Results from CONNECT-BY query: ID N P CYCLE Path 1 A 0 /1A 2 B A 0 /1A/2B 4 D B 0 /1A/2B/4D 6 E D 0 /1A/2B/4D/6E 7 F E 0 /1A/2B/4D/6E/7F 8 C F 1 /1A/2B/4D/6E/7F/8C 3 C A 0 /1A/3C 5 D C 0 /1A/3C/5D 6 E D 0 /1A/3C/5D/6E 7 F E 1 /1A/3C/5D/6E/7F 10 rows selected. The rows that are missing are: ID N P CYCLE PATH 8 C F 0 /1A/3C/5D/6E/7F/8C 5 D C 0 /1A/2B/4D/6E/7F/8C/5D (denote this as MISSING-ROWS) Another thing to note is that the CTE query has returned CYCLE=0 for all rows, while the CONNECT BY one has returned two rows with CYCLE=1: 8 C F 1 /1A/2B/4D/6E/7F/8C 7 F E 1 /1A/3C/5D/6E/7F Note that these rows are parents of rows in MISSING-ROWS. I don't think the issue is caused by breadth-first vs depth-first traversal differences. The issue is caused by us not understanding how Oracle's NOCYCLE clause works. Take a look at the original-dataset.jpg, and PATH columns of MISSING-ROWS We see that the path forms the entire loop (or "cycle"). Oracle's NOCYCLE seems to be about *not* constructing the entire loop. To rule out breadth-first vs depth-first differences, let's make an example where there's just one loop: create table g3(a int, b int); insert into g3 values (1, 2); insert into g3 values (2, 3); insert into g3 values (3, 4); insert into g3 values (4, 5); insert into g3 values (5, 6); insert into g3 values (6, 7); insert into g3 values (7, 4); select a, CONNECT_BY_ISCYCLE as cycle, SYS_CONNECT_BY_PATH(a, '/') "Path" from g3 start with a=1 connect by NOCYCLE prior b= a; A CYCLE Path 1 0 /1 2 0 /1/2 3 0 /1/2/3 4 0 /1/2/3/4 5 0 /1/2/3/4/5 6 1 /1/2/3/4/5/6 The row with a=7 is not listed. If it did, the path would have had a complete cycle. It is a bit counter intuitive that /1/2/3/4/5/6 has CYCLE=1, while actually it does not form a cycle. It's one step short of doing so. The documentation is vague about this, but here's somebody thinking the same on the internet: http://www.sql.ru/forum/255418/zadachka-anagrammy-na-sql?hl=%e0%ed%e0%e3%f0… : While posting messages on MetaLink, I think I finally got the logic of NOCYCLE. Clue was right there in the name itself. NOCYCLE means do not cycle. So if you have a hierarchical query without NOCYCLE and a cycle is detected Oracle does not know how to handle it and throws ORA-01436. If NOCYCLE is used Oracle simply cuts of part of a branch right at loop point without raising an error (another word it simply does not follow the loop). And that explains why CONNECT_BY_ISCYCLE is set to 1 for the last row in first iteration of the loop. Otherwise, we would not get 1 at all. Now next logical step would be to implement CYCLE. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Win32 - Sysvar 'optimizer_switch' failed 'def_val < my_set_bits(typelib.count)'
by jerome brauge 11 Aug '17

11 Aug '17

Hi All, Build of last 10.2-ext fail on Win32 with message : Sysvar 'optimizer_switch' failed 'def_val < my_set_bits(typelib.count)' The problem seems to be in sys_vars.ic (Sys_var_set) : > > { > option.var_type|= GET_SET; > global_var(ulonglong)= def_val; > SYSVAR_ASSERT(typelib.count > 0); > SYSVAR_ASSERT(typelib.count <= 64); > SYSVAR_ASSERT(def_val <= my_set_bits(typelib.count)); Patch b14e2b04 introduce a new optimizer_switch and now 'optimizer_switch' has 32 values. my_set_bits works with unit32. A new function like this static inline uint64 my_set_bits64(int n) { return (((1ULL << (n - 1)) - 1) << 1) | 1; } might solve this issue. Regards.

1 0

[Maria-developers] Virtual columns in sql_mode=ORACLE
by jerome brauge 11 Aug '17

11 Aug '17

Hello Alexander, Something seems to be wrong in sql_yacc_ora.yy. (10.2-ext branch) We can't create a table with a virtual column because the grammar try to reduce the rule sp_cursor_name_and_offset. CREATE TABLE t1 (c1 INTEGER, c2 INTEGER AS (c1 MOD 10) VIRTUAL) ENGINE=InnoDb; ERROR 1324 (42000): Undefined CURSOR: c1 Regards, Jérôme.

2 1

[Maria-developers] Default value on column in sql_mode=ORACLE
by jerome brauge 11 Aug '17

11 Aug '17

Hello Alexander, Do you know why STRICT_TRANS_TABLES is not set in sql_mode=ORACLE ? The following script should not work: set sql_mode=oracle; create table t1 (c1 integer not null,c2 integer not null); insert into t1 (c1) values(1); Query OK, 1 row affected, 1 warning Regards, Jérôme.

2 1

Re: [Maria-developers] MDEV-13439: Database permissions are not enough to run a subquery with GROUP BY within a view
by Sergei Golubchik 09 Aug '17

09 Aug '17

Hi, Oleksandr! On Aug 07, Oleksandr Byelkin wrote: > revision-id: cf66c30ce592ad88d447568272bdced2046f2906 (mariadb-10.2.7-20-gcf66c30ce59) > parent(s): bcc10a5a447805ce64aa13ee6a037c1618219616 > committer: Oleksandr Byelkin > timestamp: 2017-08-07 13:42:35 +0200 > message: > > MDEV-13439: Database permissions are not enough to run a subquery with GROUP BY within a view > > The bug is result adding ability to have derived tables inside views. > Fixed checks should be a switch between view/derived or select derived and information schema. > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 77f502d3a7b..9bffcbe6a9e 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -7594,8 +7594,11 @@ bool check_grant(THD *thd, ulong want_access, TABLE_LIST *tables, > /* > It is subquery in the FROM clause. VIEW set t_ref->derived after > table opening, but this function always called before table opening. > + > + NOTE: is_derived() can't be used here bacause subquery in this case typo "bacause" > + the FROM clase (derived tables) can be not be marked yet. > */ > - if (!t_ref->referencing_view) > + if (t_ref->is_anonymous_derived_table() || t_ref->schema_table) > { > /* > If it's a temporary table created for a subquery in the FROM > diff --git a/sql/sql_derived.cc b/sql/sql_derived.cc > index 649f745fdc4..8f8d051a976 100644 > --- a/sql/sql_derived.cc > +++ b/sql/sql_derived.cc > @@ -822,10 +822,12 @@ bool mysql_derived_prepare(THD *thd, LEX *lex, TABLE_LIST *derived) > table->derived_select_number= first_select->select_number; > table->s->tmp_table= INTERNAL_TMP_TABLE; > #ifndef NO_EMBEDDED_ACCESS_CHECKS > - if (derived->referencing_view) > + if (derived->is_view()) > table->grant= derived->grant; > else > { > + DBUG_ASSERT(derived->is_derived()); > + DBUG_ASSERT(derived->is_anonymous_derived_table()); Hmm, if derived->is_derived() is always true here, why there's an if() below? If it's redundant - please, remove it. > table->grant.privilege= SELECT_ACL; > if (derived->is_derived()) > derived->grant.privilege= SELECT_ACL; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] commit_checkpoint_request() vs. thd_get_durability_property() (in relation to MDEV-11937)
by Kristian Nielsen 08 Aug '17

08 Aug '17

Monty asked me to fix MDEV-11937. This particular one is a performance regression in InnoDB commit. But there is a wider problem that I thought I should explain, so it can be perhaps avoided in the future. MariaDB and MySQL use different mechanisms for storage engines to avoid having to fsync during commit when binlog is enabled. In MariaDB, storage engines implement the commit_checkpoint_request() handlerton method. In MySQL, storage engines call thd_get_durability_property() to check if they can avoid fsync. The problem is that somehow the thd_get_durability_property() function was introduced into MariaDB code, but it is completely non-functional. So now there is code in InnoDB, TokuDB and RocksDB that calls this function and does not work correctly. This lead to performance regression due to extra fsync() calls. This seems to me a serious problem. Now new code can be merged and compile fine, where in reality it is wrong. There really should not be two separate and different mechanisms for the same thing, and certainly not with one of them non-functional. The "expected" approach would be to remove thd_get_durability_property() and update storage engines to use the corresponding MariaDB APIs (commit_ordered() and commit_checkpoint_request()). This should not be hard. A simple commit_checkpoint_request() implementation can just fsync all transactions immediately (similar to what MySQL does). A more detailed implementation can avoid any extra fsyncs, and just asynchroneously notify the upper layer with commit_checkpoint_notify_ha() later when such fsync happens normally (this is what InnoDB does). See comments in handler.h for details. Or was the intention to eventually replace the whole MariaDB binlog group commit implementation with the MySQL one, to make MariaDB less divergent? This would require a number of changes to MariaDB binlog and replication. The binlog recovery code should be replaced (MySQL does not have the ability to recover from more than one binlog). The binlog group commit code must be replaced as well, and the commit_ordered() mechanism removed. I think this would also require a re-design of the MariaDB in-order parallel replication. MySQL has some optional mechanism for in-order, but my understanding is that it is not sufficient to do optimistic parallel replication. I am not intimately familiar with the MySQL code though. I hope this helps. The wider problem behind MDEV-11937 is one of policy more than one of code bugs, so not too much else I can do to address it. ----------------------------------------------------------------------- Incidentally, I noticed some code in InnoDB trx0trx.cc: /* We set the HA_IGNORE_DURABILITY during prepare phase of binlog group commit to not flush redo log for every transaction here. So that we can flush prepared records of transactions to redo log in a group right before writing them to binary log during flush stage of binlog group commit. */ So the idea is to do group prepare with the same group of transactions that will later group commit to the binlog. In MariaDB, this concept does not exist. Storage engine prepares are allowed to run in parallel and in any order compared to binlog commit. So the InnoDB group prepare can include more transactions than participate in the binlog commit (but also less, of course). IIRC, the important thing is to ensure that all transactions are durably prepared in storage engines before being written to the binlog. In MariaDB, there is MYSQL_LOG_BIN::group_commit_queue that holds the list of transactions to be group committed to the binlog. A similar mechanism in MariaDB might use the group_commit_queue as the set of transactions to send to storage engines for group prepare. But if we wait for a prepare fsync() after building this list, some transactions that could prepare during this fsync may be unnecesarily delayed to the next binlog commit. The MySQL 5.7 code grabs the list of transactions to binlog group commit, and then flushes the log in _all_ storage engines, unconditionally. That just seems horrybly wrong, but I can't see any other way to read the code (from MYSQL_BIN_LOG::process_flush_stage_queue()): ha_flush_logs(NULL, true); It even does so while holding LOCK_log :-( I guess the MySQL idea is that there is only one storage engine anyway, InnoDB. - Kristian.

3 3

Re: [Maria-developers] 4c26b71c77d: MDEV-8960: Can't refer the same column twice in one ALTER TABLE
by Sergei Golubchik 04 Aug '17

04 Aug '17

Hi, Jan! A question and a suggestion, see below On Jul 27, jan wrote: > revision-id: 4c26b71c77d58fbeb3192e2142ef66efe5f122dd (mariadb-10.0.31-43-g4c26b71c77d) > parent(s): a8c1817846eb9f3fd8738e80db87cff642b9a78b > author: Jan Lindström > committer: Jan Lindström > timestamp: 2017-07-27 13:17:13 +0300 > message: > > MDEV-8960: Can't refer the same column twice in one ALTER TABLE > > Problem was that if column was created in alter table when > it was refered again it was not tried to find from list > of current columns. > > mysql_prepare_alter_table: > There is two cases > (1) If alter table adds a new column and then later alter > changes the field definition, there was no check from > list of new columns, instead an incorrect error was given. > (2) If alter table adds a new column and then later alter > changes the default, there was no check from list of > new columns, instead an incorrect error was given. > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 2eff8fd5e2f..3ba3ec6847e 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -7503,9 +7503,25 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > { > if (def->change && ! def->field) > { > - my_error(ER_BAD_FIELD_ERROR, MYF(0), def->change, > - table->s->table_name.str); > - goto err; > + /* > + Check if there is modify for newly added field. > + */ > + Create_field *find; > + find_it.rewind(); > + while((find=find_it++)) > + { > + if (!my_strcasecmp(system_charset_info,find->field_name, def->field_name)) > + break; > + } > + > + if (find && !find->field) Can it be here that find->field != NULL? I don't see how. But if yes, the ER_BAD_FIELD_ERROR looks rather weird in that case. If not - simple if (find) would be less confusing. > + find_it.remove(); > + else > + { > + my_error(ER_BAD_FIELD_ERROR, MYF(0), def->change, > + table->s->table_name.str); > + goto err; > + } > } > /* > Check that the DATE/DATETIME not null field we are going to add is > @@ -7571,6 +7587,29 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > find_it.after(def); // Put column after this > } > } > + /* > + Check if there is alter for newly added field. > + */ > + alter_it.rewind(); > + Alter_column *alter; > + while ((alter=alter_it++)) > + { > + if (!my_strcasecmp(system_charset_info,def->field_name, alter->name)) > + break; > + } > + if (alter) > + { > + if (def->sql_type == MYSQL_TYPE_BLOB) > + { > + my_error(ER_BLOB_CANT_HAVE_DEFAULT, MYF(0), def->change); > + goto err; > + } > + if ((def->def=alter->def)) // Use new default > + def->flags&= ~NO_DEFAULT_VALUE_FLAG; > + else > + def->flags|= NO_DEFAULT_VALUE_FLAG; > + alter_it.remove(); > + } Please, move this into a function. Don't just copy 20 lines around. For example if (apply_alter_column(alter_it, def, field->field_name)) goto err; > } > if (alter_info->alter_list.elements) > { Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Review 2 of compressed columns
by Michael Widenius 04 Aug '17

04 Aug '17

Hi! Here is a review for the patches marked with fixup! MDEV-11371 - column compression > commit f1a3cd40ecc4c144ac32d43ab5c1e88c8e4fae7e > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Mon Jul 31 16:07:39 2017 +0400 > > fixup! MDEV-11371 - column compression > > In worst case we have to store data length + 1 for compressed data, > we should increase field length by 1 internally so that data is not > truncated. > > diff --git a/mysql-test/t/column_compression.test b/mysql-test/t/column_compression.test > index 256e96c..d3f8481 100644 > --- a/mysql-test/t/column_compression.test > +++ b/mysql-test/t/column_compression.test > @@ -54,8 +54,8 @@ SHOW CREATE TABLE t1; > --cat_file $MYSQLD_DATADIR/test/t1.CSV > DROP TABLE t1; > > ---echo # Test fields with length equal to data length > -CREATE TABLE t1(a VARCHAR(10) COMPRESSED); > +--echo # Test fields that don't fit data > +CREATE TABLE t1(a VARCHAR(9) COMPRESSED); > --error ER_DATA_TOO_LONG > INSERT INTO t1 VALUES(REPEAT('a', 10)); > INSERT INTO t1 VALUES(REPEAT(' ', 10)); Please also insert a correct row, and not only edge cases. (You may do it already, but I can't see that in the diff) > @@ -70,3 +70,12 @@ INSERT INTO t1 VALUES(REPEAT(' ', 255)); > SET column_compression_threshold=DEFAULT; > SELECT a, LENGTH(a) FROM t1; > DROP TABLE t1; > + > +--echo # Corner case: VARCHAR(255) COMPRESSED must have 2 bytes pack length > +CREATE TABLE t1(a VARCHAR(255) COMPRESSED); > +SHOW CREATE TABLE t1; > +SET column_compression_threshold=300; > +INSERT INTO t1 VALUES(REPEAT('a', 255)); > +SET column_compression_threshold=DEFAULT; > +SELECT a, LENGTH(a) FROM t1; > +DROP TABLE t1; Please insert also a row without column_compression_threshold, just to cover all cases in the test case. > diff --git a/sql/field.cc b/sql/field.cc > index 216fb95..e1dc193 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -7555,7 +7555,8 @@ void Field_varstring::sql_type(String &res) const > length= cs->cset->snprintf(cs,(char*) res.ptr(), > res.alloced_length(), "%s(%d)", > (has_charset() ? "varchar" : "varbinary"), > - (int) field_length / charset()->mbmaxlen); > + (int) field_length / charset()->mbmaxlen - > + MY_TEST(compression_method())); > res.length(length); > if ((thd->variables.sql_mode & (MODE_MYSQL323 | MODE_MYSQL40)) && > has_charset() && (charset()->state & MY_CS_BINSORT)) > @@ -10015,6 +10016,8 @@ void Column_definition::create_length_to_internal_length(void) > case MYSQL_TYPE_STRING: > case MYSQL_TYPE_VARCHAR: > length*= charset->mbmaxlen; > + if (sql_type == MYSQL_TYPE_VARCHAR && compression_method()) > + length++; Add a comment: /* Compressed columns needs one extra byte to store the compression method. This byte is invisible to the end user, but not for the storage engine. */ > key_length= length; > pack_length= calc_pack_length(sql_type, length); > break; <cut> > commit 274ab73a89573554d07a6de1deb846ad47eb1004 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Thu Jul 27 13:06:33 2017 +0400 > > fixup! MDEV-11371 - column compression > > Fixed BLOB COMPRESSED -> BLOB COMPRESSED replication. > > The problem was that Field_blob::unpack gets compressed data and calls > Field_blob_compressed::store(), which compressed it again. Fixed by calling > Field_blob::set_ptr() instead of ::store(). > > Note that now no data copy is performed and blob is pointing to data allocated > by replication routines. It used to work this way before 06fb8c2d10, but still > should be reviewed carefully to make sure no copy is actually needed. > patch ok. > commit 93e00970100503c3f365063d9261119f096bee48 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Thu Jul 27 11:44:20 2017 +0400 > > Get rid of Field::do_save_field_metadata() > > It doesn't serve any purpose, but generates extra virtual function call. > ok > commit e1247d4991bb4ebd3f98788c7bfe31a34faf7b69 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Thu Jul 27 10:58:47 2017 +0400 > > fixup! MDEV-11371 - column compression > > This is an addition to "Fixed compressed columns to work with partitioning". > > Mroonga calls key_copy() on write/delete, when column is not in read_set. > Adjust read_set for the duration of val_str(). > > diff --git a/sql/field.cc b/sql/field.cc > index ec82984..ab96ee9 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -7659,8 +7659,12 @@ uint Field_varstring::get_key_image(uchar *buff, uint length, > { > String val; > uint local_char_length; > + my_bitmap_map *old_map; > > + old_map= dbug_tmp_use_all_columns(table, table->read_set); > val_str(&val, &val); > + dbug_tmp_restore_column_map(table->read_set, old_map); > + Shouldn't this be in mronga code and not in MariaDB? Looking at the original code, we didn't check for read_set before either for get_key_image, so I assume this is ok. > local_char_length= val.charpos(length / field_charset->mbmaxlen); > if (local_char_length < val.length()) > val.length(local_char_length); > commit 94040156d8007663beb6785ca653d41b26758301 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Wed Jul 26 17:28:57 2017 +0400 > > fixup! MDEV-11371 - column compression > > Get rid of hardcoded "8" (zlib compression method id) by moving towards > support for multiple compression methods. > commit 6c8b1e563c7456eb0b8ca11a621333237de37ec4 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Tue Jul 25 13:27:07 2017 +0400 > > fixup! MDEV-11371 - column compression > > Fixed compressed columns to work with partitioning. > > Reviewer note: this patch reuses get_key_image()/set_key_image() of > Field_varstring to avoid code duplication. Cons: this adds virtual function > calls val_str() and store(). > ok > commit 9c85ad8d105e74baae0b6aba0db01b4d6aac7ba6 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Thu Jul 20 16:57:46 2017 +0400 > > fixup! MDEV-11371 - column compression ok. The only thing left is now to fix replication to handle compressed columns. We also have agreed to handle the case where the master and slave have different COMPRESSED column attributes for VARCHAR and BLOB. Regards, Monty

1 0

[Maria-developers] Debian 9 support for MaxScale
by Martin Toth 03 Aug '17

03 Aug '17

Hi all, I would like to ask if there is some release date for fixing https://jira.mariadb.org/browse/MXS-1292 . Debian 9 should be included as a build target ASAP. I am installing only D9 systems on new servers and I am unable to use builds for MaxScale, so its unusable in D9. Thanks. Martin

3 7

Re: [Maria-developers] 1 Transaction and several threads
by Jan Lindström 03 Aug '17

03 Aug '17

Hi, If we talk about bulk load I would write directly .ibd file format pages, now as InnoDB has always some order between pages doing this in parallel has challenges, we can read and parse input, sort it, reformat to InnoDB pages, write these pages (maybe you could do this also parallel). To speed this up, I would remove secondary keys that are not needed from table and rebuild them after bulk load. R: Jan On Thu, Aug 3, 2017 at 12:09 PM, Oleksandr Byelkin <sanja(a)mariadb.com> wrote: > Hi! > > I am thinking about efficient inserting in a table using several threads > (LOAD), is it possible (and how difficult if it does) to make inserts with > help of several threads abut inside one InnoDB transaction (maybe one XA > transaction)? >

3 2

[Maria-developers] Git jira integration plugin
by Daniel Black 03 Aug '17

03 Aug '17

For those that haven't seen it. https://www.npmjs.com/package/jira-git

1 0

[Maria-developers] Patch for MDEV-13417 - UPDATE produces wrong values if an updated column is later used as an update source
by jerome brauge 02 Aug '17

02 Aug '17

Hello, Here is a patch for MDEV-13417 (on branch 10.2-ext). Can anyone review it ? Best regards, Jérôme.

1 0

[Maria-developers] HAVE_LONG_LONG check
by Aleksey Midenkov 20 Jul '17

20 Jul '17

Do we need HAVE_LONG_LONG in item.cc: #ifdef HAVE_LONG_LONG case MYSQL_TYPE_LONGLONG: field= new (mem_root) Field_longlong((uchar*) 0, max_length, null_ptr, 0, Field::NONE, name, 0, unsigned_flag); break; #endif ? There is no such check in field.cc: case MYSQL_TYPE_LONGLONG: return new (mem_root) Field_longlong(ptr,field_length,null_pos,null_bit, unireg_check, field_name, f_is_zerofill(pack_flag) != 0, f_is_dec(pack_flag) == 0);

2 1

Re: [Maria-developers] 0aa6217: MDEV-13065 rpl.rpl_mdev-11092 fails sporadically in buildbot
by Sergei Golubchik 19 Jul '17

19 Jul '17

Hi, Sachin! On Jul 19, sachin wrote: > revision-id: 0aa621722d74d317f39b4c463800959ee84a6097 (mariadb-10.0.31-33-g0aa6217) As discussed, it should probably go into 5.5 > parent(s): 2f07709aed863ea1852a78a0d6ab718e49df8cca > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2017-07-19 11:59:43 +0530 > message: > > MDEV-13065 rpl.rpl_mdev-11092 fails sporadically in buildbot > > Problem > rpl.rpl_mdev-11092 fails in buildbot because after starting slave in > wait_for_slave_sql_error_and_skip.inc slave is started but there may be > chances that we have not skipped the last error and Last_SQL_Errno is > still not zero untill the end of rpl_end.inc , which will compare > Last_SQL_Errno to 0. So in this this case rpl_mdev-11092 fails. > > Solution > After starting slave in wait_for_slave_sql_error_and_skip.inc we will wait for > Last_SQL_Errno to become 0. > > --- > mysql-test/include/wait_for_slave_sql_error_and_skip.inc | 15 +++++++++++++++ > 1 file changed, 15 insertions(+) > > diff --git a/mysql-test/include/wait_for_slave_sql_error_and_skip.inc b/mysql-test/include/wait_for_slave_sql_error_and_skip.inc > index 9246c18..60c8319 100644 > --- a/mysql-test/include/wait_for_slave_sql_error_and_skip.inc > +++ b/mysql-test/include/wait_for_slave_sql_error_and_skip.inc > @@ -66,6 +66,21 @@ if (!$slave_skip_counter) { > } > source include/start_slave.inc; > > +# Wait for 5 seconds Last_SQL_Errno to become zero > +--let $wait_counter=50 > +while ($wait_counter) > +{ > + let $wait_condition= query_get_value("SHOW SLAVE STATUS", Last_SQL_Errno, 1); > + if ($wait_condition == $slave_sql_errno) > + { > + real_sleep 0.1; > + dec $wait_counter; > + } > + if (!$wait_condition) > + { > + --let $wait_counter = 0; > + } > +} I'd suggest to use include/wait_for_slave_param.inc here, like let $slave_param= Last_SQL_Errno; let $slave_param_value= 0; source include/wait_for_slave_param.inc; and put a comment before it, like # start_slave.inc returns when Slave_SQL_Running=Yes. But the slave # thread sets it before clearing Last_SQL_Errno. So we have to wait # for Last_SQL_Errno=0 separately. > --let $include_filename= wait_for_slave_sql_error_and_skip.inc [errno=$slave_sql_errno] > --source include/end_include_file.inc Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 05b678bc8c1: MDEV-12489 The select stmt may fail due to "having clause is ambiguous" unexpected
by Oleksandr Byelkin 12 Jul '17

12 Jul '17

Hi, Sergei! Am 12.07.2017 um 18:13 schrieb serg(a)mariadb.org: > revision-id: 05b678bc8c166c9015ca3dc3ccd9e9c89094f3ca (mariadb-5.5.56-42-g05b678bc8c1) > parent(s): c83d6ff881dcfd1cc68124a9663c2a31ca7e2dce > author: Sergei Golubchik > committer: Sergei Golubchik > timestamp: 2017-07-12 17:46:19 +0200 > message: > > MDEV-12489 The select stmt may fail due to "having clause is ambiguous" unexpected > > only compare field references with an alias (from the SELECT clause) > when this reference doesn't specify an explicit table name part. > OK to push! [skip]

1 0

Re: [Maria-developers] [Commits] be55bbc2b20: MDEV-7826 Server crashes in Item_subselect::enumerate_field_refs_processor
by Oleksandr Byelkin 12 Jul '17

12 Jul '17

Hi, Sergei! Am 12.07.2017 um 12:49 schrieb serg(a)mariadb.org: > revision-id: be55bbc2b206f2fe295ad4179b23791c3119fe02 (mariadb-5.5.56-39-gbe55bbc2b20) > parent(s): c5975eaea174460e774e77717e972a8d32e6c8de > author: Sergei Golubchik > committer: Sergei Golubchik > timestamp: 2017-07-12 12:49:29 +0200 > message: > > MDEV-7826 Server crashes in Item_subselect::enumerate_field_refs_processor > > upper->item can be NULL if we're referring to an aggregate function OK to push! [skip]

1 0

Re: [Maria-developers] [Commits] c5975eaea17: MDEV-7339 Server crashes in Item_func_trig_cond::val_int
by Oleksandr Byelkin 12 Jul '17

12 Jul '17

Hi! Am 12.07.2017 um 10:39 schrieb serg(a)mariadb.org: > revision-id: c5975eaea174460e774e77717e972a8d32e6c8de (mariadb-5.5.56-38-gc5975eaea17) > parent(s): f305a7ce4bccbd56520d874e1d81a4f29bc17a96 > author: Sergei Golubchik > committer: Sergei Golubchik > timestamp: 2017-07-12 10:39:02 +0200 > message: > > MDEV-7339 Server crashes in Item_func_trig_cond::val_int > > Item_in_subselect::pushed_cond_guards[] array is allocated only when > left_expr->maybe_null. And it is used (for row expressions) when > left_expr->element_index(i)->maybe_null. > > For left_expr being a multi-column subquery, its maybe_null is > always false when the subquery doesn't use tables (see > Item_singlerow_subselect::fix_length_and_dec() > and subselect_single_select_engine::fix_length_and_dec()), > otherwise it's always true. > > But row elements can be NULL regardless, so let's always allocate > pushed_cond_guards for multi-column subqueries, no matter whether > its maybe_null was forced to true or false. OK to push! [skip]

1 0

Re: [Maria-developers] [Commits] 7801756: MDEV-13826: Floating point exception in Filesort_tracker::print_json_members(Json_writer*)
by Sergey Petrunia 12 Jul '17

12 Jul '17

Hi Varun, On Wed, Jul 12, 2017 at 12:43:02AM +0530, Varun wrote: > revision-id: 780175646bb9312184d8a676e86b125f663b5156 (mariadb-10.2.2-591-g7801756) > parent(s): 3904014ed37cc2b58844e8d24a5d6aaa69d80f59 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-07-12 00:41:52 +0530 > message: > > MDEV-13826: Floating point exception in Filesort_tracker::print_json_members(Json_writer*) > > Whenever Filesort_tracker has r_loops=0, r_ouptut_rows would be 0, so we should add the value zero to the member "r_output_rows" expliciytly > > --- > mysql-test/r/analyze_format_json.result | 60 +++++++++++++++++++++++++++++++++ > mysql-test/t/analyze_format_json.test | 11 ++++++ > sql/sql_analyze_stmt.cc | 11 +++--- > 3 files changed, 78 insertions(+), 4 deletions(-) > > diff --git a/mysql-test/r/analyze_format_json.result b/mysql-test/r/analyze_format_json.result > index 686da92..436b13b 100644 > --- a/mysql-test/r/analyze_format_json.result > +++ b/mysql-test/r/analyze_format_json.result > @@ -757,3 +757,63 @@ ANALYZE > } > } > drop table t1,t2,t3; > +# > +# MDEV-13286: Floating point exception in Filesort_tracker::print_json_members(Json_writer*) > +# > +create table t0(a int); > +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); > +create table t1 (a int, b int, c int); > +insert into t1 select a,a,a from t0; > +create table t2 as select * from t1; > +analyze format=json select a, (select t2.b from t2 where t2.a<t1.a order by t2.c limit 1) from t1 where t1.a<0; > +ANALYZE > +{ > + "query_block": { > + "select_id": 1, > + "r_loops": 1, > + "r_total_time_ms": 0.0324, > + "table": { > + "table_name": "t1", > + "access_type": "ALL", > + "r_loops": 1, > + "rows": 10, > + "r_rows": 10, > + "r_total_time_ms": 0.0098, > + "filtered": 100, > + "r_filtered": 0, > + "attached_condition": "t1.a < 0" > + }, > + "subqueries": [ > + { > + "expression_cache": { > + "state": "uninitialized", > + "r_loops": 0, > + "query_block": { > + "select_id": 2, > + "read_sorted_file": { > + "r_rows": null, > + "filesort": { > + "sort_key": "t2.c", > + "r_loops": 0, > + "r_limit": "(varied across executions)", > + "r_used_priority_queue": false, Look at the "table" field below: the parameters whose values are not defined because of r_loop=0 had value of null (e.g. r_filtered, r_rows). Let's follow that pattern here: r_loops:null, r_limit:null, r_used_priority_queue: null. Please change this. Ok to push after this is addressed. > + "r_output_rows": 0, > + "table": { > + "table_name": "t2", > + "access_type": "ALL", > + "r_loops": 0, > + "rows": 10, > + "r_rows": null, > + "filtered": 100, > + "r_filtered": null, > + "attached_condition": "t2.a < t1.a" > + } > + } > + } > + } > + } > + } > + ] > + } > +} > +drop table t0,t1,t2; > diff --git a/mysql-test/t/analyze_format_json.test b/mysql-test/t/analyze_format_json.test > index 88a9077..02603bc 100644 > --- a/mysql-test/t/analyze_format_json.test > +++ b/mysql-test/t/analyze_format_json.test > @@ -212,3 +212,14 @@ GROUP BY sq ORDER BY gc; > > drop table t1,t2,t3; > > + > +--echo # > +--echo # MDEV-13286: Floating point exception in Filesort_tracker::print_json_members(Json_writer*) > +--echo # > +create table t0(a int); > +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); > +create table t1 (a int, b int, c int); > +insert into t1 select a,a,a from t0; > +create table t2 as select * from t1; > +analyze format=json select a, (select t2.b from t2 where t2.a<t1.a order by t2.c limit 1) from t1 where t1.a<0; > +drop table t0,t1,t2; > diff --git a/sql/sql_analyze_stmt.cc b/sql/sql_analyze_stmt.cc > index cc6e0f8..3e49af1 100644 > --- a/sql/sql_analyze_stmt.cc > +++ b/sql/sql_analyze_stmt.cc > @@ -43,15 +43,18 @@ void Filesort_tracker::print_json_members(Json_writer *writer) > } > > writer->add_member("r_used_priority_queue"); > - if (r_used_pq == get_r_loops()) > - writer->add_bool(true); > - else if (r_used_pq == 0) > + if (r_used_pq == 0) > writer->add_bool(false); > + else if (r_used_pq == get_r_loops()) > + writer->add_bool(true); > else > writer->add_str(varied_str); > > - writer->add_member("r_output_rows").add_ll((longlong) rint(r_output_rows / > + if (r_output_rows) > + writer->add_member("r_output_rows").add_ll((longlong) rint(r_output_rows / > get_r_loops())); > + else > + writer->add_member("r_output_rows").add_ll((longlong)0); > > if (sort_passes) > { > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] RFC: new replication feature "per-engine mysql.gtid_slave pos"
by Kristian Nielsen 11 Jul '17

11 Jul '17

I plan to implement MDEV-12179, per-engine mysql.gtid_slave_pos. Here is a description of the high-level design, as a request for comments and/or suggestion for changes. The purpose of this is to fix a serious performance issue in replication when multiple storage engines are used. Every replicated transaction updates the replication GTID position in table mysql.gtid_slave_pos. There is a large overhead if that table is using another storage engine than the rest of the transaction, as then a cross-engine XA transaction needs to be done. But if more than one storage engine is used on a server, some replicated transactions will necessarily end up being cross-engine. This problem becomes more interesting to fix with the new interest in MyRocks. Though it already exists also for eg. people using MyISAM or TokuDB simultaneously with InnoDB, for example. The basic idea is to create multiple copies of the table, eg. mysql.gtid_slave_pos_innodb and mysql.gtid_slave_pos_myrocks. This way, an InnoDB transaction can update the InnoDB version of the table, and similar for other engines. To enable the feature, the DBA would create extra copies of the table for the engines she wants: CREATE TABLE mysql.gtid_slave_pos_myrocks LIKE mysql.gtid_slave_pos; ALTER TABLE mysql.gtid_slave_pos_myrocks ENGINE=myrocks; The server will look for all tables called mysql.gtid_slave_pos* at startup, as well as on each START SLAVE. This method makes migration from earlier versions simple. MariaDB already has the ability to work with multiple redundant rows in the mysql.gtid_slave_pos table. The column "sub_id" provides a version to identify the most recent entry. So the server just needs to read multiple tables (if present), and pick the newest version amongst all rows. Thus, the new feature is compatible with older versions without any schema changes. I think it is better than automatically creating copies of the table for all engines; it is often the case that an engine (eg. TokuDB) is available in the server binary, but the user does not want any tables created in that engine. This means that no speedup is obtained if the user does not create additional mysql.gtid_slave_pos* tables. (However, things should still work fine, thanks to XA synchronising cross-engine transactions). - Kristian.

6 12

[Maria-developers] mariadb_config broken in 10.2.6
by Jeroen Ooms 07 Jul '17

07 Jul '17

Running 'mariadb_config --libs' returns invalid output. See https://github.com/Homebrew/homebrew-core/issues/15333

3 3

[Maria-developers] Confusion between variable / alias and row type.
by jerome brauge 07 Jul '17

07 Jul '17

Hello Alexander, I think there is a regression introduced with row type. Attached stored procedure creation fail with error : ERROR 1105 (HY000) at line 9: 'a' is not a row variable. Regards, Jérôme.

2 2

[Maria-developers] Review of PERCENTILE_CONT && PERCENTILE_DISC
by Vicențiu Ciorbaru 04 Jul '17

04 Jul '17

Hi Varun, Here's the review for your patch. > diff --git a/mysql-test/t/percentile.test b/mysql-test/t/percentile.test > new file mode 100644 > index 00000000000..0958fc05e7d > --- /dev/null > +++ b/mysql-test/t/percentile.test > @@ -0,0 +1,41 @@ Whenever you are adding a test case, please add the result file too. It makes things easier to review. > +CREATE TABLE student (name CHAR(10), test double, score TINYINT); > + > +INSERT INTO student VALUES > +('Chun', 0, null), ('Chun', 0, 4), > +('Esben', 1, null), ('Esben', 1, null), > +('Kaolin', 0.5, 56), ('Kaolin', 0.5, 88), > +('Tatiana', 0.8, 2), ('Tatiana', 0.8, 1); > + > + > + > +select name, percentile_disc(0.6) within group(order by score) over (partition by name) from student; > +select name, percentile_disc(test) within group(order by score) over (partition by name) from student; > +select name, percentile_disc(0.4) within group(order by score) over (partition by name) from student; > + > + > +#select name, percentile_cont(null) within group(order by score) over (partition by name) from student; > +#select name, cume_dist() over (partition by name order by score) from student; > + > + > +#normal parsing > +#select percentile_cont(0.5) within group(order by score) over w1 from student > +#window w1 AS (partition by name); > + > +# no partition clause > +#select percentile_cont(0.5) within group(order by score) over () from student; > + > + > +# only one sort allowed > +#select percentile_cont(0.5) within group(order by score) over (partition by name); > + > +#parameter value should be in the range of 0 to 1 > +#select percentile_cont(1.5) within group(order by score) over (partition by name); > + > + > +# > +#select rank() over (partition by name order by score ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) from student; > + > + > + > +drop table student; > + > diff --git a/sql/item.h b/sql/item.h > index 76ce4aa935f..3df4ec4f358 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -5215,6 +5217,14 @@ class Cached_item_item : public Cached_item > cmp(); > item= save; > } This is not used in the patch. Is there a need for it any more? > + Item* get_item() > + { > + return item; > + } Add an empty line here here. > + void clear() > + { > + null_value= false; > + } > }; > > class Cached_item_str :public Cached_item_item > @@ -5236,6 +5250,12 @@ class Cached_item_real :public Cached_item_item > Cached_item_real(Item *item_par) :Cached_item_item(item_par),value(0.0) {} > bool cmp(void); > int cmp_read_only(); Space after ). Mark this function as const > + double get_value(){ return value;} I don't quite get what this clear is supposed to do. I'd expect clear to set the null_value to true, not to false. Same for other functions in the diff. Add a comment explaining this. > + void clear() > + { Space after = > + value=0.0; > + null_value= false; > + } > }; > > class Cached_item_int :public Cached_item_item > @@ -5245,6 +5265,12 @@ class Cached_item_int :public Cached_item_item > Cached_item_int(Item *item_par) :Cached_item_item(item_par),value(0) {} > bool cmp(void); > int cmp_read_only(); Space after ). Mark this function as const. > + longlong get_value(){ return value;} > + void clear() > + { Space after = > + value=0.0; > + null_value= false; > + } > }; > > > @@ -5255,6 +5281,12 @@ class Cached_item_decimal :public Cached_item_item > Cached_item_decimal(Item *item_par); > bool cmp(void); > int cmp_read_only(); Space after ) Mark this function as const. > + my_decimal *get_value(){ return &value;}; > + void clear() > + { > + null_value= false; > + my_decimal_set_zero(&value); > + } > }; > > class Cached_item_field :public Cached_item > diff --git a/sql/item_windowfunc.cc b/sql/item_windowfunc.cc > index 6ab903a81bb..b711642c52c 100644 > --- a/sql/item_windowfunc.cc > +++ b/sql/item_windowfunc.cc > @@ -93,6 +93,17 @@ Item_window_func::fix_fields(THD *thd, Item **ref) > my_error(ER_NO_ORDER_LIST_IN_WINDOW_SPEC, MYF(0), window_func()->func_name()); > return true; > } > + > + if (only_single_element_order_list()) > + { Mark such comments as TODO, like so: TODO(varunt): <message> > + // need to change the error, the error should say that we have more than one element in the order list > + if (window_spec->order_list->elements != 1) > + { > + my_error(ER_NO_ORDER_LIST_IN_WINDOW_SPEC, MYF(0), window_func()->func_name()); > + return true; > + } > + } > + > /* > TODO: why the last parameter is 'ref' in this call? What if window_func > decides to substitute itself for something else and does *ref=.... ? > @@ -179,6 +190,16 @@ void Item_sum_dense_rank::setup_window_func(THD *thd, Window_spec *window_spec) > clear(); > } > > I don't see any added value for this extra setup_percentile_func, when it's only I guess you can just put the code in directly. > +void Item_sum_percentile_disc::setup_window_func(THD *thd, Window_spec *window_spec) > +{ > + setup_percentile_func(thd, window_spec->order_list); > +} > + This function does nothing. get_handler_by_cmp_type returns an item and then you don't do anything with it. > +void Item_sum_percentile_disc::set_type_handler(Window_spec *window_spec) > +{ > + type_handler()->get_handler_by_cmp_type(window_spec->order_list->first->item[0]->result_type()); > +} > + > bool Item_sum_dense_rank::add() > { > if (peer_tracker->check_if_next_group() || first_add) > diff --git a/sql/item_windowfunc.h b/sql/item_windowfunc.h > index 77cbd556e60..5179d1832e8 100644 > --- a/sql/item_windowfunc.h > +++ b/sql/item_windowfunc.h > @@ -16,10 +17,13 @@ class Group_bound_tracker > > Group_bound_tracker(THD *thd, SQL_I_List<ORDER> *list) > { As discussed on slack, lets keep this Group_bound_tracker as is. Better yet, add a DBUG_ASSERT(list). > - for (ORDER *curr = list->first; curr; curr=curr->next) > + if (list) > { > - Cached_item *tmp= new_Cached_item(thd, curr->item[0], TRUE); > - group_fields.push_back(tmp); > + for (ORDER *curr = list->first; curr; curr=curr->next) > + { > + Cached_item *tmp= new_Cached_item(thd, curr->item[0], TRUE); > + group_fields.push_back(tmp); > + } > } > } > > @@ -71,6 +75,19 @@ class Group_bound_tracker > return 0; > } > Add a comment as to what this function does. I don't quite like how this function works. cmp has a side effect of overwriting the cached item with the compared value. In addition this function stops if we encounter a null value and the group fields are now partially updated. I assume this is only for a single element order list. Why not make something specialised for that and not use this cached_item construct. > + bool compare_with_cache_for_null_values() > + { > + List_iterator<Cached_item> li(group_fields); > + Cached_item *ptr; > + while ((ptr= li++)) > + { > + ptr->cmp(); > + if (ptr->null_value) > + return true; > + } > + return false; > + } > + > private: > List<Cached_item> group_fields; > /* > @@ -678,6 +704,138 @@ class Item_sum_ntile : public Item_sum_window_with_row_count > ulong current_row_count_; > }; > > +class Item_sum_percentile_disc : public Item_sum_cume_dist, > + public Type_handler_hybrid_field_type > +{ > +public: > + Item_sum_percentile_disc(THD *thd, Item* arg) : Item_sum_cume_dist(thd, arg), > + Type_handler_hybrid_field_type(&type_handler_longlong), > + value(NULL), result_value(NULL), val_calculated(FALSE) {} > + > + double val_real() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; Look at Item_sum_hybrid_simple. See how Item_cache * works there. Make use of that instead of Cached_item_xxx. > + return ((Cached_item_real*) result_value)->get_value(); > + } > + > + longlong val_int() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; Same comment as for val_real() > + return ((Cached_item_int*) result_value)->get_value(); > + } > + > + my_decimal* val_decimal(my_decimal* dec) > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; Same comment as for val_decimal() > + return ((Cached_item_decimal*) result_value)->get_value(); > + } > + Lets try implementing this with Item_cache, instead of cached_item. Cached_item is designed for comparison, while Item_cache is used for storing something just to cache it. > + bool add() > + { > + Item *arg = get_arg(0); > + if (arg->is_null()) > + return true; > + /* > + need to ensure that the Item arg is constant across the entire partition > + and its value ranges between [0,1] > + */ > + value->cmp(); > + > + /* for the null values of the row, we dont count take those rows in account for calculating > + the CUME_DIST */ > + > + if(value->null_value) > + return false; > + > + Item_sum_cume_dist::add(); > + double val1= Item_sum_cume_dist::val_real(); > + /* need to check type and return value accordingly*/ > + double val2 =arg->val_real_from_decimal(); > + > + /* use Cached_item to do the comparision using cmp_read_only() */ > + > + if( val1 >= val2 && !val_calculated) > + { > + val_calculated= true; > + result_value->cmp(); > + return false; > + } > + return false; > + } > + > + enum Sumfunctype sum_func() const > + { > + return PERCENTILE_DISC_FUNC; > + } > + > + void clear() > + { > + val_calculated= false; > + value->clear(); > + result_value->clear(); > + Item_sum_cume_dist::clear(); > + } > + > + const char*func_name() const > + { > + return "percentile_disc"; > + } > + > + void update_field() {} > + void set_type_handler(Window_spec *window_spec); > + const Type_handler *type_handler() const > + {return Type_handler_hybrid_field_type::type_handler();} > + > + void fix_length_and_dec() > + { > + decimals = 10; // TODO-cvicentiu find out how many decimals the standard > + // requires. > + } > + > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_sum_percentile_disc>(thd, mem_root, this); } > + void setup_window_func(THD *thd, Window_spec *window_spec); > + void setup_percentile_func(THD *thd, SQL_I_List<ORDER> *list) > + { > + value= new_Cached_item(thd, list->first->item[0], FALSE); > + result_value= new_Cached_item(thd, list->first->item[0], FALSE); > + } > + void cleanup() > + { > + if (value) > + { > + delete value; > + value= NULL; > + } > + if(result_value) > + { > + delete result_value; > + result_value= NULL; > + } > + Item_sum_num::cleanup(); > + } > + > +private: > + Cached_item *value; > + Cached_item *result_value; > + bool val_calculated; > +}; > + > > class Item_window_func : public Item_func_or_sum > { > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index 74e1ad25183..9fda4520a05 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -868,9 +873,39 @@ class Partition_read_cursor : public Table_read_cursor > } > return 0; > } This is not something we want in Partition_read_cursor. Move the contents of it to the most derived class that needs it. > + bool next_func(ha_rows *counter) > + { > + if (next()) > + return true; > + if (!check_for_null_row()) > + { > + (*counter)++; > + } > + return false; > + } This is also not something we want in Partition_read_cursor. > + bool fetch_func(ha_rows *counter) > + { > + if (fetch()) > + return true; > + if (!check_for_null_row()) > + { > + (*counter)++; > + } > + return false; > + } Null row here has no meaning. Partition_read_cursor looks at the bound_tracker for partitions. What "row" are we talking about here. Then again, this logic can probably go into our "special" frame cursor below. > + bool check_for_null_row() > + { > + if (!end_of_partition) > + { Check compare_with_cache_for_null_values comment. > + if (order_tracker.compare_with_cache_for_null_values()) > + return true; > + } > + return false; > + } > > private: > Group_bound_tracker bound_tracker; Not something we want in Partition_read_cursor. If you need an order_tracker define a different class. > + Group_bound_tracker order_tracker; > bool end_of_partition; > }; > > @@ -1102,7 +1137,7 @@ class Frame_range_n_top : public Frame_cursor > SQL_I_List<ORDER> *partition_list, > SQL_I_List<ORDER> *order_list, > bool is_preceding_arg, Item *n_val_arg) : So this gets reverted as discussed on slack. > - cursor(thd, partition_list), n_val(n_val_arg), item_add(NULL), > + cursor(thd, partition_list, NULL), n_val(n_val_arg), item_add(NULL), > is_preceding(is_preceding_arg) > { > DBUG_ASSERT(order_list->elements == 1); > @@ -1241,7 +1276,7 @@ class Frame_range_n_bottom: public Frame_cursor > SQL_I_List<ORDER> *partition_list, > SQL_I_List<ORDER> *order_list, > bool is_preceding_arg, Item *n_val_arg) : So this gets reverted as discussed on slack. > - cursor(thd, partition_list), n_val(n_val_arg), item_add(NULL), > + cursor(thd, partition_list, NULL), n_val(n_val_arg), item_add(NULL), > is_preceding(is_preceding_arg), added_values(false) > { > DBUG_ASSERT(order_list->elements == 1); > @@ -1371,7 +1406,7 @@ class Frame_range_current_row_bottom: public Frame_cursor > Frame_range_current_row_bottom(THD *thd, > SQL_I_List<ORDER> *partition_list, > SQL_I_List<ORDER> *order_list) : So this gets reverted as discussed on slack. > - cursor(thd, partition_list), peer_tracker(thd, order_list) > + cursor(thd, partition_list, NULL), peer_tracker(thd, order_list) > { > } > > @@ -1658,6 +1693,35 @@ class Frame_unbounded_following_set_count : public Frame_unbounded_following > } > }; > As discussed on slack, this needs to be renamed to something that makes sense. > +class Frame_unbounded_following_set_count_special : public Frame_unbounded_following_set_count > +{ > +public: > + Frame_unbounded_following_set_count_special( > + THD *thd, > + SQL_I_List<ORDER> *partition_list, SQL_I_List<ORDER> *order_list) : > + Frame_unbounded_following_set_count(thd, partition_list, order_list) > + {} > + next_partition can have the logic of fetch_func directly in it. > + void next_partition(ha_rows rownum) > + { > + ha_rows num_rows_in_partition= 0; > + if (cursor.fetch_func(&num_rows_in_partition)) > + return; > + Dangerous to have a while loop like this. Lets put next_func code here directly. > + /* Walk to the end of the partition, find how many rows there are. */ > + while (!cursor.next_func(&num_rows_in_partition)); > + > + List_iterator_fast<Item_sum> it(sum_functions); > + Item_sum* item; > + while ((item= it++)) > + { > + Item_sum_window_with_row_count* item_with_row_count = > + static_cast<Item_sum_window_with_row_count *>(item); > + item_with_row_count->set_row_count(num_rows_in_partition); > + } > + } > +}; > + > ///////////////////////////////////////////////////////////////////////////// > // ROWS-type frame bounds > ///////////////////////////////////////////////////////////////////////////// > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index a0bbf39b138..6c64477f08d 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -10671,6 +10676,45 @@ simple_window_func: > } > ; > > +inverse_distribution_function: > + inverse_distribution_function_def WITHIN GROUP_SYM > + '(' > + { Select->prepare_add_window_spec(thd); } > + order_by_single_element_list ')' OVER_SYM > + '(' opt_window_ref opt_window_partition_clause ')' > + { > + LEX *lex= Lex; > + if (Select->add_window_spec(thd, lex->win_ref, > + Select->group_list, > + Select->order_list, > + NULL)) > + MYSQL_YYABORT; > + $$= new (thd->mem_root) Item_window_func(thd, (Item_sum *) $1, > + thd->lex->win_spec); > + if ($$ == NULL) > + MYSQL_YYABORT; > + if (Select->add_window_func((Item_window_func *) $$)) > + MYSQL_YYABORT; > + } > + ; > + > +inverse_distribution_function_def: > + PERCENTILE_CONT_SYM '(' expr ')' Indentation for { and } is not correct here. Please check this file and do the same. > + { mark it as TODO(varun): > + //Not yet started implementing > + } > + | PERCENTILE_DISC_SYM '(' expr ')' Same, check indentation for { and } > + { > + $$= new (thd->mem_root) Item_sum_percentile_disc(thd, $3); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + ; > + Indentation check! > +order_by_single_element_list: > + ORDER_SYM BY order_list > + ; > + > window_name: > ident > {

1 0

[Maria-developers] MDEV-13240 Wrong warning with MAX(datetime_field) OVER (...)
by Alexander Barkov 04 Jul '17

04 Jul '17

Hello Vicentiu, can you please review a patch for MDEV-13240? The patch is for bb-10.2-ext / 10.3. Thanks.

1 0

Re: [Maria-developers] [Commits] 2d6649aa24c: MDEV-13186: main.win failure post MDEV-12336
by Oleksandr Byelkin 04 Jul '17

04 Jul '17

Hi, Vicentiu! Am 27.06.2017 um 11:31 schrieb vicentiu: > revision-id: 2d6649aa24c77321df349de5cee576dab1e344f4 (mariadb-10.2.6-55-g2d6649aa24c) > parent(s): c036d5ada7a5816d25915a2c6e0b6836b0cd6743 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2017-06-27 12:26:38 +0300 > message: > > MDEV-13186: main.win failure post MDEV-12336 > > During statement preparation st_order::item gets set to a value in > ref_ptr_array. During statement execution we were overriding that value, > causing subsequent checks for window functions to return true. > > Whenever we do any setting from ref_ptr_array, make sure to always > store the value in all_fields as well. > > For function items containing window functions, as MDEV-12336 has > discovered, we don't need to create a separate Item_direct_ref or > Item_aggregate_ref as they will be computed directly from the top-level > item once the window function argument columns are computed. OK to push but add comment before IF with return explaining it. > > --- > mysql-test/r/win.result | 12 ++++++++++++ > mysql-test/t/win.test | 10 ++++++++++ > sql/item.cc | 6 +++--- > 3 files changed, 25 insertions(+), 3 deletions(-) > > diff --git a/mysql-test/r/win.result b/mysql-test/r/win.result > index ec83998aa98..a67c336e5b4 100644 > --- a/mysql-test/r/win.result > +++ b/mysql-test/r/win.result > @@ -3085,3 +3085,15 @@ select max(id), rank() over (order by max(id)) from t1 where id < 3; > max(id) rank() over (order by max(id)) > 2 1 > drop table t1; > +# > +# main.win failure post MDEV-12336 > +# > +create table t(a decimal(35,10), b int); > +insert into t values (1, 10), (2, 20), (3, 30); > +prepare stmt from "SELECT (CASE WHEN sum(t.a) over (partition by t.b)=1 THEN 1000 ELSE 300 END) AS a FROM t"; > +execute stmt; > +a > +1000 > +300 > +300 > +drop table t; > diff --git a/mysql-test/t/win.test b/mysql-test/t/win.test > index 95d32c5bd14..0ecb2b35074 100644 > --- a/mysql-test/t/win.test > +++ b/mysql-test/t/win.test > @@ -1877,3 +1877,13 @@ select count(max(id)) over (order by max(id)) from t1 where id < 3; > select max(id), rank() over (order by max(id)) from t1 where id < 3; > > drop table t1; > + > +--echo # > +--echo # main.win failure post MDEV-12336 > +--echo # > +create table t(a decimal(35,10), b int); > +insert into t values (1, 10), (2, 20), (3, 30); > + > +prepare stmt from "SELECT (CASE WHEN sum(t.a) over (partition by t.b)=1 THEN 1000 ELSE 300 END) AS a FROM t"; > +execute stmt; > +drop table t; > diff --git a/sql/item.cc b/sql/item.cc > index df615b5ace9..05e5fd1ce45 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -1920,6 +1920,9 @@ void Item::split_sum_func2(THD *thd, Ref_ptr_array ref_pointer_array, > point to the temporary table. > */ > split_sum_func(thd, ref_pointer_array, fields, split_flags); > + if (type() == FUNC_ITEM) { > + return; > + } > } > else > { > @@ -1979,9 +1982,6 @@ void Item::split_sum_func2(THD *thd, Ref_ptr_array ref_pointer_array, > &ref_pointer_array[el], 0, name)))) > return; // fatal_error is set > } > - else if (type() == FUNC_ITEM && > - ((Item_func *) this)->with_window_func) > - return; > else > { > if (!(item_ref= (new (thd->mem_root) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Project Progress Report[PHASE-1]
by ANKIT KUMAR 30 Jun '17

30 Jun '17

Hi, I have newly started contributing to opensource projects, so these 3.5 weeks have been really awesome from learning point of view. Everyday I learnt something new, kept experimenting various approaches to a particular problem. I have come a long way as I am about to finish 2 subtasks among 4(4th being optional as it already does what it's expected to). Whole MariaDB community has been really helpful, and special thanks to my Mentor without whom support it wouldn't have been possible. Just recollecting everything I did in this phase- 1. Learnt how bison works, how grammar is written, how actions can be written(w.r.t to extended syntax), how parsing is done, 2. Implemented Extended CREATE USER and GRANT USER syntax. 3. Created tests for the acceptance of newly implemented grammar. 4. Designed in-memory structures that will store value of plugins and perform AND/OR operations as per precedence and authentication will be provided accordingly. 5. Wrote actions for this complex syntax(fixed some of the errors, some are yet to be fixed). In the remaining 3-4 days of Phase-1, I will try to complete this bridge between grammar and in-memory structure after which storing data in memory will be done. Now coming on to phase-2 deliverables, I am planning to finish off with Storage on Disk. We have found a simplest solution to get this done(probably in a week or 2), and in the remaining days, we will check the working of this new feature. I will then migrate this code to 10.3 branch. (Phase-3 for unexpected delay) This project involve lots of thinking than actual coding, so I think I have maintained good pace throughout. I believe I will maintain this enthusiasm and confidence level in upcoming phases too. Hopefully, I will get a chance to continue with this project. Kind regards, Ankit Kumar

2 2

[Maria-developers] Fwd: ecbad0a07a8: MDEV-13189: Window functions crash when using INTERVAL
by Vicențiu Ciorbaru 28 Jun '17

28 Jun '17

Hi Sergey, Can you please review this change that makes window functions work with INTERVAL function? Thanks, Vicentiu ---------- Forwarded message --------- From: vicentiu <vicentiu(a)mariadb.org> Date: Wed, 28 Jun 2017 at 21:08 Subject: ecbad0a07a8: MDEV-13189: Window functions crash when using INTERVAL To: <commits(a)mariadb.org> revision-id: ecbad0a07a848c1f9f474c4c749c87b6bfe7986c (mariadb-10.2.6-74-gecbad0a07a8) parent(s): 31ba0fa48d27715e82258b1e74401093e0ee17a2 author: Vicențiu Ciorbaru committer: Vicențiu Ciorbaru timestamp: 2017-06-28 21:07:55 +0300 message: MDEV-13189: Window functions crash when using INTERVAL Interval function makes use of Item_row. Item_row did not correctly mark with_window_func flag according to its arguments. Fix it by making Item_row aware of this flag. --- mysql-test/r/win.result | 14 ++++++++++++++ mysql-test/t/win.test | 9 +++++++++ sql/item_row.cc | 1 + 3 files changed, 24 insertions(+) diff --git a/mysql-test/r/win.result b/mysql-test/r/win.result index 952bd82ac7f..7f8777ed491 100644 --- a/mysql-test/r/win.result +++ b/mysql-test/r/win.result @@ -3127,3 +3127,17 @@ NULL 1 0 1 0 1 2 0 1 drop table t1; +# +# MDEV-13189: Window functions crash when using INTERVAL function +# +create table t1(i int); +insert into t1 values (1),(2),(10),(20),(30); +select sum(i) over (order by i), interval(sum(i) over (order by i), 10, 20) +from t1; +sum(i) over (order by i) interval(sum(i) over (order by i), 10, 20) +1 0 +3 0 +13 1 +33 2 +63 2 +drop table t1; diff --git a/mysql-test/t/win.test b/mysql-test/t/win.test index 45653b18682..aff717e3151 100644 --- a/mysql-test/t/win.test +++ b/mysql-test/t/win.test @@ -1914,3 +1914,12 @@ select max(i) over (order by i), max(i) over (order by i) is not null from t1; drop table t1; + +--echo # +--echo # MDEV-13189: Window functions crash when using INTERVAL function +--echo # +create table t1(i int); +insert into t1 values (1),(2),(10),(20),(30); +select sum(i) over (order by i), interval(sum(i) over (order by i), 10, 20) +from t1; +drop table t1; diff --git a/sql/item_row.cc b/sql/item_row.cc index ddbb0736d54..fc484f560ee 100644 --- a/sql/item_row.cc +++ b/sql/item_row.cc @@ -63,6 +63,7 @@ bool Item_row::fix_fields(THD *thd, Item **ref) } maybe_null|= item->maybe_null; with_sum_func= with_sum_func || item->with_sum_func; + with_window_func = with_window_func || item->with_window_func; with_field= with_field || item->with_field; with_subselect|= item->with_subselect; }

1 0

[Maria-developers] MDEV-13197 Parser refactoring for CREATE VIEW, TRIGGER, SP, UDF, EVENT
by Alexander Barkov 28 Jun '17

28 Jun '17

Hello Sanja, Can you please review a patch for MDEV-13197? It's a dependency change for MDEV-10591 Oracle-style packages, which I'm currently working on. The patch is for bb-10.2-ext. Thanks.

1 0

[Maria-developers] MDEV-12179: Per-engine mysql.gtid_slave_pos: auto-configuring/packaging
by Sergey Petrunia 26 Jun '17

26 Jun '17

Hello, Suppose there is a transactional storage engine that is shipped as a loadable module. The examples are MyRocks and TokuDB. I think it the default behavior for such engine after MDEV-12179 should be that the engine is listed in @@gtid_pos_auto_engines. That is, as soon as a transaction that uses the storage engine is replicated, mysql.slave_gtid_pos{_$engine} is created for it. gtid_pos_auto_engines makes it easy to do this manually, but it's difficult to achieve as default behavior. Here's an explanation why. Loadable storage engines are put into their separate packages, for example mariadb-plugin-rocksdb_10.2.7+maria-zesty_amd64.deb. The two important files in the package are : ./etc/mysql/mariadb.conf.d/rocksdb.cnf ./usr/lib/mysql/plugin/ha_rocksdb.so The first one is a .cnf that gets included into the server's cnf file. It would be nice to put there something like SET global gtid_pos_auto_engines= CONCAT(@@gtid_pos_auto_engines, 'rocksdb'); but that is not possible (it's a .cnf not .sql) Possible solutions: == [Ab]use the Storage Engine API == Make the storage engine put itself in @@gtid_pos_auto_engines. I think this is technically possible, but - It's a storage engine API abuse - how does one explain @@gtid_pos_auto_engines to the users? == Let gtid_pos_auto_engines include unknown engine == - Allow @@gtid_pos_auto_engines to list unknown engines - set the default to include all known transactional storage engines, that is it should be : gtid_pos_auto_engines='InnoDB,TokuDB,RocksDB' // (*) and the server should allow this even if it is built without TokuDB or RocksDB (*) -- Aria is not needed as it is not a "true" transactional storage engine as far as interplay with the binlog is concerned. == Let packaging use some other solution == I suppose it's possible for packaging system to run some script after a package has been installed. that script might find/adjust my.cnf file to have the desired @@gtid_pos_auto_engines setting. This solution might be complex and fragile, though. Any thoughts ? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] exact commands to produce similar mariadb release tarball
by Phil Sweeney 26 Jun '17

26 Jun '17

Hi there, I'm trying to do some custom builds of MariaDB for myself, but want to emulate the release builds as much as possible. I'm doing this on Ubuntu 16.04 LTS and following steps here: https://mariadb.com/kb/en/mariadb/creating-the-mariadb-binary-tarball/ However, my tarball ends up being 227MB compared to the downloads that are ~500MB, plus it won't start because it's missing jemalloc. Where can I find info about the environment and the commandline used to build the official releases e.g. mariadb-10.1.24-linux-glibc_214-x86_64.tar.gz (535.6MB) Thanks! Phil

2 2

Re: [Maria-developers] [Commits] ee73b33: MDEV-8840: ANALYZE FORMAT=JSON produces wrong data with BKA
by Sergey Petrunia 25 Jun '17

25 Jun '17

Hi Varun, On Sat, Jun 17, 2017 at 05:35:41PM +0530, Varun wrote: > revision-id: ee73b331d97d33ae7aa1403a4ac7312f2d261754 (mariadb-10.1.20-323-gee73b33) > parent(s): 056bab0880544d91ea67d18fe8db65b4f6625482 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-06-17 17:34:07 +0530 > message: > > MDEV-8840: ANALYZE FORMAT=JSON produces wrong data with BKA > > The issue was that r_loops, r_rows and r_filtered in ANALYZE FORMAT= JSON were not > calculated for the table on which we were performing the MRR scan in the BKA join > Fixed this by adding respective counter in the JOIN_CACHE_MRR::open and JOIN_CACHE::next > So I apply the patch and run this example (the dataset is from explain_json.test, also pasted below): MariaDB [j5]> analyze format=json select * from t3,t4 where t3.a=t4.a and (t4.c+1 < t3.b+1)\G *************************** 1. row *************************** ANALYZE: { "query_block": { "select_id": 1, "r_loops": 1, "r_total_time_ms": 0.8859, "table": { "table_name": "t3", "access_type": "ALL", "r_loops": 1, "rows": 10, "r_rows": 10, "r_total_time_ms": 0.0202, "filtered": 100, "r_filtered": 100, "attached_condition": "t3.a is not null" }, "block-nl-join": { "table": { "table_name": "t4", "access_type": "ref", "possible_keys": ["a"], "key": "a", "key_length": "5", "used_key_parts": ["a"], "ref": ["j5.t3.a"], "r_loops": 1, "rows": 1, "r_rows": 11, "r_total_time_ms": 0.0379, "filtered": 100, "r_filtered": 90.909 }, "buffer_type": "flat", "buffer_size": "256Kb", "join_type": "BKA", "mrr_type": "Rowid-ordered scan", "attached_condition": "t4.c + 1 < t3.b + 1", "r_filtered": 0 } } } I think, these value are wrong: > "r_rows": 11, > "r_filtered": 90.909 r_rows is actually 10, r_filtered=100 (no attached condition) I guess wrong r_filtered is caused by r_rows being wrong. ## Dataset: set storage_engine=myisam; set default_storage_engine=myisam; create table t1(a int); insert into t1 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table t2(a int); insert into t2 select A.a + B.a* 10 + C.a * 100 from t1 A, t1 B, t1 C; create table t3(a int, b int); insert into t3 select a,a from t1; create table t4(a int, b int, c int, filler char(100), key (a,b)); insert into t4 select a,a,a, 'filler-data' from t2; set optimizer_switch='mrr=on'; set join_cache_level=6; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 62021f3: MDEV-12070 - Introduce thd_query_safe() from MySQL 5.7
by Sergei Golubchik 23 Jun '17

23 Jun '17

Hi, Sergey! On Jun 22, Sergey Vojtovich wrote: > revision-id: 62021f391a42c5577190aa43cb8ad91e56235b46 (mariadb-10.2.6-57-g62021f3) > parent(s): 557e1bd472612848a42e772c1fb6f8ed32ab33b4 > committer: Sergey Vojtovich > timestamp: 2017-06-22 17:15:10 +0400 > message: > > MDEV-12070 - Introduce thd_query_safe() from MySQL 5.7 > > Merged relevant part of MySQL revision: > https://github.com/mysql/mysql-server/commit/565d20b44f24fcc855dc616164d87b… > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index f8cf829..db65eb3 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -4529,6 +4529,32 @@ extern "C" LEX_STRING * thd_query_string (MYSQL_THD thd) > return(&thd->query_string.string); > } > > + > +/** > + Get the current query string for the thread. > + > + @param thd The MySQL internal thread pointer > + @param buf Buffer where the query string will be copied > + @param buflen Length of the buffer > + > + @return Length of the query > + > + @note This function is thread safe as the query string is > + accessed under mutex protection and the string is copied > + into the provided buffer. @see thd_query_string(). > +*/ > + > +extern "C" size_t thd_query_safe(MYSQL_THD thd, char *buf, size_t buflen) > +{ > + mysql_mutex_lock(&thd->LOCK_thd_data); > + size_t len= MY_MIN(buflen - 1, thd->query_length()); > + memcpy(buf, thd->query(), len); > + mysql_mutex_unlock(&thd->LOCK_thd_data); > + buf[len]= '\0'; > + return len; > +} Okay. But as neither thd_query_string() nor thd_query() are part of the plugin API, I'd suggest to remove them, they're inherently unsafe and should not be used. Few other engines use thd_query_string() need to be fixed too (but they bypass the plugin API and we don't promise stability for internal functions). Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] [Commits] 922f970: MDEV-12882 - Assertion failed in MDL_context::upgrade_shared_lock
by Sergei Golubchik 23 Jun '17

23 Jun '17

Hi, Sergey! On Jun 22, Sergey Vojtovich wrote: > revision-id: 922f9700a7fb084e3f7d6cff56e27e24fc603ed7 (mariadb-10.2.6-58-g922f970) > parent(s): 62021f391a42c5577190aa43cb8ad91e56235b46 > committer: Sergey Vojtovich > timestamp: 2017-06-22 17:35:36 +0400 > message: > > MDEV-12882 - Assertion failed in MDL_context::upgrade_shared_lock > > Relaxed assertion (in MySQL it was removed). > For "LOCK TABLES t1 WRITE CONCURRENT, t1 READ" upgrade lock to weakest > existing suitable lock, which is MDL_SHARED_NO_READ_WRITE. > > diff --git a/mysql-test/t/mdl.test b/mysql-test/t/mdl.test > new file mode 100644 > index 0000000..5a74ae5 > --- /dev/null > +++ b/mysql-test/t/mdl.test > @@ -0,0 +1,15 @@ Hm. I'd totally expect main.mdl test to exist for years :) > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 412cd1d..7c910e8 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2787,6 +2787,7 @@ static bool lock_tables_open_and_lock_tables(THD *thd, TABLE_LIST *tables) > ! table->prelocking_placeholder && > table->table->file->lock_count() == 0) > { > + enum enum_mdl_type lock_type; > /* > In case when LOCK TABLE ... READ LOCAL was issued for table with > storage engine which doesn't support READ LOCAL option and doesn't > @@ -2799,9 +2800,12 @@ static bool lock_tables_open_and_lock_tables(THD *thd, TABLE_LIST *tables) > deadlock_handler.init(); > thd->push_internal_handler(&deadlock_handler); > > + lock_type= table->table->mdl_ticket->get_type() == MDL_SHARED_WRITE ? > + MDL_SHARED_NO_READ_WRITE : MDL_SHARED_READ_ONLY; > + > bool result= thd->mdl_context.upgrade_shared_lock( > table->table->mdl_ticket, > - MDL_SHARED_READ_ONLY, > + lock_type, > thd->variables.lock_wait_timeout); This only works if you lock WRITE, then READ. If you do it the other way around, you'll get an error. Isn't it inconsistent? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] Condition pushdown into derived tables and outer joins.
by Sergey Petrunia 23 Jun '17

23 Jun '17

Hello Igor, While reviewing MDEV-12845, I've found this piece in sql_select.cc: while ((tbl= li++)) { /* Do not push conditions from where into materialized inner tables of outer joins: this is not valid. */ if (tbl->is_materialized_derived()) { /* Do not push conditions from where into materialized inner tables of outer joins: this is not valid. */ if (!tbl->is_inner_table_of_outer_join()) Q1: Is int meaningful to have two identical comments? Q2: It is not valid push parts of WHERE, but it is valid to push the parts of ON expression. An example: explain format=json select * from t21 LEFT JOIN (select a,b, COUNT(*) as CNT from t22 group by a,b) TBL ON (t21.a=TBL.a AND TBL.a<5); here, "TBL.a<5" can be pushed down into the subquery, but it is not. EXPLAIN EXTENDED confirms it: "query_block": { "select_id": 1, "const_condition": "1", "table": { "table_name": "t21", "access_type": "ALL", "rows": 10, "filtered": 100 }, "table": { "table_name": "<derived2>", "access_type": "ref", "possible_keys": ["key0"], "key": "key0", "key_length": "5", "used_key_parts": ["a"], "ref": ["test.t21.a"], "rows": 2, "filtered": 100, "attached_condition": "trigcond(trigcond(t21.a < 5 and t21.a is not null))", "materialized": { "query_block": { "select_id": 2, "filesort": { "sort_key": "t22.a, t22.b", "temporary_table": { "table": { "table_name": "t22", "access_type": "ALL", "rows": 10, "filtered": 100 } } } } } } Just in case, the commands to create the dataset were: create table ten(a int); insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table t21 (a int, b int, c int); insert into t21 select a,a,a from ten; create table t22 (a int, b int, c int); insert into t22 select a,a,a from ten; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Project Progress Report [Week 3]
by ANKIT KUMAR 20 Jun '17

20 Jun '17

Hi, In this week, we worked on in-memory structures:- Which representation to choose(prefix or expression tree), How to store values and how authentication will be done. I have prepared pseudocode to implement this subtask and have made implemented some part of code. Hopefully by 25th May, I will finish off with >50% of the project. I will declare Phase-2 deliverables next Monday(26 May). Kind regards, Ankit Kumar

2 1

Re: [Maria-developers] MDEV-12620 - set lock_wait_timeout = 1; flush tables with read lock;
by Sergei Golubchik 16 Jun '17

16 Jun '17

Hi, Sergey! On Jun 16, Sergey Vojtovich wrote: > revision-id: 056bab0880544d91ea67d18fe8db65b4f6625482 (mariadb-10.1.24-22-g056bab0) > parent(s): 58f87a41bd8de7370cc05c41977fadc685826c9e > committer: Sergey Vojtovich > timestamp: 2017-06-16 15:47:46 +0400 > message: > > MDEV-12620 - set lock_wait_timeout = 1;flush tables with read lock; > lock not released after timeout > > Release GRL if FLUSH TABLES phase failed. > > diff --git a/mysql-test/t/mdl_sync.test b/mysql-test/t/mdl_sync.test > index 8e80978..c479c2a 100644 > --- a/mysql-test/t/mdl_sync.test > +++ b/mysql-test/t/mdl_sync.test > @@ -4806,6 +4806,30 @@ disconnect con2; > disconnect con3; > > +--echo # > +--echo # MDEV-12620 - set lock_wait_timeout = 1;flush tables with read lock; > +--echo # lock not released after timeout > +--echo # > +CREATE TABLE t1(a INT) ENGINE=InnoDB; > +SET debug_sync='open_tables_after_open_and_process_table SIGNAL ready WAIT_FOR go'; > +send SELECT * FROM t1; > + > +connect (con1,localhost,root,,); > +SET debug_sync='now WAIT_FOR ready'; > +# lock_wait_timeout should be 0 in 10.3, so that we don't have to wait at all what's special in 10.3? > +SET lock_wait_timeout=1; > +--error ER_LOCK_WAIT_TIMEOUT > +FLUSH TABLES WITH READ LOCK; > +SET debug_sync='now SIGNAL go'; > + > +connection default; > +reap; > +SET debug_sync='RESET'; > +DROP TABLE t1; What should happen without a bug fix? DROP TABLE hangs? > + > +disconnect con1; > + > # Check that all connections opened by test cases in this file are really > # gone so execution of other tests won't be affected by their presence. > --source include/wait_until_count_sessions.inc > diff --git a/sql/sql_reload.cc b/sql/sql_reload.cc > index 376dfeb..d68ce96 100644 > --- a/sql/sql_reload.cc > +++ b/sql/sql_reload.cc > @@ -253,7 +253,8 @@ bool reload_acl_and_cache(THD *thd, unsigned long long options, > NOTE: my_error() has been already called by reopen_tables() within > close_cached_tables(). > */ > - result= 1; > + thd->global_read_lock.unlock_global_read_lock(thd); > + return 1; > } Sure. Looks correct. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] MDEV-10306 Wrong results with combination of CONCAT, SUBSTR and CONVERT in subquery
by Alexander Barkov 16 Jun '17

16 Jun '17

Hello Sergei, Please review a fix for MDEV-10306. Thanks!

4 13

Re: [Maria-developers] [Commits] 01b2bb79bb78e42108e32af02f1418a660ca861d Fixed the bug mdev-13064.
by Oleksandr Byelkin 16 Jun '17

16 Jun '17

Hi, Igor! Am 15.06.2017 um 23:43 schrieb Igor Babaev: > commit 01b2bb79bb78e42108e32af02f1418a660ca861d > Author: Igor Babaev <igor(a)askmonty.org> > Commit: Igor Babaev <igor(a)askmonty.org> > > Fixed the bug mdev-13064. > > This is another attempt to fix the bug mdev-12992. > This patch introduces st_select_lex::context_analysis_place for > the place in SELECT where context analysis is currently performed. > It's similar to st_select_lex::parsing_place, but it is used at It is OK to push. > the preparation stage. > --- > mysql-test/r/func_group.result | 9 +++++++++ > mysql-test/t/func_group.test | 13 +++++++++++++ > sql/item.cc | 5 ++--- > sql/item_windowfunc.cc | 2 +- > sql/sql_lex.h | 1 + > sql/sql_select.cc | 33 ++++++++++++++++++--------------- > 6 files changed, 44 insertions(+), 19 deletions(-) > > diff --git a/mysql-test/r/func_group.result b/mysql-test/r/func_group.result > index cb97ea2..c4e991e 100644 > --- a/mysql-test/r/func_group.result > +++ b/mysql-test/r/func_group.result > @@ -2447,3 +2447,12 @@ DROP TABLE t1; > # > # End of 10.1 tests > # > +# > +# MDEV-13064: assertion `n < m_size' fails in Item::split_sum_func2() > +# > +create table t1 (i int) engine=MyISAM; > +insert into t1 value (1),(2); > +select count(*)+sleep(0) from t1; > +count(*)+sleep(0) > +2 > +drop table t1; > diff --git a/mysql-test/t/func_group.test b/mysql-test/t/func_group.test > index 1e75099..8bbc9e6 100644 > --- a/mysql-test/t/func_group.test > +++ b/mysql-test/t/func_group.test > @@ -1691,3 +1691,16 @@ DROP TABLE t1; > --echo # > --echo # End of 10.1 tests > --echo # > + > +--echo # > +--echo # MDEV-13064: assertion `n < m_size' fails in Item::split_sum_func2() > +--echo # > + > +create table t1 (i int) engine=MyISAM; > +insert into t1 value (1),(2); > + > +select count(*)+sleep(0) from t1; > + > +drop table t1; > + > + > diff --git a/sql/item.cc b/sql/item.cc > index 8db5cfc..df615b5 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -490,8 +490,7 @@ int Item::save_str_value_in_field(Field *field, String *result) > command => we should check thd->lex->current_select on zero (thd->lex > can be uninitialised) > */ > - if (thd->lex->current_select && > - thd->stmt_arena->is_stmt_prepare_or_first_sp_execute()) > + if (thd->lex->current_select) > { > enum_parsing_place place= > thd->lex->current_select->parsing_place; > @@ -5513,7 +5512,7 @@ bool Item_field::fix_fields(THD *thd, Item **reference) > > SELECT_LEX *select= thd->lex->current_select; > thd->change_item_tree(reference, > - select->parsing_place == IN_GROUP_BY && > + select->context_analysis_place == IN_GROUP_BY && > alias_name_used ? *rf->ref : rf); > > /* > diff --git a/sql/item_windowfunc.cc b/sql/item_windowfunc.cc > index 59a22c6..27b0072 100644 > --- a/sql/item_windowfunc.cc > +++ b/sql/item_windowfunc.cc > @@ -71,7 +71,7 @@ > { > DBUG_ASSERT(fixed == 0); > > - enum_parsing_place place= thd->lex->current_select->parsing_place; > + enum_parsing_place place= thd->lex->current_select->context_analysis_place; > > if (!(place == SELECT_LIST || place == IN_ORDER_BY)) > { > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index face3bc..0c55ffc 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -855,6 +855,7 @@ class st_select_lex: public st_select_lex_node > /* reserved for exists 2 in */ > uint select_n_reserved; > enum_parsing_place parsing_place; /* where we are parsing expression */ > + enum_parsing_place context_analysis_place; /* where we are in prepare */ > bool with_sum_func; /* sum function indicator */ > > ulong table_join_options; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 4c64cf2..6c56f01 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -648,17 +648,15 @@ void remove_redundant_subquery_clauses(st_select_lex *subq_select_lex) > > thd->lex->allow_sum_func|= (nesting_map)1 << select->nest_level; > > - save_place= thd->lex->current_select->parsing_place; > - thd->lex->current_select->parsing_place= IN_ORDER_BY; > + save_place= thd->lex->current_select->context_analysis_place; > + thd->lex->current_select->context_analysis_place= IN_ORDER_BY; > res= res || setup_order(thd, ref_pointer_array, tables, fields, all_fields, > order); > - thd->lex->current_select->parsing_place= save_place; > - thd->lex->allow_sum_func&= ~((nesting_map)1 << select->nest_level); > - save_place= thd->lex->current_select->parsing_place; > - thd->lex->current_select->parsing_place= IN_GROUP_BY; > + thd->lex->allow_sum_func&= ~((nesting_map)1 << select->nest_level); > + thd->lex->current_select->context_analysis_place= IN_GROUP_BY; > res= res || setup_group(thd, ref_pointer_array, tables, fields, all_fields, > group, hidden_group_fields); > - thd->lex->current_select->parsing_place= save_place; > + thd->lex->current_select->context_analysis_place= save_place; > thd->lex->allow_sum_func|= (nesting_map)1 << select->nest_level; > res= res || setup_windows(thd, ref_pointer_array, tables, fields, all_fields, > win_specs, win_funcs); > @@ -712,6 +710,7 @@ void remove_redundant_subquery_clauses(st_select_lex *subq_select_lex) > if (select_lex->handle_derived(thd->lex, DT_PREPARE)) > DBUG_RETURN(1); > > + thd->lex->current_select->context_analysis_place= NO_MATTER; > thd->lex->current_select->is_item_list_lookup= 1; > /* > If we have already executed SELECT, then it have not sense to prevent > @@ -801,12 +800,13 @@ void remove_redundant_subquery_clauses(st_select_lex *subq_select_lex) > > ref_ptrs= ref_ptr_array_slice(0); > > - enum_parsing_place save_place= thd->lex->current_select->parsing_place; > - thd->lex->current_select->parsing_place= SELECT_LIST; > + enum_parsing_place save_place= > + thd->lex->current_select->context_analysis_place; > + thd->lex->current_select->context_analysis_place= SELECT_LIST; > if (setup_fields(thd, ref_ptrs, fields_list, MARK_COLUMNS_READ, > &all_fields, 1)) > DBUG_RETURN(-1); > - thd->lex->current_select->parsing_place= save_place; > + thd->lex->current_select->context_analysis_place= save_place; > > if (setup_without_group(thd, ref_ptrs, tables_list, > select_lex->leaf_tables, fields_list, > @@ -22235,14 +22235,16 @@ int setup_order(THD *thd, Ref_ptr_array ref_pointer_array, TABLE_LIST *tables, > List<Item> &fields, List<Item> &all_fields, ORDER *order, > bool from_window_spec) > { > - enum_parsing_place parsing_place= thd->lex->current_select->parsing_place; > + enum_parsing_place context_analysis_place= > + thd->lex->current_select->context_analysis_place; > thd->where="order clause"; > for (; order; order=order->next) > { > if (find_order_in_list(thd, ref_pointer_array, tables, order, fields, > all_fields, FALSE, from_window_spec)) > return 1; > - if ((*order->item)->with_window_func && parsing_place != IN_ORDER_BY) > + if ((*order->item)->with_window_func && > + context_analysis_place != IN_ORDER_BY) > { > my_error(ER_WINDOW_FUNCTION_IN_WINDOW_SPEC, MYF(0)); > return 1; > @@ -22284,7 +22286,8 @@ int setup_order(THD *thd, Ref_ptr_array ref_pointer_array, TABLE_LIST *tables, > List<Item> &fields, List<Item> &all_fields, ORDER *order, > bool *hidden_group_fields, bool from_window_spec) > { > - enum_parsing_place parsing_place= thd->lex->current_select->parsing_place; > + enum_parsing_place context_analysis_place= > + thd->lex->current_select->context_analysis_place; > *hidden_group_fields=0; > ORDER *ord; > > @@ -22300,14 +22303,14 @@ int setup_order(THD *thd, Ref_ptr_array ref_pointer_array, TABLE_LIST *tables, > all_fields, TRUE, from_window_spec)) > return 1; > (*ord->item)->marker= UNDEF_POS; /* Mark found */ > - if ((*ord->item)->with_sum_func && parsing_place == IN_GROUP_BY) > + if ((*ord->item)->with_sum_func && context_analysis_place == IN_GROUP_BY) > { > my_error(ER_WRONG_GROUP_FIELD, MYF(0), (*ord->item)->full_name()); > return 1; > } > if ((*ord->item)->with_window_func) > { > - if (parsing_place == IN_GROUP_BY) > + if (context_analysis_place == IN_GROUP_BY) > my_error(ER_WRONG_PLACEMENT_OF_WINDOW_FUNCTION, MYF(0)); > else > my_error(ER_WINDOW_FUNCTION_IN_WINDOW_SPEC, MYF(0)); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Review for MDEV-12666: CURRENT_ROLE() and DATABASE() does not work in a view
by Alexander Barkov 13 Jun '17

13 Jun '17

Hi Vicentiu, The patch looks fine. Thanks! I have only small comments below: > > [Commits] 974064cde3c: MDEV-12666: CURRENT_ROLE() and DATABASE() does not work in a view > vicentiu vicentiu at mariadb.org > Thu Jun 8 10:11:38 EEST 2017 > > Previous message (by thread): [Commits] 7f2d3c3: MDEV-11084 Select statement with partition selection against MyISAM table opens all partitions. > Next message (by thread): [Commits] d03abc71a49: MDEV-12609: Allow suppression of InnoDB log messages about reserving extents > Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] > > revision-id: 974064cde3c1cd1be9157bc11ce410604807f68b (mariadb-10.0.31-2-g974064cde3c) > parent(s): 54cd5bc703da08be45894495d7f3e2c8117617cd > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2017-06-08 10:10:07 +0300 > message: > > MDEV-12666: CURRENT_ROLE() and DATABASE() does not work in a view > > The problem lies in how CURRENT_ROLE is defined. The > Item_func_current_role inherits from Item_func_sysconst, which defines > a safe_charset_converter to be a const_charset_converter. > > During view creation, if there is no role previously set, the current_role() > function returns NULL. > > This is captured on item instantiation and the > const_charset_converter call subsequently returns an Item_null. > In turn, the function is replaced with Item_null and the view is > then created with an Item_null instead of Item_func_current_role. > > Without this patch, the first SHOW CREATE VIEW from the testcase would > have a where clause of WHERE role_name = NULL, while the second SHOW > CREATE VIEW would show a correctly created view. > > The same applies for the DATABASE function, as it can change as well. > > There is an additional problem with CURRENT_ROLE() when used in a > prepared statement. During prepared statement creation we used to set > the string_value of the function to the current role as well as the > null_value flag. During execution, if CURRENT_ROLE was not null, the > null_value flag was never set to not-null during fix_fields. > > Item_func_current_user however can never be NULL so it did not show this > problem in a view before. At the same time, the CURRENT_USER() can not > be changed between prepared statement execution and creation so the > implementation where the value is stored during fix_fields is > sufficient. > > Note also that DATABASE() function behaves differently during prepared > statements. See bug 25843 for details or commit > 7e0ad09edff587dadc3e9855fc81e1b7de8f2199 > <cut> > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -2344,6 +2344,7 @@ String *Item_func_database::val_str(String *str) > } > else > str->copy(thd->db, thd->db_length, system_charset_info); > + null_value= 0; > return str; > } > > @@ -2378,6 +2379,30 @@ bool Item_func_user::init(const char *user, const char *host) > } > > > +Item *Item_func_sysconst::safe_charset_converter(CHARSET_INFO *tocs) > +{ > + /* > + During view or prepared statement creation, the item should not > + make use of const_charset_converter as it would imply substitution > + with constant items which is not correct. Functions can have different > + values during view creation and view execution based on context. > + > + Return the identical item during view creation and prepare. > + */ > + if (current_thd->lex->context_analysis_only & > + (CONTEXT_ANALYSIS_ONLY_PREPARE | CONTEXT_ANALYSIS_ONLY_VIEW)) > + return this; > + return const_charset_converter(tocs, true, fully_qualified_func_name()); I propose to avoid code duplication and do like this: if (!Item_func_sysconst::const_item()) return this; return return const_charset_converter(tocs, true, fully_qualified_func_name()); > +} > + > +bool Item_func_sysconst::const_item() const > +{ > + if (current_thd->lex->context_analysis_only & > + (CONTEXT_ANALYSIS_ONLY_PREPARE | CONTEXT_ANALYSIS_ONLY_VIEW)) > + return false; > + return true; > +} > + > bool Item_func_user::fix_fields(THD *thd, Item **ref) > { > return (Item_func_sysconst::fix_fields(thd, ref) || > @@ -2403,20 +2428,25 @@ bool Item_func_current_role::fix_fields(THD *thd, Item **ref) > > Security_context *ctx= context->security_ctx > ? context->security_ctx : thd->security_ctx; > - > if (ctx->priv_role[0]) > { > if (str_value.copy(ctx->priv_role, strlen(ctx->priv_role), > system_charset_info)) > return 1; > - > str_value.mark_as_const(); > + null_value= maybe_null= 0; > return 0; > } > null_value= maybe_null= 1; > return 0; > } > > +String* Item_func_current_role::val_str(String * str) > +{ > + DBUG_ASSERT(fixed == 1); > + return null_value ? NULL : &str_value; > +} > + Why did you move val_str() from *.h to *.cc ? The method is just two lines, I'd leave it in *.h <cut>

1 0

[Maria-developers] MDEV-12962 Testing MDEV-10142
by Alexander Barkov 13 Jun '17

13 Jun '17

Hi Andrii, Thanks for looking into this. Please find my comments inline: > > > Below are mostly minor notes for which clarification is needed. > > Conceptual: are we sure that ORACLE mode should invalidate native MariaDB constructs? We need some consistency and in my understanding following should still work in Oracle mode: > ELSE / ELSEIF > % operation (if possible) I'm not sure. Monty, what do you think? > > > 2. MDEV-10343 sql_mode=ORACLE: Providing compatibility for basic SQL data types > > following are valid build-in datatypes declaration in oracle which cause syntax error at the moment (needs to be documented or created another task): > > these probably had to be implemented inside MDEV-10343 > > create or replace table t1(a binary_float); > create or replace table t1(a binary_double); > create or replace table t1(a long raw); > create or replace table t1(a NCLOB); > these need more effort or maybe just must be documented or separate task is needed > create or replace table t1(a varchar(30 byte)); > create or replace table t1(a varchar(30 char)); > create or replace table t1(a timestamp with local time zone); > create or replace table t1(a timestamp with time zone); > create or replace table t1(a interval day); > create or replace table t1(a interval year); > create or replace table t1(a BFILE); > create or replace table t1(a rowid); > create or replace table t1(a urowid(5)); > this should give an error (oracle will return ORA-02017: integer value required ) > MariaDB [test]> create or replace table t1(a number(2.2)); > Query OK, 0 rows affected (0.24 sec) There is a task for TIMESTAMP WITH TIME ZONE: MDEV-10018 Timestamp with time zone For other data types, should we create one task for every data type? > > > 3. MDEV-10411 Providing compatibility for basic PL/SQL constructs > 3.6 Assignment operator > It doesn't sound intuitive that one may write "wait_timeout:=a", but "a:=wait_timeout" shows syntax error. > > begin declare a int; begin a:=wait_timeout; end; end;; One needs to use @ when referring a system variable in the right side: a:=@wait_timeout; It's the same in the default mode: SET wait_timeount=100; but: SET xxx=@wait_timeout; Not sure if we should do anything with that. > > > 3.7 Variable declarations > Oracle allows NOT NULL in variable declaration. Maybe we should allow that syntax as well? > > begin declare a NOT NULL int; begin a:=1; a:=NULL; end; end;; I have created a task for this: MDEV-13078 NOT NULL routine variables > > > 10. MDEV-10585 EXECUTE IMMEDIATE > In ORACLE the command supports 'RETURNING INTO' clause, we should probably document the difference? > > sql_stmt := 'UPDATE emp SET sal = 2000 WHERE empno = :1 > > RETURNING sal INTO :2'; > > EXECUTE IMMEDIATE sql_stmt USING emp_id RETURNING INTO salary; In Oracle, EXECUTE IMMEDIATE also has the "INTO" clause: sql_stmt := 'SELECT * FROM emp WHERE empno = :id'; EXECUTE IMMEDIATE sql_stmt INTO emp_rec USING emp_id; Should we create two new tasks, for RETURNING INTO, and for INTO? > > 12. MDEV-10588 TRUNCATE TABLE t1 DROP|REUSE STORAGE > Following clauses should be allowed in TRUNCATE command as well: > [ PRESERVE MATERIALIZED VIEW LOG | PURGE MATERIALIZED VIEW LOG ] Should we create a task to ignore this syntax? > > 18. MDEV-10577 sql_mode=ORACLE: %TYPE in variable declarations > It is not clear what is expected to happen if table's column definition is altered after stored program creation. Nothing. Data type resolution is done during execution.

1 0

[Maria-developers] Progress report of the Project[Week 2]
by ANKIT KUMAR 12 Jun '17

12 Jun '17

Hi, In this week, I managed to do few contributions and finally completed subtask-1. These are few things that I did this week- 1. Extended CREATE USER syntax- Now there will be multiple authentication mechanisms per user. 2. Created tests for the newly implemented syntax and some tests for previously existing functonalities. Need to give final touch up to the code and will commit tomorrow. Now I will proceed to subtask-2 Week-3[TODO's]- 1. Implement in-memory structures for this new syntax-[Providing it to ACL_USER] 2. Will be able to write data in memory(By the end of Phase-1 evaluation). Kind regards, Ankit Kumar.

1 0

[Maria-developers] libmariadb.so.3 compatibility
by Michal Schorm 08 Jun '17

08 Jun '17

Hello, I'm currently quite confused about the *libmariadb* library. In MariaDB 10.1.24 there are: " libmysqlclient_r.so -> libmysqlclient.so " " libmysqlclient.so -> libmysqlclient.so.18 " " libmysqlclient.so.18 -> libmysqlclient.so.18.0.0 " " libmysqlclient.so.18.0.0 " In MariaDB 10.2.6 there are: " libmariadb.so -> libmariadb.so.3 " In MariaDB-connector-c v2.3.3 there are: " libmysqlclient.so -> libmariadb.so " " libmysqlclient_r.so -> libmariadb.so " " libmariadb.so -> libmariadb.so.2 " " libmariadb.so.2 " -- 1) is "libmariadb.so.2" compatibile with "libmysqlclient.so.18.0.0"? Both API and ABI? 2) is "libmariadb.so.3" compatibile with "libmysqlclient.so.18.0.0"? Both API and ABI? 3) is "libmariadb.so.2" compatibile with "libmariadb.so.3"? Both API and ABI? Many packages in Fedora depends on "libmysqlclient.so". I would like to pack 10.2 to Fedora, but I can't before all those packages move to the libmariadb library. It would be considerably easier transfer for me, if all those libraries were compatibile, so I could provide symlinks (simmilar as in the connector C), and provide time for all those packages in Fedora to move safely. -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

2 2

Re: [Maria-developers] [Commits] 4d6b16f: Fixed the bug mdev-12838.
by Sergey Petrunia 07 Jun '17

07 Jun '17

Hello Igor, On Wed, May 31, 2017 at 10:45:07AM -0700, IgorBabaev wrote: > revision-id: 4d6b16f10f9991821c128c382d32eb85248b3926 (mariadb-5.5.56-14-g4d6b16f) > parent(s): b8405c853fa30002d164d5fe2b4f8ea8979c09b8 > author: Igor Babaev > committer: Igor Babaev > timestamp: 2017-05-31 10:45:07 -0700 > message: > > Fixed the bug mdev-12838. > ... > +bool JOIN_TAB::keyuse_is_valid_for_access_in_chosen_plan(JOIN *join, > + KEYUSE *keyuse) > +{ > + if (!access_from_tables_is_allowed(keyuse->used_tables, > + join->sjm_lookup_tables)) > + return false; > + if (join->sjm_scan_tables & table->map) > + return true; > + table_map keyuse_sjm_scan_tables= keyuse->used_tables & > + join->sjm_scan_tables; > + if (!keyuse_sjm_scan_tables) > + return true; > + uint sjm_tab_nr= 0; > + while (!(keyuse_sjm_scan_tables & 1 << sjm_tab_nr)) The above "1" is a 32-bit number, while it should have the same bit length as table_map (that is, 64). Please use "table_map(1)". BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

[Maria-developers] the Lock on supremum is neccessary ?
by jhx1008 07 Jun '17

07 Jun '17

hi everyone: I use sysbench to do some tests on MySQL 5.7.x, and then I got some problems with the lock added on supremum, I hope someone can give me some advice. my table like this: CREATE TABLE `sbtest1` ( `id` int(11) NOT NULL AUTO_INCREMENT, `k` int(11) NOT NULL DEFAULT '0', `c` char(120) NOT NULL DEFAULT '', `pad` char(60) NOT NULL DEFAULT '', PRIMARY KEY (`id`), ) ENGINE=InnoDB; secondary index is not neccessary, I do not create it!! I have two sessions: session1: begin; select * from sbtest1 where id=123456 for update session2: insert into sbtest1 values(123450, 0,'aaa','bbb'); the records in the table like below: (...),(123440, 264454, xxxxxx, xxxxx), (123456, 251028, xxxxx, xxxxxx),(....) in my opinion, the record with the id=123456 existed, so session1 add the X lock on this record, session2 add the insert intention lock between(123440, 123456), these two locks are not conflicted. session2 will not be blocked by session1. unfortunately, in my case, session2 is blocked by session1, what happens? when session2 is blocked, I do "select * from information_schema.INNODB_LOCKS" use another session, It shows : mysql> select * from information_schema.INNODB_LOCKS; +----------------+-------------+-----------+-----------+------------------+------------+------------+-----------+----------+------------------------+ | lock_id | lock_trx_id | lock_mode | lock_type | lock_table | lock_index | lock_space | lock_page | lock_rec | lock_data | +----------------+-------------+-----------+-----------+------------------+------------+------------+-----------+----------+------------------------+ | 2388:29:2772:1 | 2388 | X | RECORD | `mydb`.`sbtest1` | PRIMARY | 29 | 2772 | 1 | supremum pseudo-record | | 2387:29:2772:1 | 2387 | X | RECORD | `mydb`.`sbtest1` | PRIMARY | 29 | 2772 | 1 | supremum pseudo-record | +----------------+-------------+-----------+-----------+------------------+------------+------------+-----------+----------+------------------------+ both the two sessions want to add X lock on "supremum pseudo-record" in the same time. then I have debugged this instance(row0sel.cc::row_search_mvcc), I got that: when a record existed on the page for the first one, if you want to add the X Lock on it, the supremum of pre-page will be add X Lock(maybe next key lock, supremum can be add gap lock only) also. In my case, the records' primary key stored like this: (..., 123440, supremum), (infimum, 123456, ...)， the id=123456 is the first record of that page, when add X Lock, first, find the pre-record id=123440, and then add the X lock after this record(id=123440) until match the record witch id=123456, so the supremum is locked also...， because of this lock(supremum)， session2 is blocked. if id=123456 is not the first record of page, session2 will not be blocked. so my doubt is: in my case, the session1 lock the supremum is necessary? if it's necessary, the lock on supremum maybe resulting the difference even if two mysql instances have the same records, just as in a replication. the record located on the page not all the same even if two instances have the same table sturcture & data(if slave define the table compressed, or innodb page_size not set as the same as master, etc..). if lock on supremum is neccessary, two instances execute the same SQLs, master is OK, but slave conflicted? on master， the two transactions may be committed in one group, but on slave maybe conflicted, slave's parallell replication is OK? also in MySQL 5.7.x, the xa prepare recorded in binlog, if on the master, execute two xa transactions: xa start 'aaa'; delete from table_name where id=xxxx; xa end 'aaa';xa prepare 'aaa'; xa start 'bbb';insert into table_name values(xxxx); end 'bbb'; xa prepare 'bbb'; xa commit 'aaa'; xa commit 'bbb' on the master, these xa not conflicted, I can committed these transaction, but on the slave, maybe conflicted sometimes, the replication is broken. how to repeat: create a table with primary key(id auto_increment is better) insert some records in this table use 3 sessions to find the first record on one page(perhaps, there are many pages, any one is OK).how to? session1: begin; select * from table_name where id=xxx for update // id=xxx must be exist session2: select * from table_name where id=xxx for update session3: when session2 is blocked, execute: show engine innodb status\G; in the transaction section, you can find some details like this: ------- TRX HAS BEEN WAITING 2 SEC FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 29 page no 2772 n bits 144 index PRIMARY of table `mydb`.`sbtest1` trx id 2388 lock_mode X insert intention waiting Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0 0: len 8; hex 73757072656d756d; asc supremum;; from the information "space id 29 page no 2772", we can find the page of this record. then test for next id until the page no become to 2773, remember this id, this record is the first one of page 2773 delete some records before this id witch get in step 3 use 2 sessions to do what i described I repeat it by sysbench(version 1.0). 1. use sysbench to create a table with 500,000 records: sysbench --table-size=500000 --tables=1 --threads=64 --create_secondary=false --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=mydb oltp_read_write.lua prepare 2. delete the records between(200000, 200057), delete from sbtest1 where id>200000 and id < 200057 3. two sessions: session1: begin;select * from sbtest1 where id=200057 for update; session2: insert into sbtest1 values(200050, 0,'a','a') session2 will be blocked by session1. Because this problem is related to the localtion of the record in the page, repeat it like my operations, do not use the compressed table and make sure the innodb_page_size = 16384. in my case, the Lock on supremum is neccessary? 2017-06-05 boyce@netease

2 4

[Maria-developers] Progress report of the Project
by ANKIT KUMAR 05 Jun '17

05 Jun '17

Hi, I am Working on MDEV-11340 and by the end of the first week of Coding Period I have managed to learn How to use Bison. Currently thinking about How to implement subtask-1 of my project. Kind Regards,

1 0

[Maria-developers] header change in MariaDB 10.2.6 and mysqlclient python binding
by Kazuhiko Shiozaki 30 May '17

30 May '17

Hi ! First of all, congratulations to MariaDB 10.2 GA. mysqlclient (python binding : https://pypi.python.org/pypi/mysqlclient/1.3.10 ) cannot be built with MariaDB 10.2.6 like the following : _mysql.c: In function '_mysql_ConnectionObject_ping': _mysql.c:1911:41: error: 'MYSQL' has no member named 'reconnect' if ( reconnect != -1 ) self->connection.reconnect = reconnect; ^ error: Setup script exited with error: command 'gcc' failed with exit status 1 This is because the change in include/mysql/mysql.h. In MariaDB 10.2.5, it is same as mariadb-10.2.5/include/mysql.h in the source tree. In MariaDB 10.2.6, it is same as mariadb-10.2.6/libmariadb/include/mysql.h in the source tree. Is this change intentional ? Should I contact mysqlclient developer for further fix of this issue ? Thanks in advance ! Kazuhiko

2 1

[Maria-developers] review for SP stack trace
by Alexander Barkov 30 May '17

30 May '17

Hello Jerome, here's Monty's review on your stack trace patch. Thank you very much. We're sorry for delay. > diff --git a/mysql-test/r/commit_1innodb.result b/mysql-test/r/commit_1innodb.result > index 1adba7b..8e40ec0 100644 > <cut> > > --- a/mysql-test/r/get_diagnostics.result > +++ b/mysql-test/r/get_diagnostics.result > @@ -595,7 +595,7 @@ SELECT var, @var; > END| > CALL p1(); > var @var > -1 1 > +2 2 > DROP PROCEDURE p1; > > What is the reason for this going from 1 to 2 ? > I would have understood if there would have been some warnings from before > in the code, but there wasn't. > > Does this come from the error from 'call p1()' earlier that does now have > a new 'note message'. > > If yes, it would be good to make this clear by adding a 'show > warnings' just before the above create procedure p1() line. > > # Setting TABLE_NAME is currently not implemented. > diff --git a/mysql-test/r/signal.result b/mysql-test/r/signal.result > index f05e357..a796cc6 100644 > --- a/mysql-test/r/signal_demo3.result > +++ b/mysql-test/r/signal_demo3.result > @@ -79,14 +79,23 @@ show warnings; > @@ -95,6 +104,4 @@ call proc_1(); > ERROR 45000: Oops in proc_1 > show warnings; > Level Code Message > -Error 1644 Oops in proc_5 > -Error 1644 Oops in proc_4 > -Error 1644 Oops in proc_3 > +Note 4068 At line 4 in demo.proc_3 > > Why has 3 error disappered and why is there a line number note for > demo.proc_3 but no error line? > > I assume this comes from the following line: > SET @@session.max_error_count = 5 > > The question is if max_error_count's should really be counted for notes too, > especially for "At line" notes. > The documentation for the variable says > "Specifies the maximum number of messages stored for display by" > > Still, the question is if we should include the 'At line' or not in counting. > (Not important for this patch, but good to think about) > > > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7484,3 +7484,5 @@ ER_UNKNOWN_SEQUENCES 42S02 > eng "Unknown SEQUENCE: '%-.300s'" > ER_UNKNOWN_VIEW 42S02 > eng "Unknown VIEW: '%-.300s'" > +ER_SP_STACK_TRACE > + eng "At line %d in %s" > > Should be %u as lineno is uint > > > Please add the "SHOW WARNINGS" and fix the error message and it's ok to push. > Thanks for a great addition to MariaDB!

2 2

[Maria-developers] Weekly Progress Report of the Project
by ANKIT KUMAR 29 May '17

29 May '17

Hi all, I am Ankit Kumar and I have been selected to work on allowing multiple authentication methods to same user. These are the few things I did this week- 1. Setted up Development environment on Windows. Resolved few issues which I was facing, some MariaDB members really helped me whenever I needed their help. 2. Read the Testing Framework Documentation of MariaDB in order to see how testing is performed. 3. I have been regularly communicating with the Mentor and other MariaDB Contributors in order in get deep insight of the problems that I would be working on during Coding Period. These are some basic things that I would need during official coding period, so I wont be wasting time on such things. Now I can get Started with my project. IRC Nick- AspiringMind

2 4

Re: [Maria-developers] GSoC NUMA support Project
by Daniel Black 29 May '17

29 May '17

On 29/05/17 06:03, Sumit Lakra wrote: > I think I am near done with dividing the buffer pool into the number of > nodes available in a NUMA machine. There is one thing left to do - to > check for the presence of 'libnuma' library in the server. Based on https://jira.mariadb.org/browse/MDEV-12924 I probably broke it in ab0e5031e41d0aa72dd77a6dced9eacc8be5c548 > As of now I > have explicitly defined HAVE_LIBNUMA in the > CMakeList.txt to check my code. CMakeList.txt in which directory? I'd suggest refering to publicly pushed code/branches for review. A branch on https://github.com/theGodlessLakra/server would be good. > Also, to test whether the control of execution reaches within a function > A, I tried to print out a message on the scree like 'fprintf(stderr, In > function A);' This seems to work in main(), my_init(), sys_var_init() > etc, but not in buf0buf.cc. The function which creates the buffer pool > is called by 'buf_pool_init()', which is called by > 'innobase_start_on_create_for_mysql()', which in turn is called by > 'innobase_init()', but I couldn't find where an d how is this function > called. Its called by function pointer -> see maria_declare_plugin in storage/innobase/handler/ha_innodb.cc which is called by the plugin initialization code. > Seems like the output to bot stdout and stderr is being > redirected somewhere and I can't find where (in code). reopen_fstreams in sql/{mysqld,log}.cc > Having this printing stuff is a quick way of testing the flow of control > and although my code is working, You can also run mysqld in gdb and the use "break" to set a breakpoint at a function/file:line and step through form there. > I would like to know about this > redirections if someone can tell me. add --verbose to the mysqld argument. > Also, which part of code calls the 'innobase_init()' function ? sql/plugin.cc:plugin_initialize in the call to plugin->init() > I have attached a screenshot of the server running on my NUMA emulated > pc with and without --numa-interleave=all. [ PID - 21016 without NUMA, > PID - 21058 with NUMA] > > At present it just ALLOCATES a buffer pool instance on each numa node. > Need to work on specific node binding and stuff.

1 0

[Maria-developers] GSoC NUMA support Project
by Sumit Lakra 28 May '17

28 May '17

I think I am near done with dividing the buffer pool into the number of nodes available in a NUMA machine. There is one thing left to do - to check for the presence of 'libnuma' library in the server. As of now I have explicitly defined HAVE_LIBNUMA in the CMakeList.txt to check my code. Also, to test whether the control of execution reaches within a function A, I tried to print out a message on the scree like 'fprintf(stderr, In function A);' This seems to work in main(), my_init(), sys_var_init() etc, but not in buf0buf.cc. The function which creates the buffer pool is called by 'buf_pool_init()', which is called by 'innobase_start_on_create_for_mysql()', which in turn is called by 'innobase_init()', but I couldn't find where an d how is this function called. Seems like the output to bot stdout and stderr is being redirected somewhere and I can't find where (in code). Having this printing stuff is a quick way of testing the flow of control and although my code is working, I would like to know about this redirections if someone can tell me. Also, which part of code calls the 'innobase_init()' function ? I have attached a screenshot of the server running on my NUMA emulated pc with and without --numa-interleave=all. [ PID - 21016 without NUMA, PID - 21058 with NUMA] At present it just ALLOCATES a buffer pool instance on each numa node. Need to work on specific node binding and stuff.

1 0

[Maria-developers] Building LibraOffice w/ MariaDB installed fails apt-get build-dep
by Howard Johnson 27 May '17

27 May '17

Hi, I'm having trouble with MariaDB as is described here: https://ask.libreoffice.org/en/question/96902/building-lo-w-mariadb-install… I suspect that there might be an issue with the virtual package or something like that. Any idea of what is going wrong or how to fix it? Thanks -- howard

2 1

Re: [Maria-developers] [Commits] df94429: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'
by Sergey Petrunia 26 May '17

26 May '17

Hi Varun, Ok to push. On Thu, May 25, 2017 at 07:41:50PM +0530, Varun wrote: > revision-id: df944293721a7ad93d115a58f2b574067d1dc79f (mariadb-10.1.20-282-gdf94429) > parent(s): a1b6128dedb4419db9fadaf94c356d3477d4e06f > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-05-25 19:40:08 +0530 > message: > > MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > was corrupted, server crashes in opt_sum_query > > extended keys feature disabled if the length of extended key is longer than MAX_KEY_LEN > > --- > mysql-test/r/innodb_ext_key.result | 81 ++++++++++++++++++++++++++++++++++++++ > mysql-test/t/innodb_ext_key.test | 51 ++++++++++++++++++++++++ > sql/table.cc | 39 +++++++++++++++++- > 3 files changed, 170 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/innodb_ext_key.result b/mysql-test/r/innodb_ext_key.result > index 1305be8..de1323e 100644 > --- a/mysql-test/r/innodb_ext_key.result > +++ b/mysql-test/r/innodb_ext_key.result > @@ -1133,5 +1133,86 @@ where index_date_updated= 10 and index_id < 800; > id select_type table type possible_keys key key_len ref rows Extra > 1 SIMPLE t2 range index_date_updated index_date_updated 13 NULL # Using index condition > drop table t0,t1,t2; > +# > +# MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > +# was corrupted, server crashes in opt_sum_query > +set @save_innodb_file_format= @@innodb_file_format; > +set @save_innodb_large_prefix= @@innodb_large_prefix; > +set global innodb_file_format = BARRACUDA; > +set global innodb_large_prefix = ON; > +CREATE TABLE t1 ( > +pk INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1024), > +PRIMARY KEY (pk), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def'); > +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk <= 4; > +MAX(t2.pk) > +2 > +drop table t1; > +CREATE TABLE t1 ( > +pk1 INT, > +pk2 INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1021), > +PRIMARY KEY (pk1,pk2), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(f2) where pk1 <= 5 and pk2 <=5 and f2 = 'abc' and f1 <= '3'; > +EXPLAIN > +{ > + "query_block": { > + "select_id": 1, > + "table": { > + "table_name": "t1", > + "access_type": "range", > + "possible_keys": ["f2"], > + "key": "f2", > + "key_length": "3070", > + "used_key_parts": ["f2", "pk1"], > + "rows": 1, > + "filtered": 100, > + "index_condition": "((t1.pk1 <= 5) and (t1.pk2 <= 5) and (t1.f2 = 'abc'))", > + "attached_condition": "(t1.f1 <= '3')" > + } > + } > +} > +drop table t1; > +CREATE TABLE t1 ( > +f2 INT, > +pk2 INT, > +f1 VARCHAR(3), > +pk1 VARCHAR(1000), > +PRIMARY KEY (pk1,pk2), > +KEY k1(pk1,f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(k1) where f2 <= 5 and pk2 <=5 and pk1 = 'abc' and f1 <= '3'; > +EXPLAIN > +{ > + "query_block": { > + "select_id": 1, > + "table": { > + "table_name": "t1", > + "access_type": "range", > + "possible_keys": ["k1"], > + "key": "k1", > + "key_length": "3011", > + "used_key_parts": ["pk1", "f2", "pk2"], > + "rows": 1, > + "filtered": 100, > + "index_condition": "((t1.f2 <= 5) and (t1.pk2 <= 5) and (t1.pk1 = 'abc'))", > + "attached_condition": "(t1.f1 <= '3')" > + } > + } > +} > +drop table t1; > set optimizer_switch=@save_ext_key_optimizer_switch; > +set global innodb_file_format = @save_innodb_file_format; > +set global innodb_large_prefix = @save_innodb_large_prefix; > SET SESSION STORAGE_ENGINE=DEFAULT; > diff --git a/mysql-test/t/innodb_ext_key.test b/mysql-test/t/innodb_ext_key.test > index bf94b7d..c8acfc5 100644 > --- a/mysql-test/t/innodb_ext_key.test > +++ b/mysql-test/t/innodb_ext_key.test > @@ -778,5 +778,56 @@ where index_date_updated= 10 and index_id < 800; > > drop table t0,t1,t2; > > + > +--echo # > +--echo # MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > +--echo # was corrupted, server crashes in opt_sum_query > + > +set @save_innodb_file_format= @@innodb_file_format; > +set @save_innodb_large_prefix= @@innodb_large_prefix; > +set global innodb_file_format = BARRACUDA; > +set global innodb_large_prefix = ON; > + > +CREATE TABLE t1 ( > + pk INT, > + f1 VARCHAR(3), > + f2 VARCHAR(1024), > + PRIMARY KEY (pk), > + KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > + > +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def'); > +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk <= 4; > +drop table t1; > + > +CREATE TABLE t1 ( > + pk1 INT, > + pk2 INT, > + f1 VARCHAR(3), > + f2 VARCHAR(1021), > + PRIMARY KEY (pk1,pk2), > + KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > + > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(f2) where pk1 <= 5 and pk2 <=5 and f2 = 'abc' and f1 <= '3'; > +drop table t1; > + > +CREATE TABLE t1 ( > +f2 INT, > +pk2 INT, > +f1 VARCHAR(3), > +pk1 VARCHAR(1000), > +PRIMARY KEY (pk1,pk2), > +KEY k1(pk1,f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(k1) where f2 <= 5 and pk2 <=5 and pk1 = 'abc' and f1 <= '3'; > +drop table t1; > + > set optimizer_switch=@save_ext_key_optimizer_switch; > +set global innodb_file_format = @save_innodb_file_format; > +set global innodb_large_prefix = @save_innodb_large_prefix; > SET SESSION STORAGE_ENGINE=DEFAULT; > diff --git a/sql/table.cc b/sql/table.cc > index 37c0b63..a1e9ebf 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1721,6 +1721,7 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > keyinfo= share->key_info; > uint primary_key= my_strcasecmp(system_charset_info, share->keynames.type_names[0], > primary_key_name) ? MAX_KEY : 0; > + KEY* key_first_info; > > if (primary_key >= MAX_KEY && keyinfo->flags & HA_NOSAME) > { > @@ -1800,19 +1801,38 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > keyinfo->name_length+1); > } > > + if (!key) > + key_first_info= keyinfo; > + > if (ext_key_parts > share->key_parts && key) > { > KEY_PART_INFO *new_key_part= (keyinfo-1)->key_part + > (keyinfo-1)->ext_key_parts; > uint add_keyparts_for_this_key= add_first_key_parts; > + uint length_bytes= 0, len_null_byte= 0, ext_key_length= 0; > + Field *field; > > /* > Do not extend the key that contains a component > defined over the beginning of a field. > */ > for (i= 0; i < keyinfo->user_defined_key_parts; i++) > - { > + { > uint fieldnr= keyinfo->key_part[i].fieldnr; > + field= share->field[keyinfo->key_part[i].fieldnr-1]; > + > + if (field->null_ptr) > + len_null_byte= HA_KEY_NULL_LENGTH; > + > + if (field->type() == MYSQL_TYPE_BLOB || > + field->real_type() == MYSQL_TYPE_VARCHAR || > + field->type() == MYSQL_TYPE_GEOMETRY) > + { > + length_bytes= HA_KEY_BLOB_LENGTH; > + } > + > + ext_key_length+= keyinfo->key_part[i].length + len_null_byte > + + length_bytes; > if (share->field[fieldnr-1]->key_length() != > keyinfo->key_part[i].length) > { > @@ -1821,6 +1841,23 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > } > } > > + if (add_keyparts_for_this_key) > + { > + for (i= 0; i < add_keyparts_for_this_key; i++) > + { > + uint pk_part_length= key_first_info->key_part[i].store_length; > + if (keyinfo->ext_key_part_map & 1<<i) > + { > + if (ext_key_length + pk_part_length > MAX_KEY_LENGTH) > + { > + add_keyparts_for_this_key= i; > + break; > + } > + ext_key_length+= pk_part_length; > + } > + } > + } > + > if (add_keyparts_for_this_key < (keyinfo->ext_key_parts - > keyinfo->user_defined_key_parts)) > { > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Cannot push branch to github
by Valentin Rakush 26 May '17

26 May '17

Hi all, I am trying to do first pull request to https://github.com/MariaDB/server but I cannot push my new branch to github. Git says "ERROR: Permission to MariaDB/server.git denied to alpes214. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists." Should I talk to somebody to get access rights first? -- Kind regards, Valentin Rakush

2 3

Re: [Maria-developers] MDEV-12803 Improve function parameter data type control
by Alexander Barkov 26 May '17

26 May '17

Hello Alexey, Thanks for your feedback. Sending a new version, as discussed in this email and on IRC. Please find attached. - Removed Item_geometry_func_importer - Renamed Item_xxx_geometry_property to Item_xxx_args_geometry - Renamed Item_xxx_dyadic_operation to Item_xxx_args_geometry_geometry. This is the new hierarchy: Item_str_func Item_geometry_func Item_geometry_func_args_geometry Item_func_centroid Item_func_envelope Item_func_boundary Item_func_spatial_decomp Item_func_spatial_decomp_n Item_func_buffer Item_func_pointonsurface Item_func_geometry_from_text Item_func_geometry_from_wkb Item_func_geometry_from_json Item_func_point Item_func_spatial_collection Item_func_spatial_operation Item_binary_func_args_geometry Item_func_as_wkb Item_real_func Item_real_func_args_geometry Item_func_x Item_func_y Item_func_area Item_func_glength Item_real_func_args_geometry_geometry Item_func_distance Item_long_func Item_long_func_args_geometry Item_func_issimple Item_func_isring Item_func_isclosed Item_func_dimension Item_func_numgeometries Item_func_numinteriorring Item_func_numpoints Item_func_srid Item_bool_func Item_bool_func_args_geometry Item_func_isempty Item_bool_func_args_geometry_geometry Item_func_spatial_relate Item_str_ascii_func Item_str_ascii_func_args_geometry Item_func_as_wkt Item_func_as_geojson Item_func_geometry_type Thanks! On 05/26/2017 11:50 AM, Alexey Botchkov wrote: >>1. It will look like a C fragment in a C++ code. >> Why is it more natural than a virtual method? > > The Item hierarchy and 'check_type' hierarchy are parallel. > So it seems natural to me if we can assign an already-prepared > parameter-validation function to the type. > >> 2. It will make the Item size bigger. > But it will make Item** virtual tables smaller. > >> 4. It will need more CPU for m_check_arguments_function initialization. > Must be really small overhead. > >> Are you afraid that I'll have to modify class hierarchy? > Yes. My only concern is that the class hierarchy becomes more complicated > and with repeating code. > > But after a while i sort of used to it so now i'm fine with the virtual > method. > > >> We have a convention that functions return true on error and false on >> success. > For the check_something function returning TRUE or FALSE doesn't mean an > 'error'. > It's just the condition fits or doesn't. > But fine, our code has many examples of 'check' functions returning > results this way. > > Typenames discussed. > So otherwise i'm ok with the patch. > > Best regards. > HF > > On Wed, May 24, 2017 at 4:22 PM, Alexander Barkov <bar(a)mariadb.org > <mailto:bar@mariadb.org>> wrote: > > Hi Alexey, > > Thank you very much for your review! > > Please see my comments below: > > > On 05/23/2017 12:16 AM, Alexey Botchkov wrote: > > Firstly I'd say i don't like the idea of the 'virtual check_arguments()' > > function. > > I belive it's more natural to have a member 'm_check_arguments_function' > > that would be set in constructor (or be an argument to the constructor) > > and the non-virtual 'check_arguments()' that would just call that member. > > This would make Item class structure much nicer and would even save us > > memory. > > > > If you have good arguments for your approach, I have some > > comments/questions to your patch :) > > I'm not sure that a function member is better. > > 1. It will look like a C fragment in a C++ code. > Why is it more natural than a virtual method? > > 2. It will make the Item size bigger. > > 3. It will need more coding. > Now we have to declare only virtual functions. > With a function member, we'll have to do: > - declare function anyway > - add a code into constructors > > 4. It will need more CPU for m_check_arguments_function initialization. > On the contrary, having a virtual methods costs nothing, > except a small initialization in VMTs during mysqld startup. > > > What is your main concern? > Are you afraid that I'll have to modify class hierarchy > for other Item_xxx_func? It should not be necessary. > We'll just have a set of protected methods on the Item_func level, > for most typical cases. > > > > > > Why this change? > > +++ b/mysql-test/t/gis-rtree.test > > @@ -61,7 +61,7 @@ while ($1) > > let $2=10; > > while ($2) > > { > > - eval DELETE FROM t2 WHERE Within(g, > > Envelope(GeometryFromWKB(Point($1 * 10 - 9, $2 * 10 - 9), Point($1 * 10, > > $2 * 10)))); > > + eval DELETE FROM t2 WHERE Within(g, > > Envelope(GeometryFromWKB(Point($1 * 10 - 9, $2 * 10 - 9), 0))); > > SELECT count(*) FROM t2; > > dec $2; > > > It passed POINT() as the second argiment to GeometryFromWKB. > The second argument is for SRID. After my changes it's not allowed. > > > > } > > > > > > It looks weird when the 'check_something' functions returns FALSE if > > the argument fits the check. I'd make them returning TRUE in this case > > as people normally expect. > > We have a convention that functions return true on error and false on > success. > > Please have a look into: > > check_db_name() > check_for_broken_triggers() > check_unique_table() > > They all return true on error. > I did the same. > > > > > > +bool Item_func::check_argument_types_scalar(uint start, uint end) const > > +{ > > + for (uint i= start; i < end; i++) > > + { > > + DBUG_ASSERT(i < arg_count); > > > > In code like this i'd rather do > > bool Item_func::check_argument_types_scalar(uint start, uint end) const > > { > > DBUG_ASSERT(end < arg_count); > > > > for (uint i= start; i < end; i++) > > { > > // no DBUG_ASSERT here > > ... > > Agree. > > > > > > > > > Some typenames feel misleading. I'd suggest changes > > Item_real_func_geometry_property -> Item_real_func_arg_geometry > > Item_int_func_geometry_property -> Item_int_func_arg_geometry > > Item_bool_func_geometry_property -> Item_bool_func_arg_geometry > > Item_str_ascii_func_geometry_property -> Item_ascii_func_arg_geometry > > Item_geometry_func_geometry_property -> Item_geometry_func_arg_geometry > > This looks like a property. > The first argument is an object of some complex type (GEOMETRY in > our case). > The other arguments are simple type additional optional parameters. > > In C++ this would look like: > > geom->IsSimple(); > geom->Buffer(10); > > > OpenGIS also calls these methods "properties", > so do we in the documentation: > > https://dev.mysql.com/doc/refman/5.7/en/gis-class-point.html > <https://dev.mysql.com/doc/refman/5.7/en/gis-class-point.html> > > > > > > > Item_binary_func_geometry_property - do we really need this type? > > Right, there is only one binary string property: AsWKB(). > > I added Item_xxx_geometry_property for regularity > (to have a symmetric class hierarchy with other "properties" ). > > Here's the class structure: > > > Item_str_func > Item_geometry_func > Item_geometry_func_geometry_property > Item_func_centroid > Item_func_envelope > Item_func_boundary > Item_func_spatial_decomp > Item_func_spatial_decomp_n > Item_func_buffer > Item_func_pointonsurface > Item_geometry_func_importer > Item_func_geometry_from_text > Item_func_geometry_from_wkb > Item_func_geometry_from_json > Item_func_point > Item_func_spatial_collection > Item_func_spatial_operation > Item_binary_func_geometry_property > Item_func_as_wkb > Item_real_func > Item_real_func_geometry_property > Item_func_x > Item_func_y > Item_func_area > Item_func_glength > Item_real_func_geometry_dyadic_operation > Item_func_distance > Item_int_func > Item_int_func_geometry_property > Item_func_issimple > Item_func_isring > Item_func_isclosed > Item_func_dimension > Item_func_numgeometries > Item_func_numinteriorring > Item_func_numpoints > Item_func_srid > Item_func_gis_debug > Item_bool_func > Item_bool_func_geometry_property > Item_func_isempty > Item_bool_func_geometry_dyadic_operation > Item_func_spatial_relate > Item_str_ascii_func > Item_str_ascii_func_geometry_property > Item_func_as_wkt > Item_func_as_geojson > Item_func_geometry_type > > Btw, do you need a program to print this hierarchy? :) > I wrote it a few years ago, and use it almost every day. > > > > > Item_geometry_func_importer - do we need this type? > > I can remove this. It does not have shared methods indeed. > > > > > Item_real_func_geometry_dyadic_operation -> Item_real_func_arg_2_geometries > > Item_bool_func_geometry_dyadic_operation -> Item_bool_func_arg_2_geometries > > It's an operation on two geometries. > My names is quite self-descriptive. > Also, it tells that: > - there is no any other GEOMETRY arguments > - there can be optional non-GEOMETRY additional arguments > > > > > > > + bool check_arguments() const > > + { > > + DBUG_ASSERT(arg_count >= 2); > > + return check_argument_types_or_binary(&type_handler_geometry, 0, 2); > > > > the check_argument_types_or_binary() does the check of the arg_count so > > no need for the DBUG_ASSERT(). > > But i don't insist on removing this line :) > > I can remove it :) > > > > > > > On Mon, May 15, 2017 at 5:56 PM, Alexander Barkov <bar(a)mariadb.org <mailto:bar@mariadb.org> > > <mailto:bar@mariadb.org <mailto:bar@mariadb.org>>> wrote: > > > > Hello Alexey, > > > > can you please review a patch for MDEV-12803? > > > > Thanks! > > > > > >

1 0

Re: [Maria-developers] [Commits] 5ac7ee3: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'
by Sergey Petrunia 25 May '17

25 May '17

Hi Varun, Good to see that the feedback from the last time was addressed. There is another issue, though. Consider a case where some (but not necessarily all) PK columns are present in the secondary key: create table t1 ( pk1 VARCHAR(1000) not null, pk2 INT not null, col2 INT not null, KEY key1(pk1, col2), PRIMARY KEY(pk1, pk2) ) CHARSET utf8 engine=innodb; insert into t1 select a,a,a from ten; In this case, key1 still has a suffix, but the suffix only includes columns that are not already present in the key. That is, the "real" definition of key1 is (pk1, col2, pk2) and not (pk1, col2, pk1, pk2) . Let's try that. explain select * from t1 force index(key1) where pk1 < 'zz' and col2 <=2 and pk2 <=4; Without the patch, I have all 3 key parts used: | { "query_block": { "select_id": 1, "table": { "table_name": "t1", "access_type": "range", "possible_keys": ["key1"], "key": "key1", "key_length": "3010", "used_key_parts": ["pk1", "col2", "pk2"], "rows": 1, "filtered": 100, "attached_condition": "t1.pk1 <= '0' and t1.col2 <= 2 and t1.pk2 <= 4", "using_index": true } } } | With the patch, I only see the explicitly defined key parts used: | { "query_block": { "select_id": 1, "table": { "table_name": "t1", "access_type": "range", "possible_keys": ["key1"], "key": "key1", "key_length": "3006", "used_key_parts": ["pk1", "col2"], "rows": 1, "filtered": 100, "attached_condition": "t1.pk1 <= '0' and t1.col2 <= 2 and t1.pk2 <= 4", "using_index": true } } } | Which I think is wrong, because the key length never exceeds MAX_KEY_LEN. This can be easily fixed by appyling this patch over your patch: --- table.cc.varun 2017-05-24 18:30:39.076341082 +0300 +++ table.cc 2017-05-24 19:04:25.637740027 +0300 @@ -2164,10 +2164,13 @@ for (i= 0; i < add_keyparts_for_this_key; i++) { uint pk_part_length= key_first_info->key_part[i].store_length; - if (ext_key_length + pk_part_length > MAX_KEY_LENGTH) + if (keyinfo->ext_key_part_map & 1<<i) { - add_keyparts_for_this_key= i; - break; + if (ext_key_length + pk_part_length > MAX_KEY_LENGTH) + { + add_keyparts_for_this_key= i; + break; + } } ext_key_length+= pk_part_length; } Do you agree with this change? Any amendments? If it's ok, please add this (together with the testcase) into your patch. I've checked MySQL 5.7 also, and they do something weird: ... insert into t1 select A.a + 1000*B.a, A.a + 1000*B.a, A.a + 1000*B.a from one_k A, ten B; explain format=json select * from t1 force index(key1) where pk1 <= '0' and col2 <=2 and pk2 <=4; EXPLAIN { "query_block": { "select_id": 1, "cost_info": { "query_cost": "1.41" }, "table": { "table_name": "t1", "access_type": "range", "possible_keys": [ "key1" ], "key": "key1", "used_key_parts": [ "pk1" ], "key_length": "3010", "rows_examined_per_scan": 1, "rows_produced_per_join": 0, "filtered": "11.11", "using_index": true, "cost_info": { "read_cost": "1.39", "eval_cost": "0.02", "prefix_cost": "1.41", "data_read_per_join": "335" }, "used_columns": [ "pk1", "pk2", "col2" ], "attached_condition": "((`test`.`t1`.`pk1` <= '0') and (`test`.`t1`.`col2` <= 2) and (`test`.`t1`.`pk2` <= 4))" } } } key_length is 3010, but used_key_parts only includes "pk1"... The numbers do not match. On Tue, May 23, 2017 at 07:08:48PM +0530, Varun wrote: > revision-id: 5ac7ee32a8e373d989bf9d665b5e01f770ac583f (mariadb-10.1.20-282-g5ac7ee3) > parent(s): a1b6128dedb4419db9fadaf94c356d3477d4e06f > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-05-23 19:00:15 +0530 > message: > > MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > was corrupted, server crashes in opt_sum_query > > extended keys feature disabled if the length of extended key is longer than MAX_KEY_LEN > > --- > mysql-test/r/innodb_ext_key.result | 51 ++++++++++++++++++++++++++++++++++++++ > mysql-test/t/innodb_ext_key.test | 38 ++++++++++++++++++++++++++++ > sql/table.cc | 36 ++++++++++++++++++++++++++- > 3 files changed, 124 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/innodb_ext_key.result b/mysql-test/r/innodb_ext_key.result > index 1305be8..c76660c 100644 > --- a/mysql-test/r/innodb_ext_key.result > +++ b/mysql-test/r/innodb_ext_key.result > @@ -1133,5 +1133,56 @@ where index_date_updated= 10 and index_id < 800; > id select_type table type possible_keys key key_len ref rows Extra > 1 SIMPLE t2 range index_date_updated index_date_updated 13 NULL # Using index condition > drop table t0,t1,t2; > +# > +# MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > +# was corrupted, server crashes in opt_sum_query > +set @save_innodb_file_format= @@innodb_file_format; > +set @save_innodb_large_prefix= @@innodb_large_prefix; > +set global innodb_file_format = BARRACUDA; > +set global innodb_large_prefix = ON; > +CREATE TABLE t1 ( > +pk INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1024), > +PRIMARY KEY (pk), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def'); > +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk <= 4; > +MAX(t2.pk) > +2 > +drop table t1; > +CREATE TABLE t1 ( > +pk1 INT, > +pk2 INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1021), > +PRIMARY KEY (pk1,pk2), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(f2) where pk1 <= 5 and pk2 <=5 and f2 = 'abc' and f1 <= '3'; > +EXPLAIN > +{ > + "query_block": { > + "select_id": 1, > + "table": { > + "table_name": "t1", > + "access_type": "range", > + "possible_keys": ["f2"], > + "key": "f2", > + "key_length": "3070", > + "used_key_parts": ["f2", "pk1"], > + "rows": 1, > + "filtered": 100, > + "index_condition": "((t1.pk1 <= 5) and (t1.pk2 <= 5) and (t1.f2 = 'abc'))", > + "attached_condition": "(t1.f1 <= '3')" > + } > + } > +} > +drop table t1; > set optimizer_switch=@save_ext_key_optimizer_switch; > +set global innodb_file_format = @save_innodb_file_format; > +set global innodb_large_prefix = @save_innodb_large_prefix; > SET SESSION STORAGE_ENGINE=DEFAULT; > diff --git a/mysql-test/t/innodb_ext_key.test b/mysql-test/t/innodb_ext_key.test > index bf94b7d..adb032f 100644 > --- a/mysql-test/t/innodb_ext_key.test > +++ b/mysql-test/t/innodb_ext_key.test > @@ -778,5 +778,43 @@ where index_date_updated= 10 and index_id < 800; > > drop table t0,t1,t2; > > + > +--echo # > +--echo # MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > +--echo # was corrupted, server crashes in opt_sum_query > + > +set @save_innodb_file_format= @@innodb_file_format; > +set @save_innodb_large_prefix= @@innodb_large_prefix; > +set global innodb_file_format = BARRACUDA; > +set global innodb_large_prefix = ON; > + > +CREATE TABLE t1 ( > + pk INT, > + f1 VARCHAR(3), > + f2 VARCHAR(1024), > + PRIMARY KEY (pk), > + KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > + > +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def'); > +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk <= 4; > +drop table t1; > + > +CREATE TABLE t1 ( > + pk1 INT, > + pk2 INT, > + f1 VARCHAR(3), > + f2 VARCHAR(1021), > + PRIMARY KEY (pk1,pk2), > + KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > + > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(f2) where pk1 <= 5 and pk2 <=5 and f2 = 'abc' and f1 <= '3'; > +drop table t1; > + > set optimizer_switch=@save_ext_key_optimizer_switch; > +set global innodb_file_format = @save_innodb_file_format; > +set global innodb_large_prefix = @save_innodb_large_prefix; > SET SESSION STORAGE_ENGINE=DEFAULT; > diff --git a/sql/table.cc b/sql/table.cc > index 37c0b63..c8c9696 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1721,6 +1721,7 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > keyinfo= share->key_info; > uint primary_key= my_strcasecmp(system_charset_info, share->keynames.type_names[0], > primary_key_name) ? MAX_KEY : 0; > + KEY* key_first_info; > > if (primary_key >= MAX_KEY && keyinfo->flags & HA_NOSAME) > { > @@ -1800,19 +1801,38 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > keyinfo->name_length+1); > } > > + if (!key) > + key_first_info= keyinfo; > + > if (ext_key_parts > share->key_parts && key) > { > KEY_PART_INFO *new_key_part= (keyinfo-1)->key_part + > (keyinfo-1)->ext_key_parts; > uint add_keyparts_for_this_key= add_first_key_parts; > + uint length_bytes= 0, len_null_byte= 0, ext_key_length= 0; > + Field *field; > > /* > Do not extend the key that contains a component > defined over the beginning of a field. > */ > for (i= 0; i < keyinfo->user_defined_key_parts; i++) > - { > + { > uint fieldnr= keyinfo->key_part[i].fieldnr; > + field= share->field[keyinfo->key_part[i].fieldnr-1]; > + > + if (field->null_ptr) > + len_null_byte= HA_KEY_NULL_LENGTH; > + > + if (field->type() == MYSQL_TYPE_BLOB || > + field->real_type() == MYSQL_TYPE_VARCHAR || > + field->type() == MYSQL_TYPE_GEOMETRY) > + { > + length_bytes= HA_KEY_BLOB_LENGTH; > + } > + > + ext_key_length+= keyinfo->key_part[i].length + len_null_byte > + + length_bytes; > if (share->field[fieldnr-1]->key_length() != > keyinfo->key_part[i].length) > { > @@ -1821,6 +1841,20 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > } > } > > + if (add_keyparts_for_this_key) > + { > + for (i= 0; i < add_keyparts_for_this_key; i++) > + { > + uint pk_part_length= key_first_info->key_part[i].store_length; > + if (ext_key_length + pk_part_length > MAX_KEY_LENGTH) > + { > + add_keyparts_for_this_key= i; > + break; > + } > + ext_key_length+= pk_part_length; > + } > + } > + > if (add_keyparts_for_this_key < (keyinfo->ext_key_parts - > keyinfo->user_defined_key_parts)) > { > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] MDEV-12803 Improve function parameter data type control
by Alexander Barkov 24 May '17

24 May '17

Hi Alexey, Thank you very much for your review! Please see my comments below: On 05/23/2017 12:16 AM, Alexey Botchkov wrote: > Firstly I'd say i don't like the idea of the 'virtual check_arguments()' > function. > I belive it's more natural to have a member 'm_check_arguments_function' > that would be set in constructor (or be an argument to the constructor) > and the non-virtual 'check_arguments()' that would just call that member. > This would make Item class structure much nicer and would even save us > memory. > > If you have good arguments for your approach, I have some > comments/questions to your patch :) I'm not sure that a function member is better. 1. It will look like a C fragment in a C++ code. Why is it more natural than a virtual method? 2. It will make the Item size bigger. 3. It will need more coding. Now we have to declare only virtual functions. With a function member, we'll have to do: - declare function anyway - add a code into constructors 4. It will need more CPU for m_check_arguments_function initialization. On the contrary, having a virtual methods costs nothing, except a small initialization in VMTs during mysqld startup. What is your main concern? Are you afraid that I'll have to modify class hierarchy for other Item_xxx_func? It should not be necessary. We'll just have a set of protected methods on the Item_func level, for most typical cases. > > Why this change? > +++ b/mysql-test/t/gis-rtree.test > @@ -61,7 +61,7 @@ while ($1) > let $2=10; > while ($2) > { > - eval DELETE FROM t2 WHERE Within(g, > Envelope(GeometryFromWKB(Point($1 * 10 - 9, $2 * 10 - 9), Point($1 * 10, > $2 * 10)))); > + eval DELETE FROM t2 WHERE Within(g, > Envelope(GeometryFromWKB(Point($1 * 10 - 9, $2 * 10 - 9), 0))); > SELECT count(*) FROM t2; > dec $2; It passed POINT() as the second argiment to GeometryFromWKB. The second argument is for SRID. After my changes it's not allowed. > } > > > It looks weird when the 'check_something' functions returns FALSE if > the argument fits the check. I'd make them returning TRUE in this case > as people normally expect. We have a convention that functions return true on error and false on success. Please have a look into: check_db_name() check_for_broken_triggers() check_unique_table() They all return true on error. I did the same. > > +bool Item_func::check_argument_types_scalar(uint start, uint end) const > +{ > + for (uint i= start; i < end; i++) > + { > + DBUG_ASSERT(i < arg_count); > > In code like this i'd rather do > bool Item_func::check_argument_types_scalar(uint start, uint end) const > { > DBUG_ASSERT(end < arg_count); > > for (uint i= start; i < end; i++) > { > // no DBUG_ASSERT here > ... Agree. > > > > Some typenames feel misleading. I'd suggest changes > Item_real_func_geometry_property -> Item_real_func_arg_geometry > Item_int_func_geometry_property -> Item_int_func_arg_geometry > Item_bool_func_geometry_property -> Item_bool_func_arg_geometry > Item_str_ascii_func_geometry_property -> Item_ascii_func_arg_geometry > Item_geometry_func_geometry_property -> Item_geometry_func_arg_geometry This looks like a property. The first argument is an object of some complex type (GEOMETRY in our case). The other arguments are simple type additional optional parameters. In C++ this would look like: geom->IsSimple(); geom->Buffer(10); OpenGIS also calls these methods "properties", so do we in the documentation: https://dev.mysql.com/doc/refman/5.7/en/gis-class-point.html > > Item_binary_func_geometry_property - do we really need this type? Right, there is only one binary string property: AsWKB(). I added Item_xxx_geometry_property for regularity (to have a symmetric class hierarchy with other "properties" ). Here's the class structure: Item_str_func Item_geometry_func Item_geometry_func_geometry_property Item_func_centroid Item_func_envelope Item_func_boundary Item_func_spatial_decomp Item_func_spatial_decomp_n Item_func_buffer Item_func_pointonsurface Item_geometry_func_importer Item_func_geometry_from_text Item_func_geometry_from_wkb Item_func_geometry_from_json Item_func_point Item_func_spatial_collection Item_func_spatial_operation Item_binary_func_geometry_property Item_func_as_wkb Item_real_func Item_real_func_geometry_property Item_func_x Item_func_y Item_func_area Item_func_glength Item_real_func_geometry_dyadic_operation Item_func_distance Item_int_func Item_int_func_geometry_property Item_func_issimple Item_func_isring Item_func_isclosed Item_func_dimension Item_func_numgeometries Item_func_numinteriorring Item_func_numpoints Item_func_srid Item_func_gis_debug Item_bool_func Item_bool_func_geometry_property Item_func_isempty Item_bool_func_geometry_dyadic_operation Item_func_spatial_relate Item_str_ascii_func Item_str_ascii_func_geometry_property Item_func_as_wkt Item_func_as_geojson Item_func_geometry_type Btw, do you need a program to print this hierarchy? :) I wrote it a few years ago, and use it almost every day. > > Item_geometry_func_importer - do we need this type? I can remove this. It does not have shared methods indeed. > > Item_real_func_geometry_dyadic_operation -> Item_real_func_arg_2_geometries > Item_bool_func_geometry_dyadic_operation -> Item_bool_func_arg_2_geometries It's an operation on two geometries. My names is quite self-descriptive. Also, it tells that: - there is no any other GEOMETRY arguments - there can be optional non-GEOMETRY additional arguments > > > + bool check_arguments() const > + { > + DBUG_ASSERT(arg_count >= 2); > + return check_argument_types_or_binary(&type_handler_geometry, 0, 2); > > the check_argument_types_or_binary() does the check of the arg_count so > no need for the DBUG_ASSERT(). > But i don't insist on removing this line :) I can remove it :) > > > On Mon, May 15, 2017 at 5:56 PM, Alexander Barkov <bar(a)mariadb.org > <mailto:bar@mariadb.org>> wrote: > > Hello Alexey, > > can you please review a patch for MDEV-12803? > > Thanks! > >

1 0

[Maria-developers] Oracle compatibility : Update with same source and target
by jerome brauge 23 May '17

23 May '17

Hello Alexander, In the same spirit of MDEV-12137, can you create a task for update with same source and target ? Regard, Jérôme.

2 3

[Maria-developers] MDEV-10142 - Replace function
by jerome brauge 23 May '17

23 May '17

Hello Alexander, Here is patch for replace function in oracle mode. Do I have to add a test for replication ? Regards, Jérôme.

2 4

Re: [Maria-developers] [Commits] 55a4858fe6f: MDEV-12666: CURRENT_ROLE() does not work in a view
by Alexander Barkov 23 May '17

23 May '17

Hi Vicentiu, It seems that Item_func_sysconst abuses const_charset_converter(). I have the following proposals: - Instead of adding a new code into Item::const_charset_converter(), can you please override Item_func_sysconst::safe_charset_converter()? The direction proposed by Sergei's looks good: Item *safe_charset_converter(THD *thd, CHARSET_INFO *tocs) { + if (thd->lex->context_analysis_only & CONTEXT_ANALYSIS_ONLY_VIEW) + return Item_str_func::safe_charset_converter(thd, tocs); return const_charset_converter(thd, tocs, true, fully_qualified_func_name() } But it probably needs to be extended to handle prepared statement. At PREPARE-time it should not call const_charset_converter(). At EXECUTE-time it should be Ok to call const_charset_converter(). So can you please change Item_func_sysconst::const_item() to report false when in views and during PREPARE-time? And then change safe_charset_converter() as follows: Item *safe_charset_converter(THD *thd, CHARSET_INFO *tocs) { if (!const_item()) return Item_str_func::safe_charset_converter(thd, tocs); return const_charset_converter(thd, tocs, true, fully_qualified_func_name() } - Please add tests for DATABASE() and CURRENT_USER - Please add tests for PS with changing context between PREPARE and EXECUTE. Thanks. On 05/22/2017 06:15 PM, vicentiu wrote: > revision-id: 55a4858fe6f3fdc63a072b93b9bc3d8e51dc5e94 (mariadb-10.0.30-76-g55a4858fe6f) > parent(s): 8d1827f5540a0def82b1e2cbc557245c3e5e14ca > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2017-05-22 17:06:01 +0300 > message: > > MDEV-12666: CURRENT_ROLE() does not work in a view > > The problem lies in how CURRENT_ROLE is defined. The > Item_func_current_role inherits from Item_func_sysconst, which defines > a safe_charset_converter to be a const_charset_converter. > > During view creation, if there is no role previously set, the current_role() > function returns NULL. > > This is captured on item instantiation and the > const_charset_converter call subsequently returns an Item_null. > In turn, the function is replaced with Item_null and the view is > then created with an Item_null instead of Item_func_current_role. > > Without this patch, the first SHOW CREATE VIEW from the testcase would > have a where clause of WHERE role_name = NULL, while the second SHOW > CREATE VIEW would show a correctly created view. > > The solution proposed is to not replace any function with an Item_null, > but instead use an Item_static_string_func. > > --- > .../suite/roles/current_role_view-12666.result | 48 ++++++++++++++++++++ > .../suite/roles/current_role_view-12666.test | 52 ++++++++++++++++++++++ > sql/item.cc | 13 +++++- > 3 files changed, 112 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/suite/roles/current_role_view-12666.result b/mysql-test/suite/roles/current_role_view-12666.result > new file mode 100644 > index 00000000000..525748b4875 > --- /dev/null > +++ b/mysql-test/suite/roles/current_role_view-12666.result > @@ -0,0 +1,48 @@ > +CREATE USER has_role@'localhost'; > +GRANT ALL PRIVILEGES ON *.* TO has_role@'localhost'; > +CREATE ROLE test_role; > +GRANT test_role TO has_role@'localhost'; > +CREATE USER no_role@'localhost'; > +GRANT ALL PRIVILEGES ON *.* TO no_role@'localhost'; > +CREATE TABLE view_role_test ( > +id int primary key, > +role_name varchar(50) > +); > +INSERT INTO view_role_test VALUES (1, 'test_role'); > +CREATE OR REPLACE > +DEFINER = no_role@localhost > +SQL SECURITY INVOKER > +VIEW v_view_role_test > +AS > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > +show create view v_view_role_test; > +View Create View character_set_client collation_connection > +v_view_role_test CREATE ALGORITHM=UNDEFINED DEFINER=`no_role`@`localhost` SQL SECURITY INVOKER VIEW `v_view_role_test` AS select `view_role_test`.`id` AS `id`,`view_role_test`.`role_name` AS `role_name` from `view_role_test` where (`view_role_test`.`role_name` = current_role()) latin1 latin1_swedish_ci > +SET ROLE test_role; > +CREATE OR REPLACE > +DEFINER = no_role@localhost > +SQL SECURITY INVOKER > +VIEW v_view_role_test_2 > +AS > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > +show create view v_view_role_test_2; > +View Create View character_set_client collation_connection > +v_view_role_test_2 CREATE ALGORITHM=UNDEFINED DEFINER=`no_role`@`localhost` SQL SECURITY INVOKER VIEW `v_view_role_test_2` AS select `view_role_test`.`id` AS `id`,`view_role_test`.`role_name` AS `role_name` from `view_role_test` where (`view_role_test`.`role_name` = current_role()) latin1 latin1_swedish_ci > +SELECT CURRENT_USER(); > +CURRENT_USER() > +root@localhost > +SELECT CURRENT_ROLE(); > +CURRENT_ROLE() > +test_role > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > +id role_name > +1 test_role > +SELECT * FROM v_view_role_test; > +id role_name > +1 test_role > +drop user has_role@'localhost'; > +drop user no_role@'localhost'; > +drop role test_role; > +drop table view_role_test; > +drop view v_view_role_test; > +drop view v_view_role_test_2; > diff --git a/mysql-test/suite/roles/current_role_view-12666.test b/mysql-test/suite/roles/current_role_view-12666.test > new file mode 100644 > index 00000000000..86c74260b6e > --- /dev/null > +++ b/mysql-test/suite/roles/current_role_view-12666.test > @@ -0,0 +1,52 @@ > +CREATE USER has_role@'localhost'; > +GRANT ALL PRIVILEGES ON *.* TO has_role@'localhost'; > + > +CREATE ROLE test_role; > +GRANT test_role TO has_role@'localhost'; > + > +CREATE USER no_role@'localhost'; > +GRANT ALL PRIVILEGES ON *.* TO no_role@'localhost'; > + > +CREATE TABLE view_role_test ( > + id int primary key, > + role_name varchar(50) > + ); > + > +INSERT INTO view_role_test VALUES (1, 'test_role'); > + > + > +CREATE OR REPLACE > +DEFINER = no_role@localhost > +SQL SECURITY INVOKER > +VIEW v_view_role_test > +AS > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > + > +show create view v_view_role_test; > + > +SET ROLE test_role; > + > +CREATE OR REPLACE > +DEFINER = no_role@localhost > +SQL SECURITY INVOKER > +VIEW v_view_role_test_2 > +AS > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > + > +show create view v_view_role_test_2; > + > + > +SELECT CURRENT_USER(); > +SELECT CURRENT_ROLE(); > + > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > +SELECT * FROM v_view_role_test; > + > + > +drop user has_role@'localhost'; > +drop user no_role@'localhost'; > +drop role test_role; > + > +drop table view_role_test; > +drop view v_view_role_test; > +drop view v_view_role_test_2; > diff --git a/sql/item.cc b/sql/item.cc > index 4ce8396f71e..2ee3a6d61dc 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -1263,9 +1263,20 @@ Item *Item::const_charset_converter(CHARSET_INFO *tocs, > DBUG_ASSERT(fixed); > StringBuffer<64>tmp; > String *s= val_str(&tmp); > - if (!s) > + > + /* Functions that return NULL during prepare must not be replaced with > + Item_null as they may return non-null values during execution. > + Ex: CURRENT_ROLE() during view creation time can be NULL if no role > + is set. */ > + if (!s && !func_name) > return new Item_null((char *) func_name, tocs); > > + if (!s) > + { > + tmp.set_ascii("", 0); > + s = &tmp; > + } > + > if (!needs_charset_converter(s->length(), tocs)) > { > if (collation.collation == &my_charset_bin && tocs != &my_charset_bin && > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

1 0

Re: [Maria-developers] [Commits] b400696: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'
by Sergey Petrunia 22 May '17

22 May '17

Hi Varun, Please find my input below. I would like again to voice my dissatisfaction that I have to point out coding style violations in pretty much every patch I get. The other error in the patch is also pretty basic, ok to have it once but I think it is reasonable to expect more polished patches to be submitted for code review. On Sat, May 20, 2017 at 01:50:42AM +0530, Varun wrote: > revision-id: b400696386529fa63e79f723bbb06c05f7bae820 (mariadb-10.2.2-416-gb400696) > parent(s): fff61e31eca83a0a9af7c30e2bcda8309e9c695a > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-05-20 01:49:34 +0530 > message: > > MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > was corrupted, server crashes in opt_sum_query > > extended keys feature disabled if the length of extended key is longer than MAX_KEY_LEN > > --- > mysql-test/r/innodb_ext_key.result | 45 ++++++++++++++++++++++++++++++++++++++ > mysql-test/t/innodb_ext_key.test | 31 ++++++++++++++++++++++++++ > sql/table.cc | 36 +++++++++++++++++++++++++++++- > 3 files changed, 111 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/innodb_ext_key.result b/mysql-test/r/innodb_ext_key.result > index 1305be8..66391d6 100644 > --- a/mysql-test/r/innodb_ext_key.result > +++ b/mysql-test/r/innodb_ext_key.result > @@ -1133,5 +1133,50 @@ where index_date_updated= 10 and index_id < 800; > id select_type table type possible_keys key key_len ref rows Extra > 1 SIMPLE t2 range index_date_updated index_date_updated 13 NULL # Using index condition > drop table t0,t1,t2; > +# > +# MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > +# was corrupted, server crashes in opt_sum_query > +CREATE TABLE t1 ( > +pk INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1024), > +PRIMARY KEY (pk), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8; > +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def'); > +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk <= 4; > +MAX(t2.pk) > +2 > +drop table t1; > +CREATE TABLE t1 ( > +pk1 INT, > +pk2 INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1021), > +PRIMARY KEY (pk1,pk2), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8; > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(f2) where pk1 <= 5 and pk2 <=5 and f2 = 'abc' and f1 <= '3'; > +EXPLAIN > +{ > + "query_block": { > + "select_id": 1, > + "table": { > + "table_name": "t1", > + "access_type": "range", > + "possible_keys": ["f2"], > + "key": "f2", > + "key_length": "3070", > + "used_key_parts": ["f2", "pk1"], > + "rows": 1, > + "filtered": 100, > + "index_condition": "t1.pk1 <= 5 and t1.pk2 <= 5 and t1.f2 = 'abc'", > + "attached_condition": "t1.f1 <= '3'" > + } > + } > +} > +drop table t1; > set optimizer_switch=@save_ext_key_optimizer_switch; > SET SESSION STORAGE_ENGINE=DEFAULT; ... > diff --git a/sql/table.cc b/sql/table.cc > index 398383e..7d88eba 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -2104,6 +2104,7 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > for (uint key=0 ; key < keys ; key++,keyinfo++) > { > uint usable_parts= 0; > + KEY* key_first_info; This variable may be used un-initialized. If I change the above line to be KEY* key_first_info=(KEY*)0x1; and run this statement: CREATE TABLE t1a ( pk INT, f1 VARCHAR(3), f2 VARCHAR(1024), f2a VARCHAR(1024), PRIMARY KEY (pk), KEY(f2), KEY(f2a) ) ENGINE=InnoDB CHARSET utf8; Then I get a crash. > keyinfo->name=(char*) share->keynames.type_names[key]; > keyinfo->name_length= strlen(keyinfo->name); > keyinfo->cache_name= > @@ -2118,19 +2119,38 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > keyinfo->name_length+1); > } > > + if(!key) Coding style: 'if (' ... > + key_first_info= keyinfo; > + > if (ext_key_parts > share->key_parts && key) > { > KEY_PART_INFO *new_key_part= (keyinfo-1)->key_part + > (keyinfo-1)->ext_key_parts; > uint add_keyparts_for_this_key= add_first_key_parts; > + uint length_bytes= 0, len_null_byte= 0, ext_key_length= 0; > + Field *field; > > /* > Do not extend the key that contains a component > defined over the beginning of a field. > */ > for (i= 0; i < keyinfo->user_defined_key_parts; i++) > - { > + { Coding style: wrong identation. > uint fieldnr= keyinfo->key_part[i].fieldnr; > + field= share->field[keyinfo->key_part[i].fieldnr-1]; > + > + if (field->null_ptr) > + len_null_byte= HA_KEY_NULL_LENGTH; > + > + if (field->type() == MYSQL_TYPE_BLOB || > + field->real_type() == MYSQL_TYPE_VARCHAR || > + field->type() == MYSQL_TYPE_GEOMETRY) > + { > + length_bytes= HA_KEY_BLOB_LENGTH; > + } > + > + ext_key_length+= keyinfo->key_part[i].length + len_null_byte > + + length_bytes; > if (share->field[fieldnr-1]->key_length() != > keyinfo->key_part[i].length) > { > @@ -2139,6 +2159,20 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > } > } > > + if (add_keyparts_for_this_key) > + { > + for (i= 0; i < add_keyparts_for_this_key; i++) > + { > + uint pk_part_length= key_first_info->key_part[i].store_length; > + if (ext_key_length + pk_part_length > MAX_KEY_LENGTH) > + { > + add_keyparts_for_this_key= i; > + break; > + } > + ext_key_length+= pk_part_length; > + } > + } > + > if (add_keyparts_for_this_key < (keyinfo->ext_key_parts - > keyinfo->user_defined_key_parts)) > { BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 55a4858fe6f: MDEV-12666: CURRENT_ROLE() does not work in a view
by Sergei Golubchik 22 May '17

22 May '17

Hi, Vicentiu! Another possible approach (that doesn't necessarily penalizes other const functions that happen to return NULL) would be for Item_func_sysconst not to behave as const during view creation. Because they really aren't, sysconst functions can change between view creation and a select from the view. The fix would be, then Item *safe_charset_converter(THD *thd, CHARSET_INFO *tocs) { + if (thd->lex->context_analysis_only & CONTEXT_ANALYSIS_ONLY_VIEW) + return Item_str_func::safe_charset_converter(thd, tocs); return const_charset_converter(thd, tocs, true, fully_qualified_func_name() } And possibly (not sure) other values of context_analysis_only would need a similar treatment here. On May 22, vicentiu wrote: > revision-id: 55a4858fe6f3fdc63a072b93b9bc3d8e51dc5e94 (mariadb-10.0.30-76-g55a4858fe6f) > parent(s): 8d1827f5540a0def82b1e2cbc557245c3e5e14ca > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2017-05-22 17:06:01 +0300 > message: > > MDEV-12666: CURRENT_ROLE() does not work in a view > > The problem lies in how CURRENT_ROLE is defined. The > Item_func_current_role inherits from Item_func_sysconst, which defines > a safe_charset_converter to be a const_charset_converter. > > During view creation, if there is no role previously set, the current_role() > function returns NULL. > > This is captured on item instantiation and the > const_charset_converter call subsequently returns an Item_null. > In turn, the function is replaced with Item_null and the view is > then created with an Item_null instead of Item_func_current_role. > > Without this patch, the first SHOW CREATE VIEW from the testcase would > have a where clause of WHERE role_name = NULL, while the second SHOW > CREATE VIEW would show a correctly created view. > > The solution proposed is to not replace any function with an Item_null, > but instead use an Item_static_string_func. > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MariaDB License Clarification
by Ryan Coe 19 May '17

19 May '17

Hello, My name is Ryan Coe and I've been maintaining the MariaDB package for Buildroot (https://buildroot.org) In commit 577915def8d39223aa23a06f898e11e82e72365a, COPYING.LESSER was removed. Our packaging currently states the MariaDB license as GPL-2.0 (server), GPL-2.0 with FLOSS exception (GPL client library), LGPL-2.0 (LGPL client library). Could I please get clarification on how the different components are licensed? Also, the GitHub page is still referencing COPYING.LESSER in the README. Thank you.

2 1

Re: [Maria-developers] strange IF in translog_flush_buffers()
by Sergei Golubchik 18 May '17

18 May '17

Hi, Sanja! On May 17, Oleksandr Byelkin wrote: > Hi! > > Actually code should work as it is, but it check unnecessary condition and > also very dificult to understand. I propose following fix (comment in it > explains why it work now and why condition is always TRUE) I cannot review it, this is between you and Monty. If you think it's ok, please push it (into 10.0). Or I can push it together with all other similar fixes. > diff --git a/storage/maria/ma_loghandler.c b/storage/maria/ma_loghandler.c > index 86e51cc3526..e680e830703 100644 > --- a/storage/maria/ma_loghandler.c > +++ b/storage/maria/ma_loghandler.c > @@ -7789,8 +7789,23 @@ void translog_flush_buffers(TRANSLOG_ADDRESS *lsn, > translog_force_current_buffer_to_finish(); > translog_buffer_unlock(buffer); > } > - else if (log_descriptor.bc.buffer->prev_last_lsn != LSN_IMPOSSIBLE) > + else > { > + if (log_descriptor.bc.buffer->last_lsn == LSN_IMPOSSIBLE) > + { > + /* > + In this case both last_lsn & prev_last_lsn are LSN_IMPOSSIBLE > + otherwise it will go in the first IF because LSN_IMPOSSIBLE less > + then any real LSN and cmp_translog_addr(*lsn, > + log_descriptor.bc.buffer->prev_last_lsn) will be TRUE > + */ > + DBUG_ASSERT(log_descriptor.bc.buffer->prev_last_lsn == > + LSN_IMPOSSIBLE); > + DBUG_PRINT("info", ("There is no LSNs yet generated => do nothing")); > + translog_unlock(); > + DBUG_VOID_RETURN; > + } > + > /* fix lsn if it was horizon */ > *lsn= log_descriptor.bc.buffer->prev_last_lsn; > DBUG_PRINT("info", ("LSN to flush fixed to prev last lsn: (%lu,0x%lx)", > @@ -7799,13 +7814,6 @@ void translog_flush_buffers(TRANSLOG_ADDRESS *lsn, > TRANSLOG_BUFFERS_NO); > translog_unlock(); > } > - else if (log_descriptor.bc.buffer->last_lsn == LSN_IMPOSSIBLE) > - { > - DBUG_PRINT("info", ("There is no LSNs yet generated => do nothing")); > - translog_unlock(); > - DBUG_VOID_RETURN; > - } > - > /* flush buffers */ > *sent_to_disk= translog_get_sent_to_disk(); > if (cmp_translog_addr(*lsn, *sent_to_disk) > 0) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-12803 Improve function parameter data type control
by Alexander Barkov 15 May '17

15 May '17

Hello Alexey, can you please review a patch for MDEV-12803? Thanks!

1 0

[Maria-developers] String functions have their item->maybe_null=true unnecessarily, blocking optimizations
by Sergey Petrunia 15 May '17

15 May '17

Hi Alexander, I was looking at https://jira.mariadb.org/browse/MDEV-11893 and found this piece of code: bool Item_str_func::fix_fields(THD *thd, Item **ref) { bool res= Item_func::fix_fields(thd, ref); /* In Item_str_func::check_well_formed_result() we may set null_value flag on the same condition as in test() below. */ maybe_null= maybe_null || thd->is_strict_mode(); return res; } This has been introduced by this cset: https://github.com/MariaDB/server/commit/af22eb35e577ef17226faf662f2cffc470… Which says: Add Item_str_func::fix_fields() implementation, and set maybe_null to TRUE if we are in the SQL mode that requires some functions to return null even if they normally do not. The patch has only one example of CHAR() function doing that. At the same time, not having NOT-NULL attribute disallows query opimizations (see MDEV-11893 for an example). I think we should have this maybe_null= maybe_null || thd->is_strict_mode(); logic on a case-by-case basis only. I wanted to check with you - are there any other known string functions that, as the patch puts it "return null even if they normally do not"? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] MDEV-11340 Allow multiple alternative authentication methods for the same user
by serg＠mariadb.org 15 May '17

15 May '17

Hi, Ankit! I've started writing down my thoughts about MDEV-11340 "Allow multiple alternative authentication methods for the same user" See MDEV-11340 and feel free to comment on it. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 8edbb1117a9: MDEV-12741: innodb.ibuf_not_empty failed in buildbot with "InnoDB: Trying to do I/O to a tablespace which does not exist"
by Marko Mäkelä 15 May '17

15 May '17

On Thu, May 11, 2017 at 11:25 AM, jan <jan.lindstrom(a)mariadb.com> wrote: > revision-id: 8edbb1117a9e1fd81fbd08b8f1d06c72efe38f44 (mariadb-10.2.5-168- > g8edbb1117a9) > parent(s): f4df8c00c957146f04db031f1f9f80aab229d1d2 > author: Jan Lindström > committer: Jan Lindström > timestamp: 2017-05-11 11:15:37 +0300 > message: > > MDEV-12741: innodb.ibuf_not_empty failed in buildbot with "InnoDB: Trying > to do I/O to a tablespace which does not exist" > > Background thread is doing ibuf merge, in buf0rea.cc > buf_read_ibuf_merge_pages(). > It first tries to get page_size and if space is not found it deletes them, > but > as we do not hold any mutexes, space can be marked as stopped between that > and buf_read_page_low() for same space. This naturally leads seen error > message on log. > > buf_read_page_low(): Add parameter ignore_missing_space = false that > is passed to fil_io() > [snip] > @@ -178,7 +180,7 @@ buf_read_page_low( > > *err = fil_io( > request, sync, page_id, page_size, 0, page_size.physical(), > - dst, bpage); > + dst, bpage, ignore_missing_space); > > if (sync) { > thd_wait_end(NULL); > Can you please add the parameter to the IORequest? It would seem to be a more appropriate place for this. (Ideally, fil_io() would only take the request as a parameter, but we do not want to do such a big refactoring right now.) Marko

1 0

Re: [Maria-developers] [Commits] e6dabcd: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'
by Sergey Petrunia 11 May '17

11 May '17

Hi Varun, So I am debugging the testcase from the bug report. I reach these lines: > if(i) > ext_key_length+= key_part->length; (Why there is no indentation btw?) and I see: (gdb) p key_part->length $40 = 3072 3072=1024*3, that is, the NULL-byte and length bytes are NOT counted. The code then proceeds to check if pk_length + ext_key_length <= MAX_KEY_LENGTH but that check could produce a wrong result if NULL- and length- bytes are not counted. The second issue is that I don't see where ext_key_length is zero'ed. I try debugging this query: CREATE TABLE t2 ( pk INT, f1 VARCHAR(3), f2 VARCHAR(1024), f2a VARCHAR(1024), PRIMARY KEY (pk), KEY(f2), KEY(f2a) ) ENGINE=InnoDB CHARSET utf8; I put a breakpoint at this line: > if (share->use_ext_keys && i && !(keyinfo->flags & HA_NOSAME) > && (pk_length + ext_key_length <= MAX_KEY_LENGTH)) (84 chars! where's the coding style again?) and I observe: (gdb) print ext_key_length $45 = 6144 This is clearly more than any [possible] key length for the table. So, this particular patch is not a solution (even if the testcase happens to pass with it). On Mon, May 08, 2017 at 04:23:11PM +0530, Varun wrote: > revision-id: e6dabcd3a7e994feaa1a1dc5d0c1a0d621272f72 (mariadb-10.2.2-387-ge6dabcd) > parent(s): 39bce67b7aa27fc7e5458f36838148f6e0f4c3af > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-05-08 16:21:32 +0530 > message: > > MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > was corrupted, server crashes in opt_sum_query > > extended keys feature disabled if the length of extended key is longer than MAX_KEY_LEN > > --- > sql/table.cc | 8 ++++++-- > 1 file changed, 6 insertions(+), 2 deletions(-) > > diff --git a/sql/table.cc b/sql/table.cc > index 398383e..554fc78 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -695,7 +695,7 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > KEY_PART_INFO *key_part= NULL; > ulong *rec_per_key= NULL; > KEY_PART_INFO *first_key_part= NULL; > - uint first_key_parts= 0; > + uint first_key_parts= 0, pk_length=0, ext_key_length=0; > > if (!keys) > { > @@ -765,6 +765,7 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > keyinfo->algorithm= first_keyinfo->algorithm; > if (new_frm_ver >= 3) > keyinfo->block_size= first_keyinfo->block_size; > + pk_length= keyinfo->key_length; > } > > keyinfo->key_part= key_part; > @@ -796,6 +797,8 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > strpos+=7; > } > key_part->store_length=key_part->length; > + if(i) > + ext_key_length+= key_part->length; > } > > /* > @@ -805,7 +808,8 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > keyinfo->ext_key_parts= keyinfo->user_defined_key_parts; > keyinfo->ext_key_flags= keyinfo->flags; > keyinfo->ext_key_part_map= 0; > - if (share->use_ext_keys && i && !(keyinfo->flags & HA_NOSAME)) > + if (share->use_ext_keys && i && !(keyinfo->flags & HA_NOSAME) > + && (pk_length + ext_key_length <= MAX_KEY_LENGTH)) > { > for (j= 0; > j < first_key_parts && keyinfo->ext_key_parts < MAX_REF_PARTS; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MDEV-10764 PL/SQL parser - Phase 2 / MDEV-12137
by jerome brauge 11 May '17

11 May '17

2 1

[Maria-developers] mariadb-java-client 1.6.0 does not work from yesterday update ?
by Krishnan, Gnanavel (Nokia - IN/Chennai) 11 May '17

11 May '17

Hi All, I was using mariadb java client jar version 1.6.0. but looks it is not present in release document Also from yesterday 1.6.0 jar gets updated not sure why Should use only 1.5.9 or 2.0.0 rc ? https://repo1.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/1.6.0/ Thank you, Gnanavel.

1 0

Re: [Maria-developers] [Commits] 39bce67: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'
by Sergey Petrunia 10 May '17

10 May '17

Hi Varun, First reaction: > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 84e4018..63636f9 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -3948,7 +3948,11 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > key_part_info->fieldnr= field; > key_part_info->offset= (uint16) sql_field->offset; > key_part_info->key_type=sql_field->pack_flag; > - uint key_part_length= sql_field->key_length; > + uint key_part_length= MY_MAX(sql_field->key_length, > + sql_field->pack_length); What does the above do? It is not clear why MAX should be used. (Let's denote the compared variables X and Y. Can X>Y and X<Y both happen in practice? This requires a comment!) > + > + if (!(sql_field->flags & NOT_NULL_FLAG)) > + key_part_length++; > > if (column->length) > { I am concerned that the above may have an undesired effect: key_part_kength counts NULL-byte while previously it did not. Let's try this example: create table t1 (pk int not null primary key, col1 int, key(col1) ) engine=innodb; Run it, and put a breakpoint below on this code: /* Check if the key segment is partial, set the key flag accordingly */ if (key_part_length != sql_field->key_length) key_info->flags|= HA_KEY_HAS_PART_KEY_SEG; You will see that for filed 'col1' HA_KEY_HAS_PART_KEY_SEG flag is now set. This didn't happen before. Is this really intended? I wasn't able to find what exactly may get wrong, but if I run this query: explain select * from t1 where col1=1; then it hangs. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MDEV-12775 Reuse data type aggregation code for hybrid functions and UNION
by Alexander Barkov 10 May '17

10 May '17

Hello Sanja, Please review a patch for MDEV-12775, also fixing a number of bugs: - MDEV-9495 Wrong field type for a UNION of a signed and an unsigned INT expression - MDEV-9497 UNION and COALESCE produce different field types for DECIMAL+INT - MDEV-12594 UNION between fixed length double columns does not always preserve scale - MDEV-12595 UNION converts INT to BIGINT - MDEV-12599 UNION is not symmetric when mixing INT and CHAR Thanks!

1 0

[Maria-developers] galera source and direction
by Daniel Black 09 May '17

09 May '17

With Codership's galera repository showing spelling fixes and test case changes as the only changes this year, and Percona's galera repository streaming ahead with improvements, is Percona's galera becoming the new base? I did a quick test though and the mtr tests all pass with Percona's galera on the 10.2 branch head.

1 0

[Maria-developers] Google Summer of Code 2017 students
by Daniel Black 05 May '17

05 May '17

Folks I'd like to welcome our Google Summer of Code students. https://summerofcode.withgoogle.com/organizations/5889423829893120/#projects Ankit Kumar - Allow multiple alternative authentication methods for the same user mentor: Sergei Golubchik Daniil Medvedev - Full DECIMAL support in ColumnStore mentor: Andrew Hutchings Sumit Lakra - NUMA GSoC Project Proposal mentors: Daniel Black and Jan Lindström Galina Shalygina - Transformation of IN predicate into IN subquery based on implementation of table value constructor mentor: Igor Babaev Please make them welcome and we are looking forward to mentoring them to develop their contributions.

1 0

[Maria-developers] MDEV-7049 MySQL#74585 - InnoDB: Failing assertion: *mbmaxlen < 5 in file ha_innodb.cc line 1904
by Alexander Barkov 04 May '17

04 May '17

Hello Marko, InnoDB uses 5 bits to store the combination of mbminlen and mbmaxlen, and its calculated as: mbmaxlen * 5 + mbminlen Note, combinations with mbminlen=0 and mbmaxlen=0 are not possible. Also, combinations with mbminlen>mbmaxlen are not possible. This is a summary: mbminlen mbmaxlen 0 1 2 3 4 0 (0) (1) (2) (3) (4) 1 (5) 6 (7) (8) (9) 2 (10) 11 12 (13) (14) 3 (15) 16 17 18 (19) 4 (20) 21 22 23 24 I marked impossible numbers with parentheses. In the range 0..24: - 10 pairs are valid - 15 pairs are not valid It should be safe to use any impossible number to mean mbmaxlen=5 & mbminlen=1 It seems there is no really a need to extend the header to use separate 3 bits for both mbminlen and mbmaxlen.

2 2

[Maria-developers] MDEV-10142 - bb-10.2-compatibility
by jerome brauge 04 May '17

04 May '17

Hello Alexander, This is the patch for CHR(). Function CHR(n) returns the character having the binary equivalent to n in the database character set. Returns null on null input. Regard, Jérôme. PS : I share this code under terms of MCA. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : mercredi 26 avril 2017 06:12 > À : jerome brauge > Cc : MariaDB Developers > Objet : Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > > Hello Jerome, > > Thanks for working on this! > > On 04/25/2017 11:42 PM, jerome brauge wrote: > > Hello Alexander, > > Some times ago, we have had the below discussion. > > I made a patch following my first idea (it's enough from our point of view) : > replace empty string by null in literals and in make_empty_string. > > I think this patch can be complementary to your suggestion of adding > > an Item_func_eq_oracle (for data inserted in default sql_mode) > > > > This patch also contains following some functions changes: > > - replace ('ab','a',null) returns 'a' instead null > > - trim / ltrim / rtrim returns null when input only contains spaces > > - lpad /rpad can accept 2 args (default padding char is space) and if > > length equals to 0, returns null -substring : if start index is equal > > to 0, act as if it's equals to 1 -add a function chr() as a synonym of > > char() (but accept only one arg) -add a function lengthb() as a > > synonym of lentgh() -change the semantic of length for oracle (where > > length() is the character length and not the byte length) > > > > What do you think about this patch ? > > > > (Perhaps I have to make 2 patchs, one for empty string, and one for > > functions changes with separate test file for each modified function) > > Right, can you please split the patch into pieces? > I suggest even more pieces than two. One patch for one function. > I'll create MDEVs, one MDEV per function and put examples how Oracle > works. > > Can we do it in the following order: > > 1. LPAD / RPAD - making the third parameter optional. > This extension will be good for all sql_modes. > I checked IBM DB2, Oracle PostgreSQL, Firebird - they all have > the third parameter to LPAD/RPAD optional. > > 2. CHR(). > > 3. LENGTH() and LENGTHB(). > > > Then we can continue with REPLACE, TRIM. SUBSTRING in no particular > order, one patch per function. > > > Some general notes: > > > 1. Unify *.yy files > > Please put sql_mode dependent code into LEX rather than *.yy files directly. > We're going to unify the two files soon. See here for details: > MDEV-12518) Unify sql_yacc.yy and sql_yacc_ora.yy > > So instead of doing: > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > $$= new (thd->mem_root) Item_func_substr(thd, $3, $5, $7); > if ($$ == NULL) > MYSQL_YYABORT; > } > > > Please do something like this in both sql_yacc.yy and sql_yacc_ora.yy: > > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > if (!($$= Lex->make_item_func_substr(thd, $3, $5, $7))) > MYSQL_YYABORT; > } > > where LEX::make_item_func_substr() is a new method, creating either > Item_func_substr or Item_func_substr_oracle depending on sql_mode. > > The same applies to all other functions, and to text_literal. > > > 2. Don't use Item::is_null() when you call val_str() anyway. > > There are two reasons that: > - Performance > is_null() involves the second val_str() call, which can be resource consuming > in case of complex functions and subselect. > > - Side effect > Expressions can have some side effects. > For example, in case if the argument is a stored function which counts how > many times it was called and stores the counter in a user variable, or writes > something to a table on every execution. > If we do is_null() followed by val_str(), the side effect will done two times. > We need one time only. > > So instead of: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > if (res = replace_value(str, > args[1]->is_null() ? &tmp_emtpystr : > rgs[1]->val_str(&tmp_value2), > args[2]->is_null() ? &tmp_emtpystr : > rgs[2]->val_str(&tmp_value3)) > > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Please do something like this: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > String *str1= args[1]->val_str(&tmp_value1); > String *str2= args[2]->val_str(&tmp_value2); > if (res= replace_value(str, > str1 ? str1 : &tmp_emptystr, > str2 ? str2 : &tmp_emptystr) > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Thanks! > > > > > > > Best regards, > > Jérôme. > > > >> -----Message d'origine----- > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 23 > >> janvier 2017 12:49 À : jerome brauge Cc : MariaDB Developers Objet : > >> Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > >> > >> Hi Jerome, > >> > >> On 01/18/2017 01:22 PM, jerome brauge wrote: > >>> Hello Alexander, > >>> Sometime ago, when I ask you about plan for MDEV-10574, you replied : > >>> > >>>> The current plan is to do these transformations: > >>>> > >>>> 1. Transform Insert > >>>> - insert values ("") -> insert values (null) > >>>> > >>>> 2. Transform Select > >>>> > >>>> - where v=x => (v <> "" and V=X) > >>>> - where v is null => (v="" or v is null) > >>>> > >>>> We didn't plan to change functions yet. Thanks for bringing this up. > >>>> We'll discuss this. > >>> > >>> I've done some tests just by changing : > >>> - insert an Item_null instead of an Item_string when $1.length==0 in > >>> rule text_literal of sql_yacc_ora.yy > >>> - return null instead of an empty string in > >>> Item_str_func::make_empty_result > >>> > >>> My first tests seem promising. > >>> > >>> Of course this solution does not allow to "see" the records created > >>> with > >> empty strings as null values. > >>> I don't see the importance of being able to do this in a transparent way. > >>> We can explicitly select these row by adding rtrim on these columns. > >>> > >>> If you are interesting, I can begin to write a test to evaluate the > >>> coverage of > >> this solution. > >> > >> I'm afraid this will fix the behavior only for literals. > >> > >> This script: > >> > >> DROP TABLE t1; > >> CREATE TABLE t1 (a VARCHAR(10), b VARCHAR(10)); INSERT INTO t1 > VALUES > >> ('',''); SELECT COUNT(*) FROM t1 WHERE a=''; SELECT COUNT(*) FROM t1 > >> WHERE a=b; > >> > >> returns 0 for both SELECT queries in Oracle, and returns 1 for both > >> SELECT queries in MariaDB, Your approach will fix the first SELECT only. > >> The second will still return 1. > >> > >> > >> For now, I'm not sure how to do this correctly. Some ways possible: > >> > >> 1. We could add new classes, e.g. Item_func_eq_oracle as a pair for > >> Item_func_eq, which will handle both empty strings and NULLs inside > >> their > >> val_int() implementations. > >> > >> 2. Or there could be some after-processing on the created Item tree > >> which will replace all things like Item_func_eq to Item_cond_and with > >> Item_func_eq and Item_func_null_predicate passed as arguments. > >> > >> Currently I'm inclined to #1, because we don't have a good mechanism > >> to clone items to implement #2 correctly. We'd need such cloning to > >> pass arguments both to Item_func_eq and Item_func_null_predicate > correctly. > >> Some places in the code don't expect that the same arguments can be > >> passed to two different Item_func instances. > >> > >> On the other hand, implementing #1 will need more changes in the > >> optimizer. > >> > >> We were going to think on this later. > >> > >>> > >>> Best regard. > >>> Jérôme. > >>>

2 6

[Maria-developers] cte page
by Galina Shalygina 04 May '17

04 May '17

мои артиклы

4 5

[Maria-developers] Please review a patch for MDEV-12665 Hybrid functions do not preserve geometry type
by Alexander Barkov 02 May '17

02 May '17

Hello Alexey, Please review a joint patch fixing these two problems: MDEV-12665 Hybrid functions do not preserve geometry type MDEV-12560 Wrong data type for SELECT NULL UNION SELECT Point(1,1) Thanks!

1 0

[Maria-developers] Race Condition in Seconds behind master.
by Sachin Setiya 02 May '17

02 May '17

Hello Everyone! I setup a classical master-slave server. Master is constantly running query like this insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); flush logs; insert into t1 values(1,1,1,1,1); {there are arround million querys} Then I executed Show slave status (couple of times in a second), and found Seconds_behind_master is changing arbitrarily MariaDB [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Queueing master event to the relay log Master_Host: localhost Master_User: root Master_Port: 16000 Connect_Retry: 60 Master_Log_File: master-bin.014595 Read_Master_Log_Pos: 4 Relay_Log_File: slave-relay-bin.029186 Relay_Log_Pos: 3557 Relay_Master_Log_File: master-bin.014593 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 3257 Relay_Log_Space: 184375 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 148 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: Current_Pos Gtid_IO_Pos: 0-1-145942 Replicate_Do_Domain_Ids: Replicate_Ignore_Domain_Ids: Parallel_Mode: conservative SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Reading event from the relay log 1 row in set (0.01 sec) MariaDB [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: localhost Master_User: root Master_Port: 16000 Connect_Retry: 60 Master_Log_File: master-bin.014632 Read_Master_Log_Pos: 3257 Relay_Log_File: slave-relay-bin.029264 Relay_Log_Pos: 1548 Relay_Master_Log_File: master-bin.014632 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 1248 Relay_Log_Space: 179996 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: Current_Pos Gtid_IO_Pos: 0-1-146322 Replicate_Do_Domain_Ids: Replicate_Ignore_Domain_Ids: Parallel_Mode: conservative SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: init 1 row in set (0.00 sec) MariaDB [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Queueing master event to the relay log Master_Host: localhost Master_User: root Master_Port: 16000 Connect_Retry: 60 Master_Log_File: master-bin.014670 Read_Master_Log_Pos: 299 Relay_Log_File: slave-relay-bin.029336 Relay_Log_Pos: 3557 Relay_Master_Log_File: master-bin.014668 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 3257 Relay_Log_Space: 184569 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 150 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: Current_Pos Gtid_IO_Pos: 0-1-146692 Replicate_Do_Domain_Ids: Replicate_Ignore_Domain_Ids: Parallel_Mode: conservative SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Reading event from the relay log -- Regards Sachin Setiya Software Engineer at MariaDB

3 2

[Maria-developers] bb-10.2-ext / rpad lpad with 2 args
by jerome brauge 02 May '17

02 May '17

Hi Alexander, This is the patch for rpad/lpad with 2 args (call of make_empty_string() in val_str() is to prepare MDEV-10574). Regard, Jérôme. PS : I share this code under terms of MCA. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : mercredi 26 avril 2017 06:12 > À : jerome brauge > Cc : MariaDB Developers > Objet : Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > > Hello Jerome, > > Thanks for working on this! > > On 04/25/2017 11:42 PM, jerome brauge wrote: > > Hello Alexander, > > Some times ago, we have had the below discussion. > > I made a patch following my first idea (it's enough from our point of view) : > replace empty string by null in literals and in make_empty_string. > > I think this patch can be complementary to your suggestion of adding > > an Item_func_eq_oracle (for data inserted in default sql_mode) > > > > This patch also contains following some functions changes: > > - replace ('ab','a',null) returns 'a' instead null > > - trim / ltrim / rtrim returns null when input only contains spaces > > - lpad /rpad can accept 2 args (default padding char is space) and if > > length equals to 0, returns null -substring : if start index is equal > > to 0, act as if it's equals to 1 -add a function chr() as a synonym of > > char() (but accept only one arg) -add a function lengthb() as a > > synonym of lentgh() -change the semantic of length for oracle (where > > length() is the character length and not the byte length) > > > > What do you think about this patch ? > > > > (Perhaps I have to make 2 patchs, one for empty string, and one for > > functions changes with separate test file for each modified function) > > Right, can you please split the patch into pieces? > I suggest even more pieces than two. One patch for one function. > I'll create MDEVs, one MDEV per function and put examples how Oracle > works. > > Can we do it in the following order: > > 1. LPAD / RPAD - making the third parameter optional. > This extension will be good for all sql_modes. > I checked IBM DB2, Oracle PostgreSQL, Firebird - they all have > the third parameter to LPAD/RPAD optional. > > 2. CHR(). > > 3. LENGTH() and LENGTHB(). > > > Then we can continue with REPLACE, TRIM. SUBSTRING in no particular > order, one patch per function. > > > Some general notes: > > > 1. Unify *.yy files > > Please put sql_mode dependent code into LEX rather than *.yy files directly. > We're going to unify the two files soon. See here for details: > MDEV-12518) Unify sql_yacc.yy and sql_yacc_ora.yy > > So instead of doing: > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > $$= new (thd->mem_root) Item_func_substr(thd, $3, $5, $7); > if ($$ == NULL) > MYSQL_YYABORT; > } > > > Please do something like this in both sql_yacc.yy and sql_yacc_ora.yy: > > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > if (!($$= Lex->make_item_func_substr(thd, $3, $5, $7))) > MYSQL_YYABORT; > } > > where LEX::make_item_func_substr() is a new method, creating either > Item_func_substr or Item_func_substr_oracle depending on sql_mode. > > The same applies to all other functions, and to text_literal. > > > 2. Don't use Item::is_null() when you call val_str() anyway. > > There are two reasons that: > - Performance > is_null() involves the second val_str() call, which can be resource consuming > in case of complex functions and subselect. > > - Side effect > Expressions can have some side effects. > For example, in case if the argument is a stored function which counts how > many times it was called and stores the counter in a user variable, or writes > something to a table on every execution. > If we do is_null() followed by val_str(), the side effect will done two times. > We need one time only. > > So instead of: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > if (res = replace_value(str, > args[1]->is_null() ? &tmp_emtpystr : > rgs[1]->val_str(&tmp_value2), > args[2]->is_null() ? &tmp_emtpystr : > rgs[2]->val_str(&tmp_value3)) > > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Please do something like this: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > String *str1= args[1]->val_str(&tmp_value1); > String *str2= args[2]->val_str(&tmp_value2); > if (res= replace_value(str, > str1 ? str1 : &tmp_emptystr, > str2 ? str2 : &tmp_emptystr) > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Thanks! > > > > > > > Best regards, > > Jérôme. > > > >> -----Message d'origine----- > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 23 > >> janvier 2017 12:49 À : jerome brauge Cc : MariaDB Developers Objet : > >> Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > >> > >> Hi Jerome, > >> > >> On 01/18/2017 01:22 PM, jerome brauge wrote: > >>> Hello Alexander, > >>> Sometime ago, when I ask you about plan for MDEV-10574, you replied : > >>> > >>>> The current plan is to do these transformations: > >>>> > >>>> 1. Transform Insert > >>>> - insert values ("") -> insert values (null) > >>>> > >>>> 2. Transform Select > >>>> > >>>> - where v=x => (v <> "" and V=X) > >>>> - where v is null => (v="" or v is null) > >>>> > >>>> We didn't plan to change functions yet. Thanks for bringing this up. > >>>> We'll discuss this. > >>> > >>> I've done some tests just by changing : > >>> - insert an Item_null instead of an Item_string when $1.length==0 in > >>> rule text_literal of sql_yacc_ora.yy > >>> - return null instead of an empty string in > >>> Item_str_func::make_empty_result > >>> > >>> My first tests seem promising. > >>> > >>> Of course this solution does not allow to "see" the records created > >>> with > >> empty strings as null values. > >>> I don't see the importance of being able to do this in a transparent way. > >>> We can explicitly select these row by adding rtrim on these columns. > >>> > >>> If you are interesting, I can begin to write a test to evaluate the > >>> coverage of > >> this solution. > >> > >> I'm afraid this will fix the behavior only for literals. > >> > >> This script: > >> > >> DROP TABLE t1; > >> CREATE TABLE t1 (a VARCHAR(10), b VARCHAR(10)); INSERT INTO t1 > VALUES > >> ('',''); SELECT COUNT(*) FROM t1 WHERE a=''; SELECT COUNT(*) FROM t1 > >> WHERE a=b; > >> > >> returns 0 for both SELECT queries in Oracle, and returns 1 for both > >> SELECT queries in MariaDB, Your approach will fix the first SELECT only. > >> The second will still return 1. > >> > >> > >> For now, I'm not sure how to do this correctly. Some ways possible: > >> > >> 1. We could add new classes, e.g. Item_func_eq_oracle as a pair for > >> Item_func_eq, which will handle both empty strings and NULLs inside > >> their > >> val_int() implementations. > >> > >> 2. Or there could be some after-processing on the created Item tree > >> which will replace all things like Item_func_eq to Item_cond_and with > >> Item_func_eq and Item_func_null_predicate passed as arguments. > >> > >> Currently I'm inclined to #1, because we don't have a good mechanism > >> to clone items to implement #2 correctly. We'd need such cloning to > >> pass arguments both to Item_func_eq and Item_func_null_predicate > correctly. > >> Some places in the code don't expect that the same arguments can be > >> passed to two different Item_func instances. > >> > >> On the other hand, implementing #1 will need more changes in the > >> optimizer. > >> > >> We were going to think on this later. > >> > >>> > >>> Best regard. > >>> Jérôme. > >>>

2 4

[Maria-developers] MDEV-12617 CASE and CASE-alike hybrid functions do not preserve exact data types
by Alexander Barkov 28 Apr '17

28 Apr '17

Hi Sanja, can you please review a patch for MDEV-12617? Thanks!

1 0

[Maria-developers] MDEV-10142 - bb-10.2-compatibility / MDEV-10574
by jerome brauge 28 Apr '17

28 Apr '17

Hello Alexander, Sometime ago, when I ask you about plan for MDEV-10574, you replied : >The current plan is to do these transformations: > >1. Transform Insert >- insert values ("") -> insert values (null) > >2. Transform Select > >- where v=x => (v <> "" and V=X) >- where v is null => (v="" or v is null) > >We didn't plan to change functions yet. Thanks for bringing this up. >We'll discuss this. I've done some tests just by changing : - insert an Item_null instead of an Item_string when $1.length==0 in rule text_literal of sql_yacc_ora.yy - return null instead of an empty string in Item_str_func::make_empty_result My first tests seem promising. Of course this solution does not allow to "see" the records created with empty strings as null values. I don't see the importance of being able to do this in a transparent way. We can explicitly select these row by adding rtrim on these columns. If you are interesting, I can begin to write a test to evaluate the coverage of this solution. Best regard. Jérôme.

2 6

[Maria-developers] MDEV-10142 - bb-10.2-compatibility
by jerome brauge 27 Apr '17

27 Apr '17

Hello Alexander, This is the patch for length/lengthb. Regard, Jérôme. PS : I share this code under terms of MCA. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : mercredi 26 avril 2017 06:12 > À : jerome brauge > Cc : MariaDB Developers > Objet : Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > > Hello Jerome, > > Thanks for working on this! > > On 04/25/2017 11:42 PM, jerome brauge wrote: > > Hello Alexander, > > Some times ago, we have had the below discussion. > > I made a patch following my first idea (it's enough from our point of view) : > replace empty string by null in literals and in make_empty_string. > > I think this patch can be complementary to your suggestion of adding > > an Item_func_eq_oracle (for data inserted in default sql_mode) > > > > This patch also contains following some functions changes: > > - replace ('ab','a',null) returns 'a' instead null > > - trim / ltrim / rtrim returns null when input only contains spaces > > - lpad /rpad can accept 2 args (default padding char is space) and if > > length equals to 0, returns null -substring : if start index is equal > > to 0, act as if it's equals to 1 -add a function chr() as a synonym of > > char() (but accept only one arg) -add a function lengthb() as a > > synonym of lentgh() -change the semantic of length for oracle (where > > length() is the character length and not the byte length) > > > > What do you think about this patch ? > > > > (Perhaps I have to make 2 patchs, one for empty string, and one for > > functions changes with separate test file for each modified function) > > Right, can you please split the patch into pieces? > I suggest even more pieces than two. One patch for one function. > I'll create MDEVs, one MDEV per function and put examples how Oracle > works. > > Can we do it in the following order: > > 1. LPAD / RPAD - making the third parameter optional. > This extension will be good for all sql_modes. > I checked IBM DB2, Oracle PostgreSQL, Firebird - they all have > the third parameter to LPAD/RPAD optional. > > 2. CHR(). > > 3. LENGTH() and LENGTHB(). > > > Then we can continue with REPLACE, TRIM. SUBSTRING in no particular > order, one patch per function. > > > Some general notes: > > > 1. Unify *.yy files > > Please put sql_mode dependent code into LEX rather than *.yy files directly. > We're going to unify the two files soon. See here for details: > MDEV-12518) Unify sql_yacc.yy and sql_yacc_ora.yy > > So instead of doing: > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > $$= new (thd->mem_root) Item_func_substr(thd, $3, $5, $7); > if ($$ == NULL) > MYSQL_YYABORT; > } > > > Please do something like this in both sql_yacc.yy and sql_yacc_ora.yy: > > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > if (!($$= Lex->make_item_func_substr(thd, $3, $5, $7))) > MYSQL_YYABORT; > } > > where LEX::make_item_func_substr() is a new method, creating either > Item_func_substr or Item_func_substr_oracle depending on sql_mode. > > The same applies to all other functions, and to text_literal. > > > 2. Don't use Item::is_null() when you call val_str() anyway. > > There are two reasons that: > - Performance > is_null() involves the second val_str() call, which can be resource consuming > in case of complex functions and subselect. > > - Side effect > Expressions can have some side effects. > For example, in case if the argument is a stored function which counts how > many times it was called and stores the counter in a user variable, or writes > something to a table on every execution. > If we do is_null() followed by val_str(), the side effect will done two times. > We need one time only. > > So instead of: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > if (res = replace_value(str, > args[1]->is_null() ? &tmp_emtpystr : > rgs[1]->val_str(&tmp_value2), > args[2]->is_null() ? &tmp_emtpystr : > rgs[2]->val_str(&tmp_value3)) > > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Please do something like this: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > String *str1= args[1]->val_str(&tmp_value1); > String *str2= args[2]->val_str(&tmp_value2); > if (res= replace_value(str, > str1 ? str1 : &tmp_emptystr, > str2 ? str2 : &tmp_emptystr) > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Thanks! > > > > > > > Best regards, > > Jérôme. > > > >> -----Message d'origine----- > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 23 > >> janvier 2017 12:49 À : jerome brauge Cc : MariaDB Developers Objet : > >> Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > >> > >> Hi Jerome, > >> > >> On 01/18/2017 01:22 PM, jerome brauge wrote: > >>> Hello Alexander, > >>> Sometime ago, when I ask you about plan for MDEV-10574, you replied : > >>> > >>>> The current plan is to do these transformations: > >>>> > >>>> 1. Transform Insert > >>>> - insert values ("") -> insert values (null) > >>>> > >>>> 2. Transform Select > >>>> > >>>> - where v=x => (v <> "" and V=X) > >>>> - where v is null => (v="" or v is null) > >>>> > >>>> We didn't plan to change functions yet. Thanks for bringing this up. > >>>> We'll discuss this. > >>> > >>> I've done some tests just by changing : > >>> - insert an Item_null instead of an Item_string when $1.length==0 in > >>> rule text_literal of sql_yacc_ora.yy > >>> - return null instead of an empty string in > >>> Item_str_func::make_empty_result > >>> > >>> My first tests seem promising. > >>> > >>> Of course this solution does not allow to "see" the records created > >>> with > >> empty strings as null values. > >>> I don't see the importance of being able to do this in a transparent way. > >>> We can explicitly select these row by adding rtrim on these columns. > >>> > >>> If you are interesting, I can begin to write a test to evaluate the > >>> coverage of > >> this solution. > >> > >> I'm afraid this will fix the behavior only for literals. > >> > >> This script: > >> > >> DROP TABLE t1; > >> CREATE TABLE t1 (a VARCHAR(10), b VARCHAR(10)); INSERT INTO t1 > VALUES > >> ('',''); SELECT COUNT(*) FROM t1 WHERE a=''; SELECT COUNT(*) FROM t1 > >> WHERE a=b; > >> > >> returns 0 for both SELECT queries in Oracle, and returns 1 for both > >> SELECT queries in MariaDB, Your approach will fix the first SELECT only. > >> The second will still return 1. > >> > >> > >> For now, I'm not sure how to do this correctly. Some ways possible: > >> > >> 1. We could add new classes, e.g. Item_func_eq_oracle as a pair for > >> Item_func_eq, which will handle both empty strings and NULLs inside > >> their > >> val_int() implementations. > >> > >> 2. Or there could be some after-processing on the created Item tree > >> which will replace all things like Item_func_eq to Item_cond_and with > >> Item_func_eq and Item_func_null_predicate passed as arguments. > >> > >> Currently I'm inclined to #1, because we don't have a good mechanism > >> to clone items to implement #2 correctly. We'd need such cloning to > >> pass arguments both to Item_func_eq and Item_func_null_predicate > correctly. > >> Some places in the code don't expect that the same arguments can be > >> passed to two different Item_func instances. > >> > >> On the other hand, implementing #1 will need more changes in the > >> optimizer. > >> > >> We were going to think on this later. > >> > >>> > >>> Best regard. > >>> Jérôme. > >>>

1 0

[Maria-developers] Please review MDEV-12588 Add Type_handler::type_handler_for_tmp_table() and Type_handler::type_handler_for_union()
by Alexander Barkov 25 Apr '17

25 Apr '17

Hello Sanja, Please review a patch for MDEV-12588. Thanks!

1 0

[Maria-developers] Please review MDEV-12426 Add Field::type_handler()
by Alexander Barkov 25 Apr '17

25 Apr '17

Hello Vicențiu, Please review a patch for: MDEV-12426 Add Field::type_handler() also fixing: MDEV-12432 Range optimizer for ENUM and SET does not return "Impossible WHERE" in some case Thank you!

2 2

[Maria-developers] MDEV-9217 Split Item::tmp_table_field_from_field_type() into virtual methods in Type_handler
by Alexander Barkov 24 Apr '17

24 Apr '17

Hello Sanja, Please review a patch for MDEV-9217. Thanks!

2 3

Re: [Maria-developers] MDEV-12506 Split Item_func_min_max::fix_length_and_dec() into methods in Type_handler
by Alexander Barkov 22 Apr '17

22 Apr '17

Hi Alexey, Thanks for your review! On 04/22/2017 06:49 PM, Alexey Botchkov wrote: > Ok to push. > I'd only move the bool Item_func_min_max::fix_attributes(Item **items, > uint nitems) function > to the item_func.h as it's just the typehandler method call now. I'm afraid this is not possible. This method uses current_thd->is_error(), and the structure of THD is not visible in item_func.h. > > Best regards. > HF > > > On Sat, Apr 15, 2017 at 7:23 AM, Alexander Barkov <bar(a)mariadb.org > <mailto:bar@mariadb.org>> wrote: > > Hello Alexey, > > Can you please review a patch for MDEV-12506? > > Thanks. > >

1 0

[Maria-developers] Refactoring in LEX_STRING and LEX_CSTRING
by Alexander Barkov 21 Apr '17

21 Apr '17

Hello all, Monty is now doing refactoring: 1. Replacing a lot of LEX_STRING to LEX_CSTRING. 2. Fixing functions and methods that have arguments of types: a. pointer "const LEX_STRING *name" and b. reference "const LEX_STRING &name". to the same style. Monty chose passing by pointer (2a). I think that: - #1 is good. - #2 is good, in the sense of changing to the same style. But I think that the choice of "passing by pointer" in #2 is bad. Passing by reference for LEX_STRING/LEX_CSTRING related things would be better. With the "passing by pointer" approach whenever one needs to pass a LEX_STRING to a function accepting LEX_CSTRING, one will have to do: void func1(const LEX_STRING *str) { LEX_CSTRING tmp= {str->str, str->length}; func2(&tmp); } I'd propose two things instead: 1. Add a class: class Lex_cstring: public LEX_CSTRING { public: Lex_cstring() {} /*Uninitialized */ Lex_cstring(const char *str, size_t length) { LEX_STRING::str= str; LEX_STRING::length= length; } Lex_cstring(const LEX_STRING *str) { LEX_STRING::str= str->str; LEX_STRING::length= str->length; } }; 2. Instead of "const LEX_CSTRING *str" we use "const Lex_cstring &str" all around C++ code. So the above example with passing a LEX_STRING to something that expect a LEX_CSTRING would be simplified to void func1(const LEX_STRING *str) { func2(str); } 3. In some classes now we have duplicate methods and constructors like this: class XXX { public: XXX(const char *str, size_t length) { ... } XXX(const LEX_STRING *str) { ... } void method(const char *str, size_t length) {...} void method(const LEX_STRING *str) {...} }; to make it possible to call like this: new XXX(str, length); new XXX(lex_str_ptr) new XXX(&lex_str) method(str, length); method(lex_str_ptr); method(&lex_str); We change this to "passing by reference" and remove duplicates: class XXX { public: XXX(const Lex_cstring &str) { ... } void method(const Lex_cstring &str) {...} }; After this change we can call like this: new XXX(Lex_cstring(str, length)); new XXX(*lex_str_ptr) new XXX(lex_str); method(Lex_cstring(str, length)); method(*lex_str_ptr); method(lex_str); Notice, not needs for separate methods accepting two arguments "const char *str, size_t length".

4 5

[Maria-developers] How to report progress for InnoDB background tasks (esp. at startup/shutdown)?
by Marko Mäkelä 18 Apr '17

18 Apr '17

Hi all, MariaDB Server features a nice-looking interface for reporting progress on long-running SQL operations: https://mariadb.com/kb/en/mariadb/progress-reporting/ MySQL 5.7 implemented something similar for reporting progress for ALTER TABLE…ALGORITHM=INPLACE. I should reimplement that for MariaDB using the existing MariaDB infrastructure. I wonder if the same interface could be used for tracking progress on periodic background tasks that might not be directly related to the execution of any SQL statements. InnoDB has a number of background tasks, such as: * purging unneeded history (removing undo log records and delete-marked index records) * rolling back recovered transactions (mainly, if the server was killed) * index defragmentation, encryption key rotation * redo log checkpoints (flush all dirty pages up to the checkpoint LSN; most interesting at shutdown) The above would need to be monitored also while the server is shutting down. I understand that at startup, we would probably need a different progress reporting interfaces for the very first InnoDB task of redo log apply. Everything else can be done in the background while the server is accepting connections. Related to this, I wonder if it is possible or feasible to change the SQL layer logic so that at shutdown, it will not issue ROLLBACK (as part of KILL QUERY) to the currently active connections, but let the subsequent startup do the rollback in the background. This should allow faster shutdowns and restarts. Best regards, Marko -- DON’T MISS M|17April 11 - 12, 2017 The Conrad Hotel New York City https://m17.mariadb.com/ Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation

2 2

[Maria-developers] Item_func_group_concat::fix_fields - indentation and check positioning?
by Daniel Black 18 Apr '17

18 Apr '17

I came across the following gcc warning (6.3.1) sql/item_sum.cc: In member function ‘virtual bool Item_func_group_concat::fix_fields(THD*, Item**)’: sql/item_sum.cc:3478:5: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation] if ((!args[i]->fixed && ^~ sql/item_sum.cc:3482:7: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘if’ with_subselect|= args[i]->with_subselect; ^~~~~~~~~~~~~~ The with_subselect line was added in 950abd526 In context it is: Item_func_group_concat::fix_fields(THD *thd, Item **ref) { ... for (i=0 ; i < arg_count ; i++) { if ((!args[i]->fixed && args[i]->fix_fields(thd, args + i)) || args[i]->check_cols(1)) return TRUE; with_subselect|= args[i]->with_subselect; } While removing the indent would fix the warning, I'd like someone to check this line should be after the IF statement rather than before.

2 1

Re: [Maria-developers] [JIRA] (MDEV-12499) ARCHIVE tables still don't show up in I_S.TABLES when storage engine is not loaded
by Jean-Francois B. Gagne 17 Apr '17

17 Apr '17

Hi Sergei, (sorry for cross-posting to MariaDB.dev and MariaDB.discuss: I think this interest both audiences.) Sergei Golubchik commented on MDEV-12499: > ----------------------------------------- > > This only happens if you created an ARCHIVE table and never used it before > the upgrade. No INSERT, no SELECT, not even {{SELECT * FROM > INFORMATION_SCHEMA.TABLES}} — nothing. > > ARCHIVE engine supports table discovery, that is, the engine is always the > authoritative source of table metadata, while {{.frm}} files are merely a > metadata cache. When you do {{CREATE TABLE}} the server does not know > whether the engine wants the table metadata to be cached. This happens when > the table is opened for the first time. > > If there is no {{.frm}} file and the ARCHIVE engine is not loaded, the > server has no way of knowing that the ARCHIVE table exists. That's why it's > not shown anywhere. > > If you do {{SELECT * FROM test.a1}} before the upgrade, the engine will > tell the server to create the {{.frm}} file and on the upgrade the plugin > will be loaded. In a practical use case, it's enough to have at least one > ARCHIVE table that was accessed at least once after it was created, and the > ARCHIVE plugin will be loaded on upgrade. > Thanks for the detailed info above. In this, you write ".frm files are merely a metadata cache", is there any sort of documentation about when .frm files are useful, useless, and might get out of sync ? This probably deserves a page in the KB (and Google does not know much on "mariadb frm file site:mariadb.com"). However, your comment above raises the following question: if "ARCHIVE engine supports table discovery, that is, the engine is always the authoritative source of table metadata", where is this information stored ? I guess that unless the plugin is loaded, this information is not accessible. This causes a chicken and egg problem which MDEV-12499 was trying to solve: how can mysql_upgrade knows if it should add BLACKHOLE and ARCHIVE to the list of plugin to load (see MDEV-11942). Finally, you marked MDEV-12499 as "Won't Fix", do you have another solution for ARCHIVE and MDEV-11942 ? Many thanks, JFG

2 1

[Maria-developers] MDEV-12514 Split Item_temporal_func::fix_length_and_dec()
by Alexander Barkov 17 Apr '17

17 Apr '17

Hi Sanja, Please review a patch for MDEV-12514, also fixing this bug: MDEV-12515 Wrong value when storing DATE_ADD() and ADDTIME() to a numeric field Thanks!

1 1

[Maria-developers] MDEV-12506 Split Item_func_min_max::fix_length_and_dec() into methods in Type_handler
by Alexander Barkov 15 Apr '17

15 Apr '17

Hello Alexey, Can you please review a patch for MDEV-12506? Thanks.

1 0

[Maria-developers] Please review patch for 12017 and 12018.
by Sachin Setiya 09 Apr '17

09 Apr '17

Hi Monty!, Please review patch for 12017 and 12018 . this patch solves both the problem. -- Regards Sachin Setiya Software Engineer at MariaDB

2 1

[Maria-developers] compilation failure: my_atomic.h
by Alexander Barkov 08 Apr '17

08 Apr '17

Hi Sergey, kvm-bintar-centos5-x86 fails to compile: http://buildbot.askmonty.org/buildbot/grid?category=main&branch=10.3 with this error: /home/buildbot/buildbot/build/include/my_atomic.h:121:2: error: #error atomic ops for this platform are not implemented Can you please fix this?

3 4

[Maria-developers] Adding "proxy protocol" support to MariaDB
by Jose 07 Apr '17

07 Apr '17

Hi! As you may know, the proxy protocol [1,2] is a way to communicate a client's IP address to a backend server sitting behind a proxy front-end. It is the equivalent of the X-Forwarded-For HTTP header, but done at Layer 4. This protocol was originally proposed by one of the HAProxy developers and its support is becoming more widespread. Percona, Amazon AWS, HAProxy, NGINX, and others. Mariadb's Jira shows an unassigned feature request for it, but with no one apparently assigned to it [3]. Percona supports proxy protocol [4]. For the configuration part, Percona defines a global variable that gives the IP address of the proxy front-ends that are authorized to send a proxy protocol header. Percona's handler code for the proxy protocol (minus the global variable handling) is in the vio/viosockets.c file [4] I looked at Percona's implementation. This file looks very similar to MariaDb's own viosockets.c file [5] so it may a potential good insertion point. Percona's software license seems to be permisive enough for reusing / adapting their code [6]. Please correct me if I'm wrong but it seems no one is currently working on this feature right now. I'd like to know if you would be interested in help side-porting Percona's proxy protocol code to MariaDB. I'll probably need some help or pointers on how to add a global variable, though. Best regards, --jose [1] Proxy protocol documentation https://blog.haproxy.com/haproxy/proxy-protocol/ [2] Proxy protocol specification www.haproxy.org/download/1.8/doc/proxy-protocol.txt [3] Mariadb feature request for proxy protocol https://jira.mariadb.org/browse/MDEV-11159?jql=project%20%3D%20MDEV%20AND%2… [3] Percona proxy protocol manual page https://www.percona.com/doc/percona-server/5.6/flexibility/proxy_protocol_s… [4] Percona proxy protocol handler code https://github.com/percona/percona-server/blob/1e2f003a5bd48763c27e37542d97… https://github.com/percona/percona-server/blob/1e2f003a5bd48763c27e37542d97… [5] mariadb potential insertion pode for proxy protocol code https://github.com/MariaDB/server/blob/bb2c1a52c61706dde8c525a8887f2d364c0d… [6] Percona software license https://www.percona.com/doc/percona-server/LATEST/copyright.html

3 3

Re: [Maria-developers] b4a223e3381: MDEV-10355 Weird error message upon CREATE TABLE with DEFAULT
by Sergei Golubchik 07 Apr '17

07 Apr '17

Hi, Jacob! On Mar 28, jacob.mathew(a)mariadb.com wrote: > revision-id: b4a223e338152c8764c980aeae59fb927eea1b0b (mariadb-10.2.4-98-gb4a223e3381) > parent(s): 3f7455c03008b2428fa9b364b1add4c36834ff71 > author: Jacob Mathew > committer: Jacob Mathew > timestamp: 2017-03-28 17:41:56 -0700 > message: > > MDEV-10355 Weird error message upon CREATE TABLE with DEFAULT > > Fixed handling of default values with cached temporal functions so that the > CREATE TABLE statement now succeeds. > Added clearing of cached function values to trigger function reevaluation > when updating default values and when updating virtual column values. > Fixed the error message. > Added quoting of date/time values in cases when this was omitted. > Added a test case. > Updated test result files that include date/time values that were > previously unquoted. > > diff --git a/mysql-test/r/func_default_between_temporal.result b/mysql-test/r/func_default_between_temporal.result > new file mode 100644 > index 00000000000..db9656f4d37 > --- /dev/null > +++ b/mysql-test/r/func_default_between_temporal.result > @@ -0,0 +1,15 @@ > +CREATE OR REPLACE TABLE t1 ( col INT DEFAULT ( 1 LIKE ( NOW() BETWEEN '2000-01-01' AND '2012-12-12' ) ) ); > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `col` int(11) DEFAULT (1 like (current_timestamp() between '2000-01-01 00:00:00' and '2012-12-12 00:00:00')) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +SET timestamp = UNIX_TIMESTAMP( '2004-04-04' ); > +INSERT INTO t1 VALUES( DEFAULT ); > +SET timestamp = DEFAULT; better use another constant here, don't rely on the current timestamp being between 2000 and 2012. I know, sounds crazy. But debian's reproducible builds (*) do everything with the system time seriously off. (*) https://wiki.debian.org/ReproducibleBuilds > +INSERT INTO t1 VALUES( DEFAULT ); > +SELECT * FROM t1; > +col > +1 > +0 > +DROP TABLE t1; > diff --git a/mysql-test/t/func_default_between_temporal.test b/mysql-test/t/func_default_between_temporal.test please, rename the test somehow or put it into default.test. "func_default" means it's about a DEFAULT() function, but your test is not. > new file mode 100644 > index 00000000000..6cbd5a342c6 > --- /dev/null > +++ b/mysql-test/t/func_default_between_temporal.test > @@ -0,0 +1,11 @@ Start the test with a header, like # # MDEV-10355 Weird error message upon CREATE TABLE with DEFAULT # > +CREATE OR REPLACE TABLE t1 ( col INT DEFAULT ( 1 LIKE ( NOW() BETWEEN '2000-01-01' AND '2012-12-12' ) ) ); > +SHOW CREATE TABLE t1; > + > +SET timestamp = UNIX_TIMESTAMP( '2004-04-04' ); > +INSERT INTO t1 VALUES( DEFAULT ); > +SET timestamp = DEFAULT; > +INSERT INTO t1 VALUES( DEFAULT ); > + > +SELECT * FROM t1; > + > +DROP TABLE t1; > diff --git a/sql/item.cc b/sql/item.cc > index c34d27fa63b..ee3e43b60ad 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -616,8 +616,9 @@ void Item::print_value(String *str) > str->append("NULL"); > else > { > - switch (result_type()) { > + switch (cmp_type()) { > case STRING_RESULT: > + case TIME_RESULT: okay, that fixes quoting. good. > append_unescaped(str, ptr->ptr(), ptr->length()); > break; > case DECIMAL_RESULT: > @@ -9433,17 +9433,26 @@ void Item_cache::store(Item *item) > > void Item_cache::print(String *str, enum_query_type query_type) > { > - if (value_cached) > + if ( example && ( example->type() == FUNC_ITEM ) && // There is a cached function item > + ( query_type & QT_ITEM_IDENT_SKIP_TABLE_NAMES ) ) // Caller is show-create-table 1. better test for QT_ITEM_CACHE_WRAPPER_SKIP_DETAILS 2. Why the test for FUNC_ITEM? I don't see any logical reason why skipping of "<cache>" should only apply to functions. and please watch the spacing around parentheses, try to use the same coding style as the rest of the code > { > - print_value(str); > - return; > + // Instead of "cache" or the cached value, print the function name > + example->print( str, query_type ); > } > - str->append(STRING_WITH_LEN("<cache>(")); > - if (example) > - example->print(str, query_type); > else > - Item::print(str, query_type); > - str->append(')'); > + { > + if (value_cached) Indentation. please, try to use the same coding style as the rest of the code > + { > + print_value(str); > + return; > + } It's not very logical that the value is not printed if one doesn't want "<cache>". I think this method could be like this: ================================== void Item_cache::print(String *str, enum_query_type query_type) { if (value_cached && !(query_type & QT_NO_DATA_EXPANSION)) { print_value(str); return; } if (!(query_type & QT_ITEM_CACHE_WRAPPER_SKIP_DETAILS)) str->append(STRING_WITH_LEN("<cache>(")); if (example) example->print(str, query_type); else Item::print(str, query_type); if (!(query_type & QT_ITEM_CACHE_WRAPPER_SKIP_DETAILS)) str->append(')'); } ================================== That is, print the value unless QT_NO_DATA_EXPANSION is specified. Print "<cache>" unless QT_ITEM_CACHE_WRAPPER_SKIP_DETAILS is specified. And Virtual_column_info::print() should also specify QT_NO_DATA_EXPANSION. > + str->append(STRING_WITH_LEN("<cache>(")); > + if (example) > + example->print(str, query_type); > + else > + Item::print(str, query_type); > + str->append(')'); > + } > } > > /** > diff --git a/sql/item.h b/sql/item.h > index 67640ce5f4d..222f8580cb4 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1653,6 +1653,7 @@ class Item: public Value_source, > { > return mark_unsupported_function(full_name(), arg, VCOL_IMPOSSIBLE); > } > + virtual bool clear_cached_function_value(void *arg) { return 0; } > virtual bool check_field_expression_processor(void *arg) { return 0; } > virtual bool check_func_default_processor(void *arg) { return 0; } > /* > @@ -5450,8 +5451,21 @@ class Item_cache: public Item_basic_constant, > } > bool check_vcol_func_processor(void *arg) > { > + if ( example ) > + { > + return example->check_vcol_func_processor( arg ); whitespace around parentheses and indentation > + } > return mark_unsupported_function("cache", arg, VCOL_IMPOSSIBLE); > } > + bool clear_cached_function_value( void *arg ) > + { > + if ( example && example->type() == Item::FUNC_ITEM ) > + { > + // Clear the cached function value to trigger reevaluation > + clear(); > + } > + return false; > + } Why did you need a dedicated clear_cached_function_value() processor? There's Item::cleanup() that should be called from fix_session_vcol_expr(). Oh, okay, I see two issues here. First, item->cleanup() does not recurse into function arguments. That's my bug, I should've used something like - vcol->expr->cleanup(); + vcol->expr->walk(&Item::cleanup_excluding_fields_processor, 0, 0); It has to be fixed, it's a genuine bug. A test case could be create table t1 (n varchar(100), cu varchar(100) default current_user(), uc varchar(100) default concat(current_user())); create definer=foo@localhost view v1 as select * from t1; insert v1 (n) values ('v1'); select * from t1; This is a slightly modified test from default_session.test file. Second, it's incorrect for Item_cache to use simply example->check_vcol_func_processor(arg); because caching requires re-fixing, even if the underlying example item doesn't. So, this should be something like bool check_vcol_func_processor(void *arg) { if ( example ) { Item::vcol_func_processor_result *res= (Item::vcol_func_processor_result*)arg; example->check_vcol_func_processor(arg); if (res->errors & VCOL_NOT_STRICTLY_DETERMINISTIC) res->errors|= VCOL_SESSION_FUNC; return false; } return mark_unsupported_function("cache", arg, VCOL_IMPOSSIBLE); } this sets VCOL_SESSION_FUNC flag, which will require item to be re-fixed for every statement. But doesn't set it for true constants. With these two fixes Item_cache::cleanup() { clear(); } should works just fine. > /** > Check if saved item has a non-NULL value. > Will cache value of saved item if not already done. > diff --git a/sql/table.cc b/sql/table.cc > index 3a08d1e49ea..9ae035e41aa 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -2802,9 +2802,9 @@ static bool fix_and_check_vcol_expr(THD *thd, TABLE *table, > > int error= func_expr->walk(&Item::check_vcol_func_processor, 0, &res); > if (error || (res.errors & VCOL_IMPOSSIBLE)) > - { // this can only happen if the frm was corrupted > + { this comment is still true, don't delete it. and there's no need to pass the type inside anymore. > my_error(ER_VIRTUAL_COLUMN_FUNCTION_IS_NOT_ALLOWED, MYF(0), res.name, > - "???", "?????"); > + vcol_type_name((enum_vcol_info_type) vcol_type), vcol->name.str); > DBUG_RETURN(1); > } > vcol->flags= res.errors; > @@ -7374,6 +7374,10 @@ int TABLE::update_virtual_fields(handler *h, enum_vcol_update_mode update_mode) > > if (update) > { > + // Clear any cached function values in the virtual field expression > + // to trigger their reevaluation > + vcol_info->expr->walk( &Item::clear_cached_function_value, 0, 0 ); // Always returns 0 There's no need to do that for every update_virtual_fields(), that is, no need to do it for every row, only needs to be done once per statement. And fix_session_vcol_expr() will do it for you if you implement Item_cache::cleanup() and set VCOL_SESSION_FUNC and fix fix_session_vcol_expr() to cleanup recursively. > + > int field_error __attribute__((unused)) = 0; > /* Compute the actual value of the virtual fields */ > if (vcol_info->expr->save_in_field(vf, 0)) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 4

Re: [Maria-developers] b7aa15458db: MDEV-11117 CHECK constraint fails on intermediate step of ALTER
by Sergei Golubchik 06 Apr '17

06 Apr '17

Hi, Jacob! On Apr 03, jacob.mathew(a)mariadb.com wrote: > revision-id: b7aa15458db95149af4f473a90009844a16e9db8 (mariadb-10.2.4-103-gb7aa15458db) > parent(s): 3c422e60bbee79bb636e65910c26ac193de70a84 > author: Jacob Mathew > committer: Jacob Mathew > timestamp: 2017-04-03 17:54:32 -0700 > message: > > MDEV-11117 CHECK constraint fails on intermediate step of ALTER > > Fixed the bug by failing the statement with an error message that explains > that an auto-increment column may not be used in an expression for a > check constraint, a default value or a generated column. > Added a test case. > > new file mode 100644 > index 00000000000..e192fd16677 > --- /dev/null > +++ b/mysql-test/t/check_constraint_autoinc.test > @@ -0,0 +1,7 @@ > +# > +# An auto-increment column may not be used in an expression for > +# a check constraint: MDEV-11117 > +# > +--echo # AUTO_INCREMENT IN VCOL > +-- error ER_AUTOINC_IN_VIRTUAL_COLUMN_FUNCTION > +create or replace table t1( c1 int auto_increment primary key, check( c1 > 0 or c1 is null ) ); you could've added this 2-line test to the existing file check_constraint.test No strong arguments either way, though. Pro: mysqltest is executed for every test file, it's faster to have less test files. Don't know if the difference is big, though. Contra: it's easier to see what has failed with one test per file. It doesn't matter now, but might matter in the future, when we'll have more automatic failure analysys. > diff --git a/sql/field.h b/sql/field.h > index 50dcb397616..4bd43551648 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -591,7 +591,8 @@ static inline const char *vcol_type_name(enum_vcol_info_type type) > #define VCOL_NON_DETERMINISTIC 2 > #define VCOL_SESSION_FUNC 4 /* uses session data, e.g. USER or DAYNAME */ > #define VCOL_TIME_FUNC 8 > -#define VCOL_IMPOSSIBLE 16 > +#define VCOL_AUTO_INC 16 > +#define VCOL_IMPOSSIBLE 32 > > #define VCOL_NOT_STRICTLY_DETERMINISTIC \ > (VCOL_NON_DETERMINISTIC | VCOL_TIME_FUNC | VCOL_SESSION_FUNC) > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 930cd2ceaa2..a78e97c2577 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7142,6 +7142,8 @@ ER_NO_EIS_FOR_FIELD > ER_WARN_AGGFUNC_DEPENDENCE > eng "Aggregate function '%-.192s)' of SELECT #%d belongs to SELECT #%d" > ukr "Агрегатна функція '%-.192s)' з SELECTу #%d належить до SELECTу #%d" > +ER_AUTOINC_IN_VIRTUAL_COLUMN_FUNCTION > + eng "Auto-increment column %`s may not be used in an expression for a check constraint, a default value or a generated column." First, please add error messages at the very end of the file. Second, there's an existing error ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED eng "Function or expression '%s' cannot be used in the %s clause of %`s" which is generally used like my_error(ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED, MYF(0), "NOW()", "CHECK", field->name); or, for subqueries my_error(ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED, MYF(0), "SELECT ...", "DEFAULT", field->name); It'd be best if you could use it. If not, at least rephrase your error message to use ... in the %s clause of %`s pattern, instead of enumerating all possible places where autoincrement is not allowed. > diff --git a/sql/table.cc b/sql/table.cc > index 6b15c06cb91..d1772933190 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -2811,6 +2811,19 @@ static bool fix_and_check_vcol_expr(THD *thd, TABLE *table, > "???", "?????"); > DBUG_RETURN(1); > } > + else if (res.errors & VCOL_AUTO_INC) > + { > + /* > + An auto_increment field may not be used in an expression for > + a check constraint, a default value or a generated column > + > + Note that this error condition is not detected during parsing > + of the statement because the field item does not have a field > + pointer at that time > + */ > + my_error(ER_AUTOINC_IN_VIRTUAL_COLUMN_FUNCTION, MYF(0), res.name); > + DBUG_RETURN(1); > + } > vcol->flags= res.errors; There is already a check for auto-increment in generated columns. I've added it in commit a418c9920047d5222a0d065343347312127b780f (see the change in item.cc). But the error message is somewhat confusing. And very confusing when applied to CHECK and DEFAULT. Yours is better, so please remove my check, no need to do it twice. Don't forget to re-run tests after that (at least vcol and gcol suites) as some error messages could change. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] mariadb-connector-c on mingw-w64 with pthreads
by Jeroen Ooms 06 Apr '17

06 Apr '17

For the language bindings for the R programming language (RMySQL) I am building mariadb-connector-c-2.3.2 on Windows using the mingw-w64 compiler. Our mingw-w64 compilers have been configured with pthreads support (same as msys2): - i686-493-posix-dwarf-rt_v3-s for win32 - x86_64-493-posix-seh-rt_v3-s for win64 Building libmariadbclient succeeds, however when I try to link the static library into the application I get these errors: C:/Rtools/mingw_64/lib/gcc/x86_64-w64-mingw32/lib/libpthread.a(libwinpthread_la-cond.o):cond.c:(.text+0x300): multiple definition of `pthread_cond_init' libmariadbclient.a(my_pthread.c.obj):my_pthread.c:(.text+0x0): first defined here Same error for 'pthread_cond_destroy', 'pthread_cond_wait', and 'pthread_cond_timedwait'. From the source code of my_pthread.c it seems that mariadb-connector-c is indeed redefining these functions on _WIN32. I guess this assumes MSVC which does not have native pthreads? Some questions: 1. What would be the correct way to configure/build mariadb-connector-c on mingw-w64 with native pthread support? 2. Is there a cmake option to disable threading alltogether? Thank you

2 1

[Maria-developers] Database corruption
by jerome brauge 05 Apr '17

05 Apr '17

Hello, We have a java tools to massively load data in a database (parallel load in different tables). With it, on bb-10.2-compatibility branch, I'm able to reproduce crash and sometimes database corruption: · on truncate (but delete works fine) · on parallel insert Crash seems occurred in innoDB code. See attached, logs of each case. Do you need more information ? Regards, Jérôme.

3 11

Re: [Maria-developers] JDBC getWarnings and stored procedure
by Daniel Black 05 Apr '17

05 Apr '17

On 03/04/17 20:21, jerome brauge wrote: > Hello, > > When we call a stored procedure in JDBC which emit warnings, > getWarnings() always returns null. > > I don’t really think that the problem is in MariaDB Connector J because > warning flag in the EOF packet is not set to true. > hasWarnings is set: https://github.com/MariaDB/mariadb-connector-j/blob/master/src/main/java/or… Its used here: https://github.com/MariaDB/mariadb-connector-j/blob/master/src/main/java/or… I suspect the bug is warningsCleared has a implicit default of false in the above class. As a workaround check, call reenableWarnings() before getWarnings(). > Mysql client behaves in the same way, by default it shows warnings count > issued by DML command but not for stored procedure. > > > > Is it a normal behavior ? > > > > Best regards, > > Jérôme. > > > > > > > > > > > > > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-developers > Post to : maria-developers(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-developers > More help : https://help.launchpad.net/ListHelp >

3 3

Re: [Maria-developers] [Commits] e446bf7: MDEV-12145: Wrong result (missing rows) on query with IN and EXISTS subqueries from InnoDB tables
by Igor Babaev 04 Apr '17

04 Apr '17

Varun, 1. The mdev # in the test case should be 12145 2. Could you please explain why do you need > @@ -9735,8 +9735,7 @@ make_join_select(JOIN *join,SQL_SELECT *select,COND *cond) > !(used_tables & tab->emb_sj_nest->sj_inner_tables)) > { > save_used_tables= used_tables; > - used_tables= join->const_table_map | OUTER_REF_TABLE_BIT | > - RAND_TABLE_BIT; > + used_tables= join->const_table_map | RAND_TABLE_BIT; > } ? I checked your test case on 5.5 only with ; > */ > if (tab == join->join_tab + join->top_join_tab_count - 1) > - current_map|= OUTER_REF_TABLE_BIT | RAND_TABLE_BIT; > + current_map|= RAND_TABLE_BIT; > used_tables|=current_map; > > if (tab->type == JT_REF && tab->quick && and it passed. If the first change is really needed you have to produce a test case where it is really critical. If this is just cosmetic change the comment should clearly say about it. 3. I wonder why the patch is applied to 10.2 if it's reported for 5.5-10.2. Regards, Igor. On 04/04/2017 05:55 AM, Varun wrote: > revision-id: e446bf7dc49b96a09e11199ad3e05fade6490279 (mariadb-10.2.3-396-ge446bf7) > parent(s): 6d417a0bad205a6bacfee10dbc46dd631b093e75 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-04-04 18:24:43 +0530 > message: > > MDEV-12145: Wrong result (missing rows) on query with IN and EXISTS subqueries from InnoDB tables > > Tables that are within SJ-Materialization nests cannot have their conditions referring to preceding non-const tables. But in this issue this was happening. > Fixed by not allowing used_tables to allow outer tables > > --- > mysql-test/r/subselect_sj2.result | 26 ++++++++++++++++++++++++++ > mysql-test/r/subselect_sj2_jcl6.result | 26 ++++++++++++++++++++++++++ > mysql-test/r/subselect_sj2_mat.result | 26 ++++++++++++++++++++++++++ > mysql-test/t/subselect_sj2.test | 24 ++++++++++++++++++++++++ > sql/sql_select.cc | 5 ++--- > 5 files changed, 104 insertions(+), 3 deletions(-) > > diff --git a/mysql-test/r/subselect_sj2.result b/mysql-test/r/subselect_sj2.result > index 948be57..14032ec 100644 > --- a/mysql-test/r/subselect_sj2.result > +++ b/mysql-test/r/subselect_sj2.result > @@ -1331,5 +1331,31 @@ WHERE a IN ( SELECT b FROM t2 LEFT JOIN v3 ON ( c = b ) ) ; > a pk b > DROP TABLE t1,t2,t3; > DROP VIEW v3; > +# > +# MDEV-11108: Assertion `uniq_tuple_length_arg <= table->file->max_key_length()' failed in SJ_TMP_TABLE::create_sj_weedout_tmp_table > +# > +CREATE TABLE t1 (f1 INT) ENGINE=InnoDB; > +INSERT INTO t1 VALUES (4),(6); > +CREATE TABLE t2 (i2 INT, KEY(i2)) ENGINE=InnoDB; > +INSERT INTO t2 VALUES (8),(7),(1); > +CREATE TABLE t3 (f3 INT, i3 INT, KEY(i3)) ENGINE=InnoDB; > +CREATE ALGORITHM=MERGE VIEW v3 AS SELECT * FROM t3; > +INSERT INTO t3 VALUES (8,0),(6,3),(2,8),(3,8),(1,6),(0,0),(1,0),(1,5); > +CREATE TABLE t4 (i4 INT, f4 INT, KEY(i4)) ENGINE=InnoDB; > +INSERT INTO t4 VALUES (0,0),(4,0),(0,2),(1,0),(2,1),(2,7),(6,3),(8,7),(8,1),(1,0),(7,2),(6,0),(8,1); > +SELECT * FROM t1 WHERE EXISTS ( SELECT * FROM t2, t3 WHERE i3 = i2 AND f1 IN ( SELECT f3 FROM v3, t4 WHERE i4 = f3 AND f4 <> f3 ) ); > +f1 > +6 > +ANALYZE TABLE t1,t2,t3,t4; > +Table Op Msg_type Msg_text > +test.t1 analyze status OK > +test.t2 analyze status OK > +test.t3 analyze status OK > +test.t4 analyze status OK > +SELECT * FROM t1 WHERE EXISTS ( SELECT * FROM t2, t3 WHERE i3 = i2 AND f1 IN ( SELECT f3 FROM v3, t4 WHERE i4 = f3 AND f4 <> f3 ) ); > +f1 > +6 > +DROP VIEW v3; > +DROP TABLE t1, t2, t3, t4; > # This must be the last in the file: > set optimizer_switch=@subselect_sj2_tmp; > diff --git a/mysql-test/r/subselect_sj2_jcl6.result b/mysql-test/r/subselect_sj2_jcl6.result > index 2955307..2aa0605 100644 > --- a/mysql-test/r/subselect_sj2_jcl6.result > +++ b/mysql-test/r/subselect_sj2_jcl6.result > @@ -1346,6 +1346,32 @@ WHERE a IN ( SELECT b FROM t2 LEFT JOIN v3 ON ( c = b ) ) ; > a pk b > DROP TABLE t1,t2,t3; > DROP VIEW v3; > +# > +# MDEV-11108: Assertion `uniq_tuple_length_arg <= table->file->max_key_length()' failed in SJ_TMP_TABLE::create_sj_weedout_tmp_table > +# > +CREATE TABLE t1 (f1 INT) ENGINE=InnoDB; > +INSERT INTO t1 VALUES (4),(6); > +CREATE TABLE t2 (i2 INT, KEY(i2)) ENGINE=InnoDB; > +INSERT INTO t2 VALUES (8),(7),(1); > +CREATE TABLE t3 (f3 INT, i3 INT, KEY(i3)) ENGINE=InnoDB; > +CREATE ALGORITHM=MERGE VIEW v3 AS SELECT * FROM t3; > +INSERT INTO t3 VALUES (8,0),(6,3),(2,8),(3,8),(1,6),(0,0),(1,0),(1,5); > +CREATE TABLE t4 (i4 INT, f4 INT, KEY(i4)) ENGINE=InnoDB; > +INSERT INTO t4 VALUES (0,0),(4,0),(0,2),(1,0),(2,1),(2,7),(6,3),(8,7),(8,1),(1,0),(7,2),(6,0),(8,1); > +SELECT * FROM t1 WHERE EXISTS ( SELECT * FROM t2, t3 WHERE i3 = i2 AND f1 IN ( SELECT f3 FROM v3, t4 WHERE i4 = f3 AND f4 <> f3 ) ); > +f1 > +6 > +ANALYZE TABLE t1,t2,t3,t4; > +Table Op Msg_type Msg_text > +test.t1 analyze status OK > +test.t2 analyze status OK > +test.t3 analyze status OK > +test.t4 analyze status OK > +SELECT * FROM t1 WHERE EXISTS ( SELECT * FROM t2, t3 WHERE i3 = i2 AND f1 IN ( SELECT f3 FROM v3, t4 WHERE i4 = f3 AND f4 <> f3 ) ); > +f1 > +6 > +DROP VIEW v3; > +DROP TABLE t1, t2, t3, t4; > # This must be the last in the file: > set optimizer_switch=@subselect_sj2_tmp; > # > diff --git a/mysql-test/r/subselect_sj2_mat.result b/mysql-test/r/subselect_sj2_mat.result > index a067422..1ea020a 100644 > --- a/mysql-test/r/subselect_sj2_mat.result > +++ b/mysql-test/r/subselect_sj2_mat.result > @@ -1333,6 +1333,32 @@ WHERE a IN ( SELECT b FROM t2 LEFT JOIN v3 ON ( c = b ) ) ; > a pk b > DROP TABLE t1,t2,t3; > DROP VIEW v3; > +# > +# MDEV-11108: Assertion `uniq_tuple_length_arg <= table->file->max_key_length()' failed in SJ_TMP_TABLE::create_sj_weedout_tmp_table > +# > +CREATE TABLE t1 (f1 INT) ENGINE=InnoDB; > +INSERT INTO t1 VALUES (4),(6); > +CREATE TABLE t2 (i2 INT, KEY(i2)) ENGINE=InnoDB; > +INSERT INTO t2 VALUES (8),(7),(1); > +CREATE TABLE t3 (f3 INT, i3 INT, KEY(i3)) ENGINE=InnoDB; > +CREATE ALGORITHM=MERGE VIEW v3 AS SELECT * FROM t3; > +INSERT INTO t3 VALUES (8,0),(6,3),(2,8),(3,8),(1,6),(0,0),(1,0),(1,5); > +CREATE TABLE t4 (i4 INT, f4 INT, KEY(i4)) ENGINE=InnoDB; > +INSERT INTO t4 VALUES (0,0),(4,0),(0,2),(1,0),(2,1),(2,7),(6,3),(8,7),(8,1),(1,0),(7,2),(6,0),(8,1); > +SELECT * FROM t1 WHERE EXISTS ( SELECT * FROM t2, t3 WHERE i3 = i2 AND f1 IN ( SELECT f3 FROM v3, t4 WHERE i4 = f3 AND f4 <> f3 ) ); > +f1 > +6 > +ANALYZE TABLE t1,t2,t3,t4; > +Table Op Msg_type Msg_text > +test.t1 analyze status OK > +test.t2 analyze status OK > +test.t3 analyze status OK > +test.t4 analyze status OK > +SELECT * FROM t1 WHERE EXISTS ( SELECT * FROM t2, t3 WHERE i3 = i2 AND f1 IN ( SELECT f3 FROM v3, t4 WHERE i4 = f3 AND f4 <> f3 ) ); > +f1 > +6 > +DROP VIEW v3; > +DROP TABLE t1, t2, t3, t4; > # This must be the last in the file: > set optimizer_switch=@subselect_sj2_tmp; > set optimizer_switch=default; > diff --git a/mysql-test/t/subselect_sj2.test b/mysql-test/t/subselect_sj2.test > index a948b08..5b745ca 100644 > --- a/mysql-test/t/subselect_sj2.test > +++ b/mysql-test/t/subselect_sj2.test > @@ -1464,5 +1464,29 @@ WHERE a IN ( SELECT b FROM t2 LEFT JOIN v3 ON ( c = b ) ) ; > DROP TABLE t1,t2,t3; > DROP VIEW v3; > > +--echo # > +--echo # MDEV-11108: Assertion `uniq_tuple_length_arg <= table->file->max_key_length()' failed in SJ_TMP_TABLE::create_sj_weedout_tmp_table > +--echo # > + > +CREATE TABLE t1 (f1 INT) ENGINE=InnoDB; > +INSERT INTO t1 VALUES (4),(6); > + > +CREATE TABLE t2 (i2 INT, KEY(i2)) ENGINE=InnoDB; > +INSERT INTO t2 VALUES (8),(7),(1); > + > +CREATE TABLE t3 (f3 INT, i3 INT, KEY(i3)) ENGINE=InnoDB; > +CREATE ALGORITHM=MERGE VIEW v3 AS SELECT * FROM t3; > +INSERT INTO t3 VALUES (8,0),(6,3),(2,8),(3,8),(1,6),(0,0),(1,0),(1,5); > + > +CREATE TABLE t4 (i4 INT, f4 INT, KEY(i4)) ENGINE=InnoDB; > +INSERT INTO t4 VALUES (0,0),(4,0),(0,2),(1,0),(2,1),(2,7),(6,3),(8,7),(8,1),(1,0),(7,2),(6,0),(8,1); > + > +SELECT * FROM t1 WHERE EXISTS ( SELECT * FROM t2, t3 WHERE i3 = i2 AND f1 IN ( SELECT f3 FROM v3, t4 WHERE i4 = f3 AND f4 <> f3 ) ); > +ANALYZE TABLE t1,t2,t3,t4; > +SELECT * FROM t1 WHERE EXISTS ( SELECT * FROM t2, t3 WHERE i3 = i2 AND f1 IN ( SELECT f3 FROM v3, t4 WHERE i4 = f3 AND f4 <> f3 ) ); > + > +DROP VIEW v3; > +DROP TABLE t1, t2, t3, t4; > + > --echo # This must be the last in the file: > set optimizer_switch=@subselect_sj2_tmp; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 151e341..135ad9a 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -9735,8 +9735,7 @@ make_join_select(JOIN *join,SQL_SELECT *select,COND *cond) > !(used_tables & tab->emb_sj_nest->sj_inner_tables)) > { > save_used_tables= used_tables; > - used_tables= join->const_table_map | OUTER_REF_TABLE_BIT | > - RAND_TABLE_BIT; > + used_tables= join->const_table_map | RAND_TABLE_BIT; > } > > /* > @@ -9744,7 +9743,7 @@ make_join_select(JOIN *join,SQL_SELECT *select,COND *cond) > It solve problem with select like SELECT * FROM t1 WHERE rand() > 0.5 > */ > if (tab == join->join_tab + join->top_join_tab_count - 1) > - current_map|= OUTER_REF_TABLE_BIT | RAND_TABLE_BIT; > + current_map|= RAND_TABLE_BIT; > used_tables|=current_map; > > if (tab->type == JT_REF && tab->quick && > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] MDEV-12238 Add Type_handler::Item_func_{plus|minus|mul|div|mod}_fix_length_and_dec()
by Alexander Barkov 04 Apr '17

04 Apr '17

Hello Alexey, Thanks for your review. Please find an updated version attached. On 03/19/2017 08:12 PM, Alexey Botchkov wrote: > Hi, Alexander. > > The patch is ok to push. > I only have two cosmetical recommendation. > 1. const Pair* find_pair() implementation belongs to sql_type.cc. A bit > too long for the .h file. Moved. > 2. I'd create the enum {NOT_COMMUTATIVE, COMMUTATIVE} and use it as the > parameter for the find_pair() and related functions. Makes code much > more understandable. As agreed on IRC, I created a new class Type_aggregator_commutative instead. Thanks! > > Best regards. > HF > > On Mon, Mar 13, 2017 at 3:41 PM, Alexander Barkov <bar(a)mariadb.org > <mailto:bar@mariadb.org>> wrote: > > Hello Alexey, > > Please review a patch for MDEV-12238. > > Thanks! > >

1 1

[Maria-developers] Please review a patch removing LEX::text_string_is_7bit
by Alexander Barkov 04 Apr '17

04 Apr '17

Hello Sergei, Can you please review a patch for bb-10.2-ext fixing the problem that in case of possible changes in look-ahead in rules using TEXT_STRING or NCHAR_STRING, Lex->text_string_is_7bit can get out of sync with $1/$2/$3, etc. Thanks!

2 2

[Maria-developers] JDBC getWarnings and stored procedure
by jerome brauge 03 Apr '17

03 Apr '17

Hello, When we call a stored procedure in JDBC which emit warnings, getWarnings() always returns null. I don't really think that the problem is in MariaDB Connector J because warning flag in the EOF packet is not set to true. Mysql client behaves in the same way, by default it shows warnings count issued by DML command but not for stored procedure. Is it a normal behavior ? Best regards, Jérôme.

1 0

[Maria-developers] Please review a patch for Mdev-12019
by Sachin Setiya 03 Apr '17

03 Apr '17

Hi Monty, Please review a patch for Mdev-12019. -- Regards Sachin Setiya Software Engineer at MariaDB

1 0

[Maria-developers] Errors in building debs
by Sergey Petrunia 01 Apr '17

01 Apr '17

Hi Vicențiu, I'm looking at the last build in bb-10.2-mariarocks http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-x86/builds/35… and I see this: + DH_BUILD_DDEBS=1 + echo + debian/autobake-deb.sh debian/autobake-deb.sh: line 80: [: i686: integer expression expected debian/autobake-deb.sh: line 80: [i686: command not found and then it eventually fails with this error: cp: cannot stat ‘debian/tmp/etc/mysql/conf.d/rocksdb_se.cnf’: No such file or I assume this is caused by your recent changes? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] [Commits] 96403c5e124: Replace some unsigned integers with size_t to avoid type mismatch.
by Sergey Petrunia 31 Mar '17

31 Mar '17

Hi Marko, Please find below the review for patch #1 of the "size_t patches": my comments are marked with 'psergey:' On Thu, Mar 30, 2017 at 02:39:50PM +0300, marko.makela(a)mariadb.com wrote: > commit 96403c5e1240342948cb88600ece14c0c8b72dbc > Author: Marko Mäkelä <marko.makela(a)mariadb.com> > Date: Sat Mar 11 22:03:31 2017 +0200 > > Replace some unsigned integers with size_t to avoid type mismatch. > > diff --git a/sql/key.cc b/sql/key.cc > index bb10e90..9e8693f 100644 > --- a/sql/key.cc > +++ b/sql/key.cc > @@ -1,4 +1,5 @@ > /* Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved. > + Copyright (c) 2017, MariaDB Corporation. > > This program is free software; you can redistribute it and/or modify > it under the terms of the GNU General Public License as published by > @@ -392,7 +393,7 @@ void field_unpack(String *to, Field *field, const uchar *rec, uint max_length, > tmp.length(charpos); > } > if (max_length < field->pack_length()) > - tmp.length(min(tmp.length(),max_length)); > + tmp.length(min(tmp.length(),static_cast<size_t>(max_length))); psergey: This is not necessary as tmp.length() returns a 'size_t', but it's still ok to have I guess. > ErrConvString err(&tmp); > to->append(err.ptr()); > } ... > diff --git a/sql/spatial.cc b/sql/spatial.cc > index 7c9d8bb..399a968 100644 > --- a/sql/spatial.cc > +++ b/sql/spatial.cc > @@ -2278,7 +2278,7 @@ int Gis_multi_line_string::geometry_n(uint32 num, String *result) const > break; > data+= length; > } > - return result->append(data, length, (uint32) 0); > + return result->append(data, length, static_cast<size_t>(0)); psergey: isn't that excessive, can one just write "0" there? > } > > > @@ -2710,8 +2710,9 @@ int Gis_multi_polygon::geometry_n(uint32 num, String *result) const > } while (--num); > if (no_data(data, 0)) // We must check last segment > return 1; > - return result->append(start_of_polygon, (uint32) (data - start_of_polygon), > - (uint32) 0); > + return result->append(start_of_polygon, > + static_cast<size_t>(data - start_of_polygon), > + static_cast<size_t>(0)); psergey: isn't that excessive, can one just write "0" there? > } > > ... > diff --git a/sql/sql_string.h b/sql/sql_string.h > index 18f5f4c..8248f6d 100644 > --- a/sql/sql_string.h > +++ b/sql/sql_string.h > @@ -531,16 +531,19 @@ class String > { > Ptr[str_length++] = c; > } > - void q_append2b(const uint32 n) > + void q_append2b(size_t n) psergey: this just appends the two lower bytes. Is this change really needed? > { > int2store(Ptr + str_length, n); > str_length += 2; > } > - void q_append(const uint32 n) > + void q_append(size_t n) > { psergey: This always appends 4 bytes. It is a little bit scary that this function will optionally discard the upper 4 bytes of the passed values. As far as I understand its usage, this is for writing the data that should always be 4 bytes, regardless of the sizeof(size_t) of the platform we're on. > int4store(Ptr + str_length, n); > str_length += 4; > } > +#if SIZEOF_SIZE_T > 4 > + void q_append(uint32 n) { q_append(static_cast<size_t>(n)); } > +#endif > void q_append(double d) > { > float8store(Ptr + str_length, d); ... > @@ -669,11 +672,11 @@ class String > { > return !sortcmp(this, other, cs); > } > - void q_net_store_length(ulonglong length) > + void q_net_store_length(size_t length) psergey: I dont think this should vary depending on sizeof(size_t). This function is used to construct data to be sent over network, check sql/protocol.cc, net_send_ok(). Apparently network data should have the same number of bits always. > { > DBUG_ASSERT(Alloced_length >= (str_length + net_length_size(length))); > char *pos= (char *) net_store_length((uchar *)(Ptr + str_length), length); > - str_length= uint32(pos - Ptr); > + str_length= static_cast<size_t>(pos - Ptr); > } > void q_net_store_data(const uchar *from, size_t length) > { BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] bb-10.2-compatibility - processlist
by jerome brauge 31 Mar '17

31 Mar '17

Hello Alexander, Here is a patch to add information provided by JDBC (setClientInfo) in information_schema.processlist (ApplicationName, ClientUser, ClientHostname). Do you think that it can be integrate ? Regards, Jérôme.

2 8

[Maria-developers] Please review: MDEV-12416 OOM in create_virtual_tmp_table() makes the server crash
by Alexander Barkov 31 Mar '17

31 Mar '17

Hello Alexey, can you please review a patch for MDEV-12416? Thanks!

1 0

[Maria-developers] Please review MDEV-12415 Remove sp_name::m_qname
by Alexander Barkov 31 Mar '17

31 Mar '17

Hello Sanja, can you please review a patch for MDEV-12415? Thanks!

1 0

[Maria-developers] bb-10.2-compatibility
by jerome brauge 30 Mar '17

30 Mar '17

Hello Alexander, On oracle when an error occurred inside a stored procedure, the error message contains the stack trace of calling procedures. It's very useful when you have tons of procedures. I wrote this little patch to do something like that by adding the stack trace as "notes". To show the correct lines number (according to result of show create procedure) , I had to correct yylineno in sql_les.cc. But now, some tests failed because the line number of errors have changed ! What is your opinion on this patch ? Regards, Jérôme.

2 3

Re: [Maria-developers] bb-10.2-compatibility
by jerome brauge 29 Mar '17

29 Mar '17

Hi Alexander, Here is the patch that corrects the line number of errors. Only tests from test suite for oracle seems to be affected. Row number are correct when test is recorded with "mysql-test-run.pl -record", but wrong with mysql or mysqltest (I don't understand why !) Regard, Jérôme. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : mardi 28 mars 2017 07:42 > À : jerome brauge > Objet : Re: bb-10.2-compatibility > > Hi Jerome, > > > On 03/28/2017 08:45 AM, jerome brauge wrote: > > Alexander, > > Due to this , when i record test for sp-cursor-rowtype.test, I have: > > > > Warning 1265 Data truncated for column 'b' at row 2 > > > > instead of > > > > Warning 1265 Data truncated for column 'b' at row 1 > > > > for test > > # Fetching a cursor into a cursor%ROWTYPE variable, with truncation > > > > I do a test with an older project to see if it's work better. > > It's Ok to record with a wrong "at row XXX" for now, as this bug is not specific > to bb-10.2-compatibility. > > When this problem is fixed in parent branches (10.0, 10.1, 10.2), we'll merge > into bb-10.2-compatibility and fix wrong "at row XXX" > results. > > > Btw, it's very likely that bb-10.2-compatibility will be merged to 10.3 soon, in > the beginning of April. > > > > > > > Jérôme. > > > >> -----Message d'origine----- > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mardi 28 mars > >> 2017 06:10 À : jerome brauge Objet : Re: bb-10.2-compatibility > >> > >> Hello Jerome, > >> > >> On 03/28/2017 12:03 AM, jerome brauge wrote: > >>> Hello Alexander, > >>> Now that my linux build is ok, I have something strange with > >>> warnings on cursors (detected in sp-cursor-rowtype.test) > >>> > >>> The following code > >>> > >>> set sql_mode=oracle; > >>> use test > >>> CREATE or replace TABLE wjbt1 (a INT, b VARCHAR(10)); CREATE or > >>> replace TABLE wjbt2 (a INT, b INT); INSERT INTO wjbt1 VALUES > >>> (10,'11x'); INSERT INTO wjbt1 VALUES (10,'12x'); delimiter $$ CREATE > >>> or replace PROCEDURE pwjb1() AS CURSOR cur1 IS SELECT b FROM > wjbt1; > >>> BEGIN DECLARE res int; BEGIN OPEN cur1; FETCH cur1 INTO res; FETCH > >>> cur1 INTO res; CLOSE cur1; END; END; $$ delimiter ; call pwjb1(); > >>> show warnings; > >>> > >>> returns: > >>> +---------+------+------------------------------------------+ > >>> | Level | Code | Message | > >>> +---------+------+------------------------------------------+ > >>> | Warning | 1265 | Data truncated for column 'res' at row 3 | > >>> | Warning > >>> | | 1265 | Data truncated for column 'res' at row 3 | > >>> +---------+------+------------------------------------------+ > >>> > >>> I done this test on Linux and Windows with the same behavious. > >>> I don't understand why. Do you have an idea ? > >> > >> > >> Truncation seems to be correct, > >> because the procedure converts '11x' and '12x' to INT. > >> > >> But 'at row 3' seems to be wrong. > >> > >> I reported the problem as: > >> > >> https://jira.mariadb.org/browse/MDEV-12377 > >> > >> Looks like an old bug. MariaDB-10.0 exposes the same behavior. > >> > >> > >>> > >>> Regards, > >>> Jérôme. > >>>

2 1

Re: [Maria-developers] CREATE SEQUENCE is coming
by Daniel Black 28 Mar '17

28 Mar '17

On 28/03/17 07:31, Vladislav Vaintroub wrote: > > > On 3/27/2017 6:15 AM, Daniel Black wrote: >> On 26/03/17 22:11, Michael Widenius wrote: >>> There is still a lot of work to fix edge cases, but in the current >>> implementation most major things seams to work... >>> >>> Comments, suggestions or questions? >> NICE! >> >> How will this value be exposed to getGeneratedKeys in >> mariadb-connector-j? > I do not think it needs to. Statement.getGeneratedKeys returns generated > auto-increment values on insert, the same ulonglong that is in OK > packet, the same thing that LAST_INSERT_ID() is returning. > no sequences involved Ah, ok. I was miss understanding how Postgres does its sequences. I found this quick summary from 2011 (date in http response header) http://www.neilconway.org/docs/sequences/ Oracle ref: http://docs.oracle.com/cd/B28359_01/server.111/b28286/pseudocolumns002.htm#… uses sequence.CURRVAL, sequence.NEXTVAL https://www.postgresql.org/docs/9.6/static/functions-sequence.html As a note itseams PREVIOUS VALUE FOR seems to be a DB2 construct. Postgres uses CURRVAL(sequence) which to me sounds more sound. I guess MDEV-10422 INSERT RETURNING is the alternative. All options work. Thanks for writing this. So it seems they show up as full table where querying them is pretty harmless? I suspect if things like pt-table-sync start trying to modify them ugly things could happen.

1 0

Re: [Maria-developers] CREATE SEQUENCE is coming
by Daniel Black 27 Mar '17

27 Mar '17

On 26/03/17 22:11, Michael Widenius wrote: > There is still a lot of work to fix edge cases, but in the current > implementation most major things seams to work... > > Comments, suggestions or questions? NICE! How will this value be exposed to getGeneratedKeys in mariadb-connector-j?

2 1

[Maria-developers] [GSOC] ColumnStore: Add full support for DECIMAL
by Даниил Медведев 27 Mar '17

27 Mar '17

Hello, I took part in GSoC last year, and closed one of MariaDB tasks. Now I want to continue programming with MariaDB developers and solve a new task. I considered some of proposed tasks, and was interested in "ColumnStore: Add full support for DECIMAL". I think this task is doable in the given time and I might solve it. P.S. My github link: https://github.com/medvdanil Regards, Daniil Medvedev

2 5

Re: [Maria-developers] 99d2448: MDEV-9255 Add generation_expression to information_schema.columns.
by Sergei Golubchik 27 Mar '17

27 Mar '17

Hi, Alexey! On Mar 27, Alexey Botchkov wrote: > revision-id: 99d24484e474ce2f52fe58fc1c053f0aa186f094 (mariadb-10.2.4-84-g99d2448) > parent(s): ad7da60deda2cf97e60118a58f50f648abde6448 > committer: Alexey Botchkov > timestamp: 2017-03-27 17:44:20 +0400 > message: > > MDEV-9255 Add generation_expression to information_schema.columns. > > Added IS_GENERATED and GENERATION_EXPRESSION columns required by > the SQL standard. ok to push! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Query on MDEC-7502
by Vishwanath Bharadwaj 27 Mar '17

27 Mar '17

Hello Mr. Sergei, What I propose is rather than dumping the table into a single bin log. What your idea proposes is we slide through the table based on primary key. This becomes a bit of a problem if there is some error on the receiver side. So we create 2 logs. We cut the chunk from one log and send it through the RB logging. This way we need not have different replication for same data. If some error occurs we can resend the file from the data that is dumped into bin logger. I hope I'm able to clear you query. If there is even a slightest doubt in my idea please feel free to contact me and suggest changes as I am still in my learning stage. Thank you:)

2 2

[Maria-developers] [GSoC2017]Project prosoal [MDEV-12296] Cassandra Storage Engine V2
by Neveralso 27 Mar '17

27 Mar '17

Hello. My name is Fangshuo Jiang. I am a first-year master student of computer technology, in Beijing University of Posts and Telecommunications. I would like to write a prososal and work on MDEV-12296:Cassandra Storage Engine V2. I was an intern at Tencent Company as a C++ back-end development engineer. I am familiar with C++ network programming, with some C++ projects holding on my GitHub. https://github.com/neveralso Look forward to more advice. Thank you. Best wishes. Fangshuo Jiang neveralso(a)gmail.com

2 1

Re: [Maria-developers] Gsoc project
by Vishwanath Bharadwaj 26 Mar '17

26 Mar '17

Hi hrishikesh, I guess we've to wait for someone who is authorized to make these decisions reply to your question. Ping Sergei Golubchik if you want to submit a proposal. Thank you:) On Mar 26, 2017 23:55, "Hrishikesh Paramane" <paramane1995hrishi(a)gmail.com> wrote: > MDEV-8100 > > > On Sun, Mar 26, 2017 at 11:54 PM, Vishwanath Bharadwaj < > bharadwajpullabhatla(a)gmail.com> wrote: > >> Hi hrishikesh, >> >> It would be helpful if you could provide us the project ticket number(Ex: >> MDEV-7502) >> >> Thank you:) >> >> On Mar 26, 2017 23:44, "Vishwanath Bharadwaj" < >> bharadwajpullabhatla(a)gmail.com> wrote: >> >>> Hi Hrishikesh, >>> >>> It would be helpful if you could provide us the project ticket number. >>> >>> Thank you:) >>> >>> On Mar 26, 2017 11:35 PM, "Hrishikesh Paramane" < >>> paramane1995hrishi(a)gmail.com> wrote: >>> >>>> Is Table function project removed from Google Summer of Code? >>>> >>>> _______________________________________________ >>>> Mailing list: https://launchpad.net/~maria-developers >>>> Post to : maria-developers(a)lists.launchpad.net >>>> Unsubscribe : https://launchpad.net/~maria-developers >>>> More help : https://help.launchpad.net/ListHelp >>>> >>>> >

1 0

[Maria-developers] Gsoc project
by Hrishikesh Paramane 26 Mar '17

26 Mar '17

Is Table function project removed from Google Summer of Code?

3 3

[Maria-developers] GSOC 2017 ideas
by Charles Cui 25 Mar '17

25 Mar '17

Hi guys, This is Charles Cui from Tsinghua University, China. I am pretty interested in the ideas listed in GSOC 2017 page on Maria DB, especially the idea "Cassandra Storage Engine V2". I have some experiences in Cassandra and MySQL, but quite new to Maria DB. I am wondering whether I can join this project and work with you guys? I speak fluent English and available during the whole summer. In case that this project is allocated to other people, I am also open to pick other projects in Maria DB. Let me know if you have questions or feedbacks. Thanks, Charles.

3 4

[Maria-developers] MariaDB Developers (Un)Conference
by Ian Gilfillan 24 Mar '17

24 Mar '17

Hi The schedule for the MariaDB Developers (Un)Conference (to be held in New York, April 9 and 10) is being fleshed out, so this is a last call to add or vote for any sessions. You can update the spreadsheet directly: https://docs.google.com/spreadsheets/d/1mUFk5gd0fd6amjxGhb7hG1xRHQdh-9qxavp… Please remember to register on the Meetup page if you're coming - our hosts BNY Mellon need a list of everyone attending in advance: https://www.meetup.com/MariaDB-Developers/events/234702611/ The MySQL New York Meetup group is also hosting an event on the evening of April 10 - see the details at: https://www.meetup.com/EffectiveMySQL/ Announcement: https://mariadb.org/2017-developers-meetup-new-york/

1 0

[Maria-developers] Why We can set binlog_format=mixed for session ?
by Sachin Setia 23 Mar '17

23 Mar '17

Hi Monty! When flashback is on why we can set binlog_format = Mixed/ statement for session scope, What is use of this and why this is allowed ? Regards sachin

1 0

[Maria-developers] Please review MDEV-12338 Split Item_type_holder::get_real_type() into virtual Item::real_type_handler()
by Alexander Barkov 23 Mar '17

23 Mar '17

Hi Vicențiu, can you please review a patch for MDEV-12338? Thanks!

1 0

Re: [Maria-developers] [Commits] 8ac272ab2c4: MDEV-12253: Buffer pool blocks are accessed after they have been freed
by Marko Mäkelä 23 Mar '17

23 Mar '17

On Mon, Mar 20, 2017 at 3:59 PM, jan <jan.lindstrom(a)mariadb.com> wrote: > diff --git a/mysql-test/suite/encryption/include/innodb-util.pl > b/mysql-test/suite/encryption/include/innodb-util.pl > index 241545dac18..5a7295682dd 100644 > --- a/mysql-test/suite/encryption/include/innodb-util.pl > +++ b/mysql-test/suite/encryption/include/innodb-util.pl > @@ -4,8 +4,11 @@ > # All the tables must be in the same database, you can call it like so: > # ib_backup_tablespaces("test", "t1", "blah", ...). > > +use strict; > +use warnings; > use File::Copy; > use File::Spec; > +use Fcntl qw(:DEFAULT :seek); > > sub ib_normalize_path { > my ($path) = @_; > @@ -124,3 +127,32 @@ sub ib_restore_ibd_files { > ib_restore_ibd_file($tmpd, $datadir, $db, $table); > } > } > + > +sub ib_corrupt_tablespace { > + my ($db, $table) = @_; > + my $datadir = $ENV{'MYSQLD_DATADIR'}; > + my $page_size = $ENV{'INNODB_PAGE_SIZE'}; > + my $ibd_file = sprintf("%s%s/%s.ibd", $datadir,$db,$table); > The sprintf() function and the separate parameters $db, $table look like obfuscation to me. Why not just write "$ENV{MYSQLD_DATADIR}/test/t1.ibd" when invoking this function in the .test file? > + print "file: $ibd_file $page_size\n"; > Why do you want to pollute the output with the page size? That would require rdiff files for different page sizes. > +# We try to corrupt FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION field > + my $len; > + > + sysopen IBD_FILE, $ibd_file, O_RDWR || die "Unable to open $ibd_file"; > + > + while ($len = sysread(IBD_FILE, $_, $page_size)) > + { > + sysseek(IBD_FILE, -$len, SEEK_CUR); > + substr($_, 26, 8) = pack("N",0xdeadbeef); > + print "Corrupted page with $chunk\n"; > + syswrite(IBD_FILE, $_, $len); > + } > + close IBD_FILE; > +} > Where is the variable $chunk defined? Why does this script execute issue output to stdout for every iteration of the loop? That would seem to make the test depend on the number of pages in the file, and thus cause result differences when using a different innodb_page_size. If the sysread() function happens to be interrupted by a signal, it could return a different amount than 0 or $page_size bytes. Thus, the operation of the loop would be nondeterministic. I would rather use a simple copying loop with a regular expression replacement, instead of this in-place modification. I would just do something like this: --exec perl -ei 's|(.{26}).{8}(.$ENV{INNODB_PAGE_SIZE})|"$1".pack("N",0xdeadbeef),"$2"|seg' $MYSQLD_DATADIR/test/t1.ibd Shorter and faster, and using fewer system calls. > diff --git a/mysql-test/suite/encryption/r/innodb-force-corrupt.result > b/mysql-test/suite/encryption/r/innodb-force-corrupt.result > new file mode 100644 > index 00000000000..bba2ec17bbc > --- /dev/null > +++ b/mysql-test/suite/encryption/r/innodb-force-corrupt.result > @@ -0,0 +1,38 @@ > +call mtr.add_suppression("InnoDB: Database page corruption on disk or a > failed.*"); > +call mtr.add_suppression("InnoDB: Corruption: Block in space_id .* in > file .* corrupted."); > +call mtr.add_suppression("InnoDB: Based on page type .*"); > +call mtr.add_suppression("InnoDB: Unable to read tablespace .* page no .* > into the buffer pool after .* attempts"); > +CALL mtr.add_suppression("InnoDB: Error: Unable to read tablespace .* > page no .* into the buffer pool after 100 attempts"); > +CALL mtr.add_suppression("InnoDB: Database page corruption on disk or a > failed"); > +CALL mtr.add_suppression("InnoDB: Space .* file test/t1 read of page .*"); > +CALL mtr.add_suppression("InnoDB: You may have to recover from a > backup."); > +CALL mtr.add_suppression("InnoDB: It is also possible that your > operatingsystem has corrupted its own file cache."); > +CALL mtr.add_suppression("InnoDB: and rebooting your computer removes the > error."); > +CALL mtr.add_suppression("InnoDB: If the corrupt page is an index page > you can also try to"); > +CALL mtr.add_suppression("InnoDB: fix the corruption by dumping, > dropping, and reimporting"); > +CALL mtr.add_suppression("InnoDB: the corrupt table. You can use CHECK"); > +CALL mtr.add_suppression("InnoDB: TABLE to scan your table for > corruption."); > +CALL mtr.add_suppression("InnoDB: See also .* about forcing recovery."); > +CALL mtr.add_suppression("InnoDB: However key management plugin or used > key_version 3221605118 is not found or used encryption algorithm or method > does not match."); > +CALL mtr.add_suppression("Marking tablespace as missing. You may drop > this table or install correct key management plugin and key file."); > +CALL mtr.add_suppression("InnoDB: Block in space_id .* in file .* > encrypted."); > Are all these suppressions really required? (The same problem is with multiple tests. +call mtr.add_suppression("mysqld: File .* not found .*"); > The suppression is looking for a substring by default (no implied $ at the end of the regular expression). The .* at the end of the regexp is clutter. > +call mtr.add_suppression(".*"); > This catch-all is absolutely not acceptable. > +--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--shutdown_server > +--source include/wait_until_disconnected.inc > Do not use the low-level commands directly. Use restart_mysqld.inc. See the encryption test cleanup that I did in 10.2 in the merge: https://github.com/MariaDB/server/commit/2d96d13ecdcab4ce16c0d8976753d8aa50… > ++--write_file $MYSQLTEST_VARDIR/keys1.txt > +1;770A8A65DA156D24EE2A093277530142 > +4;770A8A65DA156D24EE2A093277530143 > +EOF > This file can be created already before the shutdown (or restart). Why not use key files in std_data? What is the benefit of creating files with identical static content in multiple --suite=encryption tests? +--exec echo "restart:--innodb-encrypt-tables --innodb-stats-persistent > --plugin-load-add=file_key_management.so --file-key-management > --file-key-management-filename=$MYSQLTEST_VARDIR/keys1.txt" > > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--enable_reconnect > +--source include/wait_until_connected_again.inc > This is missing disable_reconnect, which is part of restart_mysqld.inc. > +--exec echo "restart:--innodb-encrypt-tables --innodb-stats-persistent > --plugin-load-add=file_key_management.so --file-key-management > --file-key-management-filename=$MYSQLTEST_VARDIR/keys2.txt" > > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--enable_reconnect > +--source include/wait_until_connected_again.inc > +--source include/restart_mysqld.inc > What is the purpose of restarting the server twice here? > +--error 1296 > +select count(*) from t1 FORCE INDEX (test) where b like 'secret%'; > +--error 1296 > +select count(*) from t2 FORCE INDEX (test) where b like 'secret%'; > +select count(*) from t3 FORCE INDEX (test) where b like 'secret%'; > Do not use numeric error codes. Refer to include/mysqld_ername.h in the build directory for the symbolic codes. 1296 is ER_GET_ERRMSG. +--remove_file $MYSQLTEST_VARDIR/keys1.txt > +--remove_file $MYSQLTEST_VARDIR/keys2.txt > + > +--write_file $MYSQLTEST_VARDIR/keys1.txt > +1;770A8A65DA156D24EE2A093277530142 > +4;770A8A65DA156D24EE2A093277530143 > +EOF > Why remove and rewrite the file with the identical content? Similar steps are being done in multiple encryption tests. Can some of the repetition be avoided? Or could we test multiple operations with one server restart? Instead of having one test to check if CHECK TABLE crashes when the keys are incorrect or not available, another to check if ALTER TABLE crashes, and so on, just restart the server once and then execute all the commands that are to be tested. That would make --suite=encryption run faster too. > +--exec echo "restart:--innodb-encrypt-tables --innodb-stats-persistent > --plugin-load-add=file_key_management.so --file-key-management > --file-key-management-filename=$MYSQLTEST_VARDIR/keys1.txt" > > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +drop table t1,t2,t3; > + > +--remove_file $MYSQLTEST_VARDIR/keys1.txt > The server was left running, tied to keys1.txt. Shouldn’t we rather restart the server at the end of the test, to have the same configuration as it was at the start of the test? > diff --git a/mysql-test/suite/encryption/t/innodb-force-corrupt.test > b/mysql-test/suite/encryption/t/innodb-force-corrupt.test > new file mode 100644 > index 00000000000..3127b8f4006 > --- /dev/null > +++ b/mysql-test/suite/encryption/t/innodb-force-corrupt.test > @@ -0,0 +1,122 @@ > +# > +# MDEV-11759: Encryption code in MariaDB 10.1/10.2 causes compatibility > problems > +# > + > +-- source include/have_innodb.inc > +-- source include/have_file_key_management_plugin.inc > +# Don't test under valgrind, memory leaks will occur > +source include/not_valgrind.inc; > Which memory leaks? Please run this with --valgrind and show. > +# Avoid CrashReporter popup on Mac > +source include/not_crashrep.inc; > Why? The test should not crash. > +# Don't test under embedded > +source include/not_embedded.inc; > +# Require InnoDB > +source include/have_innodb.inc; > I would omit this kind of redundant comments. I’d add a comment to explain that embedded server tests do not support restarting. > +# Test could open crash reporter on Windows > +# if compiler set up > +source include/not_windows.inc; > This is totally unacceptable. We must test under Windows on Buildbot. > +call mtr.add_suppression("InnoDB: Database page corruption on disk or a > failed.*"); > +call mtr.add_suppression("InnoDB: Corruption: Block in space_id .* in > file .* corrupted."); > +call mtr.add_suppression("InnoDB: Based on page type .*"); > +call mtr.add_suppression("InnoDB: Unable to read tablespace .* page no .* > into the buffer pool after .* attempts"); > +CALL mtr.add_suppression("InnoDB: Error: Unable to read tablespace .* > page no .* into the buffer pool after 100 attempts"); > +CALL mtr.add_suppression("InnoDB: Database page corruption on disk or a > failed"); > +CALL mtr.add_suppression("InnoDB: Space .* file test/t1 read of page .*"); > +CALL mtr.add_suppression("InnoDB: You may have to recover from a > backup."); > +CALL mtr.add_suppression("InnoDB: It is also possible that your > operatingsystem has corrupted its own file cache."); > +CALL mtr.add_suppression("InnoDB: and rebooting your computer removes the > error."); > +CALL mtr.add_suppression("InnoDB: If the corrupt page is an index page > you can also try to"); > +CALL mtr.add_suppression("InnoDB: fix the corruption by dumping, > dropping, and reimporting"); > +CALL mtr.add_suppression("InnoDB: the corrupt table. You can use CHECK"); > +CALL mtr.add_suppression("InnoDB: TABLE to scan your table for > corruption."); > +CALL mtr.add_suppression("InnoDB: See also .* about forcing recovery."); > +CALL mtr.add_suppression("InnoDB: However key management plugin or used > key_version 3221605118 is not found or used encryption algorithm or method > does not match."); > +CALL mtr.add_suppression("Marking tablespace as missing. You may drop > this table or install correct key management plugin and key file."); > +CALL mtr.add_suppression("InnoDB: Block in space_id .* in file .* > encrypted."); > Are all these suppressions really necessary? Some of the first 4 suppressions are matching same strings as some of the later suppressions. > +let $MYSQLD_DATADIR=`select @@datadir`; > +let INNODB_PAGE_SIZE=`select @@innodb_page_size`; > +let MYSQLD_DATADIR=`select @@datadir`; > Why is MYSQLD_DATADIR assigned both to an mtr internal variable and an environment variable? The latter should suffice. > +--source include/shutdown_mysqld.inc > +--source include/wait_until_disconnected.inc > The wait_until_disconnected.inc is redundant and should be removed. > +--disable_result_log > +--error 1296 > +SELECT * FROM t1; > +# > +# There is change that index is marked as corrupted here > +# > +--error 0,1712 > +SELECT * FROM t2; > chance != change. Why would t2 sometimes be not corrupted? In which way does it fail if the 0, is removed? Use symbolic error codes, not numeric ones. > +--error 1296 > +SELECT * FROM t3; > +--enable_result_log > + > +--source include/shutdown_mysqld.inc > +--source include/wait_until_disconnected.inc > Again, the wait_until_disconnected.inc should be removed. > + > +--echo # Restore the original t1.ibd > +--remove_file $MYSQLD_DATADIR/test/t1.ibd > +--move_file $MYSQLD_DATADIR/test/t1.ibd.backup > $MYSQLD_DATADIR/test/t1.ibd > +--move_file $MYSQLD_DATADIR/test/t2.ibd.backup > $MYSQLD_DATADIR/test/t2.ibd > +--move_file $MYSQLD_DATADIR/test/t3.ibd.backup > $MYSQLD_DATADIR/test/t3.ibd > + > +--source include/start_mysqld.inc > + > +DROP TABLE t1,t2,t3; > diff --git a/mysql-test/suite/encryption/t/innodb-redo-badkey.test b/mysql-test/suite/encryption/t/innodb-redo-badkey.test > new file mode 100644 > index 00000000000..34ffeb1b59a > --- /dev/null > +++ b/mysql-test/suite/encryption/t/innodb-redo-badkey.test > @@ -0,0 +1,121 @@ > +-- source include/have_innodb.inc > +-- source include/have_file_key_management_plugin.inc > +# embedded does not support restart > +-- source include/not_embedded.inc > +# Don't test this under valgrind, memory leaks will occur. > +--source include/not_valgrind.inc > +# Avoid CrashReporter popup on Mac > +--source include/not_crashrep.inc > +--source include/not_windows.inc > The not_windows.inc is not acceptable. I would say that crashes or memory leaks are not acceptable either. > +--source ../../suite/innodb/include/no_checkpoint_start.inc > + > +--disable_warnings > +SET GLOBAL innodb_file_format = `Barracuda`; > +SET GLOBAL innodb_file_per_table = ON; > +--enable_warnings > + > +create table t1(a int not null primary key auto_increment, c char(250), b > blob) engine=innodb row_format=compressed encrypted=yes > encryption_key_id=10; > +create table t2(a int not null primary key auto_increment, c char(250), b > blob) engine=innodb row_format=compressed; > +create table t3(a int not null primary key auto_increment, c char(250), b > blob) engine=innodb encrypted=yes encryption_key_id=10; > +create table t4(a int not null primary key auto_increment, c char(250), b > blob) engine=innodb; > +CREATE INDEX test ON t1 (b(10)); > +CREATE INDEX test ON t2 (b(10)); > +CREATE INDEX test ON t3 (b(10)); > +CREATE INDEX test ON t4 (b(10)); > + > +--disable_query_log > +--let $i = 20 > +begin; > +while ($i) > +{ > + insert into t1(c,b) values (repeat('secret1',20), > repeat('secret2',6000)); > + dec $i; > +} > +commit; > +--enable_query_log > + > +begin; > +insert into t2 select * from t1; > +insert into t3 select * from t1; > +insert into t4 select * from t1; > +commit; > + > +--remove_file $MYSQLTEST_VARDIR/keys1.txt > +let SEARCH_FILE= $MYSQLTEST_VARDIR/log/my_restart.err; > + > +# > +# We test redo log page read at recv_read_page using > +# incorrect keys from std_data/keys.txt. If checkpoint > +# happens we will skip this test. If no checkpoint > +# happens, InnoDB refuses to start as used > +# encryption key is incorrect. > +# > +SET GLOBAL innodb_buf_flush_list_now = 1; > +begin; > +update t1 set c = repeat('secret3', 20); > +update t2 set c = repeat('secret4', 20); > +update t3 set c = repeat('secret4', 20); > +update t4 set c = repeat('secret4', 20); > +insert into t1 (c,b) values (repeat('secret5',20), > repeat('secret6',6000)); > +insert into t2 (c,b) values (repeat('secret7',20), > repeat('secret8',6000)); > +insert into t3 (c,b) values (repeat('secret9',20), > repeat('secre10',6000)); > +insert into t4 (c,b) values (repeat('secre11',20), > repeat('secre12',6000)); > +COMMIT; > +let $cleanup= drop table t1,t2,t3,t4; > +--let CLEANUP_IF_CHECKPOINT= $cleanup; > +--source ../../suite/innodb/include/no_checkpoint_end.inc > Do we really need to execute all this code while not allowing any redo log checkpoint? It would seem to me that this test would be skipped most of the time. Where is the file my_restart.err created? +--write_file $MYSQLTEST_VARDIR/keys1.txt > +1;770A8A65DA156D24EE2A093277530997 > +10;770A8A65DA156D24EE2A093277530142 > +EOF > + > +--echo # restart > +--error 1 > +-- source include/start_mysqld.inc > +--source include/kill_mysqld.inc > This looks very suspicious. Why is the start and kill needed? And when would --source return an error? Why would we want to kill the server here? A shutdown would make the test more deterministic. I think that we would want to ensure that InnoDB refused to start up after the start_mysqld.inc. > diff --git a/mysql-test/suite/encryption/t/innodb-redo-nokeys.test > b/mysql-test/suite/encryption/t/innodb-redo-nokeys.test > new file mode 100644 > index 00000000000..bfe5cd78e5b > --- /dev/null > +++ b/mysql-test/suite/encryption/t/innodb-redo-nokeys.test > @@ -0,0 +1,98 @@ > +-- source include/have_innodb.inc > +-- source include/have_file_key_management_plugin.inc > +# embedded does not support restart > +-- source include/not_embedded.inc > +# Don't test this under valgrind, memory leaks will occur. > +--source include/not_valgrind.inc > +# Avoid CrashReporter popup on Mac > +--source include/not_crashrep.inc > +--source include/not_windows.inc > + > +--write_file $MYSQLTEST_VARDIR/keys1.txt > +1;770A8A65DA156D24EE2A093277530997 > +10;770A8A65DA156D24EE2A093277530999 > +EOF > This file is very similar to the other added tests. I would prefer to have one test that tests multiple things across a server restart, instead of having a large number of tests that duplicate a lot of boilerplate code and make it hard to follow the actual purpose of the test. Can you please merge some of the tests? +--source ../../suite/innodb/include/no_checkpoint_start.inc > + > +create table t1(a int not null primary key auto_increment, c char(200), b > blob) engine=innodb row_format=compressed encrypted=yes > encryption_key_id=10; > +create table t2(a int not null primary key auto_increment, c char(200), b > blob) engine=innodb row_format=compressed; > +create table t3(a int not null primary key auto_increment, c char(200), b > blob) engine=innodb encrypted=yes encryption_key_id=10; > +create table t4(a int not null primary key auto_increment, c char(200), b > blob) engine=innodb; > +CREATE INDEX test ON t1 (b(10)); > +CREATE INDEX test ON t2 (b(10)); > +CREATE INDEX test ON t3 (b(10)); > +CREATE INDEX test ON t4 (b(10)); > + > +--disable_query_log > +--let $i = 20 > +begin; > +while ($i) > +{ > + insert into t1(c,b) values (repeat('secret1',20), > repeat('secret2',6000)); > + dec $i; > +} > +commit; > +--enable_query_log > + > +begin; > +insert into t2 select * from t1; > +insert into t3 select * from t1; > +insert into t4 select * from t1; > +commit; > + > +--remove_file $MYSQLTEST_VARDIR/keys1.txt > + > +# > +# We test redo log page read at recv_read_page using > +# incorrect keys from std_data/keys.txt. If checkpoint > +# happens we will skip this test. If no checkpoint > +# happens, InnoDB refuses to start as used > +# encryption key is not found. > +# > +SET GLOBAL innodb_flush_log_at_trx_commit=1; > +begin; > +update t1 set c = repeat('secret3', 20); > +update t2 set c = repeat('secret4', 20); > +update t3 set c = repeat('secret4', 20); > +update t4 set c = repeat('secret4', 20); > +insert into t1 (c,b) values (repeat('secret5',20), > repeat('secret6',6000)); > +insert into t2 (c,b) values (repeat('secret7',20), > repeat('secret8',6000)); > +insert into t3 (c,b) values (repeat('secret9',20), > repeat('secre10',6000)); > +insert into t4 (c,b) values (repeat('secre11',20), > repeat('secre12',6000)); > +COMMIT; > +let $cleanup= drop table t1,t2,t3,t4; > +--let CLEANUP_IF_CHECKPOINT= $cleanup; > +--source ../../suite/innodb/include/no_checkpoint_end.inc > Is it really necessary to place the no_checkpoint_start.inc before the table creation statements? Why are there separate CREATE TABLE and CREATE INDEX statements for the empty tables? > diff --git a/mysql-test/suite/innodb/t/innodb_bug14147491.test > b/mysql-test/suite/innodb/t/innodb_bug14147491.test > index 5776b2c2e37..d16a44e8a55 100644 > --- a/mysql-test/suite/innodb/t/innodb_bug14147491.test > +++ b/mysql-test/suite/innodb/t/innodb_bug14147491.test > @@ -7,6 +7,7 @@ > call mtr.add_suppression("InnoDB: Database page corruption on disk or a > failed.*"); > call mtr.add_suppression("InnoDB: Corruption: Block in space_id .* in > file .* corrupted."); > call mtr.add_suppression("InnoDB: Based on page type .*"); > +call mtr.add_suppression("InnoDB: Unable to read tablespace .* page no .* > into the buffer pool after .* attempts"); > > # Don't test under valgrind, memory leaks will occur > source include/not_valgrind.inc; > I think that InnoDB is already noisy enough. Please fix the code instead of adding one more warning that is not really helping users. @@ -4096,6 +4096,11 @@ btr_estimate_number_of_different_key_vals( > > page = btr_cur_get_page(&cursor); > > + if (index->table->file_unreadable || > index->table->corrupted) { > + mtr_commit(&mtr); > + goto exit_loop; > + } > + > rec = page_rec_get_next(page_get_infimum_rec(page)); > > if (!page_rec_is_supremum(rec)) { > When is this code covered? Why are we not testing this before assigning page? That variable is not being used after the goto. In fact, the variable "page" should be declared inside the for loop body. diff --git a/storage/innobase/btr/btr0pcur.cc b/storage/innobase/btr/ > btr0pcur.cc > index 01d2e1bb8e2..237e4e8d109 100644 > --- a/storage/innobase/btr/btr0pcur.cc > +++ b/storage/innobase/btr/btr0pcur.cc > @@ -416,6 +416,11 @@ btr_pcur_move_to_next_page( > cursor->old_stored = BTR_PCUR_OLD_NOT_STORED; > > page = btr_pcur_get_page(cursor); > + > + if (!page) { > + return; > + } > + > next_page_no = btr_page_get_next(page, mtr); > space = buf_block_get_space(btr_pcur_get_block(cursor)); > zip_size = buf_block_get_zip_size(btr_pcur_get_block(cursor)); > @@ -425,11 +430,16 @@ btr_pcur_move_to_next_page( > next_block = btr_block_get(space, zip_size, next_page_no, > cursor->latch_mode, > btr_pcur_get_btr_cur(cursor)->index, > mtr); > + > + if (!next_block) { > + return; > + } > + > next_page = buf_block_get_frame(next_block); > #ifdef UNIV_BTR_DEBUG > ut_a(page_is_comp(next_page) == page_is_comp(page)); > ut_a(btr_page_get_prev(next_page, mtr) > - == buf_block_get_page_no(btr_pcur_get_block(cursor))); > + == buf_block_get_page_no(btr_pcur_get_block(cursor))); > #endif /* UNIV_BTR_DEBUG */ > next_block->check_index_page_at_flush = TRUE; > Are these changes really necessary? In which tests would the added "return" statements be executed? These conditions look like they could hurt performance. > diff --git a/storage/innobase/buf/buf0buf.cc b/storage/innobase/buf/ > buf0buf.cc > index c7e2183fa7b..066a68da8c0 100644 > --- a/storage/innobase/buf/buf0buf.cc > +++ b/storage/innobase/buf/buf0buf.cc > @@ -2358,7 +2358,18 @@ buf_page_get_zip( > /* Page not in buf_pool: needs to be read from file */ > > ut_ad(!hash_lock); > - buf_read_page(space, zip_size, offset, NULL); > + dberr_t err = buf_read_page(space, zip_size, offset); > + > + if (err != DB_SUCCESS) { > + ib_logf(IB_LOG_LEVEL_ERROR, > + "Reading compressed page " ULINTPF > + " from tablespace " ULINTPF > + " failed with error: %s (%d).", > + offset, space, ut_strerr(err), err); > + > + goto err_exit; > + } > + > > #if defined UNIV_DEBUG || defined UNIV_BUF_DEBUG > ut_a(++buf_dbg_counter % 5771 || buf_validate()); > Return the error to the only caller of this function, and let that function report the error. With the patch as it is, 2 messages would be reported for the single error. Reporting the ut_strerr(err) alone is sufficient; nobody is interested in the enum dberr_t numeric values. DBUG_EXECUTE_IF( > "innodb_page_corruption_retries", > - retries = BUF_PAGE_READ_MAX_RETRIES; > + goto force_fail; > ); > This change is wrong. The idea of the debug instrumentation is to speed up the failure (not retry reading the corrupted page for 100 times). The test must not crash even when this instrumentation is not present. > + /* Try to set table as corrupted instead of > + asserting. */ > + if (space > TRX_SYS_SPACE && > + dict_set_corrupted_by_space(space)) { > + return (NULL); > } > Why should we commit suicide when a page in the system tablespace is corrupted? I would just return NULL from here in any case. - if (!bpage->encrypted) { > - ut_ad(buf_pool->n_pend_reads > 0); > - buf_pool->n_pend_reads--; > + /* After this point bpage can't be referenced. */ > + buf_LRU_free_one_page(bpage); > > - buf_pool_mutex_exit(buf_pool); > - } > + ut_ad(buf_pool->n_pend_reads > 0); > + buf_pool->n_pend_reads--; > > - return(ret); > + buf_pool_mutex_exit(buf_pool); > } > Elsewhere, MariaDB 10.1 is using atomic memory access for the field buf_pool->n_pend_reads. We must be consistent. If some reader or writer is using atomic memory access and no buf_pool->mutex at all, then every access must use atomic memory access, no matter if holding buf_pool->mutex. @@ -4708,31 +4696,12 @@ buf_page_io_complete( > /* If page space id is larger than > TRX_SYS_SPACE > (0), we will attempt to mark the > corresponding > table as corrupted instead of crashing > server */ > - if (bpage->space > TRX_SYS_SPACE > - && buf_mark_space_corrupt(bpage)) { > - return(false); > + if (bpage->space > TRX_SYS_SPACE) { > + buf_mark_space_corrupt(bpage); > + return(err); > } else { > - if (!bpage->encrypted) { > - ib_logf(IB_LOG_LEVEL_ERROR, > - "Ending processing > because of a corrupt database page."); > - > - ut_error; > - } > - > - ib_push_warning(innobase_get_trx(), > DB_DECRYPTION_FAILED, > - "Table in tablespace %lu > encrypted." > - "However key management > plugin or used key_id %lu is not found or" > - " used encryption > algorithm or method does not match." > - " Can't continue opening > the table.", > - bpage->space, > bpage->key_version); > - > - if (bpage->encrypted && > bpage->space > TRX_SYS_SPACE) { > - > buf_mark_space_corrupt(bpage); > - } else { > - ut_error; > - } > - > - return(false); > + ib_logf(IB_LOG_LEVEL_FATAL, > + "Ending processing because > of a corrupt database page."); > } > } > } > Again, why commit suicide if a page in the system tablespace cannot be read? I think that we should try our best to allow any data to be rescue, and not crash early. diff --git a/storage/innobase/buf/buf0flu.cc b/storage/innobase/buf/ > buf0flu.cc > index 3f03ee8256f..e805585ed1e 100644 > --- a/storage/innobase/buf/buf0flu.cc > +++ b/storage/innobase/buf/buf0flu.cc > @@ -956,7 +956,10 @@ buf_flush_write_block_low( > > /* true means we want to evict this page from the > LRU list as well. */ > - buf_page_io_complete(bpage, true); > + dberr_t err = buf_page_io_complete(bpage, true); > + > + /* This is write path, encryption should not fail here. */ > + ut_ad(err == DB_SUCCESS); > } > > /* Increment the counter of I/O operations used > I think I already noted in an earlier review that a warning for the unused variable "err" would be issued for the non-debug build. Please build both non-debug and debug, and pay attention to the warnings! @@ -444,23 +464,23 @@ High-level function which reads a page asynchronously > from a file to the > buffer buf_pool if it is not already there. Sets the io_fix flag and sets > an exclusive lock on the buffer frame. The flag is cleared and the x-lock > released by the i/o-handler thread. > -@return TRUE if page has been read in, FALSE in case of failure */ > +@param[in] space Tablespace id > +@param[in] offset Page no */ > UNIV_INTERN > -ibool > +void > buf_read_page_async( > -/*================*/ > - ulint space, /*!< in: space id */ > - ulint offset) /*!< in: page number */ > + ulint space, > + ulint offset) > Add a function comment that this function is only used for restoring buffer pool dumps, and that we do not care if the page exists or if the read succeeded. @@ -728,16 +748,30 @@ buf_read_ahead_linear( > count += buf_read_page_low( > &err, false, > ibuf_mode, > - space, zip_size, FALSE, > tablespace_version, i, NULL); > - if (err == DB_TABLESPACE_DELETED) { > - ut_print_timestamp(stderr); > - fprintf(stderr, > - " InnoDB: Warning: in" > - " linear readahead trying to > access\n" > - "InnoDB: tablespace %lu page > %lu,\n" > - "InnoDB: but the tablespace does > not" > - " exist or is just being > dropped.\n", > - (ulong) space, (ulong) i); > + space, zip_size, FALSE, > tablespace_version, i); > + > + switch(err) { > + case DB_SUCCESS: > + break; > + case DB_TABLESPACE_DELETED: > + ib_logf(IB_LOG_LEVEL_WARN, > + "In random" > + "readahead trying to access" > + "tablespace " ULINTPF " page " > ULINTPF > + " but the tablespace does not" > + " exist or is just being dropped.", > + space, i); > + break; > + > + case DB_DECRYPTION_FAILED: > + ib_logf(IB_LOG_LEVEL_ERROR, > + "In random readahead read page " > ULINTPF > + " from tablespace " ULINTPF > + " but page is still encrypted.", > + i, space); > + break; > + default: > + ut_error; > } > You seem to have missed my earlier review comment where I noted that the message about "linear readahead" is misleadingly replaced by messages mentioning "random readahead". > @@ -829,6 +863,13 @@ buf_read_ibuf_merge_pages( > page_nos[i], > zip_size, FALSE); > } > + > + if (err == DB_DECRYPTION_FAILED) { > + ib_logf(IB_LOG_LEVEL_ERROR, > + "Read page " ULINTPF " from tablespace " > ULINTPF > + " for insert buffer but page was > encrypted.", > + page_nos[i], space_ids[i]); > + } > } > > os_aio_simulated_wake_handler_threads(); > It would be clearer to say in the error message that we were unable to decrypt the page. > @@ -2192,7 +2192,7 @@ dict_load_table_low( > > (*table)->id = mach_read_from_8(field); > > - (*table)->ibd_file_missing = FALSE; > + (*table)->file_unreadable = FALSE; > > return(NULL); > } > @@ -2379,7 +2379,7 @@ dict_load_table( > "Table '%s' tablespace is set as discarded.", > table_name); > > - table->ibd_file_missing = TRUE; > + table->file_unreadable = true; > > } else if (!fil_space_for_table_exists_in_mem( > table->space, name, false, true, heap, > @@ -2387,7 +2387,7 @@ dict_load_table( > > if (DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY)) { > /* Do not bother to retry opening temporary > tables. */ > - table->ibd_file_missing = TRUE; > + table->file_unreadable = true; > > } else { > if (!(ignore_err & DICT_ERR_IGNORE_RECOVER_LOCK)) { > @@ -2422,7 +2422,7 @@ dict_load_table( > /* We failed to find a sensible > tablespace file */ > > - table->ibd_file_missing = TRUE; > + table->file_unreadable = TRUE; > } > if (filepath) { > mem_free(filepath); > This is not consistent. Sometimes you are using true, sometimes TRUE or FALSE for the unsigned:1 bit-field. Either way is fine, but be consistent. > diff --git a/storage/innobase/fil/fil0crypt.cc b/storage/innobase/fil/ > fil0crypt.cc > index 7bef91f47ff..6a54ec6df35 100644 > --- a/storage/innobase/fil/fil0crypt.cc > +++ b/storage/innobase/fil/fil0crypt.cc > @@ -541,6 +541,11 @@ fil_parse_write_crypt_data( > fil_space_release(space); > } > > + /* Check is used key found from encryption plugin */ > + if (crypt_data->should_encrypt() && !crypt_data->is_key_found()) { > + return NULL; > + } > + > return ptr; > } > > As far as I can understand, this is prematurely ending the redo log recovery, without returning any error! This is not acceptable. Please add a proper test for this and ensure that an error is being returned. > diff --git a/storage/innobase/fil/fil0fil.cc b/storage/innobase/fil/ > fil0fil.cc > index 627a63aaae0..079803eedf3 100644 > --- a/storage/innobase/fil/fil0fil.cc > +++ b/storage/innobase/fil/fil0fil.cc > @@ -6292,6 +6292,8 @@ fil_aio_wait( > mutex_enter(&fil_system->mutex); > > fil_node_complete_io(fil_node, fil_system, type); > + ulint purpose = fil_node->space->purpose; > + ulint space_id = fil_node->space->id; > > mutex_exit(&fil_system->mutex); > > Why assign fil_node->space->purpose to a local variable? It is only referenced once. > @@ -6303,9 +6305,29 @@ fil_aio_wait( > deadlocks in the i/o system. We keep tablespace 0 data files always > open, and use a special i/o thread to serve insert buffer > requests. */ > > - if (fil_node->space->purpose == FIL_TABLESPACE) { > + if (purpose == FIL_TABLESPACE) { > srv_set_io_thread_op_info(segment, "complete io for buf > page"); > - buf_page_io_complete(static_cast<buf_page_t*>(message)); > + buf_page_t* bpage = static_cast<buf_page_t*>(message); > + ulint offset = bpage ? bpage->offset : ULINT_UNDEFINED; > + > + dberr_t err = buf_page_io_complete(bpage); > + > + if (err != DB_SUCCESS) { > + ib_log_level_t level = IB_LOG_LEVEL_FATAL; > + > + /* In crash recovery set log corruption on > + and produce only an error to fail InnoDB startup. > */ > + if (recv_recovery_is_on()) { > + recv_sys->found_corrupt_log = true; > + level = IB_LOG_LEVEL_ERROR; > + } > + > + ib_logf(level, > + "%s operation failed for tablespace " > + ULINTPF " offset " ULINTPF " error=%s > (%d).", > + type == OS_FILE_WRITE ? "Write" : "Read", > + space_id, offset, ut_strerr(err),err); > + } > } else { > srv_set_io_thread_op_info(segment, "complete io for log"); > log_io_complete(static_cast<log_group_t*>(message)); > Can this really ever fail for writes? Please add a debug assertion and simplify the message. There is no need to display the numeric value of dberr_t; the ut_strerr() suffices. diff --git a/storage/innobase/include/btr0pcur.ic > b/storage/innobase/include/btr0pcur.ic > index 1cd13824542..87b48de6389 100644 > --- a/storage/innobase/include/btr0pcur.ic > +++ b/storage/innobase/include/btr0pcur.ic > @@ -357,6 +357,11 @@ btr_pcur_move_to_next( > return(FALSE); > } > > + if (cursor->btr_cur.index->table->corrupted || > + cursor->btr_cur.index->table->file_unreadable) { > + return(FALSE); > + } > + > btr_pcur_move_to_next_page(cursor, mtr); > > return(TRUE); > Is this change really needed? Can we merge the "corrupted" flag to "file_unreadable" to make the check less expensive? diff --git a/storage/innobase/include/log0recv.h b/storage/innobase/include/ > log0recv.h > index 9ca1c46d72b..1ff5cf721bf 100644 > --- a/storage/innobase/include/log0recv.h > +++ b/storage/innobase/include/log0recv.h > @@ -274,9 +274,10 @@ recv_sys_var_init(void); > #endif /* !UNIV_HOTBACKUP */ > /** Apply the hash table of stored log records to persistent data pages. > @param[in] last_batch whether the change buffer merge will be > - performed as part of the operation */ > + performed as part of the operation > +@return DB_SUCCESS or DB_DECRYPTION_FAILED */ > UNIV_INTERN > -void > +dberr_t > recv_apply_hashed_log_recs(bool last_batch); > Please try to keep the return value as void, or alternatively add MY_ATTRIBUTE((warn_unused_result)) and fix all callers. I do not think it is easy to fix log_checkpoint(), > diff --git a/storage/innobase/log/log0recv.cc b/storage/innobase/log/ > log0recv.cc > index 45fd42848e8..e56d4a46650 100644 > --- a/storage/innobase/log/log0recv.cc > +++ b/storage/innobase/log/log0recv.cc > @@ -1298,6 +1298,10 @@ recv_parse_or_apply_log_rec_body( > break; > case MLOG_FILE_WRITE_CRYPT_DATA: > ptr = const_cast<byte*>(fil_parse_write_crypt_data(ptr, > end_ptr, block)); > + > + if (!ptr) { > + recv_sys->found_corrupt_log = TRUE; > + } > break; > default: > ptr = NULL; > If block==NULL, ptr=NULL is returned on EOF. We do not want to have random bogus failures in redo log recovery! Even if a redo log record happens to be split across multiple redo log segments, we must parse the whole redo log! > @@ -3549,7 +3553,7 @@ row_truncate_table_for_mysql( > "create a new tablespace", > table->name); > > - table->ibd_file_missing = 1; > + table->file_unreadable = 1; > err = DB_ERROR; > goto funct_exit; > } > Here we are assigning yet another literal to the field. Earlier, FALSE, TRUE, and true have been observed. Be consistent. Or even better, add a method void dict_table_t::set_unreadable(bool unreadable = true) and use it. > diff --git a/storage/innobase/row/row0sel.cc b/storage/innobase/row/ > row0sel.cc > index dab1bc58db5..8f73ed4d49a 100644 > --- a/storage/innobase/row/row0sel.cc > +++ b/storage/innobase/row/row0sel.cc > @@ -3714,11 +3714,12 @@ row_search_for_mysql( > > return(DB_TABLESPACE_DELETED); > > - } else if (prebuilt->table->ibd_file_missing) { > + } else if (prebuilt->table->file_unreadable && > + fil_space_get(prebuilt->table->space) == NULL) { > > return(DB_TABLESPACE_NOT_FOUND); > > - } else if (prebuilt->table->is_encrypted) { > + } else if (prebuilt->table->file_unreadable) { > > return(DB_DECRYPTION_FAILED); > } else if (!prebuilt->index_usable) { > I would add an method bool dict_table_t::is_file_missing() const and use it instead of duplicating the code. Marko -- DON’T MISS M|17 April 11 - 12, 2017 The Conrad Hotel New York City https://m17.mariadb.com/ Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation

1 0

[Maria-developers] Queries on ticket MDEV-7502(GSOC)
by Vishwanath Bharadwaj 22 Mar '17

22 Mar '17

Hi, Vishwanath! Hello MR. Sergei, I am Vishwanath Bharadwaj and I've contacted you earlier regarding the project. With the help of the code Kristian provided in the comment section of the ticket I was able to review the code of the project proceedings till date. I was very much impressed by the way it has been implemented. I don't have any major queries (just some minor issues on the implementation of the semaphores). I've also familiarized myself with some code of both statement based binary logging and row- based logging. I would also like to get into the details of what the provisioning mode permissions are? And the method of using the primary key to identify the remaining chunk is not quite effective because weare limiting the options only to tables with primary keys. This is quite a unique and interesting project and I'm looking forward to working and discussing with you on the same. Thank you:)

2 1