- developers - lists.mariadb.org

Re: [Maria-developers] InnoDB blob for primary key
by Sachin Setia 19 May '16

19 May '16

Hi Sergei! As i told you i was prototyping for hash table It is done around 90% apart from one thing when hash is same how to get record from .myd file when i have offset of record so currently i am skipping it But it is very fast i do not know why this is so fast here are results of employee database salary table definition CREATE TABLE salaries ( emp_no INT NOT NULL, salary blob NOT NULL, from_date DATE NOT NULL, to_date DATE NOT NULL, FOREIGN KEY (emp_no) REFERENCES employees (emp_no) ON DELETE CASCADE, PRIMARY KEY (emp_no, from_date) ) ; And query is MariaDB [employees]> select distinct salary from salaries; Result with out using hash table +--------+ 85814 rows in set (2 min 24.76 sec) Result with using hash table | 39420 | +--------+ 85809 rows in set (6.24 sec) ( number of rows are not equal but this can be solved if i get record by offset) I am sure there is something wrong.The whole hash table is in memory like wise the b tree of hash is in memory but why there is so much improvement. Please sir check the prototype and tell if i am wrong .thanks Regards sachin On Mon, May 2, 2016 at 11:43 AM, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Sachin! > > On May 02, Sachin Setia wrote: > > I am sorry sir Currently my exam are going on > > But i am working on prototype of second project. Will be done by tommorow > > Regards > > sachin > > Sure thing, that's totally fine! > > Still, in the future, if you plan to go silent for a while (exam or you > just want to relax for a few days or something else :) - please drop me > a short email and then I will know that you didn't disappear from the > project. Thanks! > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org >

2 4

Re: [Maria-developers] 1e883e9: MDEV-5535: Cannot reopen temporary table
by Sergei Golubchik 19 May '16

19 May '16

Hi, Nirbhay! Here's a review of MDEV-5535. Sorry, it took a while - this was a big patch. The approach is fine, my comments are mostly about details. On Mar 07, Nirbhay Choubey wrote: > diff --git a/mysql-test/r/temp_table.result b/mysql-test/r/temp_table.result > index ee0b3ab..94b02a6 100644 > --- a/mysql-test/r/temp_table.result > +++ b/mysql-test/r/temp_table.result > @@ -308,3 +308,185 @@ show status like 'com_drop%table'; > Variable_name Value > Com_drop_table 2 > Com_drop_temporary_table 1 > +# > +# MDEV-5535: Cannot reopen temporary table > +# > +DROP DATABASE IF EXISTS temp_db; > +CREATE DATABASE temp_db; > +USE temp_db; > +# > +# SHOW TABLES do not list temporary tables. > +# > +CREATE TEMPORARY TABLE temp_t1(i INT) ENGINE=INNODB; > +INSERT INTO temp_t1 VALUES(1); > +SELECT * FROM temp_t1; > +i > +1 > +SHOW TABLES; > +Tables_in_temp_db > +DROP TABLE temp_t1; > +# > +# Create and drop a temporary table. > +# > +CREATE TEMPORARY TABLE temp_t1(i INT) ENGINE=INNODB; > +INSERT INTO temp_t1 VALUES(1); > +SELECT * FROM temp_t1; > +i > +1 > +DROP TABLE temp_t1; > +SELECT * FROM temp_t1; > +ERROR 42S02: Table 'temp_db.temp_t1' doesn't exist > +# > +# Create a tempporary table and base table with same name and DROP TABLE. > +# > +CREATE TABLE t1(c1 VARCHAR(20)) ENGINE=INNODB; > +INSERT INTO t1 VALUES("BASE TABLE"); > +CREATE TEMPORARY TABLE t1(c1 VARCHAR(20)) ENGINE=INNODB; > +INSERT INTO t1 VALUES("TEMPORARY TABLE"); > +SELECT * FROM t1; > +c1 > +TEMPORARY TABLE > +DROP TABLE t1; > +SELECT * FROM t1; > +c1 > +BASE TABLE > +DROP TABLE t1; > +SELECT * FROM t1; > +ERROR 42S02: Table 'temp_db.t1' doesn't exist > +# > +# Create a tempporary table and base table with same name and DROP TEMPORARY > +# TABLE. > +# > +CREATE TABLE t1(c1 VARCHAR(20)) ENGINE=INNODB; > +INSERT INTO t1 VALUES("BASE TABLE"); > +CREATE TEMPORARY TABLE t1(c1 VARCHAR(20)) ENGINE=INNODB; > +INSERT INTO t1 VALUES("TEMPORARY TABLE"); > +SELECT * FROM t1; > +c1 > +TEMPORARY TABLE > +DROP TEMPORARY TABLE t1; > +SELECT * FROM t1; > +c1 > +BASE TABLE > +DROP TEMPORARY TABLE t1; > +ERROR 42S02: Unknown table 'temp_db.t1' > +SELECT * FROM t1; > +c1 > +BASE TABLE > +DROP TABLE t1; > +# > +# Create a temporary table and drop its parent database. > +# > +USE temp_db; > +CREATE TEMPORARY TABLE temp_t1(i INT) ENGINE=INNODB; > +INSERT INTO temp_t1 VALUES (1); > +DROP DATABASE temp_db; > +CREATE DATABASE temp_db; > +USE temp_db; > +DROP TEMPORARY TABLE temp_t1; > +# > +# Similar to above, but this time with a base table with same name. > +# > +USE temp_db; > +CREATE TABLE t1(i INT)ENGINE=INNODB; > +CREATE TEMPORARY TABLE t1(i INT) ENGINE=INNODB; > +INSERT INTO t1 VALUES (1); > +DROP DATABASE temp_db; > +CREATE DATABASE temp_db; > +USE temp_db; > +DROP TEMPORARY TABLE t1; > +DROP TABLE t1; > +ERROR 42S02: Unknown table 'temp_db.t1' > +# > +# Create a temporary table within a function. > +# > +USE temp_db; > +CREATE FUNCTION f1() RETURNS INT > +BEGIN > +DROP TEMPORARY TABLE IF EXISTS temp_t1; > +CREATE TEMPORARY TABLE temp_t1(i INT) ENGINE=INNODB; > +INSERT INTO `temp_t1` VALUES(1); > +RETURN (SELECT COUNT(*) FROM temp_t1); > +END| > +SELECT f1(); > +f1() > +1 > +SELECT * FROM temp_t1; > +i > +1 > +DROP TABLE temp_t1; > +CREATE TEMPORARY TABLE `temp_t1`(i INT) ENGINE=INNODB; > +SELECT f1(); > +f1() > +1 > +SELECT * FROM temp_t1; > +i > +1 > +DROP FUNCTION f1; > +# > +# Create and drop a temporary table within a function. > +# > +CREATE FUNCTION f2() RETURNS INT > +BEGIN > +DROP TEMPORARY TABLE IF EXISTS temp_t1; > +CREATE TEMPORARY TABLE temp_t1(i INT) ENGINE=INNODB; > +INSERT INTO temp_t1 VALUES(1); > +DROP TABLE temp_t1; > +RETURN 0; > +END| > +ERROR HY000: Explicit or implicit commit is not allowed in stored function or trigger. > +# > +# Create a temporary table within a function and select it from another > +# function. > +# > +CREATE FUNCTION f2() RETURNS INT > +BEGIN > +DROP TEMPORARY TABLE IF EXISTS temp_t1; > +CREATE TEMPORARY TABLE temp_t1 (i INT) ENGINE=INNODB; > +INSERT INTO temp_t1 VALUES (1); > +RETURN f2_1(); > +END| > +CREATE FUNCTION f2_1() RETURNS INT > +RETURN (SELECT COUNT(*) FROM temp_t1)| > +DROP TEMPORARY TABLE temp_t1; > +SELECT f2(); > +f2() > +1 > +DROP TEMPORARY TABLE temp_t1; > +DROP FUNCTION f2; > +# > +# Create temporary table like base table. > +# > +CREATE TABLE t1(i INT) ENGINE=INNODB; > +INSERT INTO t1 VALUES(1); > +CREATE TEMPORARY TABLE temp_t1 LIKE t1; > +SELECT * FROM temp_t1; > +i > +CREATE TEMPORARY TABLE t1 LIKE t1; > +ERROR 42000: Not unique table/alias: 't1' > +DROP TABLE temp_t1, t1; > +# > +# Create temporary table as base table. > +# > +CREATE TABLE t1(i INT) ENGINE=INNODB; > +INSERT INTO t1 VALUES(1); > +CREATE TEMPORARY TABLE temp_t1 AS SELECT * FROM t1; > +SELECT * FROM temp_t1; > +i > +1 > +DROP TABLE temp_t1, t1; > +# > +# Reopen temporary table > +# > +CREATE TEMPORARY TABLE temp_t1(i int)ENGINE=INNODB; > +INSERT INTO temp_t1 VALUES(1), (2); > +SELECT * FROM temp_t1 a, temp_t1 b; > +i i > +1 1 > +1 2 > +2 1 > +2 2 > +DROP TABLE temp_t1; > +# Cleanup > +DROP DATABASE temp_db; > +# MDEV-5535: End of tests What's that? You have lots of tests for temporary tables - not that I mind the number, but they are mostly unrelated to MDEV-5535. Only the one last test is about the new feature. I'd expect it to be the other way around - many tests for the new functionality and few, if at all - for the unmodified behavior. > diff --git a/mysql-test/r/temp_table_frm.result b/mysql-test/r/temp_table_frm.result > index 19c6638..58e9a0a 100644 > --- a/mysql-test/r/temp_table_frm.result > +++ b/mysql-test/r/temp_table_frm.result > @@ -16,6 +16,6 @@ variable_name session_status.variable_value - t1.variable_value > OPENED_FILES 0 > OPENED_PLUGIN_LIBRARIES 0 > OPENED_TABLE_DEFINITIONS 2 > -OPENED_TABLES 1 > +OPENED_TABLES 2 why did it change? UPDATE: now, I see - you close and open temporary tables all the time. See my comment about it below. > OPENED_VIEWS 0 > drop table t1; > diff --git a/mysql-test/suite/handler/handler.inc b/mysql-test/suite/handler/handler.inc > index c71dc53..ca67b62 100644 > --- a/mysql-test/suite/handler/handler.inc > +++ b/mysql-test/suite/handler/handler.inc > @@ -489,7 +489,7 @@ handler t1 open as a1; > handler a1 read a=(1); > handler a1 read a next; > handler a1 read a next; > ---error ER_CANT_REOPEN_TABLE > +#--error ER_CANT_REOPEN_TABLE remove it, don't comment. And, please, fix the comment before this test. > select a,b from t1; > handler a1 read a prev; > handler a1 read a prev; > diff --git a/mysql-test/suite/multi_source/status_vars.result b/mysql-test/suite/multi_source/status_vars.result > index 12917f9..50d500e 100644 > --- a/mysql-test/suite/multi_source/status_vars.result > +++ b/mysql-test/suite/multi_source/status_vars.result > @@ -76,22 +76,34 @@ start slave; > include/wait_for_slave_to_start.inc > set binlog_format = statement; > create temporary table tmp1 (i int) engine=MyISAM; > +show status like 'Slave_open_temp_table_definitions'; > +Variable_name Value > +Slave_open_temp_table_definitions 1 > show status like 'Slave_open_temp_tables'; > Variable_name Value > -Slave_open_temp_tables 1 > +Slave_open_temp_tables 0 why did it change? > set default_master_connection = 'master1'; > +show status like 'Slave_open_temp_table_definitions'; > +Variable_name Value > +Slave_open_temp_table_definitions 1 > show status like 'Slave_open_temp_tables'; > Variable_name Value > -Slave_open_temp_tables 1 > +Slave_open_temp_tables 0 > set binlog_format = statement; > create temporary table tmp1 (i int) engine=MyISAM; > +show status like 'Slave_open_temp_table_definitions'; > +Variable_name Value > +Slave_open_temp_table_definitions 2 > show status like 'Slave_open_temp_tables'; > Variable_name Value > -Slave_open_temp_tables 2 > +Slave_open_temp_tables 0 > set default_master_connection = ''; > +show status like 'Slave_open_temp_table_definitions'; > +Variable_name Value > +Slave_open_temp_table_definitions 2 > show status like 'Slave_open_temp_tables'; > Variable_name Value > -Slave_open_temp_tables 2 > +Slave_open_temp_tables 0 > include/reset_master_slave.inc > include/reset_master_slave.inc > include/reset_master_slave.inc > diff --git a/mysql-test/suite/multi_source/status_vars.test b/mysql-test/suite/multi_source/status_vars.test > index d1cfda7..f9d2e85 100644 > --- a/mysql-test/suite/multi_source/status_vars.test > +++ b/mysql-test/suite/multi_source/status_vars.test > @@ -107,9 +107,11 @@ create temporary table tmp1 (i int) engine=MyISAM; > > --connection slave > --sync_with_master 0,'master1' > +show status like 'Slave_open_temp_table_definitions'; > show status like 'Slave_open_temp_tables'; You can do show status like 'Slave_open_temp_table%'; instead. here and everywhere. > > set default_master_connection = 'master1'; > +show status like 'Slave_open_temp_table_definitions'; > show status like 'Slave_open_temp_tables'; > > --connect (master2,127.0.0.1,root,,,$SERVER_MYPORT_2) > diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc > index 7bdd9c1..426989c 100644 > --- a/sql/rpl_rli.cc > +++ b/sql/rpl_rli.cc > @@ -1060,24 +1061,14 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, > > void Relay_log_info::close_temporary_tables() > { > - TABLE *table,*next; > DBUG_ENTER("Relay_log_info::close_temporary_tables"); > > - for (table=save_temporary_tables ; table ; table=next) > + if (sql_driver_thd) > { > - next=table->next; > - > - /* Reset in_use as the table may have been created by another thd */ > - table->in_use=0; > - /* > - Don't ask for disk deletion. For now, anyway they will be deleted when > - slave restarts, but it is a better intention to not delete them. > - */ > - DBUG_PRINT("info", ("table: 0x%lx", (long) table)); > - close_temporary(table, 1, 0); > + sql_driver_thd->temporary_tables.close_tables(true); Hmm, really? see that comment above in the old code: /* Reset in_use as the table may have been created by another thd */ and now you assume that all temptables belong to sql_driver_thd? but sql_driver_thd comment says THD for the main sql thread, the one that starts threads to process slave requests. If there is only one thread, then this THD is also used for SQL processing. and indeed in parallel replication there can be many sql threads, all with their own temporary tables. > } > - save_temporary_tables= 0; > - slave_open_temp_tables= 0; > + save_temp_table_shares= 0; > + > DBUG_VOID_RETURN; > } > > @@ -1753,6 +1744,10 @@ void rpl_group_info::cleanup_context(THD *thd, bool error) > } > m_table_map.clear_tables(); > slave_close_thread_tables(thd); > + > + /* Cleanup temporary tables. */ > + thd->temporary_tables.cleanup(); > + why this was not needed before? > if (error) > { > thd->mdl_context.release_transactional_locks(); > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > index cf0b8e8..45346cd 100644 > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -2929,6 +2929,7 @@ void sp_head::add_mark_lead(uint ip, List<sp_instr> *leads) > thd->get_stmt_da()->set_overwrite_status(false); > } > thd_proc_info(thd, "closing tables"); > + /* main.temp_table: check this */ what does that mean? > close_thread_tables(thd); > thd_proc_info(thd, 0); > > @@ -2970,8 +2971,7 @@ void sp_head::add_mark_lead(uint ip, List<sp_instr> *leads) > open_tables stage. > */ > if (!res || !thd->is_error() || > - (thd->get_stmt_da()->sql_errno() != ER_CANT_REOPEN_TABLE && can ER_CANT_REOPEN_TABLE still happen somewhere? > - thd->get_stmt_da()->sql_errno() != ER_NO_SUCH_TABLE && > + (thd->get_stmt_da()->sql_errno() != ER_NO_SUCH_TABLE && > thd->get_stmt_da()->sql_errno() != ER_NO_SUCH_TABLE_IN_ENGINE && > thd->get_stmt_da()->sql_errno() != ER_UPDATE_TABLE_USED)) > thd->stmt_arena->state= Query_arena::STMT_EXECUTED; > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 6656a84..e73ada2 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -4230,7 +4234,8 @@ void THD::restore_backup_open_tables_state(Open_tables_backup *backup) > Before we will throw away current open tables state we want > to be sure that it was properly cleaned up. > */ > - DBUG_ASSERT(open_tables == 0 && temporary_tables == 0 && > + DBUG_ASSERT(open_tables == 0 && > + temporary_tables.is_empty() && temporary_tables.is_empty() is not the same as temporary_tables == 0, because it takes into account thd->rgi_slave > derived_tables == 0 && > lock == 0 && > locked_tables_mode == LTM_NONE && > diff --git a/sql/sql_class.h b/sql/sql_class.h > index ae240ae..02b5739 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -799,6 +800,13 @@ enum killed_type > volatile int64 local_memory_used; > /* Memory allocated for global usage */ > volatile int64 global_memory_used; > + > + /* Open temporary tables. */ > + ulong open_temporary_tables; this one doesn't seem to be used anywhere > + /* Number of temporary table definitions opened by THD. */ > + ulong opened_temporary_table_definitions; > + /* Number of temporary tables opened by THD. */ > + ulong opened_temporary_tables; > } STATUS_VAR; > > /* > @@ -3506,13 +3525,13 @@ class THD :public Statement, > */ > DBUG_PRINT("debug", > ("temporary_tables: %s, in_sub_stmt: %s, system_thread: %s", > - YESNO(temporary_tables), YESNO(in_sub_stmt), > + YESNO(temporary_tables.is_empty()), YESNO(in_sub_stmt), > show_system_thread(system_thread))); > if (in_sub_stmt == 0) > { > if (wsrep_binlog_format() == BINLOG_FORMAT_ROW) > set_current_stmt_binlog_format_row(); > - else if (temporary_tables == NULL) > + else if (temporary_tables.is_empty()) temporary_tables.is_empty() is not the same as temporary_tables == NULL > set_current_stmt_binlog_format_stmt(); > } > DBUG_VOID_RETURN; > diff --git a/sql/sql_handler.cc b/sql/sql_handler.cc > index e8ade81..52b14e1 100644 > --- a/sql/sql_handler.cc > +++ b/sql/sql_handler.cc > @@ -180,7 +180,7 @@ static void mysql_ha_close_table(SQL_HANDLER *handler) > table->file->ha_index_or_rnd_end(); > table->query_id= thd->query_id; > table->open_by_handler= 0; > - mark_tmp_table_for_reuse(table); > + //mark_tmp_table_for_reuse(table); why? > } > my_free(handler->lock); > handler->init(); > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index aa2cae5..c594ef1 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -4051,16 +4051,16 @@ static TABLE *create_table_from_items(THD *thd, > } > else > { > - if (open_temporary_table(thd, create_table)) > - { > - /* > - This shouldn't happen as creation of temporary table should make > - it preparable for open. Anyway we can't drop temporary table if > - we are unable to find it. > - */ > - DBUG_ASSERT(0); > - } > - DBUG_ASSERT(create_table->table == create_info->table); > + /* > + TODO: Explain why we do not need to call THD::wait_for_prior_commit() > + here? > + */ So? > + TABLE *tab= create_info->table; > + tab->query_id= thd->query_id; > + thd->thread_specific_used= true; > + create_table->updatable= true; > + create_table->table= tab; > + tab->init(thd, create_table); hmm, you don't create a table here. how comes? > } > } > else > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 52dcc7e..64227d4 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -23,7 +23,7 @@ > // set_handler_table_locks, > // lock_global_read_lock, > // make_global_read_lock_block_commit > -#include "sql_base.h" // find_temporary_table > +#include "sql_base.h" // Temporary_tables::find_table really? it's in temporary_tables.h, not in sql_base.h anymore > #include "sql_cache.h" // QUERY_CACHE_FLAGS_SIZE, query_cache_* > #include "sql_show.h" // mysqld_list_*, mysqld_show_*, > // calc_sum_of_all_status > @@ -5730,6 +5731,8 @@ static bool do_execute_sp(THD *thd, sp_head *sp) > > /* Free tables */ > close_thread_tables(thd); > + /* Do not close HANDLER tables. */ > + thd->temporary_tables.close_tables(false); why this wasn't here before? > #ifdef WITH_WSREP > thd->wsrep_consistency_check= NO_CONSISTENCY_CHECK; > #endif /* WITH_WSREP */ > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 40032c3..f568432 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -4608,13 +4602,14 @@ handler *mysql_create_frm_image(THD *thd, > which was created. > @param[out] key_count Number of keys in table which was created. > > - If one creates a temporary table, this is automatically opened > - > Note that this function assumes that caller already have taken > exclusive metadata lock on table being created or used some other > way to ensure that concurrent operations won't intervene. > mysql_create_table() is a wrapper that can be used for this. > > + In case of temporary tables, the TABLE_SHARE is added to thd's > + temporary_tables_share list. > + hmm, so you've decided not to open the table automatically... > @retval 0 OK > @retval 1 error > @retval -1 table existed but IF EXISTS was used > @@ -4820,17 +4813,12 @@ int create_table_impl(THD *thd, > create_info->table= 0; > if (!frm_only && create_info->tmp_table()) > { > - /* > - Open a table (skipping table cache) and add it into > - THD::temporary_tables list. > - */ > - > - TABLE *table= open_table_uncached(thd, create_info->db_type, frm, path, > - db, table_name, true, true); > + TABLE *table= > + thd->temporary_tables.create_and_use_table(create_info->db_type, frm, > + path, db, table_name, true); > > if (!table) > { > - (void) rm_temporary_table(create_info->db_type, path); why not? because create_and_use_table() is atomic? > goto err; > } > > diff --git a/sql/table.h b/sql/table.h > index ab39603..8b7c665 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -600,6 +601,7 @@ struct TABLE_STATISTICS_CB > struct TABLE_SHARE > { > TABLE_SHARE() {} /* Remove gcc warning */ > + TABLE_SHARE *next, *prev; I'd rather not add two pointers to TABLE_SHARE just for the sake of temporary tables. Use something else, please. May be List<> will work for you? Or, like struct TMP_TABLE_SHARE: TABLE_SHARE { TMP_TABLE_SHARE *next, *prev; } > > /** Category of this table. */ > TABLE_CATEGORY table_category; > diff --git a/sql/table_cache.cc b/sql/table_cache.cc > index 2dd368a..b307841 100644 > --- a/sql/table_cache.cc > +++ b/sql/table_cache.cc > @@ -589,6 +589,7 @@ void tdc_unlock_share(TDC_element *element) > key Table cache key > key_length Length of key > flags operation: what to open table or view > + out_table TABLE for the requested table please, put unrelated comment fixes in a separate commit (it's very easy to do with git citool) > > IMPLEMENTATION > Get a table definition from the table definition cache. > diff --git a/sql/table_cache.h b/sql/table_cache.h > index 2c5b0fc..ac36269 100644 > --- a/sql/table_cache.h > +++ b/sql/table_cache.h > @@ -1,3 +1,5 @@ > +#ifndef TABLE_CACHE_H_INCLUDED > +#define TABLE_CACHE_H_INCLUDED and this one too. commit all unrelated changes separately > /* Copyright (c) 2000, 2012, Oracle and/or its affiliates. > Copyright (c) 2010, 2011 Monty Program Ab > Copyright (C) 2013 Sergey Vojtovich and MariaDB Foundation > diff --git a/storage/myisam/mi_close.c b/storage/myisam/mi_close.c > index f0a82bc..23e01ef 100644 > --- a/storage/myisam/mi_close.c > +++ b/storage/myisam/mi_close.c > @@ -67,7 +67,7 @@ int mi_close(register MI_INFO *info) > if (share->kfile >= 0 && > flush_key_blocks(share->key_cache, share->kfile, > &share->dirty_part_map, > - ((share->temporary || share->deleting) ? > + ((share->deleting) ? Why? in all your myisam/aria changes (this one and others) you've enabled updating of on-disk data for temporary tables. like, flush keycache blocks, update file header, etc. Why should it be done persistently on disk? Temporary tables are not persistent, if mariadb crashes nobody will care what old temporary table files will contain. It should be enough to have all up-to-date information in memory. > FLUSH_IGNORE_CHANGED : > FLUSH_RELEASE))) > error=my_errno; > diff --git a/sql/temporary_tables.h b/sql/temporary_tables.h > new file mode 100644 > index 0000000..c345bea > --- /dev/null > +++ b/sql/temporary_tables.h > @@ -0,0 +1,112 @@ > +#ifndef TEMPORARY_TABLES_H > +#define TEMPORARY_TABLES_H > +/* > + Copyright (c) 2016 MariaDB Corporation > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA > +*/ > + > +#define TMP_TABLE_KEY_EXTRA 8 > + > +class Temporary_tables { > +public: > + Temporary_tables() : m_thd(0), m_table_shares(0), m_opened_tables(0) {} > + bool init(THD *thd); > + bool is_empty(); > + bool reset(); > + bool cleanup(); > + TABLE_SHARE *create_table(handlerton *hton, LEX_CUSTRING *frm, > + const char *path, const char *db, > + const char *table_name); > + TABLE_SHARE *find_table(const TABLE_LIST *tl); > + TABLE_SHARE *find_table_reduced_key_length(const char *key, uint key_length); > + TABLE_SHARE *find_table(const char *db, const char *table_name); > + TABLE *find_open_table(const char *db, const char *table_name); > + TABLE *create_and_use_table(handlerton *hton, LEX_CUSTRING *frm, > + const char *path, const char *db, > + const char *table_name, bool open_in_engine); > + TABLE *open_table(TABLE_SHARE *share, const char *alias, bool open_in_engine); > + bool open_table(TABLE_LIST *tl); > + bool open_tables(TABLE_LIST *tl); > + bool close_tables(bool all); > + bool rename_table(TABLE *table, const char *db, const char *table_name); > + bool drop_table(TABLE *table, bool *is_trans, bool delete_in_engine); > + void mark_tables_as_free_for_reuse(); > + > +private: > + uint create_table_def_key(char *key, > + const char *db, > + const char *table_name); > + TABLE_SHARE *find_table(const char *key, uint key_length); > + TABLE *find_open_table(const char *key, uint key_length); > + TABLE *open_table(const char *db, const char *table_name); > + bool close_table(TABLE *table); > + bool rm_temporary_table(handlerton *hton, const char *path); > + bool wait_for_prior_commit(); > + bool write_query_log_events(); > + void lock_tables(); > + void unlock_tables(); > + > + /* > + Return true if the table was created explicitly. > + */ > + bool is_user_table(TABLE_SHARE *share) > + { > + const char *name= share->table_name.str; > + return strncmp(name, tmp_file_prefix, tmp_file_prefix_length); > + } Is that right? I can do CREATE TEMPORARY TABLE `#sql-foo` (a int). the correct test needs to look at, for example, table->s->path or some flag, may be... Why wouldn't it use share->tmp_table? > + > + /* List operations */ > + template <class T> > + void link(T **list, T *element) > + { > + element->next= *list; > + if (element->next) > + element->next->prev= element; > + *list= element; > + (*list)->prev= 0; > + } > + > + template <class T> > + void unlink(T **list, T *element) > + { > + if (element->prev) > + { > + element->prev->next= element->next; > + if (element->prev->next) > + element->next->prev= element->prev; > + } > + else > + { > + DBUG_ASSERT(element == *list); > + > + *list= element->next; > + if (*list) > + element->next->prev= 0; > + } > + } > + > + uint tmpkeyval(TABLE_SHARE *share) > + { > + return uint4korr(share->table_cache_key.str + > + share->table_cache_key.length - 4); > + } > + > +private: > + THD *m_thd; Hmm, is that necessary? Temporary_tables class is instantiated in only one place - inside the THD. So Temporary_tables always belongs to its thd, and storing a point to the parent THD in the Temporary_tables you basically add another void* to already big THD class, while thid pointer stores no useful information at all. A hackish way to fix is is to replace m_thd with (((char*)this) - offsetof(THD, temporary_tables)) I'm not suggesting this hack, it is just to prove that m_thd is redundant. I hope you'll find a nicer and safer solution :) By the way, any increase in the THD size will most probably cause a test failure on labrador, you'll need to increase the DEFAULT_THREAD_STACK to fix it. Please check that - even if you remove m_thd there is still a m_table_shares pointer that increases THD size. > + TABLE_SHARE *m_table_shares; > + TABLE *m_opened_tables; > +}; > + > +#endif /* TEMPORARY_TABLES_H */ > diff --git a/sql/temporary_tables.cc b/sql/temporary_tables.cc > new file mode 100644 > index 0000000..5d8bce6 > --- /dev/null > +++ b/sql/temporary_tables.cc > @@ -0,0 +1,1265 @@ > +/* > + Copyright (c) 2016 MariaDB Corporation > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA > +*/ > + > +#include "sql_acl.h" /* TMP_TABLE_ACLS */ > +#include "sql_base.h" /* free_io_cache, > + get_table_def_key, > + tdc_create_key */ > +#include "lock.h" /* mysql_lock_remove */ > +#include "log_event.h" /* Query_log_event */ > +#include "sql_show.h" /* append_identifier */ > +#include "sql_handler.h" /* mysql_ha_rm_temporary_tables */ > +#include "temporary_tables.h" /* Temporary_tables */ > +#include "rpl_rli.h" /* rpl_group_info */ > + > + > +/* > + Initialize the Temporary_tables object. Currently it always returns > + false (success). > + > + @param thd [IN] Thread handle > + > + @return false Success > + true Error > +*/ > +bool Temporary_tables::init(THD *thd) > +{ > + DBUG_ENTER("Temporary_tables::init"); > + this->m_thd= thd; > + DBUG_RETURN(false); > +} > + > + > +/* > + Check whether temporary tables exist. The decision is made based on the > + existence of TABLE_SHAREs. > + > + @return false Temporary tables exist > + true No temporary table exist > +*/ > +bool Temporary_tables::is_empty() > +{ > + bool result; > + > + if (!m_thd) > + { > + return true; > + } > + > + rpl_group_info *rgi_slave= m_thd->rgi_slave; > + > + if (rgi_slave) > + { > + result= (rgi_slave->rli->save_temp_table_shares == NULL) ? true : false; > + } > + else > + { > + result= (m_table_shares == NULL) ? true : false; > + } > + > + return result; > +} > + > + > +/* > + Reset the Temporary_tables object. Currently, it always returns > + false (success). > + > + @return false Success > + true Error > +*/ > +bool Temporary_tables::reset() > +{ > + DBUG_ENTER("Temporary_tables::reset"); > + m_table_shares= 0; > + m_opened_tables= 0; > + DBUG_RETURN(false); > +} > + > + > +/* > + Cleanup the session's temporary tables by closing all open temporary tables > + as well as freeing the respective TABLE_SHAREs. > + It also writes "DROP TEMPORARY TABLE .." query log events to the binary log. > + > + Currently, it always returns false (success). > + > + @return false Success > + true Error > +*/ > +bool Temporary_tables::cleanup() > +{ > + DBUG_ENTER("Temporary_tables::cleanup"); > + > + TABLE_SHARE *share; > + TABLE_SHARE *next; > + lock_tables(); old code used to have here: if (!thd->temporary_tables) DBUG_RETURN(FALSE); DBUG_ASSERT(!thd->rgi_slave); you don't do either, instead you do lock_tables() (which is only needed if thd->rgi_slave != NULL). How comes? > + > + /* > + Ensure we don't have open HANDLERs for tables we are about to close. > + This is necessary when close_temporary_tables() is called as part s/close_temporary_tables/Temporary_tables::cleanup/ in the comment > + of execution of BINLOG statement (e.g. for format description event). > + */ > + mysql_ha_rm_temporary_tables(m_thd); > + > + // Close all open temporary tables. > + close_tables(true); > + > + // Write DROP TEMPORARY TABLE query log events to binary log. > + if (!m_thd->rgi_slave) another if (!m_thd->rgi_slave) while the old code had DBUG_ASSERT instead > + { > + write_query_log_events(); > + } > + > + // Free all TABLE_SHARES. > + share= m_table_shares; > + > + while (share) { > + next= share->next; > + rm_temporary_table(share->db_type(), share->path.str); > + > + /* Delete the share from table share list */ > + unlink<TABLE_SHARE>(&m_table_shares, share); > + > + free_table_share(share); > + my_free(share); > + > + /* Decrement Slave_open_temp_table_definitions status variable count. */ > + if (m_thd->rgi_slave) > + { > + thread_safe_decrement32(&slave_open_temp_table_definitions); > + } why did you put write_query_log_events() in a separate function? now you need to iterate the list of tables twice. > + > + share= next; > + } > + > + reset(); > + > + unlock_tables(); > + > + DBUG_RETURN(false); > +} > + > + > +/* > + Create a temporary table. > + > + @param hton [in] Handlerton > + @param frm [in] Binary frm image > + @param path [in] File path (without extension) > + @param db [in] Schema name > + @param table_name [in] Table name > + > + @return Success A pointer to table share object > + Failure NULL > +*/ > +TABLE_SHARE *Temporary_tables::create_table(handlerton *hton, LEX_CUSTRING *frm, > + const char *path, const char *db, > + const char *table_name) > +{ > + DBUG_ENTER("Temporary_tables::create_table"); > + > + TABLE_SHARE *share= NULL; > + char key_cache[MAX_DBKEY_LENGTH], *saved_key_cache, *tmp_path; > + uint key_length; > + int res; > + > + lock_tables(); > + > + if (wait_for_prior_commit()) > + { > + goto end; /* Failure */ > + } > + > + /* Create the table definition key for the temporary table. */ > + key_length= create_table_def_key(key_cache, db, table_name); > + > + if (!(share= (TABLE_SHARE *) my_malloc(sizeof(TABLE_SHARE) + strlen(path) + > + 1 + key_length, MYF(MY_WME)))) > + { > + goto end; /* Out of memory */ > + } > + > + tmp_path= (char *)(share + 1); > + saved_key_cache= strmov(tmp_path, path) + 1; > + memcpy(saved_key_cache, key_cache, key_length); > + > + init_tmp_table_share(m_thd, share, saved_key_cache, key_length, > + strend(saved_key_cache) + 1, tmp_path); > + > + share->db_plugin= ha_lock_engine(m_thd, hton); > + > + /* > + Prefer using frm image over file. The image might not be available in > + ALTER TABLE, when the discovering engine took over the ownership (see > + TABLE::read_frm_image). > + */ > + res= (frm->str) > + ? share->init_from_binary_frm_image(m_thd, false, frm->str, frm->length) > + : open_table_def(m_thd, share, GTS_TABLE | GTS_USE_DISCOVERY); > + > + if (res) > + { > + /* > + No need to lock share->mutex as this is not needed for temporary tables. > + */ > + free_table_share(share); > + my_free(share); > + share= NULL; > + goto end; > + } > + > + share->m_psi= PSI_CALL_get_table_share(true, share); > + > + /* Add share to the head of table share list. */ > + link<TABLE_SHARE>(&m_table_shares, share); > + > + /* Increment Slave_open_temp_table_definitions status variable count. */ > + if (m_thd->rgi_slave) > + { > + thread_safe_increment32(&slave_open_temp_table_definitions); > + } > + > +end: > + unlock_tables(); > + > + DBUG_RETURN(share); > +} > + > + > +/* > + Lookup the TABLE_SHARE using the given db/table_name.The server_id and > + pseudo_thread_id used to generate table definition key is taken from > + m_thd (see create_table_def_key()). Return NULL is none found. > + > + @return Success A pointer to table share object > + Failure NULL > +*/ > +TABLE_SHARE *Temporary_tables::find_table(const char *db, > + const char *table_name) > +{ > + DBUG_ENTER("Temporary_tables::find_table"); > + > + TABLE_SHARE *share; > + char key[MAX_DBKEY_LENGTH]; > + uint key_length; > + > + key_length= create_table_def_key(key, db, table_name); > + share= find_table(key, key_length); > + > + DBUG_RETURN(share); > +} > + > + > +/* > + Lookup TABLE_SHARE using the specified TABLE_LIST element. Return NULL is none > + found. > + > + @return Success A pointer to table share object > + Failure NULL > +*/ > +TABLE_SHARE *Temporary_tables::find_table(const TABLE_LIST *tl) > +{ > + DBUG_ENTER("Temporary_tables::find_table"); > + > + TABLE_SHARE *share; > + const char *tmp_key; > + char key[MAX_DBKEY_LENGTH]; > + uint key_length; > + > + key_length= get_table_def_key(tl, &tmp_key); > + memcpy(key, tmp_key, key_length); > + int4store(key + key_length, m_thd->variables.server_id); > + int4store(key + key_length + 4, m_thd->variables.pseudo_thread_id); > + key_length += TMP_TABLE_KEY_EXTRA; > + > + share= find_table(key, key_length); > + > + DBUG_RETURN(share); > +} > + > + > +/* > + Lookup TABLE_SHARE using the specified table definition key. Return NULL is > + none found. > + > + @return Success A pointer to table share object > + Failure NULL > +*/ > +TABLE_SHARE *Temporary_tables::find_table(const char *key, > + uint key_length) > +{ > + DBUG_ENTER("Temporary_tables::find_table"); > + > + TABLE_SHARE *share; > + TABLE_SHARE *result= NULL; > + > + lock_tables(); > + > + for (share= m_table_shares; share; share= share->next) > + { > + if (share->table_cache_key.length == key_length && > + !(memcmp(share->table_cache_key.str, key, key_length))) > + { > + result= share; > + break; > + } > + } > + > + unlock_tables(); > + > + DBUG_RETURN(result); > +} > + > + > +/* > + Lookup TABLE_SHARE based on the specified key. This key, however, is not > + the usual key used for temporary tables. It does not contain server_id & > + pseudo_thread_id. This function is essentially used use to check whether > + there is any temporary table which _shadows_ a base table. > + (see: Query_cache::send_result_to_client()) > + > + @return Success A pointer to table share object > + Failure NULL > +*/ > +TABLE_SHARE *Temporary_tables::find_table_reduced_key_length(const char *key, > + uint key_length) > +{ > + DBUG_ENTER("Temporary_tables::find_table_reduced_key_length"); > + > + TABLE_SHARE *share; > + TABLE_SHARE *result= NULL; > + > + lock_tables(); > + > + for (share= m_table_shares; share; share= share->next) > + { > + if ((share->table_cache_key.length - TMP_TABLE_KEY_EXTRA) == key_length && > + !(memcmp(share->table_cache_key.str, key, key_length))) > + { > + result= share; > + break; > + } > + } > + > + unlock_tables(); > + > + DBUG_RETURN(result); > +} > + > + > +/* > + Lookup the list of opened temporary tables using the specified > + db/table_name. Return NULL is none found. > + > + @return Success A pointer to table object > + Failure NULL > +*/ > +TABLE *Temporary_tables::find_open_table(const char *db, > + const char *table_name) > +{ > + DBUG_ENTER("Temporary_tables::find_open_table"); > + > + TABLE *table; > + char key[MAX_DBKEY_LENGTH]; > + uint key_length; > + > + if (wait_for_prior_commit()) > + { > + DBUG_RETURN(NULL); /* Failure */ > + } > + > + key_length= create_table_def_key(key, db, table_name); > + > + table= find_open_table(key, key_length); > + > + DBUG_RETURN(table); > +} > + > + > +/* > + Lookup the list of opened temporary tables using the specified > + key. Return NULL is none found. > + > + @return Success A pointer to table object > + Failure NULL > +*/ > +TABLE *Temporary_tables::find_open_table(const char *key, > + uint key_length) > +{ > + DBUG_ENTER("Temporary_tables::find_open_table"); > + > + TABLE *table, *result= NULL; > + > + for (table= m_opened_tables; table; table= table->next) > + { > + if (table->s->table_cache_key.length == key_length && > + !(memcmp(table->s->table_cache_key.str, key, key_length))) > + { > + result= table; > + break; > + } > + } > + > + DBUG_RETURN(result); > +} > + > + > +/* > + Create a temporary table, open it and return the TABLE handle. > + > + @param hton [in] Handlerton > + @param frm [in] Binary frm image > + @param path [in] File path (without extension) > + @param db [in] Schema name > + @param table_name [in] Table name > + @param open_in_engine [in] Whether open table in SE > + > + > + @return Success A pointer to table object > + Failure NULL > +*/ > +TABLE *Temporary_tables::create_and_use_table(handlerton *hton, better create_and_open() > + LEX_CUSTRING *frm, > + const char *path, > + const char *db, > + const char *table_name, > + bool open_in_engine) > +{ > + DBUG_ENTER("Temporary_tables::create_and_use_table"); > + > + TABLE_SHARE *share; > + TABLE *table; > + > + if (wait_for_prior_commit()) > + { > + DBUG_RETURN(NULL); /* Failure */ > + } > + > + if (!(share= create_table(hton, frm, path, db, table_name))) > + { > + DBUG_RETURN(NULL); > + } > + > + if ((table= open_table(share, table_name, open_in_engine))) > + { > + DBUG_RETURN(table); > + } > + > + DBUG_RETURN(NULL); > +} > + > + > +/* > + Open a table from the specified TABLE_SHARE with the given alias. > + > + @param share [in] Table share > + @param alias [in] Table alias > + @param open_in_engine [in] Whether open table in SE > + > + @return Success A pointer to table object > + Failure NULL > +*/ > +TABLE *Temporary_tables::open_table(TABLE_SHARE *share, > + const char *alias, > + bool open_in_engine) > +{ > + DBUG_ENTER("Temporary_tables::open_table"); > + > + TABLE *table; > + > + if (wait_for_prior_commit()) > + { > + DBUG_RETURN(NULL); /* Failure */ > + } > + > + if (!(table= (TABLE *) my_malloc(sizeof(TABLE), MYF(MY_WME)))) > + { > + DBUG_RETURN(NULL); /* Out of memory */ > + } > + > + if (open_table_from_share(m_thd, share, alias, > + (open_in_engine) ? > + (uint) (HA_OPEN_KEYFILE | HA_OPEN_RNDFILE | > + HA_GET_INDEX) : 0, > + (uint) (READ_KEYINFO | COMPUTE_TYPES | > + EXTRA_RECORD), > + ha_open_options, > + table, > + open_in_engine ? false : true)) > + { > + my_free(table); > + DBUG_RETURN(NULL); > + } > + > + table->reginfo.lock_type= TL_WRITE; /* Simulate locked */ > + table->grant.privilege= TMP_TABLE_ACLS; > + share->tmp_table= (table->file->has_transactions() ? > + TRANSACTIONAL_TMP_TABLE : NON_TRANSACTIONAL_TMP_TABLE); > + > + table->pos_in_table_list= 0; > + table->query_id= m_thd->query_id; > + > + lock_tables(); > + > + /* Add table to the head of table list. */ > + link<TABLE>(&m_opened_tables, table); > + > + /* Increment Slave_open_temp_table_definitions status variable count. */ > + if (m_thd->rgi_slave) > + { > + thread_safe_increment32(&slave_open_temp_tables); > + } > + > + unlock_tables(); > + > + DBUG_PRINT("tmptable", ("Opened table: '%s'.'%s' 0x%lx", table->s->db.str, > + table->s->table_name.str, (long) table)); > + DBUG_RETURN(table); > +} > + > + > +/* > + Lookup the table share list and open a table based on db/table_name. > + Return NULL if none found. > + > + @param db [in] Schema name > + @param table_name [in] Table name > + > + @return Success A pointer to table object > + Failure 0 > +*/ > +TABLE *Temporary_tables::open_table(const char *db, > + const char *table_name) > +{ > + DBUG_ENTER("Temporary_tables::open_table"); > + > + TABLE *result= 0; > + TABLE_SHARE *share; > + > + if ((share= find_table(db, table_name))) > + { > + result= open_table(share, table_name, true); > + } why this open_table(db, table_name) is doing so much less than open_table(table_list) below? > + > + DBUG_RETURN(result); > +} > + > + > +/* > + Lookup the table share list and open a table based on the specified > + TABLE_LIST element. Return false if the table was opened successfully. > + > + @param tl [in] TABLE_LIST > + > + @return false Success > + true Failure > +*/ > +bool Temporary_tables::open_table(TABLE_LIST *tl) > +{ > + DBUG_ENTER("Temporary_tables::open_table"); > + > + TABLE *table= NULL; > + TABLE_SHARE *share; > + > + /* > + Code in open_table() assumes that TABLE_LIST::table can be non-zero only > + for pre-opened temporary tables. > + */ > + DBUG_ASSERT(tl->table == NULL); > + > + /* > + This function should not be called for cases when derived or I_S > + tables can be met since table list elements for such tables can > + have invalid db or table name. > + Instead Temporary_tables::open_tables() should be used. > + */ > + DBUG_ASSERT(!tl->derived && !tl->schema_table); > + > + if (wait_for_prior_commit()) > + { > + DBUG_RETURN(true); /* Failure */ > + } why do you have it here? The old open_temporary_table() only had it below, you have it twice. > + > + lock_tables(); > + > + if (tl->open_type == OT_BASE_ONLY || m_table_shares == NULL) > + { > + DBUG_PRINT("info", ("skip_temporary is set or no temporary tables")); > + unlock_tables(); > + DBUG_RETURN(false); > + } > + > + unlock_tables(); > + > + if ((share= find_table(tl)) && > + (table= open_table(share, tl->get_table_name(), true))) Why would you open temporary tables all the time? old code didn't do that. temporary tables were automatically opened when created and automatically dropped when closed. I understand that you *might* need to open a second instance of a temporary table (but perhaps this can be avoided too), but why would close them again? just keep them open for reuse. In that case your open_table will just pick one unused TABLE from the m_opened_tables list > + { > + if (wait_for_prior_commit()) > + { > + DBUG_RETURN(true); /* Failure */ > + } > + > +#ifdef WITH_PARTITION_STORAGE_ENGINE > + if (tl->partition_names) > + { > + /* Partitioned temporary tables is not supported. */ > + DBUG_ASSERT(!table->part_info); > + my_error(ER_PARTITION_CLAUSE_ON_NONPARTITIONED, MYF(0)); > + DBUG_RETURN(true); > + } > +#endif > + > + table->query_id= m_thd->query_id; > + m_thd->thread_specific_used= true; > + /* It is neither a derived table nor non-updatable view. */ > + tl->updatable= true; > + tl->table= table; > + table->init(m_thd, tl); > + DBUG_RETURN(false); > + } > + > + if (!table && > + tl->open_type == OT_TEMPORARY_ONLY && > + tl->open_strategy == TABLE_LIST::OPEN_NORMAL) > + { > + my_error(ER_NO_SUCH_TABLE, MYF(0), tl->db, tl->table_name); > + DBUG_RETURN(true); > + } > + > + DBUG_RETURN(false); > +} > + > + > +/* > + Pre-open temporary tables corresponding to table list elements. > + > + @note One should finalize process of opening temporary tables > + by calling open_tables(). This function is responsible > + for table version checking and handling of merge tables. > + > + @param tl [in] TABLE_LIST > + > + @return false On success. If a temporary table exists > + for the given element, tl->table is set. > + true On error. my_error() has been called. > +*/ > +bool Temporary_tables::open_tables(TABLE_LIST *tl) > +{ > + DBUG_ENTER("Temporary_tables::open_tables"); > + > + TABLE_LIST *first_not_own; > + > + if (wait_for_prior_commit()) > + { > + DBUG_RETURN(NULL); /* Failure */ > + } why do you wait_for_prior_commit here? old open_temporary_tables didn't do that, because every individual open_table() below waits for prior commit too. > + > + first_not_own= m_thd->lex->first_not_own_table(); > + > + for (TABLE_LIST *table= tl; > + table && table != first_not_own; > + table= table->next_global) > + { > + if (table->derived || table->schema_table) > + { > + /* > + Derived and I_S tables will be handled by a later call to open_tables(). > + */ > + continue; > + } > + > + if ((m_thd->temporary_tables.open_table(table))) > + { > + DBUG_RETURN(true); > + } > + } > + > + DBUG_RETURN(false); > +} > + > + > +/* > + Close a temporary table. > + > + @param table [in] Table handle > + > + @return false Success > + true Error > +*/ > +bool Temporary_tables::close_table(TABLE *table) > +{ > + DBUG_ENTER("Temporary_tables::close_table"); > + DBUG_PRINT("tmptable", ("closing table: '%s'.'%s' 0x%lx alias: '%s'", > + table->s->db.str, table->s->table_name.str, > + (long) table, table->alias.c_ptr())); > + > + /* Delete the table from table list */ > + unlink<TABLE>(&m_opened_tables, table); > + > + free_io_cache(table); > + closefrm(table, false); > + my_free(table); > + > + /* Decrement Slave_open_temp_table_definitions status variable count. */ > + if (m_thd->rgi_slave) > + { > + thread_safe_decrement32(&slave_open_temp_tables); > + } > + > + DBUG_RETURN(false); > +} > + > +/* > + Close all the opened table. When 'all' is set to false, tables opened by > + handlers and ones with query_id different than that of m_thd will not be > + be closed. Currently, false (success) is always returned. > + > + @param all [in] Whether to close all tables? > + > + @return false Success > + true Failure > +*/ > +bool Temporary_tables::close_tables(bool all) > +{ > + TABLE *table; > + TABLE *next; > + > + table= m_opened_tables; > + > + while(table) { > + next= table->next; > + > + if (all || ((table->query_id == m_thd->query_id) && > + !(table->open_by_handler))) > + { > + mysql_lock_remove(m_thd, m_thd->lock, table); > + close_table(table); > + } > + > + table= next; > + } > + return false; > +} > + > + > +/* > + Write query log events with "DROP TEMPORARY TABLES .." for each pseudo > + thread to the binary log. > + > + @return false Success > + true Error > +*/ > +bool Temporary_tables::write_query_log_events() > +{ > + DBUG_ENTER("Temporary_tables::write_query_log_events"); > + DBUG_ASSERT(!m_thd->rgi_slave); > + > + TABLE_SHARE *share; > + TABLE_SHARE *next; > + TABLE_SHARE *prev_share; > + // Assume thd->variables.option_bits has OPTION_QUOTE_SHOW_CREATE. > + bool was_quote_show= true; > + bool error= 0; > + bool found_user_tables= false; > + // Better add "IF EXISTS" in case a RESET MASTER has been done. > + const char stub[]= "DROP /*!40005 TEMPORARY */ TABLE IF EXISTS "; > + char buf[FN_REFLEN]; > + > + /* > + Return in case there are no temporary tables or binary logging is > + disabled. > + */ > + if (!(m_table_shares && mysql_bin_log.is_open())) > + { > + DBUG_RETURN(false); > + } > + > + String s_query(buf, sizeof(buf), system_charset_info); > + s_query.copy(stub, sizeof(stub) - 1, system_charset_info); > + > + /* > + Insertion sort of temporary tables by pseudo_thread_id to build ordered > + list of sublists of equal pseudo_thread_id. > + */ why does it have to be sorted? (I know that the old code also sorted the list, but I still cannot understand why) > + > + for (prev_share= m_table_shares, share= prev_share->next; > + share; > + prev_share= share, share= share->next) > + { > + TABLE_SHARE *prev_sorted; /* Same as for prev_share */ > + TABLE_SHARE *sorted; > + > + if (is_user_table(share)) > + { > + if (!found_user_tables) > + found_user_tables= true; > + > + for (prev_sorted= NULL, sorted= m_table_shares; > + sorted != share; > + prev_sorted= sorted, sorted= sorted->next) > + { > + if (!is_user_table(sorted) || > + tmpkeyval(sorted) > tmpkeyval(share)) > + { > + /* > + Move into the sorted part of the list from the unsorted. > + */ > + prev_share->next= share->next; > + share->next= sorted; > + if (prev_sorted) > + { > + prev_sorted->next= share; > + } > + else > + { > + m_table_shares= share; > + } > + share= prev_share; > + break; > + } > + } > + } > + } > + > + /* > + We always quote db, table names though it is slight overkill. > + */ it's not an overkill, it's really needed > + if (found_user_tables && > + !(was_quote_show= MY_TEST(m_thd->variables.option_bits & > + OPTION_QUOTE_SHOW_CREATE))) > + { > + m_thd->variables.option_bits |= OPTION_QUOTE_SHOW_CREATE; > + } > + > + /* > + Scan sorted temporary tables to generate sequence of DROP. > + */ > + for (share= m_table_shares; share; share= next) > + { > + if (is_user_table(share)) > + { > + bool save_thread_specific_used= m_thd->thread_specific_used; > + my_thread_id save_pseudo_thread_id= m_thd->variables.pseudo_thread_id; > + char db_buf[FN_REFLEN]; > + String db(db_buf, sizeof(db_buf), system_charset_info); > + > + /* > + Set pseudo_thread_id to be that of the processed table. > + */ > + m_thd->variables.pseudo_thread_id= tmpkeyval(share); > + > + db.copy(share->db.str, share->db.length, system_charset_info); > + /* > + Reset s_query() if changed by previous loop. > + */ > + s_query.length(sizeof(stub) - 1); > + > + /* > + Loop forward through all tables that belong to a common database > + within the sublist of common pseudo_thread_id to create single > + DROP query. > + */ > + for (; > + share && is_user_table(share) && > + tmpkeyval(share) == m_thd->variables.pseudo_thread_id && > + share->db.length == db.length() && > + memcmp(share->db.str, db.ptr(), db.length()) == 0; > + share= next) > + { > + /* > + We are going to add ` around the table names and possible more > + due to special characters. > + */ > + append_identifier(m_thd, &s_query, share->table_name.str, > + strlen(share->table_name.str)); you don't need strlen(share->table_name.str) because you can write share->table_name.length instead :) (may be this applies to other strlen's too) > + s_query.append(','); > + next= share->next; > + } > + > + m_thd->clear_error(); > + CHARSET_INFO *cs_save= m_thd->variables.character_set_client; > + m_thd->variables.character_set_client= system_charset_info; > + m_thd->thread_specific_used= true; > + > + Query_log_event qinfo(m_thd, s_query.ptr(), > + s_query.length() - 1 /* to remove trailing ',' */, > + false, true, false, 0); > + qinfo.db= db.ptr(); > + qinfo.db_len= db.length(); > + m_thd->variables.character_set_client= cs_save; > + > + m_thd->get_stmt_da()->set_overwrite_status(true); > + if ((error= (mysql_bin_log.write(&qinfo) || error))) > + { > + /* > + If we're here following THD::cleanup, thence the connection > + has been closed already. So lets print a message to the > + error log instead of pushing yet another error into the > + stmt_da. > + > + Also, we keep the error flag so that we propagate the error > + up in the stack. This way, if we're the SQL thread we notice > + that close_temporary_tables failed. (Actually, the SQL > + thread only calls close_temporary_tables while applying old s/close_temporary_tables/Temporary_tables::cleanup/ in the comment > + Start_log_event_v3 events.) > + */ > + sql_print_error("Failed to write the DROP statement for " > + "temporary tables to binary log"); > + } > + > + m_thd->get_stmt_da()->set_overwrite_status(false); > + m_thd->variables.pseudo_thread_id= save_pseudo_thread_id; > + m_thd->thread_specific_used= save_thread_specific_used; > + } > + else > + { > + next= share->next; > + } > + } > + > + if (!was_quote_show) > + { > + /* > + Restore option. > + */ > + m_thd->variables.option_bits&= ~OPTION_QUOTE_SHOW_CREATE; > + } > + > + DBUG_RETURN(error); > +} > + > + > +/* > + Rename a temporary table. > + > + @param table [in] Table handle > + @param db [in] New schema name > + @param table_name [in] New table name > + > + @return false Success > + true Error > +*/ > +bool Temporary_tables::rename_table(TABLE *table, > + const char *db, > + const char *table_name) > +{ > + DBUG_ENTER("Temporary_tables::rename_table"); > + > + char *key; > + uint key_length; > + TABLE_SHARE *share= table->s; > + > + if (!(key= (char *) alloc_root(&share->mem_root, MAX_DBKEY_LENGTH))) > + { > + DBUG_RETURN(true); > + } > + > + /* > + Temporary tables are renamed by simply changing their table definition key. > + */ > + key_length= create_table_def_key(key, db, table_name); > + share->set_table_cache_key(key, key_length); > + > + DBUG_RETURN(false); > +} > + > + > +/* > + Drop a temporary table. > + > + Try to locate the table in the list of thd->temporary_tables. > + If the table is found: > + - If the table is being used by some outer statement, i.e. > + ref_count > 1, we only close the given table and return. > + - If the table is locked with LOCK TABLES or by prelocking, > + unlock it and remove it from the list of locked tables > + (THD::lock). Currently only transactional temporary tables > + are locked. > + - Close the temporary table, remove its .FRM. > + - Remove the table share from the list of temporary table shares. > + > + This function is used to drop user temporary tables, as well as > + internal tables created in CREATE TEMPORARY TABLE ... SELECT > + or ALTER TABLE. Even though part of the work done by this function > + is redundant when the table is internal, as long as we > + link both internal and user temporary tables into the same > + temporary tables list, it's impossible to tell here whether > + we're dealing with an internal or a user temporary table. why is it impossible? there is is_user_table() function. I know that it's broken, but it should be fixed - it's used elsewhere. > + > + @param thd [in] Thread handler > + @param table [in] Temporary table to be deleted > + @param is_trans [out] Is set to the type of the table: > + transactional (e.g. innodb) as true or > + non-transactional (e.g. myisam) as false. fix the comment to match the function prototype, please. don't forget to explain why do you need delete_in_engine parameter (e.g. delete_in_engine is false in ALTER TABLE) > + > + @retval 0 the table was found and dropped successfully. > + @retval -1 the table is in use by a outer query retval is wrong > +*/ > + > + > +/* > + @return false Table was either dropped or closed in > + case multiple open tables were found > + referring the table share. > + true Error > +*/ > +bool Temporary_tables::drop_table(TABLE *table, > + bool *is_trans, > + bool delete_in_engine) rename 'delete_in_engine' please, because it also includes removing the frm file, not only "in engine". I cannot think of a good name, sorry :( > +{ > + DBUG_ENTER("Temporary_tables::drop_table"); > + > + TABLE_SHARE *share; > + handlerton *hton; > + uint ref_count= 0; > + bool result; > + > + DBUG_ASSERT(table); > + DBUG_PRINT("tmptable", ("Dropping table: '%s'.'%s'", > + table->s->db.str, table->s->table_name.str)); > + old code had here /* Table might be in use by some outer statement. */ if (table->query_id && table->query_id != thd->query_id) { my_error(ER_CANT_REOPEN_TABLE, MYF(0), table->alias.c_ptr()); DBUG_RETURN(-1); } the intention is not to allow a temporary table being dropped from a trigger or a stored function, if the table is used by the outer statement. How do you solve this now? Apparently you do, somehow, because there is a test case for it. > + lock_tables(); > + > + if (is_trans) > + *is_trans= table->file->has_transactions(); > + > + share= table->s; > + hton= share->db_type(); > + > + /* > + Iterate over the list of open tables to find the number of tables > + referencing this table share. > + */ > + for (TABLE *tab= m_opened_tables; tab; tab= tab->next) > + { > + if (tab->s == share) > + { > + ref_count ++; Strictly speaking, you can store a list of TABLE's in the TABLE_SHARE - that is, make TABLE_SHARE have a pointer to the list of its own TABLE's. That'd be particularly easy, if you introduce TMP_TABLE_SHARE as I've mentioned above But on the other hand, we don't expect many temporary tables opened at the same time, so this list is supposed to be short, right? Then this optimization doesn't matter practically. > + } > + } > + > + DBUG_ASSERT(ref_count > 0); > + > + /* > + If LOCK TABLES list is not empty and contains this table, unlock the table > + and remove the table from this list. > + */ > + mysql_lock_remove(m_thd, m_thd->lock, table); > + > + if (close_table(table)) > + { > + result= true; > + goto end; > + } > + > + /* There are other tables referencing this table share. */ > + if (ref_count > 1) > + { > + result= false; so, it is not considered an error? why? > + goto end; > + } > + > + if (delete_in_engine) > + { > + rm_temporary_table(hton, share->path.str); > + } > + > + /* Delete the share from table share list */ > + unlink<TABLE_SHARE>(&m_table_shares, share); > + > + free_table_share(share); > + my_free(share); > + > + /* Decrement Slave_open_temp_table_definitions status variable count. */ > + if (m_thd->rgi_slave) > + { > + thread_safe_decrement32(&slave_open_temp_table_definitions); > + } > + > + result= false; > + > +end: > + unlock_tables(); > + > + DBUG_RETURN(result); > +} > + > + > +/* > + Create a table definition key. > + > + @param key [out] Buffer for the key to be created (must > + be of size MAX_DBKRY_LENGTH) > + @param db [in] Database name > + @param table_name [in] Table name > + > + @return Key length. > + > + @note > + The table key is create from: > + db + \0 > + table_name + \0 > + > + Additionally, we add the following to make each temporary table unique on > + the slave. > + > + 4 bytes of master thread id > + 4 bytes of pseudo thread id > +*/ > + > +uint Temporary_tables::create_table_def_key(char *key, const char *db, > + const char *table_name) > +{ > + DBUG_ENTER("Temporary_tables::create_table_def_key"); > + > + uint key_length; > + > + key_length= tdc_create_key(key, db, table_name); > + int4store(key + key_length, m_thd->variables.server_id); > + int4store(key + key_length + 4, m_thd->variables.pseudo_thread_id); > + key_length += TMP_TABLE_KEY_EXTRA; > + > + DBUG_RETURN(key_length); > +} > + > + > +/** > + Delete a temporary table. > + > + @param base [in] Handlerton for table to be deleted. > + @param path [in] Path to the table to be deleted (i.e. path > + to its .frm without an extension). > + > + @return false Success > + true Error > +*/ > +bool Temporary_tables::rm_temporary_table(handlerton *base, const char *path) > +{ > + bool error= false; > + handler *file; > + char frm_path[FN_REFLEN + 1]; > + > + DBUG_ENTER("Temporary_tables::rm_temporary_table"); > + > + strxnmov(frm_path, sizeof(frm_path) - 1, path, reg_ext, NullS); > + if (mysql_file_delete(key_file_frm, frm_path, MYF(0))) > + error= true; > + > + file= get_new_handler((TABLE_SHARE*) 0, current_thd->mem_root, base); > + if (file && file->ha_delete_table(path)) > + { > + error= true; > + sql_print_warning("Could not remove temporary table: '%s', error: %d", > + path, my_errno); > + } > + > + delete file; > + DBUG_RETURN(error); > +} > + > + > +bool Temporary_tables::wait_for_prior_commit() > +{ > + DBUG_ENTER("Temporary_tables::wait_for_prior_commit"); > + > + /* > + Temporary tables are not safe for parallel replication. They were > + designed to be visible to one thread only, so have no table locking. > + Thus there is no protection against two conflicting transactions > + committing in parallel and things like that. > + > + So for now, anything that uses temporary tables will be serialised > + with anything before it, when using parallel replication. > + > + TODO: We might be able to introduce a reference count or something > + on temp tables, and have slave worker threads wait for it to reach > + zero before being allowed to use the temp table. Might not be worth > + it though, as statement-based replication using temporary tables is > + in any case rather fragile. > + */ > + if (m_thd->rgi_slave && > + m_thd->rgi_slave->is_parallel_exec && > + m_thd->wait_for_prior_commit()) > + { > + DBUG_RETURN(true); > + } > + > + DBUG_RETURN(false); > +} > + > + > +void Temporary_tables::mark_tables_as_free_for_reuse() { > + TABLE *table; > + TABLE *next; > + > + DBUG_ENTER("mark_temp_tables_as_free_for_reuse"); > + > + if (m_thd->query_id == 0) > + { > + /* Thread has not executed any statement and has not used any tmp tables */ > + DBUG_VOID_RETURN; > + } > + > + lock_tables(); > + > + if (!m_thd->temporary_tables.is_empty()) do you need that? it checks whether m_table_shares or rgi_slave->rli->save_temp_table_shares is NULL. But after lock_tables() m_table_shares is equal to rgi_slave->rli->save_temp_table_shares, so it's enough to check m_table_shares. And if m_table_shares is NULL then m_opened_tables is also NULL, so the whole if() condition is redundant it only duplicates this while() below. > + { > + > + table= m_opened_tables; > + > + while(table) { > + next= table->next; > + > + if ((table->query_id == m_thd->query_id) && ! table->open_by_handler) > + { > + mysql_lock_remove(m_thd, m_thd->lock, table); > + close_table(table); I don't get it. Old code had mark_tmp_table_for_reuse() here, you have mysql_lock_remove and close_table. But temporary tables aren't locked, are they? And constantly closing/reopening temporary tables is a waste of resources, you can keep them open until they're dropped. > + } > + > + table= next; > + } > + } > + > + unlock_tables(); Old code had here: if (rgi_slave) { /* Temporary tables are shared with other by sql execution threads. As a safety messure, clear the pointer to the common area. */ thd->temporary_tables= 0; } shouldn't you have put it into unlock_tables() ? (yes, it was "messure" in the old comment, not my typo :) > + > + DBUG_VOID_RETURN; > +} > + > + > +void Temporary_tables::lock_tables() > +{ > + rpl_group_info *rgi_slave= m_thd->rgi_slave; > + if (rgi_slave) > + { > + mysql_mutex_lock(&rgi_slave->rli->data_lock); > + m_table_shares= rgi_slave->rli->save_temp_table_shares; > + } > +} > + > + > +void Temporary_tables::unlock_tables() > +{ > + rpl_group_info *rgi_slave= m_thd->rgi_slave; > + if (rgi_slave) > + { > + rgi_slave->rli->save_temp_table_shares= m_table_shares; > + mysql_mutex_unlock(&rgi_slave->rli->data_lock); > + } > +} > + Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 4

[Maria-developers] Effect of gcc optimization levels
by Sergey Vojtovich 17 May '16

17 May '16

Hi Wlad, Last week I promised to benchmark effect of various gcc optimization levels. Here're results: -O0 418TPS -Os 663TPS -O1 759TPS -O2 807TPS -O3 812TPS -O3 -unroll2 -ip 812TPS (default) -Ofast 819TPS Benchmark was done on a Sandy Bridge system running Ubuntu 12.10. Single thread OLTP RO, performance schema off. Regards, Sergey

1 0

[Maria-developers] Please review MDEV-6353 my_ismbchar() and my_mbcharlen() refactoring
by Alexander Barkov 17 May '16

17 May '16

Hi Sergei, Please review a partial patch for MDEV-6353. It removes my_mbcharlen() in all remaining pieces of the code, except LOAD DATA and SELECT INTO OUTFILE. I'll do LOAD DATA and SELECT INTO OUTFILE in a separate patch. Thanks.

2 4

[Maria-developers] MDEV-10035 DBUG_ASSERT on CREATE VIEW v1 AS SELECT * FROM t1 FOR UPDATE
by Alexander Barkov 16 May '16

16 May '16

Hi Sanja, Sergei, Sanja, you asked me to review a patch for MDEV-10035: revision-id: 4ffe2295e78538dde93df078421726f0c5a7d2a2 (mariadb-10.2.0-29-g4ffe229) parent(s): b79944950e5e5db40cf7ad49061edf5f105512c4 committer: Oleksandr Byelkin timestamp: 2016-05-15 15:25:33 +0200 message: I earlier also proposed the same idea to disallow FOR UPDATE, and even created a patch disallowing this in the parser syntactically (see attached). But Sergei worried that we should not do it this way and proposed some other solutions. Please see Sergei's comments in: MDEV-10063 VIEWs and subqueries with FOR UPDATE Sergei, are you still in doubts? Thanks.

3 9

Re: [Maria-developers] 7e70658: MDEV-9947: COM_MULTI united response
by Sergei Golubchik 14 May '16

14 May '16

Hi, Sanja! Looks ok to me. You're simply disable net_flush as far as I can see. For comments and questions, see below... On May 09, Oleksandr Byelkin wrote: > revision-id: 7e70658d1e88b4a255e3c28426f8259e5e596a85 (mariadb-10.2.0-20-g7e70658) > parent(s): c0a59b46be5be341bd6ffc9fe188a236ced46522 > committer: Oleksandr Byelkin > timestamp: 2016-05-09 15:26:18 +0200 > message: > > MDEV-9947: COM_MULTI united response > diff --git a/sql/protocol.cc b/sql/protocol.cc > index 9e52870..0bf2a05 100644 > --- a/sql/protocol.cc > +++ b/sql/protocol.cc > @@ -208,7 +208,8 @@ bool net_send_error(THD *thd, uint sql_errno, > const char *err, > bool > net_send_ok(THD *thd, > uint server_status, uint statement_warn_count, > - ulonglong affected_rows, ulonglong id, const char > *message) > + ulonglong affected_rows, ulonglong id, const char > *message, > + bool postpond_flush) "postpone" or "delay" or even (looking at how you use it) "skip" > { > NET *net= &thd->net; > uchar buff[MYSQL_ERRMSG_SIZE+10],*pos; > diff --git a/sql/sql_error.h b/sql/sql_error.h > index e03c3dd..98c8d77 100644 > --- a/sql/sql_error.h > +++ b/sql/sql_error.h > @@ -704,6 +704,12 @@ class Diagnostics_area > const char *message() const > { DBUG_ASSERT(m_status == DA_ERROR || m_status == DA_OK); return > m_message; } > > + bool postpond_flush() const > + { DBUG_ASSERT(m_status == DA_OK); return m_postpond_flush; } What if one uses COM_MULTI to send two commands and the first fails (results in an error). Will the second be executed? (This is a question about *sending*, client-to-server, not about MDEV-9947) > + > + void set_postpond_flush() > + { m_postpond_flush= TRUE; } > + > uint sql_errno() const > { DBUG_ASSERT(m_status == DA_ERROR); return m_sql_errno; } > > @@ -857,6 +863,8 @@ class Diagnostics_area > /** Set to make set_error_status after set_{ok,eof}_status > possible. */ > bool m_can_overwrite_status; A comment would've been nice... > + bool m_postpond_flush; > + > /** Message buffer. Can be used by OK or ERROR status. */ > char m_message[MYSQL_ERRMSG_SIZE]; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] Cassandra v2 Project Blog and git setup.
by Charles Muurmu 12 May '16

12 May '16

Hi everyone, I am not used to blogging but I have created a wordpress blog on which I will be making posts about my progress on this project with Spetrunia. Please feel free to criticize and suggest corrections. So far I have forked the code to my Github and my work will be based on 10.1. <https://github.com/charlesmuurmu/server> I have build the code on OS X El Capitan. I have also created a branch in which I will be working. I am currently going through some documentation and tutorials on the Datastax C++ driver. Regards, Biggy

2 3

Re: [Maria-developers] GSoC Student Introduction
by Sergey Petrunia 11 May '16

11 May '16

On Wed, May 11, 2016 at 08:26:49PM +0300, Sergey Petrunia wrote: > On Wed, Apr 27, 2016 at 07:44:11AM +0100, Charles Muurmu wrote: > > I am Charles Muurmu also known as laserlight on IRC and I was selected for > > GSoC 2016 to work on migrating MariaDB Cassandra storage engine from the > > Thrift API to Datastax driver. I will be mentored by Spetrunia. My goals > > for this community bonding period will be get myself ready by accomplishing > > the following tasks: > > > > Fork and clone the server code. > > > > Try to build the code. > > > > Setup a blog on which I will post weekly reports on progress. > > > > Study the Datastax C++ driver API and documentation. > > When CassandraSE V1 was in development, I was unable to use CQL, because Thrift API was such that the entire CQL resultset had to be materialized before it could be sent to the client. That is, running "select * from column_family" would cause the cassandra node and/or MySQL to die from attempting to use too much memory. I'm interested to learn how this is resolved in the Datastax driver. > > Study the ha_cassandra.xx code and storage engines architecture. > > > Good plan. > > Hopefully the last two items will give a better idea about how to structure the > coding (what to do as the first, second, etc milestone). Another question to think about is data models. Use of Thrift API calls in CassandraSE V1 meant that - secondary indexes could not be used - there were some other limitation on what datatypes could be used (I don't remember what they were exactly). Now, if we are using CQL, it should be possible to use those. Maybe there's also something else that I'm not aware of. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Please review MDEV-10051 Fix subselect to return a syntax error instead of "Incorrect usage of UNION and LIMIT"
by Alexander Barkov 11 May '16

11 May '16

Hello Sergei, Please review the next step for MDEV-8909 union parser cleanup Now it's MDEV-10051, and it fixes subselects. Note, I decided to fix derived tables in a separate patch later. So there will be one more patch blocking MDEV-8909. Thanks!

2 1

[Maria-developers] MDEV-10036 sql_yacc.yy: Split select_part2 to disallow syntactically bad constructs with INTO, PROCEDURE, UNION
by Alexander Barkov 10 May '16

10 May '16

Hi Sergei, Please review a patch for MDEV-10036. This is the next step for: MDEV-8909 union parser cleanup Thanks!

2 2

[Maria-developers] Please review MDEV-10030 sql_yacc.yy: Split table_expression and remove PROCEDURE from create_select, select_paren_derived, select_derived2, query_specification
by Alexander Barkov 09 May '16

09 May '16

Hi Sergei, Please review a patch for MDEV-10030, which is a part of MDEV-8909 union parser cleanup. Thanks!

2 4

[Maria-developers] GSoC 2016: Starting blog
by Галина Шалыгина 07 May '16

07 May '16

Hi everyone, I have created blog, where I'm going to write everything about my progress on project "Pushing conditions into non-mergeable views and derived tables in MariaDB". I've already written a post on what I've done, so if you are interested, you can check it out HERE <http://gsocmariadbshagalla.blogspot.ru/>. Best wishes, Galina Shalygina.

1 0

[Maria-developers] Joinable threads
by Sergey Vojtovich 06 May '16

06 May '16

Hi Monty, I vaguely recall you told me once that you like detached threads more than joinable, specifically for service threads. Could you remind me what were the reasons behind it? I need this to fix https://jira.mariadb.org/browse/MDEV-9994 properly. Thanks, Sergey

2 4

Re: [Maria-developers] [MariaDB/server] MDEV-9857: line cache size (to 10.1) (#169)
by Kristian Nielsen 06 May '16

06 May '16

Sergey Vojtovich <notifications(a)github.com> writes: > Since Alexey's patch wasn't yet accepted by MySQL we need his > confirmation that we can take it under BSD-new or MCA > terms. @akopytov, could you confirm? That _really_ is completely ridiculous! So here we have a patch. You will gladly accept it from MySQL/Oracle under GPL v2, without any special conditions. But the very same patch, submitted directly by the original author, you will not take??!? Now, obviously this is not something Sergey Vojtovich invented on his own, I doubt he even agrees personally. So Otto, you specifically asked me to be more vocal about MariaDB Foundation issue. Can we please have your personal opinions on this idea of BSD/contributer agreement for selected potential contributers? I have from the very start of MariaDB been strongly opposed to any form of contributer agreements or reciprocal licensing. Fortunately, we never really had it in practice, since all major contributions (MySQL@Oracle, XtraDB, oqgraph, sphinx, tokudb, galera, ...) were GPL only. But recently we have this message to potential contributers that the MariaDB Foundation "needs" to have BSD or MCA. This is extremely disrespectful for at least two reasons: - There clearly is no such "need", since the Foundation receives GPL rights only to a large part of new contributions to MariaDB - for example from Galera, MySQL@Oracle, Percona. - Selected contributers are except from this requirement. For example, according to my information, large parts of the MariaDB code to which the company SkySQL/MariaDB Corporation has full rights, they are _not_ required to provide BSD/MCA for to the MariaDB Foundation. Otto, I strongly encourage you to look to other successful Free Software projects such as the Linux Kernel, where an important part of the success is the deliberate choice of GPL-only _without_ any reciprocal licensing. I urge you to re-consider, and take a personal stance against this BSD/MCA nonsense, and work strongly for an official GPL-only contributer policy. Meanwhile, I encourage all contributers to _not_ accept this BSD/MCA requirement. If you disagree, be aware that there is no such strong requirement, and the major part of contributions to MariaDB are done without this requirement. If your contribution is accepted, but under condition of BSD/MCA, and you disagree, ask on the mailing list for someone else with commit rights to push your patch. I am sure there are several that will be willing to do so. - Kristian.

3 5

[Maria-developers] MDEV-9712 Performance degradation of nested NULLIF
by Alexander Barkov 05 May '16

05 May '16

Hello Sergei, Please have a look into a draft fix for MDEV-9712. The problem happens because in 10.1 execution time for various recursive stages (like walk, update_used_table and propagate_equal_fields) in NULLIF is O(recursion_level^2), because complexity is doubled on every recursion level when we copy args[0] to args[2]. This fix simplifies some stages to make them faster. It reduced execution time from 144 seconds to 12 seconds. The idea is just not to iterate through args[2] when it's just a copy of args[0] in these stages: - Item_func_nullif::walk - Item_func_nullif::update_used_tables I'm not fully sure that skipping args[2] is always correct though. For some transformers it may not be correct. <offtopic> Btw, walk() can be called from the partitioning code when arg_count is still 2 rather than 3. Looks suspicious. Perhaps there is a hidden bug behind that. </offtopic> The remaining heavy piece of the code is: Item* propagate_equal_fields(THD *thd, const Context &ctx, COND_EQUAL *cond) { Context cmpctx(ANY_SUBST, cmp.compare_type(), cmp.compare_collation()); args[0]->propagate_equal_fields_and_change_item_tree(thd, cmpctx, cond, &args[0]); args[1]->propagate_equal_fields_and_change_item_tree(thd, cmpctx, cond, &args[1]); args[2]->propagate_equal_fields_and_change_item_tree(thd, Context_identity(), cond, &args[2]); return this; } We cannot use the same approach here, because args[0] and args[2] are propagated with a different context (ANY_SUBST vs IDENTITY_SUBST). Not sure what to do with this. Perhaps, data types could have an extra attribute to tell that ANY_SUBST and IDENITTY_SUBST are equal. So, for example, for non-ZEROFILL integers, ANY_SUBST and IDENTITY_SUBST do the same thing and thus propagation could be done one time only. But this would be a partial solution, for a limited number of data types. Another option is to leave propagation as is. Another option is do nothing at all. If we rewrite the reported query as CASE, it will look heavy indeed. So it's Ok to expect heavy execution complexity. Thanks.

2 3

[Maria-developers] UNION + INTO OUTFILE is confusing
by Alexander Barkov 04 May '16

04 May '16

Hi Sergei, If I run this query: SELECT 1 UNION SELECT 2 INTO OUTFILE 'test.txt'; It creates a file 'test.txt' with this content: 1 2 Looks fine so far. Now if I do "rm test.txt" and execute another query (with parentheses): (SELECT 1) UNION (SELECT 2 INTO OUTFILE 'test.txt'); it still puts the same two records from both UNION parts! This looks confusing, as parentheses a kind of assume that only the right UNION part is to be exported. Exporting only one UNION part is not supported, IIRC. So the above query should probably return a syntax error. But now there is a new trouble. There is no syntax like this: (SELECT ...) UNION (SELECT ...) INTO OUTFILE 'test.txt'; So instead of: (SELECT ...) UNION (SELECT ... INTO OUTFILE 'test.txt'); one will have to write this: SELECT * INTO OUTFILE 'test.txt' FROM ((SELECT ...) UNION (SELECT ...)) AS t1; The latter looks longer, but at least it's not confusing as the former.

2 2

Re: [Maria-developers] [Commits] b1ddc7d: MDEV-9487: Server crashes in Time_and_counter_tracker::incr_loops with UNION in ALL subquery
by Sergey Petrunia 04 May '16

04 May '16

Hi Sanja, I have one comment: the patch introduces JOIN::in_optimize which can have values of 1) not initialized and 2) false. This is clearly redundant. Ok to push after the above is addressed. On Sun, Feb 21, 2016 at 10:12:25PM +0100, OleksandrByelkin wrote: > revision-id: b1ddc7d546e6b147838af72dd03f86a8b272fdf0 (mariadb-10.1.11-18-gb1ddc7d) > parent(s): fd8e846a3b049903706267d58e6d8e61eea97df8 > committer: Oleksandr Byelkin > timestamp: 2016-02-21 22:12:25 +0100 > message: > > MDEV-9487: Server crashes in Time_and_counter_tracker::incr_loops with UNION in ALL subquery > > Do not mark subquery as inexpensive when it is not optimized. > > --- > mysql-test/r/derived_view.result | 6 +++--- > mysql-test/r/subselect.result | 20 +++++++++++++++++--- > mysql-test/r/subselect_no_exists_to_in.result | 20 +++++++++++++++++--- > mysql-test/r/subselect_no_mat.result | 20 +++++++++++++++++--- > mysql-test/r/subselect_no_opts.result | 20 +++++++++++++++++--- > mysql-test/r/subselect_no_scache.result | 20 +++++++++++++++++--- > mysql-test/r/subselect_no_semijoin.result | 20 +++++++++++++++++--- > mysql-test/r/type_year.result | 1 + > mysql-test/t/subselect.test | 11 +++++++++++ > sql/item_subselect.cc | 20 ++++++++++++++++---- > sql/sql_select.h | 2 ++ > 11 files changed, 135 insertions(+), 25 deletions(-) > > diff --git a/mysql-test/r/derived_view.result b/mysql-test/r/derived_view.result > index 639942f..5783247 100644 > --- a/mysql-test/r/derived_view.result > +++ b/mysql-test/r/derived_view.result > @@ -1101,7 +1101,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra > 1 PRIMARY t3 system NULL NULL NULL NULL 1 100.00 > 2 SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL No tables used > Warnings: > -Note 1003 select 6 AS `a`,5 AS `b` from `test`.`t3` where (not(<expr_cache><6,5>(<in_optimizer>((6,5),<exists>(select 7,5 having (trigcond(((<cache>(6) = 7) or isnull(7))) and trigcond(((<cache>(5) = 5) or isnull(5))) and trigcond(<is_not_null_test>(7)) and trigcond(<is_not_null_test>(5)))))))) > +Note 1003 select 6 AS `a`,5 AS `b` from `test`.`t3` where 1 > SELECT t.a,t.b FROM t3 RIGHT JOIN ((SELECT * FROM t1) AS t, t2) ON t2.b != 0 > WHERE (t.a,t.b) NOT IN (SELECT 7, 5); > a b > @@ -1115,7 +1115,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra > 1 PRIMARY t3 system NULL NULL NULL NULL 1 100.00 > 3 SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL No tables used > Warnings: > -Note 1003 select 6 AS `a`,5 AS `b` from `test`.`t3` where (not(<expr_cache><6,5>(<in_optimizer>((6,5),<exists>(select 7,5 having (trigcond(((<cache>(6) = 7) or isnull(7))) and trigcond(((<cache>(5) = 5) or isnull(5))) and trigcond(<is_not_null_test>(7)) and trigcond(<is_not_null_test>(5)))))))) > +Note 1003 select 6 AS `a`,5 AS `b` from `test`.`t3` where 1 > SELECT t.a,t.b FROM t3 RIGHT JOIN (v1 AS t, t2) ON t2.b != 0 > WHERE (t.a,t.b) NOT IN (SELECT 7, 5); > a b > @@ -1129,7 +1129,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra > 1 PRIMARY t3 system NULL NULL NULL NULL 1 100.00 > 2 SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL No tables used > Warnings: > -Note 1003 select 6 AS `a`,5 AS `b` from `test`.`t3` where (not(<expr_cache><6,5>(<in_optimizer>((6,5),<exists>(select 7,5 having (trigcond(((<cache>(6) = 7) or isnull(7))) and trigcond(((<cache>(5) = 5) or isnull(5))) and trigcond(<is_not_null_test>(7)) and trigcond(<is_not_null_test>(5)))))))) > +Note 1003 select 6 AS `a`,5 AS `b` from `test`.`t3` where 1 > DROP VIEW v1; > DROP TABLE t1,t2,t3; > # > diff --git a/mysql-test/r/subselect.result b/mysql-test/r/subselect.result > index 75c8597..90d8f9f 100644 > --- a/mysql-test/r/subselect.result > +++ b/mysql-test/r/subselect.result > @@ -6805,7 +6805,9 @@ FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR alias1.a = 'y' > HAVING field>'B' AND ( 'Moscow' ) IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible HAVING > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > +1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) AS field > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > @@ -6817,8 +6819,8 @@ SELECT MAX( alias2.a ) > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR ('Moscow') IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY alias1 index a a 19 NULL 11 Using where; Using index > -1 PRIMARY alias2 ref a a 19 test.alias1.a 2 Using index > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > 1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) > @@ -7133,3 +7135,15 @@ sq > NULL > drop view v2; > drop table t1,t2; > +# > +# MDEV-9487: Server crashes in Time_and_counter_tracker::incr_loops > +# with UNION in ALL subquery > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f VARCHAR(8)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('foo'); > +SELECT f FROM t1 WHERE f > ALL ( SELECT 'bar' UNION SELECT 'baz' ); > +f > +foo > +drop table t1; > +SET NAMES default; > diff --git a/mysql-test/r/subselect_no_exists_to_in.result b/mysql-test/r/subselect_no_exists_to_in.result > index e6238af..dcceb61 100644 > --- a/mysql-test/r/subselect_no_exists_to_in.result > +++ b/mysql-test/r/subselect_no_exists_to_in.result > @@ -6805,7 +6805,9 @@ FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR alias1.a = 'y' > HAVING field>'B' AND ( 'Moscow' ) IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible HAVING > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > +1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) AS field > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > @@ -6817,8 +6819,8 @@ SELECT MAX( alias2.a ) > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR ('Moscow') IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY alias1 index a a 19 NULL 11 Using where; Using index > -1 PRIMARY alias2 ref a a 19 test.alias1.a 2 Using index > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > 1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) > @@ -7133,6 +7135,18 @@ sq > NULL > drop view v2; > drop table t1,t2; > +# > +# MDEV-9487: Server crashes in Time_and_counter_tracker::incr_loops > +# with UNION in ALL subquery > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f VARCHAR(8)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('foo'); > +SELECT f FROM t1 WHERE f > ALL ( SELECT 'bar' UNION SELECT 'baz' ); > +f > +foo > +drop table t1; > +SET NAMES default; > set optimizer_switch=default; > select @@optimizer_switch like '%exists_to_in=off%'; > @@optimizer_switch like '%exists_to_in=off%' > diff --git a/mysql-test/r/subselect_no_mat.result b/mysql-test/r/subselect_no_mat.result > index 70edc64..4e9f750 100644 > --- a/mysql-test/r/subselect_no_mat.result > +++ b/mysql-test/r/subselect_no_mat.result > @@ -6800,7 +6800,9 @@ FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR alias1.a = 'y' > HAVING field>'B' AND ( 'Moscow' ) IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible HAVING > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > +1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) AS field > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > @@ -6812,8 +6814,8 @@ SELECT MAX( alias2.a ) > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR ('Moscow') IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY alias1 index a a 19 NULL 11 Using where; Using index > -1 PRIMARY alias2 ref a a 19 test.alias1.a 2 Using index > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > 1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) > @@ -7126,6 +7128,18 @@ sq > NULL > drop view v2; > drop table t1,t2; > +# > +# MDEV-9487: Server crashes in Time_and_counter_tracker::incr_loops > +# with UNION in ALL subquery > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f VARCHAR(8)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('foo'); > +SELECT f FROM t1 WHERE f > ALL ( SELECT 'bar' UNION SELECT 'baz' ); > +f > +foo > +drop table t1; > +SET NAMES default; > set optimizer_switch=default; > select @@optimizer_switch like '%materialization=on%'; > @@optimizer_switch like '%materialization=on%' > diff --git a/mysql-test/r/subselect_no_opts.result b/mysql-test/r/subselect_no_opts.result > index c89fd13..4b09fd2 100644 > --- a/mysql-test/r/subselect_no_opts.result > +++ b/mysql-test/r/subselect_no_opts.result > @@ -6796,7 +6796,9 @@ FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR alias1.a = 'y' > HAVING field>'B' AND ( 'Moscow' ) IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible HAVING > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > +1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) AS field > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > @@ -6808,8 +6810,8 @@ SELECT MAX( alias2.a ) > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR ('Moscow') IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY alias1 index a a 19 NULL 11 Using where; Using index > -1 PRIMARY alias2 ref a a 19 test.alias1.a 2 Using index > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > 1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) > @@ -7124,4 +7126,16 @@ sq > NULL > drop view v2; > drop table t1,t2; > +# > +# MDEV-9487: Server crashes in Time_and_counter_tracker::incr_loops > +# with UNION in ALL subquery > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f VARCHAR(8)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('foo'); > +SELECT f FROM t1 WHERE f > ALL ( SELECT 'bar' UNION SELECT 'baz' ); > +f > +foo > +drop table t1; > +SET NAMES default; > set @optimizer_switch_for_subselect_test=null; > diff --git a/mysql-test/r/subselect_no_scache.result b/mysql-test/r/subselect_no_scache.result > index b12bf21..322e64d 100644 > --- a/mysql-test/r/subselect_no_scache.result > +++ b/mysql-test/r/subselect_no_scache.result > @@ -6811,7 +6811,9 @@ FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR alias1.a = 'y' > HAVING field>'B' AND ( 'Moscow' ) IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible HAVING > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > +1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) AS field > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > @@ -6823,8 +6825,8 @@ SELECT MAX( alias2.a ) > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR ('Moscow') IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY alias1 index a a 19 NULL 11 Using where; Using index > -1 PRIMARY alias2 ref a a 19 test.alias1.a 2 Using index > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > 1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) > @@ -7139,6 +7141,18 @@ sq > NULL > drop view v2; > drop table t1,t2; > +# > +# MDEV-9487: Server crashes in Time_and_counter_tracker::incr_loops > +# with UNION in ALL subquery > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f VARCHAR(8)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('foo'); > +SELECT f FROM t1 WHERE f > ALL ( SELECT 'bar' UNION SELECT 'baz' ); > +f > +foo > +drop table t1; > +SET NAMES default; > set optimizer_switch=default; > select @@optimizer_switch like '%subquery_cache=on%'; > @@optimizer_switch like '%subquery_cache=on%' > diff --git a/mysql-test/r/subselect_no_semijoin.result b/mysql-test/r/subselect_no_semijoin.result > index 54f145d..d380528 100644 > --- a/mysql-test/r/subselect_no_semijoin.result > +++ b/mysql-test/r/subselect_no_semijoin.result > @@ -6796,7 +6796,9 @@ FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR alias1.a = 'y' > HAVING field>'B' AND ( 'Moscow' ) IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible HAVING > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > +1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) AS field > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > @@ -6808,8 +6810,8 @@ SELECT MAX( alias2.a ) > FROM t1 AS alias1, t1 AS alias2, t1 AS alias3 > WHERE alias1.a = alias2.a OR ('Moscow') IN ( SELECT a FROM t1 ); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY alias1 index a a 19 NULL 11 Using where; Using index > -1 PRIMARY alias2 ref a a 19 test.alias1.a 2 Using index > +1 PRIMARY alias1 index a a 19 NULL 11 Using index > +1 PRIMARY alias2 index a a 19 NULL 11 Using where; Using index; Using join buffer (flat, BNL join) > 1 PRIMARY alias3 index NULL a 19 NULL 11 Using index; Using join buffer (flat, BNL join) > 2 SUBQUERY t1 index_subquery a a 19 const 1 Using index; Using where > SELECT MAX( alias2.a ) > @@ -7124,5 +7126,17 @@ sq > NULL > drop view v2; > drop table t1,t2; > +# > +# MDEV-9487: Server crashes in Time_and_counter_tracker::incr_loops > +# with UNION in ALL subquery > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f VARCHAR(8)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('foo'); > +SELECT f FROM t1 WHERE f > ALL ( SELECT 'bar' UNION SELECT 'baz' ); > +f > +foo > +drop table t1; > +SET NAMES default; > set @optimizer_switch_for_subselect_test=null; > set @join_cache_level_for_subselect_test=NULL; > diff --git a/mysql-test/r/type_year.result b/mysql-test/r/type_year.result > index 842a16e..204cec2 100644 > --- a/mysql-test/r/type_year.result > +++ b/mysql-test/r/type_year.result > @@ -387,6 +387,7 @@ a > 00 > select a from t1 where a=(select 2000 from dual where 1); > a > +00 > select a from t1 where a=y2k(); > a > 00 > diff --git a/mysql-test/t/subselect.test b/mysql-test/t/subselect.test > index a862870..f71c215 100644 > --- a/mysql-test/t/subselect.test > +++ b/mysql-test/t/subselect.test > @@ -5989,3 +5989,14 @@ SELECT ( SELECT MIN(v2.f2) FROM t1 ) AS sq FROM v2 GROUP BY sq; > > drop view v2; > drop table t1,t2; > + > +--echo # > +--echo # MDEV-9487: Server crashes in Time_and_counter_tracker::incr_loops > +--echo # with UNION in ALL subquery > +--echo # > +SET NAMES utf8; > +CREATE TABLE t1 (f VARCHAR(8)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('foo'); > +SELECT f FROM t1 WHERE f > ALL ( SELECT 'bar' UNION SELECT 'baz' ); > +drop table t1; > +SET NAMES default; > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index d4e3e6c..f45d83d 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -561,22 +561,34 @@ bool Item_subselect::is_expensive() > for (SELECT_LEX *sl= unit->first_select(); sl; sl= sl->next_select()) > { > JOIN *cur_join= sl->join; > + > + /* not optimized subquery */ > if (!cur_join) > - continue; > + return true; > + > + /* very simple subquery */ > + if (!cur_join->tables_list && !sl->first_inner_unit()) > + return false; > + > + /* > + If the subquery is not optimised or in the process of optimization > + it supposed to be expensive > + */ > + if (!cur_join->optimized || cur_join->in_optimize) > + return true; > > /* > Subqueries whose result is known after optimization are not expensive. > Such subqueries have all tables optimized away, thus have no join plan. > */ > - if (cur_join->optimized && > - (cur_join->zero_result_cause || !cur_join->tables_list)) > + if ((cur_join->zero_result_cause || !cur_join->tables_list)) > return false; > > /* > If a subquery is not optimized we cannot estimate its cost. A subquery is > considered optimized if it has a join plan. > */ > - if (!(cur_join->optimized && cur_join->join_tab)) > + if (!cur_join->join_tab) > return true; > > if (sl->first_inner_unit()) > diff --git a/sql/sql_select.h b/sql/sql_select.h > index 9f90473..e57844fd 100644 > --- a/sql/sql_select.h > +++ b/sql/sql_select.h > @@ -1292,6 +1292,7 @@ class JOIN :public Sql_alloc > OPTIMIZATION_IN_PROGRESS=1, > OPTIMIZATION_DONE=2}; > bool optimized; ///< flag to avoid double optimization in EXPLAIN > + bool in_optimize; > bool initialized; ///< flag to avoid double init_execution calls > > Explain_select *explain; > @@ -1380,6 +1381,7 @@ class JOIN :public Sql_alloc > ref_pointer_array_size= 0; > zero_result_cause= 0; > optimized= 0; > + in_optimize= 0; > have_query_plan= QEP_NOT_PRESENT_YET; > initialized= 0; > cleaned= 0; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 2

Re: [Maria-developers] c75cbc1: MDEV-717 LP:1003679 - Wrong binlog order on concurrent DROP schema and
by Sergei Golubchik 04 May '16

04 May '16

Hi, Alexey! convention below: when I start a comment with "btw" it means that you don't need to change anything, it's just a thought I had when looking at that line in the patch. On May 03, Alexey Botchkov wrote: > revision-id: c75cbc19806223be8e750c27ba0bd4b17ef61f54 (mariadb-10.1.13-24-gc75cbc1) > parent(s): 94cd0f6c9b3b04db67501ef29d470f32527ceda2 > committer: Alexey Botchkov > timestamp: 2016-05-03 13:20:12 +0400 > message: > > MDEV-717 LP:1003679 - Wrong binlog order on concurrent DROP schema and > CREATE function. > > Test case added. That required setting some DEBUG_SYNC points. > > diff --git a/mysql-test/suite/binlog/t/binlog_mdev717.test b/mysql-test/suite/binlog/t/binlog_mdev717.test > new file mode 100644 > index 0000000..5e8cce6 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_mdev717.test > @@ -0,0 +1,49 @@ > +# MDEV-717 LP:1003679 - Wrong binlog order on concurrent DROP schema and CREATE function. > + > +--source include/have_log_bin.inc > +RESET MASTER; > + > +disable_warnings; > +DROP DATABASE IF EXISTS mysqltest; > +enable_warnings; btw, generally this isn't necessary anymore, after-test checks verify that every test cleans up after itself. > +connect(con1,localhost,root,,); btw, this works also without extra commas: connect(con1,localhost,root); > +connection default; > + > +CREATE DATABASE mysqltest; > +SET DEBUG_SYNC= "wait_drop_db_name_locked SIGNAL locked WAIT_FOR release"; > +--send DROP DATABASE mysqltest; > +connection con1; > +SET DEBUG_SYNC= "now WAIT_FOR locked"; Hm... If the default connection signals "locked" before you start waiting for it now - will your test still work? You can test it by adding a sleep right after "connection con1". (don't commit the sleep, of course, it's just to way to verify whether the test case is valid) Alternatively, in many places tests wait for a connection to reach the certain debug_sync point with, like: let $wait_condition= select count(*) = 1 from information_schema.processlist where state like "debug sync point: wait_drop_db_name_locked%"; source include/wait_condition.inc; > +SET DEBUG_SYNC= "sp_create_routine_started SIGNAL release"; > +--error 0,ER_BAD_DB_ERROR why? Do you mean that even with your sync points the specific execution order if not guaranteed? > +CREATE FUNCTION mysqltest.f1() RETURNS INT RETURN 1; > +connection default; > +--reap > + > +CREATE DATABASE mysqltest; > +SET DEBUG_SYNC= "wait_drop_db_name_locked SIGNAL locked1 WAIT_FOR release1"; > +--send DROP DATABASE mysqltest; > +connection con1; > +SET DEBUG_SYNC= "now WAIT_FOR locked1"; > +SET DEBUG_SYNC= "create_event_started SIGNAL release1"; > +--error 0,ER_BAD_DB_ERROR > +CREATE EVENT mysqltest.e1 ON SCHEDULE EVERY 15 MINUTE DO BEGIN END; > +connection default; > +--reap > + > +CREATE DATABASE mysqltest; > +CREATE EVENT mysqltest.e1 ON SCHEDULE EVERY 15 MINUTE DO BEGIN END; > +SET DEBUG_SYNC= "wait_drop_db_name_locked SIGNAL locked1 WAIT_FOR release1"; > +--send DROP DATABASE mysqltest; > +connection con1; > +SET DEBUG_SYNC= "now WAIT_FOR locked1"; > +SET DEBUG_SYNC= "update_event_started SIGNAL release1"; > +--error 0,ER_BAD_DB_ERROR > +ALTER EVENT mysqltest.e1 ON SCHEDULE EVERY 20 MINUTE DO BEGIN END; > +connection default; > +--reap > + > +SET DEBUG_SYNC= "RESET"; > +--source include/show_binlog_events.inc > + > diff --git a/sql/events.cc b/sql/events.cc > index 77dbb8b..3c50d6e 100644 > diff --git a/sql/sp.cc b/sql/sp.cc > index a518b52..c2e3fd2 100644 > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -1043,6 +1043,8 @@ sp_create_routine(THD *thd, stored_procedure_type type, sp_head *sp) > DBUG_ASSERT(type == TYPE_ENUM_PROCEDURE || > type == TYPE_ENUM_FUNCTION); > > + DEBUG_SYNC(thd, "sp_create_routine_started"); > + could you use "before_wait_locked_pname" sync point instead? It's at the beginning of lock_object_name(). > /* Grab an exclusive MDL lock. */ > if (lock_object_name(thd, mdl_type, sp->m_db.str, sp->m_name.str)) > { > diff --git a/sql/sql_db.cc b/sql/sql_db.cc > index 2ba67cb..fdb49f2 100644 > --- a/sql/sql_db.cc > +++ b/sql/sql_db.cc > @@ -837,6 +837,8 @@ mysql_rm_db_internal(THD *thd,char *db, bool if_exists, bool silent) > if (lock_schema_name(thd, dbnorm)) > DBUG_RETURN(true); > > + DEBUG_SYNC(thd, "wait_drop_db_name_locked"); > + Could you use "after_wait_locked_schema_name" sync point instead? It's the one at the end of lock_schema_name(). > length= build_table_filename(path, sizeof(path) - 1, db, "", "", 0); > strmov(path+length, MY_DB_OPT_FILE); // Append db option file name > del_dbopt(path); // Remove dboption hash entry Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] [Commits] a6fb98f: MDEV-6368: assertion xid_seqno > trx_sys_cur_xid_seqno
by Jan Lindström 03 May '16

03 May '16

Hi Nirbhay, I'm not expert on this code area but some questions, comments below. On Tue, May 3, 2016 at 1:26 AM, Nirbhay Choubey <nirbhay(a)mariadb.com> wrote: > revision-id: a6fb98fd74ab6f14fab0c1ef4630b010ff28880f > (mariadb-10.1.13-6-ga6fb98f) > parent(s): f8cdb9e7e8fb692f8eb3b9bb4792cd60653e0eb8 > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-05-02 18:26:58 -0400 > message: > > MDEV-6368: assertion xid_seqno > trx_sys_cur_xid_seqno > > - Validate the specified wsrep_start_position value by also > checking the return status of wsrep->sst_received. This also > ensures that changes in wsrep_start_position is not allowed > when the node is not in JOINING state. > - Do not allow decrease in seqno within same UUID. > - The initial checkpoint in SEs should be [0...:-1]. > > > diff --git a/sql/wsrep_sst.cc b/sql/wsrep_sst.cc > index 4f6cc51..44fdd03 100644 > --- a/sql/wsrep_sst.cc > +++ b/sql/wsrep_sst.cc > @@ -264,42 +264,91 @@ void wsrep_sst_complete (const wsrep_uuid_t* > sst_uuid, > mysql_mutex_unlock (&LOCK_wsrep_sst); > } > > -void wsrep_sst_received (wsrep_t* const wsrep, > +/* > + If wsrep provider is loaded, inform that the new state snapshot > + has been received. Also update the local checkpoint. > + > + @param wsrep [IN] wsrep handle > + @param uuid [IN] Initial state UUID > + @param seqno [IN] Initial state sequence number > + @param state [IN] Always NULL, also ignored by wsrep > provider (?) > + @param state_len [IN] Always 0, also ignored by wsrep > provider (?) > + @param implicit [IN] Whether invoked implicitly due to > SST > + (true) or explicitly because if > change > + in wsrep_start_position by user > (false). > + @return false Success > + true Error > + > +*/ > +bool wsrep_sst_received (wsrep_t* const wsrep, > const wsrep_uuid_t& uuid, > wsrep_seqno_t const seqno, > const void* const state, > - size_t const state_len) > Why not const size_t state_len ? Use of const here is confusing. > + size_t const state_len, > + bool const implicit) > { > - wsrep_get_SE_checkpoint(local_uuid, local_seqno); > + /* > + To keep track of whether the local uuid:seqno should be updated. > Also, note > + that local state (uuid:seqno) is updated/checkpointed only after we > get an > + OK from wsrep provider. By doing so, the values remain consistent > across > + the server & wsrep provider. > + */ > + bool do_update= false; > + > + // Get the locally stored uuid:seqno. > + wsrep_get_SE_checkpoint(local_uuid, local_seqno); > What if this fails, no error handling ? > > - if (memcmp(&local_uuid, &uuid, sizeof(wsrep_uuid_t)) || > - local_seqno < seqno || seqno < 0) > + if (memcmp(&local_uuid, &uuid, sizeof(wsrep_uuid_t)) || > + local_seqno < seqno) > + { > + do_update= true; > + } > + else if (local_seqno > seqno) > + { > + WSREP_WARN("SST position can't be set in past. Requested: %lld, > Current: " > + " %lld.", (long long)seqno, (long long)local_seqno); > + /* > + If we are here because of SET command, simply return true (error) > instead of > + aborting. > + */ > + if (implicit) > { > - wsrep_set_SE_checkpoint(uuid, seqno); > - local_uuid = uuid; > - local_seqno = seqno; > + WSREP_WARN("Can't continue."); > + unireg_abort(1); > } > - else if (local_seqno > seqno) > + else > { > - WSREP_WARN("SST postion is in the past: %lld, current: %lld. " > - "Can't continue.", > - (long long)seqno, (long long)local_seqno); > - unireg_abort(1); > + return true; > } > + } > > #ifdef GTID_SUPPORT > - wsrep_init_sidno(uuid); > + wsrep_init_sidno(uuid); > #endif /* GTID_SUPPORT */ > > - if (wsrep) > - { > - int const rcode(seqno < 0 ? seqno : 0); > - wsrep_gtid_t const state_id = { > - uuid, (rcode ? WSREP_SEQNO_UNDEFINED : seqno) > - }; > + if (wsrep) > + { > + int const rcode(seqno < 0 ? seqno : 0); > When seqno can be < 0 ? > + wsrep_gtid_t const state_id= {uuid, > + (rcode ? WSREP_SEQNO_UNDEFINED : seqno)}; > + > + wsrep_status_t ret= wsrep->sst_received(wsrep, &state_id, state, > + state_len, rcode); > > - wsrep->sst_received(wsrep, &state_id, state, state_len, rcode); > + if (ret != WSREP_OK) > + { > + return true; > } > + } > + > + // Now is the good time to update the local state and checkpoint. > + if (do_update) { > + wsrep_set_SE_checkpoint(uuid, seqno); > Again, I would add error handling. > + local_uuid= uuid; > + local_seqno= seqno; > + } > + > + return false; > } > > // Let applier threads to continue > @@ -308,7 +357,7 @@ void wsrep_sst_continue () > if (sst_needed) > { > WSREP_INFO("Signalling provider to continue."); > - wsrep_sst_received (wsrep, local_uuid, local_seqno, NULL, 0); > + wsrep_sst_received (wsrep, local_uuid, local_seqno, NULL, 0, true); > You should check the return code. > } > } > > diff --git a/sql/wsrep_var.cc b/sql/wsrep_var.cc > index 9c01e54..bd94190 100644 > > > bool wsrep_start_position_update (sys_var *self, THD* thd, enum_var_type > type) > { > - WSREP_INFO ("wsrep_start_position var submitted: '%s'", > - wsrep_start_position); > - // since this value passed wsrep_start_position_check, don't check > anything > - // here > - wsrep_set_local_position (wsrep_start_position, true); > - return 0; > + // Print a confirmation that wsrep_start_position has been updated. > + WSREP_INFO ("wsrep_start_position set to '%s'", wsrep_start_position); > + return false; > Hmm, normally I thought that system variable set would go like if (variable_check()) variable_update() but now update does not do nothing ? > } > void wsrep_start_position_init (const char* val) > @@ -164,7 +194,7 @@ void wsrep_start_position_init (const char* val) > return; > } > > - wsrep_set_local_position (val, false); > + wsrep_set_local_position (val, strlen(val), false); > Add error handling. > } > > static bool refresh_provider_options() > diff --git a/storage/innobase/trx/trx0sys.cc > b/storage/innobase/trx/trx0sys.cc > index 76c8613..1625e8d 100644 > --- a/storage/innobase/trx/trx0sys.cc > +++ b/storage/innobase/trx/trx0sys.cc > @@ -349,10 +349,10 @@ void read_wsrep_xid_uuid(const XID* xid, unsigned > char* buf) > unsigned char xid_uuid[16]; > long long xid_seqno = read_wsrep_xid_seqno(xid); > read_wsrep_xid_uuid(xid, xid_uuid); > - if (!memcmp(xid_uuid, trx_sys_cur_xid_uuid, 8)) > + if (!memcmp(xid_uuid, trx_sys_cur_xid_uuid, 16) && > + (xid_seqno > -1)) > Does this mean that you ignore uuid if it is exactly the same, or is there error handling on else? R: Jan

2 1

[Maria-developers] GSoC 2016 GTID support for MariaDB MHA
by Charles Dirk 03 May '16

03 May '16

Hi everyone, I have created a wordpress blog HERE <https://mariadbmhagtidgsoc2016.wordpress.com/> on which I will post regularly about my progress with this project. Feel free to check it out. Cheers, Charles.

1 0

[Maria-developers] GSoC Community Bonding
by Rebecca Tucker 03 May '16

03 May '16

Hi Everyone, I am Becca and I will be working with Lixun Peng as mentor on adding support for GTID in the mysqlbinlog tool for GSoC 2016. I will base my work on 10.1 which I have cloned and build successfully on OS X El Capitan. I am currently studying the mysqlbinlog.cc code to see how we can extend the my_options array for GTID support and next I will study the GTID code to understand its data structures. I look forward to an awesome summer at the MariaDB foundation and thanks for giving me this opportunity. Cheers, Becca.

2 1

Re: [Maria-developers] ad8b439: MDEV-9618 solaris sparc build fails on 10.1.
by Sergei Golubchik 03 May '16

03 May '16

Hi, Alexey! On May 02, Alexey Botchkov wrote: > revision-id: ad8b4399f2fcd1d5e9dcdb8b2a8832cb3a6745ae (mariadb-10.1.13-23-gad8b439) > parent(s): ad4239cc3dc7ad5f6f264e1fb3cf6d24084bda90 > committer: Alexey Botchkov > timestamp: 2016-05-02 12:03:39 +0400 > message: > > MDEV-9618 solaris sparc build fails on 10.1. > > Compiler on Solaris is sensitive to C/C++ call models > differences, so it fails if we try to mix these two in > '?' operator. > Fixed by trying to keep the call models uniformity with the > proper amount of 'extern "C"' hints. > > --- > include/my_crypt.h | 6 ++++++ > include/mysql/plugin_audit.h | 7 +++++++ > include/mysql/plugin_encryption.h | 9 +++++++++ > .../example_key_management_plugin.cc | 2 +- > plugin/file_key_management/file_key_management_plugin.cc | 2 +- > sql/encryption.cc | 14 ++++++++++---- > unittest/sql/mf_iocache-t.cc | 2 +- > 7 files changed, 35 insertions(+), 7 deletions(-) > > diff --git a/include/my_crypt.h b/include/my_crypt.h > index e1e94c9..db280ca 100644 > --- a/include/my_crypt.h > +++ b/include/my_crypt.h > @@ -82,6 +82,12 @@ static inline uint my_aes_ctx_size(enum my_aes_mode mode __attribute__((unused)) > return MY_AES_CTX_SIZE; > } > > +static inline uint my_aes_ctx_size_for_handler(unsigned int a, > + unsigned int b __attribute__((unused))) > +{ > + return my_aes_ctx_size((enum my_aes_mode) a); > +} > + This won't be inlined, because you store a pointer to this function in a structure. Better use the cast, as before. > int my_random_bytes(uchar* buf, int num); > > #ifdef __cplusplus > diff --git a/include/mysql/plugin_audit.h b/include/mysql/plugin_audit.h > index 31589f0..e96f743 100644 > --- a/include/mysql/plugin_audit.h > +++ b/include/mysql/plugin_audit.h > @@ -23,6 +23,10 @@ > > #include "plugin.h" > > +#ifdef __cplusplus > +extern "C" { > +#endif I agree, in general. But why does your short patch have nothing about audit api? > #define MYSQL_AUDIT_CLASS_MASK_SIZE 1 > > #define MYSQL_AUDIT_INTERFACE_VERSION 0x0302 > @@ -174,5 +178,8 @@ struct st_mysql_audit > unsigned long class_mask[MYSQL_AUDIT_CLASS_MASK_SIZE]; > }; > > +#ifdef __cplusplus > +} > +#endif > > #endif > diff --git a/include/mysql/plugin_encryption.h b/include/mysql/plugin_encryption.h > index 3f35c2b..d748c4f 100644 > --- a/include/mysql/plugin_encryption.h > +++ b/include/mysql/plugin_encryption.h > @@ -27,6 +27,10 @@ > > #include <mysql/plugin.h> > > +#ifdef __cplusplus > +extern "C" { > +#endif ...etc I agree that these extern "C" are the correct fix. On the other hand, I don't see a point in moving casts from the assignment to a new variable definition. So, I'd suggest to keep all extern "C" hunks in the patch, but remove cast avoidance changes. Then ok to push! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] 9058071: MDEV-9362: InnoDB tables using DATA_DIRECTORY created using MySQL 5.6 do not work with MariaDB 10.1
by Sergei Golubchik 02 May '16

02 May '16

Hi, Jan! I have various comments, see below. Nothing big, nothing that's really the approach you've taken. In fact, I like where it's going. But this is a huge and intrusive patch. I wonder, should we rather do it in 10.2? See below, you yourself write: On Feb 25, Jan Lindström wrote: > > Caution: Please take backups of your database before migrating 10.1.12. This is not what users normally do when upgrading from a GA version to the next minor GA version. Anyway, see comments about the patch below: > diff --git a/mysql-test/suite/encryption/r/innodb-bad-key-change.result b/mysql-test/suite/encryption/r/innodb-bad-key-change.result > index cf97918..39e08f4 100644 > --- a/mysql-test/suite/encryption/r/innodb-bad-key-change.result > +++ b/mysql-test/suite/encryption/r/innodb-bad-key-change.result > @@ -36,8 +36,7 @@ SELECT * FROM t1; > ERROR HY000: Got error 192 'Table encrypted but decryption failed. This could be because correct encryption management plugin is not loaded, used encryption key is not available or encryption method does not match.' from InnoDB > SHOW WARNINGS; > Level Code Message > -Warning 1812 Tablespace is missing for table 'test/t1' > -Warning 192 Table test/t1 is encrypted but encryption service or used key_id 2 is not available. Can't continue reading table. > +Warning 192 Table test/t1 is encrypted but encryption service or used key_id is not available. Can't continue reading table. why did that happen? > Error 1296 Got error 192 'Table encrypted but decryption failed. This could be because correct encryption management plugin is not loaded, used encryption key is not available or encryption method does not match.' from InnoDB > DROP TABLE t1; > # Start server with keys.txt > diff --git a/storage/innobase/include/dict0tableoptions.h b/storage/innobase/include/dict0tableoptions.h > new file mode 100644 > index 0000000..def0e39 > --- /dev/null > +++ b/storage/innobase/include/dict0tableoptions.h > @@ -0,0 +1,127 @@ > +/***************************************************************************** > + > +Copyright (c) 2016, MariaDB Corporation. > + > +This program is free software; you can redistribute it and/or modify it under > +the terms of the GNU General Public License as published by the Free Software > +Foundation; version 2 of the License. > + > +This program is distributed in the hope that it will be useful, but WITHOUT > +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS > +FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. > + > +You should have received a copy of the GNU General Public License along with > +this program; if not, write to the Free Software Foundation, Inc., > +51 Franklin Street, Suite 500, Boston, MA 02110-1335 USA > + > +*****************************************************************************/ > + > +/**************************************************//** > +@file include/dict0tableoptions.h > +Definitions for the system table SYS_TABLE_OPTIONS > + > +Created 22/01/2006 Jan Lindström > +*******************************************************/ > + > +#ifndef dict0tableoptions_h > +#define dict0tableoptions_h > + > +#include "univ.i" > + > +/** Data structure to hold contents of SYS_TABLE_OPTIONS */ This is a move in the interesting direction. One of the ideas I was toying with for a few years is to remove .frm files for InnoDB tables. In MariaDB a storage engine can live without frm files if the engine supports table discovery. This is rather easy to implement, but InnoDB does not store the complete table definition, so the discovery will be lossy. Now with your SYS_TABLE_OPTIONS table we're getting closer to having the complete table definition saved inside InnoDB. And one step closer to frm-less InnoDB tables. > +struct dict_tableoptions_t{ > + table_id_t table_id; why do you need table_id in the option structure? > + /* true if table page compressed */ > + bool page_compressed; > + /* Used compression level if set */ > + ulonglong page_compression_level; > + /* dict0types.h: ATOMIC_WRITES_DEFAULT, _ON, or _OFF */ > + atomic_writes_t atomic_writes; > + /* fil0crypt.h: FIL_SPACE_ENCRYPTION_DEFAULT, _ON, or _OFF */ > + fil_encryption_t encryption; > + /* Used encryption key identifier if set */ > + ulonglong encryption_key_id; > + /* true if table options need to be stored */ why wouldn't they need to be? > + bool need_stored; > +}; > + > +/********************************************************************//** > +This function parses a SYS_TABLE_OPTIONS record, extracts necessary > +information from the record and returns it to the caller. > +@return error message or NULL if successfull */ > +UNIV_INTERN > +const char* > +dict_process_sys_tableoptions( > +/*==========================*/ > + mem_heap_t* heap, /*!< in/out: heap memory */ > + const rec_t* rec, /*!< in: current SYS_TABLE_OPTIONS rec */ > + dict_tableoptions_t* table_options); /*!< out: table options */ > + > +/********************************************************************//** > +Gets the table options from SYS_TABLE_OPTIONS based on table_id > +@return true if found, false if not found */ > +UNIV_INTERN > +bool > +dict_get_table_options( > +/*===================*/ > + table_id_t table_id, /*!< in: table id */ > + dict_tableoptions_t* options, /*!< out:table options */ > + bool fixed); /*!< in: can we fix the > + dictionary ? */ > + > +/********************************************************************//** > +Gets the table options from SYS_TABLE_OPTIONS > +@return true if found, false if not found */ > +UNIV_INTERN > +bool > +dict_get_table_options( > +/*===================*/ > + dict_table_t* table, /*!< in/out: table */ > + bool fixed); /*!< in: can we fix the > + dictionary ? */ > + > +/********************************************************************//** > +Update the record in SYS_TABLE_OPTIONS. > +@return DB_SUCCESS if OK, dberr_t if the update failed */ > +UNIV_INTERN > +dberr_t > +dict_update_tableoptions( > +/*=====================*/ > + const dict_table_t* table); /*!< in: table object */ > + > +/********************************************************************//** > +Insert record into SYS_TABLE_OPTIONS > +@return DB_SUCCESS if OK, dberr_t if the insert failed */ > +UNIV_INTERN > +dberr_t > +dict_insert_tableoptions( > +/*=====================*/ > + const dict_table_t* table, /*!< in: table object */ > + bool fixed); /*!< in: can we fix the > + dictionary ? */ > + > +/********************************************************************//** > +Update the table flags in SYS_TABLES. > +@return DB_SUCCESS if OK, dberr_t if the update failed */ > +UNIV_INTERN > +dberr_t > +dict_update_table_flags( > +/*=====================*/ > + const dict_table_t* table, /*!< in: table object */ > + bool fixed); /*!< in: can we fix the > + dictionary ? */ > + > +/********************************************************************//** > +Delete the record in SYS_TABLE_OPTIONS. > +@return DB_SUCCESS if OK, dberr_t if the update failed */ > +UNIV_INTERN > +dberr_t > +dict_delete_tableoptions( > +/*=====================*/ > + const dict_table_t* table, /*!< in: table object */ > + trx_t* trx, /*!< in: trx */ > + bool fixed); /*!< in: can we fix the > + dictionary ? */ > + > +#endif /* dict0tableoptions_h */ > + > diff --git a/storage/innobase/dict/dict0tableoptions.cc b/storage/innobase/dict/dict0tableoptions.cc > new file mode 100644 > index 0000000..527c5e3 > --- /dev/null > +++ b/storage/innobase/dict/dict0tableoptions.cc > @@ -0,0 +1,482 @@ > +/***************************************************************************** > + > +Copyright (c) 2016, MariaDB Corporation. > + > +This program is free software; you can redistribute it and/or modify it under > +the terms of the GNU General Public License as published by the Free Software > +Foundation; version 2 of the License. > + > +This program is distributed in the hope that it will be useful, but WITHOUT > +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS > +FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. > + > +You should have received a copy of the GNU General Public License along with > +this program; if not, write to the Free Software Foundation, Inc., > +51 Franklin Street, Suite 500, Boston, MA 02110-1335 USA > + > +*****************************************************************************/ > + > +/**************************************************//** > +@file dict/dict0tableoptions.cc > +Function implementations for the system table SYS_TABLE_OPTIONS > + > +Created 22/01/2006 Jan Lindström > +*******************************************************/ > + > +#include "mysql_version.h" > +#include "btr0pcur.h" > +#include "btr0btr.h" > +#include "page0page.h" > +#include "mach0data.h" > +#include "dict0dict.h" > +#include "dict0boot.h" > +#include "dict0stats.h" > +#include "dict0mem.h" > +#include "rem0cmp.h" > +#include "srv0start.h" > +#include "srv0srv.h" > +#include "dict0crea.h" > +#include "dict0priv.h" > +#include "ha_prototypes.h" /* innobase_casedn_str() */ > +#include "fts0priv.h" > +#include "dict0tableoptions.h" > +#include "dict0load.h" > +#include "row0mysql.h" > + > +/********************************************************************//** > +This function parses a SYS_TABLE_OPTIONS record, extracts necessary > +information from the record and returns it to the caller. > +@return error message or NULL if successfull */ > +UNIV_INTERN > +const char* > +dict_process_sys_tableoptions( > +/*==========================*/ > + mem_heap_t* heap, /*!< in/out: heap memory */ > + const rec_t* rec, /*!< in: current SYS_TABLE_OPTIONS rec */ > + dict_tableoptions_t* table_options) /*!< out: table options */ > +{ > + const byte* field; > + ulint len=0; > + > + if (rec_get_deleted_flag(rec, 0)) { > + return("delete-marked record in SYS_TABLE_OPTIONS"); > + } > + > + if (rec_get_n_fields_old(rec) != DICT_NUM_FIELDS__SYS_TABLEOPTIONS) { > + return("wrong number of columns in SYS_TABLE_OPTIONS record"); > + } How does it work future-wise? How do you add new options to the table, say, in 10.2? > diff --git a/storage/innobase/api/api0api.cc b/storage/innobase/api/api0api.cc > index 739ea9f..4af32a5 100644 > --- a/storage/innobase/api/api0api.cc > +++ b/storage/innobase/api/api0api.cc > @@ -270,7 +271,7 @@ ib_open_table_by_name( > dict_table_t* table; > > table = dict_table_open_on_name(name, FALSE, FALSE, > - DICT_ERR_IGNORE_NONE); > + DICT_ERR_IGNORE_NONE, NULL); What does that mean that you pass NULL as the last argument? You won't know whether the table is encrypted or compressed - so you, basically won't be able to read the table? > > if (table != NULL && table->ibd_file_missing) { > table = NULL; > diff --git a/storage/innobase/buf/buf0buf.cc b/storage/innobase/buf/buf0buf.cc > index f4e7c0d..f1953af 100644 > --- a/storage/innobase/buf/buf0buf.cc > +++ b/storage/innobase/buf/buf0buf.cc > @@ -4677,7 +4678,7 @@ buf_page_io_complete( > "However key management plugin or used key_id %lu is not found or" > " used encryption algorithm or method does not match." > " Can't continue opening the table.", > - bpage->key_version); > + bpage->space, bpage->key_version); This seems to be an unrelated bugfix. And it seems to be already fixed in the latest 10.1 > > if (bpage->space > TRX_SYS_SPACE) { > if (corrupted) { > diff --git a/storage/innobase/dict/dict0dict.cc b/storage/innobase/dict/dict0dict.cc > index 2b72835..13f1ee6 100644 > --- a/storage/innobase/dict/dict0dict.cc > +++ b/storage/innobase/dict/dict0dict.cc > @@ -1187,6 +1189,16 @@ dict_table_open_on_name( > > /* If table is encrypted return table */ > if (ignore_err == DICT_ERR_IGNORE_NONE > + && !table->is_encrypted && options) { hmm, where else table->is_encrypted can be set? > + if ((options->encryption == FIL_SPACE_ENCRYPTION_ON || > + (options->encryption == FIL_SPACE_ENCRYPTION_DEFAULT && srv_encrypt_tables)) > + && !encryption_key_id_exists((unsigned int)options->encryption_key_id)) { > + table->is_encrypted = true; > + } > + } > + > + /* If table is encrypted return table */ > + if (ignore_err == DICT_ERR_IGNORE_NONE > && table->is_encrypted) { > /* Make life easy for drop table. */ > if (table->can_be_evicted) { > diff --git a/storage/innobase/dict/dict0load.cc b/storage/innobase/dict/dict0load.cc > index d8bd0a6..2b732f1 100644 > --- a/storage/innobase/dict/dict0load.cc > +++ b/storage/innobase/dict/dict0load.cc > @@ -56,7 +64,8 @@ static const char* SYSTEM_TABLE_NAME[] = { > "SYS_FOREIGN", > "SYS_FOREIGN_COLS", > "SYS_TABLESPACES", > - "SYS_DATAFILES" > + "SYS_DATAFILES", > + "SYS_TABLE_OPTIONS" May be "SYS_MARIADB_OPTIONS" ? Not really InnoDB sys table style, I agree, but guarantees no conflicts in the future. > }; > > /* If this flag is TRUE, then we will load the cluster index's (and tables') > @@ -2365,6 +2386,11 @@ dict_load_table( > btr_pcur_close(&pcur); > mtr_commit(&mtr); > > + if (table && options) { no need to do if (table), because table is guaranteed to be not NULL here > + ut_ad(table->table_options); > + memcpy(table->table_options, options, sizeof(dict_tableoptions_t)); > + } > + > if (table->space == 0) { > /* The system tablespace is always available. */ > } else if (table->flags2 & DICT_TF2_DISCARDED) { > diff --git a/storage/innobase/dict/dict0mem.cc b/storage/innobase/dict/dict0mem.cc > index 1724ac0..39d0d55 100644 > --- a/storage/innobase/dict/dict0mem.cc > +++ b/storage/innobase/dict/dict0mem.cc > @@ -85,7 +86,6 @@ dict_mem_table_create( > mem_heap_t* heap; > > ut_ad(name); > - ut_a(dict_tf_is_valid(flags)); why? > ut_a(!(flags2 & ~DICT_TF2_BIT_MASK)); > > heap = mem_heap_create(DICT_HEAP_SIZE); > diff --git a/storage/innobase/fil/fil0fil.cc b/storage/innobase/fil/fil0fil.cc > index 928c72c..4f75465 100644 > --- a/storage/innobase/fil/fil0fil.cc > +++ b/storage/innobase/fil/fil0fil.cc > @@ -716,27 +717,6 @@ fil_node_open_file( > ut_error; > } > > - if (UNIV_UNLIKELY(space->flags != flags)) { > - fprintf(stderr, > - "InnoDB: Error: table flags are 0x%lx" > - " in the data dictionary\n" > - "InnoDB: but the flags in file %s are 0x%lx!\n", > - space->flags, node->name, flags); do you still the check somewhere where SYS_TABLE_OPTIONS is compared with frm? > - > - ut_error; > - } > - > - if (UNIV_UNLIKELY(space->flags != flags)) { > - if (!dict_tf_verify_flags(space->flags, flags)) { > - fprintf(stderr, > - "InnoDB: Error: table flags are 0x%lx" > - " in the data dictionary\n" > - "InnoDB: but the flags in file %s are 0x%lx!\n", > - space->flags, node->name, flags); > - ut_error; > - } > - } > - > if (size_bytes >= FSP_EXTENT_SIZE * UNIV_PAGE_SIZE) { > /* Truncate the size to whole extent size. */ > size_bytes = ut_2pow_round(size_bytes, > @@ -1917,11 +1909,6 @@ fil_check_first_page( > flags = mach_read_from_4(FSP_HEADER_OFFSET + FSP_SPACE_FLAGS + page); > > if (UNIV_PAGE_SIZE != fsp_flags_get_page_size(flags)) { > - fprintf(stderr, > - "InnoDB: Error: Current page size %lu != " > - " page size on page %lu\n", > - UNIV_PAGE_SIZE, fsp_flags_get_page_size(flags)); > - why? > return("innodb-page-size mismatch"); > } > > @@ -3379,8 +3362,7 @@ fil_create_new_single_table_tablespace( > ulint size, /*!< in: the initial size of the > tablespace file in pages, > must be >= FIL_IBD_FILE_INITIAL_SIZE */ > - fil_encryption_t mode, /*!< in: encryption mode */ > - ulint key_id) /*!< in: encryption key_id */ > + const dict_table_t* table) /*!< in: table or NULL */ what does NULL mean here? > { > os_file_t file; > ibool ret; > @@ -3640,6 +3627,140 @@ fil_report_bad_tablespace( > (ulong) expected_id, (ulong) expected_flags); > } > > +/****************************************************************** > +Set flags for a tablespace */ > +static > +void > +fil_space_set_fsp_flags( > +/*=====================*/ > + ulint id, /*!< in: space id */ > + uint flags) /*!< in: fsp flags */ > +{ > + fil_space_t* space; > + > + ut_ad(fil_system); > + > + mutex_enter(&fil_system->mutex); > + > + space = fil_space_get_by_id(id); > + > + if (space != NULL) { > + space->flags = flags; > + } > + > + mutex_exit(&fil_system->mutex); > +} > + > +/****************************************************************** > +Update tablespace (fsp) flags on page 0 a bit more verbose comment would've been be nice. like, say that it's used to migrate from mariadb-flags-in-page-0 to sys_table_options approach. > +@return true if successfull, false if not */ > +static > +void > +fil_update_page0( > +/*=============*/ > + byte* page0, /*!< in: page 0 or NULL */ why do you need this page0 argument if it is *always* NULL? > + os_file_t data_file, /*!< in: data file */ > + ulint space, /*!< in: space id */ > + ulint flags, /*!< in: old fsp flags */ > + ulint fsp_flags)/*!< in: new fsp flags */ > +{ > + ulint psize = FSP_FLAGS_GET_PAGE_SSIZE_MARIADB(flags); > + ulint zip_size = FSP_FLAGS_GET_ZIP_SSIZE(flags); > + > + if (!psize) { > + psize = UNIV_PAGE_SIZE_ORIG; > + } > + > + fsp_flags = fsp_flags_set_page_size(fsp_flags, psize); > + > + if (flags != fsp_flags) { > + > + ib_logf(IB_LOG_LEVEL_INFO, > + "InnoDB: Adjusted space_id %lu tablespace flags from %lu to %lu.\n", > + space, flags, fsp_flags); > + > + mtr_t mtr; > + mtr_start(&mtr); > + > + if (!page0) { > + buf_block_t* block = buf_page_get_gen(space, > + zip_size, 0, > + RW_X_LATCH, > + NULL, > + BUF_GET, > + __FILE__, __LINE__, > + &mtr); > + page0 = buf_block_get_frame(block); > + } > + > + mlog_write_ulint(page0 + FSP_HEADER_OFFSET + FSP_SPACE_FLAGS, > + fsp_flags, MLOG_4BYTES, &mtr); > + /* Redo log this as bytewise update to page 0 > + followed by an MLOG_FILE_WRITE_FSP_FLAGS */ > + byte* log_ptr = mlog_open(&mtr, 11 + 8); > + if (log_ptr != NULL) { > + log_ptr = mlog_write_initial_log_record_fast( > + page0, > + MLOG_FILE_WRITE_FSP_FLAGS, > + log_ptr, &mtr); > + mach_write_to_4(log_ptr, fsp_flags); > + log_ptr += 4; > + mach_write_to_4(log_ptr, space); > + log_ptr += 4; > + mlog_close(&mtr, log_ptr); > + } > + > + mtr_commit(&mtr); > + lsn_t end_lsn = mtr.end_lsn; > + buf_flush_init_for_writing(page0, NULL, end_lsn); > + flags = mach_read_from_4(FSP_HEADER_OFFSET + FSP_SPACE_FLAGS + page0); > + ut_ad(flags == fsp_flags); > + > + /* Flush dirty page to the storage */ > + ulint n_pages = 0; > + ulint sum_pages = 0; > + bool success = false; > + do { > + success = buf_flush_list(ULINT_MAX, end_lsn, &n_pages); > + buf_flush_wait_batch_end(NULL, BUF_FLUSH_LIST); > + sum_pages += n_pages; > + } while (!success); > + > + fil_space_set_fsp_flags(space, fsp_flags); > + } > +} > + > +/****************************************************************** > +Parse a MLOG_FILE_WRITE_FSP_FLAGS log entry > +@return position on log buffer */ > +UNIV_INTERN > +byte* > +fil_parse_write_fsp_flags( > +/*======================*/ > + byte* ptr, /*!< in: Log entry start */ > + byte* end_ptr,/*!< in: Log entry end */ > + buf_block_t* block) /*!< in: buffer block */ > +{ > + /* check that redo log entry is complete */ > + uint entry_size = 4 + 4; // size of flags + space_id > + > + if (end_ptr - ptr < entry_size){ > + return NULL; > + } > + > + ulint flags = mach_read_from_4(ptr); > + ptr += 4; > + ulint space_id = mach_read_from_4(ptr); > + ptr += 4; > + > + ut_a(fsp_flags_is_valid(flags)); > + > + /* update fil_space memory cache with flags */ > + fil_space_set_fsp_flags(space_id, flags); > + > + return ptr; > +} > + > /********************************************************************//** > Tries to open a single-table tablespace and optionally checks that the > space id in it is correct. If this does not succeed, print an error message > diff --git a/storage/innobase/fts/fts0fts.cc b/storage/innobase/fts/fts0fts.cc > index cc4a4f9..f81fd31 100644 > --- a/storage/innobase/fts/fts0fts.cc > +++ b/storage/innobase/fts/fts0fts.cc > @@ -1981,7 +1982,13 @@ fts_create_one_index_table( > dict_mem_table_add_col(new_table, heap, "ilist", DATA_BLOB, > 4130048, 0); > > - error = row_create_table_for_mysql(new_table, trx, false, FIL_SPACE_ENCRYPTION_DEFAULT, FIL_DEFAULT_ENCRYPTION_KEY); > + /* Get default encryption key id if set */ > + if (new_table && new_table->table_options && new_table cannot be NULL here > + new_table->table_options->encryption_key_id == 0) { > + new_table->table_options->encryption_key_id = innobase_get_default_encryption_key_id(trx); do you want to do it here? may be better do it in row_create_table_for_mysql? > + } > + > + error = row_create_table_for_mysql(new_table, trx, false); > > if (error != DB_SUCCESS) { > trx->error_state = error; > diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc > index 527c5be..3d5aaee 100644 > --- a/storage/innobase/handler/ha_innodb.cc > +++ b/storage/innobase/handler/ha_innodb.cc > @@ -5459,6 +5460,43 @@ ha_innobase::innobase_initialize_autoinc() > } > > /*****************************************************************//** > +Set up the table options structure based on frm. */ > +UNIV_INTERN > +void > +ha_innobase::set_table_options( > +/*===========================*/ > + THD* thd, /*!< in: thd */ > + TABLE* table, /*!< in: table */ > + dict_tableoptions_t* options) /*!< in: table options */ > +{ > + ha_table_option_struct* moptions = table->s->option_struct; > + > + options->page_compressed = moptions->page_compressed; > + options->page_compression_level = moptions->page_compression_level; > + options->atomic_writes = (atomic_writes_t)moptions->atomic_writes; > + options->encryption = (fil_encryption_t)moptions->encryption; > + options->encryption_key_id = moptions->encryption_key_id; > + > + if (options->encryption_key_id == 0) { > + options->encryption_key_id = THDVAR(thd, default_encryption_key_id); > + } No, this is not how it works. The *server* sets the encryption_key_id based on the default_encryption_key_id. The engine does not need to bother. > + > + if (options->page_compression_level == 0) { > + options->page_compression_level = page_zip_level; > + } same here > + > + /* Table options should be stored if they are not same as > + defaults */ > + if (moptions->page_compressed || > + (options->atomic_writes == ATOMIC_WRITES_ON || > + options->atomic_writes == ATOMIC_WRITES_OFF) || > + (options->encryption == FIL_SPACE_ENCRYPTION_OFF || > + options->encryption == FIL_SPACE_ENCRYPTION_ON)) { > + options->need_stored = true; > + } > +} > + > +/*****************************************************************//** > Creates and opens a handle to a table which already exists in an InnoDB > database. > @return 1 if error, 0 if success */ > @@ -11603,6 +11647,10 @@ ha_innobase::check_table_options( > return "ENCRYPTION_KEY_ID"; > > } > + > + table_options->need_stored = true; > + > + table_options->need_stored = true; typo > } > > /* Check atomic writes requirements */ > @@ -20422,3 +20483,25 @@ ib_push_warning( > my_free(buf); > va_end(args); > } > + > +/********************************************************************//** > +Helper function to get default_encryption_key_id from THD > +(trx->mysql_thd). > +@return default_encryption_key_id from THD or > +FIL_DEFAULT_ENCRYPTION_KEY */ > +UNIV_INTERN > +ulint > +innobase_get_default_encryption_key_id( See comment above. You _probably_ don't need this method > +/*===================================*/ > + trx_t* trx) /*! in: trx */ > +{ > + ulint key_id = FIL_DEFAULT_ENCRYPTION_KEY; > + > + if (trx && trx->mysql_thd) { > + THD *thd = (THD *)trx->mysql_thd; > + > + key_id = THDVAR(thd, default_encryption_key_id); > + } > + > + return (key_id); > +} > diff --git a/storage/innobase/handler/i_s.cc b/storage/innobase/handler/i_s.cc > index ef69e7d..75baa9d 100644 > --- a/storage/innobase/handler/i_s.cc > +++ b/storage/innobase/handler/i_s.cc > @@ -9130,3 +9129,230 @@ UNIV_INTERN struct st_maria_plugin i_s_innodb_sys_semaphore_waits = > STRUCT_FLD(version_info, INNODB_VERSION_STR), > STRUCT_FLD(maturity, MariaDB_PLUGIN_MATURITY_BETA), > }; > + > +/** SYS_TABLE_OPTIONS ************************************************/ > +/* Fields of the dynamic table INFORMATION_SCHEMA.INNODB_SYS_TABLE_OPTIONS */ > +static ST_FIELD_INFO innodb_sys_tableoptions_fields_info[] = Good idea! > +{ > + // SYS_TABLE_OPTIONS_TABLE_ID 0 > + {STRUCT_FLD(field_name, "TABLE_ID"), > + STRUCT_FLD(field_length, MY_INT64_NUM_DECIMAL_DIGITS), > + STRUCT_FLD(field_type, MYSQL_TYPE_LONGLONG), > + STRUCT_FLD(value, 0), > + STRUCT_FLD(field_flags, MY_I_S_UNSIGNED), > + STRUCT_FLD(old_name, ""), > + STRUCT_FLD(open_method, SKIP_OPEN_TABLE)}, > + > + // SYS_TABLE_OPTIONS_PAGE_COMPRESSED 1 > + {STRUCT_FLD(field_name, "PAGE_COMPRESSED"), > + STRUCT_FLD(field_length, 7), > + STRUCT_FLD(field_type, MYSQL_TYPE_STRING), > + STRUCT_FLD(value, 0), > + STRUCT_FLD(field_flags, 0), > + STRUCT_FLD(old_name, ""), > + STRUCT_FLD(open_method, SKIP_OPEN_TABLE)}, > + > + // SYS_TABLE_OPTIONS_PAGE_COMPRESSION_LEVEL 2 > + {STRUCT_FLD(field_name, "PAGE_COMPRESSION_LEVEL"), > + STRUCT_FLD(field_length, MY_INT32_NUM_DECIMAL_DIGITS), > + STRUCT_FLD(field_type, MYSQL_TYPE_LONG), > + STRUCT_FLD(value, 0), > + STRUCT_FLD(field_flags, MY_I_S_UNSIGNED), > + STRUCT_FLD(old_name, ""), > + STRUCT_FLD(open_method, SKIP_OPEN_TABLE)}, > + > + // SYS_TABLE_OPTIONS_ATOMIC_WRITES 3 > + {STRUCT_FLD(field_name, "ATOMIC_WRITES"), > + STRUCT_FLD(field_length, 9), > + STRUCT_FLD(field_type, MYSQL_TYPE_STRING), > + STRUCT_FLD(value, 0), > + STRUCT_FLD(field_flags, 0), > + STRUCT_FLD(old_name, ""), > + STRUCT_FLD(open_method, SKIP_OPEN_TABLE)}, > + > + // SYS_TABLE_OPTIONS_ENCRYPTED 4 > + {STRUCT_FLD(field_name, "ENCRYPTED"), > + STRUCT_FLD(field_length, 9), > + STRUCT_FLD(field_type, MYSQL_TYPE_STRING), > + STRUCT_FLD(value, 0), > + STRUCT_FLD(field_flags, 0), > + STRUCT_FLD(old_name, ""), > + STRUCT_FLD(open_method, SKIP_OPEN_TABLE)}, > + > + // SYS_TABLE_OPTIONS_ENCRYPTION_KEY_ID 5 > + {STRUCT_FLD(field_name, "ENCRYPTION_KEY_ID"), > + STRUCT_FLD(field_length, MY_INT32_NUM_DECIMAL_DIGITS), > + STRUCT_FLD(field_type, MYSQL_TYPE_LONG), > + STRUCT_FLD(value, 0), > + STRUCT_FLD(field_flags, MY_I_S_UNSIGNED), > + STRUCT_FLD(old_name, ""), > + STRUCT_FLD(open_method, SKIP_OPEN_TABLE)}, > + > + END_OF_ST_FIELD_INFO > +}; > + > +/*******************************************************************//** > +Function to go through each record in SYS_TABLE_OPTIONS table, and fill the > +information_schema.innodb_sys_table_options table with related table information > +@return 0 on success */ > +static > +int > +i_s_dict_fill_sys_table_options( > +/*============================*/ > + THD* thd, /*!< in: thread */ > + TABLE_LIST* tables, /*!< in/out: tables to fill */ > + Item* ) /*!< in: condition (not used) */ > +{ > + btr_pcur_t pcur; > + const rec_t* rec; > + mem_heap_t* heap; > + mtr_t mtr; > + > + DBUG_ENTER("i_s_sys_table_options_fill_table"); > + RETURN_IF_INNODB_NOT_STARTED(tables->schema_table_name); > + > + /* deny access to user without PROCESS_ACL privilege */ > + if (check_global_access(thd, PROCESS_ACL, true)) { > + DBUG_RETURN(0); > + } > + > + heap = mem_heap_create(1000); > + mutex_enter(&(dict_sys->mutex)); > + mtr_start(&mtr); > + > + rec = dict_startscan_system(&pcur, &mtr, SYS_TABLE_OPTIONS); > + > + while (rec) { > + const char* err_msg; > + dict_tableoptions_t options; > + > + /* Create and populate a dict_tableoptions_t structure with > + information from SYS_TABLE_OPTIONS row */ > + memset(&options, 0, sizeof(dict_tableoptions_t)); > + > + err_msg = dict_process_sys_tableoptions( > + heap, rec, &options); > + > + mtr_commit(&mtr); > + mutex_exit(&dict_sys->mutex); > + > + if (!err_msg) { > + Field** fields = tables->table->field; > + > + OK(fields[SYS_TABLE_OPTIONS_TABLE_ID]->store((longlong) options.table_id)); > + OK(fields[SYS_TABLE_OPTIONS_PAGE_COMPRESSION_LEVEL]->store((ulint)options.page_compression_level)); > + OK(fields[SYS_TABLE_OPTIONS_ENCRYPTION_KEY_ID]->store((ulint)options.encryption_key_id)); > + > + if (options.page_compressed) { > + OK(field_store_string(fields[SYS_TABLE_OPTIONS_PAGE_COMPRESSED], "YES")); > + } else { > + OK(field_store_string(fields[SYS_TABLE_OPTIONS_PAGE_COMPRESSED], "NO")); > + } > + > + if (options.encryption == FIL_SPACE_ENCRYPTION_DEFAULT) { > + OK(field_store_string(fields[SYS_TABLE_OPTIONS_ENCRYPTED], "DEFAULT")); > + } else if (options.encryption == FIL_SPACE_ENCRYPTION_ON) { > + OK(field_store_string(fields[SYS_TABLE_OPTIONS_ENCRYPTED], "ON")); > + } else { > + OK(field_store_string(fields[SYS_TABLE_OPTIONS_ENCRYPTED], "OFF")); > + } > + > + if (options.atomic_writes == ATOMIC_WRITES_DEFAULT) { > + OK(field_store_string(fields[SYS_TABLE_OPTIONS_ATOMIC_WRITES], "DEFAULT")); > + } else if (options.atomic_writes == ATOMIC_WRITES_ON) { > + OK(field_store_string(fields[SYS_TABLE_OPTIONS_ATOMIC_WRITES], "ON")); > + } else { > + OK(field_store_string(fields[SYS_TABLE_OPTIONS_ATOMIC_WRITES], "OFF")); > + } > + > + OK(schema_table_store_record(thd, tables->table)); > + > + } else { > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_CANT_FIND_SYSTEM_REC, "%s", > + err_msg); > + } > + > + mem_heap_empty(heap); > + > + /* Get the next record */ > + mutex_enter(&dict_sys->mutex); > + mtr_start(&mtr); > + rec = dict_getnext_system(&pcur, &mtr); > + } > + > + mtr_commit(&mtr); > + mutex_exit(&dict_sys->mutex); > + mem_heap_free(heap); > + > + DBUG_RETURN(0); > +} > +/*******************************************************************//** > +Bind the dynamic table INFORMATION_SCHEMA.INNODB_SYS_TABLE_OPTIONS > +@return 0 on success */ > +static > +int > +innodb_sys_table_options_init( > +/*============================*/ > + void* p) /*!< in/out: table schema object */ > +{ > + ST_SCHEMA_TABLE* schema; > + > + DBUG_ENTER("innodb_sys_table_options_init"); > + > + schema = (ST_SCHEMA_TABLE*) p; > + > + schema->fields_info = innodb_sys_tableoptions_fields_info; > + schema->fill_table = i_s_dict_fill_sys_table_options; > + > + DBUG_RETURN(0); > +} > + > +UNIV_INTERN struct st_maria_plugin i_s_innodb_sys_table_options = > +{ > + /* the plugin type (a MYSQL_XXX_PLUGIN value) */ > + /* int */ > + STRUCT_FLD(type, MYSQL_INFORMATION_SCHEMA_PLUGIN), > + > + /* pointer to type-specific plugin descriptor */ > + /* void* */ > + STRUCT_FLD(info, &i_s_info), > + > + /* plugin name */ > + /* const char* */ > + STRUCT_FLD(name, "INNODB_SYS_TABLE_OPTIONS"), > + > + /* plugin author (for SHOW PLUGINS) */ > + /* const char* */ > + STRUCT_FLD(author, maria_plugin_author), > + > + /* general descriptive text (for SHOW PLUGINS) */ > + /* const char* */ > + STRUCT_FLD(descr, "InnoDB SYS_TABLE_OPTIONS"), > + > + /* the plugin license (PLUGIN_LICENSE_XXX) */ > + /* int */ > + STRUCT_FLD(license, PLUGIN_LICENSE_GPL), > + > + /* the function to invoke when plugin is loaded */ > + /* int (*)(void*); */ > + STRUCT_FLD(init, innodb_sys_table_options_init), > + > + /* the function to invoke when plugin is unloaded */ > + /* int (*)(void*); */ > + STRUCT_FLD(deinit, i_s_common_deinit), > + > + /* plugin version (for SHOW PLUGINS) */ > + /* unsigned int */ > + STRUCT_FLD(version, INNODB_VERSION_SHORT), > + > + /* struct st_mysql_show_var* */ > + STRUCT_FLD(status_vars, NULL), > + > + /* struct st_mysql_sys_var** */ > + STRUCT_FLD(system_vars, NULL), > + > + /* Maria extension */ > + STRUCT_FLD(version_info, INNODB_VERSION_STR), > + STRUCT_FLD(maturity, MariaDB_PLUGIN_MATURITY_BETA), > +}; > diff --git a/storage/innobase/include/dict0dict.h b/storage/innobase/include/dict0dict.h > index b15d364..fa65884 100644 > --- a/storage/innobase/include/dict0dict.h > +++ b/storage/innobase/include/dict0dict.h > @@ -944,15 +948,8 @@ dict_tf_set( > ulint* flags, /*!< in/out: table */ > rec_format_t format, /*!< in: file format */ > ulint zip_ssize, /*!< in: zip shift size */ > - bool remote_path, /*!< in: table uses DATA DIRECTORY > + bool remote_path); /*!< in: table uses DATA DIRECTORY > */ > - bool page_compressed,/*!< in: table uses page compressed > - pages */ > - ulint page_compression_level, /*!< in: table page compression > - level */ > - ulint atomic_writes) /*!< in: table atomic > - writes option value*/ > - __attribute__((nonnull)); Why do you remove nonnull attributes? They help gcc to optimize the code > /********************************************************************//** > Convert a 32 bit integer table flags to the 32 bit integer that is > written into the tablespace header at the offset FSP_SPACE_FLAGS and is > diff --git a/storage/innobase/include/dict0dict.ic b/storage/innobase/include/dict0dict.ic > index a3a3446..3580081 100644 > --- a/storage/innobase/include/dict0dict.ic > +++ b/storage/innobase/include/dict0dict.ic > @@ -685,11 +629,7 @@ dict_sys_tables_type_validate( > ulint zip_ssize = DICT_TF_GET_ZIP_SSIZE(type); > ulint atomic_blobs = DICT_TF_HAS_ATOMIC_BLOBS(type); > ulint unused = DICT_TF_GET_UNUSED(type); > - ulint page_compression = DICT_TF_GET_PAGE_COMPRESSION(type); > - ulint page_compression_level = DICT_TF_GET_PAGE_COMPRESSION_LEVEL(type); By the way, what about page compression in MySQL InnoDB? where is that stored? will you support those tables? > - ulint atomic_writes = DICT_TF_GET_ATOMIC_WRITES(type); > - > - ut_a(atomic_writes <= ATOMIC_WRITES_OFF); > + ulint unused_mariadb = DICT_TF_GET_UNUSED_MARIADB(type); > > /* The low order bit of SYS_TABLES.TYPE is always set to 1. > If the format is UNIV_FORMAT_B or higher, this field is the same > diff --git a/storage/innobase/include/fil0fil.h b/storage/innobase/include/fil0fil.h > index 7fe0cc8..bc7f317 100644 > --- a/storage/innobase/include/fil0fil.h > +++ b/storage/innobase/include/fil0fil.h > @@ -39,7 +39,7 @@ Created 10/25/1995 Heikki Tuuri > #include "ibuf0types.h" > #include "log0log.h" > #endif /* !UNIV_HOTBACKUP */ > - > +#include "my_crypt.h" why? > #include <list> > > // Forward declaration > diff --git a/storage/innobase/srv/srv0start.cc b/storage/innobase/srv/srv0start.cc > index d7b37b5..29ce93d 100644 > --- a/storage/innobase/srv/srv0start.cc > +++ b/storage/innobase/srv/srv0start.cc > @@ -2537,7 +2543,16 @@ innobase_start_or_create_for_mysql(void) > > sys_datafiles_created = true; > > - /* This function assumes that SYS_DATAFILES exists */ > + err = dict_create_or_check_sys_table_options(); > + > + if (err != DB_SUCCESS) { > + return(err); > + } > + > + sys_table_options_created = true; > + > + /* This function assumes that SYS_DATAFILES > + and SYS_TABLE_OPTIONS exists */ "exist" > dict_check_tablespaces_and_store_max_id(dict_check); > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

[Maria-developers] GSoC 2016 Community Bonding
by Charles Dirk 02 May '16

02 May '16

Hi everyone, I am Charles Dirk, also Razor on IRC and am glad to be selected as a Google summer of Code student with MariaDB. I will be working on Providing GTID support in MHA and I will be mentored by Byte. I look forward to contributing to and learning much from the MariaDB developer community this summer. I am currently reading some documentation on KB to get more pointers on this task while waiting to hear from my Mentor. Cheers, Charles.

2 1

[Maria-developers] [GSOC Students] Allow to contribute
by Даниил Медведев 02 May '16

02 May '16

Hello everyone! I was selected for GSoC to work with MariaDB, with task NO PAD Collatons. Me profile on github is medvdanil, and i'm planning to code in my own branch 10.2-MDEV-9711 or 10.1-MDEV-9711. Which version I should select 10.1 or 10.2? And I ask to give me access to create one of those branches and commit to it. Regards, Daniil Medvedev

4 4

Re: [Maria-developers] 95e1c46: MDEV-9898 SET ROLE NONE can crash mysqld.
by Sergei Golubchik 02 May '16

02 May '16

Hi, Alexey! On May 02, Alexey Botchkov wrote: > Fine. > But can you please explain the logic here? When connected to the server, > the user is authenticated with the > find_user_or_anon(). But when we check permissions for the SET ROLE NONE, > we use find_user_wild(). > Why is that? Because 'user' is thd->security_ctx->priv_user in this case, not thd->security_ctx->user. find_user_or_anon() works with user/host pair, using wildcard matching for both user and host values. While find_user_exact() works with priv_user/priv_host pairs, using exact matching for both user and host values. The very weird find_user_wild() works with inconsistent priv_user/host pairs, using wildcards only for the host, not for the user. This pair make no sense to me, but this is the historical MySQL behavior that I didn't risk to break :( It's MDEV-5233 Regards, Sergei

1 0

[Maria-developers] GSoC 2016: Introduction letter of Galina Shalygina
by Галина Шалыгина 01 May '16

01 May '16

Hi everyone! My name is Galina Shalygina and I was selected for GSoC to work with MariaDB company. I'm from St Petersburg, at Russia and I study at St Petersburg State University with major in Analytical Information Systems. I'm in my third year at the University. My nick on IRC is galina and on github it's shagalla. I've already worked with MariaDB Server, and my contribution on non-recursive CTE was accepted by MariaDB. Now I'm finishing and implementation of recursive CTE. My project for GsoC 2016 is called "Pushing conditions into non-mergeable views and derived tables in MariaDB ". I plan to implement the optimizations that push conditions into non-mergeable (such as grouping) views/derived tables. I will be mentored by igor_seattle and spetrunia. During the community bonding period I'm going to: 1. Clone server code and build it; 2. Read documentation; 3. Study all developments on my theme; 4. Start some kind of blog where I'll write reports during work period; 5. For this project I will need to be able to clone expressions, so I'll start to work on solving this problem. I'm looking forward to working on this project. It would be a great experience for me! Best wishes, Galina Shalygina.

1 0

Re: [Maria-developers] f9f290b: MDEV-9851: CREATE USER w/o IDENTIFIED BY clause causes crash when using cracklib plugin
by Sergei Golubchik 29 Apr '16

29 Apr '16

Hi, Nirbhay! On Mar 31, Nirbhay Choubey wrote: > revision-id: f9f290b6828eeb57cba611d006d2a9301dc52244 (mariadb-10.1.13-3-gf9f290b) > parent(s): f4d5fe277599da4549c97c660f324c88cf9a2542 > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-03-31 18:03:44 -0400 > message: > > MDEV-9851: CREATE USER w/o IDENTIFIED BY clause causes crash when using cracklib plugin > > Add a check for NULL password. > > diff --git a/plugin/cracklib_password_check/cracklib_password_check.c b/plugin/cracklib_password_check/cracklib_password_check.c > index c593173..c192cdf 100644 > --- a/plugin/cracklib_password_check/cracklib_password_check.c > +++ b/plugin/cracklib_password_check/cracklib_password_check.c > @@ -33,7 +33,8 @@ static int crackme(MYSQL_LEX_STRING *username, MYSQL_LEX_STRING *password) > if ((host= strchr(user, '@'))) > *host++= 0; > > - if ((res= FascistCheckUser(password->str, dictionary, user, host))) > + if ((password->str == NULL) || // No password > + (res= FascistCheckUser(password->str, dictionary, user, host))) > { > my_printf_error(ER_NOT_VALID_PASSWORD, "cracklib: %s", > MYF(ME_JUST_WARNING), res); You forgot to fix the simple_password_check plugin. And if all plugins need to do the same check - it's a strong indication that this should've been done in the server. So, please, fix this in sql_acl.cc instead. Like this: - struct validation_data data= { &user->user, &user->pwtext }; + struct validation_data data= { &user->user, user->pwtext.str ? &user->pwtext : &empy_lex_str }; Ok to push with this fix and your test case. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 95e1c46: MDEV-9898 SET ROLE NONE can crash mysqld.
by Sergei Golubchik 29 Apr '16

29 Apr '16

Hi, Alexey! On Apr 28, Alexey Botchkov wrote: > revision-id: 95e1c46c1709e7244332a025a527cacceab7da60 (mariadb-10.1.13-14-g95e1c46) > parent(s): 646c4cea58afbb369021a3d7b5ccbbf6aed708d4 > committer: Alexey Botchkov > timestamp: 2016-04-28 13:36:36 +0400 > message: > > MDEV-9898 SET ROLE NONE can crash mysqld. > The check_user_can_set_role() used find_user_exact() to get the > permissions for the SET ROLE NONE command. > Which returned NULL too often, for instance when user > authenticated as 'user'@'%'. > Now we use find_user_or_anon() instead. No, this is wrong. SET ROLE NONE should use the same rule as SET ROLE name. And the latter uses acl_user->wild_eq(user, host, ip). Thus SET ROLE NONE should find the user with ACL_USER::wild_eq, that is, it should use find_user_wild(). Ok to push with find_user_wild(). Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MariaDB packages for Alpine Linux
by Guillaume Lefranc 29 Apr '16

29 Apr '16

Hello, there is an outstanding request in docker-library/mariadb for Alpine support: https://github.com/docker-library/mariadb/issues/53 What I know is that we currently don't build packages for Alpine. Alpine is going to be the platform of choice for Docker containers in the future (you can read about the various advantages of Alpine here: https://www.brianchristner.io/docker-is-moving-to-alpine-linux/) Federico Razzoli has managed to make MariaDB 10.1 work in Alpine with some caveats (no tokudb, no oqgraph, no galera). You can look at his work here https://hub.docker.com/r/santec/mariadb-alpine/. I admit I don't know about portability of Galera lib on Alpine - should probably talk to Codership about that. Is there any way we can make this happen?

1 0

[Maria-developers] Why reserved keywords: UTC_TIME, UTC_DATE, UTC_DATETIME
by Alexander Barkov 29 Apr '16

29 Apr '16

Hello, I noticed that UTC_TIME, UTC_DATE, UTC_DATETIME are reserved keywords: MariaDB [test]> SELECT * FROM utc_time; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'utc_time' at line 1 I think they should not. - These keywords are not mentioned in the SQL standard - They are not involved in any complex grammar that would prevent them from being non-reserved keywords Looks like a bug... Any objections to make them non-reserved keywords in 10.2?

6 8

Re: [Maria-developers] GSoC 2016: Unique indexes for blobs
by Sergei Golubchik 27 Apr '16

27 Apr '16

Hi, Shubham! On Apr 24, Shubham Barai wrote: > Hi Sergei, > > I have created github repository with a new branch for my prototype. Great, what's the url? btw, please, use "Reply to all", so that the whole list could see your progress. Also google recommends you to blog regularly about what you're doing: http://write.flossmanuals.net/gsocstudentguide/time-management-for-students/ but if you don't like that, you can send weekly progress report emails instead. Whatever suits you best. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 6

[Maria-developers] GSoC Student Introduction
by Charles Muurmu 27 Apr '16

27 Apr '16

Hi everyone, I am Charles Muurmu also known as laserlight on IRC and I was selected for GSoC 2016 to work on migrating MariaDB Cassandra storage engine from the Thrift API to Datastax driver. I will be mentored by Spetrunia. My goals for this community bonding period will be get myself ready by accomplishing the following tasks: Fork and clone the server code. Try to build the code. Setup a blog on which I will post weekly reports on progress. Study the Datastax C++ driver API and documentation. Study the ha_cassandra.xx code and storage engines architecture. I believe these will prepare me for my first task when coding begins which will be to integrate the Datastax C++ driver into the MariaDB Server build system. I equally look forward to more instructions from my Mentor on how we will work on this task and lastly, I would love to congratulate the other students selected with MariaDB. Cheers, laserlight.

1 0

Re: [Maria-developers] [Commits] c204e9b: MDEV-9610 Trigger on normal table can't insert into CONNECT engine table - Access Denied
by Alexander Barkov 26 Apr '16

26 Apr '16

Hi Sergei, On 04/25/2016 08:42 PM, serg(a)mariadb.org wrote: > revision-id: c204e9bd63e7f6ce40e83b1f7fc0b4607a131009 (mariadb-10.0.24-43-gc204e9b) > parent(s): ce38adddfa91949b30956abd0e4cab201effcdef > author: Sergei Golubchik > committer: Sergei Golubchik > timestamp: 2016-04-25 18:41:31 +0200 > message: > > MDEV-9610 Trigger on normal table can't insert into CONNECT engine table - Access Denied > > in case of prelocking, don't check table->grant.privilege > in handler::external_lock(), do it in > handler::start_stmt(). > > --- > storage/connect/ha_connect.cc | 58 ++++++++++++++++++++-- > storage/connect/ha_connect.h | 1 + > storage/connect/mysql-test/connect/r/grant3.result | 5 ++ > storage/connect/mysql-test/connect/t/grant3.test | 11 ++++ > 4 files changed, 70 insertions(+), 5 deletions(-) > > diff --git a/storage/connect/ha_connect.cc b/storage/connect/ha_connect.cc > index 645d000..1032567 100644 > --- a/storage/connect/ha_connect.cc > +++ b/storage/connect/ha_connect.cc > @@ -4054,6 +4054,14 @@ int ha_connect::delete_all_rows() > } // end of delete_all_rows > > > +static void issue_access_denied_error(THD *thd) > +{ > + status_var_increment(thd->status_var.access_denied_errors); > + my_error(access_denied_error_code(thd->password), MYF(0), > + thd->security_ctx->priv_user, thd->security_ctx->priv_host, > + (thd->password ? ER(ER_YES) : ER(ER_NO))); > +} > + > bool ha_connect::check_privileges(THD *thd, PTOS options, char *dbn) > { > const char *db= (dbn && *dbn) ? dbn : NULL; > @@ -4124,12 +4132,9 @@ bool ha_connect::check_privileges(THD *thd, PTOS options, char *dbn) > */ > if (!table || !table->mdl_ticket || table->mdl_ticket->get_type() == MDL_EXCLUSIVE) > return check_access(thd, FILE_ACL, db, NULL, NULL, 0, 0); > - if (table->grant.privilege & FILE_ACL) > + if (thd->lex->requires_prelocking() || table->grant.privilege & FILE_ACL) > return false; > - status_var_increment(thd->status_var.access_denied_errors); > - my_error(access_denied_error_code(thd->password), MYF(0), > - thd->security_ctx->priv_user, thd->security_ctx->priv_host, > - (thd->password ? ER(ER_YES) : ER(ER_NO))); > + issue_access_denied_error(thd); > return true; > > // This is temporary until a solution is found > @@ -4146,6 +4151,46 @@ bool ha_connect::check_privileges(THD *thd, PTOS options, char *dbn) > return true; > } // end of check_privileges > > +bool ha_connect::check_cached_privileges(THD *thd) > +{ > + PTOS options= GetTableOptionStruct(); > + TABTYPE type=GetRealType(options); > + > +#ifdef NO_EMBEDDED_ACCESS_CHECKS > + return false; > +#endif > + > + switch (type) { > + case TAB_DOS: > + case TAB_FIX: > + case TAB_BIN: > + case TAB_CSV: > + case TAB_FMT: > + case TAB_DBF: > + case TAB_XML: > + case TAB_INI: > + case TAB_VEC: > + case TAB_JSON: > + if (!options->filename || !*options->filename) > + return false; > + /* Fall through to check FILE_ACL */ > + case TAB_ODBC: > + case TAB_MYSQL: > + case TAB_DIR: > + case TAB_MAC: > + case TAB_WMI: > + case TAB_OEM: > + if (table->grant.privilege & FILE_ACL) > + return false; > + issue_access_denied_error(thd); > + return true; > + default: The patch looks fine. I'd suggest two things. 1. Don't use "default" Can you please list all possible type value instead of "default"? Otherwise, when a new table type is added the next time, it will be easier to forget to update check_cached_privileges() accordingly. 2. Try to avoid duplicate code. Btw, I'd probably try to remove the duplicate switch, by adding a new method like this: enum Extra_acl { CONNECT_SECURE_FILE_PATH= 1, CONNECT_FILE_ACL= 1<<1; } int ha_connect::extra_acl_needed(THD *thd, PTOS options) { ... switch (GetRealType(options)) { ... case TAB_DOS: case TAB_FIX: case TAB_BIN: case TAB_CSV: case TAB_FMT: case TAB_DBF: case TAB_XML: case TAB_INI: case TAB_VEC: case TAB_JSON: return (options->filename && *options->filename) ? CONNECT_SECURE_FILE_PATH : 0; case TAB_ODBC: case TAB_MYSQL: case TAB_DIR: case TAB_MAC: case TAB_WMI: case TAB_OEM: #ifdef NO_EMBEDDED_ACCESS_CHECKS return 0; #endif return CONNECT_FILE_ACL; ... } ... } And then reuse it in both ha_connect::check_privilege() and ha_connect::check_cached_privileges(). The latter would look about like this: bool ha_connect::check_cached_privileges(THD *thd) { PTOS options= GetTableOptionStruct(); if (!(extra_acl_needed(options) & CONNECT_FILE_ACL) || (table->grant.privilege & FILE_ACL)) return false; issue_access_denied_error(thd); return true; } > + return false; > + } > + return false; > + > +} // end of check_cached_privileges > + > // Check that two indexes are equivalent > bool ha_connect::IsSameIndex(PIXDEF xp1, PIXDEF xp2) > { > @@ -4308,6 +4353,9 @@ int ha_connect::start_stmt(THD *thd, thr_lock_type lock_type) > PGLOBAL g= GetPlug(thd, xp); > DBUG_ENTER("ha_connect::start_stmt"); > > + if (check_cached_privileges(thd)) > + DBUG_RETURN(HA_ERR_INTERNAL_ERROR); > + > // Action will depend on lock_type > switch (lock_type) { > case TL_WRITE_ALLOW_WRITE: > diff --git a/storage/connect/ha_connect.h b/storage/connect/ha_connect.h > index 05cc872..fae804b 100644 > --- a/storage/connect/ha_connect.h > +++ b/storage/connect/ha_connect.h > @@ -537,6 +537,7 @@ int index_prev(uchar *buf); > > protected: > bool check_privileges(THD *thd, PTOS options, char *dbn); > + bool check_cached_privileges(THD *thd); It looks like the number of leading spaces is wrong here. > MODE CheckMode(PGLOBAL g, THD *thd, MODE newmode, bool *chk, bool *cras); > char *GetDBfromName(const char *name); > > diff --git a/storage/connect/mysql-test/connect/r/grant3.result b/storage/connect/mysql-test/connect/r/grant3.result > new file mode 100644 > index 0000000..2f9d37b > --- /dev/null > +++ b/storage/connect/mysql-test/connect/r/grant3.result > @@ -0,0 +1,5 @@ > +create table tcon (i int) engine=Connect table_type=DOS file_name='tcon.dos'; > +create table tin (i int); > +create trigger tr after insert on tin for each row insert into tcon values (new.i); > +insert into tin values (1); > +drop table tin,tcon; > diff --git a/storage/connect/mysql-test/connect/t/grant3.test b/storage/connect/mysql-test/connect/t/grant3.test > new file mode 100644 > index 0000000..9f05ca7 > --- /dev/null > +++ b/storage/connect/mysql-test/connect/t/grant3.test > @@ -0,0 +1,11 @@ > +# > +# MDEV-9610 Trigger on normal table can't insert into CONNECT engine table - Access Denied > +# > +create table tcon (i int) engine=Connect table_type=DOS file_name='tcon.dos'; > +create table tin (i int); > +create trigger tr after insert on tin for each row insert into tcon values (new.i); > +insert into tin values (1); > +drop table tin,tcon; > + > +let datadir=`select @@datadir`; > +remove_file $datadir/test/tcon.dos; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

2 1

Re: [Maria-developers] Deadlock with STOP SLAVE and LOCK_active_mi
by Michael Widenius 26 Apr '16

26 Apr '16

Hi! >>> Any suggestions for how this is supposed to work? Or is it just broken >>> by design, but saved because normally slave threads do not need to >>> access SHOW STATUS or system variables? <cut> > Hm. So I went through all of the code that references LOCK_active_mi to try > and understand this. It seems there are two main uses: > > 1. As you say, to serialise admin commands. I think the list is: end_slave() > (server shutdown); CHANGE MASTER; START/STOP SLAVE; START/STOP ALL SLAVES; > RESET SLAVE; FLUSH SLAVE. > > 2. To protect access to master_info_index, eg. to prevent a Master_info from > disappearing while it is accessed. When reading about this, I think it would be better to have a counter in Master_info if someone is using it and only delete if if the counter is zero. Would be trivial to add an increment of the counter in get_master_info(). The counter could either be protected by the LOCK_active_mi or one of the mutexes in Master_info like data_lock. > There is a comment in rpl_rli.h that LOCK_active_mi protects > Relay_log_info::inited, but I did not understand it - maybe it is wrong? Setting inited is indirectly protected by LOCK_active_mi, as init_all_master_info() which calls init_master_info(), is protected by LOCK_active_mi. Looking at the code, I didn't however see any need to protect inited as this is an internal flag that is always 1 after start. It's main usage is to avoid some re-initialization of relay logs on retries. > So the idea would be to make a new lock for (1), and keep LOCK_active_mi for > (2). > > Actually, using a lock for (1) seems a bad idea - STOP SLAVE can take a long > time to run, and a mutex makes it unkillable. A condition variable might be > better (to make STOP SLAVE killable). But I guess that is a separate issue, > it would not affect the possibility of deadlocks. Agree that a condition variable would be better, but not critical as one has already a problem if stop_slave doesn't work. > Both LOCK_active_mi and the new LOCK_serialize_replication_admin_commands > must by themselves prevent master_info_index from having elements added or > removed. This is needed for start_all_slaves() and stop_all_slaves() to work > correctly, I think. Why both? Isn't it enough to first take LOCK_serialize_replication_admin_commands and then take LOCK_active_mi if one needs to access master_info_index ? > Something will need to be done for remove_master_info(). Currently, it also > tries to stop slave threads (from within free_key_master_info()) while > holding LOCK_active_mi. It seems that threads should be stopped before > attempting to remove their mi from master_info_index, probably? To do that we would need to: - Add a flag in Master_info that it's not usable anymore and change this flag only under LOCK_active_mi. get_master_info() could wait (on a condition) if this flag is set. - Add a counter that Master_info is in use. - Add a function to release Master_info. - Call terminate_slave_threads() outside of free_key_master_info() > Actually, there would still be the deadlock problem after introducing the > new lock, because it is possible for a slave thread to run CHANGE MASTER! > (and I think START SLAVE / RESET SLAVE as well). But that is probably a bug, > does not seem good that this is possible. I guess these need a check to > throw an ER_LOCK_OR_ACTIVE_TRANSACTION error, like STOP SLAVE. Wouldn't the new lock LOCK_serialize_replication_admin_commands fix that? > I think I also found another lock problem while reading the code, for > MASTER_POS_WAIT() og SHOW BINLOG EVENTS. They both grab a Master_info under > LOCK_active_mi, but then proceed to access it after releasing the lock. I do > not see anything that prevents the Master_info to disappear under their feet > if RESET SLAVE ALL is run in another thread? But again, that's a separate > issue, I suppose. Probably there are some other tricky deadlock issues > lurking here. I checked the function mysql_show_binlog_events() but could not find any access to Master_info after mysql_mutex_unlock(&LOCK_active_mi) was released. > I don't know. It seems splitting into a new > LOCK_serialize_replication_admin_commands could solve the deadlock issue, > maybe, if the other problems mentioned above are addressed. It seems very > hard to convince oneself that this would not introduce new problems > somewhere, the locking in replication is really complex and feels very > fragile :-/ It doesn't really feel like something one would want to do in > 10.1 (certainly not 10.0), but maybe 10.2? > > It's kind of a very edge-case problem (and seems to have been there for a > _long_ time). Still, hanging the mysqld process hard, requiring kill -9, is > also not nice... Adding two new flags, one if master_info is in use and one if it's unusable, shouldn't be that hard to make reasonable safe in 10.1 I am now back in Finland and working. Feel free to call me to discuss this any time. Regards, Monty

2 2

[Maria-developers] Problem that some variables can be changed without stopping the slave IO thread
by Kristian Nielsen 25 Apr '16

25 Apr '16

Monty, This patch of yours introduced a change where before both slave threads must be stopped to change @@replicate_events_marked_for_skip, but after only the SQL thread is checked: ----------------------------------------------------------------------- commit 572560f38c248d5020f0e63aeb3f8905cd568208 Author: Michael Widenius <monty(a)askmonty.org> Date: Wed Oct 3 01:44:54 2012 +0300 Changed SHOW_FUNC variabels that don't return SHOW_ARRAY to SHOW_SIMPLE_FUNC. + Master_info_index::give_error_if_slave_running() + + @return + TRUE If some slave is running. An error is printed + FALSE No slave is running +*/ + +bool Master_info_index::give_error_if_slave_running() +{ + DBUG_ENTER("warn_if_slave_running"); + mysql_mutex_assert_owner(&LOCK_active_mi); + + for (uint i= 0; i< master_info_hash.records; ++i) + { + Master_info *mi; + mi= (Master_info *) my_hash_element(&master_info_hash, i); + if (mi->rli.slave_running != MYSQL_SLAVE_NOT_RUN) + { + my_error(ER_SLAVE_MUST_STOP, MYF(0), (int) mi->connection_name.length, + mi->connection_name.str); + DBUG_RETURN(TRUE); + } + } + DBUG_RETURN(FALSE); +} bool Sys_var_replicate_events_marked_for_skip::global_update(THD *thd, set_var *var) ... - /* Slave threads must be stopped to change the variable. */ + mysql_mutex_unlock(&LOCK_global_system_variables); mysql_mutex_lock(&LOCK_active_mi); - lock_slave_threads(active_mi); - init_thread_mask(&thread_mask, active_mi, 0 /*not inverse*/); - if (thread_mask) // We refuse if any slave thread is running - { - my_error(ER_SLAVE_MUST_STOP, MYF(0)); - result= true; - } - else + if (!master_info_index->give_error_if_slave_running()) ----------------------------------------------------------------------- So instead of checking both IO and SQL thread, now it checks only SQL thread (mi->rli.slave_running). Why was this change made? Was it by mistake? Unfortunately, I did not notice this before, so I subsequently used the same code (give_error_if_slave_running()) for the following variables: gtid_slave_pos slave_parallel_threads slave_domain_parallel_threads replicate_events_marked_for_skip gtid_ignore_duplicates This means that all of the above variables can be changed while the IO thread is still running. This was never the intention! There are some complex interactions between the master, the IO thread, and the SQL thread, particularly with respect to the slave's GTID position, and parts of the code are specifically written on the assumption that certain things cannot change without first stopping all replication threads. (I do not have the full overview of exactly what can break, I think that would require a careful review of large parts of the code, but I fear that it can corrupt not just the replication GTID position and possibly even internal server structures). So I am wondering what to do ... it is easy to fix give_error_if_slave_running() to check for both threads, but this is a change in behaviour (even if it should match documentation) that could easily break administration scripts, eg. after a 10.0 automatic security-fix upgrade. I also know that eg. Jean François has written about how he uses the feature of being able to not stop the IO thread in some cases. On the other hand, this can cause serious problems for users, as they have no real way of knowing not to attempt to change these variables without stopping IO thread first, when there is no error given. MDEV-9138 describes a complex scenario where slave position gets corrupted from this. Any suggestions? Thanks, - Kristian.

1 0

[Maria-developers] newest CVEs in mysql | mariadb vulnerable?
by Christian Rebischke 23 Apr '16

23 Apr '16

Hello, I just want to inform everybody about the newest mysql CVEs: Apr 21 CVE-2016-0666 Apr 21 CVE-2016-0663 Apr 21 CVE-2016-0665 Apr 21 CVE-2016-0661 Apr 21 CVE-2016-0662 Apr 21 CVE-2016-0659 Apr 21 CVE-2016-0658 Apr 21 CVE-2016-0657 Apr 21 CVE-2016-0656 Apr 21 CVE-2016-0654 Apr 21 CVE-2016-0655 Apr 21 CVE-2016-0653 Apr 21 CVE-2016-0652 Apr 21 CVE-2016-0651 Apr 21 CVE-2016-0650 Apr 21 CVE-2016-0649 Apr 21 CVE-2016-0648 Apr 21 CVE-2016-0647 Apr 21 CVE-2016-0646 Apr 21 CVE-2016-0644 Apr 21 CVE-2016-0643 Apr 21 CVE-2016-0642 Apr 21 CVE-2016-0641 Apr 21 CVE-2016-0639 Apr 21 CVE-2016-0640 Is there a list of current CVEs for mariadb there I can compare them? Best regards Chris

2 1

Re: [Maria-developers] InnoDB blob for primary key
by Sachin Setia 23 Apr '16

23 Apr '16

Hello Sergei! By adaptive hashing I refer to something like in innodb adaptive hashing http://dev.mysql.com/doc/refman/5.7/en/innodb-adaptive-hash.html On Tue, Apr 19, 2016 at 6:55 PM, sachin setiya <sachinsetia1001(a)gmail.com> wrote: > Hello > > > ---------- Forwarded message ---------- > From: Sachin Setia <sachinsetia1001(a)gmail.com> > To: maria-developers(a)lists.launchpad.net > Cc: > Date: Mon, 11 Apr 2016 15:03:24 +0530 > Subject: InnoDB blob for primary key > Hello Developers, > Hi this is sachin.Actually i was currently experimenting with with > blob uniques in innodb > there is three main problems > 1.Unique blob > 2.Blob Forigen key > 3.Blob primary key > > 1. For blob unique we can simply store hash in unclustered index > 2. Blob Forigen key i am currently working on it > 3. Blob primary key :- for this i thought we create a 4 byte column > which stores the hash of blob primary key.Internally this column will work > as > primary key and key for clustered index. I already successufully tested > this > here is my output > > MariaDB [sachin]> create table t4 (abc blob primary key); > Query OK, 0 rows affected (0.10 sec) > > MariaDB [sachin]> insert into t4 values('sachin'); > Query OK, 1 row affected (0.01 sec) > > MariaDB [sachin]> insert into t4 values('sachin'); > ERROR 1062 (23000): Duplicate entry 'sachin' for key 'PRIMARY' > MariaDB [sachin]> insert into t4 values('sachin setiya'); > Query OK, 1 row affected (0.00 sec) > > MariaDB [sachin]> insert into t4 values('sachin setiya'); > ERROR 1062 (23000): Duplicate entry 'sachin setiya' for key 'PRIMARY' > > MariaDB [sachin]> desc t4; > +-------+------+------+-----+---------+-------+ > | Field | Type | Null | Key | Default | Extra | > +-------+------+------+-----+---------+-------+ > | abc | blob | NO | PRI | NULL | | > +-------+------+------+-----+---------+-------+ > 1 row in set (0.01 sec) > > MariaDB [sachin]> select * from t4; > +---------------+ > | abc | > +---------------+ > | sachin | > | sachin setiya | > +---------------+ > 2 rows in set (0.01 sec) > > @Sergei hi! Actually i invested arround 2 months in mariadb So for me now > it does not matter either i got selected in gsoc i want to do work in > innodb blob unique from today.Please sir allow me to do this > > I am including the patch file and t4.ibd and t4.frm file > Regards > sachin > > > > > ---------- Forwarded message ---------- > From: "Jan Lindström" <jan.lindstrom(a)mariadb.com> > To: Sachin Setia <sachinsetia1001(a)gmail.com> > Cc: "maria-developers(a)lists.launchpad.net" < > maria-developers(a)lists.launchpad.net> > Date: Tue, 12 Apr 2016 14:22:29 +0300 > Subject: Re: [Maria-developers] InnoDB blob for primary key > Hi, > > Sachin, naturally you may continue, this is open source, please read > https://mariadb.com/kb/en/mariadb/community-contributing-to-the-mariadb-pro… > From InnoDB point of view there is issue if you add a new system generated > column to the row. Firstly, existing tables will not have that column and > secondly it will effect e.g maximum row size and external tools like backup. > > R: Jan > > On Mon, Apr 11, 2016 at 12:33 PM, Sachin Setia <sachinsetia1001(a)gmail.com> > wrote: > >> Hello Developers, >> Hi this is sachin.Actually i was currently experimenting with with >> blob uniques in innodb >> there is three main problems >> 1.Unique blob >> 2.Blob Forigen key >> 3.Blob primary key >> >> 1. For blob unique we can simply store hash in unclustered index >> 2. Blob Forigen key i am currently working on it >> 3. Blob primary key :- for this i thought we create a 4 byte column >> which stores the hash of blob primary key.Internally this column will >> work as >> primary key and key for clustered index. I already successufully tested >> this >> here is my output >> >> MariaDB [sachin]> create table t4 (abc blob primary key); >> Query OK, 0 rows affected (0.10 sec) >> >> MariaDB [sachin]> insert into t4 values('sachin'); >> Query OK, 1 row affected (0.01 sec) >> >> MariaDB [sachin]> insert into t4 values('sachin'); >> ERROR 1062 (23000): Duplicate entry 'sachin' for key 'PRIMARY' >> MariaDB [sachin]> insert into t4 values('sachin setiya'); >> Query OK, 1 row affected (0.00 sec) >> >> MariaDB [sachin]> insert into t4 values('sachin setiya'); >> ERROR 1062 (23000): Duplicate entry 'sachin setiya' for key 'PRIMARY' >> >> MariaDB [sachin]> desc t4; >> +-------+------+------+-----+---------+-------+ >> | Field | Type | Null | Key | Default | Extra | >> +-------+------+------+-----+---------+-------+ >> | abc | blob | NO | PRI | NULL | | >> +-------+------+------+-----+---------+-------+ >> 1 row in set (0.01 sec) >> >> MariaDB [sachin]> select * from t4; >> +---------------+ >> | abc | >> +---------------+ >> | sachin | >> | sachin setiya | >> +---------------+ >> 2 rows in set (0.01 sec) >> >> @Sergei hi! Actually i invested arround 2 months in mariadb So for me now >> it does not matter either i got selected in gsoc i want to do work in >> innodb blob unique from today.Please sir allow me to do this >> >> I am including the patch file and t4.ibd and t4.frm file >> Regards >> sachin >> >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers(a)lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp >> >> > > > ---------- Forwarded message ---------- > From: Sergei Golubchik <vuvova(a)gmail.com> > To: Sachin Setia <sachinsetia1001(a)gmail.com> > Cc: > Date: Wed, 13 Apr 2016 00:04:22 +0200 > Subject: Re: [Maria-developers] InnoDB blob for primary key > Hi, Sachin! > > Just a very quick comment now. We're having a company meeting > (and a meetup - > http://www.meetup.com/MariaDB-Developers-Berlin-Meetup/events/230026151/) > in Berlin, I'll send you a longer reply when this is all over (in a > couple of days). > > It might be an interesting option not to hack InnoDB, but to add > this feature (long UNIQUE keys) on the upper level. That is, in sql/ > directory. Automatically add a new BIGINT column, but hide it from > the user (not show it in SHOW COLUMNS and in SELECT *). Create > an index for it (also hidden). Put the blob's hash into this column. > And on collisions retrieve both rows and compare blobs. Just like MyISAM > is doing, but not on the storage engine level, but in the > engine-independent fashion, something that works for all engines. > > Want to look at this approach? > > MySQL 5.7 has something similar (long unique constraints on the sql > layer), but they've done it only for internal temporary tables (like > MyISAM's MI_UNIQUE is only used for internal temporary tables), > and it's not directly available for the user. Still, it might give you > some hinst about how to implement this feature on the sql layer. > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org > > On Apr 11, Sachin Setia wrote: > > Hello Developers, > > Hi this is sachin.Actually i was currently experimenting with with > > blob uniques in innodb > > there is three main problems > > 1.Unique blob > > 2.Blob Forigen key > > 3.Blob primary key > > > > 1. For blob unique we can simply store hash in unclustered index > > 2. Blob Forigen key i am currently working on it > > 3. Blob primary key :- for this i thought we create a 4 byte column > > which stores the hash of blob primary key.Internally this column will > work > > as > > primary key and key for clustered index. I already successufully tested > > this > > here is my output > > > > MariaDB [sachin]> create table t4 (abc blob primary key); > > Query OK, 0 rows affected (0.10 sec) > > > > MariaDB [sachin]> insert into t4 values('sachin'); > > Query OK, 1 row affected (0.01 sec) > > > > MariaDB [sachin]> insert into t4 values('sachin'); > > ERROR 1062 (23000): Duplicate entry 'sachin' for key 'PRIMARY' > > MariaDB [sachin]> insert into t4 values('sachin setiya'); > > Query OK, 1 row affected (0.00 sec) > > > > MariaDB [sachin]> insert into t4 values('sachin setiya'); > > ERROR 1062 (23000): Duplicate entry 'sachin setiya' for key 'PRIMARY' > > > > MariaDB [sachin]> desc t4; > > +-------+------+------+-----+---------+-------+ > > | Field | Type | Null | Key | Default | Extra | > > +-------+------+------+-----+---------+-------+ > > | abc | blob | NO | PRI | NULL | | > > +-------+------+------+-----+---------+-------+ > > 1 row in set (0.01 sec) > > > > MariaDB [sachin]> select * from t4; > > +---------------+ > > | abc | > > +---------------+ > > | sachin | > > | sachin setiya | > > +---------------+ > > 2 rows in set (0.01 sec) > > > > @Sergei hi! Actually i invested arround 2 months in mariadb So for me now > > it does not matter either i got selected in gsoc i want to do work in > > innodb blob unique from today.Please sir allow me to do this > > > > I am including the patch file and t4.ibd and t4.frm file > > Regards > > sachin > > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > > index dfce503..efd6f22 100644 > > --- a/sql/sql_table.cc > > +++ b/sql/sql_table.cc > > @@ -3244,7 +3244,7 @@ mysql_prepare_create_table(THD *thd, > HA_CREATE_INFO *create_info, > > !(sql_field->charset= find_bin_collation(sql_field->charset))) > > DBUG_RETURN(TRUE); > > > > - /* > > + /* > > Convert the default value from client character > > set into the column character set if necessary. > > */ > > @@ -3874,11 +3874,11 @@ mysql_prepare_create_table(THD *thd, > HA_CREATE_INFO *create_info, > > if (f_is_geom(sql_field->pack_flag) && sql_field->geom_type == > > Field::GEOM_POINT) > > column->length= MAX_LEN_GEOM_POINT_FIELD; > > - if (!column->length) > > - { > > - my_error(ER_BLOB_KEY_WITHOUT_LENGTH, MYF(0), > column->field_name.str); > > - DBUG_RETURN(TRUE); > > - } > > +// if (!column->length) //change > > +// { > > +// my_error(ER_BLOB_KEY_WITHOUT_LENGTH, MYF(0), > column->field_name.str); > > +// DBUG_RETURN(TRUE); > > +// } > > } > > #ifdef HAVE_SPATIAL > > if (key->type == Key::SPATIAL) > > @@ -3992,9 +3992,9 @@ mysql_prepare_create_table(THD *thd, > HA_CREATE_INFO *create_info, > > } > > else if (key_part_length == 0 && (sql_field->flags & > NOT_NULL_FLAG)) > > { > > - my_error(ER_WRONG_KEY_COLUMN, MYF(0), file->table_type(), > > - column->field_name.str); > > - DBUG_RETURN(TRUE); > > +// my_error(ER_WRONG_KEY_COLUMN, MYF(0), file->table_type(), > > +// column->field_name.str); > > +// DBUG_RETURN(TRUE); > > } > > if (key_part_length > file->max_key_part_length() && > > key->type != Key::FULLTEXT) > > diff --git a/storage/innobase/dict/dict0dict.cc > b/storage/innobase/dict/dict0dict.cc > > index c51deb0..f6e94af 100644 > > --- a/storage/innobase/dict/dict0dict.cc > > +++ b/storage/innobase/dict/dict0dict.cc > > @@ -1315,7 +1315,7 @@ dict_table_add_to_cache( > > #define BIG_ROW_SIZE 1024 > > > > ut_ad(mutex_own(&(dict_sys->mutex))); > > - > > + > > dict_table_add_system_columns(table, heap); > > > > table->cached = TRUE; > > diff --git a/storage/xtradb/dict/dict0dict.cc > b/storage/xtradb/dict/dict0dict.cc > > index 206038d..d8223d7 100644 > > --- a/storage/xtradb/dict/dict0dict.cc > > +++ b/storage/xtradb/dict/dict0dict.cc > > @@ -1251,7 +1251,7 @@ dict_table_open_on_name( > > Adds system columns to a table object. */ > > UNIV_INTERN > > void > > -dict_table_add_system_columns( > > +dict_table_add_system_columns( //work > > /*==========================*/ > > dict_table_t* table, /*!< in/out: table */ > > mem_heap_t* heap) /*!< in: temporary heap */ > > @@ -1266,16 +1266,25 @@ dict_table_add_system_columns( > > etc.) and as the last columns of the table memory object. > > The clustered index will not always physically contain all > > system columns. */ > > - > > +// dict_mem_table_add_col(table,heap,"DB_BLOB_HASH",DATA_INT > > +// ,DATA_NOT_NULL,DATA_BLOB_HASH_LEN); > > dict_mem_table_add_col(table, heap, "DB_ROW_ID", DATA_SYS, > > DATA_ROW_ID | DATA_NOT_NULL, > > DATA_ROW_ID_LEN); > > #if DATA_ROW_ID != 0 > > #error "DATA_ROW_ID != 0" > > #endif > > +// dict_mem_table_add_col(table,heap,"DB_BLOB_HASH",DATA_SYS, > > +// > DATA_BLOB_HASH|DATA_NOT_NULL, > > +// > DATA_BLOB_HASH_LEN); > > + //just a rough trick to get it working > > + > > +// if(*table->name=='y'){ > > dict_mem_table_add_col(table, heap, "DB_TRX_ID", DATA_SYS, > > DATA_TRX_ID | DATA_NOT_NULL, > > DATA_TRX_ID_LEN); > > +// } > > + > > #if DATA_TRX_ID != 1 > > #error "DATA_TRX_ID != 1" > > #endif > > @@ -1310,7 +1319,11 @@ dict_table_add_to_cache( > > ulint row_len; > > > > ut_ad(dict_lru_validate()); > > - > > +// bool x =false;//break > > +// if(x){ > > +// dict_mem_table_add_col(table,heap,"DB_BLOB_HASH",DATA_INT > > +// > ,DATA_NOT_NULL,4); > > +// } > > /* The lower limit for what we consider a "big" row */ > > #define BIG_ROW_SIZE 1024 > > > > @@ -3075,7 +3088,7 @@ dict_index_build_internal_clust( > > /* Copy the fields of index */ > > dict_index_copy(new_index, index, table, 0, index->n_fields); > > > > - if (dict_index_is_univ(index)) { > > + if (dict_index_is_univ(index)) { //work > > /* No fixed number of fields determines an entry uniquely > */ > > > > new_index->n_uniq = REC_MAX_N_FIELDS; > > @@ -3124,7 +3137,7 @@ dict_index_build_internal_clust( > > DATA_ROLL_PTR), > > 0); > > > > - for (i = 0; i < trx_id_pos; i++) { > > + for (i = 0; i < trx_id_pos; i++) {//work i think i need to > do some stuff > > > > ulint fixed_size = dict_col_get_fixed_size( > > dict_index_get_nth_col(new_index, i), > > diff --git a/storage/xtradb/dict/dict0load.cc > b/storage/xtradb/dict/dict0load.cc > > index d6ed8ac..8cc59f8 100644 > > --- a/storage/xtradb/dict/dict0load.cc > > +++ b/storage/xtradb/dict/dict0load.cc > > @@ -2276,7 +2276,7 @@ dictionary cache. > > ibd_file_missing flag TRUE in the table object we return */ > > UNIV_INTERN > > dict_table_t* > > -dict_load_table( > > +dict_load_table( //work for corruped table > > /*============*/ > > const char* name, /*!< in: table name in the > > databasename/tablename format */ > > @@ -2337,7 +2337,7 @@ dict_load_table( > > btr_pcur_close(&pcur); > > mtr_commit(&mtr); > > mem_heap_free(heap); > > - > > + > > return(NULL); > > } > > > > diff --git a/storage/xtradb/dict/dict0mem.cc > b/storage/xtradb/dict/dict0mem.cc > > index a4f6cd6..6cbe556 100644 > > --- a/storage/xtradb/dict/dict0mem.cc > > +++ b/storage/xtradb/dict/dict0mem.cc > > @@ -101,11 +101,11 @@ dict_mem_table_create( > > memcpy(table->name, name, strlen(name) + 1); > > table->is_system_db = dict_mem_table_is_system(table->name); > > table->space = (unsigned int) space; > > - table->n_cols = (unsigned int) (n_cols + DATA_N_SYS_COLS); > > - > > + table->n_cols = (unsigned int) (n_cols + DATA_N_SYS_COLS); //work > > + > > table->cols = static_cast<dict_col_t*>( > > mem_heap_alloc(heap, > > - (n_cols + DATA_N_SYS_COLS) > > + (n_cols + DATA_N_SYS_COLS)//work > > * sizeof(dict_col_t))); > > > > ut_d(table->magic_n = DICT_TABLE_MAGIC_N); > > diff --git a/storage/xtradb/handler/ha_innodb.cc > b/storage/xtradb/handler/ha_innodb.cc > > index e5edf76..3a8dc91 100644 > > --- a/storage/xtradb/handler/ha_innodb.cc > > +++ b/storage/xtradb/handler/ha_innodb.cc > > @@ -5983,7 +5983,7 @@ ha_innobase::open( > > ibool par_case_name_set = FALSE; > > char par_case_name[FN_REFLEN]; > > dict_err_ignore_t ignore_err = DICT_ERR_IGNORE_NONE; > > - > > + ibool is_blob_primary_key=false; > > DBUG_ENTER("ha_innobase::open"); > > > > UT_NOT_USED(mode); > > @@ -6031,13 +6031,20 @@ ha_innobase::open( > > > > /* Get pointer to a table object in InnoDB dictionary cache */ > > ib_table = dict_table_open_on_name(norm_name, FALSE, TRUE, > ignore_err); > > - > > + //int number_of_columns = dict_table_get_n_user_cols(ib_table); > > + //if(ib_table->) > > + if(ib_table){ > > + int number_of_columns = > dict_table_get_n_user_cols(ib_table); > > + > if(!innobase_strcasecmp(ib_table->col_names,"DB_BLOB_HASH")){ > > + is_blob_primary_key=true; > > + number_of_columns--;//stodo i think we need to add > flag for blob primary key to make checking easier > > + } > > if (ib_table > > - && ((!DICT_TF2_FLAG_IS_SET(ib_table, DICT_TF2_FTS_HAS_DOC_ID) > > - && table->s->stored_fields != > dict_table_get_n_user_cols(ib_table)) > > + && ((!DICT_TF2_FLAG_IS_SET(ib_table, DICT_TF2_FTS_HAS_DOC_ID)) > > + && table->s->stored_fields != number_of_columns) //work > > || (DICT_TF2_FLAG_IS_SET(ib_table, DICT_TF2_FTS_HAS_DOC_ID) > > && (table->s->fields > > - != dict_table_get_n_user_cols(ib_table) - 1)))) { > > + != dict_table_get_n_user_cols(ib_table) - 1))) { > > ib_logf(IB_LOG_LEVEL_WARN, > > "table %s contains %lu user defined columns " > > "in InnoDB, but %lu columns in MySQL. Please " > > @@ -6054,7 +6061,7 @@ ha_innobase::open( > > ib_table = NULL; > > is_part = NULL; > > } > > - > > + } > > if (UNIV_UNLIKELY(ib_table && ib_table->is_corrupt && > > srv_pass_corrupt_table <= 1)) { > > free_share(share); > > @@ -6254,12 +6261,12 @@ ha_innobase::open( > > /* Looks like MySQL-3.23 sometimes has primary key number != 0 */ > > primary_key = table->s->primary_key; > > key_used_on_scan = primary_key; > > - > > + if(!is_blob_primary_key){ > > if (!innobase_build_index_translation(table, ib_table, share)) { > > sql_print_error("Build InnoDB index translation table > for" > > " Table %s failed", name); > > } > > - > > + } > > /* Allocate a buffer for a 'row reference'. A row reference is > > a string of bytes of length ref_length which uniquely specifies > > a row in our table. Note that MySQL may also compare two row > > @@ -6314,7 +6321,11 @@ ha_innobase::open( > > for (uint i = 0; i < table->s->keys; i++) { > > dict_index_t* index; > > index = innobase_get_index(i); > > - if (dict_index_is_clust(index)) { > > + if (dict_index_is_clust(index)) { //work > > + if(is_blob_primary_key){ > > + ref_length=4; //hash length > > + continue; > > + } > > ref_length = > > > table->key_info[i].key_length; > > } > > @@ -7795,7 +7806,8 @@ build_template_field( > > > > //ut_ad(field == table->field[i]); > > ut_ad(clust_index->table == index->table); > > - > > +//work here we go > > +//todo it should get the next column defs > > col = dict_table_get_nth_col(index->table, i); > > > > templ = prebuilt->mysql_template + prebuilt->n_template++; > > @@ -8138,7 +8150,8 @@ ha_innobase::build_template( > > continue; > > } > > } > > - > > + > if(!innobase_strcasecmp(clust_index->fields->name,"DB_BLOB_HASH")) > > + i++; > > build_template_field(prebuilt, clust_index, index, > > table, field, i); > > } > > @@ -11065,6 +11078,7 @@ create_table_def( > > ulint doc_id_col = 0; > > ibool has_doc_id_col = FALSE; > > mem_heap_t* heap; > > + bool is_blob_primary_key=false; > > > > DBUG_ENTER("create_table_def"); > > DBUG_PRINT("enter", ("table_name: %s", table_name)); > > @@ -11126,6 +11140,12 @@ create_table_def( > > table->fts->doc_col = doc_id_col; > > } > > } else { > > + if(form->key_info[0].key_part->length==0 && //change > > + form->key_info[0].key_part->key_type|MYSQL_TYPE_BLOB){ > > + s_cols++; > > + is_blob_primary_key=true; > > + } > > + > > table = dict_mem_table_create(table_name, 0, s_cols, > > flags, flags2); > > } > > @@ -11143,6 +11163,13 @@ create_table_def( > > table->data_dir_path = NULL; > > } > > heap = mem_heap_create(1000); > > + //work > > + //add one more column for hash > > + > > + if(is_blob_primary_key){ > > + dict_mem_table_add_col(table,heap,"DB_BLOB_HASH", > > + DATA_INT,1283,4); > > + } > > > > for (i = 0; i < n_cols; i++) { > > Field* field = form->field[i]; > > @@ -11222,7 +11249,7 @@ create_table_def( > > err = DB_ERROR; > > goto error_ret; > > } > > - > > + > > dict_mem_table_add_col(table, heap, > > field->field_name, > > col_type, > > @@ -11347,7 +11374,7 @@ create_index( > > the length of the key part versus the column. */ > > > > Field* field = NULL; > > - > > +//work > > for (ulint j = 0; j < form->s->fields; j++) { > > > > field = form->field[j]; > > @@ -11396,7 +11423,12 @@ create_index( > > } > > > > field_lengths[i] = key_part->length; > > - > > + if(form->key_info[0].key_part->length==0 && //change > > + form->key_info[0].key_part->key_type|MYSQL_TYPE_BLOB){ > > + dict_mem_index_add_field( > > + index,"DB_BLOB_HASH", 0); > > + continue; > > + } > > dict_mem_index_add_field( > > index, key_part->field->field_name, prefix_len); > > } > > diff --git a/storage/xtradb/include/data0type.h > b/storage/xtradb/include/data0type.h > > index 111664b..0f510e8 100644 > > --- a/storage/xtradb/include/data0type.h > > +++ b/storage/xtradb/include/data0type.h > > @@ -147,6 +147,9 @@ be less than 256 */ > > #define DATA_ROLL_PTR 2 /* rollback data pointer: 7 bytes > */ > > #define DATA_ROLL_PTR_LEN 7 > > > > +//#define DATA_BLOB_HASH 3 // hash coloumn for blob primay key > > +//#define DATA_BLOB_HASH_LEN 4 //used as a clustered index > > + > > #define DATA_N_SYS_COLS 3 /* number of system columns > defined above */ > > > > #define DATA_FTS_DOC_ID 3 /* Used as FTS DOC ID column */ > > diff --git a/storage/xtradb/que/que0que.cc > b/storage/xtradb/que/que0que.cc > > index e2dc023..076785d 100644 > > --- a/storage/xtradb/que/que0que.cc > > +++ b/storage/xtradb/que/que0que.cc > > @@ -1030,7 +1030,7 @@ que_thr_step( > > que_node_print_info(node); > > } > > #endif > > - if (type & QUE_NODE_CONTROL_STAT) { > > + if (type & QUE_NODE_CONTROL_STAT) { //improve > > if ((thr->prev_node != que_node_get_parent(node)) > > && que_node_get_next(thr->prev_node)) { > > > > diff --git a/storage/xtradb/row/row0ins.cc > b/storage/xtradb/row/row0ins.cc > > index d02e179..c58d7fb 100644 > > --- a/storage/xtradb/row/row0ins.cc > > +++ b/storage/xtradb/row/row0ins.cc > > @@ -137,7 +137,7 @@ row_ins_alloc_sys_fields( > > /*=====================*/ > > ins_node_t* node) /*!< in: insert node */ > > { > > - dtuple_t* row; > > + dtuple_t* row; //work > > dict_table_t* table; > > mem_heap_t* heap; > > const dict_col_t* col; > > @@ -3118,7 +3118,7 @@ row_ins_index_entry_set_vals( > > > > n_fields = dtuple_get_n_fields(entry); > > > > - for (i = 0; i < n_fields; i++) { > > + for (i = 0; i < n_fields; i++) {//see > > dict_field_t* ind_field; > > dfield_t* field; > > const dfield_t* row_field; > > @@ -3169,8 +3169,8 @@ row_ins_index_entry_step( > > > > ut_ad(dtuple_check_typed(node->row)); > > > > - row_ins_index_entry_set_vals(node->index, node->entry, node->row); > > - > > + row_ins_index_entry_set_vals(node->index, node->entry, > node->row);//explore > > +//need to see who sets the entry in row > > ut_ad(dtuple_check_typed(node->entry)); > > > > err = row_ins_index_entry(node->index, node->entry, thr); > > @@ -3208,7 +3208,7 @@ row_ins_alloc_row_id_step( > > > > /* Fill in row id value to row */ > > > > - row_id = dict_sys_get_new_row_id(); > > + row_id = dict_sys_get_new_row_id(); //work > > > > dict_sys_write_row_id(node->row_id_buf, row_id); > > } > > diff --git a/storage/xtradb/row/row0mysql.cc > b/storage/xtradb/row/row0mysql.cc > > index 9427b20..b28e778 100644 > > --- a/storage/xtradb/row/row0mysql.cc > > +++ b/storage/xtradb/row/row0mysql.cc > > @@ -510,7 +510,15 @@ row_mysql_store_col_in_innobase_format( > > > > return(buf); > > } > > - > > +//this is just dummy function > > +int dummuy_hash(byte * ptr,int length){ > > + int val=0; > > + for(int i=0;i<length;i++){ > > + val =val*8+(int )*ptr; > > + ptr++; > > + } > > + return val; > > +} > > /**************************************************************//** > > Convert a row in the MySQL format to a row in the Innobase format. Note > that > > the function to convert a MySQL format key value to an InnoDB dtuple is > > @@ -532,15 +540,22 @@ row_mysql_convert_row_to_innobase( > > const mysql_row_templ_t*templ; > > dfield_t* dfield; > > ulint i; > > - > > + ibool is_blob_primary_key=false; > > + ibool is_first_template=true; > > ut_ad(prebuilt->template_type == ROW_MYSQL_WHOLE_ROW); > > - ut_ad(prebuilt->mysql_template); > > - > > - for (i = 0; i < prebuilt->n_template; i++) { > > + ut_ad(prebuilt->mysql_template);//work > > + > if(!innobase_strcasecmp(prebuilt->table->col_names,"DB_BLOB_HASH")){ > > + is_blob_primary_key =true; > > + } > > + for (i = 0; i < prebuilt->n_template; i++) { //change add one > > > > templ = prebuilt->mysql_template + i; > > + if(is_blob_primary_key&&is_first_template){ > > + dfield = dtuple_get_nth_field(row, i+1); > > + is_first_template=false; > > + }else{ > > dfield = dtuple_get_nth_field(row, i); > > - > > + } > > if (templ->mysql_null_bit_mask != 0) { > > /* Column may be SQL NULL */ > > > > @@ -565,6 +580,17 @@ row_mysql_convert_row_to_innobase( > > next_column: > > ; > > } > > + if(is_blob_primary_key){ > > + //get the first field and set the hash > > + dfield_t * hash_field=dtuple_get_nth_field(row,0); > > + dfield_t * blob_field=dtuple_get_nth_field(row,1); > > + byte * hash = static_cast<byte > *>(mem_heap_zalloc(prebuilt->heap,4)); > > + mach_write_to_4(hash,dummuy_hash((byte > *)blob_field->data,blob_field->len)); > > + //4 is hash length > > + hash_field->len=4; > > + hash_field->data=hash; > > + //hash_field->type > > + } > > > > /* If there is a FTS doc id column and it is not user supplied ( > > generated by server) then assign it a new doc id. */ > > @@ -831,7 +857,7 @@ row_create_prebuilt( > > > > prebuilt->search_tuple = dtuple_create(heap, > search_tuple_n_fields); > > > > - ref = dtuple_create(heap, ref_len); > > + ref = dtuple_create(heap, ref_len);//work > > > > dict_index_copy_types(ref, clust_index, ref_len); > > > > @@ -1057,7 +1083,7 @@ row_get_prebuilt_insert_row( > > > > dict_table_copy_types(row, table); > > > > - ins_node_set_new_row(node, row); > > + ins_node_set_new_row(node, row);//explore > > > > prebuilt->ins_graph = static_cast<que_fork_t*>( > > que_node_get_parent( > > @@ -1353,7 +1379,7 @@ row_insert_for_mysql( > > row_get_prebuilt_insert_row(prebuilt); > > node = prebuilt->ins_node; > > > > - row_mysql_convert_row_to_innobase(node->row, prebuilt, mysql_rec); > > + row_mysql_convert_row_to_innobase(node->row, prebuilt, > mysql_rec);//debug > > > > savept = trx_savept_take(trx); > > > > diff --git a/storage/xtradb/row/row0row.cc > b/storage/xtradb/row/row0row.cc > > index be786f9..a1dd465 100644 > > --- a/storage/xtradb/row/row0row.cc > > +++ b/storage/xtradb/row/row0row.cc > > @@ -83,7 +83,7 @@ row_build_index_entry_low( > > entry, dict_index_get_n_unique_in_tree(index)); > > } > > > > - for (i = 0; i < entry_len; i++) { > > + for (i = 0; i < entry_len; i++) { //explore need to see how it > works for simple > > const dict_field_t* ind_field > > = dict_index_get_nth_field(index, i); > > const dict_col_t* col > > > > ---------- Forwarded message ---------- > From: Sachin Setia <sachinsetia1001(a)gmail.com> > To: maria-developers(a)lists.launchpad.net > Cc: > Date: Wed, 13 Apr 2016 21:17:03 +0530 > Subject: Re: [Maria-developers] InnoDB blob for primary key > Hello Jan! > > Actually I was going through the problems raised by you > > 1>Firstly, existing tables will not have that column > > Yes Sir, you are right that existing table woudn't have DB_BLOB_HASH > column, but this is fine because they indexed there blob column then > they must have provided blob length, and if so then length of key_part > would not be zero and so need to make changes in patch to detect this. > And also by doing I can also eliminate another problem what if existing > table have a coloumn named 'DB_BLOB_HASH' > > 2>secondly it will effect e.g maximum row size > For this i can increase reclength by 4 in pack_header function(but i > am not sure whether this will work or not) > > 3>and external tools like backup > Here, I am assuming that by backup you mean mysqldump actually i tried > this by create a table with blob coloum (length was given ) and than > changing mariadb binary(to changed code) and .sql file of backup was > working fine > > But I think there are more problems with this this prototype > 1>Existing table with coloumn named 'DB_BLOB_HASH' .We can solve this > as written above but some code in patch compares first coloumn name > with 'DB_BLOB_HASH' so this will create problem Sir, can we add a flag > in dict_table_t->flag so that we can reliable check for blob hash coloumn > > 2>This is very bad problem , what happen if if data is different and crc > hash of data is same.And this will occur when we have medium amount of > data like 100 thousand records.To avoid this we can do like > > 2.1> use crc 64 reduces probability > 2.2> use two different algorithm for hash and treat these two algoriths > hash for uniquesness.Make collusion probability very very less > > But the main problem is what would happen if collusion occurs to solve > this i have two ideas > 2.3> like 2.2 but instead of using two algorithm use one and add column > for number of times this hash is repeated collusion frequency will be zero > i think this will work fine but need to test it > 2.4> whole prototype is crap start again with some different method. Can > you > please suggest some differnt way of doing this > Regards > sachin > > On Tue, Apr 12, 2016 at 4:52 PM, Jan Lindström > <jan.lindstrom(a)mariadb.com> wrote: > > Hi, > > > > Sachin, naturally you may continue, this is open source, please read > > > https://mariadb.com/kb/en/mariadb/community-contributing-to-the-mariadb-pro… > > From InnoDB point of view there is issue if you add a new system > generated > > column to the row. Firstly, existing tables will not have that column and > > secondly it will effect e.g maximum row size and external tools like > backup. > > > > R: Jan > > > > On Mon, Apr 11, 2016 at 12:33 PM, Sachin Setia < > sachinsetia1001(a)gmail.com> > > wrote: > >> > >> Hello Developers, > >> Hi this is sachin.Actually i was currently experimenting with with > >> blob uniques in innodb > >> there is three main problems > >> 1.Unique blob > >> 2.Blob Forigen key > >> 3.Blob primary key > >> > >> 1. For blob unique we can simply store hash in unclustered index > >> 2. Blob Forigen key i am currently working on it > >> 3. Blob primary key :- for this i thought we create a 4 byte column > >> which stores the hash of blob primary key.Internally this column will > work > >> as > >> primary key and key for clustered index. I already successufully tested > >> this > >> here is my output > >> > >> MariaDB [sachin]> create table t4 (abc blob primary key); > >> Query OK, 0 rows affected (0.10 sec) > >> > >> MariaDB [sachin]> insert into t4 values('sachin'); > >> Query OK, 1 row affected (0.01 sec) > >> > >> MariaDB [sachin]> insert into t4 values('sachin'); > >> ERROR 1062 (23000): Duplicate entry 'sachin' for key 'PRIMARY' > >> MariaDB [sachin]> insert into t4 values('sachin setiya'); > >> Query OK, 1 row affected (0.00 sec) > >> > >> MariaDB [sachin]> insert into t4 values('sachin setiya'); > >> ERROR 1062 (23000): Duplicate entry 'sachin setiya' for key 'PRIMARY' > >> > >> MariaDB [sachin]> desc t4; > >> +-------+------+------+-----+---------+-------+ > >> | Field | Type | Null | Key | Default | Extra | > >> +-------+------+------+-----+---------+-------+ > >> | abc | blob | NO | PRI | NULL | | > >> +-------+------+------+-----+---------+-------+ > >> 1 row in set (0.01 sec) > >> > >> MariaDB [sachin]> select * from t4; > >> +---------------+ > >> | abc | > >> +---------------+ > >> | sachin | > >> | sachin setiya | > >> +---------------+ > >> 2 rows in set (0.01 sec) > >> > >> @Sergei hi! Actually i invested arround 2 months in mariadb So for me > now > >> it does not matter either i got selected in gsoc i want to do work in > >> innodb blob unique from today.Please sir allow me to do this > >> > >> I am including the patch file and t4.ibd and t4.frm file > >> Regards > >> sachin > >> > >> > >> > >> _______________________________________________ > >> Mailing list: https://launchpad.net/~maria-developers > >> Post to : maria-developers(a)lists.launchpad.net > >> Unsubscribe : https://launchpad.net/~maria-developers > >> More help : https://help.launchpad.net/ListHelp > >> > > > > > ---------- Forwarded message ---------- > From: Sachin Setia <sachinsetia1001(a)gmail.com> > To: maria-developers(a)lists.launchpad.net > Cc: > Date: Wed, 13 Apr 2016 22:19:44 +0530 > Subject: Re: [Maria-developers] InnoDB blob for primary key > Hello Sergei > Sorry I did not see your mail. Actually i was thinking something like > this before implementing the prototype but if i am more closer to > innodb the more performance i will i get. I will definitively think > about it. > Regards > sachin > > On Wed, Apr 13, 2016 at 9:17 PM, Sachin Setia <sachinsetia1001(a)gmail.com> > wrote: > > Hello Jan! > > > > Actually I was going through the problems raised by you > > > > 1>Firstly, existing tables will not have that column > > > > Yes Sir, you are right that existing table woudn't have DB_BLOB_HASH > > column, but this is fine because they indexed there blob column then > > they must have provided blob length, and if so then length of key_part > > would not be zero and so need to make changes in patch to detect this. > > And also by doing I can also eliminate another problem what if existing > > table have a coloumn named 'DB_BLOB_HASH' > > > > 2>secondly it will effect e.g maximum row size > > For this i can increase reclength by 4 in pack_header function(but i > > am not sure whether this will work or not) > > > > 3>and external tools like backup > > Here, I am assuming that by backup you mean mysqldump actually i tried > > this by create a table with blob coloum (length was given ) and than > > changing mariadb binary(to changed code) and .sql file of backup was > > working fine > > > > But I think there are more problems with this this prototype > > 1>Existing table with coloumn named 'DB_BLOB_HASH' .We can solve this > > as written above but some code in patch compares first coloumn name > > with 'DB_BLOB_HASH' so this will create problem Sir, can we add a flag > > in dict_table_t->flag so that we can reliable check for blob hash coloumn > > > > 2>This is very bad problem , what happen if if data is different and crc > > hash of data is same.And this will occur when we have medium amount of > > data like 100 thousand records.To avoid this we can do like > > > > 2.1> use crc 64 reduces probability > > 2.2> use two different algorithm for hash and treat these two algoriths > > hash for uniquesness.Make collusion probability very very less > > > > But the main problem is what would happen if collusion occurs to solve > > this i have two ideas > > 2.3> like 2.2 but instead of using two algorithm use one and add column > > for number of times this hash is repeated collusion frequency will be > zero > > i think this will work fine but need to test it > > 2.4> whole prototype is crap start again with some different method. Can > you > > please suggest some differnt way of doing this > > Regards > > sachin > > > > On Tue, Apr 12, 2016 at 4:52 PM, Jan Lindström > > <jan.lindstrom(a)mariadb.com> wrote: > >> Hi, > >> > >> Sachin, naturally you may continue, this is open source, please read > >> > https://mariadb.com/kb/en/mariadb/community-contributing-to-the-mariadb-pro… > >> From InnoDB point of view there is issue if you add a new system > generated > >> column to the row. Firstly, existing tables will not have that column > and > >> secondly it will effect e.g maximum row size and external tools like > backup. > >> > >> R: Jan > >> > >> On Mon, Apr 11, 2016 at 12:33 PM, Sachin Setia < > sachinsetia1001(a)gmail.com> > >> wrote: > >>> > >>> Hello Developers, > >>> Hi this is sachin.Actually i was currently experimenting with with > >>> blob uniques in innodb > >>> there is three main problems > >>> 1.Unique blob > >>> 2.Blob Forigen key > >>> 3.Blob primary key > >>> > >>> 1. For blob unique we can simply store hash in unclustered index > >>> 2. Blob Forigen key i am currently working on it > >>> 3. Blob primary key :- for this i thought we create a 4 byte column > >>> which stores the hash of blob primary key.Internally this column will > work > >>> as > >>> primary key and key for clustered index. I already successufully tested > >>> this > >>> here is my output > >>> > >>> MariaDB [sachin]> create table t4 (abc blob primary key); > >>> Query OK, 0 rows affected (0.10 sec) > >>> > >>> MariaDB [sachin]> insert into t4 values('sachin'); > >>> Query OK, 1 row affected (0.01 sec) > >>> > >>> MariaDB [sachin]> insert into t4 values('sachin'); > >>> ERROR 1062 (23000): Duplicate entry 'sachin' for key 'PRIMARY' > >>> MariaDB [sachin]> insert into t4 values('sachin setiya'); > >>> Query OK, 1 row affected (0.00 sec) > >>> > >>> MariaDB [sachin]> insert into t4 values('sachin setiya'); > >>> ERROR 1062 (23000): Duplicate entry 'sachin setiya' for key 'PRIMARY' > >>> > >>> MariaDB [sachin]> desc t4; > >>> +-------+------+------+-----+---------+-------+ > >>> | Field | Type | Null | Key | Default | Extra | > >>> +-------+------+------+-----+---------+-------+ > >>> | abc | blob | NO | PRI | NULL | | > >>> +-------+------+------+-----+---------+-------+ > >>> 1 row in set (0.01 sec) > >>> > >>> MariaDB [sachin]> select * from t4; > >>> +---------------+ > >>> | abc | > >>> +---------------+ > >>> | sachin | > >>> | sachin setiya | > >>> +---------------+ > >>> 2 rows in set (0.01 sec) > >>> > >>> @Sergei hi! Actually i invested arround 2 months in mariadb So for me > now > >>> it does not matter either i got selected in gsoc i want to do work in > >>> innodb blob unique from today.Please sir allow me to do this > >>> > >>> I am including the patch file and t4.ibd and t4.frm file > >>> Regards > >>> sachin > >>> > >>> > >>> > >>> _______________________________________________ > >>> Mailing list: https://launchpad.net/~maria-developers > >>> Post to : maria-developers(a)lists.launchpad.net > >>> Unsubscribe : https://launchpad.net/~maria-developers > >>> More help : https://help.launchpad.net/ListHelp > >>> > >> > > > ---------- Forwarded message ---------- > From: Sergei Golubchik <serg(a)mariadb.org> > To: Sachin Setia <sachinsetia1001(a)gmail.com> > Cc: > Date: Thu, 14 Apr 2016 20:53:51 +0200 > Subject: Re: [Maria-developers] InnoDB blob for primary key > Hi, Sachin! > > On Apr 13, Sachin Setia wrote: > > Hello Sergei > > Sorry I did not see your mail. Actually i was thinking something like > > this before implementing the prototype but if i am more closer to > > innodb the more performance i will i get. I will definitively think > > about it. > > Great! > > Could you please tell me (mailing list, that is) what you think before > next Monday (before April 18h, that is)? > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org > > > ---------- Forwarded message ---------- > From: Sachin Setia <sachinsetia1001(a)gmail.com> > To: maria-developers(a)lists.launchpad.net > Cc: > Date: Fri, 15 Apr 2016 18:37:15 +0530 > Subject: Re: [Maria-developers] InnoDB blob for primary key > Hi Sergei! > > Actually I was going through the mysql source code for unique long > constraints > in file sql_tmp_table.cc in function create_tmp_table they make a new field > and a new key(hash_key) and pass this table obejct to storage > engine.They actually > refer this field as a hash field > On the time of insert they call bool check_unique_constraint(TABLE > *table) function > which first calculate the hash and store it in field then they see for > duplicate > hash and retrive ha_index_next_same if records are not same then record > > We can do the same thing in mariadb by adding one more field and key in > mysql_prepare_create_table in this we check for blob with unlimited > length or varchar for length > greater then internal storage engine by doing this in > mysql_prepare_create_table there > will be no issues of frm file inconsistance. > > In case of insert first we will fill the hash field in fill_record > function of sql_base.cc > by first calculating the hash. Then we will retrive the index map > using ha_index_read_map > if returened value is zero then we will comapare two records and if > they match then we will through error > I am not sure where to place this code either in fill_record or later > Or i can simple just > fill hash in field in fill_record and then check for duplicates later on. > > Current I am not sure how to hide columns from user.Sir, can you > suggest me where to look > > But there is one problem we can make unique key by this approch but > not primary key because primary key > is clustered and hashes can collide so i think we can't use hash field > as primary key. To overcome this problem > I have one idea instead of storing just hash we can make hash field > length 10 bytes and in last two bytes > we can store short int which tells how much time hash is repeated this > can make hash unique > in case of collusion. And also we are not doing more computation > because we already retrive > all records with same hashes. > What do you think of this idea?. > And there is one more problem how to make it foreign key. > > Will send you a prototype code tomorrow. > Regards > sachin > > On Fri, Apr 15, 2016 at 12:23 AM, Sergei Golubchik <serg(a)mariadb.org> > wrote: > > Hi, Sachin! > > > > On Apr 13, Sachin Setia wrote: > >> Hello Sergei > >> Sorry I did not see your mail. Actually i was thinking something like > >> this before implementing the prototype but if i am more closer to > >> innodb the more performance i will i get. I will definitively think > >> about it. > > > > Great! > > > > Could you please tell me (mailing list, that is) what you think before > > next Monday (before April 18h, that is)? > > > > Regards, > > Sergei > > Chief Architect MariaDB > > and security(a)mariadb.org > > > ---------- Forwarded message ---------- > From: Sachin Setia <sachinsetia1001(a)gmail.com> > To: maria-developers(a)lists.launchpad.net > Cc: > Date: Sat, 16 Apr 2016 19:37:45 +0530 > Subject: Re: [Maria-developers] InnoDB blob for primary key > Hi Sergei! > As I already told you i was building prototype.It is some what completed > apart from one thing comparing of two field values. the difficulty is how > to get data length of field from table->record[1]. I will try to solve it. > One more thing actually i got how mysql hide field. For example condsider > three > fields hash,data,data. mysql field pointer point at second field not at > hash > field and hash field ptr is stored in table->hash_field > can we do something similar to store hash fields(if we make array of > hashes in > case of more than one unique).But will adding member variable cause > problem? > what do you think? > Regards > sachin > > On Fri, Apr 15, 2016 at 6:37 PM, Sachin Setia <sachinsetia1001(a)gmail.com> > wrote: > > Hi Sergei! > > > > Actually I was going through the mysql source code for unique long > constraints > > in file sql_tmp_table.cc in function create_tmp_table they make a new > field > > and a new key(hash_key) and pass this table obejct to storage > > engine.They actually > > refer this field as a hash field > > On the time of insert they call bool check_unique_constraint(TABLE > > *table) function > > which first calculate the hash and store it in field then they see for > duplicate > > hash and retrive ha_index_next_same if records are not same then record > > > > We can do the same thing in mariadb by adding one more field and key in > > mysql_prepare_create_table in this we check for blob with unlimited > > length or varchar for length > > greater then internal storage engine by doing this in > > mysql_prepare_create_table there > > will be no issues of frm file inconsistance. > > > > In case of insert first we will fill the hash field in fill_record > > function of sql_base.cc > > by first calculating the hash. Then we will retrive the index map > > using ha_index_read_map > > if returened value is zero then we will comapare two records and if > > they match then we will through error > > I am not sure where to place this code either in fill_record or later > > Or i can simple just > > fill hash in field in fill_record and then check for duplicates later on. > > > > Current I am not sure how to hide columns from user.Sir, can you > > suggest me where to look > > > > But there is one problem we can make unique key by this approch but > > not primary key because primary key > > is clustered and hashes can collide so i think we can't use hash field > > as primary key. To overcome this problem > > I have one idea instead of storing just hash we can make hash field > > length 10 bytes and in last two bytes > > we can store short int which tells how much time hash is repeated this > > can make hash unique > > in case of collusion. And also we are not doing more computation > > because we already retrive > > all records with same hashes. > > What do you think of this idea?. > > And there is one more problem how to make it foreign key. > > > > Will send you a prototype code tomorrow. > > Regards > > sachin > > > > On Fri, Apr 15, 2016 at 12:23 AM, Sergei Golubchik <serg(a)mariadb.org> > wrote: > >> Hi, Sachin! > >> > >> On Apr 13, Sachin Setia wrote: > >>> Hello Sergei > >>> Sorry I did not see your mail. Actually i was thinking something like > >>> this before implementing the prototype but if i am more closer to > >>> innodb the more performance i will i get. I will definitively think > >>> about it. > >> > >> Great! > >> > >> Could you please tell me (mailing list, that is) what you think before > >> next Monday (before April 18h, that is)? > >> > >> Regards, > >> Sergei > >> Chief Architect MariaDB > >> and security(a)mariadb.org > > > ---------- Forwarded message ---------- > From: Sachin Setia <sachinsetia1001(a)gmail.com> > To: maria-developers(a)lists.launchpad.net > Cc: > Date: Mon, 18 Apr 2016 20:12:39 +0530 > Subject: Re: [Maria-developers] InnoDB blob for primary key > Hello Sergei! > Hi Actually I was thinking about how to implement > blob as a foreign key.Foreign has to be unique which > we can already implement. To make it foreign key > we can either store unique hash or the whole blob > column. > But I am not sure much people > want to copy so long blob data in reference table. > Second option would be use blob hash as a reference key. > But user can not directly us hash as a reference key > because that is hidden. > What I was thinking of a clear to programmer way of > using blob hash. Suppose user can directly create > blob hash column ,use that column as a primary key or > foreign key.Like > create table t1(abc blob , blob hash(abc))//this will just create blob > hash column > create table t1(abc blob,unique(blob hash(abc))) // this will create > unique blob hash column > and similar for primary key and foreign key > user can enter hash value if they have some good algorithm > or if they do not give any value we will automatically > create and store hash. What do you think? sir. > Regards > sachin > > > > On Sat, Apr 16, 2016 at 7:37 PM, Sachin Setia <sachinsetia1001(a)gmail.com> > wrote: > >> Hi Sergei! >> As I already told you i was building prototype.It is some what completed >> apart from one thing comparing of two field values. the difficulty is how >> to get data length of field from table->record[1]. I will try to solve >> it. >> One more thing actually i got how mysql hide field. For example >> condsider three >> fields hash,data,data. mysql field pointer point at second field not at >> hash >> field and hash field ptr is stored in table->hash_field >> can we do something similar to store hash fields(if we make array of >> hashes in >> case of more than one unique).But will adding member variable cause >> problem? >> what do you think? >> Regards >> sachin >> >> On Fri, Apr 15, 2016 at 6:37 PM, Sachin Setia <sachinsetia1001(a)gmail.com> >> wrote: >> > Hi Sergei! >> > >> > Actually I was going through the mysql source code for unique long >> constraints >> > in file sql_tmp_table.cc in function create_tmp_table they make a new >> field >> > and a new key(hash_key) and pass this table obejct to storage >> > engine.They actually >> > refer this field as a hash field >> > On the time of insert they call bool check_unique_constraint(TABLE >> > *table) function >> > which first calculate the hash and store it in field then they see for >> duplicate >> > hash and retrive ha_index_next_same if records are not same then record >> > >> > We can do the same thing in mariadb by adding one more field and key in >> > mysql_prepare_create_table in this we check for blob with unlimited >> > length or varchar for length >> > greater then internal storage engine by doing this in >> > mysql_prepare_create_table there >> > will be no issues of frm file inconsistance. >> > >> > In case of insert first we will fill the hash field in fill_record >> > function of sql_base.cc >> > by first calculating the hash. Then we will retrive the index map >> > using ha_index_read_map >> > if returened value is zero then we will comapare two records and if >> > they match then we will through error >> > I am not sure where to place this code either in fill_record or later >> > Or i can simple just >> > fill hash in field in fill_record and then check for duplicates later >> on. >> > >> > Current I am not sure how to hide columns from user.Sir, can you >> > suggest me where to look >> > >> > But there is one problem we can make unique key by this approch but >> > not primary key because primary key >> > is clustered and hashes can collide so i think we can't use hash field >> > as primary key. To overcome this problem >> > I have one idea instead of storing just hash we can make hash field >> > length 10 bytes and in last two bytes >> > we can store short int which tells how much time hash is repeated this >> > can make hash unique >> > in case of collusion. And also we are not doing more computation >> > because we already retrive >> > all records with same hashes. >> > What do you think of this idea?. >> > And there is one more problem how to make it foreign key. >> > >> > Will send you a prototype code tomorrow. >> > Regards >> > sachin >> > >> > On Fri, Apr 15, 2016 at 12:23 AM, Sergei Golubchik <serg(a)mariadb.org> >> wrote: >> >> Hi, Sachin! >> >> >> >> On Apr 13, Sachin Setia wrote: >> >>> Hello Sergei >> >>> Sorry I did not see your mail. Actually i was thinking something like >> >>> this before implementing the prototype but if i am more closer to >> >>> innodb the more performance i will i get. I will definitively think >> >>> about it. >> >> >> >> Great! >> >> >> >> Could you please tell me (mailing list, that is) what you think before >> >> next Monday (before April 18h, that is)? >> >> >> >> Regards, >> >> Sergei >> >> Chief Architect MariaDB >> >> and security(a)mariadb.org >> > > > > ---------- Forwarded message ---------- > From: Sergei Golubchik <serg(a)mariadb.org> > To: Sachin Setia <sachinsetia1001(a)gmail.com> > Cc: maria-developers(a)lists.launchpad.net > Date: Mon, 18 Apr 2016 22:32:25 +0200 > Subject: Re: [Maria-developers] InnoDB blob for primary key > Hi, Sachin! > > On Apr 15, Sachin Setia wrote: > > Hi Sergei! > > > > Actually I was going through the mysql source code for unique long > > constraints in file sql_tmp_table.cc in function create_tmp_table they > > make a new field and a new key(hash_key) and pass this table obejct to > > storage engine.They actually refer this field as a hash field On the > > time of insert they call bool check_unique_constraint(TABLE *table) > > function which first calculate the hash and store it in field then > > they see for duplicate hash and retrive ha_index_next_same if records > > are not same then record > > Right. Very good! > > > We can do the same thing in mariadb by adding one more field and key in > > mysql_prepare_create_table in this we check for blob with unlimited > > length or varchar for length greater then internal storage engine by > > doing this in mysql_prepare_create_table there will be no issues of frm > > file inconsistance. > > > > In case of insert first we will fill the hash field in fill_record > > function of sql_base.cc by first calculating the hash. Then we will > > retrive the index map using ha_index_read_map if returened value is zero > > then we will comapare two records and if they match then we will through > > error I am not sure where to place this code either in fill_record or > > later Or i can simple just fill hash in field in fill_record and then > > check for duplicates later on. > > MariaDB supports "virtual columns", see > https://mariadb.com/kb/en/mariadb/virtual-computed-columns/ > > So, it might be enough to mark this hash column as virtual, and will be > automatically calculated when necessary (in fill_record, etc). > > > Current I am not sure how to hide columns from user.Sir, can you suggest > > me where to look > > This would need a new flag per field, like "hidden". And in all commands > that use a list of columns (SHOW, INFORMATION_SCHEMA tables, SELECT *, > INSERT table VALUE (...), etc) - this column should be skipped. > > But don't worry, I don't think this will be a problem for you. You can, > of course, start from not hiding this column and implement this hiddden > flag later. > > > But there is one problem we can make unique key by this approch but not > > primary key because primary key is clustered and hashes can collide so i > > think we can't use hash field as primary key. > > Of course. > > > To overcome this problem I have one idea instead of storing just hash > > we can make hash field length 10 bytes and in last two bytes we can > > store short int which tells how much time hash is repeated this can > > make hash unique in case of collusion. And also we are not doing more > > computation because we already retrive all records with same hashes. > > What do you think of this idea?. And there is one more problem how to > > make it foreign key. > > I would say, there is no need to bother. Let's just say that UNIQUE for > blobs is only a constraint, can not be used as a primary key, can not be > used as a foreign key. This is a reasonable limitation, I think :) > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org > > > ---------- Forwarded message ---------- > From: Sergei Golubchik <serg(a)mariadb.org> > To: Sachin Setia <sachinsetia1001(a)gmail.com> > Cc: maria-developers(a)lists.launchpad.net > Date: Mon, 18 Apr 2016 22:46:42 +0200 > Subject: Re: [Maria-developers] InnoDB blob for primary key > Hi, Sachin! > > On Apr 16, Sachin Setia wrote: > > Hi Sergei! > > As I already told you i was building prototype.It is some what > > completed apart from one thing comparing of two field values. the > > difficulty is how to get data length of field from table->record[1]. > > I will try to solve it. > > Do you mean, a length of the blob value? > Try field->get_length() > > > One more thing actually i got how mysql hide field. For example > > condsider three fields hash,data,data. mysql field pointer point at > > second field not at hash field and hash field ptr is stored in > > table->hash_field can we do something similar to store hash fields(if > > we make array of hashes in case of more than one unique).But will > > adding member variable cause problem? what do you think? > > Well, yes, that would work. > But I see few other features where field hiding might be useful, > and in these cases this simple trick with moving table->field pointer > will not work, unfortunately. > > So we might need something more complex. But, again, this is not a most > important part of this project, so don't start from it. > > In fact, I'm rather interested to know about the adaptive hashing that > you've mentioned. Can you tell me more about it, please? > > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > > index ac2162b..291a3e2 100644 > > --- a/sql/sql_base.cc > > +++ b/sql/sql_base.cc > > @@ -8956,6 +8964,31 @@ fill_record(THD *thd, TABLE *table, Field **ptr, > List<Item> &values, > > goto err; > > field->set_explicit_default(value); > > } > > + table->file->ha_index_init(0,0); > > + res = > table->file->ha_index_read_map(table->record[1],hash_field->ptr,HA_WHOLE_KEY,HA_READ_KEY_EXACT); > > + while(!res){ > > + //compare the record if not sure how to compare it so just > assume it works > > + diff = table->record[1]-table->record[0]; > > + src_length = blob_field->data_length(); > > + //dest_length = > blob_field->data_length(table->record[1]); // i dont know how to get the > length from record 1 > > + // so i am enable to do this > > + // then we can comapare records using > > + //field->cmp_max > > + //this is mysql code > > + /* if (!(table->distinct ? > > + table_rec_cmp(table) : > > + group_rec_cmp(table->group, table->record[0], > table->record[1]))) > > + return false; // skip it > > + res= table->file->ha_index_next_same(table->record[1], > > + table->hash_field->ptr, > > + sizeof(hash)); */ > > + //fetch the next record > > + res= table->file->ha_index_next_same(table->record[1], > > + hash_field->ptr, > > + 8); > > + > > + } > > not quite, you should check unique constraints in fill_record(). This > should happen when the row is actually inserted or updated. A good place > for this check, I'd say, is handler::ha_write_row() and > handler::ha_update_row(). > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org > > > ---------- Forwarded message ---------- > From: Sergei Golubchik <serg(a)mariadb.org> > To: Sachin Setia <sachinsetia1001(a)gmail.com> > Cc: maria-developers(a)lists.launchpad.net > Date: Mon, 18 Apr 2016 22:51:57 +0200 > Subject: Re: [Maria-developers] InnoDB blob for primary key > Hi, Sachin! > > On Apr 18, Sachin Setia wrote: > > Hi Actually I was thinking about how to implement blob as a foreign > > key.Foreign has to be unique which we can already implement. To make > > it foreign key we can either store unique hash or the whole blob > > column. But I am not sure much people want to copy so long blob data > > in reference table. > > Agree :) > > > Second option would be use blob hash as a reference key. > > But user can not directly us hash as a reference key > > because that is hidden. > > What I was thinking of a clear to programmer way of > > using blob hash. Suppose user can directly create > > blob hash column ,use that column as a primary key or > > foreign key.Like > > create table t1(abc blob , blob hash(abc))//this will just create blob > hash > > column > > create table t1(abc blob,unique(blob hash(abc))) // this will create > unique > > blob hash column > > and similar for primary key and foreign key > > user can enter hash value if they have some good algorithm > > or if they do not give any value we will automatically > > create and store hash. What do you think? sir. > > Mixed feelings. First, I wrote in an earlier email about using virtual > columns for that. In this line of thoughts, a user-specified hash > function is absolutely possible and logical. > > On the other hand, I don't see why anyone would need that - a hash is > not guaranteed to be unique, no matter what algorithm one uses. > > And, as I wrote in an earlier email, I don't think that primary/foreign > keys on blobs is a feature worth spending time on. > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org > >

2 6

[Maria-developers] GSoC 2016:Unique indexes for blobs
by Shubham Barai 23 Apr '16

23 Apr '16

Hello everyone, I am Shubham Barai, a 3rd-year undergraduate student in computer science at Maharashtra Institute of technology, Pune, India. I am interested in the project "Unique indexes for blobs".I have read the description of the project given on the link https://jira.mariadb.org/browse/MDEV-371. MyISAM and innoDB do allow users to create unique indexes on blobs with some limitations on prefix length. So end user can create an index on blobs using "create table" or "create index "statement. example:CREATE TABLE some_table (blob_column BLOB, UNIQUE (blob_column(700))); For MyISAM tables, the prefix limit is 1000 bytes and for innoDB limit is 767 bytes. After creating an index, I ran some queries like "explain select distinct blob_column from some_table" and found out that server executes this query with the help of the temporary table but for other data types like int, the server executes the query with the help of the index. I want to know what exactly is the task for this project.Any help would be greatly appreciated Thank You, Shubham.

4 18

[Maria-developers] Fwd: Do we have anywhere some code that copies from one internal tmp table into another?
by Igor Babaev 23 Apr '16

23 Apr '16

Hi, Please help me if you are aware of such code. In my case the tables are similar. I understand that it's easy to write this code. (Read rows from the source table one by one updating the destination table). Still I would prefer to use the existing code. Regards, Igor.

2 2

Re: [Maria-developers] 745b522: MDEV-8889 - Assertion `next_insert_id == 0' failed in handler::ha_external_lock
by Sergei Golubchik 22 Apr '16

22 Apr '16

Hi, Sergey! I believe this fix is ok... On Apr 21, Sergey Vojtovich wrote: > revision-id: 745b5226e68d376eed709d930f72c3fbc7f07d2a (mariadb-10.0.24-26-g745b522) > parent(s): 072ca71d26487817d025ee97955e6360c3c5c086 > committer: Sergey Vojtovich > timestamp: 2016-04-21 16:51:00 +0400 > message: > > MDEV-8889 - Assertion `next_insert_id == 0' failed in handler::ha_external_lock > > There was a race condition between delayed insert thread and connection thread > actually performing INSERT/REPLACE DELAYED. It was triggered by concurrent > INSERT/REPLACE DELAYED and statements that flush the same table either > explicitely or implicitely (like FLUSH TABLE, ALTER TABLE, ...). > > This race condition was caused by a gap in delayed thread shutdown logic, > which allowed concurrent connection running INSERT/REPLACE DELAYED to change > essential data consequently leaving table in semi-consistent state. > > Specifically query thread could decrease "tables_in_use" reference counter in > this gap, causing delayed insert thread to shutdown without releasing auto > increment and table lock. > > Fixed by extending condition so that delayed insert thread won't shutdown > until there're locked tables. s/until/if/ > Also removed volatile qualifier from tables_in_use and stacked_inserts since > they're supposed to be protected by mutexes. It looks like stacked_inserts is accessed without a mutex at the end of Delayed_insert::handle_inserts(). Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] cd216b1: MDEV-9142 :Adding Constraint with no database reference
by Sergei Golubchik 22 Apr '16

22 Apr '16

Hi, Jan! On Apr 07, Jan Lindström wrote: > revision-id: cd216b1ab577ffe00253ec135d6194051f7ecfa0 (mariadb-5.5.48-15-gcd216b1) > parent(s): 11b77e9b18a8d97063b4c4a96e40bf9c75bd0e8b > committer: Jan Lindström > timestamp: 2016-04-07 10:47:46 +0300 > message: > > MDEV-9142 :Adding Constraint with no database reference > results in ERROR 1046 (3D000) at line 13: No database selected. > > Use database from create table to foreign key database if > nothing else is given. Ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] failing tests
by Alexander Barkov 21 Apr '16

21 Apr '16

Hi Olivier, Vladislav, Please have a look into MDEV-9739 Assertion `m_status == DA_ERROR || m_status == DA_OK' failed in Diagnostics_area::message() ; connect.xml* tests fail in buildbot The crash appeared after this commit by Vladislav: commit 3a24f1cf8426409a69619982e84ca498d581f8a1 Author: Vladislav Vaintroub <vvaintroub(a)googlemail.com> Date: Mon Feb 22 12:48:03 2016 +0100 MDEV-9307 - provide info about DATA/INDEX directory in INFORMATION_SCHEMA.TA BLES (in CREATE_OPTIONS column) Vladislav thinks that the actual problem resides in the ConnectSE code, in a wrong use of my_error(). Can you please take over the problem from this point? Thanks. On 03/21/2016 12:54 PM, Alexander Barkov wrote: > Hi Olivier, > > > On 03/20/2016 02:10 AM, Olivier Bertrand wrote: >> Hello Alexander, >> >> Le 19/03/2016 14:59, Alexander Barkov a écrit : >>> Hi Olivier, >>> >>> On 03/19/2016 02:18 AM, Olivier Bertrand wrote: >>>> Hello Alexander, >>>> >>>> I went into linux ubuntu and pass the tests but all XML tests were >>>> skipped saying "need libxml2". However I could check was libxml2 was >>>> installed indeed. Don't know what to do next. >>> Please make sure that libxml2-dev is installed. >> It was not and I installed it. However it did not change anything. > > Please make sure to remove CMakeCache.txt and run cmake again from > scratch. Does this help? > > >> The problem is that libxml2 is not recognized as installed when >> compiling Connect and then the XML type is not available. >> (trying to create an XML table fails saying "invalid table type") >> >> Btw, how do the tests fail and for what reason? > > For some reasons it crashes in DEBUG build when doing this query: > > > SELECT count(*) FROM INFORMATION_SCHEMA.TABLES WHERE > TABLE_SCHEMA='test' AND TABLE_NAME='t1' AND ENGINE='CONNECT' > AND CREATE_OPTIONS LIKE '%`table_type`=XML%' AND CREATE_OPTIONS > LIKE '%xmlsup=libxml2%' > > > with this stack trace: > > > sql/sql_error.h:705(Diagnostics_area::message() const)[0x563787dda10a] > sql/sql_select.cc:2669(JOIN::exec_inner())[0x563787dd97b5] > sql/sql_select.cc:3453(mysql_select(THD*, Item***, TABLE_LIST*, unsigned > int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, > st_order*, unsigned long long, select_result*, st_select_lex_unit*, > st_select_lex*))[0x563787ddcd1b] > sql/sql_select.cc:384(handle_select(THD*, LEX*, select_result*, unsigned > long))[0x563787dd2811] > sql/sql_parse.cc:5936(execute_sqlcom_select)[0x563787da2ae4] > sql/sql_parse.cc:2962(mysql_execute_command(THD*))[0x563787d98625] > sql/sql_parse.cc:7336(mysql_parse(THD*, char*, unsigned int, > Parser_state*))[0x563787da615d] > sql/sql_parse.cc:1490(dispatch_command(enum_server_command, THD*, char*, > unsigned int))[0x563787d94846] > sql/sql_parse.cc:1109(do_command(THD*))[0x563787d93576] > sql/sql_connect.cc:1349(do_handle_one_connection(THD*))[0x563787ec8a6c] > sql/sql_connect.cc:1262(handle_one_connection)[0x563787ec87bb] > perfschema/pfs.cc:1862(pfs_spawn_thread)[0x56378827a1da] > /lib64/libpthread.so.0(+0x760a)[0x7f427b8aa60a] > /lib64/libc.so.6(clone+0x6d)[0x7f427959fa4d] > > > Can you please try to compile a DEBUG version on Linux and check > if you get the same problem? > > In the meanwhile I'll try to find the commit which caused this problem. > > Thanks. > > >>> >>> >>>> Olivier >>>> >>>> Le 18/03/2016 14:46, Alexander Barkov a écrit : >>>>> Hello Olivier, >>>>> >>>>> a few tests are failing in the main 10.1 tree on Linux: >>>>> >>>>> connect.xml >>>>> connect.xml_grant >>>>> connect.xml_html >>>>> connect.xml_mdev5261 >>>>> connect.xml_mult >>>>> >>>>> Can you please have a look? >>>>> >>>>> Thanks! >>

3 4

[Maria-developers] Dynamic column expression limitat - 252 characters
by Sergey Petrunia 21 Apr '16

21 Apr '16

https://mariadb.com/kb/en/mariadb/virtual-computed-columns/ says the expression cannot exceed 252 characters in length. I've got a complaint about this limitation. Do we have any plans to remove it? (Does Oracle have it in their variant of this feature?) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 2

[Maria-developers] MDEV-9372 select 100 between 1 and 9223372036854775808 returns false
by Alexander Barkov 20 Apr '16

20 Apr '16

Hi Sergei, Please review a patch for MDEV-9372. Thanks.

2 1

Re: [Maria-developers] [Commits] 308cee547f6573a9825b5f08f938d4452d0095d9 Fixed bug mdev-9931.
by Oleksandr Byelkin 20 Apr '16

20 Apr '16

Hi, Igor! On 20.04.2016 00:37, Igor Babaev wrote: > commit 308cee547f6573a9825b5f08f938d4452d0095d9 > Author: Igor Babaev <igor(a)askmonty.org> > Commit: Igor Babaev <igor(a)askmonty.org> > > Fixed bug mdev-9931. > > When the specification of a WITH table referred to a view > that used a based table with the same name as the WITH table > the server went into an infinite loop because it erroneously > resolved the reference to the base table as the reference to > the WITH table. > > With tables used in a view cannot be searched for beyond the > scope the view. > --- It looks OK to push. I'd name it 'is_view_border' (because it can be more units which belon to a view), but if you think 'is_view' is better, OK to leave it as is. > mysql-test/r/cte_nonrecursive.result | 13 +++++++++++++ > mysql-test/t/cte_nonrecursive.test | 14 ++++++++++++++ > sql/sql_cte.cc | 5 ++++- > sql/sql_lex.cc | 1 + > sql/sql_lex.h | 1 + > sql/sql_view.cc | 2 ++ > 6 files changed, 35 insertions(+), 1 deletions(-) > > diff --git a/mysql-test/r/cte_nonrecursive.result b/mysql-test/r/cte_nonrecursive.result > index df64115..a9c13f3 100644 > --- a/mysql-test/r/cte_nonrecursive.result > +++ b/mysql-test/r/cte_nonrecursive.result > @@ -746,3 +746,16 @@ with t(f1,f1) as (select * from t1 where b >= 'c') > select t1.b from t2,t1 where t1.a = t2.c; > ERROR 42S21: Duplicate column name 'f1' > drop table t1,t2; > +# > +# Bug mdev-9937: View used in the specification of with table > +# refers to the base table with the same name > +# > +create table t1 (a int); > +insert into t1 values (20), (30), (10); > +create view v1 as select * from t1 where a > 10; > +with t1 as (select * from v1) select * from t1; > +a > +20 > +30 > +drop view v1; > +drop table t1; > diff --git a/mysql-test/t/cte_nonrecursive.test b/mysql-test/t/cte_nonrecursive.test > index 5a6e07e..e3164f5 100644 > --- a/mysql-test/t/cte_nonrecursive.test > +++ b/mysql-test/t/cte_nonrecursive.test > @@ -434,3 +434,17 @@ with t(f1,f1) as (select * from t1 where b >= 'c') > select t1.b from t2,t1 where t1.a = t2.c; > > drop table t1,t2; > + > +--echo # > +--echo # Bug mdev-9937: View used in the specification of with table > +--echo # refers to the base table with the same name > +--echo # > + > +create table t1 (a int); > +insert into t1 values (20), (30), (10); > +create view v1 as select * from t1 where a > 10; > + > +with t1 as (select * from v1) select * from t1; > + > +drop view v1; > +drop table t1; > diff --git a/sql/sql_cte.cc b/sql/sql_cte.cc > index 1203a4c..77f0bcf 100644 > --- a/sql/sql_cte.cc > +++ b/sql/sql_cte.cc > @@ -512,7 +512,10 @@ bool With_element::prepare_unreferenced(THD *thd) > { > With_clause *with_clause=sl->get_with_clause(); > if (with_clause && (found= with_clause->find_table_def(table))) > - return found; > + return found; > + /* Do not look for the table's definition beyond the scope of the view */ > + if (sl->master_unit()->is_view) > + break; > } > return found; > } > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 6056b03..de345b4 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2078,6 +2078,7 @@ void st_select_lex_unit::init_query() > found_rows_for_union= 0; > insert_table_with_stored_vcol= 0; > derived= 0; > + is_view= false; > with_clause= 0; > with_element= 0; > columns_are_renamed= false; > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index 0b4f0da..10247bd 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -645,6 +645,7 @@ class st_select_lex_unit: public st_select_lex_node { > derived tables/views handling. > */ > TABLE_LIST *derived; > + bool is_view; > /* With clause attached to this unit (if any) */ > With_clause *with_clause; > /* With element where this unit is used as the specification (if any) */ > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 41fd5b7..b66f678 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1612,6 +1612,8 @@ bool mysql_make_view(THD *thd, TABLE_SHARE *share, TABLE_LIST *table, > sl->context.error_processor_data= (void *)table; > } > > + table->select_lex->master_unit()->is_view= true; > + > /* > check MERGE algorithm ability > - algorithm is not explicit TEMPORARY TABLE > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] MDEV-9745 Crash with CASE WHEN TRUE THEN COALESCE(CAST(NULL AS UNSIGNED)) ELSE 4 END
by Alexander Barkov 19 Apr '16

19 Apr '16

Hello Sergei, please review a patch for MDEV-9745. This is actually a backport of MDEV-9653 from 10.1 to 5.5. Thanks.

2 1

Re: [Maria-developers] 21265d5: MDEV-9943 - TokuDB fails to compile with gcc 5.2.1
by Sergei Golubchik 19 Apr '16

19 Apr '16

Hi, Sergey! On Apr 19, Sergey Vojtovich wrote: > revision-id: 21265d58c1ea65f5c21a56cf669321f39066d7b5 (mariadb-5.5.48-19-g21265d5) > parent(s): 6c0e231c0282b43d6a46a4983f5971e960d3b8ca > committer: Sergey Vojtovich > timestamp: 2016-04-19 16:16:13 +0400 > message: > > MDEV-9943 - TokuDB fails to compile with gcc 5.2.1 > > For some reason check_cxx_compiler_flag() passes result variable name down to > compiler: > https://github.com/Kitware/CMake/blob/master/Modules/CheckCXXSourceCompiles… > > But compiler doesn't permit dashes in macro name, like in > -DHAVE_CXX_-fimplicit-templates. > > Workarounded by renaming HAVE_CXX_-fimplicit-templates to > HAVE_CXX_IMPLICIT_TEMPLAES. > > diff --git a/storage/tokudb/ft-index/cmake_modules/TokuSetupCompiler.cmake b/storage/tokudb/ft-index/cmake_modules/TokuSetupCompiler.cmake > index 99629e4..4ae7b63 100644 > --- a/storage/tokudb/ft-index/cmake_modules/TokuSetupCompiler.cmake > +++ b/storage/tokudb/ft-index/cmake_modules/TokuSetupCompiler.cmake > @@ -103,8 +103,8 @@ set_cflags_if_supported( > > if (CMAKE_CXX_FLAGS MATCHES -fno-implicit-templates) > # must append this because mysql sets -fno-implicit-templates and we need to override it > - check_cxx_compiler_flag(-fimplicit-templates HAVE_CXX_-fimplicit-templates) > - if (HAVE_CXX_-fimplicit-templates) > + check_cxx_compiler_flag(-fimplicit-templates HAVE_CXX_IMPLICIT_TEMPLATES) > + if (HAVE_CXX_IMPLICIT_TEMPLATES) > set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fimplicit-templates") > endif () > endif() I'd rather fix other similar places too. If you agree, ok to push with the patch below. --- a/storage/tokudb/ft-index/cmake_modules/TokuSetupCompiler.cmake +++ b/storage/tokudb/ft-index/cmake_modules/TokuSetupCompiler.cmake @@ -61,12 +61,13 @@ endmacro(set_cflags_if_supported_named) ## adds a compiler flag if the compiler supports it macro(set_cflags_if_supported) foreach(flag ${ARGN}) - check_c_compiler_flag(${flag} HAVE_C_${flag}) - if (HAVE_C_${flag}) + STRING(REGEX REPLACE "[-,= ]" "_" res ${flag}) + check_c_compiler_flag(${flag} HAVE_C_${res}) + if (HAVE_C_${res}) set(CMAKE_C_FLAGS "${flag} ${CMAKE_C_FLAGS}") endif () - check_cxx_compiler_flag(${flag} HAVE_CXX_${flag}) - if (HAVE_CXX_${flag}) + check_cxx_compiler_flag(${flag} HAVE_CXX_${res}) + if (HAVE_CXX_${res}) set(CMAKE_CXX_FLAGS "${flag} ${CMAKE_CXX_FLAGS}") endif () endforeach(flag) @@ -75,8 +76,9 @@ endmacro(set_cflags_if_supported) ## adds a linker flag if the compiler supports it macro(set_ldflags_if_supported) foreach(flag ${ARGN}) - check_cxx_compiler_flag(${flag} HAVE_${flag}) - if (HAVE_${flag}) + STRING(REGEX REPLACE "[-,= ]" "_" res ${flag}) + check_cxx_compiler_flag(${flag} HAVE_${res}) + if (HAVE_${res}) set(CMAKE_EXE_LINKER_FLAGS "${flag} ${CMAKE_EXE_LINKER_FLAGS}") set(CMAKE_SHARED_LINKER_FLAGS "${flag} ${CMAKE_SHARED_LINKER_FLAGS}") endif () Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 6c0e231: MDEV-9945 - main.kill_processlist-6619 fails sporadically
by Sergei Golubchik 19 Apr '16

19 Apr '16

Hi, Sergey! On Apr 19, Sergey Vojtovich wrote: > revision-id: 6c0e231c0282b43d6a46a4983f5971e960d3b8ca (mariadb-5.5.48-18-g6c0e231) > parent(s): 18ff6f654b1ca595fc5c0eebddbc1d0c5ee0d09e > committer: Sergey Vojtovich > timestamp: 2016-04-19 14:05:52 +0400 > message: > > MDEV-9945 - main.kill_processlist-6619 fails sporadically > > SHOW PROCESSLIST output can be affected by not completed concurrent queries. > Removed this affected SHOW PROCESSLIST since it doesn't seem to affect original > problem. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 18ff6f6: MDEV-9944 - main.events_2 fails sporadically
by Sergei Golubchik 19 Apr '16

19 Apr '16

Hi, Sergey! On Apr 19, Sergey Vojtovich wrote: > revision-id: 18ff6f654b1ca595fc5c0eebddbc1d0c5ee0d09e (mariadb-5.5.48-17-g18ff6f6) > parent(s): 6fd54c01bb5bc480497b143d63181837148ff47f > committer: Sergey Vojtovich > timestamp: 2016-04-19 12:38:00 +0400 > message: > > MDEV-9944 - main.events_2 fails sporadically > > Fixed wait condition in test case to actually wait for get_lock() completion > (not for lock acquisition as it was before). This removes sporadic extra row in > subsequent processlist queries. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] InnoDB blob for primary key
by Sachin Setia 18 Apr '16

18 Apr '16

Hi Sergei! Actually I was going through the mysql source code for unique long constraints in file sql_tmp_table.cc in function create_tmp_table they make a new field and a new key(hash_key) and pass this table obejct to storage engine.They actually refer this field as a hash field On the time of insert they call bool check_unique_constraint(TABLE *table) function which first calculate the hash and store it in field then they see for duplicate hash and retrive ha_index_next_same if records are not same then record We can do the same thing in mariadb by adding one more field and key in mysql_prepare_create_table in this we check for blob with unlimited length or varchar for length greater then internal storage engine by doing this in mysql_prepare_create_table there will be no issues of frm file inconsistance. In case of insert first we will fill the hash field in fill_record function of sql_base.cc by first calculating the hash. Then we will retrive the index map using ha_index_read_map if returened value is zero then we will comapare two records and if they match then we will through error I am not sure where to place this code either in fill_record or later Or i can simple just fill hash in field in fill_record and then check for duplicates later on. Current I am not sure how to hide columns from user.Sir, can you suggest me where to look But there is one problem we can make unique key by this approch but not primary key because primary key is clustered and hashes can collide so i think we can't use hash field as primary key. To overcome this problem I have one idea instead of storing just hash we can make hash field length 10 bytes and in last two bytes we can store short int which tells how much time hash is repeated this can make hash unique in case of collusion. And also we are not doing more computation because we already retrive all records with same hashes. What do you think of this idea?. And there is one more problem how to make it foreign key. Will send you a prototype code tomorrow. Regards sachin On Fri, Apr 15, 2016 at 12:23 AM, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Sachin! > > On Apr 13, Sachin Setia wrote: >> Hello Sergei >> Sorry I did not see your mail. Actually i was thinking something like >> this before implementing the prototype but if i am more closer to >> innodb the more performance i will i get. I will definitively think >> about it. > > Great! > > Could you please tell me (mailing list, that is) what you think before > next Monday (before April 18h, that is)? > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org

2 5

[Maria-developers] Deadlock with STOP SLAVE and LOCK_active_mi
by Kristian Nielsen 18 Apr '16

18 Apr '16

So while looking at MDEV-9573, I found this code in mysql_execute_command() for STOP SLAVE: mysql_mutex_lock(&LOCK_active_mi); if ((mi= (master_info_index-> get_master_info(&lex_mi->connection_name, Sql_condition::WARN_LEVEL_ERROR)))) if (!stop_slave(thd, mi, 1/* net report*/)) my_ok(thd); mysql_mutex_unlock(&LOCK_active_mi); So basically, this code is holding LOCK_active_mi for the entire duration of the STOP SLAVE operation. This seems completely broken. Anything in a slave thread that tries to take LOCK_active_mi will then deadlock with the STOP SLAVE operation. It is simple enough to trigger, testcase (with sleep-injecting patch) at the end of the email. This uses INFORMATION_SCHEMA.SESSSION_STATUS; I could imagine accessing a number of system variables could trigger it as well. >From a quick check, it looks like this has been like this forever, eg. 5.1 has similar code. Any suggestions for how this is supposed to work? Or is it just broken by design, but saved because normally slave threads do not need to access SHOW STATUS or system variables? - Kristian. ----------------------------------------------------------------------- diff --git a/sql/mysqld.cc b/sql/mysqld.cc index ced9225..2358d6b 100644 --- a/sql/mysqld.cc +++ b/sql/mysqld.cc @@ -7765,6 +7765,7 @@ static int show_heartbeat_period(THD *thd, SHOW_VAR *var, char *buff, var->type= SHOW_CHAR; var->value= buff; +if (thd->slave_thread) my_sleep(5000000); mysql_mutex_lock(&LOCK_active_mi); if (master_info_index) { ----------------------------------------------------------------------- --source include/have_innodb.inc --source include/master-slave.inc --connection master CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=InnoDB; --save_master_pos --connection slave --sync_with_master --source include/stop_slave.inc --connection master INSERT INTO t1 SELECT VARIABLE_VALUE FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='COM_COMMIT'; --save_master_pos --connection slave --source include/start_slave.inc # Wait a bit to hit a sleep() in code after taking LOCK_active_mi. SELECT SLEEP(2); STOP SLAVE; SELECT * FROM t1 ORDER BY a; START SLAVE; --sync_with_master SELECT * FROM t1 ORDER BY a; # Clean up. --connection master DROP TABLE t1; --source include/rpl_end.inc -----------------------------------------------------------------------

2 2

[Maria-developers] MDEV-9521 Least function returns 0000-00-00 for null date columns instead of null
by Alexander Barkov 18 Apr '16

18 Apr '16

Hi Sergei, Please review a patch for MDEV-9521. Thanks.

2 3

[Maria-developers] Enabling DBUG_* macros
by Игорь Пашев 18 Apr '16

18 Apr '16

Hi all :-) How can I enable all those nice DBUG_* macros and/or see their output? I'm configuring MariaDB with cmake -DCMAKE_BUILD_TYPE=Debug, but it seems I don't have those macros.

2 1

[Maria-developers] nariadb 10.1.13 fails with openssl on gentoo
by Benny Pedersen 17 Apr '16

17 Apr '16

if openssl works for othres i like to know a working my.cnf to make it work, i have added my ssql same way as used in dovecot / postfix, no ssl error in mysql, but openssl s_client -showcerts -connect 127.0.0.1:3306 says ssl23 fails, at best i see ssl3 tlsv1 fails, output is CONNECTED(00000003) --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 7 bytes and written 308 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None Start Time: 1460845475 Timeout : 300 (sec) Verify return code: 0 (ok) --- how to debug it more from here ?

3 5

Re: [Maria-developers] 56ef213: MDEV-9836 Connection lost when using SSL
by Sergei Golubchik 16 Apr '16

16 Apr '16

Hi, Vlad! On Mar 30, wlad(a)mariadb.com wrote: > revision-id: 56ef213b8be7c91331168df002c4f69d54a26081 (mariadb-5.5.48-15-g56ef213) > parent(s): 11b77e9b18a8d97063b4c4a96e40bf9c75bd0e8b > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2016-03-30 23:09:57 +0200 > message: > > MDEV-9836 Connection lost when using SSL > > Don't read from socket in yassl in SSL_pending(). > Just return size of the buffered unprocessed data. As far as I understand, you make it to return the size of buffered *processed* data. Which is correct, according to OpenSSL documentation: SSL_pending() returns the number of bytes which have been processed, buffered and are available inside ssl for immediate read. If you agree with my understanding, please fix the commit comment to say "processed" and add a reference to the manual (a quote as above or "as specified in the OpenSSL manual" or whatever you prefer). Then ok push. > diff --git a/extra/yassl/src/ssl.cpp b/extra/yassl/src/ssl.cpp > index 9516e8b..2346533 100644 > --- a/extra/yassl/src/ssl.cpp > +++ b/extra/yassl/src/ssl.cpp > @@ -1471,10 +1471,6 @@ int SSL_peek(SSL* ssl, void* buffer, int sz) > > int SSL_pending(SSL* ssl) > { > - // Just in case there's pending data that hasn't been processed yet... > - char c; > - SSL_peek(ssl, &c, 1); > - > return ssl->bufferedData(); > } > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Error compiling MariaDB 10.1.13
by Alexey Strokach 14 Apr '16

14 Apr '16

I am having trouble compiling MariaDB as a conda package on Ubuntu 14.04. I pushed the conda recipe that I am using as well as the the stdout and stderr output to a github repo <https://github.com/ostrokach/conda-recipes-extra/tree/master/mariadb-10>. In brief, I am compiling MariaDB using CFLAGS="-I$PREFIX/include -I$PREFIX/include/ncurses" \ CPPFLAGS="-I$PREFIX/include -I$PREFIX/include/ncurses" \ CXXFLAGS="-I$PREFIX/include -I$PREFIX/include/ncurses" \ LDFLAGS="-L$PREFIX/lib" \ cmake . \ -DBUILD_CONFIG=mysql_release \ -DCMAKE_PREFIX_PATH:PATH="$PREFIX" \ -DCMAKE_INSTALL_PREFIX:PATH="$PREFIX" \ -DMYSQL_DATADIR:PATH="$PREFIX/data" make make install where $PREFIX is the location of the conda build environment containing the following packages: requirements: build: - cmake - bison - ncurses - zlib - lzo - jemalloc ==3.6.0 - aio - readline - openssl - zeromq I am using the system gcc compiler: gcc (Ubuntu 4.8.5-2ubuntu1~14.04.1) 4.8.5 Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. I get the following error at 91% during make: + CFLAGS='-I/home/user/anaconda3/envs/_build/include -I/home/user/anaconda3/envs/_build/include/ncurses' + CPPFLAGS='-I/home/user/anaconda3/envs/_build/include -I/home/user/anaconda3/envs/_build/include/ncurses' + CXXFLAGS='-I/home/user/anaconda3/envs/_build/include -I/home/user/anaconda3/envs/_build/include/ncurses' + LDFLAGS=-L/home/user/anaconda3/envs/_build/lib + cmake . -DBUILD_CONFIG=mysql_release -DCMAKE_PREFIX_PATH:PATH=/home/user/anaconda3/envs/_build -DCMAKE_INSTALL_PREFIX:PATH=/home/user/anaconda3/envs/_build -DMYSQL_DATADIR:PATH=/home/user/anaconda3/envs/_build/data + make /usr/bin/ar: creating /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/libmysql/libmysqlclient.a /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/sql/sql_yacc.yy:1022.1-12: warning: deprecated directive, use ‘%pure-parser’ [-Wdeprecated] %pure_parser /* We have threads */ ^^^^^^^^^^^^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/storage/mroonga/vendor/groonga/lib/expr.c: In function ‘grn_expr_exec’: /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/storage/mroonga/vendor/groonga/lib/expr.c:3543:1: warning: const/copy propagation disabled: 21688 basic blocks and 63352 registers [-Wdisabled-optimization] } ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/storage/mroonga/vendor/groonga/lib/expr.c:3543:1: warning: PRE disabled: 21688 basic blocks and 63352 registers [-Wdisabled-optimization] /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/storage/mroonga/vendor/groonga/lib/expr.c:3543:1: warning: const/copy propagation disabled: 21688 basic blocks and 63352 registers [-Wdisabled-optimization] /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/storage/mroonga/vendor/groonga/lib/expr.c:3543:1: warning: const/copy propagation disabled: 19113 basic blocks and 64360 registers [-Wdisabled-optimization] CMake Warning (dev) in CMakeLists.txt: No cmake_minimum_required command is present. A line of code such as cmake_minimum_required(VERSION 3.5) should be added at the top of the file. The version specified may be lower if you wish to support older CMake versions for this project. For more information run "cmake --help-policy CMP0000". This warning is for project developers. Use -Wno-dev to suppress it. /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/storage/tokudb/PerconaFT/xz/src/build_lzma/src/liblzma/lz/lz_encoder.c: In function ‘fill_window’: /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/storage/tokudb/PerconaFT/xz/src/build_lzma/src/liblzma/lz/lz_encoder.c:89:9: warning: variable ‘in_used’ set but not used [-Wunused-but-set-variable] size_t in_used; ^ In file included from /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/plugin/auth_gssapi/client_plugin.cc:37:0: /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/include/mysql/client_plugin.h:64:9: warning: ‘_mysql_client_plugin_declaration_’ initialized and declared ‘extern’ [enabled by default] _mysql_client_plugin_declaration_ = { \ ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/plugin/auth_gssapi/client_plugin.cc:101:1: note: in expansion of macro ‘mysql_declare_client_plugin’ mysql_declare_client_plugin(AUTHENTICATION) ^ troff: fatal error: can't find macro file m troff: fatal error: can't find macro file m /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc: In function ‘int com_status(String*, char*)’: /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4738:13: error: ‘A_BOLD’ was not declared in this scope vidattr(A_BOLD); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4738:19: error: ‘vidattr’ was not declared in this scope vidattr(A_BOLD); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4740:13: error: ‘A_NORMAL’ was not declared in this scope vidattr(A_NORMAL); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4808:13: error: ‘A_BOLD’ was not declared in this scope vidattr(A_BOLD); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4808:19: error: ‘vidattr’ was not declared in this scope vidattr(A_BOLD); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4810:13: error: ‘A_NORMAL’ was not declared in this scope vidattr(A_NORMAL); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc: In function ‘int put_info(const char*, INFO_TYPE, uint, const char*)’: /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4905:47: error: ‘setupterm’ was not declared in this scope (void) setupterm((char *)0, 1, (int *) 0); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4918:15: error: ‘A_STANDOUT’ was not declared in this scope vidattr(A_STANDOUT); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4918:25: error: ‘vidattr’ was not declared in this scope vidattr(A_STANDOUT); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4937:15: error: ‘A_BOLD’ was not declared in this scope vidattr(A_BOLD); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4937:21: error: ‘vidattr’ was not declared in this scope vidattr(A_BOLD); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4939:13: error: ‘A_NORMAL’ was not declared in this scope vidattr(A_NORMAL); ^ /home/user/anaconda3/conda-bld/work/server-mariadb-10.1.13/client/mysql.cc:4939:21: error: ‘vidattr’ was not declared in this scope vidattr(A_NORMAL); ^ make[2]: *** [client/CMakeFiles/mysql.dir/mysql.cc.o] Error 1 make[1]: *** [client/CMakeFiles/mysql.dir/all] Error 2 make: *** [all] Error 2 Using Anaconda Cloud api site https://api.anaconda.org Command failed: /bin/bash -x -e /home/user/anaconda/conda-recipes-extra/mariadb-10/build.sh Any help would be much appreciated, Alexey

2 1

[Maria-developers] InnoDB blob for primary key
by Sachin Setia 13 Apr '16

13 Apr '16

Hello Developers, Hi this is sachin.Actually i was currently experimenting with with blob uniques in innodb there is three main problems 1.Unique blob 2.Blob Forigen key 3.Blob primary key 1. For blob unique we can simply store hash in unclustered index 2. Blob Forigen key i am currently working on it 3. Blob primary key :- for this i thought we create a 4 byte column which stores the hash of blob primary key.Internally this column will work as primary key and key for clustered index. I already successufully tested this here is my output MariaDB [sachin]> create table t4 (abc blob primary key); Query OK, 0 rows affected (0.10 sec) MariaDB [sachin]> insert into t4 values('sachin'); Query OK, 1 row affected (0.01 sec) MariaDB [sachin]> insert into t4 values('sachin'); ERROR 1062 (23000): Duplicate entry 'sachin' for key 'PRIMARY' MariaDB [sachin]> insert into t4 values('sachin setiya'); Query OK, 1 row affected (0.00 sec) MariaDB [sachin]> insert into t4 values('sachin setiya'); ERROR 1062 (23000): Duplicate entry 'sachin setiya' for key 'PRIMARY' MariaDB [sachin]> desc t4; +-------+------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+------+------+-----+---------+-------+ | abc | blob | NO | PRI | NULL | | +-------+------+------+-----+---------+-------+ 1 row in set (0.01 sec) MariaDB [sachin]> select * from t4; +---------------+ | abc | +---------------+ | sachin | | sachin setiya | +---------------+ 2 rows in set (0.01 sec) @Sergei hi! Actually i invested arround 2 months in mariadb So for me now it does not matter either i got selected in gsoc i want to do work in innodb blob unique from today.Please sir allow me to do this I am including the patch file and t4.ibd and t4.frm file Regards sachin

2 3

Re: [Maria-developers] 4c51152: MDEV-8931: (server part of) session state tracking
by Sergei Golubchik 08 Apr '16

08 Apr '16

Hi, Sanja! Summary: Lots of comments below, many changes needed. When you copy a big feature from MySQL next time - please, take a more critical look at what you're merging. On Mar 28, OleksandrByelkin wrote: > revision-id: 4c51152d9f43e271e17a2bc266f5887ce092c00f (mariadb-10.1.8-187-g4c51152) > parent(s): 69b5c4a4220c8fa98bbca8b3f935b2a3735e19ac > committer: Oleksandr Byelkin > timestamp: 2016-03-28 19:19:54 +0200 > message: > > MDEV-8931: (server part of) session state tracking > > diff --git a/include/mysql_com.h b/include/mysql_com.h > index c13999a..c6608b7 100644 > --- a/include/mysql_com.h > +++ b/include/mysql_com.h > @@ -520,6 +544,30 @@ enum enum_mysql_set_option > MYSQL_OPTION_MULTI_STATEMENTS_OFF > }; > > +/* > + Type of state change information that the server can include in the Ok > + packet. > + Note : 1) session_state_type shouldn't go past 255 (i.e. 1-byte boundary). > + 2) Modify the definition of SESSION_TRACK_END when a new member is > + added. > +*/ > +enum enum_session_state_type > +{ > + SESSION_TRACK_SYSTEM_VARIABLES, /* Session system variables */ support comes in the next patch, I know > + SESSION_TRACK_SCHEMA, /* Current schema */ suported, I presume > + SESSION_TRACK_STATE_CHANGE, /* track session state changes */ suported, I presume > + SESSION_TRACK_GTIDS, not suported, I presume > + SESSION_TRACK_TRANSACTION_CHARACTERISTICS, /* Transaction chistics */ > + SESSION_TRACK_TRANSACTION_STATE /* Transaction state */ to be imlemented later, I suppose > +}; > + > +#define SESSION_TRACK_BEGIN SESSION_TRACK_SYSTEM_VARIABLES > + > +#define SESSION_TRACK_END SESSION_TRACK_TRANSACTION_STATE > + > +#define IS_SESSION_STATE_TYPE(T) \ > + (((int)(T) >= SESSION_TRACK_BEGIN) && ((T) <= SESSION_TRACK_END)) > + > #define net_new_transaction(net) ((net)->pkt_nr=0) > > #ifdef __cplusplus > diff --git a/mysql-test/r/mysqld--help.result b/mysql-test/r/mysqld--help.result > index a35693e..f69b615 100644 > --- a/mysql-test/r/mysqld--help.result > +++ b/mysql-test/r/mysqld--help.result > @@ -903,6 +903,11 @@ The following options may be given as the first argument: > files within specified directory > --server-id=# Uniquely identifies the server instance in the community > of replication partners > + --session-track-schema > + Track changes to the 'default schema'. > + (Defaults to on; use --skip-session-track-schema to disable.) > + --session-track-state-change > + Track changes to the 'session state'. > --show-slave-auth-info > Show user and password in SHOW SLAVE HOSTS on this > master. > diff --git a/mysql-test/suite/sys_vars/r/session_track_schema_basic.result b/mysql-test/suite/sys_vars/r/session_track_schema_basic.result > new file mode 100644 > index 0000000..be4b892 > --- /dev/null > +++ b/mysql-test/suite/sys_vars/r/session_track_schema_basic.result > @@ -0,0 +1,116 @@ we don't need these "basic" sysvar tests anymore. The test that checks for them is disabled. sysvar_* tests are now used to show basic sysvar characteristics > +# > +# Variable name : session_track_schema > +# Scope : Global & Session > +# > +# Global - default > +SELECT @@global.session_track_schema; > +@@global.session_track_schema > +1 > +# Session - default > +SELECT @@session.session_track_schema; > +@@session.session_track_schema > +1 > + > +# via INFORMATION_SCHEMA.GLOBAL_VARIABLES > +SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE VARIABLE_NAME LIKE 'track_current%'; > +VARIABLE_NAME VARIABLE_VALUE > +# via INFORMATION_SCHEMA.SESSION_VARIABLES > +SELECT * FROM INFORMATION_SCHEMA.SESSION_VARIABLES WHERE VARIABLE_NAME LIKE 'track_current%'; > +VARIABLE_NAME VARIABLE_VALUE > +SET @global_saved_tmp = @@global.session_track_schema; > + > +# Altering global variable's value > +SET @@global.session_track_schema = 0; > +SELECT @@global.session_track_schema; > +@@global.session_track_schema > +0 > +SELECT @@session.session_track_schema; > +@@session.session_track_schema > +1 > +SET @@global.session_track_schema = TrUe; > +SELECT @@global.session_track_schema; > +@@global.session_track_schema > +1 > +SELECT @@session.session_track_schema; > +@@session.session_track_schema > +1 > +SET @@global.session_track_schema = FaLsE; > +SELECT @@global.session_track_schema; > +@@global.session_track_schema > +0 > +SELECT @@session.session_track_schema; > +@@session.session_track_schema > +1 > + > +# Altering session variable's value > +SET @@session.session_track_schema = 0; > +SELECT @@global.session_track_schema; > +@@global.session_track_schema > +0 > +SELECT @@session.session_track_schema; > +@@session.session_track_schema > +0 > + > +# Variables' values in a new session. > +# Global - expect 0 > +SELECT @@global.session_track_schema; > +@@global.session_track_schema > +0 > + > +# Session - expect 0 > +SELECT @@session.session_track_schema; > +@@session.session_track_schema > +0 > + > +# Switching to the default connection. > +SELECT @@global.session_track_schema; > +@@global.session_track_schema > +0 > +SELECT @@session.session_track_schema; > +@@session.session_track_schema > +0 > + > +# Test if DEFAULT is working as expected. > +SET @@global.session_track_schema = DEFAULT; > +SET @@session.session_track_schema = DEFAULT; > + > +# Global - expect 1 > +SELECT @@global.session_track_schema; > +@@global.session_track_schema > +1 > +# Session - expect 1 > +SELECT @@session.session_track_schema; > +@@session.session_track_schema > +1 > + > +# Variables' values in a new session (con2). > +SELECT @@global.session_track_schema; > +@@global.session_track_schema > +1 > +SELECT @@session.session_track_schema; > +@@session.session_track_schema > +1 > + > +# Altering session should not affect global. > +SET @@session.session_track_schema = FALSE; > +SELECT @@global.session_track_schema; > +@@global.session_track_schema > +1 > +SELECT @@session.session_track_schema; > +@@session.session_track_schema > +0 > + > +# Variables' values in a new session (con3). > +# Altering global should not affect session. > +SET @@global.session_track_schema = OFF; > +SELECT @@global.session_track_schema; > +@@global.session_track_schema > +0 > +SELECT @@session.session_track_schema; > +@@session.session_track_schema > +1 > + > +# Switching to the default connection. > +# Restoring the original values. > +SET @@global.session_track_schema = @global_saved_tmp; > +# End of tests. > diff --git a/mysql-test/suite/sys_vars/t/session_track_schema_basic.test b/mysql-test/suite/sys_vars/t/session_track_schema_basic.test > new file mode 100644 > index 0000000..220ae35 > --- /dev/null > +++ b/mysql-test/suite/sys_vars/t/session_track_schema_basic.test > @@ -0,0 +1,105 @@ another meaningless basic test > +--source include/not_embedded.inc > + > +--echo # > +--echo # Variable name : session_track_schema > +--echo # Scope : Global & Session > +--echo # > + > +--echo # Global - default > +SELECT @@global.session_track_schema; > +--echo # Session - default > +SELECT @@session.session_track_schema; > +--echo > + > +--echo # via INFORMATION_SCHEMA.GLOBAL_VARIABLES > +--disable_warnings > +SELECT * FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE VARIABLE_NAME LIKE 'track_current%'; > +--enable_warnings > + > +--echo # via INFORMATION_SCHEMA.SESSION_VARIABLES > +--disable_warnings > +SELECT * FROM INFORMATION_SCHEMA.SESSION_VARIABLES WHERE VARIABLE_NAME LIKE 'track_current%'; > +--enable_warnings > + > +# Save the global value to be used to restore the original value. > +SET @global_saved_tmp = @@global.session_track_schema; > +--echo > + > +--echo # Altering global variable's value > +SET @@global.session_track_schema = 0; > +SELECT @@global.session_track_schema; > +SELECT @@session.session_track_schema; > + > +SET @@global.session_track_schema = TrUe; > +SELECT @@global.session_track_schema; > +SELECT @@session.session_track_schema; > + > +SET @@global.session_track_schema = FaLsE; > +SELECT @@global.session_track_schema; > +SELECT @@session.session_track_schema; > +--echo > + > +--echo # Altering session variable's value > +SET @@session.session_track_schema = 0; > +SELECT @@global.session_track_schema; > +SELECT @@session.session_track_schema; > +--echo > + > +--echo # Variables' values in a new session. > +connect (con1,"127.0.0.1",root,,test,$MASTER_MYPORT,); > + > +--echo # Global - expect 0 > +SELECT @@global.session_track_schema; > +--echo > +--echo # Session - expect 0 > +SELECT @@session.session_track_schema; > +--echo > + > +--echo # Switching to the default connection. > +connection default; > + > +SELECT @@global.session_track_schema; > +SELECT @@session.session_track_schema; > +--echo > + > +--echo # Test if DEFAULT is working as expected. > +SET @@global.session_track_schema = DEFAULT; > +SET @@session.session_track_schema = DEFAULT; > +--echo > + > +--echo # Global - expect 1 > +SELECT @@global.session_track_schema; > +--echo # Session - expect 1 > +SELECT @@session.session_track_schema; > +--echo > + > +--echo # Variables' values in a new session (con2). > +connect (con2,"127.0.0.1",root,,test,$MASTER_MYPORT,); > + > +SELECT @@global.session_track_schema; > +SELECT @@session.session_track_schema; > +--echo > + > +--echo # Altering session should not affect global. > +SET @@session.session_track_schema = FALSE; > +SELECT @@global.session_track_schema; > +SELECT @@session.session_track_schema; > +--echo > + > +--echo # Variables' values in a new session (con3). > +connect (con3,"127.0.0.1",root,,test,$MASTER_MYPORT,); > + > +--echo # Altering global should not affect session. > +SET @@global.session_track_schema = OFF; > +SELECT @@global.session_track_schema; > +SELECT @@session.session_track_schema; > +--echo > + > +--echo # Switching to the default connection. > +connection default; > + > +--echo # Restoring the original values. > +SET @@global.session_track_schema = @global_saved_tmp; > + > +--echo # End of tests. > + > diff --git a/sql/protocol.cc b/sql/protocol.cc > index 9e52870..c1e66d7 100644 > --- a/sql/protocol.cc > +++ b/sql/protocol.cc > @@ -197,6 +197,8 @@ bool net_send_error(THD *thd, uint sql_errno, const char *err, > @param affected_rows Number of rows changed by statement > @param id Auto_increment id for first row (if used) > @param message Message to send to the client (Used by mysql_status) > + @param eof_indentifier when true [FE] will be set in OK header > + else [00] will be used eh, I understand that's what MySQL has, and we want to be protocol-compatible, but can you put a meaningful comment here, please? Not like i++; // increment i by 1 and a better name for the parameter would be nice too. > > @return > @retval FALSE The message was successfully sent > @@ -221,9 +233,32 @@ net_send_ok(THD *thd, > DBUG_RETURN(FALSE); > } > > - buff[0]=0; // No fields > - pos=net_store_length(buff+1,affected_rows); > - pos=net_store_length(pos, id); > + start= buff; > + > + /* > + Use 0xFE packet header if eof_identifier is true > + unless we are talking to old client > + */ fix this comment, please > + if (eof_identifier && > + (thd->client_capabilities & CLIENT_DEPRECATE_EOF)) > + buff[0]= 254; > + else > + buff[0]= 0; I don't understand this, because the comment doesn't explain anything > + > + /* affected rows */ > + pos= net_store_length(buff + 1, affected_rows); > + > + /* last insert id */ > + pos= net_store_length(pos, id); > + > + if ((thd->client_capabilities & CLIENT_SESSION_TRACK) && > + thd->session_tracker.enabled_any() && > + thd->session_tracker.changed_any()) > + { > + server_status |= SERVER_SESSION_STATE_CHANGED; > + state_changed= true; I'd rather have trackers to set server_status when they're changed, instead of iterating the list of trackers here twice for every statement just because we apparently have free CPU cycles to burn > + } > + > if (thd->client_capabilities & CLIENT_PROTOCOL_41) > { > DBUG_PRINT("info", > @@ -247,9 +282,45 @@ net_send_ok(THD *thd, > } > thd->get_stmt_da()->set_overwrite_status(true); > > + if ((thd->client_capabilities & CLIENT_SESSION_TRACK)) > + { > + /* the info field */ > + if (state_changed || (message && message[0])) > + pos= net_store_data(pos, (uchar*) message, message ? strlen(message) : 0); > + /* session state change information */ > + if (unlikely(state_changed)) > + { > + store.set_charset(thd->variables.collation_database); > + > + /* > + First append the fields collected so far. In case of malloc, memory > + for message is also allocated here. > + */ > + store.append((const char *)start, (pos - start), MYSQL_ERRMSG_SIZE); no, don't use "buf or store, depending on whether session tracking is used" that's ridiculous (and an extra memcpy) always use store, remove buf. declare store as StringBuffer<MYSQL_ERRMSG_SIZE+10> store(thd->variables.collation_database); > + > + /* .. and then the state change information. */ > + thd->session_tracker.store(thd, store); > + > + start= (uchar *) store.ptr(); > + pos= start+store.length(); > + } > + } > - if (message && message[0]) > + else if (message && message[0]) > + { > + /* the info field, if there is a message to store */ > pos= net_store_data(pos, (uchar*) message, strlen(message)); > - error= my_net_write(net, buff, (size_t) (pos-buff)); > + } > + > + /* OK packet length will be restricted to 16777215 bytes */ > + if (((size_t) (pos - start)) > MAX_PACKET_LENGTH) > + { > + net->error= 1; I'm surprised, this doesn't have a comment "setting net->error to 1" would be quite in style with what I've seen so far :) > + net->last_errno= ER_NET_OK_PACKET_TOO_LARGE; > + my_error(ER_NET_OK_PACKET_TOO_LARGE, MYF(0)); > + DBUG_PRINT("info", ("OK packet too large")); > + DBUG_RETURN(1); > + } > + error= my_net_write(net, start, (size_t) (pos - start)); > if (!error) > error= net_flush(net); > > @@ -559,7 +630,20 @@ bool Protocol::send_ok(uint server_status, uint statement_warn_count, > bool Protocol::send_eof(uint server_status, uint statement_warn_count) > { > DBUG_ENTER("Protocol::send_eof"); > - const bool retval= net_send_eof(thd, server_status, statement_warn_count); > + bool retval; > + /* > + Normally end of statement reply is signaled by OK packet, but in case > + of binlog dump request an EOF packet is sent instead. Also, old clients > + expect EOF packet instead of OK > + */ > +#ifndef EMBEDDED_LIBRARY > + if ((thd->client_capabilities & CLIENT_DEPRECATE_EOF) && > + (thd->get_command() != COM_BINLOG_DUMP )) > + retval= net_send_ok(thd, server_status, statement_warn_count, 0, 0, NULL, > + true); > + else > +#endif > + retval= net_send_eof(thd, server_status, statement_warn_count); better keep the old code here, just net_send_eof(). and in net_send_eof() you do, like if (thd->client_capabilities & CLIENT_DEPRECATE_EOF) return net_send_ok(..., 0, 0, NULL, true); > DBUG_RETURN(retval); > } > > @@ -1551,8 +1641,14 @@ bool Protocol_binary::send_out_parameters(List<Item_param> *sp_params) > /* Restore THD::server_status. */ > thd->server_status&= ~SERVER_PS_OUT_PARAMS; > > - /* Send EOF-packet. */ > - net_send_eof(thd, thd->server_status, 0); > + if (thd->client_capabilities & CLIENT_DEPRECATE_EOF) > + ret= net_send_ok(thd, > + (thd->server_status | SERVER_PS_OUT_PARAMS | > + SERVER_MORE_RESULTS_EXISTS), > + 0, 0, 0, NULL, true); > + else > + /* In case of old clients send EOF packet. */ > + ret= net_send_eof(thd, thd->server_status, 0); > > /* > Reset SERVER_MORE_RESULTS_EXISTS bit, because this is the last packet note that unlike 5.7 you we have this "Reset SERVER_MORE_RESULTS_EXISTS" *after* net_send_eof() (bugfix for MDEV-4604) 1. you don't need "| SERVER_MORE_RESULTS_EXISTS" for net_send_ok() 2. you might try to use only net_send_eof() here too (with both flags set) > diff --git a/sql/session_tracker.h b/sql/session_tracker.h > new file mode 100644 > index 0000000..7477f96 > --- /dev/null > +++ b/sql/session_tracker.h > @@ -0,0 +1,255 @@ > +#ifndef SESSION_TRACKER_INCLUDED > +#define SESSION_TRACKER_INCLUDED > + > +/* Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > +#include "m_string.h" > +#include "thr_lock.h" > + > +/* forward declarations */ > +class THD; > +class set_var; > +class String; > + > + > +enum enum_session_tracker > +{ > + SESSION_SYSVARS_TRACKER, /* Session system variables */ > + CURRENT_SCHEMA_TRACKER, /* Current schema */ > + SESSION_STATE_CHANGE_TRACKER, > + SESSION_GTIDS_TRACKER, /* Tracks GTIDs */ > + TRANSACTION_INFO_TRACKER /* Transaction state */ why not to use the same enum from mysql_com.h ? > +}; > + > +#define SESSION_TRACKER_END TRANSACTION_INFO_TRACKER > + > + > +/** > + State_tracker > + ------------- > + An abstract class that defines the interface for any of the server's > + 'session state change tracker'. A tracker, however, is a sub- class of > + this class which takes care of tracking the change in value of a part- > + icular session state type and thus defines various methods listed in this > + interface. The change information is later serialized and transmitted to > + the client through protocol's OK packet. > + > + Tracker system variables :- > + A tracker is normally mapped to a system variable. So in order to enable, > + disable or modify the sub-entities of a tracker, the user needs to modify > + the respective system variable either through SET command or via command > + line option. As required in system variable handling, this interface also > + includes two functions to help in the verification of the supplied value > + (ON_CHECK) and the updation (ON_UPDATE) of the tracker system variable, > + namely - check() and update(). this is illogical. check() and update() do not belong in the base State_tracker > +*/ > + > +class State_tracker > +{ > +protected: > + /** Is tracking enabled for a particular session state type ? */ > + bool m_enabled; > + > + /** Has the session state type changed ? */ > + bool m_changed; > + > +public: > + /** Constructor */ > + State_tracker() : m_enabled(false), m_changed(false) > + {} > + > + /** Destructor */ > + virtual ~State_tracker() > + {} > + > + /** Getters */ > + bool is_enabled() const > + { return m_enabled; } > + > + bool is_changed() const > + { return m_changed; } > + > + /** Called in the constructor of THD*/ > + virtual bool enable(THD *thd)= 0; > + > + /** To be invoked when the tracker's system variable is checked (ON_CHECK). */ > + virtual bool check(THD *thd, set_var *var)= 0; > + > + /** To be invoked when the tracker's system variable is updated (ON_UPDATE).*/ > + virtual bool update(THD *thd)= 0; > + > + /** Store changed data into the given buffer. */ > + virtual bool store(THD *thd, String &buf)= 0; > + > + /** Mark the entity as changed. */ > + virtual void mark_as_changed(THD *thd, LEX_CSTRING *name)= 0; > +}; > + > + > +/** > + Session_tracker > + --------------- > + This class holds an object each for all tracker classes and provides > + methods necessary for systematic detection and generation of session > + state change information. > +*/ > + > +class Session_tracker > +{ > +private: > + State_tracker *m_trackers[SESSION_TRACKER_END + 1]; > + > + /* The following two functions are private to disable copying. */ > + /** Copy constructor */ > + Session_tracker(Session_tracker const &other) > + { > + DBUG_ASSERT(FALSE); > + } > + > + /** Copy assignment operator */ > + Session_tracker& operator= (Session_tracker const &rhs) > + { > + DBUG_ASSERT(FALSE); > + return *this; > + } > + > +public: judging by other comments in this file (above and below), I'm surprised this keyword does not have a comment, like /* declare the rest of the object public. that makes the methods below accessible from the other code, not only from this object itself */ Serisouly, why do these comments explain basic C++ features? or repeat whatever the function is obviously doing (like, get_tracker() or enabled_any()) And non-trivial functions are not commented, for example server_boot_verify() > + > + /** Constructor */ > + Session_tracker() > + {} > + > + /** Destructor */ > + ~Session_tracker() > + { > + } > + /** > + Initialize Session_tracker objects and enable them based on the > + tracker_xxx variables' value that the session inherit from global > + variables at the time of session initialization (see plugin_thdvar_init). > + */ > + void init(const CHARSET_INFO *char_set); > + void enable(THD *thd); > + bool server_boot_verify(const CHARSET_INFO *char_set, LEX_STRING var_list); > + > + /** Returns the pointer to the tracker object for the specified tracker. */ > + State_tracker *get_tracker(enum_session_tracker tracker) const; > + > + /** Checks if m_enabled flag is set for any of the tracker objects. */ > + bool enabled_any(); > + > + /** Checks if m_changed flag is set for any of the tracker objects. */ > + bool changed_any(); > + > + /** > + Stores the session state change information of all changes session state > + type entities into the specified buffer. > + */ > + void store(THD *thd, String &main_buf); > + void deinit() > + { > + for (int i= 0; i <= SESSION_TRACKER_END; i ++) > + delete m_trackers[i]; > + } > +}; > + > +/* > + Session_state_change_tracker > + ---------------------------- > + This is a boolean tracker class that will monitor any change that contributes > + to a session state change. > + Attributes that contribute to session state change include: > + - Successful change to System variables > + - User defined variables assignments > + - temporary tables created, altered or deleted > + - prepared statements added or removed > + - change in current database > +*/ > + > +class Session_state_change_tracker : public State_tracker Why is this one in the header, not in session_tracker.cc? > +{ > +private: > + > + void reset(); > + > +public: > + Session_state_change_tracker(); > + bool enable(THD *thd); > + bool check(THD *thd, set_var *var) > + { return false; } > + bool update(THD *thd); > + bool store(THD *thd, String &buf); > + void mark_as_changed(THD *thd, LEX_CSTRING *tracked_item_name); > + bool is_state_changed(THD*); > + void ensure_enabled(THD *thd) > + {} > +}; > + > + > +/** > + Transaction_state_tracker > + ---------------------- > + This is a tracker class that enables & manages the tracking of > + current transaction info for a particular connection. > +*/ > + > +/** > + Transaction state (no transaction, transaction active, work attached, etc.) > +*/ > +enum enum_tx_state { > + TX_EMPTY = 0, ///< "none of the below" > + TX_EXPLICIT = 1, ///< an explicit transaction is active > + TX_IMPLICIT = 2, ///< an implicit transaction is active > + TX_READ_TRX = 4, ///< transactional reads were done > + TX_READ_UNSAFE = 8, ///< non-transaction reads were done > + TX_WRITE_TRX = 16, ///< transactional writes were done > + TX_WRITE_UNSAFE = 32, ///< non-transactional writes were done > + TX_STMT_UNSAFE = 64, ///< "unsafe" (non-deterministic like UUID()) stmts > + TX_RESULT_SET = 128, ///< result-set was sent > + TX_WITH_SNAPSHOT= 256, ///< WITH CONSISTENT SNAPSHOT was used > + TX_LOCKED_TABLES= 512 ///< LOCK TABLES is active > +}; > + > +/** > + Transaction access mode > +*/ > +enum enum_tx_read_flags { > + TX_READ_INHERIT = 0, ///< not explicitly set, inherit session.tx_read_only > + TX_READ_ONLY = 1, ///< START TRANSACTION READ ONLY, or tx_read_only=1 > + TX_READ_WRITE = 2, ///< START TRANSACTION READ WRITE, or tx_read_only=0 > +}; > + > +/** > + Transaction isolation level > +*/ > +enum enum_tx_isol_level { > + TX_ISOL_INHERIT = 0, ///< not explicitly set, inherit session.tx_isolation > + TX_ISOL_UNCOMMITTED = 1, > + TX_ISOL_COMMITTED = 2, > + TX_ISOL_REPEATABLE = 3, > + TX_ISOL_SERIALIZABLE= 4 > +}; > + > +/** > + Transaction tracking level > +*/ > +enum enum_session_track_transaction_info { > + TX_TRACK_NONE = 0, ///< do not send tracker items on transaction info > + TX_TRACK_STATE = 1, ///< track transaction status > + TX_TRACK_CHISTICS = 2 ///< track status and characteristics > +}; Better remove all declarations related to transaction tracking and add them later in a separate patch (with the complete transaction tracking feature). > + > +#endif /* SESSION_TRACKER_INCLUDED */ > diff --git a/sql/session_tracker.cc b/sql/session_tracker.cc > new file mode 100644 > index 0000000..6abff2a > --- /dev/null > +++ b/sql/session_tracker.cc > @@ -0,0 +1,454 @@ > +/* Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved. and "Copyright(c) 2016, MariaDB" because it's not 1:1 copy > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > + > +#include "session_tracker.h" > + > +#include "hash.h" > +#include "table.h" > +#include "rpl_gtid.h" > +#include "sql_class.h" > +#include "sql_show.h" > +#include "sql_plugin.h" > +//#include "xa.h" this is for SESSION_TRACK_TRANSACTION_STATE too I suppose > + > +static void store_lenenc_string(String &to, const char *from, > + size_t length); > + > +class Dummy_tracker : public State_tracker > +{ > +bool enable(THD *thd __attribute__((unused))) why __attribute__((unused)) if it is used? and why is it used, if the tracker is dummy? btw, seeing how this Dummy_tracker is used, I'd rather rename it to Not_implemented_tracker > + { return update(thd); } > + bool check(THD *thd __attribute__((unused)), > + set_var *var __attribute__((unused))) > + { return false; } it's C++, you don't need __attribute__((unused)), write bool check(THD *, set_var *) { return false; } > + bool update(THD *thd __attribute__((unused))) > + { return false; } > + bool store(THD *thd __attribute__((unused)), > + String &buf __attribute__((unused))) > + > + { return false; } > + void mark_as_changed(THD *thd __attribute__((unused)), > + LEX_CSTRING *tracked_item_name __attribute__((unused))) > + {} > + > +}; > + > + > +/** > + Current_schema_tracker > + ---------------------- > + This is a tracker class that enables & manages the tracking of current > + schema for a particular connection. > +*/ > + > +class Current_schema_tracker : public State_tracker > +{ > +private: > + bool schema_track_inited; > + void reset(); > + > +public: > + > + /** Constructor */ > + Current_schema_tracker() > + { > + schema_track_inited= false; > + } > + > + bool enable(THD *thd) > + { return update(thd); } > + bool check(THD *thd, set_var *var) > + { return false; } > + bool update(THD *thd); > + bool store(THD *thd, String &buf); > + void mark_as_changed(THD *thd, LEX_CSTRING *tracked_item_name); > +}; > + > +/* To be used in expanding the buffer. */ > +static const unsigned int EXTRA_ALLOC= 1024; > + > +/////////////////////////////////////////////////////////////////////////////// > + > +/** > + @brief Enable/disable the tracker based on @@session_track_schema's value. > + > + @param thd [IN] The thd handle. > + > + @return > + false (always) > +*/ > + > +bool Current_schema_tracker::update(THD *thd) > +{ > + m_enabled= (thd->variables.session_track_schema)? true: false; > + return false; > +} > + > + > +/** > + @brief Store the schema name as length-encoded string in the specified > + buffer. Once the data is stored, we reset the flags related to > + state-change (see reset()). > + > + > + @param thd [IN] The thd handle. > + @paran buf [INOUT] Buffer to store the information to. > + > + @return > + false Success > + true Error > +*/ > + > +bool Current_schema_tracker::store(THD *thd, String &buf) > +{ > + ulonglong db_length, length; > + > + length= db_length= thd->db_length; > + length += net_length_size(length); > + > + uchar *to= (uchar *) buf.prep_append(net_length_size(length) + 1, > + EXTRA_ALLOC); > + > + /* Session state type (SESSION_TRACK_SCHEMA) */ > + to= net_store_length(to, (ulonglong)SESSION_TRACK_SCHEMA); > + > + /* Length of the overall entity. */ > + to= net_store_length(to, length); > + > + /* Length of the changed current schema name. */ > + net_store_length(to, db_length); > + > + /* Current schema name (length-encoded string). */ > + store_lenenc_string(buf, thd->db, thd->db_length); > + > + reset(); > + > + return false; > +} > + > + > +/** > + @brief Mark the tracker as changed. > + > + @param name [IN] Always null. > + > + @return void > +*/ > + > +void Current_schema_tracker::mark_as_changed(THD *thd, > + LEX_CSTRING *tracked_item_name > + __attribute__((unused))) you don't need __attribute__((unused)) here, it's C++ besides, as I've just found, we compile with -Wno-unused-parameter (this was merged from MySQL, perhaps we should remove that flag) > +{ > + m_changed= true; > + thd->lex->safe_to_cache_query= 0; > +} > + > + > +/** > + @brief Reset the m_changed flag for next statement. > + > + @return void > +*/ > + > +void Current_schema_tracker::reset() > +{ > + m_changed= false; > +} > + > + > +/////////////////////////////////////////////////////////////////////////////// > + > +/** Constructor */ > +Session_state_change_tracker::Session_state_change_tracker() > +{ > + m_changed= false; > +} > + > +/** > + @brief Initiate the value of m_enabled based on > + @@session_track_state_change value. > + > + @param thd [IN] The thd handle. > + @return false (always) > + > +**/ > + > +bool Session_state_change_tracker::enable(THD *thd) > +{ > + m_enabled= (thd->variables.session_track_state_change)? true: false; > + return false; > +} > + > +/** > + @Enable/disable the tracker based on @@session_track_state_change value. > + > + @param thd [IN] The thd handle. > + @return false (always) > + > +**/ > + > +bool Session_state_change_tracker::update(THD *thd) > +{ > + return enable(thd); > +} you're kidding, right? For Dummy_tracker and Current_schema_tracker enable() calls upate(), for Session_state_change_tracker update() calls enable(). and no matter what method calls what, it's always one_method(THD *thd) { return other_method(thd); } other_method(THD *thd) { m_enabled= (thd->variables.session_track_XXX)? true: false; } why two methods enable/update and not one? why "? true : false", and not m_enabled=thd->variables.session_track_XXX (hint: m_enabled is bool, ?true:false is the same as cast to bool anyway) and why to copy the value in the first place??? Just use the value from thd->variables.session_track_XXX > + > +/** > + @brief Store the 1byte boolean flag in the specified buffer. Once the > + data is stored, we reset the flags related to state-change. If > + 1byte flag valie is 1 then there is a session state change else > + there is no state change information. > + > + @param thd [IN] The thd handle. > + @paran buf [INOUT] Buffer to store the information to. > + > + @return > + false Success > + true Error > +**/ > + > +bool Session_state_change_tracker::store(THD *thd, String &buf) > +{ > + /* since its a boolean tracker length is always 1 */ > + const ulonglong length= 1; > + > + uchar *to= (uchar *) buf.prep_append(3,EXTRA_ALLOC); > + > + /* format of the payload is as follows: > + [ tracker type] [length] [1 byte flag] */ Noooo! Do not copy the payload format logic to every tracker. Keep it in one place (preferrably, in the protocol) > + /* Session state type (SESSION_TRACK_STATE_CHANGE) */ > + to= net_store_length(to, (ulonglong)SESSION_TRACK_STATE_CHANGE); > + > + /* Length of the overall entity it is always 1 byte */ > + to= net_store_length(to, length); > + > + /* boolean tracker will go here */ > + *to= (is_state_changed(thd) ? '1' : '0'); > + > + reset(); > + > + return false; > +} > + > +/** > + @brief Mark the tracker as changed and associated session > + attributes accordingly. > + > + @param name [IN] Always null. > + @return void > +*/ > + > +void Session_state_change_tracker::mark_as_changed(THD *thd, > + LEX_CSTRING *tracked_item_name) > +{ > + /* do not send the boolean flag for the tracker itself > + in the OK packet */ > + if(tracked_item_name && > + (strncmp(tracked_item_name->str, "session_track_state_change", 26) == 0)) > + m_changed= false; Grrr. 1. Why not? 2. what if it is SET @foobar=5, @@session_track_state_change=ON; > + else > + { > + m_changed= true; > + thd->lex->safe_to_cache_query= 0; why? > + } > +} > + > +/** > + @brief Reset the m_changed flag for next statement. > + > + @return void > +*/ > + > +void Session_state_change_tracker::reset() > +{ > + m_changed= false; > +} > + > +/** > + @brief find if there is a session state change > + > + @return > + true - if there is a session state change > + false - if there is no session state change > +**/ > + > +bool Session_state_change_tracker::is_state_changed(THD* thd) > +{ > + return m_changed; > +} > + > +/////////////////////////////////////////////////////////////////////////////// > + > +/** > + @brief Initialize session tracker objects. > + > + @param char_set [IN] The character set info. > + > + @return void > +*/ > + > +void Session_tracker::init(const CHARSET_INFO *char_set) > +{ > + m_trackers[SESSION_SYSVARS_TRACKER]= > + new (std::nothrow) Dummy_tracker; > + m_trackers[CURRENT_SCHEMA_TRACKER]= > + new (std::nothrow) Current_schema_tracker; > + m_trackers[SESSION_STATE_CHANGE_TRACKER]= > + new (std::nothrow) Session_state_change_tracker; > + m_trackers[SESSION_GTIDS_TRACKER]= > + new (std::nothrow) Dummy_tracker; > + m_trackers[TRANSACTION_INFO_TRACKER]= > + new (std::nothrow) Dummy_tracker; > +} > + > +/** > + @brief Enables the tracker objects. > + > + @param thd [IN] The thread handle. > + > + @return void > +*/ > +void Session_tracker::enable(THD *thd) > +{ > + for (int i= 0; i <= SESSION_TRACKER_END; i ++) > + m_trackers[i]->enable(thd); > +} > + > +/** > + @brief Method called during the server startup to verify the contents > + of @@session_track_system_variables. I see. Please remove it from this patch. It should've been added in the next (sysvar) patch, but as it's a wrong approach in general, I hope we'll do it correctly. > + > + @param char_set The character set info. > + @param var_list Value of @@session_track_system_variables. > + > + @return false Success > + true failure > +*/ > +bool Session_tracker::server_boot_verify(const CHARSET_INFO *char_set, > + LEX_STRING var_list) > +{ > + return false; > +} > + > + > + > +/** > + @brief Returns the pointer to the tracker object for the specified tracker. > + > + @param tracker [IN] Tracker type. > + > + @return Pointer to the tracker object. > +*/ > + > +State_tracker * > +Session_tracker::get_tracker(enum_session_tracker tracker) const > +{ > + return m_trackers[tracker]; > +} make that inline in session_tracker.h > + > + > +/** > + @brief Checks if m_enabled flag is set for any of the tracker objects. > + > + @return > + true - At least one of the trackers is enabled. > + false - None of the trackers is enabled. > + > +*/ > + > +bool Session_tracker::enabled_any() > +{ > + for (int i= 0; i <= SESSION_TRACKER_END; i ++) > + { > + if (m_trackers[i]->is_enabled()) > + return true; > + } > + return false; > +} remove enabled_any() and changed_any(). See my comments where these methods are used. > + > +/** > + @brief Checks if m_changed flag is set for any of the tracker objects. > + > + @return > + true At least one of the entities being tracker has > + changed. > + false None of the entities being tracked has changed. > +*/ > + > +bool Session_tracker::changed_any() > +{ > + for (int i= 0; i <= SESSION_TRACKER_END; i ++) > + { > + if (m_trackers[i]->is_changed()) > + return true; > + } > + return false; > +} > + > + > +/** > + @brief Store all change information in the specified buffer. > + > + @param thd [IN] The thd handle. > + @param buf [OUT] Reference to the string buffer to which the state > + change data needs to be written. > + > + @return void > +*/ > + > +void Session_tracker::store(THD *thd, String &buf) > +{ > + /* Temporary buffer to store all the changes. */ > + String temp; > + size_t length; > + > + /* Get total length. */ > + for (int i= 0; i <= SESSION_TRACKER_END; i ++) > + { > + if (m_trackers[i]->is_changed()) > + m_trackers[i]->store(thd, temp); > + } > + > + length= temp.length(); > + /* Store length first.. */ > + char *to= buf.prep_append(net_length_size(length), EXTRA_ALLOC); > + net_store_length((uchar *) to, length); > + > + /* .. and then the actual info. */ > + buf.append(temp); Nice. Copy all data into a temporary malloced buffer, them memcopy it again into the malloced real buffer. At least use StringBuffer here. it would be good to avoid memcpy too, but I only see two possible approaches, both not perfect: 1. assume it's, say, less than 251 byte in the most common case. put the data directly into buf, starting from buf.ptr() + 1. when done, put the length into the reserved byte. If the length is more than 251 - memmove everything to free more bytes. 2. iterate all session trackers twice. first - to get the length, second - to put the actual data ah, yes, and don't use references (String &buf) use pointers (String *buf) > +} > + > + > +/** > + @brief Stores the given string in length-encoded format into the specified > + buffer. > + > + @param to [IN] Buffer to store the given string in. > + @param from [IN] The give string to be stored. > + @param length [IN] Length of the above string. > + > + @return void. > +*/ > + > +static > +void store_lenenc_string(String &to, const char *from, size_t length) why not make it a method of String ? Like, append_lenenc() ? > +{ > + char *ptr; > + ptr= to.prep_append(net_length_size(length), EXTRA_ALLOC); > + net_store_length((uchar *) ptr, length); > + to.append(from, length); > +} > + > diff --git a/sql/set_var.cc b/sql/set_var.cc > index b5430c5..a55fc54 100644 > --- a/sql/set_var.cc > +++ b/sql/set_var.cc > @@ -199,8 +199,24 @@ bool sys_var::update(THD *thd, set_var *var) > (on_update && on_update(this, thd, OPT_GLOBAL)); > } > else > - return session_update(thd, var) || > + { > + bool ret= session_update(thd, var) || > (on_update && on_update(this, thd, OPT_SESSION)); > + > + /* > + Make sure we don't session-track variables that are not actually > + part of the session. tx_isolation and and tx_read_only for example > + exist as GLOBAL, SESSION, and one-shot ("for next transaction only"). > + */ > + if (var->type == OPT_SESSION) > + { > + if ((!ret) && > + thd->session_tracker.get_tracker(SESSION_STATE_CHANGE_TRACKER)->is_enabled()) > + thd->session_tracker.get_tracker(SESSION_STATE_CHANGE_TRACKER)->mark_as_changed(thd, &var->var->name); don't check for is_enabled here. Alternatives: 1. just do mark_as_changed unconditionally. this works if mark_as_changed is cheap and has no side-effects. neither is true now, but I don't understand why. 2. create a wrapper method, like Session_tracker::mark_as_changed(tracker) at least you won't need to copy this long line in every place where a state changes. doing 1 is preferrable, but I don't know if it's easily possible (because I don't understand why it's done that way now) > + } > + > + return ret; > + } > } > > uchar *sys_var::session_value_ptr(THD *thd, const LEX_STRING *base) > @@ -965,6 +983,8 @@ int set_var_collation_client::update(THD *thd) > thd->variables.character_set_results= character_set_results; > thd->variables.collation_connection= collation_connection; > thd->update_charset(); > + if (thd->session_tracker.get_tracker(SESSION_STATE_CHANGE_TRACKER)->is_enabled()) > + thd->session_tracker.get_tracker(SESSION_STATE_CHANGE_TRACKER)->mark_as_changed(thd, NULL); > thd->protocol_text.init(thd); > thd->protocol_binary.init(thd); > return 0; what about set_var_password::update and set_var_role::update ? Wouldn't it be better to mark_as_changed in sql_set_variables instead of doing that in individual sys_var descendands? > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 0d168e9..933f060 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7154,3 +7154,8 @@ ER_WRONG_ORDER_IN_WITH_CLAUSE > eng "The definition of the table '%s' refers to the table '%s' defined later in a non-recursive WITH clause" > ER_RECURSIVE_QUERY_IN_WITH_CLAUSE > eng "Recursive queries in WITH clause are not supported yet" > +ER_DUP_LIST_ENTRY > + eng "Duplicate entry '%-.192s'." really, a bug? remove this. having the same value twice is not a bug: MariaDB (test)> set sql_mode='ansi,ansi'; Query OK, 0 rows affected (0.00 sec) Let's be consistent > +ER_NET_OK_PACKET_TOO_LARGE 08S01 > + eng "OK packet too large" > + ukr "Пакет OK надто великий" > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > index d58b51a..6205dd6 100644 > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -2975,6 +2975,13 @@ sp_lex_keeper::reset_lex_and_exec_core(THD *thd, uint *nextp, > > reinit_stmt_before_use(thd, m_lex); > > +#ifndef EMBEDDED_LIBRARY why? > + if ((thd->client_capabilities & CLIENT_SESSION_TRACK) && > + thd->session_tracker.enabled_any() && > + thd->session_tracker.changed_any()) > + thd->lex->safe_to_cache_query= 0; why? > +#endif > + > if (open_tables) > res= instr->exec_open_and_lock_tables(thd, m_lex->query_tables); > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index e3b7056..bf70594 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1461,6 +1461,11 @@ void THD::init(void) > /* Initialize the Debug Sync Facility. See debug_sync.cc. */ > debug_sync_init_thread(this); > #endif /* defined(ENABLED_DEBUG_SYNC) */ > + > + /* Initialize session_tracker and create all tracker objects */ > + session_tracker.init(this->charset()); > + session_tracker.enable(this); 1. you don't need charset here, it's unused 2. thus you don't need to invoke init() explicitly at all, do it from the Session_tracker constructor. In fact, move init functionality into the constructor. > + > apc_target.init(&LOCK_thd_data); > DBUG_VOID_RETURN; > } > @@ -1620,6 +1625,12 @@ void THD::cleanup(void) > /* All metadata locks must have been released by now. */ > DBUG_ASSERT(!mdl_context.has_locks()); > > + /* > + Destroy trackers only after finishing manipulations with transaction > + state to avoid issues with Transaction_state_tracker. > + */ > + session_tracker.deinit(); may be you can do that from the destructor too > + > apc_target.destroy(); > cleanup_done=1; > DBUG_VOID_RETURN; > diff --git a/sql/sql_class.h b/sql/sql_class.h > index be652e6..063198e 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -681,6 +682,11 @@ typedef struct system_variables > > my_bool pseudo_slave_mode; > > + char *track_sysvars_ptr; no track_sysvars_ptr in this patch please > + my_bool session_track_schema; > + my_bool session_track_state_change; > + ulong session_track_transaction_info; and no session_track_transaction_info either > + > } SV; > > /** > diff --git a/sql/sql_db.cc b/sql/sql_db.cc > index 2ba67cb..53719e5 100644 > --- a/sql/sql_db.cc > +++ b/sql/sql_db.cc > @@ -1034,7 +1034,18 @@ mysql_rm_db_internal(THD *thd,char *db, bool if_exists, bool silent) > it to 0. > */ > if (thd->db && cmp_db_names(thd->db, db) && !error) > - mysql_change_db_impl(thd, NULL, 0, thd->variables.collation_server); > + { > + mysql_change_db_impl(thd, NULL, 0, thd->variables.collation_server);\ eh? trailing backslash? > + /* > + Check if current database tracker is enabled. If so, set the 'changed' flag. > + */ > + if (thd->session_tracker.get_tracker(CURRENT_SCHEMA_TRACKER)->is_enabled()) > + { > + LEX_CSTRING dummy= { C_STRING_WITH_LEN("") }; > + dummy.length= dummy.length*1; WAT??? > + thd->session_tracker.get_tracker(CURRENT_SCHEMA_TRACKER)->mark_as_changed(thd, &dummy); alternatively, make mark_as_changed to take const char* and size_t, then you won't need to create LEX_CSTRING alternatively, create LEX_CSTRING dummy statically and use it always alternatively, use empty_lex_string btw, what's the point of using an *empty string* as an argument? other trackers accept NULL. why empty string is better? and anyway, why to construct and pass down an argument that is ignored anyway??? > + } > + } > my_dirend(dirp); > DBUG_RETURN(error); > } > @@ -1588,6 +1597,18 @@ bool mysql_change_db(THD *thd, const LEX_STRING *new_db_name, bool force_switch) > > mysql_change_db_impl(thd, &new_db_file_name, db_access, db_default_cl); > > +done: > + /* > + Check if current database tracker is enabled. If so, set the 'changed' flag. > + */ > + if (thd->session_tracker.get_tracker(CURRENT_SCHEMA_TRACKER)->is_enabled()) > + { > + LEX_CSTRING dummy= { C_STRING_WITH_LEN("") }; > + dummy.length= dummy.length*1; > + thd->session_tracker.get_tracker(CURRENT_SCHEMA_TRACKER)->mark_as_changed(thd, &dummy); same > + } > + if (thd->session_tracker.get_tracker(SESSION_STATE_CHANGE_TRACKER)->is_enabled()) > + thd->session_tracker.get_tracker(SESSION_STATE_CHANGE_TRACKER)->mark_as_changed(thd, NULL); > DBUG_RETURN(FALSE); > } > > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index dbe1967..244ff5b 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -322,6 +322,8 @@ static void unlock_variables(THD *thd, struct system_variables *vars); > static void cleanup_variables(struct system_variables *vars); > static void plugin_vars_free_values(sys_var *vars); > static void restore_ptr_backup(uint n, st_ptr_backup *backup); > +#define my_intern_plugin_lock(A,B) intern_plugin_lock(A,B) > +#define my_intern_plugin_lock_ci(A,B) intern_plugin_lock(A,B) remove that > static plugin_ref intern_plugin_lock(LEX *lex, plugin_ref plugin); > static void intern_plugin_unlock(LEX *lex, plugin_ref plugin); > static void reap_plugins(void); > @@ -2776,22 +2778,23 @@ static void update_func_double(THD *thd, struct st_mysql_sys_var *var, > System Variables support > ****************************************************************************/ > > - > -sys_var *find_sys_var(THD *thd, const char *str, uint length) > +sys_var *find_sys_var_ex(THD *thd, const char *str, size_t length, > + bool throw_error, bool locked) revert all sql_plugin.cc and sql_plugin.h changes. if you need them for sysvars - do them in the second patch > { > sys_var *var; > sys_var_pluginvar *pi= NULL; > plugin_ref plugin; > - DBUG_ENTER("find_sys_var"); > + DBUG_ENTER("find_sys_var_ex"); > > - mysql_mutex_lock(&LOCK_plugin); > + if (!locked) > + mysql_mutex_lock(&LOCK_plugin); > mysql_rwlock_rdlock(&LOCK_system_variables_hash); > if ((var= intern_find_sys_var(str, length)) && > (pi= var->cast_pluginvar())) > { > mysql_rwlock_unlock(&LOCK_system_variables_hash); > LEX *lex= thd ? thd->lex : 0; > - if (!(plugin= intern_plugin_lock(lex, plugin_int_to_ref(pi->plugin)))) > + if (!(plugin= my_intern_plugin_lock(lex, plugin_int_to_ref(pi->plugin)))) > var= NULL; /* failed to lock it, it must be uninstalling */ > else > if (!(plugin_state(plugin) & PLUGIN_IS_READY)) > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index 8e5ab71..03f3a93 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -2755,7 +2755,13 @@ void mysql_sql_stmt_prepare(THD *thd) > thd->stmt_map.erase(stmt); > } > else > + { > + /* send the boolean tracker in the OK packet when rewrite the comment into something that makes sense, please > + @@session_track_state_change is set to ON */ > + if (thd->session_tracker.get_tracker(SESSION_STATE_CHANGE_TRACKER)->is_enabled()) > + thd->session_tracker.get_tracker(SESSION_STATE_CHANGE_TRACKER)->mark_as_changed(thd, NULL); > my_ok(thd, 0L, 0L, "Statement prepared"); > + } > > DBUG_VOID_RETURN; > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] [Commits] ca24c9d: MDEV-9383: Server fails to read master.info after upgrade 10.0 -> 10.1
by Kristian Nielsen 07 Apr '16

07 Apr '16

Hi Nirbhay, Do you want to review this patch for MDEV-9383? It concerns your code added for do_domain_ids=(...). - Kristian. Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > revision-id: ca24c9d1671e90e43daa483af32fc19891d1a646 (mariadb-10.1.13-8-gca24c9d) > parent(s): 4b6a3518e4dc9088d1f42cd9bc487d06137d2760 > committer: Kristian Nielsen > timestamp: 2016-04-07 14:44:29 +0200 > message: > > MDEV-9383: Server fails to read master.info after upgrade 10.0 -> 10.1 > > In some cases, MariaDB 10.0 could write a master.info file that was read > incorrectly by 10.1 and could cause server to fail to start after an upgrade. > > (If writing a new master.info file that is shorter than the old, extra > junk may remain at the end of the file. This is handled properly in > 10.1 with an END_MARKER line, but this line is not written by > 10.0. The fix here is to make 10.1 robust at reading the master.info > files written by 10.0). > > Fix several things around reading master.info and read_mi_key_from_file(): > > - read_mi_key_from_file() did not distinguish between a line with and > without an eqals '=' sign. > > - If a line was empty, read_mi_key_from_file() would incorrectly return > the key from the previous call. > > - An extra using_gtid=X line left-over by MariaDB 10.0 might incorrectly > be read and overwrite the correct value. > > - Fix incorrect usage of strncmp() which should be strcmp(). > > - Add test cases. > > --- > mysql-test/std_data/bad2_master.info | 35 +++++ > mysql-test/std_data/bad3_master.info | 37 +++++ > mysql-test/std_data/bad4_master.info | 35 +++++ > mysql-test/std_data/bad5_master.info | 35 +++++ > mysql-test/std_data/bad6_master.info | 36 +++++ > mysql-test/std_data/bad_master.info | 35 +++++ > .../suite/rpl/r/rpl_upgrade_master_info.result | 88 +++++++++++ > .../suite/rpl/t/rpl_upgrade_master_info.test | 163 +++++++++++++++++++++ > sql/rpl_mi.cc | 81 +++++----- > 9 files changed, 512 insertions(+), 33 deletions(-) > > diff --git a/mysql-test/std_data/bad2_master.info b/mysql-test/std_data/bad2_master.info > new file mode 100644 > index 0000000..6172256 > --- /dev/null > +++ b/mysql-test/std_data/bad2_master.info > @@ -0,0 +1,35 @@ > +33 > +mysql-bin.000001 > +4 > +127.0.0.1 > +root > + > +3310 > +60 > +0 > + > + > + > + > + > +0 > +1800.000 > + > +0 > + > +0 > + > + > + > + > + > + > + > + > + > + > + > + > + > +using_gtid=1 > +=0 > diff --git a/mysql-test/std_data/bad3_master.info b/mysql-test/std_data/bad3_master.info > new file mode 100644 > index 0000000..6e632cd > --- /dev/null > +++ b/mysql-test/std_data/bad3_master.info > @@ -0,0 +1,37 @@ > +33 > +mysql-bin.000001 > +4 > +127.0.0.1 > +root > + > +3310 > +60 > +0 > + > + > + > + > + > +0 > +1800.000 > + > +0 > + > +0 > + > + > + > + > + > + > + > + > + > + > + > + > + > +using_gtid=1 > + > + > +0 > diff --git a/mysql-test/std_data/bad4_master.info b/mysql-test/std_data/bad4_master.info > new file mode 100644 > index 0000000..87572ef > --- /dev/null > +++ b/mysql-test/std_data/bad4_master.info > @@ -0,0 +1,35 @@ > +33 > +mysql-bin.000001 > +4 > +127.0.0.1 > +root > + > +3310 > +60 > +0 > + > + > + > + > + > +0 > +1800.000 > + > +0 > + > +0 > + > + > + > + > + > + > + > + > + > + > + > + > + > +using_gtid=1 > +d=1 > diff --git a/mysql-test/std_data/bad5_master.info b/mysql-test/std_data/bad5_master.info > new file mode 100644 > index 0000000..4ea8113 > --- /dev/null > +++ b/mysql-test/std_data/bad5_master.info > @@ -0,0 +1,35 @@ > +33 > +mysql-bin.000001 > +4 > +127.0.0.1 > +root > + > +3310 > +60 > +0 > + > + > + > + > + > +0 > +1800.000 > + > +0 > + > +0 > + > + > + > + > + > + > + > + > + > + > + > + > + > +using_gtid=1 > +using_gtid > diff --git a/mysql-test/std_data/bad6_master.info b/mysql-test/std_data/bad6_master.info > new file mode 100644 > index 0000000..0f48f48 > --- /dev/null > +++ b/mysql-test/std_data/bad6_master.info > @@ -0,0 +1,36 @@ > +33 > +mysql-bin.000001 > +4 > +127.0.0.1 > +root > + > +3310 > +60 > +0 > + > + > + > + > + > +0 > +1800.000 > + > +0 > + > +0 > + > + > + > + > + > + > + > + > + > + > + > + > + > +using_gtid=1 > +END_MARKER > +do_domain_ids=20 Hulubulu!!?! > diff --git a/mysql-test/std_data/bad_master.info b/mysql-test/std_data/bad_master.info > new file mode 100644 > index 0000000..1541fdf > --- /dev/null > +++ b/mysql-test/std_data/bad_master.info > @@ -0,0 +1,35 @@ > +33 > +mysql-bin.000001 > +4 > +127.0.0.1 > +root > + > +3310 > +60 > +0 > + > + > + > + > + > +0 > +1800.000 > + > +0 > + > +0 > + > + > + > + > + > + > + > + > + > + > + > + > + > +using_gtid=1 > + > diff --git a/mysql-test/suite/rpl/r/rpl_upgrade_master_info.result b/mysql-test/suite/rpl/r/rpl_upgrade_master_info.result > new file mode 100644 > index 0000000..f966f18 > --- /dev/null > +++ b/mysql-test/suite/rpl/r/rpl_upgrade_master_info.result > @@ -0,0 +1,88 @@ > +include/master-slave.inc > +[connection master] > +*** MDEV-9383: Server fails to read master.info after upgrade 10.0 -> 10.1 *** > +include/stop_slave.inc > +CHANGE MASTER TO master_use_gtid=CURRENT_POS; > +include/rpl_stop_server.inc [server_number=2] > +include/rpl_start_server.inc [server_number=2] > +CREATE TABLE t1 (a INT PRIMARY KEY); > +INSERT INTO t1 VALUES (1); > +include/save_master_gtid.inc > +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_1; > +include/start_slave.inc > +include/sync_with_master_gtid.inc > +SELECT * FROM t1; > +a > +1 > +include/stop_slave.inc > +include/rpl_stop_server.inc [server_number=2] > +include/rpl_start_server.inc [server_number=2] > +INSERT INTO t1 VALUES (2); > +include/save_master_gtid.inc > +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_1; > +include/start_slave.inc > +include/sync_with_master_gtid.inc > +SELECT * FROM t1 ORDER BY a; > +a > +1 > +2 > +include/stop_slave.inc > +include/rpl_stop_server.inc [server_number=2] > +include/rpl_start_server.inc [server_number=2] > +INSERT INTO t1 VALUES (3); > +include/save_master_gtid.inc > +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_1; > +include/start_slave.inc > +include/sync_with_master_gtid.inc > +SELECT * FROM t1 ORDER BY a; > +a > +1 > +2 > +3 > +include/stop_slave.inc > +include/rpl_stop_server.inc [server_number=2] > +include/rpl_start_server.inc [server_number=2] > +INSERT INTO t1 VALUES (4); > +include/save_master_gtid.inc > +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_1; > +include/start_slave.inc > +include/sync_with_master_gtid.inc > +SELECT * FROM t1 ORDER BY a; > +a > +1 > +2 > +3 > +4 > +include/stop_slave.inc > +include/rpl_stop_server.inc [server_number=2] > +include/rpl_start_server.inc [server_number=2] > +INSERT INTO t1 VALUES (5); > +include/save_master_gtid.inc > +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_1; > +include/start_slave.inc > +include/sync_with_master_gtid.inc > +SELECT * FROM t1 ORDER BY a; > +a > +1 > +2 > +3 > +4 > +5 > +include/stop_slave.inc > +include/rpl_stop_server.inc [server_number=2] > +include/rpl_start_server.inc [server_number=2] > +INSERT INTO t1 VALUES (6); > +include/save_master_gtid.inc > +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_1; > +include/start_slave.inc > +include/sync_with_master_gtid.inc > +SELECT * FROM t1 ORDER BY a; > +a > +1 > +2 > +3 > +4 > +5 > +6 > +DROP TABLE t1; > +include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_upgrade_master_info.test b/mysql-test/suite/rpl/t/rpl_upgrade_master_info.test > new file mode 100644 > index 0000000..e81e7c0 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_upgrade_master_info.test > @@ -0,0 +1,163 @@ > +--source include/master-slave.inc > + > +--echo *** MDEV-9383: Server fails to read master.info after upgrade 10.0 -> 10.1 *** > + > +--connection slave > +--source include/stop_slave.inc > +CHANGE MASTER TO master_use_gtid=CURRENT_POS; > +--let $datadir= `SELECT @@datadir` > + > +--let $rpl_server_number= 2 > +--source include/rpl_stop_server.inc > + > +--remove_file $datadir/master.info > +--copy_file $MYSQL_TEST_DIR/std_data/bad_master.info $datadir/master.info > + > +--let $rpl_server_number= 2 > +--source include/rpl_start_server.inc > + > +--source include/wait_until_connected_again.inc > + > +--connection master > +CREATE TABLE t1 (a INT PRIMARY KEY); > +INSERT INTO t1 VALUES (1); > +--source include/save_master_gtid.inc > + > +--connection slave > +# Fix the port after we replaced master.info. > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1; > +--source include/start_slave.inc > +--source include/sync_with_master_gtid.inc > +SELECT * FROM t1; > + > +--source include/stop_slave.inc > + > +--let $rpl_server_number= 2 > +--source include/rpl_stop_server.inc > + > +--remove_file $datadir/master.info > +--copy_file $MYSQL_TEST_DIR/std_data/bad2_master.info $datadir/master.info > + > +--let $rpl_server_number= 2 > +--source include/rpl_start_server.inc > + > +--source include/wait_until_connected_again.inc > + > +--connection master > +INSERT INTO t1 VALUES (2); > +--source include/save_master_gtid.inc > + > +--connection slave > +# Fix the port after we replaced master.info. > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1; > +--source include/start_slave.inc > +--source include/sync_with_master_gtid.inc > +SELECT * FROM t1 ORDER BY a; > + > +--source include/stop_slave.inc > + > +--let $rpl_server_number= 2 > +--source include/rpl_stop_server.inc > + > +--remove_file $datadir/master.info > +--copy_file $MYSQL_TEST_DIR/std_data/bad3_master.info $datadir/master.info > + > +--let $rpl_server_number= 2 > +--source include/rpl_start_server.inc > + > +--source include/wait_until_connected_again.inc > + > +--connection master > +INSERT INTO t1 VALUES (3); > +--source include/save_master_gtid.inc > + > +--connection slave > +# Fix the port after we replaced master.info. > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1; > +--source include/start_slave.inc > +--source include/sync_with_master_gtid.inc > +SELECT * FROM t1 ORDER BY a; > + > +--source include/stop_slave.inc > + > +--let $rpl_server_number= 2 > +--source include/rpl_stop_server.inc > + > +--remove_file $datadir/master.info > +--copy_file $MYSQL_TEST_DIR/std_data/bad4_master.info $datadir/master.info > + > +--let $rpl_server_number= 2 > +--source include/rpl_start_server.inc > + > +--source include/wait_until_connected_again.inc > + > +--connection master > +INSERT INTO t1 VALUES (4); > +--source include/save_master_gtid.inc > + > +--connection slave > +# Fix the port after we replaced master.info. > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1; > +--source include/start_slave.inc > +--source include/sync_with_master_gtid.inc > +SELECT * FROM t1 ORDER BY a; > + > +--source include/stop_slave.inc > + > +--let $rpl_server_number= 2 > +--source include/rpl_stop_server.inc > + > +--remove_file $datadir/master.info > +--copy_file $MYSQL_TEST_DIR/std_data/bad5_master.info $datadir/master.info > + > +--let $rpl_server_number= 2 > +--source include/rpl_start_server.inc > + > +--source include/wait_until_connected_again.inc > + > +--connection master > +INSERT INTO t1 VALUES (5); > +--source include/save_master_gtid.inc > + > +--connection slave > +# Fix the port after we replaced master.info. > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1; > +--source include/start_slave.inc > +--source include/sync_with_master_gtid.inc > +SELECT * FROM t1 ORDER BY a; > + > +--source include/stop_slave.inc > + > +--let $rpl_server_number= 2 > +--source include/rpl_stop_server.inc > + > +--remove_file $datadir/master.info > +--copy_file $MYSQL_TEST_DIR/std_data/bad6_master.info $datadir/master.info > + > +--let $rpl_server_number= 2 > +--source include/rpl_start_server.inc > + > +--source include/wait_until_connected_again.inc > + > +--connection master > +INSERT INTO t1 VALUES (6); > +--source include/save_master_gtid.inc > + > +--connection slave > +# Fix the port after we replaced master.info. > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1; > +--source include/start_slave.inc > +--source include/sync_with_master_gtid.inc > +SELECT * FROM t1 ORDER BY a; > + > + > +# Cleanup > +--connection master > +DROP TABLE t1; > +--source include/rpl_end.inc > diff --git a/sql/rpl_mi.cc b/sql/rpl_mi.cc > index df72134..706824c 100644 > --- a/sql/rpl_mi.cc > +++ b/sql/rpl_mi.cc > @@ -205,43 +205,56 @@ void init_master_log_pos(Master_info* mi) > > /** > Parses the IO_CACHE for "key=" and returns the "key". > + If no '=' found, returns the whole line (for END_MARKER). > > @param key [OUT] Key buffer > @param max_size [IN] Maximum buffer size > @param f [IN] IO_CACHE file > + @param found_equal [OUT] Set true if a '=' was found. > > @retval 0 Either "key=" or '\n' found > @retval 1 EOF > */ > -static int read_mi_key_from_file(char *key, int max_size, IO_CACHE *f) > +static int read_mi_key_from_file(char *key, int max_size, IO_CACHE *f, > + bool *found_equal) > { > int i= 0, c; > - char *last_p; > > DBUG_ENTER("read_key_from_file"); > > - while (((c= my_b_get(f)) != '\n') && (c != my_b_EOF)) > + *found_equal= false; > + if (max_size <= 0) > + DBUG_RETURN(1); > + for (;;) > { > - last_p= key + i; > - > - if (i < max_size) > + if (i >= max_size-1) > { > - if (c == '=') > - { > - /* We found '=', replace it by 0 and return. */ > - *last_p= 0; > - DBUG_RETURN(0); > - } > - else > - *last_p= c; > + key[i] = '\0'; > + DBUG_RETURN(0); > + } > + c= my_b_get(f); > + if (c == my_b_EOF) > + { > + DBUG_RETURN(1); > + } > + else if (c == '\n') > + { > + key[i]= '\0'; > + DBUG_RETURN(0); > + } > + else if (c == '=') > + { > + key[i]= '\0'; > + *found_equal= true; > + DBUG_RETURN(0); > + } > + else > + { > + key[i]= c; > + ++i; > } > - ++i; > } > - > - if (c == my_b_EOF) > - DBUG_RETURN(1); > - > - DBUG_RETURN(0); > + /* NotReached */ > } > > enum { > @@ -539,6 +552,10 @@ file '%s')", fname); > if (lines >= LINE_FOR_LAST_MYSQL_FUTURE) > { > uint i; > + bool got_eq; > + bool seen_using_gtid= false; > + bool seen_do_domain_ids=false, seen_ignore_domain_ids=false; > + > /* Skip lines used by / reserved for MySQL >= 5.6. */ > for (i= LINE_FOR_FIRST_MYSQL_5_6; i <= LINE_FOR_LAST_MYSQL_FUTURE; ++i) > { > @@ -551,11 +568,12 @@ file '%s')", fname); > for "key=" and returns the "key" if found. The "value" can then the > parsed on case by case basis. The "unknown" lines would be ignored to > facilitate downgrades. > + 10.0 does not have the END_MARKER before any left-overs at the end > + of the file. So ignore any but the first occurrence of a key. > */ > - while (!read_mi_key_from_file(buf, sizeof(buf), &mi->file)) > + while (!read_mi_key_from_file(buf, sizeof(buf), &mi->file, &got_eq)) > { > - /* using_gtid */ > - if (!strncmp(buf, STRING_WITH_LEN("using_gtid"))) > + if (got_eq && !seen_using_gtid && !strcmp(buf, "using_gtid")) > { > int val; > if (!init_intvar_from_file(&val, &mi->file, 0)) > @@ -566,15 +584,13 @@ file '%s')", fname); > mi->using_gtid= Master_info::USE_GTID_SLAVE_POS; > else > mi->using_gtid= Master_info::USE_GTID_NO; > - continue; > + seen_using_gtid= true; > } else { > sql_print_error("Failed to initialize master info using_gtid"); > goto errwithmsg; > } > } > - > - /* DO_DOMAIN_IDS */ > - if (!strncmp(buf, STRING_WITH_LEN("do_domain_ids"))) > + else if (got_eq && !seen_do_domain_ids && !strcmp(buf, "do_domain_ids")) > { > if (mi->domain_id_filter.init_ids(&mi->file, > Domain_id_filter::DO_DOMAIN_IDS)) > @@ -582,11 +598,10 @@ file '%s')", fname); > sql_print_error("Failed to initialize master info do_domain_ids"); > goto errwithmsg; > } > - continue; > + seen_do_domain_ids= true; > } > - > - /* IGNORE_DOMAIN_IDS */ > - if (!strncmp(buf, STRING_WITH_LEN("ignore_domain_ids"))) > + else if (got_eq && !seen_ignore_domain_ids && > + !strcmp(buf, "ignore_domain_ids")) > { > if (mi->domain_id_filter.init_ids(&mi->file, > Domain_id_filter::IGNORE_DOMAIN_IDS)) > @@ -595,9 +610,9 @@ file '%s')", fname); > "ignore_domain_ids"); > goto errwithmsg; > } > - continue; > + seen_ignore_domain_ids= true; > } > - else if (!strncmp(buf, STRING_WITH_LEN("END_MARKER"))) > + else if (!got_eq && !strcmp(buf, "END_MARKER")) > { > /* > Guard agaist extra left-overs at the end of file, in case a later > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 2

[Maria-developers] First MariaDB Berlin Meetup
by Colin Charles 06 Apr '16

06 Apr '16

Hi! Many of the core server developers for MariaDB Server and MariaDB MaxScale will be having a meetup in Berlin, on April 12 2016 at Wikimedia’s offices. Feel free to drop by, have a drink, say hi, and listen to the upcoming plans http://www.meetup.com/MariaDB-Developers-Berlin-Meetup/events/230026151/ Looking forward to seeing you if you’re local to Berlin, and if you have friends that would be interested, please feel free to forward this note to them Thanks Kind Regards, Colin Charles -- Colin Charles, http://bytebot.net/blog/ twitter: @bytebot | skype: colincharles "First they ignore you, then they laugh at you, then they fight you, then you win." -- Mohandas Gandhi

1 0

[Maria-developers] Please review MDEV-9823 LOAD DATA INFILE silently truncates incomplete byte sequences
by Alexander Barkov 05 Apr '16

05 Apr '16

Hi Sergei, Please review a patch for MDEV-9823. This is a prerequisite for the current sprint task: MDEV-6353 my_ismbchar() and my_mbcharlen() refactoring Thanks.

2 3

[Maria-developers] Window functions: compare_window_spec_joined_lists() has a bug
by Sergey Petrunia 05 Apr '16

05 Apr '16

Hello Igor, I'm trying this example: create table t1 (a int, b int); insert into t1 values (1,1),(2,2),(3,3); select rank() over (order by a), rank() over (order by b) from t1; I put a breakpoint in order_window_funcs_by_window_specs(), at this call: int cmp= compare_window_spec_joined_lists(win_spec_prev, win_spec_curr); and I see that: (gdb) p win_spec1->partition_list $66 = (SQL_I_List<st_order> *) 0x7fff5c013038 (gdb) p win_spec2->partition_list $68 = (SQL_I_List<st_order> *) 0x7fff5c013038 // same as above (gdb) p win_spec2->partition_list[0] $70 = {<Sql_alloc> = {<No data fields>}, elements = 0, first = 0x0, next = 0x7fff5c013040} // Ok these are both empty lusts. (gdb) p win_spec2->order_list $72 = (SQL_I_List<st_order> *) 0x7fff5c013498 (gdb) p win_spec1->order_list $74 = (SQL_I_List<st_order> *) 0x7fff5c013050 (gdb) p win_spec2->order_list->first $76 = (st_order *) 0x7fff5c013428 (gdb) p win_spec1->order_list->first $78 = (st_order *) 0x7fff5c012fe0 // Ok these are different lists, as expected. Then, the code in compare_window_spec_joined_lists does this: { win_spec1->join_partition_and_order_lists(); win_spec2->join_partition_and_order_lists(); This will cause both window specs to have the same lists, and I will get item_window_func->marker=0 for the second item (instead of SORTORDER_CHANGE_FLAG that I expect). BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Please review MDEV-9842 LOAD DATA INFILE does not work well with a TEXT column when using sjis
by Alexander Barkov 01 Apr '16

01 Apr '16

Hi Sergei, Please review a patch for MDEV-9842. This is a prerequisite for: MDEV-6353 my_ismbchar() and my_mbcharlen() refactoring Thanks.

2 1

[Maria-developers] Please review MDEV-9842 LOAD DATA INFILE does not work well with a TEXT column when using sjis
by Alexander Barkov 31 Mar '16

31 Mar '16

Hi Sergei, Please review a patch for MDEV-9842. This is a prerequisite for: MDEV-6353 my_ismbchar() and my_mbcharlen() refactoring Thanks.

1 0

Re: [Maria-developers] yet another init.d script for running multiple mariald instances on the same host.
by Sergei Golubchik 31 Mar '16

31 Mar '16

Hi, Andrii! On Mar 21, Andrii Petrenko wrote: > Hello Sergey, > > I have created one more script for running mariadb (or mysql) instacnces > on the same host. For different instances i'm using different defaults > file. > > i'd like ti share it with you, or maria team -- IMHO it is more useful > than existent script. > > https://github.com/aplsms/Multi-MariaDB Yes, good idea - using scripts like /etc/init.d/mysql-1, etc. This can be a useful trick in some cases. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-9841: Window functions: embedded server fails to start due to error code conflicts
by Sergey Petrunia 31 Mar '16

31 Mar '16

Hi Serg, We added a number of new error messages for window functions, and now server error codes have collided with the client. The text in MDEV-9841 has the details. Sanja mentioned that you were aware of the possibility of this and had a solution. Can we discuss it sometime? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Please review MDEV-9811 and MDEV-9824 (dependencies for MDEV-6353)
by Alexander Barkov 31 Mar '16

31 Mar '16

Hi Sergei, Please review a patch fixing MDEV-9811 and MDEV-9824. They are prerequisites for: MDEV-6353 my_ismbchar() and my_mbcharlen() refactoring Thanks.

2 1

[Maria-developers] Unsubscribe
by Abdul Hakeem 31 Mar '16

31 Mar '16

1 0

Re: [Maria-developers] MDEV-7773, Aggregate Stored Functions
by Soham Mankad 30 Mar '16

30 Mar '16

Respected Sir, I have found some of ways to implement this project using C++, SQL, java, or C. As I had mentioned earlier, I was searching for some of database systems that have this facility or in some way can implement it. The database system that I discovered while learning for this topic is HP Vertica and in this system the basic approach is creating user defined library saved as .so for C/C++ or .jarr for java and loading it using CREATE LIBRARY command. When we call SQL function HP Vertica passes data values to the code in the library to process it. Syntax: CREATE [ OR REPLACE ] AGGREGATE FUNCTION [[db-name.]schema.]function-name ... AS LANGUAGE 'language' NAME 'factory' LIBRARY library_name; example using this method is as given below => CREATE LIBRARY AggregateFunctions AS '/opt/vertica/sdk/examples/build/AggregateFunctions.so'; CREATE LIBRARY => CREATE AGGREGATE FUNCTION ag_avg AS LANGUAGE 'C++' NAME 'AverageFactory' library AggregateFunctions; CREATE AGGREGATE FUNCTION => CREATE AGGREGATE FUNCTION ag_cat AS LANGUAGE 'C++' NAME 'ConcatenateFactory' library AggregateFunctions; CREATE AGGREGATE FUNCTION => \x Expanded display is on. select * from user_functions; The only drawback in this method that this can be implemented using C++ only for HP Vertica. The another method that I discovered is given in the latest update of oracle i.e Oracle 9i (9) the refrence link for same is http://www.oracle-developer.net/display.php?id=215 According to this we require two components to implement aggregate functions 1. an object type specification and body; and 2. a PL/SQL function. There is a development framework for data cartridges (think of this as a template) provided by Oracle. This provides the structure of the object type and its methods (down to the detail of how we name them) and also defines how we create our PL/SQL function. For example, our object type will contain the following:attribute(s) for holding state information. These can be of any existing built-in or user-defined datatype; - a mandatory ODCIAggregateInitialize static method to reset the state attributes at the start of an aggregation; - a mandatory ODCIAggregateIterate member method to apply each input value to the running aggregate value; - a mandatory ODCIAggregateTerminate member method to return the final result of the aggregate; and - an optional ODCIAggregateMerge member method, used to combine the results of more than one stream of aggregation (for example, with parallel query) before returning a result. More detailed information, explanation and examples are given in the link that I have mentioned above. Please let me know whether I am working in right direction or not and whether the methods I have mentioned can be used or not to implement the project. Regards, Soham Mankad

2 2

[Maria-developers] MDEV-8360 Clean-up CHARSET_INFO: strnncollsp: diff_if_only_endspace_difference
by Alexander Barkov 30 Mar '16

30 Mar '16

Hi Sergei, please review a patch for MDEV-8360. Btw, shouldn't we remove: #define HA_END_SPACE_ARE_EQUAL 512 in ./include/my_base.h ? It's not really needed. Thanks.

2 1

Re: [Maria-developers] [Commits] d40d68f: Convert percent_rank to work with cursors
by Sergey Petrunia 30 Mar '16

30 Mar '16

On Mon, Mar 28, 2016 at 10:58:55PM +0300, Vicentiu Ciorbaru wrote: > revision-id: d40d68f23602be9886c1b502fdad9d23bdc9a0fb (mariadb-10.1.8-187-gd40d68f) > parent(s): bf18dac08fe0ae18975158e786fee097883949d4 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2016-03-28 22:51:42 +0300 > message: > > Convert percent_rank to work with cursors > > The percent_rank function now is compatible with the cursor algorithm. > We no longer need a special implementation for it to work. > > --- > sql/item_windowfunc.h | 77 +++++++++++------------ > sql/sql_window.cc | 165 +++++++++++++++++--------------------------------- > 2 files changed, 91 insertions(+), 151 deletions(-) > > diff --git a/sql/item_windowfunc.h b/sql/item_windowfunc.h > index 6f91bc8..1dd483c 100644 > --- a/sql/item_windowfunc.h > +++ b/sql/item_windowfunc.h > @@ -317,12 +317,18 @@ class Item_context > NOTE: All two pass window functions need to implement > this interface. > */ > -class Item_sum_window_with_context : public Item_sum_num, > - public Item_context > +class Item_sum_window_with_row_count : public Item_sum_num > { > public: > - Item_sum_window_with_context(THD *thd) > - : Item_sum_num(thd), Item_context() {} > + Item_sum_window_with_row_count(THD *thd) : Item_sum_num(thd), > + partition_row_count_(0){} > + > + void set_row_count(ulonglong count) { partition_row_count_ = count; } > + > + protected: > + longlong get_row_count() { return partition_row_count_; } > + private: > + ulonglong partition_row_count_; > }; > > /* > @@ -336,12 +342,11 @@ class Item_sum_window_with_context : public Item_sum_num, > This is held within the row_count context. > - Second pass to compute rank of current row and the value of the function > */ > -class Item_sum_percent_rank: public Item_sum_window_with_context, > - public Window_context_row_count > +class Item_sum_percent_rank: public Item_sum_window_with_row_count > { > public: > Item_sum_percent_rank(THD *thd) > - : Item_sum_window_with_context(thd), cur_rank(1) {} > + : Item_sum_window_with_row_count(thd), cur_rank(1) {} > > longlong val_int() > { > @@ -359,14 +364,9 @@ class Item_sum_percent_rank: public Item_sum_window_with_context, > We can not get the real value without knowing the number of rows > in the partition. Don't divide by 0. > */ > - if (!get_context_()) > - { > - // Calling this kind of function with a context makes no sense. > - DBUG_ASSERT(0); > - return 0; > - } > - > - longlong partition_rows = get_context_()->get_field_context(result_field); > + ulonglong partition_rows = get_row_count(); > + null_value= partition_rows > 0 ? false : true; > + > return partition_rows > 1 ? > static_cast<double>(cur_rank - 1) / (partition_rows - 1) : 0; > } > @@ -381,25 +381,6 @@ class Item_sum_percent_rank: public Item_sum_window_with_context, > return "percent_rank"; > } > > - bool create_window_context() > - { > - // TODO-cvicentiu: Currently this means we must make sure to delete > - // the window context. We can potentially allocate this on the THD memroot. > - // At the same time, this is only necessary for a small portion of the > - // query execution and it does not make sense to keep it for all of it. > - context_ = new Window_context_row_count(); > - if (context_ == NULL) > - return true; > - return false; > - } > - > - void delete_window_context() > - { > - if (context_) > - delete get_context_(); > - context_ = NULL; > - } > - > void update_field() {} > > void clear() > @@ -428,13 +409,6 @@ class Item_sum_percent_rank: public Item_sum_window_with_context, > void cleanup() > { > peer_tracker.cleanup(); > - Item_sum_window_with_context::cleanup(); > - } > - > - /* Helper function so that we don't cast the context every time. */ > - Window_context_row_count* get_context_() > - { > - return static_cast<Window_context_row_count *>(context_); > } > }; > > @@ -517,6 +491,27 @@ class Item_window_func : public Item_func_or_sum > } > } > > + bool requires_partition_size() const > + { > + switch (window_func()->sum_func()) { > + case Item_sum::PERCENT_RANK_FUNC: > + case Item_sum::CUME_DIST_FUNC: > + return true; > + default: > + return false; > + } > + } > + > + bool requires_peer_size() const > + { > + switch (window_func()->sum_func()) { > + case Item_sum::CUME_DIST_FUNC: > + return true; > + default: > + return false; > + } > + } This function seem not to be used anywhere? > + > bool is_order_list_mandatory() const > { > switch (window_func()->sum_func()) { > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index e8e226b..1bfac7a 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -865,15 +865,18 @@ class Frame_unbounded_preceding : public Frame_cursor > } > }; > > + > /* > UNBOUNDED FOLLOWING frame bound > */ > > class Frame_unbounded_following : public Frame_cursor > { > - Table_read_cursor cursor; > > +protected: > + Table_read_cursor cursor; > Group_bound_tracker bound_tracker; > + > public: > void init(THD *thd, READ_RECORD *info, SQL_I_List<ORDER> *partition_list, > SQL_I_List<ORDER> *order_list) > @@ -910,6 +913,35 @@ class Frame_unbounded_following : public Frame_cursor > }; > > > +class Frame_unbounded_following_set_count : public Frame_unbounded_following > +{ > + void next_partition(longlong rownum, Item_sum* item) > + { > + ulonglong num_rows_in_partition= 0; > + if (!rownum) > + { > + /* Read the first row */ > + if (cursor.get_next()) > + return; > + num_rows_in_partition++; > + } > + > + /* Remember which partition we are in */ > + bound_tracker.check_if_next_group(); > + /* Walk to the end of the partition, find how many rows there are. */ > + while (!cursor.get_next()) > + { > + if (bound_tracker.check_if_next_group()) > + break; > + num_rows_in_partition++; > + } > + > + Item_sum_window_with_row_count* item_with_row_count = > + static_cast<Item_sum_window_with_row_count *>(item); > + item_with_row_count->set_row_count(num_rows_in_partition); > + } > +}; > + > ///////////////////////////////////////////////////////////////////////////// > // ROWS-type frame bounds > ///////////////////////////////////////////////////////////////////////////// > @@ -1212,18 +1244,37 @@ Frame_cursor *get_frame_cursor(Window_frame *frame, bool is_top_bound) > return NULL; > } > > +void add_extra_frame_cursors(List<Frame_cursor> *cursors, > + const Item_sum *window_func) > +{ > + switch (window_func->sum_func()) > + { > + case Item_sum::CUME_DIST_FUNC: > + cursors->push_back(new Frame_unbounded_preceding); What is the reason for having this cursor at all? > + cursors->push_back(new Frame_range_current_row_bottom); > + break; This way to structure the code looks wrong. Why should add_extra_frame_cursors know that some functions (e.g. CUME_DIST) need a cursor, and some don't? I think it would be more natural if CUME_DIST knew that it needs certain kinds of cursor(s). Maybe, we should put off this question for now, and get back to it when we have more window functions. (If we implement LEAD and LAG, we will need to create Frame_rows_n_preceding and Frame_rows_n_following for them. Will this be done in this function too?) Another question (possibly wrong): why is CUME_DIST mentioned here, while PERCENT_RANK isn't? At the first glance they will need the same kind of cursors, wont they? > + default: > + cursors->push_back(new Frame_unbounded_preceding); > + cursors->push_back(new Frame_rows_current_row_bottom); > + } > +} > + > List<Frame_cursor> get_window_func_required_cursors( > const Item_window_func* item_win) > { > List<Frame_cursor> result; I don't think is's good idea to pass around List objects by value. - there is no need to do this. - List template is not fully specified w.r.t what happens when you copy-assign it. I don't consider the situation with two List objects sharing the list but not sharing e.g. the value of List::elements meaningful. because if that, I would pass a pointer or reference to the list and have the code populate the passed list. > > + if (item_win->requires_partition_size()) > + result.push_back(new Frame_unbounded_following_set_count); > + > /* > If it is not a regular window function that follows frame specifications, > specific cursors are required. > */ > if (item_win->is_frame_prohibited()) > { > - DBUG_ASSERT(0); // TODO-cvicentiu not-implemented yet. > + add_extra_frame_cursors(&result, item_win->window_func()); > + return result; > } > > /* A regular window function follows the frame specification. */ > @@ -1283,7 +1334,6 @@ bool compute_window_func_with_frames(Item_window_func *item_win, > > List<Frame_cursor> cursors= get_window_func_required_cursors(item_win); > > - > List_iterator_fast<Frame_cursor> it(cursors); > Frame_cursor *c; > while((c= it++)) > @@ -1362,107 +1412,6 @@ bool compute_window_func_with_frames(Item_window_func *item_win, > } > > > -bool compute_two_pass_window_functions(Item_window_func *item_win, > - TABLE *table, READ_RECORD *info) > -{ > - /* Perform first pass. */ > - > - // TODO-cvicentiu why not initialize the record for when we need, _in_ > - // this function. > - READ_RECORD *info2= new READ_RECORD(); > - int err; > - bool is_error = false; > - bool first_row= true; > - clone_read_record(info, info2); > - Item_sum_window_with_context *window_func= > - static_cast<Item_sum_window_with_context *>(item_win->window_func()); > - uchar *rowid_buf= (uchar*) my_malloc(table->file->ref_length, MYF(0)); > - > - is_error= window_func->create_window_context(); > - /* Unable to allocate a new context. */ > - if (is_error) > - return true; > - > - Window_context *context = window_func->get_window_context(); > - /* > - The two pass algorithm is as follows: > - We have a sorted table according to the partition and order by clauses. > - 1. Scan through the table till we reach a partition boundary. > - 2. For each row that we scan, add it to the context. > - 3. Once the partition boundary is met, do a second scan through the > - current partition and use the context information to compute the value for > - the window function for that partition. > - 4. Reset the context. > - 5. Repeat from 1 till end of table. > - */ > - > - bool done = false; > - longlong rows_in_current_partition = 0; > - // TODO handle end of table updating. > - while (!done) > - { > - > - if ((err= info->read_record(info))) > - { > - done = true; > - } > - > - bool partition_changed= done || item_win->check_if_partition_changed(); > - // The first time we always have a partition changed. Ignore it. > - if (first_row) > - { > - partition_changed= false; > - first_row= false; > - } > - > - if (partition_changed) > - { > - /* > - We are now looking at the first row for the next partition, or at the > - end of the table. Either way, we must remember this position for when > - we finish doing the second pass. > - */ > - table->file->position(table->record[0]); > - memcpy(rowid_buf, table->file->ref, table->file->ref_length); > - > - for (longlong row_number = 0; row_number < rows_in_current_partition; > - row_number++) > - { > - if ((err= info2->read_record(info2))) > - { > - is_error= true; > - break; > - } > - window_func->add(); > - // Save the window function into the table. > - item_win->save_in_field(item_win->result_field, true); > - err= table->file->ha_update_row(table->record[1], table->record[0]); > - if (err && err != HA_ERR_RECORD_IS_THE_SAME) > - { > - is_error= true; > - break; > - } > - } > - > - if (is_error) > - break; > - > - rows_in_current_partition= 0; > - window_func->clear(); > - context->reset(); > - > - // Return to the beginning of the new partition. > - table->file->ha_rnd_pos(table->record[0], rowid_buf); > - } > - rows_in_current_partition++; > - context->add_field_to_context(item_win->result_field); > - } > - > - window_func->delete_window_context(); > - delete info2; > - my_free(rowid_buf); > - return is_error; > -} > > > /* Make a list that is a concation of two lists of ORDER elements */ > @@ -1527,16 +1476,12 @@ bool Window_func_runner::setup(THD *thd) > compute_func= compute_window_func_values; > break; > } > - case Item_sum::PERCENT_RANK_FUNC: > - case Item_sum::CUME_DIST_FUNC: > - { > - compute_func= compute_two_pass_window_functions; > - break; > - } > case Item_sum::COUNT_FUNC: > case Item_sum::SUM_BIT_FUNC: > case Item_sum::SUM_FUNC: > case Item_sum::AVG_FUNC: > + case Item_sum::PERCENT_RANK_FUNC: > + case Item_sum::CUME_DIST_FUNC: > { > /* > Frame-aware window function computation. It does one pass, but > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Fwd: Gsoc 2016 Mdev 371 Unique index for blob
by Sachin Setia 29 Mar '16

29 Mar '16

Hi Sergei Did you see the patch.Actually the patch works for every combination of blob and varchars table definition. Can you please tell what extra modification I should make Regards sachin On Mon, Mar 28, 2016 at 9:34 PM, Sachin Setia <sachinsetia1001(a)gmail.com> wrote: > > ---------- Forwarded message ---------- > From: Sachin Setia <sachinsetia1001(a)gmail.com> > Date: Mon, Mar 28, 2016 at 9:13 PM > Subject: Re: [Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob > To: maria-developers(a)lists.launchpad.net > > > I am Sorry I did not see this mail.So I did not send the prototype link in > proposal > > On Thu, Mar 24, 2016 at 10:41 PM, Sergei Golubchik <serg(a)mariadb.org> > wrote: > >> Hi, Sachin! >> >> As you might have noticed on the mailing list, there is another student >> who wants to do this project. >> >> It would help us choose, if you'd put your prototype code on github and >> put the link to the repository into your final proposal. >> >> Thanks! >> >> On Mar 15, Sachin Setia wrote: >> > Dear Developers, >> > >> > I was doing prototype for this project as Mr Sergei Golubchik suggested >> > >> >> Regards, >> Sergei >> Chief Architect MariaDB >> and security(a)mariadb.org >> > > >

2 1

[Maria-developers] Getting restarted with mariadb and a port to AIX
by Michael Felt 28 Mar '16

28 Mar '16

Hi all, This week I added a few issues that I felt were clear enough to report directly. This one I am not so sure of as it concerns two areas I know too little about: a) cmake (how to get an equivalent of GNU autotools ./configure --help) b) cmake (how to make it show the commands it is actually running (so I could add -bnoquiet and/or -bloadmap:XXX.map) c) 'strategic' grepping through C++ sources to find where an undefined label is calling something. After I 'patch' aka bandaid mariadb-5.5.48.0/sql-common/mysql_async.c (see https://jira.mariadb.org/browse/MDEV-9814) I get loader errors - and I cannot, in ways I could with an autotools generated 'make' see how to debug. Q) are there variables (e.g., V=1) I can pass to make so that the commands it performs are more verbose Q) have any ideas why these symbols are undefined? Thanks, Michael

2 1

[Maria-developers] Unique Index for blob
by Sachin Setia 28 Mar '16

28 Mar '16

Hello Devlopers Hi this is sachin. I applied for gsoc 2016(applied by name sachin setiya) My topic was unique index for blob(MyISAM) But i already completed some part of it .it is working for blobs varchar for definitions like create table tbl(abc varchar(2000) , xyz blob ,unique(abc,xyz)); I have corrected the table2myisam function . For documentation you can refer to my gsoc proposal If am doing wrong please let me know. I am attaching the patch file TODO Second thing which i need to do is to tweak check_definition function to correctly check for definition currently I am commenting the function code Regards sachin

1 1

Re: [Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob
by Sachin Setia 28 Mar '16

28 Mar '16

I am Sorry I did not see this mail.So I did not send the prototype link in proposal On Thu, Mar 24, 2016 at 10:41 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Sachin! > > As you might have noticed on the mailing list, there is another student > who wants to do this project. > > It would help us choose, if you'd put your prototype code on github and > put the link to the repository into your final proposal. > > Thanks! > > On Mar 15, Sachin Setia wrote: > > Dear Developers, > > > > I was doing prototype for this project as Mr Sergei Golubchik suggested > > > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org >

1 0

[Maria-developers] Please review MDEV-6353 my_ismbchar() and my_mbcharlen() refactoring
by Alexander Barkov 28 Mar '16

28 Mar '16

Hi Sergei, Please review a partial patch for MDEV-6353. It removes my_mbcharlen() in all remaining pieces of the code, except LOAD DATA and SELECT INTO OUTFILE. I'll do LOAD DATA and SELECT INTO OUTFILE in a separate patch. Thanks.

1 0

Re: [Maria-developers] Google Summer of Code 2016
by Soham Mankad 26 Mar '16

26 Mar '16

Respected sir, I am Soham Mankad and thanks Sergei sir for directing to me this developers list. As I had mailed you before I have gone through the basic requirements and goals of the project "Aggregate Stored Functions" and also thinking on varoius syntax and languages that can be used like hsql, Postgresql, or mysql. Also as you had mentioned in a reply to Vikas Gupta still i have not found any data base with this facility existing uptill now. Also I have seen the syntax and possible ways given by you in knowledge base of the project. The most basic form that you have mentioned i.e using nested commands i.e returning sum adding two numbers. I am also learning about various aspects of create function commands in different languages loke read only or returning which rows or coloumns so that optimum result will be given. I have submitted my proposal for this project on Gsoc'16 website. I would definitely appreciate an opportunity to work on this project. Regards, Soham Mankad.

1 0

[Maria-developers] Window functions: why some functions are not in select_lex->window_funcs?
by Sergey Petrunia 24 Mar '16

24 Mar '16

Hello Igor, I was trying to move JOIN::process_window_functions() from using join->fields_list to using join->select_lex->window_funcs. And I discovered that join->select_lex->window_funcs does not contain window function items that use window definition. Is this intentional? Example: create table t0 (a int); insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table t1 (pk int, c int); insert into t1 select a+1,1 from t0; update t1 set c=2 where pk not in (1,2,3,4); select pk, c, count(*) over (partition by c order by pk rows between 2 preceding and 2 following) as CNT from t1 Here: #1 0x0000555555af5db6 in AGGR_OP::end_send (this=0x7fff4c0086c8) (gdb) p join->select_lex->window_funcs $69 = {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x7fff4c005d18, last = 0x7fff4c005d18, elements = 1}, <No data fields>} But: select pk, c, count(*) over w1 as CNT from t1 window w1 as (partition by c order by pk rows between 2 preceding and 2 following); (gdb) p join->select_lex->window_funcs $74 = {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x555556d48820, last = 0x55555ab10b70, elements = 0}, <No data fields>} BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

Re: [Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob
by Sachin Setia 23 Mar '16

23 Mar '16

Hello Sergei Actually I was prototyping for blob and varchar for aria and myisam storage engine. My prototype worked for complex definations like craete table(abc int primary key, blob_col blob unique, varchar_col varchar(1000) unique) engine=myisam; Solved the issue of frm file incosistance. As you suggested for doing it for innodb i am current working on it.Innodb does not natively support hash based index. when we run select distinct column from tbl; it use create_internal_tmp_table() which uses maria storage engine for creating tmp table. But query like this works MariaDB [sachin]> create table iu2(abc blob unique); Query OK, 0 rows affected (0.04 sec) MariaDB [sachin]> insert into iu2 values(1); Query OK, 1 row affected (0.03 sec) MariaDB [sachin]> insert into iu2 values(1); ERROR 1062 (23000): Duplicate entry '1' for key 'abc' this query does not use hash but it simply compares values Will write a proposal shortly. On Tue, Mar 22, 2016 at 4:20 PM, Sachin Setia <sachinsetia1001(a)gmail.com> wrote: > Hello Sergi > Actually I was prototyping for blob and varchar for aria and myisam > storage engine. > My prototype worked for complex definations like > craete table(abc int primary key, blob_col blob unique, varchar_col > varchar(1000) unique) engine=myisam; > Solved the issue of frm file incosistance. > > As you suggested for doing it for innodb i am current working on it.Innodb > does not natively support hash based index. > when we run select distinct column from tbl; > it use create_internal_tmp_table() which uses maria storage engine for > creating tmp table. > But query like this works > MariaDB [sachin]> create table iu2(abc blob unique); > Query OK, 0 rows affected (0.04 sec) > > MariaDB [sachin]> insert into iu2 values(1); > Query OK, 1 row affected (0.03 sec) > > MariaDB [sachin]> insert into iu2 values(1); > ERROR 1062 (23000): Duplicate entry '1' for key 'abc' > this query does not use hash but it simply compares values > Will write a proposal shortly. > > > Regards > sachin > > On Sat, Mar 19, 2016 at 1:52 AM, Sergei Golubchik <serg(a)mariadb.org> > wrote: > >> Hi, Sachin! >> >> On Mar 18, Sachin Setia wrote: >> > >> > ERROR 1030 (HY000): Got error 190 "Incompatible key or row definition >> > between the MariaDB .frm file and the information in the storage engine. >> > You have to dump an" from storage engine MyISAM >> > >> > We are getting this becuase in mi_create for each unique_def it creates >> one >> > keydef and writes it.And this creates two problem >> > 1. frm keydef algorithm is different from saved kefdef algorithm(always >> > zero) for the time I have solved this problem . >> > >> > 2.Mismatch between keydef's keysegs the reason for this is when >> mi_create >> > creates keyseg for unique_def it did not keeps the orignal uniquedef's >> > keyseg parameters in mind like language start length which creates >> problem >> > in check_definition function in ha_myisam.cc.I am currently working on >> it >> > Once again sorry for this foolish mistake. >> > Regars >> > sachin >> >> No problem, everyone makes mistakes :) >> >> It's a prototype, after all. It's much more important that you >> understand how the code works and why it doesn't work. >> >> Regards, >> Sergei >> Chief Architect MariaDB >> and security(a)mariadb.org >> > >

2 3

[Maria-developers] More memory allocation for key seg
by Sachin Setia 23 Mar '16

23 Mar '16

Hello Devlopers I was debugging the source code of mariadb. My query was MariaDB [sachin]> create table pro2(abc int primary key,xyz int , ass int ,unique(xyz,ass))engine=myisam; In table2myisam function of file ha_myisam.cc i found that we are allocating more memory for keysegments if (!(my_multi_malloc(MYF(MY_WME), recinfo_out, (share->fields * 2 + 2) * sizeof(MI_COLUMNDEF), keydef_out, share->keys * sizeof(MI_KEYDEF), &keyseg, (share->key_parts + share->keys) * sizeof(HA_KEYSEG), ^ here why this NullS))) And we donot use these extra key segments because suceeding for loop use only total number of share->key_parts keysegments. And same in ha_maria file.Is this a bug or I am missing something Regards sachin

2 2

[Maria-developers] Please review MDEV-9604 crash in Item::save_in_field with empty enum value
by Alexander Barkov 22 Mar '16

22 Mar '16

Hi Sergei, It fixes the reported crash, and additionally inconsistencies when comparing temporal columns to empty strings: In some cases it returned rows, in other cases it returned no rows, depending on: - the engine - the type of comparison arguments (columns vs literals). - presence of an index Now everything works in the same way. Note, this additional problem is not 10.1 specific, it existed in 10.0. But 10.0 and 10.1 worked differently in certain cases. Thanks!

2 4

Re: [Maria-developers] Aggregate stored functions [MDEV-7773]
by Sergei Golubchik 22 Mar '16

22 Mar '16

Hi, Varun! On Mar 21, Varun Gupta wrote: > Hi Sergei, > > 4) :create an Item_sum_sp to be able to use aggregate stored functions in > queries. > Does this mean we can have the aggregate stored functions in the SELECT , > WHERE etc clauses?can you give me more details about this. Yes, of course. That's how functions are used, right? :) SELECT AVG(x), COUNT(y), MAX(z) FROM t1 WHERE aaa=12 GROUP BY bbb; for example. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] 29e99a7: MDEV-717 LP:1003679 - Wrong binlog order on concurrent DROP schema and
by Sergei Golubchik 21 Mar '16

21 Mar '16

Hi, Alexey! In general the patch is pretty good. But I've had few comments, mostly about bad choice of error messages. It's a historical behavoir that you decided not to change. If you'd prefer to leave it as is, that's ok, but then, please, create a new MDEV about these error messages. And add test cases :) On Mar 20, Alexey Botchkov wrote: > revision-id: 29e99a701817da56499df9126830d9533e4d66f3 (mariadb-10.1.12-26-g29e99a7) > parent(s): 7cb16dc2a369058b31df1b3999989f6beff94d07 > committer: Alexey Botchkov > timestamp: 2016-03-20 23:57:26 +0400 > message: > > MDEV-717 LP:1003679 - Wrong binlog order on concurrent DROP schema and > CREATE function. > The cause of the issue is when DROP DATABASE takes > metadata lock and is in progress through it's > execution, a concurrently running CREATE FUNCTION checks > for the existence of database which it succeeds and then it > waits on the metadata lock. Once DROP DATABASE writes to > BINLOG and finally releases the metadata lock on schema > object, the CREATE FUNCTION waiting on metadata lock gets > in it's code path and succeeds and writes to binlog. Please structure the commit comment as <first line - the subject> <empty line> <the rest - the body> That's the convention and various tools, for example, github itself, use it to parse and display the commit subject and body. > --- > sql/events.cc | 29 ++++++++++++++------ > sql/sp.cc | 81 +++++++++++++++++++++++++++++++++++++------------------- > sql/sp.h | 2 +- > sql/sql_parse.cc | 56 +++++++-------------------------------- What, no test cases at all? Please, add tests for everything you've fixed: - create event - update event with moving to a new database > diff --git a/sql/sp.cc b/sql/sp.cc > index 6ec5914..11ba186 100644 > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -1040,7 +1045,22 @@ sp_create_routine(THD *thd, stored_procedure_type type, sp_head *sp) > > /* Grab an exclusive MDL lock. */ > if (lock_object_name(thd, mdl_type, sp->m_db.str, sp->m_name.str)) > - DBUG_RETURN(SP_OPEN_TABLE_FAILED); > + { > + my_error(ER_SP_STORE_FAILED, MYF(0), SP_TYPE_STRING(type), sp->m_name.str); No better error message here? May be ER_BAD_DB_ERROR? I understand that the old behavior is ER_SP_STORE_FAILED here. > + DBUG_RETURN(TRUE); > + } > + > + /* > + Check that a database directory with this name > + exists. Design note: This won't work on virtual databases > + like information_schema. > + */ > + if (check_db_dir_existence(sp->m_db.str)) > + { > + my_error(ER_BAD_DB_ERROR, MYF(0), sp->m_db.str); > + DBUG_RETURN(TRUE); > + } > + > > /* Reset sql_mode during data dictionary operations. */ > thd->variables.sql_mode= 0; > @@ -1049,7 +1069,9 @@ sp_create_routine(THD *thd, stored_procedure_type type, sp_head *sp) > thd->count_cuted_fields= CHECK_FIELD_WARN; > > if (!(table= open_proc_table_for_update(thd))) > - ret= SP_OPEN_TABLE_FAILED; > + { > + my_error(ER_SP_STORE_FAILED, MYF(0), SP_TYPE_STRING(type),sp->m_name.str); better error message, may be? > + } > else > { > /* Checking if the routine already exists */ > @@ -1268,7 +1289,8 @@ sp_create_routine(THD *thd, stored_procedure_type type, sp_head *sp) > &(thd->lex->definer->host), > saved_mode)) > { > - ret= SP_INTERNAL_ERROR; > + my_error(ER_SP_STORE_FAILED, MYF(0), > + SP_TYPE_STRING(type), sp->m_name.str); That's OOM, can be a better error message too. Btw, could you reverse the return value of show_create_sp() please? > goto done; > } > /* restore sql_mode when binloging */ > @@ -1277,9 +1299,14 @@ sp_create_routine(THD *thd, stored_procedure_type type, sp_head *sp) > if (thd->binlog_query(THD::STMT_QUERY_TYPE, > log_query.ptr(), log_query.length(), > FALSE, FALSE, FALSE, 0)) > - ret= SP_INTERNAL_ERROR; > + { > + my_error(ER_SP_STORE_FAILED, MYF(0), > + SP_TYPE_STRING(type), sp->m_name.str); that's very confusing. "Failed to CREATE %s %s" when the creation has actually succeeded. The same applies to the OOM error above. > + goto done; > + } > thd->variables.sql_mode= 0; > } > + ret= FALSE; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Randomness in TokuDB builds
by Otto Kekäläinen 21 Mar '16

21 Mar '16

Hello TokuDB experts! There is some randomness, maybe some unique build id in every TokuDB build that breaks the reproducable builds of it in Debian. Check out: https://tests.reproducible-builds.org/dbd/unstable/amd64/mariadb-10.0_10.0.… I can't find any mentions of a build id or similar in the sources. Can somebody help me figure out where the randomness stems from, and how I should patch the source to make the build deterministic? - Otto

2 1

[Maria-developers] Please review MDEV-9516 type error when setting session variable
by Alexander Barkov 21 Mar '16

21 Mar '16

Hello Sergei, Please review a patch for MDEV-9516. Thanks.

1 0

Re: [Maria-developers] 8d675d5: MDEV-6058 MySQL Bug #11766693: LOG-SLOW-ADMIN-STATEMENTS AND
by Sergei Golubchik 21 Mar '16

21 Mar '16

Hi, Alexey! On Mar 21, Alexey Botchkov wrote: > revision-id: 8d675d5764e7d8ecad19fafdd167bee9a41d1dc7 (mariadb-10.1.12-27-g8d675d5) > parent(s): 94768542115289272996f6450b7d3cafa75dcead > committer: Alexey Botchkov > timestamp: 2016-03-21 13:02:09 +0400 > message: > > MDEV-6058 MySQL Bug #11766693: LOG-SLOW-ADMIN-STATEMENTS AND > LOG-SLOW-SLAVE-STATEMENTS NOT DISPLAYED. > These parameters were moved from the command line options to the > system variables section. Treatment of the > opt_log_slow_slave_statements changed to let the > dynamic change of the variable. Thanks! Looks good, just one question: > diff --git a/mysql-test/suite/rpl/r/rpl_slow_query_log.result b/mysql-test/suite/rpl/r/rpl_slow_query_log.result > index 79b83b4..871722e 100644 > --- a/mysql-test/suite/rpl/r/rpl_slow_query_log.result > +++ b/mysql-test/suite/rpl/r/rpl_slow_query_log.result > @@ -76,6 +76,18 @@ ALTER TABLE t1 ADD INDEX id1(a); > INSERT INTO t1 values(1, sleep(3)); > ### Assertion is good. Both Master and Slave exhibit the > ### same number of queries in slow log: 1 > +******************************************************************** > +**** TRUNCATE the slow log then check whether runtime changes of > +**** log_slow_slave_statements work without slave restart. > +******************************************************************** > +SET @old_log_slow_slave_statements= @@global.log_slow_slave_statements; > +SET @@global.log_slow_slave_statements = off; > +TRUNCATE mysql.slow_log; > +INSERT INTO t1 values(1, sleep(3));; > +0 > +SET @@global.log_slow_slave_statements = on; > +INSERT INTO t1 values(1, sleep(3));; > +1 > SET @@global.log_output= @old_log_output; > SET @@global.long_query_time= @old_long_query_time; > DROP TABLE t1; > diff --git a/mysql-test/suite/rpl/t/rpl_slow_query_log.test b/mysql-test/suite/rpl/t/rpl_slow_query_log.test > index faf037a..3505883 100644 > --- a/mysql-test/suite/rpl/t/rpl_slow_query_log.test > +++ b/mysql-test/suite/rpl/t/rpl_slow_query_log.test > @@ -299,6 +299,39 @@ if ($master_slow_query == $slave_slow_query) > -- echo ### same number of queries in slow log: $master_slow_query > } > > +-- echo ******************************************************************** > +-- echo **** TRUNCATE the slow log then check whether runtime changes of > +-- echo **** log_slow_slave_statements work without slave restart. > +-- echo ******************************************************************** > + > +SET @old_log_slow_slave_statements= @@global.log_slow_slave_statements; > +SET @@global.log_slow_slave_statements = off; > +TRUNCATE mysql.slow_log; > + > +-- connection master > + > +--disable_warnings > +-- eval $slow_query; > +--enable_warnings > +sync_slave_with_master; > + > +-- connection slave > + > +eval SELECT sql_text FROM mysql.slow_log WHERE sql_text like '$slow_query'; > + > +SET @@global.log_slow_slave_statements = on; > + > +-- connection master > + > +--disable_warnings > +-- eval $slow_query; > +--enable_warnings > +sync_slave_with_master; > + > +-- connection slave > + > +eval SELECT sql_text FROM mysql.slow_log WHERE sql_text like '$slow_query'; Did you run this test? It seems the result doesn't match the test. The result looks like as if you'd have eval SELECT COUNT(*) FROM ... Please, run the full test suite before pushing. > + > -- connection master > SET @@global.log_output= @old_log_output; > SET @@global.long_query_time= @old_long_query_time; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-9369 IN operator with ( num, NULL ) gives inconsistent result
by Sergei Golubchik 21 Mar '16

21 Mar '16

Hi, Alexander! On Feb 08, Alexander Barkov wrote: > Hello Sergei, > > Please review a patch for MDEV-9369. > > Briefly, it fixes cmp_item::cmp() to return three possible values: > FALSE, TRUE, UNKNOWN. These values are then recursively pulled to > the very top level, to the owner Item_xxx. > > The owner Item_func_in (as well as Item_func_case and Item_func_equal) > then further "translate" these tree-values logic into the traditional > val_int()+null_value combination using some additional information. > > For example, Item_func_in::val_int() uses "negated" as an additional > information. > > The patch looks Ok for me. Looks ok to me too. A couple of stylistic comments, see below. ok to push afterwards. > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index d5f5087..39003de 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -3837,15 +3843,15 @@ void cmp_item_decimal::store_value(Item *item) > /* val may be zero if item is nnull */ > if (val && val != &value) > my_decimal2decimal(val, &value); > + set_null_value(item->null_value); > } > > > int cmp_item_decimal::cmp(Item *arg) > { > my_decimal tmp_buf, *tmp= arg->val_decimal(&tmp_buf); > - if (arg->null_value) > - return 1; > - return my_decimal_cmp(&value, tmp); > + return (m_null_value || arg->null_value) ? if you access m_null_value directly, you don't need a setter for it. it'll be less confusing if you replace set_null_value(X) with m_null_value=X. alternatively, you can add a getter is_null_value(). But I personally wouldn't do it, cmp_xxx is a small set of simple classes, no need to get too enterprisey there. > + UNKNOWN : (my_decimal_cmp(&value, tmp) != 0); > } > > > @@ -3892,10 +3900,10 @@ cmp_item *cmp_item_datetime::make_same() > } > > > -bool Item_func_in::nulls_in_row() > +bool Item_func_in::list_contains_null() I found the old name clear enough. But ok, whatever... > { > Item **arg,**arg_end; > - for (arg= args+1, arg_end= args+arg_count; arg != arg_end ; arg++) > + for (arg= args + 1, arg_end= args+arg_count; arg != arg_end ; arg++) > { > if ((*arg)->null_inside()) > return 1; > @@ -4010,6 +4018,32 @@ void Item_func_in::fix_length_and_dec() > } > } > > + /* > + First conditions for bisection to be possible: > + 1. All types are similar, and > + 2. All expressions in <in value list> are const > + */ > + bool bisection_possible= > + type_cnt == 1 && // 1 > + const_itm; // 2 > + if (bisection_possible) > + { > + /* > + In the presence of NULLs, the correct result of evaluating this item > + must be UNKNOWN or FALSE. To achieve that: > + - If type is scalar, we can use bisection and the "have_null" boolean. > + - If type is ROW, we will need to scan all of <in value list> when > + searching, so bisection is impossible. Unless: > + 3. UNKNOWN and FALSE are equivalent results > + 4. Neither left expression nor <in value list> contain any NULL value > + */ > + > + if (cmp_type == ROW_RESULT && > + !((is_top_level_item() && !negated) || // 3 > + (!list_contains_null() && !args[0]->maybe_null))) // 4 > + bisection_possible= false; I think the condition will be easier to understand if you remove the negation of that complex condition in parentheses: if (cmp_type == ROW_RESULT && ((!is_top_level_item() || negated) && // 3 (list_contains_null() || args[0]->maybe_null))) // 4 I find this ^^^ much easier to understand. > + } > + > if (type_cnt == 1) > { > if (cmp_type == STRING_RESULT && Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] Google Summer of Code [MySql -> MsSql]
by Bo Zhu 20 Mar '16

20 Mar '16

Hi, everyone. I am Bo, a second year master student in Clemson university, SC. I had ever implemented a small relational database and java compiler. I am really interested in the MaxScale filter to translate the Mysql query to MsSql one. In this task, we need to write some C++ code as a MaxScale's filter, right ? I think it is about syntax translation which is a compiler front-end problem. I have shared my proposal draft and am really looking forward to get some feedback.

1 0

[Maria-developers] MaxScale filter to real Microsoft SQL Server syntax
by Wepngong Benaiah 20 Mar '16

20 Mar '16

> > Hello everyone, I am Wepngong Ngeh Benaiah, a 4th year Computer Engineering student at the University of Buea, Cameroon. I have been looking at gsoc projects on https://mariadb.com/kb/en/mariadb/google-summer-of-code-2016/ and got interested in *MaxScale filter to real Microsoft SQL Server syntax [https://jira.mariadb.org/browse/MXS-1 <https://jira.mariadb.org/browse/MXS-1>]. * I will love to contribute to MariaDB by means of GSOC2016. I have drafted a proposal and submitted to google and will love to get feedback on this Thanks in advance -- Buea,Cameroon University of Buea Faculty of Engineering and Technology Computer Engineering Software Engineering * Wepngong Ngeh Benaiah* *https://github.com/bwepngong <https://github.com/bwepngong>*

1 0

[Maria-developers] GSoC 2016 CONC-125
by Christian Massey 19 Mar '16

19 Mar '16

Hi everyone, I am Christian Massey a computer science student at York University and I wish to work on Providing import and export functions for popular data formats like JSON, XML , PHP, ... for Connector/C and MariaDB Server. I have been inspecting the code to understand how the dynamic column packed string is encoded and I see kind on big endian encoding( Correct me if I am wrong please). Also I see that the old format already work like arrays and it uses 3 bits for encoding type which is the main reason why not all types in the enum are supported in this old format. DBUG_ASSERT(((type - 1) & (~7)) == 0); /* fit in 3 bits */ (Again please correct me if I am wrong. The encoding is not easy to understand) My Question is do we need to create a completely new type to support arrays or do we change the old format which already work like arrays in a way but lacks support for some types(I don't know which of the enums yet). Please, I wish to write a proposal for this project and hope to discuss more with mentors for more clarification on how we are going to implement this as it seems to be a very important if not the most critical part of this project. Cheers, Kris.

2 2

[Maria-developers] Aggregate stored functions [MDEV-7773]
by Varun Gupta 19 Mar '16

19 Mar '16

Well i have checked out the syntax for the databases for aggregate functions. I thought using the syntax similar to that of HSQL would be good. In it if we are having N tuples then we are computing the values for the N rows and then after that we make another call and return the value for the function. Well this seems very reasonable to do. Using PostgreSQL <http://www.postgresql.org/docs/8.3/static/sql-createaggregate.html> we need to have two regular functions but if we have an aggregate function like Count(column_name) , then this would be done in just a single function , I guess we would not need two functions for it . Regards, Varun

2 11

Re: [Maria-developers] b1a1e5b: MDEV-6058 MySQL Bug #11766693: LOG-SLOW-ADMIN-STATEMENTS AND LOG-SLOW-SLAVE-STATEMENTS NOT DISPLAYED.
by Sergei Golubchik 18 Mar '16

18 Mar '16

Hi, Alexey! On Mar 18, Alexey Botchkov wrote: > revision-id: b1a1e5b3d3d5ab95c596580de07daf38a3efa039 (mariadb-10.1.12-19-gb1a1e5b) > parent(s): 4aac51db9a55a11574cbfa60484d4329d35b6f2c > committer: Alexey Botchkov > timestamp: 2016-03-18 11:47:05 +0400 > message: > > MDEV-6058 MySQL Bug #11766693: LOG-SLOW-ADMIN-STATEMENTS AND LOG-SLOW-SLAVE-STATEMENTS NOT DISPLAYED. > These parameters were moved from the command line options to the > system variables section. Treatment of the > opt_log_slow_slave_statements changed to let the dynamic change > of the variable. > > --- > mysql-test/r/variables.result | 10 ++ > mysql-test/suite/rpl/r/rpl_slow_query_log.result | 10 ++ > mysql-test/suite/rpl/t/rpl_slow_query_log.test | 43 ++++++++ > .../r/log_slow_admin_statements_basic.result | 76 ++++++++++++++ > .../r/log_slow_admin_statements_func.result | 46 +++++++++ > .../r/log_slow_slave_statements_basic.result | 76 ++++++++++++++ > .../t/log_slow_admin_statements_basic.test | 110 +++++++++++++++++++++ > .../sys_vars/t/log_slow_admin_statements_func.test | 61 ++++++++++++ > .../t/log_slow_slave_statements_basic.test | 110 +++++++++++++++++++++ > mysql-test/t/variables.test | 11 +++ I suppose you didn't run the full test suite. Otherwise you would've noticed that sysvar_* results were changed. Also, because we have sysvar_* tests, we no longer create <varname>_basic tests for every new variable, they are no longer useful. Starting from 10.1 you only need to test the functionality of new variables, not their basic properties. > sql/log_event.cc | 10 ++ > sql/mysqld.cc | 8 -- > sql/sys_vars.cc | 13 +++ > 13 files changed, 576 insertions(+), 8 deletions(-) > > diff --git a/mysql-test/r/variables.result b/mysql-test/r/variables.result > index fef3e4a..e46e65f 100644 > --- a/mysql-test/r/variables.result > +++ b/mysql-test/r/variables.result > @@ -1797,3 +1797,13 @@ select * from information_schema.session_variables where variable_name='sql_mode > VARIABLE_NAME VARIABLE_VALUE > SQL_MODE ANSI_QUOTES > End of 5.5 tests > +show variables like 'log_slow%statements'; > +Variable_name Value > +log_slow_admin_statements OFF > +log_slow_slave_statements OFF > +select * from information_schema.global_variables where variable_name like 'log_slow_admin_statements'; > +VARIABLE_NAME VARIABLE_VALUE > +LOG_SLOW_ADMIN_STATEMENTS OFF > +select * from information_schema.global_variables where variable_name like 'log_slow_slave_statements'; > +VARIABLE_NAME VARIABLE_VALUE > +LOG_SLOW_SLAVE_STATEMENTS OFF You can remove this test. > diff --git a/mysql-test/suite/rpl/t/rpl_slow_query_log.test b/mysql-test/suite/rpl/t/rpl_slow_query_log.test > index faf037a..de336fc 100644 > --- a/mysql-test/suite/rpl/t/rpl_slow_query_log.test > +++ b/mysql-test/suite/rpl/t/rpl_slow_query_log.test > @@ -299,6 +299,49 @@ if ($master_slow_query == $slave_slow_query) > -- echo ### same number of queries in slow log: $master_slow_query > } > > +-- echo ******************************************************************** > +-- echo **** TRUNCATE the slow log then check whether runtime changes of > +-- echo **** log_slow_slave_statements work without slave restart. > +-- echo ******************************************************************** > + > +SET @old_log_slow_slave_statements= @@global.log_slow_slave_statements; > +SET @@global.log_slow_slave_statements = off; > +TRUNCATE mysql.slow_log; > + > +-- connection master > + > +--disable_warnings > +-- eval $slow_query; > +--enable_warnings > +sync_slave_with_master; > + > +-- connection slave > + > +let $slave_slow_query= `SELECT count(*) = 1 FROM mysql.slow_log WHERE sql_text like '$slow_query'`; > +if ($slave_slow_query) > +{ > + SELECT * FROM mysql.slow_log; > + die "Assertion failed! Slow query FOUND in slow query log. Bailing out!"; > +} please, NO! If you want to test whether a certain row is present in a certain table you do SELECT columns FROM table; may be with a WHERE clause. But never if (`SELECT`) { die; } > + > +SET @@global.log_slow_slave_statements = on; > + > +-- connection master > + > +--disable_warnings > +-- eval $slow_query; > +--enable_warnings > +sync_slave_with_master; > + > +-- connection slave > + > +let $slave_slow_query= `SELECT count(*) = 1 FROM mysql.slow_log WHERE sql_text like '$slow_query'`; > +if (!$slave_slow_query) > +{ > + SELECT * FROM mysql.slow_log; > + die "Assertion failed! Slow query NOT FOUND in slow query log. Bailing out!"; > +} Same. > + > -- connection master > SET @@global.log_output= @old_log_output; > SET @@global.long_query_time= @old_long_query_time; > diff --git a/mysql-test/suite/sys_vars/r/log_slow_admin_statements_basic.result b/mysql-test/suite/sys_vars/r/log_slow_admin_statements_basic.result > new file mode 100644 > index 0000000..1cd9135 > --- /dev/null > +++ b/mysql-test/suite/sys_vars/r/log_slow_admin_statements_basic.result Remove this useless test, please > diff --git a/mysql-test/suite/sys_vars/r/log_slow_slave_statements_basic.result b/mysql-test/suite/sys_vars/r/log_slow_slave_statements_basic.result > new file mode 100644 > index 0000000..b9e88ce > --- /dev/null > +++ b/mysql-test/suite/sys_vars/r/log_slow_slave_statements_basic.result > @@ -0,0 +1,76 @@ And this one, too. > diff --git a/mysql-test/suite/sys_vars/t/log_slow_admin_statements_func.test b/mysql-test/suite/sys_vars/t/log_slow_admin_statements_func.test > new file mode 100644 > index 0000000..78e4d35 > --- /dev/null > +++ b/mysql-test/suite/sys_vars/t/log_slow_admin_statements_func.test > + > +# ALTER, OPTIMIZE, CHECK and DROP operations should be logged in slow query log. > +SELECT sql_text FROM mysql.slow_log WHERE sql_text LIKE '%TABLE log_slow_admin_statements%'; this is how one checks whether a query was logged. > diff --git a/sql/log_event.cc b/sql/log_event.cc > index e99ef16..9793f7d 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -4428,6 +4428,16 @@ int Query_log_event::do_apply_event(rpl_group_info *rgi, > if (thd->m_digest != NULL) > thd->m_digest->reset(thd->m_token_array, max_digest_length); > > + if (thd->slave_thread && > + (thd->variables.sql_log_slow != opt_log_slow_slave_statements)) > + { > + /* > + The log_slow_slave_statements variable can be changed dynamically, > + so we have to set the sql_log_slow respectively. > + */ > + thd->variables.sql_log_slow= opt_log_slow_slave_statements; > + } Really? Why not if (thd->slave_thread) thd->variables.sql_log_slow= opt_log_slow_slave_statements; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] failing tests
by Alexander Barkov 18 Mar '16

18 Mar '16

Hello Olivier, a few tests are failing in the main 10.1 tree on Linux: connect.xml connect.xml_grant connect.xml_html connect.xml_mdev5261 connect.xml_mult Can you please have a look? Thanks!

2 1

[Maria-developers] Please review MDEV-9653 Assertion `length || !scale' failed in uint my_decimal_length_to_precision(uint, uint, bool)
by Alexander Barkov 18 Mar '16

18 Mar '16

Hello Sergei, Please review a patch for: MDEV-9653 Assertion `length || !scale' failed in uint my_decimal_length_to_precision(uint, uint, bool) Thanks!

2 6

Re: [Maria-developers] 72b709a: MDEV-9701: CREATE VIEW with GROUP BY or ORDER BY and constant produces invalid definition
by Sergei Golubchik 17 Mar '16

17 Mar '16

Hi, Sanja! On Mar 16, OleksandrByelkin wrote: > revision-id: 72b709ac7503ae6dd2b5e9049322fefb90b0ebbe (mariadb-10.1.12-16-g72b709a) > parent(s): 9b53d84d14a9b031d193f6beae382a232aa738e3 > committer: Oleksandr Byelkin > timestamp: 2016-03-16 19:49:17 +0100 > message: > > MDEV-9701: CREATE VIEW with GROUP BY or ORDER BY and constant produces invalid definition > > Fixed printing integer constant in the ORDER clause (MySQL solution) > Removed workaround for double resolving counter in the ORDER. > > --- > mysql-test/r/view.result | 19 +++++++++++++++++++ > mysql-test/t/view.test | 17 +++++++++++++++++ > sql/sql_lex.cc | 32 ++++++++++++-------------------- > sql/sql_select.cc | 6 +++++- > sql/sql_union.cc | 11 +++++++++++ > 5 files changed, 64 insertions(+), 21 deletions(-) > > diff --git a/mysql-test/r/view.result b/mysql-test/r/view.result > index 8ee72c3..d4a2a9b 100644 > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index fd84109..31fc5f9 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2689,30 +2689,22 @@ void st_select_lex::print_order(String *str, > { > if (order->counter_used) > { > - if (!(query_type & QT_VIEW_INTERNAL)) > - { > char buffer[20]; > size_t length= my_snprintf(buffer, 20, "%d", order->counter); > str->append(buffer, (uint) length); you can replace these three lines with str->append_ulonglong(order->counter); > } > else > { > - /* replace numeric reference with expression */ > + /* replace numeric reference with equivalent for ORDER constant */ > if (order->item[0]->type() == Item::INT_ITEM && > order->item[0]->basic_const_item()) > { > - char buffer[20]; > - size_t length= my_snprintf(buffer, 20, "%d", order->counter); > - str->append(buffer, (uint) length); > /* make it expression instead of integer constant */ > - str->append(STRING_WITH_LEN("+0")); > + str->append(STRING_WITH_LEN("''")); > } > else > (*order->item)->print(str, query_type); > } > - } > - else > - (*order->item)->print(str, query_type); > if (!order->asc) > str->append(STRING_WITH_LEN(" desc")); > if (order->next) > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index fff24b9..f14e8b1 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -21874,7 +21874,11 @@ find_order_in_list(THD *thd, Item **ref_pointer_array, TABLE_LIST *tables, > */ > if (order_item->type() == Item::INT_ITEM && order_item->basic_const_item()) > { /* Order by position */ > - uint count= (uint) order_item->val_int(); > + uint count; > + if (order->counter_used) how can this happen? > + count= order->counter; // counter was once resolved > + else > + count= (uint) order_item->val_int(); > if (!count || count > fields.elements) > { > my_error(ER_BAD_FIELD_ERROR, MYF(0), > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index c835083..8145cec 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -1161,11 +1161,22 @@ List<Item> *st_select_lex_unit::get_unit_column_types() > return &sl->item_list; > } > > + > +static void cleanup_order(ORDER *order) > +{ > + for (; order; order= order->next) > + order->counter_used= 0; > +} > + > + > bool st_select_lex::cleanup() > { > bool error= FALSE; > DBUG_ENTER("st_select_lex::cleanup()"); > > + cleanup_order(order_list.first); > + cleanup_order(group_list.first); why is it needed? > if (join) > { > DBUG_ASSERT((st_select_lex*)join->select_lex == this); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] [Commits] 1e542ec: MDEV-9382: After updating mariadb server apt-configure fails
by Sergei Golubchik 17 Mar '16

17 Mar '16

Hi, Nirbhay! Ok to push. And, please, check whether /etc/default/mariadb needs to be mentioned in the corresponding (mariadb on debian/ubuntu) knowledgebase pages. On Mar 16, Nirbhay Choubey wrote: > revision-id: 1e542ecd8af6f78db767a9b3da016b38d1ec3970 (mariadb-10.1.12-6-g1e542ec) > parent(s): b3bde93149e7135507fd7abbca2786d77552953d > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-03-16 19:39:19 -0400 > message: > > MDEV-9382: After updating mariadb server apt-configure fails > > When acting as a Galera receiver node, server startup may take > more than 30 secs (the current default) as it has to wait for > SST/IST operation to complete besides spending some time doing > wsrep recovery. > > Fixed by raising the default value of MYSQLD_STARTUP_TIMEOUT > to 60 secs. Also sourced /etc/default/mariadb into the init > script so that it can be used to set MYSQLD_STARTUP_TIMEOUT. > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] NO PAD Collation for MariaDB
by Melissa Rose 17 Mar '16

17 Mar '16

Hello everyone, I am Melisa Rose from University of Leeds UK and I wish to contribute to MariaDB under GSoC 2016. I am interested in the project involving adding NO PAD collations to default character sets(MDEV-9711 <https://jira.mariadb.org/browse/MDEV-9711>). Apparently, this project does not require thorough understanding of the Server Internal code and the Instructions highlighted by Alexander Barkov on how to implement this are straight forward. I will be submitting a proposal for this project hoping i'm linked to the assigned mentor soon. Look forward to an awesome summer with MariaDB. Kind regards, Melisa Rose.

2 3

Re: [Maria-developers] [Maria-discuss] Collations, trailing spaces and unique indexes
by Alexander Barkov 17 Mar '16

17 Mar '16

Hi Binarus, On 03/16/2016 06:13 PM, Binarus wrote: >> I just wanted to split the task into these steps: >> >> 1. Default and _bin collations. >> This is what MDEV-9711 is about. >> >> 2. Unicode Collation Algorithm (UCA) based collations >> (e.g. utf8_unicode_nopad_ci) >> >> >> 3. The rest (not covered by the previous steps) >> This will include latin1_general_nopad_cs, latin2_czech_nopad_cs >> >> >> This is why asked in the previous letter if utf8_unicode_nopad_ci would >> work for you. >> Also, which MariaDB version do you need this for? Will 10.1 work? >> > > Thank you very much for this explanation. > > Regarding the version: Our O/S usually is Debian which generally uses software which is quite old. This means that I will have to compile MariaDB myself in every case. > The current project where this new collations are needed will probably take about three months before being finished. It would be very nice if we could have the new collations until then. > In the meantime, we'll continue development with MySQL and switch to MariaDB as soon as the new collations are available. Excellent. The fact that you compile from sources makes the things easier. I guess we can add new collations officially into 10.2. Then after 3 month if you'll feel that 10.2 is not mature enough for you, we can make a custom non-official 10.1 tree with the new collations back-ported (it should be trivial to do). > > ci collations would be sufficient as a starting point. Nevertheless, I think there must be a nopad collation for every collation which already exists. > Personally, since we are using utf8 (and no other encoding) in every part of every project, I would prefer to have *all utf8* nopad collations before >having any other nopad collation. But that's just my personal opinion. >We will be grateful for any progress regardless of the order of things being implemented. Thanks. Then we're flexible to implement #2 first, and wait for #1 to be done under terms of Google Summer Of Code 2016. > > Binarus > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp >

1 0

Re: [Maria-developers] [Maria-discuss] Collations, trailing spaces and unique indexes
by Binarus 17 Mar '16

17 Mar '16

Alexander, I'd like to continue our conversation from Maria-discuss. >> - I have found my_strnxfrm_unicode in the same file and will need more time to make my opinion of how difficult it will be >> (I don't know what a weight is, so I currently try to understand what the function does at all). > > This function is used to create sort keys for non-indexed ORDER BY, > for these cases: > > - ORDER BY on an expression > - ORDER BY on a column that does not have an index > > > The idea is exactly the same with the C function strxfrm. > See "man strxfrm". > > The code implements non-indexed ORDER BY in filesort.cc > as follows: > > 1. It calls *_strnxfrm_* functions for all records and converts > CHAR/VARCHAR/TEXT values into their fixed length binary sortable keys. > > 2 . Then executes binary sorting on these keys. > > By the way, fixing this function might be tricky. > > > Currently my_strnxfrm_unicode() pads the tail using weights of the SPACE > character. > The NO PAD version will need to pad the tail using a weight which is > less than the weight of the smallest possible character. > > This should be easy for UCA bases collations (e.g. > utf8_unicode_nopad_ci), because the smallest possible > character in UCA collations is "U+0009 HORIZONTAL TABULATION", > and its weight is 0x0201. So we can just pad the sort key > using a smaller value 0x0200. > > > But I'm not sure yet what to do with 8-bit collations, > which usually use 0x00 as weight for the smallest character. > So we don't have a smaller value. > There are two options here: > > 1. Pad with 0x00. But this will mean that 'aaa<min>' and just 'aaa' > will have unpredictable order when doing ORDER BY without an index > (where <min> is the smallest possible character in the collation). > > As the smallest character in non-UCA collations is usually > "U+0000 NULL", this will mean that 'aaa\0' and just 'aaa' > will have unpredictable order. > > 2. Reserve extra bytes at the end of the key, to store the true length, so > - 'aaa\0' will have the key '4141410004' > - 'aaa' will have the key '4141410003', and therefore will always > be sorted before 'aaa\0'. > > I'm inclined towards #2, to have consistent ORDER BY behavior > with and without indexes. Wow, thanks for explaining. I now have learned that I probably can't assist you in solving that problem (I haven't been in unicode, weights, sorting and character sets very deep until now). But if it helps, you eventually could break the problem into two steps: 1) Fix the problem with unique indexes 2) Fix ordering for any possible situation The problems we have encountered in real life so far are all related to 1), i.e. we would be happy if unique indexes would consider 'a' and 'a ' being different values. In contrast, ordering is not that important for us. I don't know what others are thinking about that, though. >> >> - My main problem: I did not find my_strnncollsp_utf8_general_ci anywhere (nor in the same neither in any other file). Where is it? > > The function name is just "my_strnncollsp_utf8". OK, thank you very much. I must have misunderstood something. >> Furthermore, studying the code has led to some questions; for example, there already seems to be a #define which controls the padding-when-comparing mode, but only for the _cs collations? > > Can you please clarify which lines do you mean? I am referring to the file ctype-utf8.c, line 5698. There is: #ifndef VARCHAR_WITH_DIFF_ENDSPACE_ARE_DIFFERENT_FOR_UNIQUE diff_if_only_endspace_difference= 0; #endif I haven't been able yet to thoroughly analyze where and how this is used, but at least somebody obviously already had some thoughts whether or not trailing spaces should make a difference in UNIQUEs :-) Regards, Binarus

2 1

Re: [Maria-developers] bf2eaa7: MDEV-9382: After updating mariadb server apt-configure fails
by Sergei Golubchik 17 Mar '16

17 Mar '16

Hi, Nirbhay! On Mar 02, Nirbhay Choubey wrote: > revision-id: bf2eaa7c1548f07ce5de13a406ff9183f082b8b3 (mariadb-10.1.12-4-gbf2eaa7) > parent(s): fd741c1a12e78035dcef9a5684b530fe09cbfdde > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-03-02 09:41:03 -0500 > message: > > MDEV-9382: After updating mariadb server apt-configure fails > > When acting as a Galera receiver node, server startup may take > more than 30 secs (the current default) as it has to wait for > SST/IST operation to complete besides spending some time doing > wsrep recovery. > > In specific case of this MDEV, the server startup timedout as > it took ~48 secs for the node to fully start. > > A viable workaround would be to adjust the MYSQLD_STARTUP_TIMEOUT > env to a higher value, but raising the default to 60 secs should > fix scenarios like these. > > --- > debian/mariadb-server-10.1.mysql.init | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/debian/mariadb-server-10.1.mysql.init b/debian/mariadb-server-10.1.mysql.init > index 9e098b4..005d0e8 100644 > --- a/debian/mariadb-server-10.1.mysql.init > +++ b/debian/mariadb-server-10.1.mysql.init > @@ -109,7 +109,7 @@ case "${1:-''}" in > /usr/bin/mysqld_safe "${@:2}" > /dev/null 2>&1 & > > # 6s was reported in #352070 to be too little > - for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-30}"); do > + for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-60}"); do > sleep 1 This doesn't fix much. Next time somebody will need 61 second and you'll adjust to 120? SST might take many minutes and it's not a good idea to have every user wait this long. Instead you should do what MDEV-8509 says. Add a line like [ -r /etc/default/mariadb ] && . /etc/default/mariadb then one can have a very personal setting of MYSQLD_STARTUP_TIMEOUT in /etc/default/mariadb Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

3 3

[Maria-developers] [GSoC 2016] NO PAD Collations
by Даниил Медведев 16 Mar '16

16 Mar '16

Hi, I am Daniil Medvedev from Lomonosov Moscow State University, a 3rd year student, pursuing my bachelor’s degree in Computer Science. I was interested in project idea NO PAD Collations, and I wish to contribute to MariaDB using my skills and ideas. Technologies that i know: C/C++, git, linux, shell, cmake, some databases. My experience: Some projects on github: https://github.com/medvdanil And I contributed one open source NoSQL DBMS on github last summer. Also I am ready to perform some opening task or do commit in your repo, if it helps me to join GSoC. Thank you, Daniil Medvedev

3 2

[Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob
by Sachin Setia 16 Mar '16

16 Mar '16

Dear Developers, I was doing prototype for this project as Mr Sergei Golubchik suggested "This task is about creating MI_UNIQUEDEF "uniques" instead of MI_KEYDEF "keys" for long unique constraints." Here is my output MariaDB [sachin]> create table tbl(int_key int primary key , blob_key blob unique); Query OK, 0 rows affected (0.03 sec) MariaDB [sachin]> insert into tbl values(1,1); Query OK, 1 row affected (0.01 sec) MariaDB [sachin]> insert into tbl values(2,1); ERROR 1062 (23000): Duplicate entry '1' for key 'blob_key' MariaDB [sachin]> insert into tbl values(2,2); Query OK, 1 row affected (0.00 sec) MariaDB [sachin]> insert into tbl values(3,3); Query OK, 1 row affected (0.00 sec) MariaDB [sachin]> select * from tbl; +---------+----------+ | int_key | blob_key | +---------+----------+ | 1 | 1 | | 2 | 2 | | 3 | 3 | +---------+----------+ 3 rows in set (0.00 sec) I am assuming that unique blob will have 0 zero key . Should i mail you the source code.Please let me know Regards sachin

2 4

[Maria-developers] Introduction - GSOF16
by Chandu Maddila 16 Mar '16

16 Mar '16

Hi Team, I'm Chandra Sekhar Maddila from India and currently masters student of BITS-Pilani. I'm pretty interested in working with MariaDB team as part of GSOF. Some of the items that I'm interested in are MXS-1, MDEV-8100, MDEV-5662, MDEV-371 Here's a quick summary of my work experience and Skill set. - Currently a student of BITS-Pilani <http://www.bits-pilani.ac.in/> (Masters, Computer Science). - Total experience of 6 years in IT industry, as a software developer (My last stint was in *Microsoft Research *[MSR] - India as a research SDE). - Have been working with designing and development of RDBMS systems : With Oracle earlier and SQL Server after joining Microsoft - Involved in various projects like :* Implementing column store indexes in SQL Server, Designing a cloud based monitoring systems for database performance monitoring, tools to migrate data from Oracle to SQL Server, Implementing HA Solution (AlwaysOn) for SQL Server etc...* - Languages/Frameworks: - Asp.Net, C#, Jquery, Javascript, HTML5, CSS, Bootstrap, Knockout, SOAP, REST, JSON, Web Services - WCF, MVVM, XAML, Windows App Development, Azure Cloud services, Azure web services - Java, J2EE, UNIX Shell scripting I worked extensively on both *UNIX and Windows platforms*. *Database domain expertise:* *Oracle* + In depth understanding of Oracle database internals and architecture (Memory management, storage engine and optimizer) + *Performance tuning*: Instance and SQL level tuning with the help of *AWR, Statspack, TKPROF* reports and Explain plans and tools like *OEM* and *TOAD* + Oracle *11g, 10g, 9i* Database Setup, Administration, Monitoring on various flavors of UNIX: RHEL, AIX, SUN Solaris, HPUX – Itanium chipset; Good understanding of Oracle internals and Oracle system architecture + Oracle *RAC, ASM and Grid infrastructure* setup, maintenance and administration; Non RAC to RAC, Non-ASM to ASM conversions; Familiarity with OS level clustering techniques like *HACMP* + All kinds of Database *Backup and Recovery* methodologies: *RMAN *(Hot, Cold, Active Database Duplication)*, Datapump, SQL Loader* etc…; Familiarity with 3rd party backup tools like *CA ArcServe* *SQL Server* + Working experience with SQL 2014, 2012, 2008R2, 2008 and 2005 on different versions of Windows + Very good at *troubleshooting complex issues and working on outages/showstoppers*. + Very good in *database performance and query tuning*: Analyzing *execution plans* and rewriting queries + Administering SQL Server High Availability and Disaster Recovery technologies like *AlwaysOn, Failover Clustering *and *Database mirroring*. + Proficiency and expertise in all varieties of *SQL Server Replication*: *Snapshot Replication, Transactional replication, Merge replication and Change tracking* + Expertise in handling all sorts of *backup technologies: Native, VSS, VDI, Backup to URL etc…* + Expert in handling *database corruption cases and recovering* the databases + Expertise in T-SQL writing stored procedures, triggers and functions. For any queries, feel free to reach out to me @ chandu.maddila(a)gmail.com Thanks, Chandra

2 1

Re: [Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob
by Sergei Golubchik 15 Mar '16

15 Mar '16

Hi, Sachin! First: please send questions like this to maria-developers(a)lists.launchpad.net it's a public mailing list dedicated to MySQL and MariaDB internals, source code, and related things. I am subscribed, so I'll see you mail there, and you may be sure I will, because it won't be accidentally catched by my spam filter, or sorted out in some obscure folder. Furthermore other subscribers will see your question and could reply if I will be not available (e.g. I could be travelling). Thank you. On Mar 14, Sachin Setia wrote: > Thank you sir for your reply but my question is what this pack flag will do > and basically what is pack record will you please suggest source file in > which i can look into to get more understanding of pack length ,pack flag > and pack record There isn't much I can answer - your question is too generic. "pack_flag" is a set of flags that is stored per field in the frm file and it describes some aspects of the field. Search for "pack_flag" in the source tree. If you ask a specific question, I'll be able to provide a more detailed answer. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] a0c06ba: Preliminary implementation for the aggregate sum function as a window function
by Sergey Petrunia 14 Mar '16

14 Mar '16

Hi Vicentiu, I think it add_helper() is a really poor choice of name when the function can do removal. Maybe add_or_remove() name would be better? On Mon, Mar 14, 2016 at 03:44:48PM +0200, Vicentiu Ciorbaru wrote: > revision-id: a0c06ba1edb54c8c4705189c0455137a85658297 (mariadb-10.1.8-154-ga0c06ba) > parent(s): ce8a0d8e19e7bf1486a2c80ff6b7a30ef35bf99f > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2016-03-14 15:42:00 +0200 > message: > > Preliminary implementation for the aggregate sum function as a window function > > This implementation does not deal with the case where removal of > elements from the window frame causes the item to turn to a null value. > > --- > mysql-test/r/win_sum.result | 42 ++++++++++++++++++++++++++++++++++++++++++ > mysql-test/t/win_sum.test | 32 ++++++++++++++++++++++++++++++++ > sql/item_sum.cc | 29 +++++++++++++++++++++++++---- > sql/item_sum.h | 4 ++++ > sql/sql_window.cc | 1 + > 5 files changed, 104 insertions(+), 4 deletions(-) > > diff --git a/mysql-test/r/win_sum.result b/mysql-test/r/win_sum.result > new file mode 100644 > index 0000000..1db6c6e > --- /dev/null > +++ b/mysql-test/r/win_sum.result > @@ -0,0 +1,42 @@ > +create table t1 ( > +pk int primary key, > +a int, > +b int, > +c real > +); > +insert into t1 values > +(101 , 0, 10, 1.1), > +(102 , 0, 10, 2.1), > +(103 , 1, 10, 3.1), > +(104 , 1, 10, 4.1), > +(108 , 2, 10, 5.1), > +(105 , 2, 20, 6.1), > +(106 , 2, 20, 7.1), > +(107 , 2, 20, 8.15), > +(109 , 4, 20, 9.15), > +(110 , 4, 20, 10.15), > +(111 , 5, NULL, 11.15), > +(112 , 5, 1, 12.25), > +(113 , 5, NULL, 13.35), > +(114 , 5, NULL, 14.50), > +(115 , 5, NULL, 15.65); > +select pk, a, b, sum(b) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING), > +sum(c) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) > +from t1; > +pk a b sum(b) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) sum(c) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) > +101 0 10 20 3.2 > +102 0 10 20 3.2 > +103 1 10 20 7.199999999999999 > +104 1 10 20 7.199999999999999 > +105 2 20 40 13.2 > +106 2 20 60 21.35 > +107 2 20 50 20.35 > +108 2 10 30 13.250000000000002 > +109 4 20 40 19.3 > +110 4 20 40 19.3 > +111 5 NULL 1 23.4 > +112 5 1 1 36.75 > +113 5 NULL 1 40.1 > +114 5 NULL 0 43.5 > +115 5 NULL 0 30.15 > +drop table t1; > diff --git a/mysql-test/t/win_sum.test b/mysql-test/t/win_sum.test > new file mode 100644 > index 0000000..3c12b08 > --- /dev/null > +++ b/mysql-test/t/win_sum.test > @@ -0,0 +1,32 @@ > +create table t1 ( > + pk int primary key, > + a int, > + b int, > + c real > +); > + > + > +insert into t1 values > +(101 , 0, 10, 1.1), > +(102 , 0, 10, 2.1), > +(103 , 1, 10, 3.1), > +(104 , 1, 10, 4.1), > +(108 , 2, 10, 5.1), > +(105 , 2, 20, 6.1), > +(106 , 2, 20, 7.1), > +(107 , 2, 20, 8.15), > +(109 , 4, 20, 9.15), > +(110 , 4, 20, 10.15), > +(111 , 5, NULL, 11.15), > +(112 , 5, 1, 12.25), > +(113 , 5, NULL, 13.35), > +(114 , 5, NULL, 14.50), > +(115 , 5, NULL, 15.65); > + > +--sorted_result > +select pk, a, b, sum(b) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING), > + sum(c) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) > + > +from t1; > + > +drop table t1; > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index c78c206..3f4853a 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -1318,25 +1318,39 @@ void Item_sum_sum::fix_length_and_dec() > bool Item_sum_sum::add() > { > DBUG_ENTER("Item_sum_sum::add"); > + add_helper(false); > + DBUG_RETURN(0); > +} > + > +void Item_sum_sum::add_helper(bool perform_removal) > +{ > + DBUG_ENTER("Item_sum_sum::add_helper"); > if (Item_sum_sum::result_type() == DECIMAL_RESULT) > { > my_decimal value; > const my_decimal *val= aggr->arg_val_decimal(&value); > if (!aggr->arg_is_null(true)) > { > - my_decimal_add(E_DEC_FATAL_ERROR, dec_buffs + (curr_dec_buff^1), > - val, dec_buffs + curr_dec_buff); > + if (perform_removal) > + my_decimal_sub(E_DEC_FATAL_ERROR, dec_buffs + (curr_dec_buff ^ 1), > + dec_buffs + curr_dec_buff, val); > + else > + my_decimal_add(E_DEC_FATAL_ERROR, dec_buffs + (curr_dec_buff ^ 1), > + val, dec_buffs + curr_dec_buff); > curr_dec_buff^= 1; > null_value= 0; > } > } > else > { > - sum+= aggr->arg_val_real(); > + if (perform_removal) > + sum-= aggr->arg_val_real(); > + else > + sum+= aggr->arg_val_real(); > if (!aggr->arg_is_null(true)) > null_value= 0; > } > - DBUG_RETURN(0); > + DBUG_VOID_RETURN; > } > > > @@ -1386,6 +1400,13 @@ my_decimal *Item_sum_sum::val_decimal(my_decimal *val) > return val_decimal_from_real(val); > } > > +void Item_sum_sum::remove() > +{ > + DBUG_ENTER("Item_sum_sum::remove"); > + add_helper(true); > + DBUG_VOID_RETURN; > +} > + > /** > Aggregate a distinct row from the distinct hash table. > > diff --git a/sql/item_sum.h b/sql/item_sum.h > index f1f5b22..24dfd9b 100644 > --- a/sql/item_sum.h > +++ b/sql/item_sum.h > @@ -773,6 +773,10 @@ class Item_sum_sum :public Item_sum_num, > return has_with_distinct() ? "sum(distinct " : "sum("; > } > Item *copy_or_same(THD* thd); > + void remove(); > + > +private: > + void add_helper(bool perform_removal); > }; > > > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index df9899e..f0d74b1 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -1584,6 +1584,7 @@ bool JOIN::process_window_functions(List<Item> *curr_fields_list) > } > case Item_sum::COUNT_FUNC: > case Item_sum::SUM_BIT_FUNC: > + case Item_sum::SUM_FUNC: > { > /* > Frame-aware window function computation. It does one pass, but > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob
by Sachin Setia 14 Mar '16

14 Mar '16

Hello Developers My name is sachin , a third year undergrad student in India.I am interested in gsoc 2016 I was debugging the maria db for following query create table c4(a1 int primary key,b1 int unique,c1 blob) engine=myisam; In mysql_prepare_create_table there is a function name prepare_create_field which sets the sql_field pack flag . for a1 field it sets the pack flag =27 . do not quiet get what it means.

2 1

Re: [Maria-developers] c0b1148: MDEV-9659 : AWS KMS encryption plugin
by Sergei Golubchik 12 Mar '16

12 Mar '16

Hi, Wlad! Just a couple of comments: On Mar 11, wlad(a)mariadb.com wrote: > + # Relax AWS C++ SDK unreasonably high requirements for CMake version. Use replace utility (from MariaDB build) > + # to patch their CMakeLists.txt Nice solution. But you might need to add a dependency, to ensure that this external project target is built after replace. Unless cmake can, somehow, derive it automatically (you can test it by building just this specific target in a clean tree). > + MY_DIR *dirp = my_dir(".", MYF(0)); > + if (!dirp) > + { > + sql_print_error("Can't scan current directory"); > + DBUG_RETURN(-1); > + } > + for (unsigned int i=0; i < dirp->number_of_files; i++) > + { > + > + KEY_INFO info; > + if (extract_id_and_version(dirp->dir_entry[i].name, &info.key_id, &info.key_version) == 0) > + { > + key_info_cache[KEY_ID_AND_VERSION(info.key_id, info.key_version)]= info; > + latest_version_cache[info.key_id]= max(info.key_version, latest_version_cache[info.key_id]); > + } > + } > + DBUG_RETURN(0); my_dir() uses malloc(), don't forget my_dirend() to free the memory. Regards, Sergei

2 1

Re: [Maria-developers] 66cd0f2: MDEV-9704: ALTER TABLE does not work from client
by Sergei Golubchik 12 Mar '16

12 Mar '16

Hi, OleksandrByelkin! On Mar 11, OleksandrByelkin wrote: > revision-id: 66cd0f2f02bb45707b38e983186ab8e7aae21da9 (mariadb-10.1.8-156-g66cd0f2) > parent(s): 75d8544c0bc611e82078b9e5f59caef9105e4892 > committer: Oleksandr Byelkin > timestamp: 2016-03-11 16:55:57 +0100 > message: > > MDEV-9704: ALTER TABLE does not work from client > > Do not try to send progress packets if client was not informed about them. > > --- > sql/sql_acl.cc | 2 -- > 1 file changed, 2 deletions(-) > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index cd44673..3ae25d5 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -11900,8 +11900,6 @@ static ulong parse_client_handshake_packet(MPVIO_EXT *mpvio, > (((ulonglong)uint4korr(net->read_pos + 28)) << 32); > client_capabilities|= ext_client_capabilities; > } > - else if (client_capabilities & CLIENT_PROGRESS_OBSOLETE) > - client_capabilities|= MARIADB_CLIENT_PROGRESS; Okay. And I think that now you don't need CLIENT_PROGRESS_OBSOLETE either. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-9659 Encryption plugin that uses AWS Key management service
by Sergei Golubchik 12 Mar '16

12 Mar '16

Hi, Wlad! On Mar 03, wlad(a)mariadb.com wrote: > revision-id: c1ed3c0267da2ece7664e9f876314979114b2186 (mariadb-10.1.11-26-gc1ed3c0) > parent(s): a62db8186e40a62366bc86f49a0195e9b687d86d > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2016-03-03 21:29:29 +0100 > message: > > MDEV-9659 Encryption plugin that uses AWS Key management service Looks pretty good, thanks! See comment below... > diff --git a/cmake/plugin.cmake b/cmake/plugin.cmake > index c24239c..361cb59 100644 > --- a/cmake/plugin.cmake > +++ b/cmake/plugin.cmake > @@ -74,7 +74,9 @@ MACRO(MYSQL_ADD_PLUGIN) > SET(compat "with${compat}") > ENDIF() > > - IF (compat STREQUAL ".") > + IF (ARG_DISABLED) > + SET(howtobuild NO) > + ELSEIF (compat STREQUAL ".") > SET(howtobuild DYNAMIC) > ELSEIF (compat STREQUAL "with.") > IF (NOT ARG_MODULE_ONLY) > @@ -122,7 +124,7 @@ MACRO(MYSQL_ADD_PLUGIN) > > # Build either static library or module > IF (PLUGIN_${plugin} MATCHES "(STATIC|AUTO|YES)" AND NOT ARG_MODULE_ONLY > - AND NOT ARG_DISABLED AND NOT ARG_CLIENT) > + AND NOT ARG_CLIENT) > > IF(CMAKE_GENERATOR MATCHES "Makefiles|Ninja") > # If there is a shared library from previous shared build, > @@ -178,8 +180,7 @@ MACRO(MYSQL_ADD_PLUGIN) > SET (mysql_optional_plugins ${mysql_optional_plugins} PARENT_SCOPE) > ENDIF() > ELSEIF(PLUGIN_${plugin} MATCHES "(DYNAMIC|AUTO|YES)" > - AND NOT ARG_STATIC_ONLY AND NOT WITHOUT_DYNAMIC_PLUGINS > - AND NOT ARG_DISABLED) > + AND NOT ARG_STATIC_ONLY AND NOT WITHOUT_DYNAMIC_PLUGINS) > > ADD_VERSION_INFO(${target} MODULE SOURCES) > ADD_LIBRARY(${target} MODULE ${SOURCES}) a bit more of the explanation would've been nice. in the changeset comment or - even better - pushing this change separately. you can select what files to include in the commit, you know... > diff --git a/plugin/aws_key_management/CMakeLists.txt b/plugin/aws_key_management/CMakeLists.txt > new file mode 100644 > index 0000000..ec9ee46 > --- /dev/null > +++ b/plugin/aws_key_management/CMakeLists.txt > @@ -0,0 +1,144 @@ > +# We build parts of AWS C++ SDK as CMake external project > +# The restrictions of the SDK (https://github.com/awslabs/aws-sdk-cpp/blob/master/README.md) > +# are > + > +# - OS : Windows,Linux or OSX > +# - C++11 compiler : VS2013+, gcc 4.7+, clang 3.3+ > +# - libcurl development package needs to be present on Unixes > +# > +# Since we're using ExternalProject that reads from git repository > +# we also require GIT to be present on the build machine > + > + > +# Give message why the building this plugin is skipped (only if -DVERBOSE is defined) > +# and and bail out. > +MACRO(SKIP_AWS_PLUGIN msg) > + IF(DEFINED VERBOSE) VERBOSE - is it used somewhere else? Or you've invented it on the spot for this plugin? I understand that it can be used in other cmake files, but is it? > + MESSAGE(STATUS "Skip aws_key_management - ${msg}") > + RETURN() > + ENDIF() > +ENDMACRO() Ok, as discussed on irc, I'd suggest to add here a check whether AWS SDK is already installed. Like, CheckLibraryExists or CheckSymbolExists. And if not - then try to download and compile it (which would require new cmake, git, curl, etc). > +IF(CMAKE_VERSION VERSION_LESS "2.8.8") > + SKIP_AWS_PLUGIN("CMake is too old") > +ENDIF() > + > +FIND_PACKAGE(Git) > +IF(NOT GIT_FOUND) > + SKIP_AWS_PLUGIN("no GIT") > +ENDIF() > + > +INCLUDE(ExternalProject) > +IF (NOT(WIN32 OR APPLE OR (CMAKE_SYSTEM_NAME MATCHES "Linux"))) > + SKIP_AWS_PLUGIN("OS unsupported by AWS SDK") > +ENDIF() > + > +IF(UNIX) > + FIND_PACKAGE(CURL) > + IF(NOT CURL_FOUND) > + SKIP_AWS_PLUGIN("AWS C++ SDK requires libcurl development package") > + ENDIF() > + SET(PIC_FLAG -fPIC) > +ENDIF() > + > +SET(CXX11_FLAGS) > +SET(OLD_COMPILER_MSG "AWS SDK requires c++11 -capable compiler (minimal supported versions are g++ 4.7, clang 3.3, VS2103)") > +IF(CMAKE_CXX_COMPILER_ID MATCHES "GNU") > + EXECUTE_PROCESS(COMMAND ${CMAKE_CXX_COMPILER} -dumpversion OUTPUT_VARIABLE GCC_VERSION) eh? I thought cmake knows compiler versions... let me see This one: CMAKE_<LANG>_COMPILER_VERSION > + IF (GCC_VERSION VERSION_LESS 4.7) > + SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") > + ENDIF() > + SET(CXX11_FLAGS "-std=c++11") Instead of checking the compiler version, I would rather suggest to check for a feature that you need to use. Like MY_CHECK_CXX_COMPILER_FLAG(-std=c++11) IF (NOT HAVE_CXX__STD_C__11) SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") ENDIF() Something like that. Not sure about HAVE_CXX__STD_C__11 name, though. > +ELSEIF (CMAKE_CXX_COMPILER_ID MATCHES "Clang") > + IF (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 3.3) > + SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") > + ENDIF() > + SET(CXX11_FLAGS "-stdlib=libc++") Same. > +ELSEIF(MSVC) > + IF (MSVC_VERSION LESS 1800) > + SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") > + ENDIF() > +ELSE() > + SKIP_AWS_PLUGIN("Compiler not supported by AWS C++ SDK") > +ENDIF() > + > +IF(WIN32) > + SET(EXTRA_SDK_CMAKE_FLAGS -DCMAKE_CXX_FLAGS_DEBUGOPT="" -DCMAKE_EXE_LINKER_FLAGS_DEBUGOPT="") For consistency you can put this inside ELSEIF(MSVC), that's where you set options for gcc and clang. > +ENDIF() > + > +FIND_PROGRAM(PERL_EXECUTABLE perl) > + > +IF(PERL_EXECUTABLE) > + # CMake minimimal version for building AWS SDK can be less than 3.1.2 > + # Downgrade this requirement. > + SET(AWS_SDK_PATCH_COMMAND > + PATCH_COMMAND ${PERL_EXECUTABLE} -i.bak -pe s/3.1.2/2.8/g ${CMAKE_BINARY_DIR}/aws-sdk-cpp/CMakeLists.txt) 1. Without perl it'll require 3.1.2, right? 2. Couldn't you do that with cmake alone? 3. Is that safe at all - you pull the latest version from git, they can start using cmake-3.1.2 features any time. > +ENDIF() > + > +ExternalProject_Add( > + aws_sdk_cpp > + GIT_REPOSITORY "https://github.com/awslabs/aws-sdk-cpp.git" > + UPDATE_COMMAND "" > + ${AWS_SDK_PATCH_COMMAND} > + SOURCE_DIR "${CMAKE_BINARY_DIR}/aws-sdk-cpp" > + CMAKE_ARGS > + -DBUILD_ONLY=aws-cpp-sdk-kms -DSTATIC_LINKING=1 > + "-DCMAKE_CXX_FLAGS_DEBUG=${CMAKE_CXX_FLAGS_DEBUG} ${PIC_FLAG}" > + "-DCMAKE_CXX_FLAGS_RELWITHDEBINFO=${CMAKE_CXX_FLAGS_RELWITHDEBINFO} ${PIC_FLAG}" > + "-DCMAKE_CXX_FLAGS_RELEASE=${CMAKE_CXX_FLAGS_RELEASE} ${PIC_FLAG}" > + "-DCMAKE_CXX_FLAGS_MINSIZEREL=${CMAKE_CXX_FLAGS_MINSIZEREL} ${PIC_FLAG}" > + ${EXTRA_SDK_CMAKE_FLAGS} > + -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR}/aws_sdk_cpp > + TEST_COMMAND "" > +) > + > +# Figure out where AWS installs SDK libraries > +# The below is defined in AWS SDK's CMakeLists.txt > +# (and their handling is weird, every OS has special install directory) > +if(WIN32) > + SET(SDK_INSTALL_BINARY_PREFIX "windows") > +ELSEIF(APPLE) > + SET(SDK_INSTALL_BINARY_PREFIX "mac") > +ELSEIF(UNIX) > + SET(SDK_INSTALL_BINARY_PREFIX "linux") > +ENDIF() > +IF(NOT APPLE) > + IF(CMAKE_SIZEOF_VOID_P EQUAL 8) > + SET(SDK_INSTALL_BINARY_PREFIX "${SDK_INSTALL_BINARY_PREFIX}/intel64") > + ELSE() > + SET(SDK_INSTALL_BINARY_PREFIX "${SDK_INSTALL_BINARY_PREFIX}/ia32") > + ENDIF() > +ENDIF() > +IF(CMAKE_CONFIGURATION_TYPES) > + SET(SDK_INSTALL_BINARY_PREFIX "${SDK_INSTALL_BINARY_PREFIX}/${CMAKE_CFG_INTDIR}") > +ENDIF() > + > +# We do not need to build the whole SDK , just 2 of its libs > +set(AWS_SDK_LIBS aws-cpp-sdk-core aws-cpp-sdk-kms) > + > +FOREACH(lib ${AWS_SDK_LIBS}) > + ADD_LIBRARY(${lib} STATIC IMPORTED) > + ADD_DEPENDENCIES(${lib} aws_sdk_cpp) > + SET(loc "${CMAKE_BINARY_DIR}/aws_sdk_cpp/lib/${SDK_INSTALL_BINARY_PREFIX}/${CMAKE_STATIC_LIBRARY_PREFIX}${lib}${CMAKE_STATIC_LIBRARY_SUFFIX}") > + SET_TARGET_PROPERTIES(${lib} PROPERTIES IMPORTED_LOCATION ${loc}) > + IF(WIN32) > + SET_TARGET_PROPERTIES(${lib} PROPERTIES IMPORTED_LINK_INTERFACE_LIBRARIES "bcrypt;winhttp;wininet;userenv") > + ELSE() > + SET_TARGET_PROPERTIES(${lib} PROPERTIES IMPORTED_LINK_INTERFACE_LIBRARIES "${SSL_LIBRARIES};${CURL_LIBRARIES}") > + ENDIF() > +ENDFOREACH() > + > +IF(CMAKE_SYSTEM_NAME MATCHES "Linux") > + # Need whole-archive , otherwise static libraries are not linked > + SET(AWS_SDK_LIBS -Wl,--whole-archive ${AWS_SDK_LIBS} -Wl,--no-whole-archive) > +ENDIF() > + > +SET_TARGET_PROPERTIES(aws_sdk_cpp PROPERTIES EXCLUDE_FROM_ALL TRUE) > +INCLUDE_DIRECTORIES(${CMAKE_BINARY_DIR}/aws_sdk_cpp/include) > +SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${CXX11_FLAGS}") > +MYSQL_ADD_PLUGIN(aws_key_management aws_key_management_plugin.cc > + MODULE_ONLY COMPONENT aws-key-management > + LINK_LIBRARIES ${AWS_SDK_LIBS} DISABLED) > diff --git a/plugin/aws_key_management/aws_key_management_plugin.cc b/plugin/aws_key_management/aws_key_management_plugin.cc > new file mode 100644 > index 0000000..fa7d042 > --- /dev/null > +++ b/plugin/aws_key_management/aws_key_management_plugin.cc > @@ -0,0 +1,611 @@ > +/* > + Copyright (c) 2016 MariaDB Corporation > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > + > +#include <my_global.h> > +#include <my_pthread.h> > +#include <my_sys.h> > +#include <mysql/plugin_encryption.h> > +#include <my_crypt.h> > +#include <string.h> > +#include <stdio.h> > +#include <stdlib.h> > +#include <mysqld_error.h> > +#include <base64.h> > +#include <map> > +#include <algorithm> > +#include <string> > +#include <vector> > +#include <iterator> > +#include <sstream> > +#include <fstream> > +#ifndef _WIN32 > +#include <sys/types.h> > +#include <dirent.h> > +#endif Use cmake checks when there are some, HAVE_DIRENT_H, HAVE_SYS_TYPES_H, HAVE_STRING_H, HAVE_STDLIB_H. Hmm, HAVE_STDLIB_H doesn't make sense to me :) could you remove it from configure.cmake, please? In a separate changeset. May be HAVE_SYS_TYPES_H too, not sure. But do use HAVE_DIRENT_H and HAVE_STDLIB_H above instead of _WIN32. > +#include <aws/core/client/AWSError.h> > +#include <aws/core/utils/logging/AWSLogging.h> > +#include <aws/core/utils/logging/ConsoleLogSystem.h> > +#include <aws/kms/KMSClient.h> > +#include <aws/kms/model/DecryptRequest.h> > +#include <aws/kms/model/DecryptResult.h> > +#include <aws/kms/model/GenerateDataKeyWithoutPlaintextRequest.h> > +#include <aws/kms/model/GenerateDataKeyWithoutPlaintextResult.h> > +#include <aws/core/utils/Outcome.h> > +using namespace std; > +using namespace Aws::KMS; > +using namespace Aws::KMS::Model; > +using namespace Aws::Utils::Logging; > +extern void sql_print_error(const char *format, ...); > +extern void sql_print_warning(const char *format, ...); > +extern void sql_print_information(const char *format, ...); > + > + > +/* Plaintext key info struct */ > +struct KEY_INFO > +{ > + unsigned int key_id; > + unsigned int key_version; > + unsigned int length; > + unsigned char data[MY_AES_MAX_KEY_LENGTH]; > + bool load_failed; /* if true, do not attempt to reload?*/ > +public: > + KEY_INFO() : key_id(0), key_version(0), length(0), load_failed(false){}; > +}; > +#define KEY_ID_AND_VERSION(key_id,version) ((longlong)key_id << 32 | version) > + > +/* Cache for the latest version, per key id */ > +static std::map<uint, uint> latest_version_cache; > + > +/* Cache for plaintext keys */ > +static std::map<ulonglong, KEY_INFO> key_info_cache; > + > +static const char *key_spec_names[]={ "AES_128", "AES_256", 0 }; > + > +/* Plugin variables */ > +static char* master_key_id; > +static unsigned long key_spec; > +static unsigned long log_level; > +static int rotate_key; > + > +/* AWS functionality*/ > +static int aws_decrypt_key(const char *path, KEY_INFO *info); > +static int aws_generate_datakey(uint key_id, uint version); > + > +static int extract_id_and_version(const char *name, uint *id, uint *ver); > +static unsigned int get_latest_key_version(unsigned int key_id); > +static unsigned int get_latest_key_version_nolock(unsigned int key_id); > +static int load_key(KEY_INFO *info); > + > +/* Mutex to serialize access to caches */ > +static pthread_mutex_t mtx; Use PSI, please. That's mysql_mutex_t, etc. > + > +static Aws::KMS::KMSClient *client; > + > +/* Redirect AWS trace to error log */ > +class MySQLLogSystem : public Aws::Utils::Logging::FormattedLogSystem > +{ > +public: > + > + using Base = FormattedLogSystem; > + MySQLLogSystem(LogLevel logLevel) : > + Base(logLevel) > + { > + } > + virtual LogLevel GetLogLevel(void) const override > + { > + return (LogLevel)log_level; > + } > + virtual ~MySQLLogSystem() > + { > + } > + > +protected: > + virtual void ProcessFormattedStatement(Aws::String&& statement) override > + { > +#ifdef _WIN32 > + /* > + On Windows, we can't use C runtime functions to write to stdout, > + because we compile with static C runtime, so plugin has a stdout > + different from server. Thus we're using WriteFile(). > + */ > + DWORD nSize= (DWORD)statement.size(); > + DWORD nWritten; > + const char *s= statement.c_str(); > + HANDLE h= GetStdHandle(STD_OUTPUT_HANDLE); > + > + WriteFile(h, s, nSize, &nWritten, NULL); > +#else > + printf("%s", statement.c_str()); printf? You have declared sql_print_information above, why wouldn't you use it here? > +#endif > + } > +}; > + > +/* > + Read filenames from the current directory. > + This is used in plugin initilization. We check for > + plugin specific names aws-kms-key.<id>.<version> > + to find out what keys/key versions are already there. > +*/ > +static vector<string> read_dir() > +{ > + vector<string> v; > + DBUG_ENTER("read_dir"); > +#ifdef _WIN32 > + WIN32_FIND_DATA ffd; > + HANDLE h= FindFirstFile("*", &ffd); > + if (h == INVALID_HANDLE_VALUE) > + { > + DBUG_RETURN(v); > + } > + do > + { > + /* handle file possibly updates caches*/ > + v.push_back(ffd.cFileName); > + } while (FindNextFile(h, &ffd)); > + FindClose(h); > +#else > + DIR *dir= opendir("."); > + if (!dir) > + { > + DBUG_RETURN(v); > + } > + struct dirent *dp; > + while ((dp= readdir(dir)) != NULL) > + { > + v.push_back(dp->d_name); > + } > + closedir(dir); > +#endif > + DBUG_RETURN(v); ok. although you could've used my_dir and save yourself some troubles :) > +} > + > + > +/* > + Plugin initialization. > + > + Create KMS client and scan datadir to find out which keys and versions > + are present. > +*/ > +static int plugin_init(void *p) > +{ > + DBUG_ENTER("plugin_init"); > + client = new KMSClient(); > + if (!client) > + { > + sql_print_error("Can not initialize KMS client"); > + DBUG_RETURN(-1); > + } > + InitializeAWSLogging(Aws::MakeShared<MySQLLogSystem>("aws_key_management_plugin", (Aws::Utils::Logging::LogLevel) log_level)); > + pthread_mutex_init(&mtx, NULL); > + vector<string> filenames= read_dir(); > + for (size_t i= 0; i < filenames.size(); i++) > + { > + KEY_INFO info; > + if (extract_id_and_version(filenames[i].c_str(), &info.key_id, &info.key_version) == 0) > + { > + key_info_cache[KEY_ID_AND_VERSION(info.key_id, info.key_version)]= info; > + latest_version_cache[info.key_id]= max(info.key_version, latest_version_cache[info.key_id]); > + } > + } > + DBUG_RETURN(0); > +} > + > + > + > +static int plugin_deinit(void *p) > +{ > + DBUG_ENTER("plugin_deinit"); > + latest_version_cache.clear(); > + key_info_cache.clear(); > + pthread_mutex_destroy(&mtx); > + delete client; > + ShutdownAWSLogging(); > + DBUG_RETURN(0); > +} > + > +/* Generate filename to store the ciphered key */ > +static void format_keyfile_name(char *buf, size_t size, uint key_id, uint version) > +{ > + snprintf(buf, size, "aws-kms-key.%u.%u", key_id, version); > +} > + > +/* Extract key id and version from file name */ > +static int extract_id_and_version(const char *name, uint *id, uint *ver) > +{ > + int len; > + int n= sscanf(name, "aws-kms-key.%u.%u%n", id, ver, &len); > + if (n == 2 && *id > 0 && *ver > 0 && len == (int)strlen(name)) > + return 0; > + return 1; > +} > + > +/* > + Decrypt key stored in aws-kms-key.<id>.<version> > + Cache the decrypted key. > +*/ > +static int load_key(KEY_INFO *info) > +{ > + int ret; > + char path[256]; > + DBUG_ENTER("load_key"); > + DBUG_PRINT("enter", ("id=%u,ver=%u", info->key_id, info->key_version)); > + format_keyfile_name(path, sizeof(path), info->key_id, info->key_version); > + ret= aws_decrypt_key(path, info); > + if (ret) > + info->load_failed= true; > + > + latest_version_cache[info->key_id]= max(latest_version_cache[info->key_id], info->key_version); > + key_info_cache[KEY_ID_AND_VERSION(info->key_id, info->key_version)]= *info; > + > + if (!ret) > + { > + sql_print_information("AWS KMS plugin: loaded key %u, version %u, key length %u bit", > + info->key_id, info->key_version,(uint)info->length*8); > + } > + else > + { > + sql_print_warning("AWS KMS plugin: key %u, version %u could not be decrypted", > + info->key_id, info->key_version); > + } > + DBUG_RETURN(ret); > +} > + > + > +/* > + Get latest version for the key. > + > + If key is not decrypted yet, this function also decrypt the key > + and error will be returned if decryption fails. > + > + The reason for that is that InnoDB crashes > + in case errors are returned by get_key(), Shall we rather fix that in InnoDB? Do you remember where it crashes? > + > + A new key will be created if it does not exist, provided there is > + valid master_key_id. > +*/ > +static unsigned int get_latest_key_version(unsigned int key_id) > +{ > + unsigned int ret; > + DBUG_ENTER("get_latest_key_version"); > + > + pthread_mutex_lock(&mtx); > + ret= get_latest_key_version_nolock(key_id); why do you need get_latest_key_version_nolock if you never use it directly? > + pthread_mutex_unlock(&mtx); > + > + DBUG_PRINT("info", ("key=%u,ret=%u", key_id, ret)); > + DBUG_RETURN(ret); > +} > + > +static unsigned int get_latest_key_version_nolock(unsigned int key_id) > +{ > + KEY_INFO info; > + uint ver; > + DBUG_ENTER("get_latest_key_version_nolock"); > + ver= latest_version_cache[key_id]; > + if (ver > 0) > + { > + info= key_info_cache[KEY_ID_AND_VERSION(key_id, ver)]; > + } > + if (info.load_failed) > + { > + /* Decryption failed previously, don't retry */ > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + } > + else if (ver > 0) > + { > + /* Key encrypted already, return it*/ may be s/encrypted already/exists/ ? > + if (info.length > 0) > + DBUG_RETURN(ver); > + } > + else // (ver == 0) > + { > + /* Generate a new key, version 1 */ > + if (!master_key_id[0]) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Can't generate key %u, because aws_key_management_master_key_id parameter is not set", > + MYF(ME_JUST_WARNING), key_id); > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + } > + if (aws_generate_datakey(key_id, 1) != 0) > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + info.key_id= key_id; > + info.key_version= 1; > + info.length= 0; > + } > + > + if (load_key(&info)) > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + DBUG_RETURN(info.key_version); > +} > + > + > +/* > + Decrypt a file with KMS > +*/ > +static int aws_decrypt_key(const char *path, KEY_INFO *info) > +{ > + DBUG_ENTER("aws_decrypt_key"); > + > + /* Read file content into memory */ > + ifstream ifs(path, ios::binary | ios::ate); > + if (!ifs.good()) > + { > + sql_print_error("can't open file %s", path); > + } > + size_t pos = (size_t)ifs.tellg(); > + std::vector<char> contents(pos); > + ifs.seekg(0, ios::beg); > + ifs.read(&contents[0], pos); > + > + /* Decrypt data the with AWS */ > + DecryptRequest request; > + Aws::Utils::ByteBuffer byteBuffer((unsigned char *)contents.data(), pos); > + request.SetCiphertextBlob(byteBuffer); > + DecryptOutcome outcome = client->Decrypt(request); > + if (!outcome.IsSuccess()) > + { > + sql_print_error("AWS KMS plugin: Decrypt failed for %s : %s", path, > + outcome.GetError().GetMessage().c_str()); > + DBUG_RETURN(-1); > + } > + Aws::Utils::ByteBuffer plaintext = outcome.GetResult().GetPlaintext(); > + size_t len = plaintext.GetLength(); > + > + if (len > (int)sizeof(info->data)) > + { > + sql_print_error("AWS KMS plugin: encoding key too large for %s", path); > + DBUG_RETURN(ENCRYPTION_KEY_BUFFER_TOO_SMALL); > + } > + memcpy(info->data, plaintext.GetUnderlyingData(), len); > + info->length= len; > + DBUG_RETURN(0); > +} > + > + > +/* Generate a new datakey and store it a file */ > +static int aws_generate_datakey(uint keyid, uint version) > +{ > + > + DBUG_ENTER("aws_generate_datakey"); > + GenerateDataKeyWithoutPlaintextRequest request; > + request.SetKeyId(master_key_id); > + request.SetKeySpec(DataKeySpecMapper::GetDataKeySpecForName(key_spec_names[key_spec])); > + > + GenerateDataKeyWithoutPlaintextOutcome outcome; > + outcome= client->GenerateDataKeyWithoutPlaintext(request); > + if (!outcome.IsSuccess()) > + { > + sql_print_error("AWS KMS plugin : GenerateDataKeyWithoutPlaintext failed : %s - %s", > + outcome.GetError().GetExceptionName().c_str(), > + outcome.GetError().GetMessage().c_str()); > + DBUG_RETURN(-1); > + } > + > + string out; > + char filename[20]; > + Aws::Utils::ByteBuffer byteBuffer = outcome.GetResult().GetCiphertextBlob(); > + > + format_keyfile_name(filename, sizeof(filename), keyid, version); > + int fd= my_open(filename, O_RDWR | O_CREAT, 0); > + if (fd < 0) > + { > + sql_print_error("AWS KMS plugin: Can't create file %s", filename); > + DBUG_RETURN(-1); > + } > + size_t len= byteBuffer.GetLength(); > + if (my_write(fd, byteBuffer.GetUnderlyingData(), len, 0) != len) > + { > + sql_print_error("AWS KMS plugin: can't write to %s", filename); > + my_close(fd, 0); > + my_delete(filename, 0); > + DBUG_RETURN(-1); > + } > + my_close(fd, 0); > + sql_print_information("AWS KMS plugin: generated encrypted datakey for key id=%u, version=%u", > + keyid, version); > + DBUG_RETURN(0); > +} > + > +/* Key rotation for a single key */ > +static int rotate_single_key(uint key_id) > +{ > + uint ver; > + ver= latest_version_cache[key_id]; > + > + if (!ver) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "key %u does not exist", MYF(ME_JUST_WARNING), key_id); > + return -1; > + } > + else if (aws_generate_datakey(key_id, ver + 1)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "Could not generate datakey for key id= %u, ver= %u", > + MYF(ME_JUST_WARNING), key_id, ver); > + return -1; > + } > + else > + { > + KEY_INFO info; > + info.key_id= key_id; > + info.key_version = ver + 1; > + if (load_key(&info)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "Could not load datakey for key id= %u, ver= %u", > + MYF(ME_JUST_WARNING), key_id, ver); > + return -1; > + } > + } > + return 0; > +} > + > +/* Key rotation for all key ids */ > +static int rotate_all_keys() > +{ > + int ret= 0; > + for (map<uint, uint>::iterator it= latest_version_cache.begin(); it != latest_version_cache.end(); it++) > + { > + ret= rotate_single_key(it->first); > + if (ret) > + break; > + } > + return ret; > +} > + > +static void update_rotate(MYSQL_THD, struct st_mysql_sys_var *, void *, const void *val) > +{ > + if (!master_key_id[0]) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "aws_key_management_master_key_id must be set to generate new data keys", MYF(ME_JUST_WARNING)); > + return; > + } > + pthread_mutex_lock(&mtx); > + rotate_key= *(int *)val; > + switch (rotate_key) > + { > + case 0: > + break; > + case -1: > + rotate_all_keys(); > + break; > + default: > + rotate_single_key(rotate_key); > + break; > + } > + rotate_key= 0; > + pthread_mutex_unlock(&mtx); > +} > + > +static unsigned int get_key( > + unsigned int key_id, > + unsigned int version, > + unsigned char* dstbuf, > + unsigned int* buflen) > +{ > + KEY_INFO info; > + > + DBUG_ENTER("get_key"); > + pthread_mutex_lock(&mtx); > + info= key_info_cache[KEY_ID_AND_VERSION(key_id, version)]; > + if (info.length == 0 && !info.load_failed) > + { > + info.key_id= key_id; > + info.key_version= version; > + load_key(&info); > + } > + pthread_mutex_unlock(&mtx); > + > + if (info.load_failed) > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + if (*buflen < info.length) > + { > + *buflen= info.length; > + DBUG_RETURN(ENCRYPTION_KEY_BUFFER_TOO_SMALL); > + } > + *buflen= info.length; > + memcpy(dstbuf, info.data, info.length); > + DBUG_RETURN(0); > +} > + > + > +/* Plugin defs */ > +struct st_mariadb_encryption aws_key_management_plugin= { > + MariaDB_ENCRYPTION_INTERFACE_VERSION, > + get_latest_key_version, > + get_key, > + // use default encrypt/decrypt functions > + 0, 0, 0, 0, 0 > +}; > + > + > +static TYPELIB key_spec_typelib = > +{ > + array_elements(key_spec_names) - 1, "", > + key_spec_names, NULL > +}; > + > +const char *log_level_names[] = > +{ > + "Off", > + "Fatal", > + "Error", > + "Warn", > + "Info", > + "Debug", > + "Trace", > + 0 > +}; > + > +static TYPELIB log_level_typelib = > +{ > + array_elements(log_level_names) - 1, "", > + log_level_names, NULL > +}; > + > +static MYSQL_SYSVAR_STR(master_key_id, master_key_id, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_MEMALLOC, > + "Key id for master encryption key. Used to create new datakeys. If not set, no new keys will be created", > + NULL, NULL, ""); > + > +static MYSQL_SYSVAR_ENUM(key_spec, key_spec, > + PLUGIN_VAR_RQCMDARG, > + "Encryption algorithm used to create new keys.", > + NULL, NULL, 0, &key_spec_typelib); > + > + > +static MYSQL_SYSVAR_ENUM(log_level, log_level, > + PLUGIN_VAR_RQCMDARG, > + "Logging for AWS API", > + NULL, NULL, 0, &log_level_typelib); > + > + > +static MYSQL_SYSVAR_INT(rotate_key, rotate_key, > + PLUGIN_VAR_RQCMDARG, > + "Set this variable to key id to perform rotation of the key. Specify -1 to rotate all keys", > + NULL, update_rotate, 0, -1, INT_MAX, 1); > + > +static struct st_mysql_sys_var* settings[]= { > + MYSQL_SYSVAR(master_key_id), > + MYSQL_SYSVAR(key_spec), > + MYSQL_SYSVAR(rotate_key), > + MYSQL_SYSVAR(log_level), > + NULL > +}; > + > +/* > + Plugin library descriptor > +*/ > +maria_declare_plugin(aws_key_management) > +{ > + MariaDB_ENCRYPTION_PLUGIN, > + &aws_key_management_plugin, > + "aws_key_management", > + "MariaDB Corporation", > + "AWS key management plugin", > + PLUGIN_LICENSE_GPL, > + plugin_init, > + plugin_deinit, > + 0x0100, > + NULL, > + settings, > + "1.0", > + MariaDB_PLUGIN_MATURITY_EXPERIMENTAL > +} > +maria_declare_plugin_end; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] GSoC 2016 MHA GTID support.
by Charles Dirk 11 Mar '16

11 Mar '16

Hi, I am a computer science student at the university of Cape Town and wish to participate in GSoC 2016 with MariaDB Foundation. I am interested if working on making MHA understand GTIDs. I am proficient in C/C+, Python/Perl and Im equally familiar with replication. I look forward to hearing from mentors soon. Cheers, Charles.

3 3

[Maria-developers] MDEV-9709 Unexpected modification of value and warning about out of range value upon ALTER
by Alexander Barkov 11 Mar '16

11 Mar '16

Hi Elena, can you please review a patch for MDEV-9709? I'm quite sure about the code change, it's trivial. Please have a look into the tests. Thanks.

1 0

[Maria-developers] GSoC 2016 Maxscale Filter
by Lisa Brinson 08 Mar '16

08 Mar '16

Hi everyone, My name is Lisa Brinson and I am a junior Computer science student at Oregon State University. I wish to participate in an open source project through GSoC 2016. I am proficient in C and C++ as well as git. I have looked through the proposed projects and I have developed interest in the Maxscale tasks especially the filters. I have been going through the filter API and have also looked at some sample filters in server/modules/filter directory. I will try to write my first filter(a hello world filter) to get familiar with the API. I will be submitting 3 proposals for the 3 Maxscale filters hoping one will be selected. Best regards, Lisa.

3 4

[Maria-developers] Questions regarding GSoC 2016 and Import / Export
by Dan Ungureanu 08 Mar '16

08 Mar '16

Hello, I am a second year student currently attending University Politehnica of Bucharest and I am interested in contributing to MariaDB during this year's Google Summer of Code. I have looked at the ideas you published in the Knowledge Base [1] and two projects caught my attention. The first one is related to the import and export process related to dynamic columns [2] and the second one is related to the GIS enhancements [3]. I am willing to implement both of them, but I am not sure whether the time will allow me properly implement both of them. I cannot estimate how much effort do these projects require as I am still trying to become acquainted with the code base. I hope I am not underestimating the projects and some guidance from one of the mentors (Oleksandr Byelkin, Georg Richter or Alexey Botchkov - Holyfoot) is certainly welcome. I will focus on "Import and export popular data formats from and to dynamic columns" and as soon as I am done with this one, hopefully I will be able to work on GIS enhancements as well. Before continuing, I would like to clarify some aspects related to these projects. Import and export popular data formats from and to dynamic columns: - Can be split in four main tasks: a) adding support for arrays (vectors) in dynamic columns. Currently, dynamic columns only store dictionaries (maps); b) implement a library that can import (parse) and export (build in a specific format) data to and from dynamic columns; c) provide interface to access to the new import and export mechanism; d) provide unit tests for the new features. - The second part (b) of the project is mostly standalone and requires little edits to the project and the third part (c) will be mostly shared by the server and the C connector. - The outcome of (b) is going to be a minimal parsing and building library for various formats. - Some of the formats that I plan to support are: JSON, XML, CSV (maybe XLS too?), PHP Array, YAML, SQL (export as INSERT INTO statement?), etc. One of the key aspects I have to keep in mind when designing the library is to provide a extensible interface to quickly implement new formats in the future. - Probably, most of my work will be located in: - ./server/sql/item_*.(cc|h) - implement user defined functions - ./server/mysys/ma_dyncol.c - provide interface for user defined functions - ./server/include/ma_dyncol.h - ./mariadb-connector-c/libmariadb/mariadb_dyncol.c - provide interface in C Connector - ./mariadb-connector-c/include/mariadb_dyncol.h GIS enhancements: - This project is based on extending the current functionality related to GIS. I am not very familiar with GIS and I still have to do more research based on documents [4], [5] and [6], but the biggest part of the project consists in defining user functions for handling GIS. - I found this compliance test suite [7] for PostGIS that may come in handy. Also, I have a few other questions related to how should I proceed: - I have already read the introductory developers guide [8], the contributing guidelines [9] and will submit the MariaDB Contributor Agreement [10] as soon as possible. Is there anything else I should do before getting started? - I found no template for Google Summer of Code applications. Is there anything in particular that mentors would like to see? I was thinking of organizing my application in multiple sections, some of the most important being: Synopsis, Benefits to the Users, Project Details, Deliverables, Project Schedule and my resume. I believe that I will receive more feedback related after I will have submitted my application. - Most issues already have an assignee. Is it fine if I try and send pull requests for some of those bugs? I could not decide between whether I should send this mail to discuss or developers mailing list. Hopefully, I chose the right one. I am sorry for my long email and thank you for attention. I am looking for feedback, so please do not hesitate to send me any suggestion you might have. [1] https://mariadb.com/kb/en/mariadb/google-summer-of-code-2016/ [2] https://jira.mariadb.org/browse/CONC-125 [3] https://jira.mariadb.org/browse/MDEV-5813 [4] http://www.opengeospatial.org/standards/sfs [5] http://portal.opengeospatial.org/files/?artifact_id=25354 [6] http://www.opengeospatial.org/compliance/ [7] https://svn.osgeo.org/postgis/trunk/extras/ogc_test_suite/README [8] https://mariadb.org/get-involved/getting-started-for-developers/ [9] https://mariadb.com/kb/en/mariadb/contributing-code/ [10] https://mariadb.com/kb/en/mariadb/mca/ Best regards, Dan Ungureanu

2 1

[Maria-developers] GSoC 2016 Maxscale Luafilter
by Pedro Moratta 08 Mar '16

08 Mar '16

Hi, My name in Pedro Mora and I am a MS Computer Science student at Milano Institute of technology. I wish to work on Maxscale filters during GSoC 2016. I am interested in working on the Lua filter because I am familiar with Lua scripting and C/C++ programming as well as working Knowledge of Maxscale. I have been looking at some sample filters but could not find the luafilter mentioned as it seems the github branch https://github.com/mariadb-corporation/MaxScale/blob/luafilter/server/modul… <https://github.com/mariadb-corporation/MaxScale/blob/luafilter/server/modul…> no longer exists. Please I would love to know where the current repo for the Luafilter is located so as to see what had been done and what we would need to enhance this summer. I look forward to feedback soon. Regards, Pedro

1 0

[Maria-developers] Enhancing mysqlbinlog (MDEV-4989 and MDEV-5993)
by Dan Ungureanu 08 Mar '16

08 Mar '16

Hello, I have been looking into MDEV-4989 [1] and MDEV-5993 [2] and as far as I know, the features requested in those two tickets have already been implemented in MySQL. However, the team decided to approach this problem in a different manner and focus more on parallel replication so the GTID implementations diverged. Now, my question is: for starters, is it enough to consider the code from MySQL as some sort of guideline when implementing these features in MariaDB? Kristian Nielsen did a great job explaining GTID in his blog posts and even though the mechanism behind the scene seems to be pretty simple, the codebase is quite massive and I have a hard time going through all of it. >From what I can understand so far, I do not believe that porting the code from MySQL to MariaDB is possible since there are a few key aspects that differ. Is that correct? I hope I made myself clear and I look forward to hearing from you. [1] https://jira.mariadb.org/browse/MDEV-4989 [2] https://jira.mariadb.org/browse/MDEV-5993 Best regards, Dan Ungureanu

2 1

Re: [Maria-developers] [Commits] e19e45b: MDEV-9651 - Simplify audit event dispatching
by Sergei Golubchik 07 Mar '16

07 Mar '16

Hi, Sergey! On Mar 01, Sergey Vojtovich wrote: > revision-id: e19e45b23c07ef8d0679ca0ed7d93b70625e1a26 (mariadb-10.1.8-141-ge19e45b) > parent(s): e9f6c8167e238107643b9ef6228ca23976afb654 > committer: Sergey Vojtovich > timestamp: 2016-03-01 10:21:42 +0400 > message: > > MDEV-9651 - Simplify audit event dispatching > > Simplified audit event dispatching call chain from: Ok to push. Btw, you there's a conveniency helper safe_strlen() that you could've used. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] GSoC 2016 Maxscale Filter Implementation.
by Jason Kravitz 07 Mar '16

07 Mar '16

Hi, I am Jay Kravitz an Informations systems student at Georgetown University and I would love to work on Maxscale during GSoC 2016. I am interested in the Filter projects specifically the filter to capture incoming operations for consumption in external sources. I have build Maxscale on Ubuntu 15 and I am currently looking at some filters in the codebase to understand the filter API. I will also submit a backup proposal for the query injection filter hoping I am given the opportunity to implement one. I look forward to hearing from anyone soon. Regards, JK.

1 0

[Maria-developers] main.delayed fails on timeout
by Alexander Barkov 03 Mar '16

03 Mar '16

Hello Monty, Sergei, Can you please have a look into: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-precise-amd64/builds… Can you please give any suggestions about what we can do to fix the problem with main.delayed failing sporadically? This problem is blocking us badly in Debian. Otto can try to disable this test temporary. But it would be nice to have a real fix. Any tips would be very appreciated. Thanks!

2 1

[Maria-developers] GSoC 2015 mysqlbinlog Enhancement.
by Rebecca Tucker 03 Mar '16

03 Mar '16

Hello everyone, I am Rebecca Tucker a MS student at University of Maryland and I wish to participate in the summer of code to work on making the mysqlbinlog tool to understand global transaction IDs. I am familiar with the tool and replication and can write documented C and C++ code. I Look forward to communicating with you soon to get more pointers on the project. Cheers, Becca.

1 0

[Maria-developers] GSOC 2016 self Introduction
by Tapan Halani 02 Mar '16

02 Mar '16

Hello all devs, I am Tapan Halani , an undergrad student from India. I am interested in contributing code to MariaDB and am taking part in GSoC 2016 also. I have already done all the prerequisite code cloning and setup etc. and am going through MDEV-4989 , trying to understand and solve it. Looking forward to have a great time contributing. Thanks Tapan

1 0

[Maria-developers] Igor please review: b6a2441: Remove all comments starting with 'psergey'
by Sergey Petrunia 02 Mar '16

02 Mar '16

Please review the below: ----- Forwarded message from Sergei Petrunia <psergey(a)askmonty.org> ----- Date: Wed, 2 Mar 2016 13:35:11 +0300 (FET) From: Sergei Petrunia <psergey(a)askmonty.org> To: commits(a)mariadb.org, sergey(a)mariadb.com Subject: b6a2441: Remove all comments starting with 'psergey' revision-id: b6a2441edf3bfce381f5dc24f5be23fce9ac36dd parent(s): b05158cc10a75196b5c0bf8dad9360608a2dd5b9 committer: Sergei Petrunia branch nick: 10.1-dbg5 timestamp: 2016-03-02 13:35:11 +0300 message: Remove all comments starting with 'psergey' --- sql/item.cc | 3 +-- sql/item_subselect.cc | 15 +-------------- sql/opt_range.cc | 4 ++-- sql/opt_subselect.cc | 1 - sql/opt_table_elimination.cc | 2 -- sql/sql_base.cc | 4 ++-- sql/sql_join_cache.cc | 7 +------ sql/sql_select.cc | 39 ++------------------------------------- sql/sql_statistics.cc | 9 --------- sql/sql_union.cc | 2 +- 10 files changed, 10 insertions(+), 76 deletions(-) diff --git a/sql/item.cc b/sql/item.cc index d6132cc..8fd45d5 100644 --- a/sql/item.cc +++ b/sql/item.cc @@ -4227,8 +4227,7 @@ static bool mark_as_dependent(THD *thd, SELECT_LEX *last, SELECT_LEX *current, DBUG_PRINT("info", ("mark_item: %p lex: %p", mark_item, last)); mark_item->depended_from= last; } - if (current->mark_as_dependent(thd, last, - /** resolved_item psergey-thu **/ mark_item)) + if (current->mark_as_dependent(thd, last, mark_item)) DBUG_RETURN(TRUE); if (thd->lex->describe & DESCRIBE_EXTENDED) { diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc index a5ef81e..e11eafe 100644 --- a/sql/item_subselect.cc +++ b/sql/item_subselect.cc @@ -243,10 +243,6 @@ bool Item_subselect::fix_fields(THD *thd_param, Item **ref) done_first_fix_fields= TRUE; inside_first_fix_fields= TRUE; upper_refs.empty(); - /* - psergey-todo: remove _first_fix_fields calls, we need changes on every - execution - */ } eliminated= FALSE; @@ -1753,7 +1749,6 @@ my_decimal *Item_in_subselect::val_decimal(my_decimal *decimal_value) Check that the right part of the subselect contains no more than one column. E.g. in SELECT 1 IN (SELECT * ..) the right part is (SELECT * ...) */ - // psergey: duplicated_subselect_card_check if (select_lex->item_list.elements > 1) { my_error(ER_OPERAND_COLUMNS, MYF(0), 1); @@ -2172,7 +2167,6 @@ bool Item_allany_subselect::is_maxmin_applicable(JOIN *join) DBUG_ENTER("Item_in_subselect::row_value_transformer"); - // psergey: duplicated_subselect_card_check if (select_lex->item_list.elements != cols_num) { my_error(ER_OPERAND_COLUMNS, MYF(0), cols_num); @@ -3600,7 +3594,6 @@ void subselect_engine::set_row(List<Item> &item_list, Item_cache **row) if (!(row[i]= Item_cache::get_cache(thd, sel_item, sel_item->cmp_type()))) return; row[i]->setup(thd, sel_item); - //psergey-backport-timours: row[i]->store(sel_item); } if (item_list.elements > 1) cmp_type= res_type= ROW_RESULT; @@ -4017,9 +4010,6 @@ int subselect_uniquesubquery_engine::index_lookup() subselect_uniquesubquery_engine::~subselect_uniquesubquery_engine() { - /* Tell handler we don't need the index anymore */ - //psergey-merge-todo: the following was gone in 6.0: - //psergey-merge: don't need this after all: tab->table->file->ha_index_end(); } @@ -4172,9 +4162,6 @@ int subselect_indexsubquery_engine::exec() uint subselect_single_select_engine::cols() { - //psergey-sj-backport: the following assert was gone in 6.0: - //DBUG_ASSERT(select_lex->join != 0); // should be called after fix_fields() - //return select_lex->join->fields_list.elements; return select_lex->item_list.elements; } @@ -5136,7 +5123,7 @@ void check_out_index_stats(JOIN *join) table_map key_used=0; table_map non_key_used= 0; - bzero(&key_start_use, sizeof(key_start_use)); //psergey-todo: safe initialization! + bzero(&key_start_use, sizeof(key_start_use)); bzero(&key_infix_use, sizeof(key_infix_use)); for (order= join->group_list; order; order= order->next) diff --git a/sql/opt_range.cc b/sql/opt_range.cc index ed7e9a5..72a851d 100644 --- a/sql/opt_range.cc +++ b/sql/opt_range.cc @@ -8640,7 +8640,7 @@ static bool remove_nonrange_trees(RANGE_OPT_PARAM *param, SEL_TREE *tree) if (key1->elements != 1) { - key2->use_count+=key1->elements-1; //psergey: why we don't count that key1 has n-k-p? + key2->use_count+=key1->elements-1; key2->increment_use_count((int) key1->elements-1); } if (key1->type == SEL_ARG::MAYBE_KEY) @@ -10104,7 +10104,7 @@ ha_rows check_quick_select(PARAM *param, uint idx, bool index_only, } DBUG_PRINT("exit", ("Records: %lu", (ulong) rows)); - DBUG_RETURN(rows); //psergey-merge:todo: maintain first_null_comp. + DBUG_RETURN(rows); } diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc index cb356fb..1a13dd6 100644 --- a/sql/opt_subselect.cc +++ b/sql/opt_subselect.cc @@ -635,7 +635,6 @@ int check_and_do_in_subquery_rewrites(JOIN *join) (oe1, oe2) IN (SELECT ie1, ie2, ie3 ...) TODO why do we have this duplicated in IN->EXISTS transformers? - psergey-todo: fix these: grep for duplicated_subselect_card_check */ if (select_lex->item_list.elements != in_subs->left_expr->cols()) { diff --git a/sql/opt_table_elimination.cc b/sql/opt_table_elimination.cc index 912ef4a..8378e8e 100644 --- a/sql/opt_table_elimination.cc +++ b/sql/opt_table_elimination.cc @@ -757,8 +757,6 @@ void eliminate_tables(JOIN *join) else { DBUG_ASSERT(!tbl->nested_join || tbl->sj_on_expr); - //psergey-todo: is the following really correct or we'll need to descend - //down all ON clauses: ? if (tbl->sj_on_expr) tables_used_on_left |= tbl->sj_on_expr->used_tables(); } diff --git a/sql/sql_base.cc b/sql/sql_base.cc index ac2162b..3d02f9c 100644 --- a/sql/sql_base.cc +++ b/sql/sql_base.cc @@ -3965,7 +3965,7 @@ thr_lock_type read_lock_type_for_table(THD *thd, goto end; } DBUG_PRINT("tcache", ("opening table: '%s'.'%s' item: %p", - tables->db, tables->table_name, tables)); //psergey: invalid read of size 1 here + tables->db, tables->table_name, tables)); (*counter)++; /* @@ -8105,7 +8105,7 @@ bool setup_tables(THD *thd, Name_resolution_context *context, Item *item= table_list->jtbm_subselect->optimizer; if (table_list->jtbm_subselect->optimizer->fix_fields(thd, &item)) { - my_error(ER_TOO_MANY_TABLES,MYF(0), static_cast<int>(MAX_TABLES)); /* psergey-todo: WHY ER_TOO_MANY_TABLES ???*/ + my_error(ER_TOO_MANY_TABLES,MYF(0), static_cast<int>(MAX_TABLES)); DBUG_RETURN(1); } DBUG_ASSERT(item == table_list->jtbm_subselect->optimizer); diff --git a/sql/sql_join_cache.cc b/sql/sql_join_cache.cc index f84440a..9f69169 100644 --- a/sql/sql_join_cache.cc +++ b/sql/sql_join_cache.cc @@ -3919,11 +3919,6 @@ int JOIN_TAB_SCAN_MRR::next() If a record in in an incremental cache contains no fields then the association for the last record in cache will be equal to cache->end_pos */ - /* - psergey: this makes no sense where HA_MRR_NO_ASSOC is used. - DBUG_ASSERT(cache->buff <= (uchar *) (*ptr) && - (uchar *) (*ptr) <= cache->end_pos); - */ if (join_tab->table->vfield) update_virtual_fields(join->thd, join_tab->table); } @@ -4591,7 +4586,7 @@ bool JOIN_CACHE_BKAH::prepare_look_for_matches(bool skip_last) { last_matching_rec_ref_ptr= next_matching_rec_ref_ptr= 0; if (no_association && - !(curr_matching_chain= get_matching_chain_by_join_key())) //psergey: added '!' + !(curr_matching_chain= get_matching_chain_by_join_key())) return 1; last_matching_rec_ref_ptr= get_next_rec_ref(curr_matching_chain); return 0; diff --git a/sql/sql_select.cc b/sql/sql_select.cc index 4759af0..878cb8b 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -2861,9 +2861,6 @@ void JOIN::exec_inner() /* Free first data from old join */ - /* - psergey-todo: this is the place of pre-mature JOIN::free call. - */ curr_join->join_free(); if (curr_join->make_simple_join(this, curr_tmp_table)) DBUG_VOID_RETURN; @@ -3001,7 +2998,6 @@ void JOIN::exec_inner() curr_join->select_distinct=0; } curr_tmp_table->reginfo.lock_type= TL_UNLOCK; - // psergey-todo: here is one place where we switch to if (curr_join->make_simple_join(this, curr_tmp_table)) DBUG_VOID_RETURN; calc_group_buffer(curr_join, curr_join->group_list); @@ -6646,12 +6642,6 @@ static void choose_initial_table_order(JOIN *join) jtab_sort_func= straight_join ? join_tab_cmp_straight : join_tab_cmp; } - /* - psergey-todo: if we're not optimizing an SJM nest, - - sort that outer tables are first, and each sjm nest follows - - then, put each [sjm_table1, ... sjm_tableN] sub-array right where - WHERE clause pushdown would have put it. - */ my_qsort2(join->best_ref + join->const_tables, join->table_count - join->const_tables, sizeof(JOIN_TAB*), jtab_sort_func, (void*)join->emb_sjm_nest); @@ -9171,7 +9161,6 @@ static bool create_ref_for_key(JOIN *join, JOIN_TAB *j, !(parent->join_tab_reexec= (JOIN_TAB*) thd->alloc(sizeof(JOIN_TAB)))) DBUG_RETURN(TRUE); /* purecov: inspected */ - // psergey-todo: here, save the pointer for original join_tabs. join_tab= parent->join_tab_reexec; table= &parent->table_reexec[0]; parent->table_reexec[0]= temp_table; table_count= top_join_tab_count= 1; @@ -10060,9 +10049,6 @@ bool TABLE_LIST::is_active_sjm() DBUG_ASSERT(inner_tab->table); current_map= inner_tab->table->map; used_tables2|= current_map; - /* - psergey: have put the -1 below. It's bad, will need to fix it. - */ COND *tmp_cond= make_cond_for_table(thd, on_expr, used_tables2, current_map, /*(inner_tab - first_tab)*/ -1, @@ -11070,11 +11056,6 @@ void check_join_cache_usage_for_tables(JOIN *join, ulonglong options, idx, prev_tab); tab->use_join_cache= MY_TEST(tab->used_join_cache_level); - /* - psergey-merge: todo: raise the question that this is really stupid that - we can first allocate a join buffer, then decide not to use it and free - it. - */ if (join->return_tab) { tab= join->return_tab; @@ -15622,12 +15603,6 @@ bool cond_is_datetime_is_null(Item *cond) @retval false cannot be used */ -/* - psergey-todo: this returns false for int_column='1234' (here '1234' is a - constant. Need to discuss this with Bar). - - See also Field::test_if_equality_guaranees_uniqueness(const Item *item); -*/ static bool test_if_equality_guarantees_uniqueness(Item *l, Item *r) { @@ -18293,10 +18268,6 @@ enum_nested_loop_state rc= evaluate_join_record(join, join_tab, error); } - /* - Note: psergey has added the 2nd part of the following condition; the - change should probably be made in 5.1, too. - */ bool skip_over= FALSE; while (rc == NESTED_LOOP_OK && join->return_tab >= join_tab) { @@ -20956,7 +20927,7 @@ uint find_shortest_key(TABLE *table, const key_map *usable_keys) !(table->file->index_flags(best_key, 0, 1) & HA_CLUSTERED_INDEX))) goto use_filesort; - if (select && // psergey: why doesn't this use a quick? + if (select && table->quick_keys.is_set(best_key) && best_key != ref_key) { key_map tmp_map; @@ -24002,11 +23973,7 @@ void JOIN_TAB::save_explain_data(Explain_table_access *eta, eta->type= tab_type; /* Build "possible_keys" value */ - // psergey-todo: why does this use thd MEM_ROOT??? Doesn't this - // break ANALYZE ? thd->mem_root will be freed, and after that we will - // attempt to print the query plan? append_possible_keys(thd->mem_root, eta->possible_keys, table, keys); - // psergey-todo: ^ check for error return code /* Build "key", "key_len", and "ref" */ if (tab_type == JT_NEXT) @@ -24054,8 +24021,6 @@ void JOIN_TAB::save_explain_data(Explain_table_access *eta, eta->hash_next_key.set(thd->mem_root, & table->key_info[index], table->key_info[index].key_length); - // psergey-todo: ^ is the above correct? are we necessarily joining on all - // columns? } if (!key_info) @@ -24211,7 +24176,7 @@ void JOIN_TAB::save_explain_data(Explain_table_access *eta, eta->push_extra(ET_OPEN_FRM_ONLY); else eta->push_extra(ET_OPEN_FULL_TABLE); - /* psergey-note: the following has a bug.*/ + if (table_list->is_table_read_plan->trivial_show_command || (table_list->is_table_read_plan->has_db_lookup_value() && table_list->is_table_read_plan->has_table_lookup_value())) diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc index 28e76ae..b22a2ae 100644 --- a/sql/sql_statistics.cc +++ b/sql/sql_statistics.cc @@ -3650,15 +3650,6 @@ double get_column_range_cardinality(Field *field, { double avg_frequency= col_stats->get_avg_frequency(); res= avg_frequency; - /* - psergey-todo: what does check for min_value, max_value mean? - min/max_value are set to NULL in alloc_statistics_for_table() and - alloc_statistics_for_table_share(). Both functions will immediately - call create_min_max_statistical_fields_for_table and - create_min_max_statistical_fields_for_table_share() respectively, - which will set min/max_value to be valid pointers, unless OOM - occurs. - */ if (avg_frequency > 1.0 + 0.000001 && col_stats->min_value && col_stats->max_value) { diff --git a/sql/sql_union.cc b/sql/sql_union.cc index c835083..b369ef0 100644 --- a/sql/sql_union.cc +++ b/sql/sql_union.cc @@ -974,7 +974,7 @@ bool st_select_lex_unit::exec() 1st execution sets certain members (e.g. select_result) to perform subquery execution rather than EXPLAIN line production. In order to reset them back, we re-do all of the actions (yes it is ugly): - */ // psergey-todo: is the above really necessary anymore?? + */ join->init(thd, item_list, fake_select_lex->options, result); saved_error= mysql_select(thd, &fake_select_lex->ref_pointer_array, &result_table_list, ----- End forwarded message ----- -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] GSoC 2016 self Introduction
by Charles Muurmu 02 Mar '16

02 Mar '16

Dear MariaDB Developers, I am Charles Muurmu and I wish to participate in GSoC 2016 (and 17 if possible) with MariaDB. First, congratulations for passing a successful application with Google. I have gone through the program time line and students applications start in two weeks, However I am interested in two projects for which I will submit 2 proposals. First, I am interested in the Cassandra storage engine V2 as I have basic knowledge on how to write storage engine plugins as well as working knowledge of cassandra. Also I will be having as backup project the GIS enhancement task which requires adding a new geometry type containing the third coordinate, writing importers and exporters for the extended geometry formats... I have two books guiding me for now: Understanding MySQL Internals by O'reilly media and MySQL plugin Development by Sergei Golubchik. While preparing for student application I will be reading mariadb Knowledge base and ask questions on #maria. I will also try to look an the DataStax C++ Driver which will be used. Thanks, Charles.

1 0

[Maria-developers] [GSoC2016] Contribution to Cassandra SE v2
by Olzhas Kaiyrakhmet 02 Mar '16

02 Mar '16

Hello, I am Olzhas Kaiyrakhmet, computer science student at Ulsan National Institute of Science and Technology, Korea. I am interested in contributing to code to MariaDB and participating in GSoC2016. I do find the project, Cassandra SE v2, very interesting for me. Recently, I worked with Cassandra cluster and did look inside of it code. Tried to do some modifications. Also, I have some experience on contributing to distributive systems and databases as research intern in university. I have a great will to contribute for this project with your assistance. Best wishes, Olzhas kaiyrakhmet

3 2

[Maria-developers] Time delayed replication
by Colin Charles 29 Feb '16

29 Feb '16

Hi! Quick question — we’ve had https://jira.mariadb.org/browse/MDEV-8341 open for quite sometime now, and MySQL 5.6 and 5.7 obviously implement this feature, time delayed replication Are you using it in production? Are you finding that you are missing it in MariaDB Server? How important would it be to you, to have this in MariaDB Server 10.2? Thanks -- Colin Charles, http://bytebot.net/blog/ twitter: @bytebot | skype: colincharles "First they ignore you, then they laugh at you, then they fight you, then you win." -- Mohandas Gandhi

2 1

Re: [Maria-developers] A proposal to deprecate syntax: SELECT a'test'
by Michael Widenius 29 Feb '16

29 Feb '16

Hi! > Hello Monty, all, > > > We discussed this with Sergei and both think that it will be a good idea > to deprecate this syntax: > > SELECT a'test'; > > where 'a' is an identifier and 'test' its alias. Is this dependent on if there is a space or not between a and 'test' ? Note that in MySQL and MariaDB, ' and " are both used for quoting strings and I don't think we can change that without affecting a lot of applications! I think it's ok to require one to have a space between a and 'test', if that would help at all. > This is a non-standard way, and it conflicts with some other important > standard SQL grammar. See below. Note that in MySQL/MariaDB, when not using ANSI mode, ' and " are equal and ` is used for identifiers, so this is expected to be non-standard. For quoting identifiers ` should be used. Of course, if you are using ANSI mode, then things should be as close to ANSI as possible. > Typified literals and a syntax conflict > --------------------------------------- > > An identifier followed by a single-quoted text string is needed > for typified literals: > > SELECT INET6'ffff::ffff'; > > We already support this for temporal literals, which is a part of > the SQL standard: > > SELECT TIME'10:10:10'; > SELECT TIMESTAMP'2001-01-01 10:20:30'; > SELECT DATE'2001-01-01'; > > Note, the conflict already exists in MySQL and MariaDB. > > > This script: > > DROP TABLE IF EXISTS t1; > CREATE TABLE t1 (date date); > INSERT INTO t1 VALUES ('2016-02-19'); > SELECT date'2001-01-01' FROM t1; > > returns: > > +------------------+ > | date'2001-01-01' | > +------------------+ > | 2001-01-01 | > +------------------+ > > Notice, it returns a DATE literal, it does not return the value of > the column t1.date using '2001-01-01' as an alias. As ansi sql doesn't allow you to use date as a column name, this isn't really as big problem as it may seam ;) Anyway, I agree that this can be confusing, but so is having date as a column namn. > Notice the difference: > > SELECT date`2001-01-01` FROM t1; > +------------+ > | 2001-01-01 | > +------------+ > | 2016-02-19 | > +------------+ > > This is confusing. Not really, as you are using different quotes and you get the expected answer. This should be same as if you used " in ANSI SQL, isn't it? > Shift-reduce conflicts > ---------------------- > By the way, the fact that we support single-quoted string as an alias > is the reason for shift-reduce conflicts in sql_yacc.yy, because > TIME/DATE/TIMESTAMP followed by 'string' can be interpreted in two ways: > - a column name followed by an alias > - an SQL-standard time literal What is the SQL-standard way to use an alias? (Sorry, can't access the standard just now): SELECT date "2001-01-01" FROM t1; or SELECT date '2001-01-01' FROM t1; And if you execute both of these queries, will you not get the same confusing result for one of them? As ' and " are identical in MySQL mode, I don't see how you can fix this without causing even more confusion. > What can be used instead > ------------------------ > We have a number of other ways to specify aliases: > > The standard ways: > > SELECT a AS test; > SELECT a "test"; > SELECT a AS "test"; Ok, this answers my question above. The standard says that an alias is a text string. In MySQL / MariaDB a text string can be specified with both ' or ". I don't think it's good idea to make alias a special case where you have to specify the text string only with ". > MySQL/MariaDB extensions: > > > SELECT a `test`; > SELECT a AS `test`; > > That should be enough. Note that this has nothing exclusively to do with alias, but with the MySQL/MariaDB extension that you can use ' instead of " as a string delimiter. > Proposal > -------- > 1. In 10.1 we add a warning when a single-quoted string is used as a > column alias. Something like this should work: > > Single quotes in a select sublist alias are deprecated. Use double > quotes instead. This would break the specification of what a text strings is in MariaDB. It would also be very confusing that text strings needs to be specified differently in different places of the SQL syntax. > 2. In 10.2 we disallow this syntax by default and add either sql_mode or > old_mode to enable the old behavior. As you can already get the behavior you want by specifying ANSI MODE, I don't think this change is necessary. I think it's also bad as it makes it harder to define what a text string is. For example: In MariaDB, a text string can be delimited with ' or ", except in the case of alias when you have to use ' I think it's better to require, as most ANSI SQL databases does, one to quote column names like 'date' and 'time' to ensure they are not mixed with operators or string prefixes. Regards, Monty

2 3

[Maria-developers] Type_handler and INET6
by Alexander Barkov 29 Feb '16

29 Feb '16

Hello Sergei, I'm sending a new version for Type_handler and INET6 for your review. There are three patches attached. 1. inet6-v65-prepare.diff.gz Includes preparatory changes, to avoid different execution paths for different data types. 2. inet6-v65-incremental.diff.gz Actually adds INET6, together with val_raw family methods, last minute changes that did not get into #1. 3. inet6-v65-full.diff.gz is a full patch that includes the previous two. Details on the patches as well as my answers to your previous review suggestions follow below. Thanks. The first patch makes preparatory changes, to avoid different execution paths and adds some helper classes needed for the API. - Adds a class Record_addr, to simplify API (pass just one value instead of 3 separate values for ptr, null_ptr and null_bit) and share some code moving initialization (e.g. for a temporary table purposes) to Record_addr constructors. Derives Field from it. - Adds a class Type_ext_attributes. This collects extra members of Item_type_holder. Currently it covers only extra attributes for GEOMETRY, DECIMAL, ENUM/SET. We'll add a generic "void *extra" pointer here, when user-defined types need even more extra attributes. This class will also be needed inside Item_hybrid_func, which currently have a bug (do not preserve geometry type and srid of the arguments). I'll fix it later. See MDEV-9405. - Adds a class Create_attr, to pass Create_field related attributes to handlers. - Adds a class Type_handler_register, to look up handlers by type code and names. - Continues unification of the code that creates temporary fields for various Items, see ::create_tmp_field() in sql_select.cc. This task is not fully finished. Some more chages are still needed. The intent is to get rid of ::create_tmp_field() and switch to use methods in Item instead. - Adds a data type handler pointer to Arg_comparator. Previously the was only "Item_result m_compare_type". - Splits Item_func_case::fix_length_and_dec() into three separate steps: fix_field_type(), fix_return_arguments(), fix_comparison_arguments(), to incorporate the type handler methods easier. - Splits Item_func_between::val_int() into virtual methods in Type_handler_xxx. - Adds virtual methods to Type_handler, to avoid different execution paths for non-traditional types: calc_pack_length(), check_column_definition(), prepare_column_definition(), make_table_field(), Item_save_in_field(), make_cache_item(), calc_pack_length(), calc_display_length(), Item_type_holder_join_attributes(), Item_func_hex_val_str_ascii(), Item_func_hybrid_field_type_val_xxx(), Item_func_between_fix_length_and_dec(), Item_func_between_val_int(), Item_sum_hybrid_fix_length_and_dec(), - Adds Type_handler_numeric, to share some code in real, int decimal handlers. - Moves field_type_for_temporal_comparison() from Item to Type_handler. - Makes some other changes in data type aggregation in various pieces of the code. This will need some more changes though. - Removes Field::result_merge_type(). Uses Type_handler::cmp_type() and Type_handler::result_type() instead. - Simplifies ::make_field() in field.cc, using Type_handler::make_table_field(). - Removes the global Arg_comparator::comparator_matrix[6][2] and uses methods in Type_handler instead. The second patch actually adds INET6 and: - Adds val_raw family methods into Items - Adds some pieces into the type aggregation code in a few places - Fixes sql_yacc.yy to handle non-traditional types, adds Cast_type and fixes Lex_cast_type_st. - Fixes Type_handler_register to search non-traditional types by name. - Disables equal expression propagation for non-traditional types (the code is not ready yet for this, e.g. Item_equal needs some changes) - Adds the INET6 handler with tests Also, I answer your questions and suggestions that you made during the first review. Please see below. >> > > === added file 'mysql-test/r/type_inet6.result' > > > --- mysql-test/r/type_inet6.result 1970-01-01 00:00:00 +0000 > > > +++ mysql-test/r/type_inet6.result 2014-07-01 11:10:33 +0000 > > > +::ff > > > +::ffff > > > +::0.255.255.255 > > > +::255.255.255.255 > > > +::ff:ffff:ffff > > > +::ffff:255.255.255.255 > > Hmm. A bit unexpected. Do you have a link to some > document that specifies how ipv6 numbers should be parsed and printed? https://en.wikipedia.org/wiki/IPv6_address says: :ffff:0:0/96 — This prefix is designated as an IPv4-mapped IPv6 address. Note, I also checked this in PostgreSQL and it works in the same way: bar=> select inet'::ffff:ffff:ffff'; inet ------------------------ ::ffff:255.255.255.255 > > > +SELECT * FROM t1 WHERE a=0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF; > > > +a > > > +ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff > > > +ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff > > > +Warnings: > > > +Warning 1105 Cast to INET6 converted '\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF' to 'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff' > > Too many warnings. I'd rather use Item_cache or something similar > to have only one actual type conversion and one warning It now creates Item_cache. So there's only one warning. > > > === modified file 'sql/field.h' > > > --- sql/field.h 2014-06-11 08:08:08 +0000 > > > +++ sql/field.h 2014-07-01 10:38:12 +0000 > > > @@ -366,6 +437,10 @@ class Field > > > virtual ~Field() {} > > > /* Store functions returns 1 on overflow and -1 on fatal error */ > > > virtual int store(const char *to, uint length,CHARSET_INFO *cs)=0; > > > + virtual int store_raw(const String *raw, enum_field_types type) > > > + { > > > + return store(raw->ptr(), raw->length(), raw->charset()); > > > + } > > How do you guarantee that the 'raw' is in the correct internal raw format? > Or is it only used after int_to_raw, str_to_raw, etc? raw representation can only be created by a method of the corresponding Type_handler_xxx, or come from "ptr" of the corresponding Field_xxx. Bad raw format means that there is a bug inside Type_handler_xxx. Btw, In the latest patch I changed it to: int store_raw_native(const char *str, uint length); Conversion (when needed) is done outside of Field. > > > +class Field_type_joiner > > It's rather "aggregator", not "joiner". > But in fact, it's not even that, it's a pair of { type, derivation }. > May be it should be Field_type_with_derivation ? :) > And a method Field_type_with_derivation::aggregate(...) It's now Type_handler_hybrid_field_type, which I earlier added in 10.1. There is no "derivation" in this version, as we discussed in Luxembough. Sigh, I'd really really prefer if we had it though. I think when field=expr is compared according to the field data type is clearer than they're compared accoring to the expr datatype. Also, comparing according to field will use the optimizer by default, without having to cast "expr" to the field data type. > > > +class Type_handler > > > +{ > > > +public: > > > + Type_handler() {} > > > + ~Type_handler() {} > > > + virtual const char *field_type_name() const = 0; > > > + virtual uint field_type_name_length() const = 0; > > may be, remove field_ prefix, or even a field_type_ prefix? > the method is still unambiguous, but much shorter and readable. > and, may be, make it non-virtual, like > > ... > private: > LEX_STRING type_name; > public: > Type_handler(const char *name_arg) > { > type_name.str= name_arg; > type_name.length= strlen(name_arg); > } > const char *name() const { return type_name.str; } > const char *name_length() const { return type_name.length; } > ... > > and your new type will do in the constructor: > > Type_handler_inet6(): Type_handler("INET6") { } > > less boilerplate code for the new types, and you know how I'm trying > to avoid boilerplate code in the APIs :) I'm using the very needed new class Name for it. It now looks simpler, I believe. > > > > + virtual enum_field_types field_type() const = 0; > > That doesn't work for loadable types. > I think that we eventually need to do with types what we've done with > storage engines. There was db_type enum, but now the engine is almost always > identified by the pointer to the handlerton. So, for types a pointer to > the Type_handler might be a unique type identifier in the server. > Replacing enum_field_types where possible. Sure, in the end we should get a code that will identify handlers by a pointer rather than field_type(). This will need a few additional steps, which I just did not have a chance to do yet: - Fix all existing Items to return type_handler() instead of field_type() as a type identifier. This will be easier after we add virtual inheritance (or templates) soon, on top of my Type_handler changes. - Get rid of all field_type() calls: split the remaining pieces of the code that use field_type() into more virtual methods in Type_handler. - Fix FRM to store new field types as a textual name rather than a numeric ID. Can we do these steps a separately? Note, it can be done after adding INET6 and UUID, before adding dynamic loadability. > > > + virtual enum_field_types fallback_field_type() const = 0; > > I don't like the idea of fallback_field_type. > You have Field_type_joiner for new types, but you fallback to > Field::field_type_merge() for old types. I'd rather not distinguish > between "old" and "new" types. Just like we have with storage engines or > authentication - there are no "native" and "plugin" storage engines (or > authentication), *all* storage engines are plugins. So there's no code like > > if (is_plugin(engine_type)) > ... > else > ... > > so, I prefer uniform code with no special checks for type handlers. > Implementations differ, but the interface is the same for all types. I nodified the code, so there is no a separate fallback_field_type() any more. But there is still cmp_type(), which is uses in very rare cases, to exchange data to/from other data types: INET6->something, something->INET6. Also, there are still some different execution paths inside the methods that aggregate data types. We'll need to discuss it. I could not come up with a simple single method for aggregation. Also, currently field type aggregation is done in different ways in various pieces of the code (hybrid functions, UNION, comparison). We should try to unify these ways. > > > + virtual Item_result cast_to_int_type() const = 0; > > Why is this abstract? It could have a reasonable default implementation > reducing the amount of boilerplate code in the Type_handler's > > > > + virtual int cmp(Item *a, Item *b) const = 0; > > > + virtual int cmp(String *a, String *b) const = 0; > > These too Not sure that exactly cast_to_int_type() can have a reasonable default. For INET6 and UUID it's DECIMAL_RESULT. For compressed URLs (MDEV-4630) and external ENUM (MDEV-4913) it's more likely to be STRING_RESULT. Anyway, I agree that we should add some reasonable defaults when possible. Perhaps we could add two abstract classes with some defaults: Type_handler | +-- Type_handler_traditional (derive all existing types from this) | +-- Type_handler_binary_fixed_length_storage (derive INET6 and UUID from this) But I'd propose not to hurry with that. We have time before we add loadabale data type plugins. Can we add UUID first? It will show how much INET6 and UUID can share and thus how much can be moved to Type_handler_binary_fixed_length_storage. > > > + virtual int save_in_field(Item_hex_hybrid *item, Field *field) const = 0; > > Why do you need that virtual? It could be always val_raw/store_raw I guess this question is not relevant any more, because we now have handlers for the traditional data types, and they need to treat hex hybrids differenty (as string for Field_str and Field_temporal, and as a longlong for the other datatypes). Note, I modified this piece slighly. It's now a virtual method in Field rather than in Type_handler: int Field::store_hex_hybrid(const char *str, uint length); > > > + virtual int set_cmp_func(Arg_comparator *cmp, Item *a, Item *b) const = 0; > > this can be removed, see below (under Type_handler_inet6::set_cmp_func) I could not get rid of it. Moreover, the traditional types now use this method. For example: bool Type_handler_int_result::set_comparator_func(Arg_comparator *cmp) const { return cmp->set_cmp_func_int(); } So now I have the same execution path for all data types here. > > > + virtual bool varbinary_to_raw(const String *from, String *to) const = 0; > > 1. This most certainly needs a comment. It took me quite a while to > understand what this method is for and why str_to_raw isn't used instead. > A comment should say that it's used when a field binary representation is > specified in a query directly was 0xHHHHHHHH... > > 2. Still, I'm not sure this method is needed. It means a different > behavior for > > where ipv6 = 0xHHHHHH... > > and > > create t1 (b binary(16)); insert t1 values (0xHHHHH....); > ... where ipv6 = a ... > Note, it's the same with what we do for numbers: CREATE OR REPLACE TABLE t1 (a INT, b BLOB); INSERT INTO t1 VALUES (0x31,0x31); SELECT COUNT(*) FROM t1 WHERE a=0x31; -- returns a row SELECT * FROM t1 WHERE a=b; -- returns no rows See below. > an alternative would be to treat any binary string as raw, > directly in str_to_raw. But a drawback would be that > > where ipv6 = "ffff::1" > > and > > where ipv6 = convert("ffff::1" to binary) > > will work differently. And it's not particularly good either. I like the idea to treat all binary strings as raw with INET6, and then with UUID. I think this is not really a drawback. Remember, when we discussed "is [VAR]BINARY a data type or a character set" around the year of 2005, we fortunately decided that they are data types, so we introduced data types [VAR]BINARY as a replacement to "CHAR CHARACTER SET binary". So [VAR]CHAR abd [VAR]BINARY are different data types! It should be OK that INET6 will communicate with two ***different*** datatypes [VAR]CHAR vs [VAR]BINARY differently. If we agree on that, I'll change the code accordingly. We preliminary agreed on IRC to make explicit and implicit conversion between a data type and BINARY use methods in Type_handler. (but I have not added them yet). > > So, it's a tradeoff, and the question is - what is "less gotcha" ? > I don't know. I have a slight preference towards removing varbinary_to_raw() > just to make the API smaller. And thinking about adding it if there would be > user complains. > > Another thought. I'm starting to doubt that we should support direct > raw values in the SQL at all. It might work for ipv6 > > where ipv6 = 0xHHHHHH > > but it certainly won't work for, say, integers: > > where int = 0xHHHHH > > ^^^ this is not raw, but a normal hexadecimal number. > So, unless you introduce a special unambiguous syntax (like RAW(0xHHHHH)), > I think that raw values should not be supported in SQL. This syntax looks useful for me: WHERE inet6_field=0xFFFF; Btw, in case of integer numbers, it's exactly the low level raw format in Field. This is how integer numbers are stored, isn't it? I didn't understand why it won't work for numbers. Note, comparison to a hex hybrid works fine for INET6 in my patch and does not break the existing data types. Numeric comparion goes through Item_hex_hybrid::val_int(). String comparison goes through Item_hex_hybrid::val_str(). INET6 comparison goes through Item_hex_hybrid::val_raw(). This is Arg_comparator::func who decides which method to use. In the future, we could probably unify this: implement val_raw() for the traditional data types, and use val_raw() for comparison instead of the usual methods like val_int(). It's more tricky what should happen in case of VARCHAR-alike or BLOB-alike data types (suppose we'll decide the standard CLOB type in the future). Should val_raw() include the length bytes? I think no, as it will assume duplication of "length", once in String::length(), and one more time in the beginning of String::ptr(). So I was thinking of "raw" as something very close to how the data is stored in Field, but it's not necessary a 100% match. Perhaps we just need a different name than "raw"... I agree we don't need the field-style low level raw format (e.g. with length bytes for a VARCHAR-alike data type). > > > + virtual Item *make_cast(MEM_ROOT *mem_root, Item *arg) const = 0; > > Hmm. This implies an Item_typecast_xxx for every type. > I would be easier to have one Item_typecast_handled, that stores > a pointer to Type_handler. Would be much easier than implementing > new Item_typecast_xxx for every type. Some data types could probably share the same Item_typecast_xxx, but I don't think it's generally doable with a single Item for all data types. There are a few reasons why I prefer a separate method in handler to create a CAST Item. - Different data types can have different set of extra attributes, which the corresponding Item_typecast_xxx should store in. It's easier to store attributes in typed structures rather than in a generic memory pointed by "void *". It will make our life easier when we start debugging in gdb :) - I plan to simplify sql_yacc.yy so the loadable datatypes and the traditional datatypes share the same CAST and field_type grammar. It will be needed soon even for the traditional data types. - A method in handler will help to add the missing CAST types easier, e.g. FLOAT, SMALLINT, etc. It looks natural that every Field_xxx has a corresponding Item_typecast_xxx. > > > + int store_raw(const String *str, enum_field_types type_arg) > > > + { > > > + if (type_arg != type()) > > > + { > > > + set_null(); > > > + return 1; // TODO: go through str/int/real/dec/date? > > > + } > > > + DBUG_ASSERT(str->length() == binary_length()); > > > + memcpy(ptr, str->ptr(), binary_length()); > > > + return 0; > > > + } > > val_raw and store_raw should not be virtual, they are always > identical for all types. Just don't forget the validity check. <offtopic> Btw, it now looks like this: int store_raw_native(const char *str, uint length) { DBUG_ASSERT(length == binary_length()); memcpy(ptr, str, binary_length()); return 0; } I moved type conversion to the upper level. </offtopic> Right, the actual "store" is often a simple memcpy() for fixed length data types like INET6. But it's not generally the case. - It can be different for non-fixed length data types. It's again about the fact that the value of val_raw() and the byte array stored in Field::ptr() can have slighly differrent formats. Variable length types will pass the values using a normal String format, without additional length bytes coded inside the array pointed by String::ptr(). - Also, in case we have a shared method to store, we'll have to have additional virtual methods for validity check, to call inside. There's no sense for two virtual calls. - Also, in some cases it will be possible to use val_raw() instead of val_int(), val_decimal(), val_str(), val_real(), get_date(), and thus share almost duplicate code that's now implemented for different Item_result types. I already mentioned Arg_comparator. But there could also be only one universal Item_cache, etc. > > > + int store_time_dec(MYSQL_TIME *ltime, uint dec) > > > + { > > > + my_decimal tmp; > > > + return store_decimal(date2my_decimal(ltime, &tmp)); > > > + } > > Numerous val_xxx and store_xxx could be replaced by xxx_to_raw > and raw_to_xxx + val_raw and store_raw. You shouldn't introduce > both val_xxx and raw_to_xxx. Same for store_xxx and xxx_to_raw Right. I don't expose raw_to_xxx publically any more. They are now internal protected Type_handler_inet6 routines. > > > > + > > > + uint size_of() const { return sizeof(*this); } > > > + }; > > > + > > > + > > > + class Arg_comparator_inet6: public Arg_comparator > > > + { > > > + public: > > > + int compare_inet6() > > > + { > > > + if ((*a)->val_raw(&value1, cmp_field_type()) || > > > + (*b)->val_raw(&value2, cmp_field_type())) > > > + goto null; > > > + if (set_null) > > > + owner->null_value= 0; > > > + DBUG_ASSERT(value1.length() == Type_handler_inet6::binary_length()); > > > + DBUG_ASSERT(value2.length() == Type_handler_inet6::binary_length()); > > > + return memcmp(value1.ptr(), value2.ptr(), Type_handler_inet6::binary_length()); > > > + null: > > > + if (set_null) > > > + owner->null_value= 1; > > > + return -1; > > > + } > > > + int compare_e_inet6() > > > + { > > > + bool a_is_null= (*a)->val_raw(&value1, cmp_field_type()); > > > + bool b_is_null= (*b)->val_raw(&value2, cmp_field_type()); > > > + if (a_is_null || b_is_null) > > > + return MY_TEST(a_is_null == b_is_null); > > > + DBUG_ASSERT(value1.length() == Type_handler_inet6::binary_length()); > > > + DBUG_ASSERT(value2.length() == Type_handler_inet6::binary_length()); > > > + return MY_TEST(memcmp(value1.ptr(), value2.ptr(), Type_handler_inet6::binary_length()) == 0); > > > + } > > we should try to get rid of one of the functions, see below > ... > ... > ... > > > + int set_cmp_func(Arg_comparator *cmp, Item *a, Item *b) const > > > + { > > > + cmp->set_func(cmp->is_owner_equal_func() ? > > > + (arg_cmp_func) &Arg_comparator_inet6::compare_e_inet6 : > > > + (arg_cmp_func) &Arg_comparator_inet6::compare_inet6, > > > + field_type()); > > Again, boilerplate code, better: > > cmp->set_func(Arg_comparator_inet6::compare_e_inet6, > Arg_comparator_inet6::compare_inet6, > field_type()); > > no "cmp->is_owner_equal_func() ?" - this can be done inside set_func() > Also, I wonder, whether every type really needs two functions, for equal func > and not. Perhaps having only one - compare_e_inet6() should be sufficient and > checks for NULL can be done in the common wrapper? > And, even more, set_cmp_func() is not necessary at all. Type_handler > should only have compare_type() and compare_e_type() virtual methods > (or only one compare_e_type()) and the implementation of set_cmp_func() > becomes identical for all types, and thus doesn't have to be virtual > anymore. One method less to implement. I agree. We should try to do that. Let's try to decompose the functions of the existing data types and the INET6 related functions in a separate patch. > > > > + return 0; > > > + } > > > + void sortlength(struct st_sort_field *order, const Item *item) const > > > + { > > > + order->result_type= item->cmp_type(); > > > + order->length= Type_handler_inet6::binary_length(); > > > + order->suffix_length= 0; > > > + order->need_strxnfrm= 0; > > > + } > > > + bool make_sort_key(struct st_sort_field *order, uchar *to) const > > > + { > > > + Item *item= order->item; > > > + String buf((char *) to, order->length, &my_charset_bin); > > > + /* > > > + make_sortkey() uses str_result(), val_result(), val_int_result(), > > > + val_decimal_result() for the standard result types. > > > + QQ: should we introduce val_raw_result() and use here instead of > > > + val_raw() ? > > Or, perhaps, we should fix make_sortkey() to use val_str, val_real, etc. > I failed to understant why *result() methos are used there. So did I. But as I had to implement the val_raw_result() methods anyway for other purposes, it was easy to use val_raw_result() here, for symmetry with the other types, and in the latest patch I still use _result_ here. It can be fixed later for all data types to use the usual val_xxx() instead. > I've even replaced them with non-*result versions and not a single test > failed. Including the test case that was added when val* methods > were replaced with val*result methods. > > > > + > > > +Type_handler_register::Type_handler_register() > > > +{ > > > + memset(m_handler, 0, sizeof(m_handler)); > > > + m_min_type= 256; > > > + m_max_type= 0; > > > + m_min_length= 256; > > > + m_max_length= 0; > > > + add(&Type_handler_inet6::s_singleton); > > Eh? I thought it's added in the Type_handler_inet6 constructor. I think in the end there should be two separate steps actually: - a constructor for Type_handler_xxx - a method to add it into the registry So the plugin loading routine will first create a handler from a dynamic library, and then put it into the registry. Now I just implemented it by a special "class Type_handler_inet6_singleton".

2 4

[Maria-developers] Unsubscribe
by Abdul Hakeem 28 Feb '16

28 Feb '16

1 0

[Maria-developers] win.test fails with --ps-protocol
by Sergey Petrunia 27 Feb '16

27 Feb '16

Hello Igor, I'm looking at the buildbot http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.2-mde… and I see that your push https://github.com/MariaDB/server/commit/0c6d753708dfe07607ab6879b4832fe6ec… causes failures for win.test with --ps-protocol: http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/4911/st… Question #1: can you fix this? Question #2: Also, stacktrace printing is broken in the tree: mysqld: /home/buildbot/buildbot/build/mariadb-10.2.0/sql/set_var.cc:1205: sys_var::where get_sys_var_value_origin(void*): Assertion `!mysqld_server_started' failed. Fatal signal 6 while backtracing Could a failure like this be caused by "the backport" that was pushed earlier? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] a849d1b: MDEV-9488 - Table cache cleanups
by Sergei Golubchik 24 Feb '16

24 Feb '16

Hi, Sergey! On Jan 29, Sergey Vojtovich wrote: > revision-id: a849d1bac80f2b38bc6bc170388e2f3cc1d4e1ad (mariadb-10.1.8-113-ga849d1b) > parent(s): 5f656d0dca8aae9486d91191ee6b6da6b9c552bf > committer: Sergey Vojtovich > timestamp: 2016-01-29 12:58:50 +0400 > message: > > MDEV-9488 - Table cache cleanups > > Remove tdc_acquire_share() helpers: they don't actually make things simpler. Hmm, okay. ok to push. Regards, Sergei

1 0

Re: [Maria-developers] eff48be: MDEV-9488 - Table cache cleanups
by Sergei Golubchik 24 Feb '16

24 Feb '16

Hi, Sergey! On Jan 29, Sergey Vojtovich wrote: > revision-id: eff48be5f1f9aedd2b0d6690d0883d05ef7f44a1 (mariadb-10.1.8-114-geff48be) > parent(s): a849d1bac80f2b38bc6bc170388e2f3cc1d4e1ad > committer: Sergey Vojtovich > timestamp: 2016-01-29 12:59:52 +0400 > message: > > MDEV-9488 - Table cache cleanups > > Cleanup now unused arguments of tdc_open_view(). ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 5f656d0: MDEV-9488 - Table cache cleanups
by Sergei Golubchik 24 Feb '16

24 Feb '16

Hi, Sergey! On Jan 29, Sergey Vojtovich wrote: > revision-id: 5f656d0dca8aae9486d91191ee6b6da6b9c552bf (mariadb-10.1.8-112-g5f656d0) > parent(s): 7b50447aa6d051b8d14bb01ef14802cb8ffee223 > committer: Sergey Vojtovich > timestamp: 2016-01-29 12:58:24 +0400 > message: > > MDEV-9488 - Table cache cleanups > > tdc_assign_new_table_id() does not relate to table cache, move it out of > table_cache.cc. ok to push Regards, Sergei

1 0

Re: [Maria-developers] 59e1aaa: MDEV-7331 - information_schema.user_variables
by Sergei Golubchik 24 Feb '16

24 Feb '16

Hi, Sergey! On Feb 24, Sergey Vojtovich wrote: > revision-id: 59e1aaa5f71cb725ce5242eb5abf2e00c6f5a793 (mariadb-10.1.8-124-g59e1aaa) > parent(s): 0485328d030f4b742dac7b667e8ed099beb9e9f2 > committer: Sergey Vojtovich > timestamp: 2016-02-24 14:31:43 +0400 > message: > > +static ST_FIELD_INFO user_variables_fields_info[] = > +{ > + { "VARIABLE_NAME", 64, MYSQL_TYPE_STRING, 0, 0, "Variable_name", 0 }, > + { "VARIABLE_VALUE", 2048, MYSQL_TYPE_STRING, 0, MY_I_S_MAYBE_NULL, "Value", 0 }, > + { 0, 0, MYSQL_TYPE_NULL, 0, 0, 0, 0 } > +}; 1. As discussed on irc, let's show everything we know about variables. Hmm, let me see what's in user_var_entry structure... The type, the character set, signed/unsigned. I think, query_id would be not very useful, though. See COLUMNS and SYSTEM_VARIABLES for appropriate column names. 2. You have defined old_name's for columns - "Variable_name" and "Value". This automatically enables SHOW USER_VARIABLES statement. See https://mariadb.com/kb/en/mariadb/information-schema-plugins-show-and-flush… You shouldn't enable SHOW if it displays exactly the same content as I_S.USER_VARIABLES. But if you'll add all the other metadata fields to the I_S table, then SHOW could be used to display a subset of the columns, e.g. only variable name and the value. Just as you have now :) 3. An interesting extension could be a FLUSH USER_VARIABLES statement that would empty the thd->user_vars hash. It's very easy to do, but I don't know if it'll be very useful :) See the url above for the API. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-7231 Field ROUTINE_DEFINITION in INFORMATION_SCHEMA.`ROUTINES` contains broken procedure body when used shielding quotes inside.
by Alexander Barkov 22 Feb '16

22 Feb '16

Hi Sergei, Please review a patch for MDEV-7231. Thanks.

2 4

[Maria-developers] Getting started
by Varun Gupta 19 Feb '16

19 Feb '16

Hey, I am Varun and this year I wish to take part in Google Summer of Code 2016 . I have been looking around the project ideas on the ideas page of MariaDB and am interested in the idea 'AGGREGATE STORE FUNCTIONS' . It would be helpful if someone could guide me where should i start looking to get more details related to this project idea. Regards

2 1

[Maria-developers] A proposal to deprecate syntax: SELECT a'test'
by Alexander Barkov 19 Feb '16

19 Feb '16

Hello Monty, all, We discussed this with Sergei and both think that it will be a good idea to deprecate this syntax: SELECT a'test'; where 'a' is an identifier and 'test' its alias. This is a non-standard way, and it conflicts with some other important standard SQL grammar. See below. Typified literals and a syntax conflict --------------------------------------- An identifier followed by a single-quoted text string is needed for typified literals: SELECT INET6'ffff::ffff'; We already support this for temporal literals, which is a part of the SQL standard: SELECT TIME'10:10:10'; SELECT TIMESTAMP'2001-01-01 10:20:30'; SELECT DATE'2001-01-01'; Note, the conflict already exists in MySQL and MariaDB. This script: DROP TABLE IF EXISTS t1; CREATE TABLE t1 (date date); INSERT INTO t1 VALUES ('2016-02-19'); SELECT date'2001-01-01' FROM t1; returns: +------------------+ | date'2001-01-01' | +------------------+ | 2001-01-01 | +------------------+ Notice, it returns a DATE literal, it does not return the value of the column t1.date using '2001-01-01' as an alias. Notice the difference: SELECT date`2001-01-01` FROM t1; +------------+ | 2001-01-01 | +------------+ | 2016-02-19 | +------------+ This is confusing. Shift-reduce conflicts ---------------------- By the way, the fact that we support single-quoted string as an alias is the reason for shift-reduce conflicts in sql_yacc.yy, because TIME/DATE/TIMESTAMP followed by 'string' can be interpreted in two ways: - a column name followed by an alias - an SQL-standard time literal What can be used instead ------------------------ We have a number of other ways to specify aliases: The standard ways: SELECT a AS test; SELECT a "test"; SELECT a AS "test"; MySQL/MariaDB extensions: SELECT a `test`; SELECT a AS `test`; That should be enough. Proposal -------- 1. In 10.1 we add a warning when a single-quoted string is used as a column alias. Something like this should work: Single quotes in a select sublist alias are deprecated. Use double quotes instead. 2. In 10.2 we disallow this syntax by default and add either sql_mode or old_mode to enable the old behavior. Sergei thinks that old_mode is something that should sooner or later be completely deleted. So perhaps sql_mode is better here, as the old behavior might be needed forever for MySQL compatibility. We'll appreciate your feedback. Thanks.

1 0

[Maria-developers] Google Summer of Code 2016
by Colin Charles 18 Feb '16

18 Feb '16

Hi! Its that time of year again and I believe that we plan to participate in Google Summer of Code 2016 again (we did so in 2013, 2014, and 2015). We are of course waiting to get accepted (we will know on Feb 29 2016) but that should not stop you, a *potential mentor* from adding your idea to the idea’s list page: https://mariadb.com/kb/en/mariadb/google-summer-of-code-2016/ Potential mentors should also see if there are Jira tasks they’d like completed: https://mariadb.atlassian.net/issues/?jql=labels%3Dgsoc16 (add more by adding the gsoc16 label) *Students* — please note that we have *not* been accepted *yet*, so asking us if you can do a project now is not necessarily useful (once we are accepted, then yes, please propose your interest) Here’s to Google Summer of Code 2016! cheers, -colin -- Colin Charles, http://bytebot.net/blog/ twitter: @bytebot | skype: colincharles "First they ignore you, then they laugh at you, then they fight you, then you win." -- Mohandas Gandhi

1 0

Re: [Maria-developers] [Commits] fd8e846: MDEV-9564 - added s390x to lib64 INSTALL_LIBDIR handling
by Sergei Golubchik 18 Feb '16

18 Feb '16

Hi, Sergey! Ok to push. I didn't like it at first, as x86_64 is amd64 on FreeBSD, x64 on Windows, and I don't know what else can there be. For example, FreeBSD doesn't use /usr/lib64. But your change is for RPM packages only - so it's fine. On Feb 18, Sergey Vojtovich wrote: > revision-id: fd8e846a3b049903706267d58e6d8e61eea97df8 (mariadb-10.1.11-17-gfd8e846) > parent(s): db5b51fb7e3360d4ebc303bd67363bc8324e86b6 > committer: Sergey Vojtovich > timestamp: 2016-02-18 11:01:22 +0400 > message: > > MDEV-9564 - added s390x to lib64 INSTALL_LIBDIR handling > > Adjusted INSTALL_LIBDIR detection so that it is set to "lib64" on any 64bit > system (not only x86_64). New condition is insipired by GNUInstallDirs cmake > module. > > --- > cmake/install_layout.cmake | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/cmake/install_layout.cmake b/cmake/install_layout.cmake > index 757166b..d1a261f 100644 > --- a/cmake/install_layout.cmake > +++ b/cmake/install_layout.cmake > @@ -137,7 +137,7 @@ SET(INSTALL_SCRIPTDIR_RPM "bin") > SET(INSTALL_SYSCONFDIR_RPM "/etc") > SET(INSTALL_SYSCONF2DIR_RPM "/etc/my.cnf.d") > # > -IF(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64") > +IF(CMAKE_SIZEOF_VOID_P EQUAL 8) > SET(INSTALL_LIBDIR_RPM "lib64") > SET(INSTALL_PLUGINDIR_RPM "lib64/mysql/plugin") > ELSE() Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Window functions code, MDEV-9543 and MDEV-8646 are now merged
by Sergey Petrunia 15 Feb '16

15 Feb '16

Hello, Following the Thursday discussion, I've managed to re-base 10.2-window_simple branch onto the bb-10.2-mdev9543. The result is now pushed into bb-10.2-mdev9543. https://github.com/MariaDB/server/commits/bb-10.2-mdev9543 Since rebase happened, 10.2-window_simple tree should be abandoned. Let's try to avoid re-bases and "take everything and re-commit" steps from now on. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] bb-10.2-mdev9543 is missing item_windowfunc.*
by Sergey Petrunia 13 Feb '16

13 Feb '16

Hello Igor, The tree at bb-10.2-mdev9543 is missing item_windowfunc.* files. Could you please add them there and push? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MDEV-9541 Selecting a time using a datetime fails
by Alexander Barkov 12 Feb '16

12 Feb '16

Hi Monty, Please have a look into the latest comments in MDEV-9541 on how PostgreSQL, SQL Server, Oracle 11i and MySQL-5.7 work when comparing temporal data types. There is no a general rule. Every vendor implements it in a different way. 1. PostgreSQL almost follows the standard, which disallows direct comparison of TIME and TIMESTAMP/DATETIME. The only exception, it allows to compare a TIME field to a string literal which looks like TIMESTAMP: SELECT * FROM t1 WHERE a='2001-01-01 10:20:30'; a | b ----------+--------------------- 10:20:30 | 2001-01-01 10:20:30 But strangely not the other way around! Comparing of a TIMESTAMP field to a string which looks like TIME does not work. I did not find an explanation for this. 2. SQL Server - Comparison of TIME + VARCHAR is done as TIME - Comparison of DATETIME + VARCHAR is done as DATETIME - Comparison of TIME and DATETIME is not allowed - Comparison of TIME and INT is not allowed - Comparison of DATETIME and INT is allowed (not sure how exactly it works though) - Temporal literals {t'10:20:30'} have data type VARCHAR, this is just a syntactic variation of a string literal '10:20:30', with no any additional difference. 3. Oracle 11i Oracle does not have the TIME data type. So mixture of TIME and TIMESTAMP is not possible. But comparison of TIMESTAMP and VARCHAR is always done as TIMESTAMP, no matter what kind expressions are involved (fields, functions, literals). 4. MySQL-5.7 It has a special code: for a TIME field compared to a non-DATE constant, the constant argument is converted to TIME. But as soon as you wrap the field say into COALESCE, or put a non-constant instead of a constant, this works in a different way. So, comparing of a TIME field to a string literal that looks like a TIMESTAMP/DATETIME works similar in PostgreSQL, SQL Server, MySQL-5.7. However, this looks only as a coincidence for this particular corner case. Perhaps we should not fix that in 10.0. It seems that MariaDB since 5.3 handles comparison of TIME in a more consistent way (less exceptions). Btw, why not to add a warning every time when an implicit non-standard type conversion happens and suggest an explicit CAST? This will encourage users to avoid doubtful constructions. Also, I noticed that the SQL Server documentation says that comparison of different data types is a subject to change in the future versions, and using this is not recommended. Perhaps we should also say something like this. I propose instead of adding a quick fix for this exception, to introduce a conception of "type strength" or "type derivation" in 10.2. The idea is simple: 1. Comparison is done according to the type of the strongest side. 2. Strengths(derivations) are as follows: - A field or an SP variable has the strongest type derivation, say 4. - Functions have derivation between field and typified constants, say 3. - Some functions that return input arguments (e.g. COALESCE, LEFT) can repeat derivations of their arguments. - A typified constant like TIME'10:20:30' has derivation 2. - A typeless constant like '10:20:30' has a very weak derivation, say 1. - An explicit NULL has the weakest derivation, say 0. 3. Only in case of equal strengths the result depends on the data types. Perhaps data types can have some sort of "minor strength", for conflict resolution. Say DATETIME/TIMESTAMP can have more minor strength over TIME or DATE. So a DATETIME field is compared to a TIME field as DATETIME rather than TIME. This will give some advantages: - Universal comparison rules for all data types - This idea will simplify adding new data types a lot. They will just follow the universal rule. Otherwise, new data types (say INET6), we'll have to have different execution paths with some exceptions and exceptions from exceptions :) - The problem mentioned in MDEV-9541 will be gone: it will compare as TIME just because it's a field compared to a typeless constant. - We'll remove a special code covering comparison of YEAR and LONGLONG fields. They will just work as before, but now because a field always wins a constant, according to the universal rules. - We'll probably remove some examples of a special code covering explicit NULLs (i.e. Item::NULL_ITEM). - A field compared to a constant will always be able to use indexes. Less chances to put the server into a heavy load because of a non-smart query executed. - The same results for a field compared to some expression, and for the same field wrapped into a hybrid function like COALESCE. - Hybrid functions with field arguments will also be able to use indexes in some cases. - Equal field propagation will be possible in more cases. - It should be easier to add a new SQL mode for the strict SQL standard compliant comparison style. I think it's good to have this option. Greetings.

1 0

[Maria-developers] MDEV-9371 select insert('a', 2, 1, 'b') doesn't return expected 'a'
by Alexander Barkov 10 Feb '16

10 Feb '16

Hello Sergei, Please review a fix for MDEV-9371. I'd suggest to push it to 10.0 instead of 5.5, as MySQL fixed this bug in 5.6 (not 5.5). Also, I think it's worthy to backport MDEV-9332 to 10.0 as well, because the patch for MDEV-9371 touches the test which we modified in MDEV-9332. Thanks!

2 2

Re: [Maria-developers] [Commits] 841c317: MDEV-7122: Assertion `0' failed in subselect_hash_sj_engine::init
by Sergey Petrunia 09 Feb '16

09 Feb '16

Hi Vicentiu, Please find some cosmetic feedback below. Ok to push after it is addressed. On Tue, Feb 09, 2016 at 04:02:58PM +0200, Vicentiu Ciorbaru wrote: > revision-id: 841c317d237453f3e6c253e2a884cad73622b59a (mariadb-5.5.47-33-g841c317) > parent(s): ad94790f468bacb2cd62d02d3a6e7f012ca72e65 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2016-02-09 16:00:18 +0200 > message: > > MDEV-7122: Assertion `0' failed in subselect_hash_sj_engine::init > > The select mentioned in the bug attempted to create a temporary table > using the maria storage engine. The table needs to have primary keys such that > duplicates can be removed. Unfortunately this use case has a longer > than allowed key and the tmp table got created without a temporary key. > We must not allow materialization for the subquery if the total key > length and key parts is greater than what the storage engine supports. > > --- > mysql-test/r/subselect.result | 12 ++++++++++++ > mysql-test/r/subselect_no_mat.result | 12 ++++++++++++ > mysql-test/r/subselect_no_opts.result | 12 ++++++++++++ > mysql-test/r/subselect_no_scache.result | 12 ++++++++++++ > mysql-test/r/subselect_no_semijoin.result | 12 ++++++++++++ > mysql-test/t/subselect.test | 9 +++++++++ > sql/opt_subselect.cc | 10 ++++++++++ > sql/sql_select.cc | 25 +++++++++++++++++++++++++ > 8 files changed, 104 insertions(+) > > diff --git a/mysql-test/r/subselect.result b/mysql-test/r/subselect.result > index ba52fd9..27e8eaa 100644 > --- a/mysql-test/r/subselect.result > +++ b/mysql-test/r/subselect.result > @@ -7081,3 +7081,15 @@ sq > NULL > deallocate prepare stmt; > drop table t1,t2,t3,t4; > +# MDEV-7122 > +# Assertion `0' failed in subselect_hash_sj_engine::init > +# > +SET SESSION big_tables=1; Please unset the variable back at the end of this testcase. We don't want the rest of the test to run with non-standard settings. > +CREATE TABLE t1(a char(255) DEFAULT '', KEY(a(10))) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; > +INSERT INTO t1 VALUES(0),(0),(0); > +SELECT * FROM t1 WHERE a IN(SELECT MIN(a) FROM t1); > +a > +0 > +0 > +0 > +DROP TABLE t1; > diff --git a/mysql-test/r/subselect_no_mat.result b/mysql-test/r/subselect_no_mat.result > index d8d5c4e..4727f9f 100644 > --- a/mysql-test/r/subselect_no_mat.result > +++ b/mysql-test/r/subselect_no_mat.result > @@ -7078,6 +7078,18 @@ sq > NULL > deallocate prepare stmt; > drop table t1,t2,t3,t4; > +# MDEV-7122 > +# Assertion `0' failed in subselect_hash_sj_engine::init > +# > +SET SESSION big_tables=1; > +CREATE TABLE t1(a char(255) DEFAULT '', KEY(a(10))) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; > +INSERT INTO t1 VALUES(0),(0),(0); > +SELECT * FROM t1 WHERE a IN(SELECT MIN(a) FROM t1); > +a > +0 > +0 > +0 > +DROP TABLE t1; > set optimizer_switch=default; > select @@optimizer_switch like '%materialization=on%'; > @@optimizer_switch like '%materialization=on%' > diff --git a/mysql-test/r/subselect_no_opts.result b/mysql-test/r/subselect_no_opts.result > index 78908bc..7c542dd 100644 > --- a/mysql-test/r/subselect_no_opts.result > +++ b/mysql-test/r/subselect_no_opts.result > @@ -7076,4 +7076,16 @@ sq > NULL > deallocate prepare stmt; > drop table t1,t2,t3,t4; > +# MDEV-7122 > +# Assertion `0' failed in subselect_hash_sj_engine::init > +# > +SET SESSION big_tables=1; > +CREATE TABLE t1(a char(255) DEFAULT '', KEY(a(10))) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; > +INSERT INTO t1 VALUES(0),(0),(0); > +SELECT * FROM t1 WHERE a IN(SELECT MIN(a) FROM t1); > +a > +0 > +0 > +0 > +DROP TABLE t1; > set @optimizer_switch_for_subselect_test=null; > diff --git a/mysql-test/r/subselect_no_scache.result b/mysql-test/r/subselect_no_scache.result > index d1de4b8..3b58b3c 100644 > --- a/mysql-test/r/subselect_no_scache.result > +++ b/mysql-test/r/subselect_no_scache.result > @@ -7087,6 +7087,18 @@ sq > NULL > deallocate prepare stmt; > drop table t1,t2,t3,t4; > +# MDEV-7122 > +# Assertion `0' failed in subselect_hash_sj_engine::init > +# > +SET SESSION big_tables=1; > +CREATE TABLE t1(a char(255) DEFAULT '', KEY(a(10))) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; > +INSERT INTO t1 VALUES(0),(0),(0); > +SELECT * FROM t1 WHERE a IN(SELECT MIN(a) FROM t1); > +a > +0 > +0 > +0 > +DROP TABLE t1; > set optimizer_switch=default; > select @@optimizer_switch like '%subquery_cache=on%'; > @@optimizer_switch like '%subquery_cache=on%' > diff --git a/mysql-test/r/subselect_no_semijoin.result b/mysql-test/r/subselect_no_semijoin.result > index 524a5dd..9246ffb 100644 > --- a/mysql-test/r/subselect_no_semijoin.result > +++ b/mysql-test/r/subselect_no_semijoin.result > @@ -7076,5 +7076,17 @@ sq > NULL > deallocate prepare stmt; > drop table t1,t2,t3,t4; > +# MDEV-7122 > +# Assertion `0' failed in subselect_hash_sj_engine::init > +# > +SET SESSION big_tables=1; > +CREATE TABLE t1(a char(255) DEFAULT '', KEY(a(10))) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; > +INSERT INTO t1 VALUES(0),(0),(0); > +SELECT * FROM t1 WHERE a IN(SELECT MIN(a) FROM t1); > +a > +0 > +0 > +0 > +DROP TABLE t1; > set @optimizer_switch_for_subselect_test=null; > set @join_cache_level_for_subselect_test=NULL; > diff --git a/mysql-test/t/subselect.test b/mysql-test/t/subselect.test > index 67eac0d..5ca21ea 100644 > --- a/mysql-test/t/subselect.test > +++ b/mysql-test/t/subselect.test > @@ -5964,3 +5964,12 @@ EXECUTE stmt; > > deallocate prepare stmt; > drop table t1,t2,t3,t4; > + > +--echo # MDEV-7122 > +--echo # Assertion `0' failed in subselect_hash_sj_engine::init > +--echo # > +SET SESSION big_tables=1; > +CREATE TABLE t1(a char(255) DEFAULT '', KEY(a(10))) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; > +INSERT INTO t1 VALUES(0),(0),(0); > +SELECT * FROM t1 WHERE a IN(SELECT MIN(a) FROM t1); > +DROP TABLE t1; > diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc > index 827290f..a39e04b 100644 > --- a/sql/opt_subselect.cc > +++ b/sql/opt_subselect.cc > @@ -830,12 +830,14 @@ bool subquery_types_allow_materialization(Item_in_subselect *in_subs) > in_subs->sjm_scan_allowed= FALSE; > > bool all_are_fields= TRUE; > + uint32 total_key_length = 0; > for (uint i= 0; i < elements; i++) > { > Item *outer= in_subs->left_expr->element_index(i); > Item *inner= it++; > all_are_fields &= (outer->real_item()->type() == Item::FIELD_ITEM && > inner->real_item()->type() == Item::FIELD_ITEM); > + total_key_length += inner->max_length; > if (outer->cmp_type() != inner->cmp_type()) > DBUG_RETURN(FALSE); > switch (outer->cmp_type()) { > @@ -866,6 +868,14 @@ bool subquery_types_allow_materialization(Item_in_subselect *in_subs) > } > } > > + /* > + Make sure that create_tmp_table will not fail due to too long keys. > + See MDEV-7122. This check is performed inside create_tmp_table also and > + we must do it so that we know the table has keys created. > + */ > + if (total_key_length > HA_MAX_KEY_LENGTH || elements > HA_MAX_KEY_SEG) > + DBUG_RETURN(FALSE); > + > in_subs->types_allow_materialization= TRUE; > in_subs->sjm_scan_allowed= all_are_fields; > DBUG_PRINT("info",("subquery_types_allow_materialization: ok, allowed")); > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index f43f506..218a4ce 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -15307,6 +15307,13 @@ create_tmp_table(THD *thd, TMP_TABLE_PARAM *param, List<Item> &fields, > if (!table->file) > goto err; > > + /* > + Temporary table storage engines must allow keys of at least > + HA_MAX_KEY_LENGT and at least HA_MAX_KEY_SEG key parts. psergey: typo. ^^^^^^^^^^^ > + */ > + DBUG_ASSERT(table->file->max_key_length() >= HA_MAX_KEY_LENGTH && > + table->file->max_key_parts() >= HA_MAX_KEY_SEG); > + > if (!using_unique_constraint) > reclength+= group_null_items; // null flag is stored separately > > @@ -15940,6 +15947,12 @@ bool create_internal_tmp_table(TABLE *table, KEY *keyinfo, > goto err; > > bzero(seg, sizeof(*seg) * keyinfo->key_parts); > + /* > + Note that a simillar check is performed during typo ^^^^^^^^^ > + subquery_types_allow_materialization. See MDEV-7122 for more details as > + to why. Whenever this changes, it must be updated there as well, for > + all tmp_table engines. > + */ > if (keyinfo->key_length > table->file->max_key_length() || > keyinfo->key_parts > table->file->max_key_parts() || > share->uniques) > @@ -16126,6 +16139,12 @@ bool create_internal_tmp_table(TABLE *table, KEY *keyinfo, > goto err; > > bzero(seg, sizeof(*seg) * keyinfo->key_parts); > + /* > + Note that a simillar check is performed during typo ^^^^^^^^^ > + subquery_types_allow_materialization. See MDEV-7122 for more details as > + to why. Whenever this changes, it must be updated there as well, for > + all tmp_table engines. > + */ > if (keyinfo->key_length > table->file->max_key_length() || > keyinfo->key_parts > table->file->max_key_parts() || > share->uniques) > @@ -16286,6 +16305,12 @@ create_internal_tmp_table_from_heap2(THD *thd, TABLE *table, > if (!(new_table.file= get_new_handler(&share, &new_table.mem_root, > new_table.s->db_type()))) > DBUG_RETURN(1); // End of memory > + /* > + Temporary table storage engines must allow keys of at least > + HA_MAX_KEY_LENGTH and at least HA_MAX_KEY_SEG key parts. > + */ > + DBUG_ASSERT(new_table.file->max_key_length() >= HA_MAX_KEY_LENGTH && > + new_table.file->max_key_parts() >= HA_MAX_KEY_SEG); > > save_proc_info=thd->proc_info; > thd_proc_info(thd, proc_info); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MTR related community contribution
by Vicențiu Ciorbaru 07 Feb '16

07 Feb '16

Hi Vlad, I'm looking at the community pull request https://github.com/grooverdan/mariadb-server/commit/a9dfee63bad7393002621f9… I have a question regarding your commit: 2943d2b7e99b4221ef481890fc9a56ad569f3985 The contribution aims to remove the hard limit of 50 unique ids as it seems to cause problems for some use cases with a lot of proceeses. It does this by either leaving it at 50 or to be equal to the number of spawned processes. I'm thinking that even setting the limit of unique ids to num_processes might cause problems if there is another mtr running in parallel and both have > 50 processes. This is probably minor but why set this limit so low? Wouldn't a larger number (say 10k) be ok? Or why not generate a random number and lock a file with that number? (if the random number is invalid, generate a new one. On average it should finish quite quickly) Thanks, Vicentiu

3 2

[Maria-developers] Invalid code in server_audit.c
by Vicențiu Ciorbaru 06 Feb '16

06 Feb '16

Hi Alexei, While investigating community contributions from Daniel Black (MDEV-9433), I came across this comment: [image: danblack]Daniel Black <https://mariadb.atlassian.net/secure/ViewProfile.jspa?name=danblack> added a comment - 2016-01-20 02:54 [plugin/server_audit/server_audit.c:402]: (error) Invalid number of character '{' when these macros are defined: 'DBUG_VOID_RETURN=return;DBUG_RETURN(a)=return a;HAVE_PSI_INTERFACE'. [plugin/server_audit/server_audit.c:402]: (error) Invalid number of character '{' when these macros are defined: 'DBUG_VOID_RETURN=return;DBUG_RETURN(a)=return a;HAVE_PSI_INTERFACE;HUGETLB_USE_PROC_MEMINFO'. [plugin/server_audit/server_audit.c:402]: (error) Invalid number of character '{' when these macros are defined: 'DBUG_VOID_RETURN=return;DBUG_RETURN(a)=return a;HAVE_PSI_INTERFACE;USE_ALARM_THREAD'. Exists in 5.5 https://github.com/MariaDB/server/blob/5.5/plugin/server_audit/server_audit… Remove second {{ I've had a look at the code mentioned there and key_LOCK_bigbuffer does not exist anywhere else. Since we're not getting any compilation issues, it means we're not compiling that bit of code due to the defines. This indeed is so, as MYSQL_DYNAMIC_PLUGIN is defined for that file on the default cmake config which leads to FLOGGER being defined. I could not figure out a way to actually compile the code with defines such that that bit of code actually compiles. Either way it would not compile, due to the missing symbol. Do you think that we should just remove the line that mentions key_LOCK_bigbuffer? Also, since this is something we have not caught during testing, can we test for this compilation configuration somehow? Vicentiu

1 0

[Maria-developers] MariaDB JIRA is moving
by Rasmus Johansson 05 Feb '16

05 Feb '16

The MariaDB JIRA instance that currently is in use for project and issue tracking will change. The current instance is hosted in Atlassian's cloud and it has worked well, but we have hit the maximum user limit of 2000 users. It's fantastic to see how many of you actually report bugs and other issues in the MariaDB project! To hit that limit also means that we have to migrate over to a self-hosted instance of JIRA. Below are important details about that change. When will the switch happen: Sat 27th of Feb What is the impact for me as a user of MariaDB's JIRA: 1. URL change - The current instance is hosted in Atalassian's cloud and has the URL mariadb.atlassian.net. The new one will be found at jira.mariadb.org starting from 28th of Feb. 2. RSS feeds - If you have been using JIRA's RSS feeds, you'll need to set up them again since the URL is changing What will happen to old JIRA: The mariadb.atlassian.net JIRA instance will be put into read-only mode and a banner will be visible on each page telling the URL of the new MariaDB JIRA instance. It will remain this way for some time since it has been referenced quite heavily from other sites. source: https://mariadb.org/mariadb-jira-is-moving/ -- Rasmus Johansson, VP Engineering MariaDB | t: +358 50 499 9589 | Skype: ratzpo

1 0

[Maria-developers] MDEV-8691 Multiple test failures on mips and mipsel platforms, suites federated and rpl
by Alexander Barkov 02 Feb '16

02 Feb '16

Hi Jan, can you please have a look into: https://mariadb.atlassian.net/browse/MDEV-8691 Otto managed to get a stack trace from mips machine. It seems to crash inside InnoDB. Thanks. #3 <signal handler called> #4 0x000000fff6b48e80 in std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*) () from /usr/lib/mips64el-linux-gnuabi64/libstdc++.so.6 #5 0x00000001207cf878 in std::__cxx11::list<unsigned char*, std::allocator<unsigned char*> >::_M_insert (__x=<synthetic pointer>, __position=..., this=0x118) at /usr/include/c++/5/bits/stl_list.h:1755 #6 std::__cxx11::list<unsigned char*, std::allocator<unsigned char*> >::push_back (__x=<synthetic pointer>, this=0x118) at /usr/include/c++/5/bits/stl_list.h:1089 #7 recv_dblwr_t::add (this=0x118, page=0xffeb004000 "\377\060~$") at /«PKGBUILDDIR»/storage/xtradb/log/log0recv.cc:4077 #8 0x00000001208df7f8 in buf_dblwr_init_or_load_pages (file=<optimized out>, path=0xfff678b029 <srv_read_only_mode> "", load_corrupt_pages=<optimized out>) at /«PKGBUILDDIR»/storage/xtradb/buf/buf0dblwr.cc:457 #9 0x000000fff66e42bc in open_or_create_data_files (sum_of_new_sizes=<synthetic pointer>, max_flushed_lsn=0xffffa89a50, min_flushed_lsn=0xffffa89a48, create_new_db=<synthetic pointer>) at /«PKGBUILDDIR»/storage/innobase/srv/srv0start.cc:1033 #10 innobase_start_or_create_for_mysql () at /«PKGBUILDDIR»/storage/innobase/srv/srv0start.cc:2105 #11 0x000000fff65f0028 in innobase_init (p=<optimized out>) at /«PKGBUILDDIR»/storage/innobase/handler/ha_innodb.cc:3400 #12 0x00000001204a17bc in ha_initialize_handlerton (plugin=0xfff5cb49f0) at /«PKGBUILDDIR»/sql/handler.cc:501 #13 0x00000001202fcafc in plugin_initialize (tmp_root=0xffffa91c40, tmp_root@entry=0x0, plugin=plugin@entry=0xfff5cb49f0, argc=0x120fdf680 <remaining_argc>, argc@entry=0x6, argv=0xfff5c7cc58, argv@entry=0xfd8, options_only=<optimized out>) at /«PKGBUILDDIR»/sql/sql_plugin.cc:1379 #14 0x00000001202fd7b0 in plugin_init (argc=0x6, argv=0xfd8, flags=<optimized out>) at /«PKGBUILDDIR»/sql/sql_plugin.cc:1659 #15 0x00000001202347ec in init_server_components () at /«PKGBUILDDIR»/sql/mysqld.cc:4850 #16 0x000000012023b83c in mysqld_main (argc=255, argv=0xff00000000) at /«PKGBUILDDIR»/sql/mysqld.cc:5445 #17 0x000000012021cf18 in main (argc=<optimized out>, argv=<optimized out>) at /«PKGBUILDDIR»/sql/main.cc:25

2 2

Re: [Maria-developers] 9f2e2d8: [MDEV-7978] Update grammar for new syntax
by Sergei Golubchik 02 Feb '16

02 Feb '16

Hi, Vicentiu! On Jan 19, Vicentiu Ciorbaru wrote: > revision-id: 9f2e2d89917709f9223fe2d4b75dc86d0297b4c4 (mariadb-10.1.8-113-g9f2e2d8) > parent(s): 83e197aa6bd3a7eb81482795732bfbdb3b415deb > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2016-01-17 17:07:21 +0200 > message: > > [MDEV-7978] Update grammar for new syntax > > Extend the syntax accepted by the grammar to account for the new create user > and alter user syntax. a couple of comments: > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 3eb6a99..c1f4249 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -7279,6 +7280,11 @@ alter: > lex->sql_command= SQLCOM_ALTER_SERVER; > lex->server_options.reset($3); > } OPTIONS_SYM '(' server_options_list ')' { } > + | ALTER opt_if_exists USER clear_privileges user_list > + require_clause resource_options > + { please, add a comment that "ALTER USER foo;" is allowed for MySQL compatibility > + Lex->sql_command= SQLCOM_ALTER_USER; > + } > ; > > ev_alter_on_schedule_completion: > @@ -15734,6 +15724,37 @@ grant_option: > } > ; > > +resource_option_list: > + resource_option_list resource_option {} > + | resource_option {} > + ; > + > +resource_options: better rename it to opt_resource_options (and opt_grant_options below) for consistency. In sql_yacc.yy optional clauses often start with opt_ and one can immediately see that they're optional in, for example | ALTER opt_if_exists USER clear_privileges user_list opt_require_clause opt_resource_options > + /* empty */ {} > + | WITH resource_option_list > + ; > + > + > +grant_options: > + /* empty */ {} > + | WITH grant_option_list {} > + ; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-9415 Crash with ConnectSE + MySQL ODBC driver + INSERT
by Alexander Barkov 02 Feb '16

02 Feb '16

Hi Georg, Can you please have a look into: https://mariadb.atlassian.net/browse/MDEV-9415 ConnectSE crashes on INSERT statements inside the ODBC driver, while UPDATE, SELECT, DELETE work fine. Any ideas how to track down the problem? Thanks.

1 1

Re: [Maria-developers] [Commits] 5dd6eb4: MDEV-9117: Client Server capability negotiation for MariaDB specific functionality
by Sergei Golubchik 01 Feb '16

01 Feb '16

Hi, Sanja! On Jan 28, OleksandrByelkin wrote: > revision-id: 5dd6eb45c418f61c749a7884d7b3e13d62f3bfb9 (mariadb-10.1.8-113-g5dd6eb4) > parent(s): 14d79a5e99f9992cabb7489203989ccdaacadb78 > committer: Oleksandr Byelkin > timestamp: 2016-01-28 14:58:12 +0100 > message: > > MDEV-9117: Client Server capability negotiation for MariaDB specific functionality Just a couple of comments: > New capability flags space. > Removed old progress flag, added new one. > > diff --git a/client/mysql.cc b/client/mysql.cc > index 3e08c87..715eed8 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -152,7 +152,7 @@ static ulong opt_max_allowed_packet, opt_net_buffer_length; > static uint verbose=0,opt_silent=0,opt_mysql_port=0, opt_local_infile=0; > static uint my_end_arg; > static char * opt_mysql_unix_port=0; > -static int connect_flag=CLIENT_INTERACTIVE; > +static ulong connect_flag=CLIENT_INTERACTIVE; ulong width is not portable, remember trubles we had with ulong sysvars? better use ulonglong for 64bit and uint for 32bit. > static my_bool opt_binary_mode= FALSE; > static int interrupted_query= 0; > static char *current_host,*current_db,*current_user=0,*opt_password=0, > diff --git a/libmysql/client_settings.h b/libmysql/client_settings.h > index b233614..2577870 100644 > --- a/libmysql/client_settings.h > +++ b/libmysql/client_settings.h > @@ -27,7 +27,7 @@ extern char * mysql_unix_port; > When adding capabilities here, consider if they should be also added to > the server's version. > */ > -#define CLIENT_CAPABILITIES (CLIENT_LONG_PASSWORD | \ > +#define CLIENT_CAPABILITIES (CLIENT_MYSQL | \ really? CLIENT_MYSQL? Is it because they aren't using Connector/C yet? > CLIENT_LONG_FLAG | \ > CLIENT_TRANSACTIONS | \ > CLIENT_PROTOCOL_41 | \ > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 2bcd351..dcb12e6 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -11730,18 +11731,27 @@ static ulong parse_client_handshake_packet(MPVIO_EXT *mpvio, > */ > DBUG_ASSERT(net->read_pos[pkt_len] == 0); > > - ulong client_capabilities= uint2korr(net->read_pos); > + ulonglong client_capabilities= uint2korr(net->read_pos); > + compile_time_assert(sizeof(client_capabilities) >= 8); > if (client_capabilities & CLIENT_PROTOCOL_41) > { > - if (pkt_len < 4) > + if (pkt_len < 32) > return packet_error; > client_capabilities|= ((ulong) uint2korr(net->read_pos+2)) << 16; > + if (!(client_capabilities & CLIENT_MYSQL)) > + { > + // it is client with mariadb extensions > + ulonglong ext_client_capabilities= > + (((ulonglong)uint4korr(net->read_pos + 28)) << 32); > + client_capabilities|= ext_client_capabilities; > + } As I wrote earlier, please add here else if (client_capabilities & CLIENT_PROGRESS_OBSOLETE) client_capabilities|= MARIADB_CLIENT_PROGRESS; > } > > /* Disable those bits which are not supported by the client. */ > + compile_time_assert(sizeof(thd->client_capabilities) >= 8); > thd->client_capabilities&= client_capabilities; > > - DBUG_PRINT("info", ("client capabilities: %lu", thd->client_capabilities)); > + DBUG_PRINT("info", ("client capabilities 1: %llu", thd->client_capabilities)); why " 1" ? > if (thd->client_capabilities & CLIENT_SSL) > { > unsigned long errptr __attribute__((unused)); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] Error creating new list
by Sonali Ellath 01 Feb '16

01 Feb '16

Hi, I built mailman-bundler using the guidelines found here <https://gitlab.com/mailman/mailman-bundler>. Server started successfully and and I created few domains. It worked well but when I tried to create a list, I received HTTP Error 500: A server error occurred. Please contact the administrator. Traceback: Traceback (most recent call last): File "/usr/lib/python3.4/wsgiref/handlers.py", line 137, in run self.result = application(self.environ, self.start_response) File "github/mailman/mailman-bundler/venv-3.4/lib/python3.4/site-packages/mailman/database/transaction.py", line 57, in wrapper rtn = function(*args, **kws) File "github/mailman/mailman-bundler/venv-3.4/lib/python3.4/site-packages/mailman/rest/wsgiapp.py", line 65, in __call__ environ, start_response) File "github/mailman/mailman-bundler/venv-3.4/lib/python3.4/site-packages/falcon/api.py", line 182, in __call__ responder(req, resp, **params) File "github/mailman/mailman-bundler/venv-3.4/lib/python3.4/site-packages/mailman/rest/lists.py", line 206, in on_post mlist = create_list(**validator(request)) File "github/mailman/mailman-bundler/venv-3.4/lib/python3.4/site-packages/mailman/app/lifecycle.py", line 81, in create_list call_name(config.mta.incoming).create(mlist) File "github/mailman/mailman-bundler/venv-3.4/lib/python3.4/site-packages/mailman/mta/postfix.py", line 69, in create self.regenerate() File "github/mailman/mailman-bundler/venv-3.4/lib/python3.4/site-packages/mailman/mta/postfix.py", line 104, in regenerate raise RuntimeError(NL.join(errors)) RuntimeError: command failure: /usr/sbin/postmap github/mailman/mailman-bundler/var/data/postfix_lmtp, 127, Key has expired command failure: /usr/sbin/postmap github/mailman/mailman-bundler/var/data/postfix_domains, 127, Key has expired Domain Details: Mail Host: lists.example.com URL Host: http://lists.example.com Please Help! Thanks - Sonali

2 1

[Maria-developers] Non-bloking API problem on non-x86 (MDEV-9112)
by Kristian Nielsen 01 Feb '16

01 Feb '16

Hi Georg, I stumbled upon the problem in the server source tree that ucontext check is broken, causing the non-blocking library to be completely disabled on all non-x86 platforms. After creating my own patch, I noticed that you alrady made a patch for this in MDEV-9112, but it does not appear to be pushed yet. Do you still want to push your patch? The patch looks fine (you can take this as review), it handles a couple things I forgot in my own patch. I think this should definitely go in 10.0 (and 10.1) - but do you think it should go in 5.5 also (not sure)? - Kristian. PS. I supplied a patch in CONC-153 for the similar problem in mariadb-connector-c, but maybe you are already aware of that as well. It was BTW cool to see that an application of mine (originally written against libmysqlclient) worked without changes on mariadb-connector-c.

1 1

Re: [Maria-developers] [Commits] 2a181ea: MDEV-8713 Add continuous binary log backup to mysqlbinlog.
by Sergei Golubchik 31 Jan '16

31 Jan '16

Hi, Alexey! On Nov 27, Alexey Botchkov wrote: > revision-id: 2a181ea7b16dcbc6dd30af5a2396760261d7d500 (mariadb-10.1.8-81-g2a181ea) > parent(s): 9d5c9379a6fd3662d4388cda177a3b1ea7b070cc > committer: Alexey Botchkov > timestamp: 2015-11-27 16:33:23 +0400 > message: > > MDEV-8713 Add continuous binary log backup to mysqlbinlog. > diff --git a/client/client_priv.h b/client/client_priv.h > index 656c8fc..44f0b21 100644 > --- a/client/client_priv.h > +++ b/client/client_priv.h > @@ -92,6 +92,7 @@ enum options_client > OPT_REPORT_PROGRESS, > OPT_SKIP_ANNOTATE_ROWS_EVENTS, > OPT_SSL_CRL, OPT_SSL_CRLPATH, > + OPT_RAW_OUTPUT, OPT_WAIT_SERVER_ID, OPT_STOP_NEVER, 1. There's no need to create named constants for OPT_RAW_OUTPUT and OPT_WAIT_SERVER_ID - they are never used anywhere. Simply put 0 in the my_options structure instead. 2. OPT_STOP_NEVER is used in the switch, so it needs a named constant. but it still doesn't have to be in client_priv.h, it's not an option common to all clients. Define it in mysqlbinlog.cc (see how mysqltest.cc and mysql_upgrade.c do it) > OPT_MAX_CLIENT_OPTION /* should be always the last */ > }; > > diff --git a/client/mysqlbinlog.cc b/client/mysqlbinlog.cc > index 5b07232..a2509b7 100644 > --- a/client/mysqlbinlog.cc > +++ b/client/mysqlbinlog.cc > @@ -96,6 +98,8 @@ static char* database= 0; > static my_bool force_opt= 0, short_form= 0, remote_opt= 0; > static my_bool debug_info_flag, debug_check_flag; > static my_bool force_if_open_opt= 1; > +static my_bool opt_raw_mode= 0, opt_stop_never= 0; > +static int64 opt_stop_never_slave_server_id= -1; longlong, not int64. Because 1) you've used GET_LL for it, and 2) we generally don't use int64 (it's only used in special cases to mean exactly "signed 64-bit integer"). > static my_bool opt_verify_binlog_checksum= 1; > static ulonglong offset = 0; > static char* host = 0; > @@ -1368,8 +1373,14 @@ static struct my_option my_options[] = > {"read-from-remote-server", 'R', "Read binary logs from a MySQL server.", > &remote_opt, &remote_opt, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, > 0, 0}, > - {"result-file", 'r', "Direct output to a given file.", 0, 0, 0, GET_STR, > - REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, > + {"raw", OPT_RAW_OUTPUT, "Requires -R. Output raw binlog data instead of SQL " > + "statements and write it to files corresponding to server logs.", What does that mean? "write it to files corresponding to server logs"? Perhaps you could clarify the help text? > + &opt_raw_mode, &opt_raw_mode, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, > + 0, 0}, > + {"result-file", 'r', "Direct output to a given file. With --raw this is a " > + "prefix for the file names.", > + &result_file_name, &result_file_name, 0, GET_STR, REQUIRED_ARG, > + 0, 0, 0, 0, 0, 0}, > {"server-id", 0, > "Extract only binlog entries created by the server having the given id.", > &server_id, &server_id, 0, GET_ULONG, > @@ -1418,6 +1429,16 @@ static struct my_option my_options[] = > "(you should probably use quotes for your shell to set it properly).", > &stop_datetime_str, &stop_datetime_str, > 0, GET_STR_ALLOC, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, > + {"stop-never", OPT_STOP_NEVER, "Wait for more data from the server " > + "instead of stopping at the end of the last log. Implicitly sets " > + "--to-last-log but instead of stopping at the end of the last log " > + "it continues to wait till the server disconnects.", I don't know why you needed to say the same thing twice. What about: Wait for more data from the server instead of stopping at the end of the last log. Implies --to-last-log. > + &opt_stop_never, &opt_stop_never, 0, > + GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, > + {"stop-never-slave-server-id", OPT_WAIT_SERVER_ID, > + "The slave server_id used for --read-from-remote-server --stop-never.", > + &opt_stop_never_slave_server_id, &opt_stop_never_slave_server_id, 0, > + GET_LL, REQUIRED_ARG, -1, -1, 0xFFFFFFFFLL, 0, 0, 0}, GET_LL, really? So, negative server_id's are allowed? Either do it like the existing --server-id option (GET_ULONG, 0 means "not set"), or, if you want to allow 0 as a valid server-id value, you can do, like case OPT_WAIT_SERVER_ID: if (argument == disabled_my_option) opt_stop_never_slave_server_id= -1; break; but really I wouldn't bother and just do like --server-id does. > {"stop-position", OPT_STOP_POSITION, > "Stop reading the binlog at position N. Applies to the last binlog " > "passed on the command line.", > @@ -1701,6 +1718,9 @@ get_one_option(int optid, const struct my_option *opt __attribute__((unused)), > print_version(); > opt_version= 1; > break; > + case OPT_STOP_NEVER: > + to_last_remote_log= 1; > + break; This is wrong, it allows someone to do mysqlbinlog --stop-never --to-last-remote-log=0 and you get incorrect configuration. Alternatively one can do mysqlbinlog --stop-never=1 --stop-never=0 and to_last_remote_log will be magically enabled. Heck, one can even do mysqlbinlog --stop-never=0 and that'll enable to_last_remote_log. The correct approach is to do if (opt_stop_never) to_last_remote_log= TRUE; and only *after* all options are read. Note that in this case you won't need OPT_STOP_NEVER constant either (see my comment earlier in this review) > case '?': > usage(); > opt_version= 1; > @@ -1922,6 +1943,240 @@ static Exit_status check_master_version() > } > > + > + > +static char out_file_name[FN_REFLEN + 1]; > + > +static Exit_status handle_event_raw_mode(PRINT_EVENT_INFO *print_event_info, > + ulong *len, > + const char* logname, uint logname_len) > +{ I'm not reviewing the logic of handle_event_raw_mode, as far as I can see you've copied it from mysqlbinlog in MySQL. Did you use the latest version of mysqlbinlog or the one from the original patch? Use the latest one, it might have bugfixes. > + const char *error_msg; > + const unsigned char *read_pos= mysql->net.read_pos + 1; > + Log_event_type type; > + DBUG_ENTER("handle_event_raw_mode"); > + DBUG_ASSERT(opt_raw_mode && remote_opt); > + > + type= (Log_event_type) read_pos[EVENT_TYPE_OFFSET]; > + > + if (type == HEARTBEAT_LOG_EVENT) > + DBUG_RETURN(OK_CONTINUE); > + > + if (type == ROTATE_EVENT || type == FORMAT_DESCRIPTION_EVENT) > + { > + Log_event *ev; > + if (!(ev= Log_event::read_log_event((const char*) read_pos , > + *len - 1, &error_msg, > + glob_description_event, > + opt_verify_binlog_checksum))) > + { > + error("Could not construct annotate event object: %s", error_msg); may be: "%s", type == ROTATE_EVENT ? "rotate":"annotate" > + DBUG_RETURN(ERROR_STOP); > + } > + /* > + If reading from a remote host, ensure the temp_buf for the > + Log_event class is pointing to the incoming stream. > + */ > + ev->register_temp_buf((char *) read_pos, FALSE); > + > @@ -2000,113 +2264,23 @@ static Exit_status dump_remote_log_entries(PRINT_EVENT_INFO *print_event_info, > break; // end of data > DBUG_PRINT("info",( "len: %lu net->read_pos[5]: %d\n", > len, net->read_pos[5])); > - if (net->read_pos[5] == ANNOTATE_ROWS_EVENT) > + if (opt_raw_mode) > { > - if (!(ev= read_remote_annotate_event(net->read_pos + 1, len - 1, > - &error_msg))) > - { > - error("Could not construct annotate event object: %s", error_msg); > - DBUG_RETURN(ERROR_STOP); > - } > + retval= handle_event_raw_mode(print_event_info, &len, > + logname, logname_len); > } > else > { > - if (!(ev= Log_event::read_log_event((const char*) net->read_pos + 1 , > - len - 1, &error_msg, > - glob_description_event, > - opt_verify_binlog_checksum))) > - { > - error("Could not construct log event object: %s", error_msg); > - DBUG_RETURN(ERROR_STOP); > - } > - /* > - If reading from a remote host, ensure the temp_buf for the > - Log_event class is pointing to the incoming stream. > - */ > - ev->register_temp_buf((char *) net->read_pos + 1, FALSE); > + retval= handle_event_text_mode(print_event_info, &len, > + logname, logname_len, old_off); > } > - > - Log_event_type type= ev->get_type_code(); > - if (glob_description_event->binlog_version >= 3 || > - (type != LOAD_EVENT && type != CREATE_FILE_EVENT)) > + if (retval != OK_CONTINUE) > { > - /* > - If this is a Rotate event, maybe it's the end of the requested binlog; > - in this case we are done (stop transfer). > - This is suitable for binlogs, not relay logs (but for now we don't read > - relay logs remotely because the server is not able to do that). If one > - day we read relay logs remotely, then we will have a problem with the > - detection below: relay logs contain Rotate events which are about the > - binlogs, so which would trigger the end-detection below. > - */ > - if (type == ROTATE_EVENT) > - { > - Rotate_log_event *rev= (Rotate_log_event *)ev; > - /* > - If this is a fake Rotate event, and not about our log, we can stop > - transfer. If this a real Rotate event (so it's not about our log, > - it's in our log describing the next log), we print it (because it's > - part of our log) and then we will stop when we receive the fake one > - soon. > - */ > - if (rev->when == 0) > - { > - if (!to_last_remote_log) > - { > - if ((rev->ident_len != logname_len) || > - memcmp(rev->new_log_ident, logname, logname_len)) > - { > - DBUG_RETURN(OK_CONTINUE); > - } > - /* > - Otherwise, this is a fake Rotate for our log, at the very > - beginning for sure. Skip it, because it was not in the original > - log. If we are running with to_last_remote_log, we print it, > - because it serves as a useful marker between binlogs then. > - */ > - continue; > - } > - len= 1; // fake Rotate, so don't increment old_off > - } > - } > - else if (type == FORMAT_DESCRIPTION_EVENT) > - { > - /* > - This could be an fake Format_description_log_event that server > - (5.0+) automatically sends to a slave on connect, before sending > - a first event at the requested position. If this is the case, > - don't increment old_off. Real Format_description_log_event always > - starts from BIN_LOG_HEADER_SIZE position. > - */ > - if (old_off != BIN_LOG_HEADER_SIZE) > - len= 1; // fake event, don't increment old_off > - } > - Exit_status retval= process_event(print_event_info, ev, old_off, logname); > - if (retval != OK_CONTINUE) > - DBUG_RETURN(retval); > + if (retval == OK_EOF) > + break; > + DBUG_RETURN(retval); > } > - else > - { > - Load_log_event *le= (Load_log_event*)ev; > - const char *old_fname= le->fname; > - uint old_len= le->fname_len; > - File file; > - Exit_status retval; > > - if ((file= load_processor.prepare_new_file_for_old_format(le,fname)) < 0) > - DBUG_RETURN(ERROR_STOP); > - > - retval= process_event(print_event_info, ev, old_off, logname); > - if (retval != OK_CONTINUE) > - { > - my_close(file,MYF(MY_WME)); > - DBUG_RETURN(retval); > - } > - retval= load_processor.load_old_format_file(net,old_fname,old_len,file); > - my_close(file,MYF(MY_WME)); > - if (retval != OK_CONTINUE) > - DBUG_RETURN(retval); > - } Hmm, when you moved all that code above into handle_event_text_mode(), you seem to have changed the behavior for to_last_remote_log. Was (comments removed for brevity): if (!to_last_remote_log) { if ((rev->ident_len != logname_len) || memcmp(rev->new_log_ident, logname, logname_len)) { DBUG_RETURN(OK_CONTINUE); } continue; } Now instead of 'continue' you return OK_CONTINUE from handle_event_text_mode() and it does not skip the code below (after the comment Let's adjust), like the old 'continue' did. > /* > Let's adjust offset for remote log as for local log to produce > similar text and to have --stop-position to work identically. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Aliasing slave_parallel_workers/slave_parallel_threads
by Colin Charles 29 Jan '16

29 Jan '16

Hi Kristian, I am referencing an old thread from March 2014 here where you don’t have an objection, but maybe this has slipped: https://lists.launchpad.net/maria-developers/msg06923.html MySQL 5.7 has been released and I’m sure you realise that it uses the variable slave_parallel_workers (http://dev.mysql.com/doc/refman/5.7/en/replication-options-slave.html#sysva…) In MariaDB Server 10.1 we use slave_parallel_threads (https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-…) From a DBA standpoint, and also from people using both databases in testing/production, this is confusing and scripts now have to have unnecessary added logic for if MariaDB or else MySQL. It would be wise for us to alias this, so that its easier from a DBA usability standpoint IMHO I planned to open up a Jira request, but since this was discussed on the list before I thought I would ask again to revive an old thread. cheers, -colin -- Colin Charles, http://bytebot.net/blog/ twitter: @bytebot | skype: colincharles "First they ignore you, then they laugh at you, then they fight you, then you win." -- Mohandas Gandhi

4 6

Re: [Maria-developers] [Commits] 56ae997: [MDEV-7978] Added show create user implementation.
by Sergei Golubchik 29 Jan '16

29 Jan '16

Hi, Vicentiu! Below are my comments on the combined patch: > diff --git a/mysql-test/r/alter_user.result b/mysql-test/r/alter_user.result > new file mode 100644 > index 0000000..2acc249 > --- /dev/null > +++ b/mysql-test/r/alter_user.result > @@ -0,0 +1,79 @@ > +select * from mysql.user where user = 'root' and host = 'localhost'; > +Host User Password Select_priv Insert_priv Update_priv Delete_priv Create_priv Drop_priv Reload_priv Shutdown_priv Process_priv File_priv Grant_priv References_priv Index_priv Alter_priv Show_db_priv Super_priv Create_tmp_table_priv Lock_tables_priv Execute_priv Repl_slave_priv Repl_client_priv Create_view_priv Show_view_priv Create_routine_priv Alter_routine_priv Create_user_priv Event_priv Trigger_priv Create_tablespace_priv ssl_type ssl_cipher x509_issuer x509_subject max_questions max_updates max_connections max_user_connections plugin authentication_string password_expired is_role default_role max_statement_time > +localhost root Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y 0 0 0 0 N N 0.000000 > +# Test syntax > +# > +# These 2 selects should have no changes from the first one. > +alter user CURRENT_USER; I'm not sure this should be a valid statement. After all ALTER TABLE t1; is not valid. There must be something after the user name > +select * from mysql.user where user = 'root' and host = 'localhost'; > +Host User Password Select_priv Insert_priv Update_priv Delete_priv Create_priv Drop_priv Reload_priv Shutdown_priv Process_priv File_priv Grant_priv References_priv Index_priv Alter_priv Show_db_priv Super_priv Create_tmp_table_priv Lock_tables_priv Execute_priv Repl_slave_priv Repl_client_priv Create_view_priv Show_view_priv Create_routine_priv Alter_routine_priv Create_user_priv Event_priv Trigger_priv Create_tablespace_priv ssl_type ssl_cipher x509_issuer x509_subject max_questions max_updates max_connections max_user_connections plugin authentication_string password_expired is_role default_role max_statement_time > +localhost root Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y 0 0 0 0 N N 0.000000 > +alter user CURRENT_USER(); > +select * from mysql.user where user = 'root' and host = 'localhost'; > +Host User Password Select_priv Insert_priv Update_priv Delete_priv Create_priv Drop_priv Reload_priv Shutdown_priv Process_priv File_priv Grant_priv References_priv Index_priv Alter_priv Show_db_priv Super_priv Create_tmp_table_priv Lock_tables_priv Execute_priv Repl_slave_priv Repl_client_priv Create_view_priv Show_view_priv Create_routine_priv Alter_routine_priv Create_user_priv Event_priv Trigger_priv Create_tablespace_priv ssl_type ssl_cipher x509_issuer x509_subject max_questions max_updates max_connections max_user_connections plugin authentication_string password_expired is_role default_role max_statement_time > +localhost root Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y 0 0 0 0 N N 0.000000 > +create user foo; > +select * from mysql.user where user = 'foo'; > +Host User Password Select_priv Insert_priv Update_priv Delete_priv Create_priv Drop_priv Reload_priv Shutdown_priv Process_priv File_priv Grant_priv References_priv Index_priv Alter_priv Show_db_priv Super_priv Create_tmp_table_priv Lock_tables_priv Execute_priv Repl_slave_priv Repl_client_priv Create_view_priv Show_view_priv Create_routine_priv Alter_routine_priv Create_user_priv Event_priv Trigger_priv Create_tablespace_priv ssl_type ssl_cipher x509_issuer x509_subject max_questions max_updates max_connections max_user_connections plugin authentication_string password_expired is_role default_role max_statement_time > +% foo N N N N N N N N N N N N N N N N N N N N N N N N N N N N N 0 0 0 0 N N 0.000000 > diff --git a/mysql-test/t/alter_user.test b/mysql-test/t/alter_user.test > new file mode 100644 > index 0000000..3a3a7d7 > --- /dev/null > +++ b/mysql-test/t/alter_user.test > @@ -0,0 +1,71 @@ > +--source include/not_embedded.inc > +--enable_connect_log > + > + > +select * from mysql.user where user = 'root' and host = 'localhost'; > +--echo # Test syntax > +--echo # > +--echo # These 2 selects should have no changes from the first one. > +alter user CURRENT_USER; > +select * from mysql.user where user = 'root' and host = 'localhost'; > +alter user CURRENT_USER(); > +select * from mysql.user where user = 'root' and host = 'localhost'; > + > +create user foo; > +select * from mysql.user where user = 'foo'; > +alter user foo; > +select * from mysql.user where user = 'foo'; > + > +--echo # Test super privilege works correctly with a read only database. > +SET @start_read_only = @@global.read_only; > +SET GLOBAL read_only=1; > +grant create user on *.* to foo; > + > +--echo # Currently no super privileges. > +connect (a, localhost, foo); > +select @@global.read_only; > + > +--error ER_OPTION_PREVENTS_STATEMENT > +alter user foo; > + > +--echo # Grant super privilege to the user. > +connection default; > +grant super on *.* to foo; > + > +--echo # We now have super privilege. We should be able to run alter user. > +connect (b, localhost, foo); > +alter user foo; > + > +connection default; > +SET GLOBAL read_only = @start_read_only; > + > +--echo # Test inexistant user. > +--error ER_CANNOT_USER > +alter user boo; > +--echo #--warning ER_CANNOT_USER > +alter if exists user boo; > + > +--echo # Test SSL related altering. eh? I don't see any SSL related altering here > +alter user foo identified by 'something'; > +select * from mysql.user where user = 'foo'; > + > +alter user foo identified by 'something2'; > +select * from mysql.user where user = 'foo'; > + > +alter user foo identified by password '*88C89BE093D4ECF72D039F62EBB7477EA1FD4D63'; > +select * from mysql.user where user = 'foo'; > + > +alter user foo identified with 'somecoolplugin'; > +select * from mysql.user where user = 'foo'; > + > +alter user foo identified with 'somecoolplugin' using 'somecoolpassphrase'; > +select * from mysql.user where user = 'foo'; > + > +--echo # Test resource limits altering. > +alter user foo with MAX_QUERIES_PER_HOUR 10 > + MAX_UPDATES_PER_HOUR 20 > + MAX_CONNECTIONS_PER_HOUR 30 > + MAX_USER_CONNECTIONS 40; > +select * from mysql.user where user = 'foo'; > +drop user foo; > +--disable_connect_log > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 96985aa..9fcd505 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -4635,7 +4640,11 @@ mysql_execute_command(THD *thd) > break; > /* Conditionally writes to binlog */ > WSREP_TO_ISOLATION_BEGIN(WSREP_MYSQL_DB, NULL, NULL) > - if (!(res= mysql_rename_user(thd, lex->users_list))) > + if (lex->sql_command == SQLCOM_ALTER_USER) > + res= mysql_alter_user(thd, lex->users_list); I don't think ALTER USER needs same privileges as RENAME USER. A user should be able to ALTER herself. We have added a check_alter_user() helper exactly for that, remember? :) > + else > + res= mysql_rename_user(thd, lex->users_list); > + if (!res) > my_ok(thd); > break; > } > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index a6c6859..1285165 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -7710,6 +7798,78 @@ static bool print_grants_for_role(THD *thd, ACL_ROLE * role) > } > > > +bool mysql_show_create_user(THD *thd, LEX_USER *lex_user) > +{ > + const char *username = safe_str(lex_user->user.str); > + const char *hostname = safe_str(lex_user->host.str); > + Protocol *protocol= thd->protocol; > + bool error= false; > + ACL_USER *acl_user; > + DBUG_ENTER("mysql_show_create_user"); > + > + // Check if the command specifies a username or not. > + if (lex_user->user.str == current_user.str) > + { > + username= thd->security_ctx->priv_user; > + hostname= thd->security_ctx->priv_host; > + } > + > + char buff[1024]; //Show create user should not take more than 1024 bytes. > + String field_name(buff, sizeof(buff), system_charset_info); There's a helper for that: StringBuffer<1024> field_name; on the other hand... you don't use field_name anywhere. > + List<Item> field_list; > + strxmov(buff, "CREATE USER for ", username, "@", hostname, NullS); > + Item_string *field = new (thd->mem_root) Item_string_ascii(thd, "", 0); > + if (!field) > + { > + my_error(ER_OUTOFMEMORY, MYF(0)); > + DBUG_RETURN(true); > + } > + Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 57bf586: MDEV-8615: Assertion `m_cpp_buf <= begin_ptr && begin_ptr <= m_cpp_buf + m_buf_length' failed in Lex_input_stream::body_utf8_start
by Sergei Golubchik 27 Jan '16

27 Jan '16

Hi, Sanja! On Jan 27, OleksandrByelkin wrote: > revision-id: 57bf58668877552ee43f4ed86689b8bbb3821c9c (mariadb-10.1.10-23-g57bf586) > parent(s): 825f51d1aab51d363dc07ec9fe0829af33063883 > committer: Oleksandr Byelkin > timestamp: 2016-01-27 00:38:36 +0100 > message: > > MDEV-8615: Assertion `m_cpp_buf <= begin_ptr && begin_ptr <= m_cpp_buf + m_buf_length' failed in Lex_input_stream::body_utf8_start > > Nothing should be done before any keyword recognized. Good! Just two comments: 1. May be it's better to rename s/pop_cs_label/pop_sp_label/ and s/pop_cs_empty_label/pop_sp_empty_label/ ? first, pop_sp_empty_label applies to all SPs, not only to compound statements, and second, pop_cs_label does not apply to compound statements at all :) 2. Your process_control_statement_label() makes no sense here. There is no need to do an if() - because in both branches you always know whether you have a label or not. Please split process_control_statement_label() in two different functions (or two yacc rules, whatever). > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 302c2fd..3c2d386 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -248,6 +248,36 @@ static bool maybe_start_compound_statement(THD *thd) > return 0; > } > > +static bool process_control_statement_label(THD *thd, LEX_STRING label) > +{ > + if (label.str != NULL) > + { > + sp_pcontext *ctx= thd->lex->spcont; > + sp_label *lab= ctx->find_label(label); > + > + if (lab) > + { > + my_error(ER_SP_LABEL_REDEFINE, MYF(0), label.str); > + return 1; > + } > + else > + { > + lab= thd->lex->spcont->push_label(thd, label, > + thd->lex->sphead->instructions()); > + lab->type= sp_label::ITERATION; > + } > + } > + else > + { > + if (maybe_start_compound_statement(thd)) > + return 1; > + /* Unlabeled controls get an empty label. */ > + thd->lex->spcont->push_label(thd, empty_lex_str, > + thd->lex->sphead->instructions()); > + } > + return 0; > +} > + > /** > Helper action for a case expression statement (the expr in 'CASE expr'). > This helper is used for 'searched' cases only. > @@ -4389,6 +4374,84 @@ sp_control_content: > } > ; > > +pop_cs_label: > + sp_opt_label > + { > + sp_label *lab; > + Lex->sphead->backpatch(lab= Lex->spcont->pop_label()); > + if ($1.str) > + { > + if (my_strcasecmp(system_charset_info, $1.str, > + lab->name.str) != 0) > + { > + my_error(ER_SP_LABEL_MISMATCH, MYF(0), $1.str); > + MYSQL_YYABORT; > + } > + } > + } > + ; > + > +pop_cs_empty_label: > + { > + sp_label *lab; > + Lex->sphead->backpatch(lab= Lex->spcont->pop_label()); > + DBUG_ASSERT(lab->name.length == 0); > + } > + ; > + > +sp_labeled_control: > + label_ident ':' LOOP_SYM > + { > + if (process_control_statement_label(thd, $1)) > + MYSQL_YYABORT; > + } > + loop_body pop_cs_label > + { } > + | label_ident ':' WHILE_SYM > + { > + if (process_control_statement_label(thd, $1)) > + MYSQL_YYABORT; > + Lex->sphead->reset_lex(thd); > + } > + while_body pop_cs_label > + { } > + | label_ident ':' REPEAT_SYM > + { > + if (process_control_statement_label(thd, $1)) > + MYSQL_YYABORT; > + } > + repeat_body pop_cs_label > + { } > + ; > + > +sp_unlabeled_control: > + LOOP_SYM > + { > + if (process_control_statement_label(thd, null_lex_str)) > + MYSQL_YYABORT; > + } > + loop_body > + pop_cs_empty_label > + { } > + | WHILE_SYM > + { > + if (process_control_statement_label(thd, null_lex_str)) > + MYSQL_YYABORT; > + Lex->sphead->reset_lex(thd); > + } > + while_body > + pop_cs_empty_label > + { } > + | REPEAT_SYM > + { > + if (process_control_statement_label(thd, null_lex_str)) > + MYSQL_YYABORT; > + } > + repeat_body > + pop_cs_empty_label > + { } > + ; > + > trg_action_time: > BEFORE_SYM > { Lex->trg_chistics.action_time= TRG_ACTION_BEFORE; } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-5513 merge audit api changes from 5.5.34.
by Alexey Botchkov 27 Jan '16

27 Jan '16

Hello Sergei, all. I didn't quite understand what exact changes in 5.5.34 are important to us, so in order to begin i'm going to mention shortly what is new in MySQL 5.7 audit plugin API. 1. New event classes added: MYSQL_AUDIT_PARSE_CLASS = 2, MYSQL_AUDIT_AUTHORIZATION_CLASS = 3, MYSQL_AUDIT_TABLE_ACCESS_CLASS = 4, MYSQL_AUDIT_GLOBAL_VARIABLE_CLASS = 5, MYSQL_AUDIT_SERVER_STARTUP_CLASS = 6, MYSQL_AUDIT_SERVER_SHUTDOWN_CLASS = 7, MYSQL_AUDIT_COMMAND_CLASS = 8, MYSQL_AUDIT_QUERY_CLASS = 9, MYSQL_AUDIT_STORED_PROGRAM_CLASS = 10, These new notifications are sent from various parts of the code. I think that part can be most difficult to keep similar between Maria and My. New data structures are declared for these classes to send data inside notify(). In my opinion these data structures only contain little information, not sufficient for many tasks. 2. Plugin can specify the CLASS/SUBCLASS filter in the st_mysql_audit structure. To implement that nubers of subclasses were changed (i mean previously known subclasses). 3. notify() function returns 'int' (it used to be 'void'). I think 2. and 3. are the reasons MySQL doesn't load plugins with older interface. 4. Notification function can affect server. - Nonzero return from the notify() breaks the query execution in some cases. - Query can be modified inside the 'notify' function in some cases. 5. Sending notification looks quite crazy in the code. All notification-sending functions share the exactly same name 'mysql_audit_notify'. Naturally the number and types of parameters are different. So that i have complications if i want to find all the places where for instance the MYSQL_AUDIT_PARSE_CLASS notifications are sent. 'grep' isn't much help anymore. 6. The code is immature. The MYSQL_AUDIT_TABLE_ACCESS_CLASS just is not supported. st_mysql_audit::class_mask doesn't always work as it's supposed to. Now my proposals for our audit API. The 'radical' proposal : make the notify() function return 'int' freeze the AUDIT_INTERFACE_VERSION forever. notify() expects the 'event_class' as a parameter. It does something with the classes it is aware of. if it gets the class it doesn't know about - it should just return. notify() at the moment is expected to get the server data from the 'const void *event' sent to it as a paramter, Which is in my opinion inconvenient for everybody. My idea is to provide a set of functions like 'ap_get_query_string(MYSQL_THD thd, const void *event)', or 'ap_get_connection_id(MYSQL_THD thd, const void *event)'. That set can be extended in newer versions of the server, but old functions never change. If a plugin connects to the old server where that new possibilities are not implemented for some reason, these function still available, just do nothing. Plugin can check if some functions are available with the server it connected to. We provide the plugin developer with an 'audit_plugin_adaptor.c' file, that is built inside the plugin. It checks the version of the server that loads the plugin and links or implements the 'ap_*' functions respectively. Also it provides proper structures for the connecting server and interceipts notification's call to prepare it for the developer's notify() function. That way the plugin once built can work with variety of servers. And getting newer 'audit_plugin_adaptor.c' is going to make your plugin working with the newer versions of a server (be it MariaDB or MySQL). Less radical proposition is to mimic MySQL's API, to make the plugin compilable both in MySQL and MariaDB environment. So we get rid of the 'maria_declare_plugin', leaving only 'mysql_declare_plugin'. The present idea of having both doesn't work well. Last option i see - get rid of the 'mysql_declare_plugin', leaving only 'maria_declare_plugin' and forget about the idea that same plugin is used in MySQL and MariaDB at the same time. In this case i'd provide the set of the beforementioned 'ap_*' functions for the plugin to get server data. As before i belive it'd be convenient, we'd never need to check the plugin API version, and once built the plugin can work with many versions of MariaDB. And we don't need the 'adaptor.c' file as the proper .h file seem to be enough. Best regards. HF

1 0

[Maria-developers] convert_tz unexpectedly returning fractional seconds
by Yitzchak Scott-Thoennes 27 Jan '16

27 Jan '16

Using the ubuntu 14.04 mariadb (5.5.46), convert_tz unexpectedly returns a time with fractional seconds if its first argument is an expression using a column. This doesn't happen with mysqldb 5.5.46. Example: select convert_tz(concat('2006-05-25 17:45:00',''),'+00:00','+00:00') no_columns_used, convert_tz(concat('2006-05-25 17:45:00',blank),'+00:00','+00:00') column_used from (select '' blank) foo \G *************************** 1. row *************************** no_columns_used: 2006-05-25 17:45:00 column_used: 2006-05-25 17:45:00.000000

2 1

[Maria-developers] MDEV-9181 (NULLIF(count(table.col)), 0) gives wrong result on 10.1.x
by Alexander Barkov 26 Jan '16

26 Jan '16

Hi, Please review a new version of a patch for MDEV-9181. Thanks.

1 0

[Maria-developers] MDEV-9332 Bug after upgrade to 10.1.10
by Alexander Barkov 26 Jan '16

26 Jan '16

Hi Sergei, Please review a patch form MDEV-9332. It does the following: - Changes the behaviour of copy_if_not_alloced in case from->Alloced_length==0 and from_length==0 to return "to" instead of "from", because "from" can point to a constant string. - Rewrites the code to make it more readable (I hope), adding comments. Thanks.

2 2

Re: [Maria-developers] [Commits] e58e69e: MDEV-6421 SQL_ERROR_LOG doesn't log comments in Events.
by Sergei Golubchik 25 Jan '16

25 Jan '16

Hi, Alexey! On Jan 25, Alexey Botchkov wrote: > revision-id: e58e69e2feed36cdbffbae18f75faf0aea028401 (mariadb-10.1.10-24-ge58e69e) > parent(s): ff8d4009a7743b99c1c13831172b08041a4b272e > committer: Alexey Botchkov > timestamp: 2016-01-25 11:28:58 +0400 > message: > > MDEV-6421 SQL_ERROR_LOG doesn't log comments in Events. > Change parser so it saves all the query line to the ';' in the > sp_instr::m_query. Looks ok to push, thanks! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] More contributors welcome to MariaDB packaging in Debian
by Otto Kekäläinen 24 Jan '16

24 Jan '16

Hello! Do you want to improve MariaDB in Debian? Do you want to contribute to Debian packaging using modern git-buildpackage tools and state of the art QA tools? Do you want to see the result of your work in the hands of numerous of Linux users who run Debian and MariaDB? I am happy to mentor anybody who want's to start contributing to Debian packaging of MariaDB. I have created some items with priority "wishlist" and title prepended with "TODO" visible at https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=mariadb-10.0#_0_6_4 These are all farily easy tasks that anybody interested in contributing could pick up and and start working on. If you want to contribute, please subscribe to the packaging teams list and show your interest there / ask questions to get directions and more help: http://lists.alioth.debian.org/pipermail/pkg-mysql-maint/ I promise to quickly review and respond to questions so that contributors feel welcomed! - Otto

3 4

[Maria-developers] bugs for beginners
by Aniket Shukla 22 Jan '16

22 Jan '16

Hello developers,I was wondering if anyone could point out some bugs for beginners in MariaDB(for the purpose of contribution) .Any help would be thankfully welcomed. Aniket

2 1

[Maria-developers] Oracle CPU January 2016
by Colin Charles 21 Jan '16

21 Jan '16

Hi! Oracle has released a Critical Patch Update: http://www.oracle.com/technetwork/topics/security/cpujan2016-2367955.html The following vulnerabilities are fixed by upgrading from MySQL 5.5.46 to 5.5.47 (picked it from the Debian maintainers list): CVE-2016-0505 CVE-2016-0546 CVE-2016-0597 CVE-2016-0598 CVE-2016-0600 CVE-2016-0606 CVE-2016-0608 CVE-2016-0609 CVE-2016-0596 CVE-2016-0616 We already have released MariaDB Server 5.5.47, so I guess its time we updated: https://mariadb.com/kb/en/mariadb/security/ (assuming that all this is fixed?) As an aside, it is interesting to see that there are CVE’s fixed for 5.7 only (see: http://www.oracle.com/technetwork/topics/security/cpujan2016-2367955.html#A…) -- Colin Charles, http://bytebot.net/blog/ twitter: @bytebot | skype: colincharles "First they ignore you, then they laugh at you, then they fight you, then you win." -- Mohandas Gandhi

1 0

Re: [Maria-developers] MDEV-8615: Assertion `m_cpp_buf <= begin_ptr && begin_ptr <= m_cpp_buf + m_buf_length' failed in Lex_input_stream::body_utf8_start
by Sergei Golubchik 21 Jan '16

21 Jan '16

Hi, Sanja! On Dec 25, OleksandrByelkin wrote: > revision-id: 684bd68c33284f6106551188d9567f35b18593ed (mariadb-10.1.9-21-g684bd68) > parent(s): 59fcd7ff2315d007045eb987da5f21abbea6f6f1 > committer: Oleksandr Byelkin > timestamp: 2015-12-25 18:39:43 +0100 > message: > > MDEV-8615: Assertion `m_cpp_buf <= begin_ptr && begin_ptr <= m_cpp_buf + m_buf_length' failed in Lex_input_stream::body_utf8_start > > Nothing should be done before any keyword recognised. > > diff --git a/mysql-test/r/mysqltest.result b/mysql-test/r/mysqltest.result > index e258b1d..a9980a7 100644 > --- a/mysql-test/r/mysqltest.result > +++ b/mysql-test/r/mysqltest.result > @@ -29,40 +29,40 @@ select 0 as "after_successful_stmt_errno" ; > after_successful_stmt_errno > 0 > garbage ; > -ERROR 42000: You have an error in your SQL syntax; <...> near 'garbage' at line 1 > +ERROR 42000: You have an error in your SQL syntax; <...> near '' at line 1 Why is that? I mean, how does the parser parse "garbage" now? > diff --git a/mysql-test/suite/funcs_1/r/storedproc.result b/mysql-test/suite/funcs_1/r/storedproc.result > index f0e1ee7..485082a 100644 > --- a/mysql-test/suite/funcs_1/r/storedproc.result > +++ b/mysql-test/suite/funcs_1/r/storedproc.result > @@ -15982,8 +15982,7 @@ insert into res_t3_itisalongname_1381742_itsaverylongname_1381742 values( 'xyz' > until count1 < 3 > END repeat label1; > END// > -ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'label1; > -END' at line 8 > +ERROR 42000: End-label label1 without match New error message is correct, great! > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index a5a62ae..74d308f 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -248,6 +248,36 @@ static bool maybe_start_compound_statement(THD *thd) > return 0; This looks quite good! But it was a bit difficult to see how the new sql_yacc.yy will look like. Did you push this patch? In what branch? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

2 1

[Maria-developers] MDEV-8930 main.func_hybrid_type with --embedded --ps-protocol: Assertion `(longlong) thd->status_var.local_memory_used >= 0' failed
by Alexander Barkov 20 Jan '16

20 Jan '16

Hi Sergei, IIRC you fixed this bug during the meeting in Amsterdam. Can you please confirm? (or just close it, if so) Thanks.

1 0

[Maria-developers] Request for the project
by Ayush Bhargav 20 Jan '16

20 Jan '16

Hi. I am studying programming in college and I wanted to have some hand on experience on larger projects. I know c++ and mysql well. Please can you provide me a project on which I can start working. Thanks. - Ayush Bhargav

2 1

Re: [Maria-developers] [Commits] cae1945: MDEV-9117: Client Server capability negotiation for MariaDB specific functionality
by Sergei Golubchik 20 Jan '16

20 Jan '16

Hi, Sanja! On Nov 25, OleksandrByelkin wrote: > revision-id: cae19452eec1d3be47fa04759dd2dda6d061543e (mariadb-10.1.8-69-gcae1945) > parent(s): 55e67c3e344317c6f349f5391e5d117ec51ae062 > committer: Oleksandr Byelkin > timestamp: 2015-11-25 18:09:39 +0100 > message: > > MDEV-9117: Client Server capability negotiation for MariaDB specific functionality > > New capability flags space. > Removed old progress flag, added new one. > > diff --git a/include/mysql_com.h b/include/mysql_com.h > index aa6ab0f..57092dc 100644 > --- a/include/mysql_com.h > +++ b/include/mysql_com.h > @@ -215,7 +216,8 @@ enum enum_server_command > /* Don't close the connection for a connection with expired password. */ > #define CLIENT_CAN_HANDLE_EXPIRED_PASSWORDS (1UL << 22) > > -#define CLIENT_PROGRESS (1UL << 29) /* Client support progress indicator */ > +#define CLIENT_PROGRESS_OBSOLETE (1UL << 29) ok > +#define CLIENT_PROGRESS 0 why is that? > #define CLIENT_SSL_VERIFY_SERVER_CERT (1UL << 30) > /* > It used to be that if mysql_real_connect() failed, it would delete any > @@ -228,6 +230,13 @@ enum enum_server_command > */ > #define CLIENT_REMEMBER_OPTIONS (1UL << 31) > > +/* MariaDB extended capability flags */ > +#define MARIADB_CLIENT_FLAGS_MASK 0xffffffff00000000ULL > +/* Client support progress indicator */ > +#define MARIADB_CLIENT_PROGRESS (1ULL << 32) > +/* Client support mariaDB extended flags */ > +#define MARIADB_CLIENT_EXTENDED_FLAGS (1ULL << 63) how can this work, if client_flag and server_capabilities are only 32-bit variables? > + > #ifdef HAVE_COMPRESS > #define CAN_CLIENT_COMPRESS CLIENT_COMPRESS > #else > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 1240d5d..d07e0a4 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -11729,18 +11731,38 @@ static ulong parse_client_handshake_packet(MPVIO_EXT *mpvio, > */ > DBUG_ASSERT(net->read_pos[pkt_len] == 0); > > - ulong client_capabilities= uint2korr(net->read_pos); > + ulonglong client_capabilities= uint2korr(net->read_pos); > + compile_time_assert(sizeof(client_capabilities) >= 8); > if (client_capabilities & CLIENT_PROTOCOL_41) > { > - if (pkt_len < 4) > + if (pkt_len < 32) > return packet_error; > client_capabilities|= ((ulong) uint2korr(net->read_pos+2)) << 16; > + if (!(client_capabilities & CLIENT_MYSQL)) > + { > + // it is client with mariadb extensions > + client_capabilities|= CLIENT_MYSQL; > + ulonglong ext_client_capabilities= > + (((ulonglong)uint4korr(net->read_pos + 28)) << 32); > + if (ext_client_capabilities & MARIADB_CLIENT_EXTENDED_FLAGS) > + client_capabilities|= ext_client_capabilities; > + else > + { > + DBUG_PRINT("error", ("CLIENT_PROTOCOL_41: on, " > + "CLIENT_LONG_PASSWORD/CLIENT_MYSQL off, " > + "but MARIADB_CLIENT_EXTENDED_FLAGS is off. " > + "flags: %llx ext flags %llx", > + client_capabilities, ext_client_capabilities)); > + return packet_error; Why do you need that (MARIADB_CLIENT_EXTENDED_FLAGS and the check)? > + } > + } here you need to check for old MariaDB clients - they will have CLIENT_MYSQL and might have CLIENT_PROGRESS_OBSOLETE - do something like else if (client_capabilities & CLIENT_PROGRESS_OBSOLETE) { client_capabilities |= MARIADB_CLIENT_PROGRESS; } > } > > diff --git a/tests/mysql_client_test.c b/tests/mysql_client_test.c > index a1a52e8..31fca0f 100644 > --- a/tests/mysql_client_test.c > +++ b/tests/mysql_client_test.c > @@ -15336,7 +15336,7 @@ static void test_mysql_insert_id() > > myheader("test_mysql_insert_id"); > > - rc= mysql_query(mysql, "drop table if exists t1"); > + rc= mysql_query(mysql, "drop table if exists t1,t2"); separate commit, please > myquery(rc); > /* table without auto_increment column */ > rc= mysql_query(mysql, "create table t1 (f1 int, f2 varchar(255), key(f1))"); > @@ -18700,7 +18700,8 @@ static void test_progress_reporting() > myheader("test_progress_reporting"); > > conn= client_connect(CLIENT_PROGRESS, MYSQL_PROTOCOL_TCP, 0); > - DIE_UNLESS(conn->client_flag & CLIENT_PROGRESS); > + if (!(conn->client_flag & CLIENT_PROGRESS)) > + return; what are you testing here? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

2 8

[Maria-developers] Contribution in MariaDB
by Sanjana Garg 18 Jan '16

18 Jan '16

Hi, I am interested in contributing to the MariaDB project. Can someone guide me on how to get started. I am proficient in C, C++, python and SQL. Thanks Sanjana Garg

2 1

[Maria-developers] Projects to contribute
by Sonali Ellath 18 Jan '16

18 Jan '16

Hi Everyone, I was looking to contribute to MariaDb for GSoC 2016, I have successfully built MariaDb from source. What should be my next step? Should I pick a project and get it done? I am proficient in C++ programming, which ones would be suitable for me? Thanks! - Sonali

2 1

[Maria-developers] MDEV-8789: fully qualified field name resolving problem
by Oleksandr Byelkin 14 Jan '16

14 Jan '16

Hi! I decided to show my finding as soon as I have them found (to give time to explain/fix, also I think that name resolving fix could possibly cause some big rewriting) but if you prefer "all togather" do not hesitate to say it. So the test suite which demostrate the problem: --------- create table t1(a int, b int); insert into t1 values (1,1),(2,2),(10,10),(20,20); create table t2(c int, d int); insert into t2 values (10,1),(20,2); # mentioning "full path" to field works here select * from test.t2 where test.t2.c <20; # and does not work here with t2 as (select a,b from t1 where a < 10) select * from t2 as tt, test.t2 where tt.a = test.t2.c; drop table t1,t2; ---------- SELECT with "WITH" clause fails complaining about undefined column in the WHERE clause. BTW if remove alias tt it complain about two tables t2 when such query: select * from test2.t2, test.t2 where test2.t2.c = test.t2.c; is legal. I am not 100% sure if it is also a bug, but possible it is.

1 3

Re: [Maria-developers] For review https://mariadb.atlassian.net/browse/MDEV-9058
by Oleksandr Byelkin 13 Jan '16

13 Jan '16

On 12.01.2016 13:09, Michael Widenius wrote: >>>>>> "Oleksandr" == Oleksandr Byelkin <sanja(a)montyprogram.com> writes: > Oleksandr> Hi! > Oleksandr> Here is full diff of the task (will be tested more when georg made it in > Oleksandr> the connectors tree) > > Oleksandr> Also it can be reviewed by parts in bb-sanja-10.2 branch on github. > > Oleksandr> Connectors part is on 10.2-georg in connectors tree. > > > diff --git a/include/mysql_com.h b/include/mysql_com.h > index 57092dc..425559f 100644 > --- a/include/mysql_com.h > +++ b/include/mysql_com.h > @@ -102,7 +102,9 @@ enum enum_server_command > COM_STMT_PREPARE, COM_STMT_EXECUTE, COM_STMT_SEND_LONG_DATA, COM_STMT_CLOSE, > COM_STMT_RESET, COM_SET_OPTION, COM_STMT_FETCH, COM_DAEMON, > /* don't forget to update const char *command_name[] in sql_parse.cc */ > - > + COM_MDB_GAP_BEG, > + COM_MDB_GAP_END=254, > + COM_MULTI, > /* Must be last */ > COM_END > }; > > Change COM_MDB_GAP_END to be 253, to ensure that COM_END is 255. > (Better to keep everything in one byte) > > Also add an compiler error check that COM_END is 255 (to ensure that one > doesn't accidently add a new define without changing COM_DB_GAP_END. OK! > > @@ -234,6 +236,8 @@ enum enum_server_command > #define MARIADB_CLIENT_FLAGS_MASK 0xffffffff00000000ULL > > Haven't seen the above define before and it's not in the current 10.1 > code. ah, it was removed wit client code, so I'll remove it if it is not used. > > /* Client support progress indicator */ > #define MARIADB_CLIENT_PROGRESS (1ULL << 32) > +/* support COM_MULTI */ > +#define MARIADB_CLIENT_COM_MULTI (1ULL << 33) > /* Client support mariaDB extended flags */ > #define MARIADB_CLIENT_EXTENDED_FLAGS (1ULL << 63) > > @@ -268,7 +272,8 @@ enum enum_server_command > MARIADB_CLIENT_PROGRESS | \ > CLIENT_PLUGIN_AUTH | \ > CLIENT_PLUGIN_AUTH_LENENC_CLIENT_DATA | \ > - CLIENT_CONNECT_ATTRS) > + CLIENT_CONNECT_ATTRS |\ > + MARIADB_CLIENT_COM_MULTI) > > You don't have MARIADB_CLIENT_COM_MULTI anywhere else in the patch. > Is that intententional ? > > If yes, you should add a comment that this is used in the LGPL C > driver code so that anyone examening the server code will not get > confused. > > As you have added flags bigger than 32, you should also change > THD::client_capabilities from ulong to ulonglong to reflect this! > > This patch doesn't include changes in sql_acl.cc to send the > MARIADB_CLIENT_COM_MULTI flag to the client. > Where are the necessary changes for this ? expanding in to 64bit was separate task (reviewed and pushed). Also we are not going to support this new features in old client, so we are signaling our connector only. I'll write a comment about it. > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index b496e4c..54884c0 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -856,6 +856,7 @@ THD::THD(bool is_wsrep_applier) > /* statement id */ 0), > rli_fake(0), rgi_fake(0), rgi_slave(NULL), > protocol_text(this), protocol_binary(this), > + last_stmt(NULL), > in_sub_stmt(0), log_all_errors(0), > binlog_unsafe_warning_flags(0), > binlog_table_maps(0), > > You don't have to set last_stmt(NULL) in THD::THD() as this function > calls init() which sets this too. ok > > @@ -1424,6 +1425,7 @@ void THD::init(void) > bzero((char *) &org_status_var, sizeof(org_status_var)); > start_bytes_received= 0; > last_commit_gtid.seq_no= 0; > + last_stmt= NULL; > status_in_global= 0; > #ifdef WITH_WSREP > wsrep_exec_mode= wsrep_applier ? REPL_RECV : LOCAL_STATE; > > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > @@ -884,6 +1115,23 @@ void cleanup_items(Item *item) > DBUG_VOID_RETURN; > } > > +static enum enum_server_command fetch_command(THD *thd, char *packet) > +{ > + enum enum_server_command > + command= (enum enum_server_command) (uchar) packet[0]; > + NET *net= &thd->net; > + DBUG_ENTER("fetch_command"); > + > > If command is 256, then the following first test can never be true and > you may get a warning from gcc. Another reason to have COM_END as 255. yes > > + if (command >= COM_END || > + (command >= COM_MDB_GAP_BEG && command <= COM_MDB_GAP_END)) > + command= COM_END; // Wrong command > + > + DBUG_PRINT("info",("Command on %s = %d (%s)", > + vio_description(net->vio), command, > + command_name[command].str)); > + DBUG_RETURN(command); > +} > + > > /** > + check COM_MULTI packet > + > + @param thd thread handle > + @param packet pointer on the packet of commands > + @param packet_length length of this packet > + > + @retval TRUE - Error > + @retval FALSE - OK > +*/ > + > +bool maria_multi_check(THD *thd, char *packet, uint packet_length) > +{ > + DBUG_ENTER("maria_multi_check"); > + while (packet_length) > + { > > You need to check for packet_length >= 3 and give the > my_message(ER_UNKNOWN_COM_ERROR... error if packet_length > 1 and <= 3 > > > + // length of command + 3 bytes where that length was stored > + uint subpacket_length= (uint3korr(packet) + 3); > + DBUG_PRINT("info", ("sub-packet length: %d command: %x", > + subpacket_length, packet[3])); > + > + if (subpacket_length == 0 || > + subpacket_length > packet_length) > > Remove check of sub_packet_length == 0; It can never be 0 as you are > adding +3 above. ok > > + { > + my_message(ER_UNKNOWN_COM_ERROR, ER_THD(thd, ER_UNKNOWN_COM_ERROR), > + MYF(0)); > + DBUG_RETURN(TRUE); > + } > + > + packet+= subpacket_length; > + packet_length-= subpacket_length; > + } > > + DBUG_RETURN(FALSE); > +} > > > @@ -1901,6 +2200,69 @@ bool dispatch_command(enum enum_server_command command, THD *thd, > general_log_print(thd, command, NullS); > my_eof(thd); > break; > + case COM_MULTI: > + if (is_com_multi) > + { > + /* we do not allow deep recursion because it is not safe */ > + my_message(ER_UNKNOWN_COM_ERROR, ER_THD(thd, ER_UNKNOWN_COM_ERROR), > + MYF(0)); > + break; > + } > > Give another error as the above one is missleading. OK > > + if (!(thd->client_capabilities & CLIENT_MULTI_RESULTS)) > + { > + /* The client does not support multiple result sets being sent back */ > + my_error(ER_COMMULTI_BADCONTEXT, MYF(0)); > + break; > + } > > How can the above be possible ? > We should not allow the client to set COM_MULTI flag if they don't > have CLIENT_MULTI_RESULT. > Better to remove the above check and document when the client can set the > COM_MULTI flag. I am not of control of client, it can be some home-made client so better to check. > > + if (maria_multi_check(thd, packet, packet_length)) > + break; > + { > + /* We have to store next length because it will be destroyed by '\0' */ > + uint next_subpacket_length= uint3korr(packet); > + unsigned char *readbuff= net->buff; > + unsigned long readbuff_max_packet= net->max_packet; > + > > Remove the above one extra empty line. > Add also some documentation why we are changing net->buff here instead > of creating another buffer and using this instead. > > Actually it's much better to create a new buffer for packet becasue > of: > > - The buffer only need to be of size packet_length, not > net->max_packet which can be much longer. > - Clearer code as you don't have to manipulate net internal structures > or restore these. > - The disadvantage is that you can't allow COM_CHANGE_USER, but I > don't think one should be able to do that anyway as this may ask for > more information from the client. OK > > > + if (!(net->buff=(uchar*) my_malloc((size_t) net->max_packet+ > + NET_HEADER_SIZE + COMP_HEADER_SIZE +1, > + MYF(MY_WME)))) > + break; > + net->buff_end=net->buff + net->max_packet; > + net->write_pos=net->read_pos = net->buff; > + > + while(packet_length) > > space after while > > + { > + uint subpacket_length= next_subpacket_length; > > Why not do as in other code and make the rest of the code easier: > subpacket_length= next_subpacket_length +3 ; ok > > + if (subpacket_length + 3 < packet_length) > + next_subpacket_length= uint3korr(packet + subpacket_length + 3); > + /* safety like in do_command() */ > + packet[subpacket_length + 3]= '\0'; > + > + enum enum_server_command subcommand= fetch_command(thd, (packet + 3)); > + > > I would almost prefer that we check for sub_commend == COM_MULTI here > instead of at start of COM_COMMAND: This would be more logical as we > may not allow all commands here (like change user). OK > > + if (is_com_multi) > + { > + /* we do not allow deep recursion because it is not safe */ > + my_message(ER_UNKNOWN_COM_ERROR, ER_THD(thd, ER_UNKNOWN_COM_ERROR), > + MYF(0)); > + break; > + } > > > + if (dispatch_command(subcommand, thd, packet + 4, > + subpacket_length - 1, TRUE)) > > Add an assert that is_error() is set if we do a break. > (As otherwise you can miss to execute commands the client expects) OK > > + break; > + > + DBUG_ASSERT(subpacket_length + 3 <= packet_length); > + packet+= (subpacket_length + 3); > + packet_length-= (subpacket_length + 3); > > + } > > I don't think you need a net_flush here as the packet is flushed by > every command the sent a result. OK > > + net_flush(net); > + my_free(net->buff); > + net->buff= readbuff; > + net->max_packet= readbuff_max_packet; > + net->buff_end=net->buff + net->max_packet; > + net->write_pos=net->read_pos = net->buff; > + > > If we use an internal buffer, you can remoev all of the above (except > the free) OK > > + if (!thd->is_error()) > + { > + thd->get_stmt_da()->reset_diagnostics_area(); > + my_ok(thd); > + } > + } > + break; > > case COM_SLEEP: > case COM_CONNECT: // Impossible here > case COM_TIME: // Impossible from client > @@ -1940,6 +2302,8 @@ bool dispatch_command(enum enum_server_command command, THD *thd, > thd->protocol->end_statement(); > query_cache_end_of_result(thd); > } > + if (drop_more_results) > + thd->server_status&= ~SERVER_MORE_RESULTS_EXISTS; > > Another way, is to just remember the status of server_status and do: > > thd->server_status= ((thd->server_status & ~SERVER_MORE_RESULTS_EXISTS) | > (org_server_status & SERVER_MORE_RESULTS_EXISTS)); > > Easier to understand than drop_more_results I am not sure that execution can't legally change setver status (if not now then in future) > > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -324,7 +324,9 @@ find_prepared_statement(THD *thd, ulong id) > prepared statements find() will return 0 if there is a named prepared > statement with such id. > */ > - Statement *stmt= thd->stmt_map.find(id); > + Statement *stmt= ((id == LAST_STMT_ID) ? > + thd->last_stmt : > + thd->stmt_map.find(id)); > > Please add a comment when id can be LAST_STMT_ID (clear to me but not > obivous for others) > > > if (stmt == 0 || stmt->type() != Query_arena::PREPARED_STATEMENT) > return NULL; > > --- a/storage/perfschema/pfs.cc > +++ b/storage/perfschema/pfs.cc > @@ -1444,19 +1444,21 @@ static void register_statement_v1(const char *category, > > for (; count>0; count--, info++) > { > - DBUG_ASSERT(info->m_name != NULL); > - len= strlen(info->m_name); > - full_length= prefix_length + len; > - if (likely(full_length <= PFS_MAX_INFO_NAME_LENGTH)) > + if(info->m_name != NULL) > > Add space after if > > Instead of doing an if, and cause a big diff, do instead: > > if (!info->m_name) > continue; > > This will remove all of the following in the diff: OK > > > { > - memcpy(formatted_name + prefix_length, info->m_name, len); > - info->m_key= register_statement_class(formatted_name, full_length, info->m_flags); > - } > - else > - { > - pfs_print_error("register_statement_v1: name too long <%s>\n", > - info->m_name); > - info->m_key= 0; > + len= strlen(info->m_name); > + full_length= prefix_length + len; > + if (likely(full_length <= PFS_MAX_INFO_NAME_LENGTH)) > + { > + memcpy(formatted_name + prefix_length, info->m_name, len); > + info->m_key= register_statement_class(formatted_name, full_length, info->m_flags); > + } > + else > + { > + pfs_print_error("register_statement_v1: name too long <%s>\n", > + info->m_name); > + info->m_key= 0; > + } > } > } > return; > > Regards, > Monty

1 1

[Maria-developers] MDEV-4691 MariaDB Kerberos/GSSAPI
by Sergei Golubchik 12 Jan '16

12 Jan '16

Hi, Wlad! Here's a review, below. Just a few comments or questions. Great to have it done finally! :) > diff --git a/plugin/auth_gssapi/CMakeLists.txt b/plugin/auth_gssapi/CMakeLists.txt > new file mode 100644 > index 0000000..e20be70 > --- /dev/null > +++ b/plugin/auth_gssapi/CMakeLists.txt > @@ -0,0 +1,33 @@ > +IF (WIN32) > + SET(USE_SSPI 1) > +ENDIF() > + > +IF(USE_SSPI) > + SET(GSSAPI_LIBRARIES secur32) > + ADD_DEFINITIONS(-DPLUGIN_SSPI) > + SET(GSSAPI_CLIENT sspi_client.cc) > + SET(GSSAPI_SERVER sspi_server.cc) > + SET(GSSAPI_ERRMSG sspi_errmsg.cc) > +ELSE() > + SET(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CMAKE_CURRENT_SOURCE_DIR}/cmake) > + FIND_PACKAGE(GSSAPI) > + IF(GSSAPI_FOUND) > + INCLUDE_DIRECTORIES(${GSSAPI_INCS}) > + ADD_DEFINITIONS(-DPLUGIN_GSSAPI) > + SET(GSSAPI_CLIENT gssapi_client.cc) > + SET(GSSAPI_SERVER gssapi_server.cc) > + SET(GSSAPI_ERRMSG gssapi_errmsg.cc) > + ELSE() > + # Can't build plugin > + RETURN() > + ENDIF() > +ENDIF () > + > + > +MYSQL_ADD_PLUGIN(auth_gssapi server_plugin.cc ${GSSAPI_SERVER} ${GSSAPI_ERRMSG} > + LINK_LIBRARIES ${GSSAPI_LIBS} > + MODULE_ONLY) please, specify a component here and below. Like "COMPONENT Kerberos" (or gssapi, whatever). It would mean that this plugin will be in a separate rpm package - it must be, because it brings additional dependencies that the server itself does not need. > + > +MYSQL_ADD_PLUGIN(auth_gssapi_client client_plugin.cc ${GSSAPI_CLIENT} ${GSSAPI_ERRMSG} > + LINK_LIBRARIES ${GSSAPI_LIBS} > + MODULE_ONLY) same here. I'm not sure whether it should be the same component as above, or a different one. also, I'm not sure whether plugin.cmake will do a correct client plugin rpm, if you'll have a different component here. > diff --git a/plugin/auth_gssapi/README.md b/plugin/auth_gssapi/README.md > new file mode 100644 > index 0000000..f20128f > --- /dev/null > +++ b/plugin/auth_gssapi/README.md > @@ -0,0 +1,129 @@ > +# GSSAPI/SSPI authentication for MariaDB > + > +This article gives instructions on configuring GSSAPI authentication plugin > +for MariaDB for passwordless login. > + > +On Unix systems, GSSAPI is usually synonymous with Kerberos authentication. > +Windows has slightly different but very similar API called SSPI, that along with Kerberos, also supports NTLM authentication. > + > +This plugin includes support for Kerberos on Unix, but also can be used as for Windows authentication with or without domain > +environment. > + > +## Server-side preparations on Unix > +To use the plugin, some preparation need to be done on the server side on Unixes. > +MariaDB server will read need access to the Kerberos keytab file, that contains service principal name for the MariaDB server. s/read need/need read/ :) > + > + > +If you are using **Unix Kerberos KDC (MIT,Heimdal)** > + > +- Create service principal using kadmin tool > + > +``` > +kadmin -q "addprinc -randkey mariadb/host.domain.com" > +``` > + > +(replace host.domain.com with fully qualified DNS name for the server host) > + > +- Export the newly created user to the keytab file > + > +``` > +kadmin -q "ktadd -k /path/to/mariadb.keytab mariadb/host.domain.com" > +``` > + > +More details can be found [here](http://www.microhowto.info/howto/create_a_service_principal_using_mit_kerberos.html) > +and [here](http://www.microhowto.info/howto/add_a_host_or_service_principal_to_a_keytab_using_mit_kerberos.html) > + > +If you are using **Windows Active Directory KDC** > +you can need to create keytab using ktpass.exe tool on Windows, map principal user to an existing domain user like this > + > +``` > +ktpass.exe /princ mariadb/host.domain.com(a)DOMAIN.COM /mapuser someuser /pass MyPas$w0rd /out mariadb.keytab /crypto all /ptype KRB5_NT_PRINCIPAL /mapop set > +``` > + > +and then transfer the keytab file to the Unix server. See [Microsoft documentation](https://technet.microsoft.com/en-us/library/cc753771.aspx) for details. > + > + > +## Server side preparations on Windows. > +Usually nothing need to be done. MariaDB server should to run on a domain joined machine, either as NetworkService account > +(which is default if it runs as service) or run under any other domain account credentials. > +Creating service principal is not required here (but you can still do it using [_setspn_](https://technet.microsoft.com/en-us/library/cc731241.aspx) tool) > + > + > +# Installing plugin > +- Start the server > + > +- On Unix, edit my the my.cnf/my.ini configuration file, set the parameter gssapi-keytab-path to point to previously > +created keytab path. > + > +``` > + gssapi-keytab-path=/path/to/mariadb.keytab > +``` > + > +- Optionally on Unix, in case the service principal name differs from default mariadb/host.domain.com@REALM, > +configure alternative principal name with > + > +``` > + gssapi-principal-name=alternative/principalname@REALM > +``` > + > +- In mysql command line client, execute > + > +``` > + INSTALL SONAME 'auth_gssapi' > +``` > + > +#Creating users > + > +Now, you can create a user for GSSAPI/SSPI authentication. CREATE USER command, for Kerberos user > +would be like this (*long* form, see below for short one) > + > +``` > +CREATE USER usr1 IDENTIFIED WITH gssapi AS 'usr1(a)EXAMPLE.COM'; > +``` > + > +(replace with real username and realm) > + > +The part after AS is mechanism specific, and needs to be ``machine\\usr1`` for Windows users identified with NTLM. > + > +You may also use alternative *short* form of CREATE USER > + > +``` > +CREATE USER usr1 IDENTIFIED WITH gssapi; > +``` > + > +If this syntax is used, realm part is used for comparison realm part is NOT used for comparison > +thus 'usr1(a)EXAMPLE.COM', 'usr1(a)EXAMPLE.CO.UK' and 'mymachine\usr1' will all identify as 'usr1'. > + > +#Login as GSSAPI user with command line clients > + > +Using command line client, do > + > +``` > +mysql --plugin-dir=/path/to/plugin-dir -u usr1 > +``` > + > +#Plugin variables > +- **gssapi-keytab-path** (Unix only) - Path to the server keytab file > +- **gssapi-principal-name** - name of the service principal. > +- **gssapi-mech-name** (Windows only) - Name of the SSPI package used by server. Can be either 'Kerberos' or 'Negotiate'. > + Defaults to 'Negotiate' (both Kerberos and NTLM users can connect) > + Set it to 'Kerberos', to prevent less secure NTLM in domain environments, but leave it as default(Negotiate) > + to allow non-domain environment (e.g if server does not run in domain enviroment). > + > + > +#Implementation > + > +Overview of the protocol between client and server > + > +1. Server : Construct gssapi-principal-name if not set in my.cnf. On Unixes defaults to hostbased name for service "mariadb". On Windows to user's or machine's domain names. > +Acquire credentials for gssapi-principal-name with ```gss_acquire_cred() / AcquireSecurityCredentials()```. > +Send packet with principal name and mech ```"gssapi-principal-name\0gssapi-mech-name\0"``` to client ( on Unix, empty string used for gssapi-mech) > + > +2. Client: execute ```gss_init_sec_context() / InitializeSecurityContext()``` passing gssapi-principal-name / gssapi-mech-name parameters. > +Send resulting GSSAPI blob to server. > + > +3. Server : receive blob from client, execute ```gss_accept_sec_context()/ AcceptSecurityContext()```, send resulting blob back to client > + > +4. Perform 2. and 3. can until both client and server decide that authentication is done, or until some error occured. If authentication was successful, GSSAPI context (an opaque structure) is generated on both client and server sides. > + > +5. Server : Client name is extracted from the context, and compared to the name provided by client(with or without realm). If name matches, plugin returns success. This is a very good documentation. But it also has to be in the KB (perhaps minus with "Implementation" part). You can put it there yourself or ask Ian to do it. There's a feature in KB that allows to "link" a KB page to a "Github flavored markup" document (meaning the markup document is the original and KB is a copy of it). Perhaps it can be useful here. > diff --git a/plugin/auth_gssapi/cmake/FindGSSAPI.cmake b/plugin/auth_gssapi/cmake/FindGSSAPI.cmake > new file mode 100644 > index 0000000..faee428 > --- /dev/null > +++ b/plugin/auth_gssapi/cmake/FindGSSAPI.cmake > @@ -0,0 +1,78 @@ > +# - Try to detect the GSSAPI support > +# Once done this will define > +# > +# GSSAPI_FOUND - system supports GSSAPI > +# GSSAPI_INCS - the GSSAPI include directory > +# GSSAPI_LIBS - the libraries needed to use GSSAPI > +# GSSAPI_FLAVOR - the type of API - MIT or HEIMDAL > + > +# Copyright (c) 2006, Pino Toscano, <toscano.pino(a)tiscali.it> > +# > +# Redistribution and use is allowed according to the terms of the BSD license. > +# For details see the accompanying COPYING-CMAKE-SCRIPTS file. There's no accompanying COPYING-CMAKE-SCRIPTS file, as far as I can see. You either need to add the license in this file or take this COPYING-CMAKE-SCRIPTS too. I'd copied the complete license into this file, I think. > + > + > +if(GSSAPI_LIBS AND GSSAPI_FLAVOR) > + > + # in cache already > + set(GSSAPI_FOUND TRUE) > + > +else(GSSAPI_LIBS AND GSSAPI_FLAVOR) > + > + find_program(KRB5_CONFIG NAMES krb5-config PATHS > + /opt/local/bin > + ONLY_CMAKE_FIND_ROOT_PATH # this is required when cross compiling with cmake 2.6 and ignored with cmake 2.4, Alex > + ) > + mark_as_advanced(KRB5_CONFIG) > + > + #reset vars > + set(GSSAPI_INCS) > + set(GSSAPI_LIBS) > + set(GSSAPI_FLAVOR) > + > + if(KRB5_CONFIG) > + > + set(HAVE_KRB5_GSSAPI TRUE) > + exec_program(${KRB5_CONFIG} ARGS --libs gssapi RETURN_VALUE _return_VALUE OUTPUT_VARIABLE GSSAPI_LIBS) > + if(_return_VALUE) > + message(STATUS "GSSAPI configure check failed.") > + set(HAVE_KRB5_GSSAPI FALSE) > + endif(_return_VALUE) > + > + exec_program(${KRB5_CONFIG} ARGS --cflags gssapi RETURN_VALUE _return_VALUE OUTPUT_VARIABLE GSSAPI_INCS) > + string(REGEX REPLACE "(\r?\n)+$" "" GSSAPI_INCS "${GSSAPI_INCS}") > + string(REGEX REPLACE " *-I" ";" GSSAPI_INCS "${GSSAPI_INCS}") > + > + exec_program(${KRB5_CONFIG} ARGS --vendor RETURN_VALUE _return_VALUE OUTPUT_VARIABLE gssapi_flavor_tmp) > + set(GSSAPI_FLAVOR_MIT) > + if(gssapi_flavor_tmp MATCHES ".*Massachusetts.*") > + set(GSSAPI_FLAVOR "MIT") > + else(gssapi_flavor_tmp MATCHES ".*Massachusetts.*") > + set(GSSAPI_FLAVOR "HEIMDAL") > + endif(gssapi_flavor_tmp MATCHES ".*Massachusetts.*") > + > + if(NOT HAVE_KRB5_GSSAPI) > + if (gssapi_flavor_tmp MATCHES "Sun Microsystems.*") > + message(STATUS "Solaris Kerberos does not have GSSAPI; this is normal.") > + set(GSSAPI_LIBS) > + set(GSSAPI_INCS) > + else(gssapi_flavor_tmp MATCHES "Sun Microsystems.*") > + message(WARNING "${KRB5_CONFIG} failed unexpectedly.") > + endif(gssapi_flavor_tmp MATCHES "Sun Microsystems.*") > + endif(NOT HAVE_KRB5_GSSAPI) > + > + if(GSSAPI_LIBS) # GSSAPI_INCS can be also empty, so don't rely on that > + set(GSSAPI_FOUND TRUE CACHE STRING "") > + message(STATUS "Found GSSAPI: ${GSSAPI_LIBS}") > + > + set(GSSAPI_INCS ${GSSAPI_INCS} CACHE STRING "") > + set(GSSAPI_LIBS ${GSSAPI_LIBS} CACHE STRING "") > + set(GSSAPI_FLAVOR ${GSSAPI_FLAVOR} CACHE STRING "") > + > + mark_as_advanced(GSSAPI_INCS GSSAPI_LIBS GSSAPI_FLAVOR) > + > + endif(GSSAPI_LIBS) > + > + endif(KRB5_CONFIG) > + > +endif(GSSAPI_LIBS AND GSSAPI_FLAVOR) > diff --git a/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm b/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm > new file mode 100644 > index 0000000..3ffc6f1 > --- /dev/null > +++ b/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm > @@ -0,0 +1,47 @@ > +package My::Suite::AuthGSSAPI; > + > +@ISA = qw(My::Suite); > + > +return "No AUTH_GSSAPI plugin" unless $ENV{AUTH_GSSAPI_SO}; > + > +return "Not run for embedded server" if $::opt_embedded_server; > + > +# Following environment variables may need to be set > +if ($^O eq "MSWin32") > +{ > + chomp(my $whoami =`whoami /UPN 2>NUL` || `whoami`); don't you need to check whether the command worked and whether it returned something valid? > + my $fullname = $whoami; > + $fullname =~ s/\\/\\\\/; # SQL escaping for backslash > + $ENV{'GSSAPI_FULLNAME'} = $fullname; > + $ENV{'GSSAPI_SHORTNAME'} = $ENV{'USERNAME'}; > +} > +else > +{ > + if (!$ENV{'GSSAPI_FULLNAME'}) > + { > + my $s = `klist |grep 'Default principal: '`; > + if ($s) > + { > + chomp($s); > + my $fullname = substr($s,19); > + $ENV{'GSSAPI_FULLNAME'} = $fullname; > + } > + } > + $ENV{'GSSAPI_SHORTNAME'} = (split /@/, $ENV{'GSSAPI_FULLNAME'}) [0]; > +} > + > + > +if (!$ENV{'GSSAPI_FULLNAME'} || !$ENV{'GSSAPI_SHORTNAME'}) > +{ > + return "Environment variable GSSAPI_SHORTNAME and GSSAPI_FULLNAME need to be set" > +} > + > +foreach $var ('GSSAPI_SHORTNAME','GSSAPI_FULLNAME','GSSAPI_KEYTAB_PATH','GSSAPI_PRINCIPAL_NAME') > +{ > + print "$var=$ENV{$var}\n"; Really? It would make mtr runs a bit too verbose. at least put it inside if ($::opt_verbose) { ... } > +} > + > +sub is_default { 1 } > + > +bless { }; > + > diff --git a/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.opt b/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.opt > new file mode 100644 > index 0000000..3077d70 > --- /dev/null > +++ b/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.opt > @@ -0,0 +1 @@ > +--loose-gssapi-keytab-path=$GSSAPI_KEYTAB_PATH --loose-gssapi-principal-name=$GSSAPI_PRINCIPAL_NAME 1. who sets GSSAPI_KEYTAB_PATH and GSSAPI_PRINCIPAL_NAME? 2. if the user, do you need to check in suite.pm whether they're set? > diff --git a/plugin/auth_gssapi/mysql-test/auth_gssapi/basic.result b/plugin/auth_gssapi/mysql-test/auth_gssapi/basic.result > new file mode 100644 > index 0000000..a859ce5 > --- /dev/null > +++ b/plugin/auth_gssapi/mysql-test/auth_gssapi/basic.result > @@ -0,0 +1,18 @@ > +INSTALL SONAME 'auth_gssapi'; > +CREATE USER GSSAPI_SHORTNAME IDENTIFIED WITH gssapi; > +SELECT USER(),CURRENT_USER(); > +USER() CURRENT_USER() > +GSSAPI_SHORTNAME@localhost GSSAPI_SHORTNAME@% > +DROP USER GSSAPI_SHORTNAME; > +CREATE USER nosuchuser IDENTIFIED WITH gssapi; > +ERROR 28000: GSSAPI name mismatch, requested 'nosuchuser', actual name 'GSSAPI_SHORTNAME' What does the real unmasked error message look like? Is it something that client can know without connecting to the server? If not - this user-friendly verbose error is an information disclosure security vulnerability > +DROP USER nosuchuser; > +CREATE USER usr1 IDENTIFIED WITH gssapi as 'GSSAPI_FULLNAME'; > +SELECT USER(),CURRENT_USER(); > +USER() CURRENT_USER() > +usr1@localhost usr1@% > +DROP USER usr1; > +CREATE USER nosuchuser IDENTIFIED WITH gssapi AS 'nosuchuser(a)EXAMPLE.COM'; > +ERROR 28000: GSSAPI name mismatch, requested 'nosuchuser(a)EXAMPLE.COM', actual name 'GSSAPI_FULLNAME' > +DROP USER nosuchuser; > +UNINSTALL SONAME 'auth_gssapi'; > diff --git a/plugin/auth_gssapi/server_plugin.cc b/plugin/auth_gssapi/server_plugin.cc > new file mode 100644 > index 0000000..64f52a3 > --- /dev/null > +++ b/plugin/auth_gssapi/server_plugin.cc > @@ -0,0 +1,175 @@ > +/* Copyright (c) 2015, Shuang Qiu, Robbie Hardwood, > + Vladislav Vaintroub & MariaDB Corporation > + > + All rights reserved. > + > + Redistribution and use in source and binary forms, with or without > + modification, are permitted provided that the following conditions are met: > + > + 1. Redistributions of source code must retain the above copyright notice, > + this list of conditions and the following disclaimer. > + > + 2. Redistributions in binary form must reproduce the above copyright notice, > + this list of conditions and the following disclaimer in the documentation > + and/or other materials provided with the distribution. > + > + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" > + AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE > + ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE > + LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR > + CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF > + SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS > + INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN > + CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) > + ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE > + POSSIBILITY OF SUCH DAMAGE. 2-clause BSD. Was it Robbie who asked for it? > +*/ > + > +/** > + @file > + > + GSSAPI authentication plugin, server side > +*/ > +#include <my_sys.h> > +#include <mysqld_error.h> > +#include <mysql/plugin_auth.h> > +#include "server_plugin.h" > +#include "common.h" > + > +/* First packet sent from server to client, contains srv_principal_name\0mech\0 */ > +static char first_packet[PRINCIPAL_NAME_MAX + MECH_NAME_MAX +2]; > +static int first_packet_len; > + > +/* > + Target name in GSSAPI/SSPI , for Kerberos it is service principal name > + (often user principal name of the server user will work) > +*/ > +char *srv_principal_name; > +char *srv_keytab_path; > +char *srv_mech_name=(char *)""; > +unsigned long srv_mech; > + > +/** > + The main server function of the GSSAPI plugin. > + */ > +static int gssapi_auth(MYSQL_PLUGIN_VIO *vio, MYSQL_SERVER_AUTH_INFO *auth_info) > +{ > + int use_full_name; > + const char *user; > + int user_len; > + > + /* Send first packet with target name and mech name */ > + if (vio->write_packet(vio, (unsigned char *)first_packet, first_packet_len)) > + { > + return CR_ERROR; > + } > + > + /* Figure out whether to use full name (as given in IDENTIFIED AS clause) > + * or just short username auth_string > + */ > + if (auth_info->auth_string_length > 0) Hmm, I wonder how that can possibly work. auth_info->auth_string can only be set when the user is known. And the user is known only after the first read... ... ok ... got it. the first packet always uses the default, mysql_native_password, plugin so when the server switches to gssapi plugin, the user name is already known. Still, in case we'll have a --default-auth-plugin server option, I'd suggest to check auth_info->auth_string_length only after the first vio->read_packet() > + { > + use_full_name= 1; > + user= auth_info->auth_string; > + user_len= auth_info->auth_string_length; > + } > + else > + { > + use_full_name= 0; > + user= auth_info->user_name; > + user_len= auth_info->user_name_length; > + } > + > + return auth_server(vio, user, user_len, use_full_name); > +} > + > +static int initialize_plugin(void *unused) > +{ > + int rc; > + rc = plugin_init(); > + if (rc) > + return rc; > + > + strcpy(first_packet, srv_principal_name); > + strcpy(first_packet + strlen(srv_principal_name) + 1,srv_mech_name); > + first_packet_len = strlen(srv_principal_name) + strlen(srv_mech_name) + 2; in the server code I'd use strmov here, like first_packet_len = strmov(strmov(first_packet, srv_principal_name) + 1, srv_mech_name) - first_packet + 1; (may be with intermediate variables, for readability). I understand that a plugin might not want to do it, to avoid depending on server code outside of the plugin API. But if that's the case, why do you include my_sys.h? And mysqld_error.h? > + > + return 0; > +} > + > +static int deinitialize_plugin(void *unused) > +{ > + return plugin_deinit(); > +} > + > +/* system variable */ > +static MYSQL_SYSVAR_STR(keytab_path, srv_keytab_path, > + PLUGIN_VAR_RQCMDARG|PLUGIN_VAR_READONLY, > + "Keytab file path (Kerberos)", > + NULL, > + NULL, > + ""); > +static MYSQL_SYSVAR_STR(principal_name, srv_principal_name, > + PLUGIN_VAR_RQCMDARG|PLUGIN_VAR_READONLY, > + "GSSAPI target name - service principal name for Kerberos authentication.", A bit inconsistent. I'd suggest either to change the first to Keytab file path for Kerberos authentication or the second to GSSAPI target name - service principal name (Kerberos) > + NULL, > + NULL, > + ""); > +#ifdef PLUGIN_SSPI > +static const char* mech_names[] = { > + "Kerberos", > + "Negotiate", > + "", > + NULL > +}; > +static TYPELIB mech_name_typelib = { > + array_elements(mech_names) - 1, > + "mech_name_typelib", > + mech_names, > + NULL > +}; > +static MYSQL_SYSVAR_ENUM(mech_name, srv_mech, > + PLUGIN_VAR_RQCMDARG|PLUGIN_VAR_READONLY, > + "GSSAPI mechanism : either Kerberos or Negotiate", You don't need "either Kerberos or Negotiate" here, simply "GSSAPI mechanism" is enough. On --help, my_getopt automatically will add ", one of: Kerberos, Negotiate". Which is consistent with other ENUM options. It only makes sense to list values explicitly, when the list contains additional information, like "GSSAPI mechanism : either Kerberos or Negotiate (automatically choose between Kerberos and NTLM)." > + NULL, > + NULL, > + 2,&mech_name_typelib); > +#endif > + > +static struct st_mysql_sys_var *system_variables[]= { > + MYSQL_SYSVAR(principal_name), > +#ifdef PLUGIN_SSPI > + MYSQL_SYSVAR(mech_name), > +#endif > +#ifdef PLUGIN_GSSAPI > + MYSQL_SYSVAR(keytab_path), > +#endif > + NULL > +}; > + > +/* Register authentication plugin */ > +static struct st_mysql_auth server_handler= { > + MYSQL_AUTHENTICATION_INTERFACE_VERSION, > + "auth_gssapi_client", > + gssapi_auth > +}; > + > +maria_declare_plugin(gssapi_server) > +{ > + MYSQL_AUTHENTICATION_PLUGIN, > + &server_handler, > + "gssapi", > + "Shuang Qiu, Robbie Harwood, Vladislav Vaintroub", > + "Plugin for GSSAPI/SSPI based authentication.", > + PLUGIN_LICENSE_BSD, > + initialize_plugin, > + deinitialize_plugin, /* destructor */ > + 0x0100, /* version */ > + NULL, /* status variables */ > + system_variables, /* system variables */ > + "1.0", > + MariaDB_PLUGIN_MATURITY_EXPERIMENTAL I'd say BETA > +} > +maria_declare_plugin_end; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

2 2

[Maria-developers] Please review MDEV-8092 Change Create_field::field from "const char *" to LEX_CSTRING
by Alexander Barkov 11 Jan '16

11 Jan '16

Hi Serg, Please review a patch for MDEV-8092. Thanks. Perhaps something interesting, about #3 in the task description. I added a debug output and collected some statistics for "mtr --suite=main", to check how these changes can potentially affect another task: MDEV-7954 my_strcasecmp_utf8() takes 0.19% in OLTP RO Here are the results: my_strcasecmp_utf8() is now called 18,709,905 times total during "mtr --suite=main" 9,200,498 of those are called to compare Field/Create_field names in Column_name::eq_name(). And 3,855,751 calls of my_strcasecmp_utf8() are now optimized away, because eq_name() immediately return "false" if the two name lengths are different. So before this change there were 18,709,905 + 3,855,751 = 22,565,656 calls of my_strcasecmp_utf8(). Looks roughly like 17% improvement in name comparison.

3 10

[Maria-developers] Log warnings when upgrading from mysql 5.6 -> mariadb 10.0
by Otto Kekäläinen 11 Jan '16

11 Jan '16

Hello! When testing mysql 5.6 -> mariadb 10.0 I noticed that syslog looks like below. Are any of these errors anything we need to care about? The databases where completely vanilla out-of-the-box, so I imagine all users doing the same upgrade will experience the same error messages. I didn't notice that it would have any actual effect on anything, these error messages are just a QA signal. Jan 11 00:13:42 sid mysqld[22723]: _BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_A Jan 11 00:13:42 sid mysqld[22723]: 160111 0:13:42 [ERROR] mysqld: Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler. Jan 11 00:13:42 sid mysqld[22723]: 2016-01-11 00:13:42 7f6caa5ba700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is BINARY(4) NOT NULL but should be INT NOT NULL (type mismatch). Jan 11 00:13:42 sid mysqld[22723]: 2016-01-11 00:13:42 7f6caa5ba700 InnoDB: Error: Fetch of persistent statistics requested for table "mysql"."gtid_slave_pos" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. Jan 11 00:13:43 sid mysqld[22723]: 160111 0:13:43 [Note] /usr/sbin/mysqld: ready for connections. Jan 11 00:13:43 sid mysqld[22723]: Version: '10.0.23-MariaDB-2' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Debian unstable Jan 11 00:13:43 sid mysql[22552]: Starting MariaDB database server: mysqld. Jan 11 00:13:43 sid systemd[1]: Started LSB: Start and stop the mysql database server daemon. Jan 11 00:13:43 sid /etc/mysql/debian-start[22771]: Upgrading MySQL tables if necessary. Jan 11 00:14:00 sid mysqld[22723]: 2016-01-11 00:14:00 7f6caa5ba700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is BINARY(4) NOT NULL but should be INT NOT NULL (type mismatch).

3 5

[Maria-developers] Use of std::string
by Vicențiu Ciorbaru 07 Jan '16

07 Jan '16

Hi Sergei, I was doing some work within sql_acl.cc, having to refactor a bit of logic regarding USER_RESOURCES and SSL related stuff and I'm thinking of making use of std::string instead of const char* to remove the need for checking NULL pointers. I noticed that we generally do not make any use of std::string within the server code, apart from storage engines. The use cases are not performance sensitive so any overhead caused by std::string I think is smaller than the benefit of not having if (!NULL) statements everywhere. I know that it is obviously based on the final result, but if it is a definite no from the start, I'll implement things using regular C pointers. Also, this is only for the new things I'm implementing, I'm not planning on refactoring everything. Sample code where we can avoid ifs: case SSL_TYPE_SPECIFIED: table->field[next_field]->store(STRING_WITH_LEN("SPECIFIED"), &my_charset_latin1); table->field[next_field+1]->store("", 0, &my_charset_latin1); table->field[next_field+2]->store("", 0, &my_charset_latin1); table->field[next_field+3]->store("", 0, &my_charset_latin1); if (lex->ssl_cipher) table->field[next_field+1]->store(lex->ssl_cipher, strlen(lex->ssl_cipher), system_charset_info); if (lex->x509_issuer) table->field[next_field+2]->store(lex->x509_issuer, strlen(lex->x509_issuer), system_charset_info); if (lex->x509_subject) table->field[next_field+3]->store(lex->x509_subject, strlen(lex->x509_subject), system_charset_info); This just becomes: case SSL_TYPE_SPECIFIED: table->field[next_field]->store(STRING_WITH_LEN("SPECIFIED"), &my_charset_latin1); table->field[next_field+1]->store(ssl_cipher.c_str(), ssl_cipher.size(), &my_charset_latin1); table->field[next_field+2]->store(x509_issuer.c_str(), x509_issuer.size(), &my_charset_latin1); table->field[next_field+3]->store(x509_subject.c_str(), x509_subject.size(), &my_charset_latin1); Thoughts? Regards, Vicentiu

2 2

[Maria-developers] Oracle vs MariaDb
by Ghazi Btissam 07 Jan '16

07 Jan '16

Hi , I have to find out in MariaDb how to implement some features used in Oracle . I have: - Load a file: in Oracle I use the external table. Is there a way (fast and efficient one ) to load a file into a table . Has MariaDb a plugin which allows to load well a specific format of files? - In my existing Oracle code I used to developp a java wrap functions which are called by the Oracle Pl/Sql procedures , is there a similar way to call external java procedure in MariaDb? For example In my Pl/Sql code I can do this calling java proc : 1- Search a files in an OS directory and insert them in a table, 2- Send an SNMP trap 3- Send a mail via SMTP - Is there an equivalent to an Oracle job in Mariadb? - Is there an equivalent to Oracle TDE (Transparent data encryption) ? -Is there an equivalent to the VPD (virtual private policy)? Do you have some usefull links to help doing this study? (FAQs tutorial links..) Many Thanks and Best Regards

2 1

[Maria-developers] Not able to run MariaDB after building from source
by Sonali Ellath 05 Jan '16

05 Jan '16

Hi, I am Sonali looking to participate in this years Google Summer of Code. I was hoping to contribute to MariaDb. I tried building MariaDb from source using instructions found here <https://mariadb.com/kb/en/mariadb/generic-build-instructions/>. Configure, make and make install were successful. But I faced problem under section Starting MariaDB for the first time. When I executed the step: > scripts/mysql_install_db --user=mysql > I received an error saying: ... > [Note] InnoDB: The InnoDB memory heap is disabled > [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins > [Note] InnoDB: Memory barrier is not used > [Note] InnoDB: Compressed tables use zlib 1.2.8 > [Note] InnoDB: Using Linux native AIO > [Note] InnoDB: Using SSE crc32 instructions > [Note] InnoDB: Initializing buffer pool, size = 128.0M > [Note] InnoDB: Completed initialization of buffer pool > [ERROR] InnoDB: Data file ./ibdata1 is of a different size 768 pages > (rounded down to MB) than specified in the .cnf file 64000 pages! > [ERROR] InnoDB: Could not open or create the system tablespace. If you > tried to add new data files to the system tablespace, and it failed here, > you should now edit innodb_data_file_path in my.cnf back to what it was, > and remove the new ibdata files InnoDB created in this failed attempt. > InnoDB only wrote those files full of zeros, but did not yet use them in > any way. But be careful: do not remove old data files which contain your > precious data! > [ERROR] Plugin 'InnoDB' init function returned error. > [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. > [ERROR] Unknown/unsupported storage engine: InnoDB > [ERROR] Aborting > > Installation of system tables failed! Examine the logs in > ./data for more information. > The problem could be conflicting information in an external > my.cnf files. You can ignore these by doing: > > shell> ./bin/scripts/mysql_install_db --defaults-file=~/.my.cnf > > You can also try to start the mysqld daemon with: > > shell> ./bin/mysqld --skip-grant --general-log & > > and use the command line tool ./bin/mysql > to connect to the mysql database and look at the grant tables: > > shell> ./bin/mysql -u root mysql > mysql> show tables; > So I tried to run mysqld but that didn't work either. I had MySql installed earlier, but I removed it because it was getting very confusing with that. I also tried to run from source as mentioned here <https://mariadb.com/kb/en/mariadb/running-mariadb-from-the-source-directory/>. But it gives the following error: > ERROR 2002 (HY000): Can't connect to local MySQL server through socket > '/tmp/mysql.sock' (111 "Connection refused") I made sure that I uncommented lines from my.cnf which had sock information under client and server. Thanks in advance Sonali

2 1

[Maria-developers] MariaDB plugin package names (in Debian) scheme: mariadb-plugin-<name>
by Otto Kekäläinen 04 Jan '16

04 Jan '16

Hello! Currenlty we have storage engine packages named mariadb-oqgraph-engine-10.0 mariadb-connect-engine-10.0 Serg suggested in the summer that the version number should be dropped from the engine/plugin package names (it is enough to have it in the package metadata). I was now also thinking that the naming scheme of the base name should change to mariadb-plugin-oqgraph mariadb-plugin-connect The motivation is that plugin is more universal than 'engine' as some plugins are not engines, and by having all plugins prepended with mariadb-plugin-* they list nicer in alphabetical listings. What do you think? I already experimented in the lines of this with http://anonscm.debian.org/cgit/pkg-mysql/mariadb-10.0.git/commit/?id=97b3c4…

2 3

[Maria-developers] TokuDB QA issues - should we fix them ourself or are any of the upstream devs listening?
by Otto Kekäläinen 04 Jan '16

04 Jan '16

Hello! While working on the Debian packaging I noticed multiple minor QA level issues that stem from problems in TokuDB: - Debian builds are not reproducable due to random embedded build id in the TokuDB plugin, see https://reproducible.debian.net/dbd/unstable/amd64/mariadb-10.0_10.0.23-1.d… (subpage of https://reproducible.debian.net/rb-pkg/unstable/amd64/mariadb-10.0.html) - Lintian complains about missing man page for binary tokuftdump. Also Lintian complains about outdated autotool files in TokuDB sources. For both see https://lintian.debian.org/maintainer/pkg-mysql-maint@lists.alioth.debian.o… I'd like to see these fixed. What is the best process to proceed with these? File an issue at mariadb.org/jira or https://github.com/percona/PerconaFT or what..? Please advice. - Otto MariaDB Foundation

3 2

[Maria-developers] Documentation for new features
by Ian Gilfillan 03 Jan '16

03 Jan '16

Hi I quite often find out about a new feature after release, either by looking through the changelog and commits, or because someone points out it's missing from the documentation. It's time-consuming trying to keep track of every change, and I miss things (especially in non-default plugins etc), so it'd be better if everyone can remember the need for documentation when adding new features/variables or changes in behaviour. Each task of this sort in JIRA could have a related documentation subtask. You can add to the knowledgebase directly (don't worry about bad English, I will fix), email me with details, or at least alert me about an impending change. thanks, ian

4 4

[Maria-developers] Should these files be included in the Debian packages?
by Otto Kekäläinen 02 Jan '16

02 Jan '16

Hello! While working on MDEV-9259 I did some comparisons about what files in general are created by the build but which are missing from the Debian packaging. The list is below. Please advise which should be included and which are OK to leave out. etc/mysql/conf.d/cassandra.cnf etc/mysql/conf.d/dialog.cnf etc/mysql/conf.d/oqgraph.cnf usr/bin/mysql_config usr/bin/mysql_embedded usr/bin/mytop usr/include/mysql/*.h usr/lib/mysql/plugin/dialog.so usr/lib/mysql/plugin/mysql_clear_password.so usr/share/mysql/mroonga/install.sql usr/share/mysql/mroonga/uninstall.sql usr/share/mysql/my-huge.cnf usr/share/mysql/my-innodb-heavy-4G.cnf usr/share/mysql/my-large.cnf usr/share/mysql/my-medium.cnf usr/share/mysql/my-small.cnf usr/share/mysql/mysqld_multi.server usr/share/mysql/mysql-log-rotate usr/share/mysql/mysql.server usr/sql-bench/*

4 6

[Maria-developers] Student looking to contribute to MariaDB
by Rishab Jaju 31 Dec '15

31 Dec '15

Hello everybody, I am a third year student looking to contribute towards MariaDB for the upcoming Google Summer of Code program. I am very proficient in C++, Java, Python and Scheme. I have done some projects in machine learning as well as NLP. My exposure to databases is limited to an undergraduate course on Database systems. What level of knowledge would I need to contribute to the project? What sort of project would be appropriate for my skill set? Thank you, Rishab Jaju.

2 1

Re: [Maria-developers] [Commits] b904ca0: MDEV-9297 - mysql_install_db launches mysqld 3x
by Sergei Golubchik 29 Dec '15

29 Dec '15

Hi, Sergey! On Dec 23, Sergey Vojtovich wrote: > revision-id: b904ca0c2e9e35ffecb314f1082e45bfb42038f3 (mariadb-10.1.8-99-gb904ca0) > parent(s): 29290a3d53a05bd7177bcf7de05e1c8fff6e9cea > committer: Sergey Vojtovich > timestamp: 2015-12-23 15:52:34 +0400 > message: > > MDEV-9297 - mysql_install_db launches mysqld 3x > > Perform database installation in one mysqld run. This makes mysql_install_db > 3x faster. > > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > index f8f36bf..9dcd23a 100644 > --- a/scripts/mysql_install_db.sh > +++ b/scripts/mysql_install_db.sh > @@ -427,7 +427,7 @@ mysqld_install_cmd_line() > > # Create the system and help tables by passing them to "mysqld --bootstrap" > s_echo "Installing MariaDB/MySQL system tables in '$ldata' ..." > -if { echo "use mysql;"; cat "$create_system_tables" "$create_system_tables2" "$fill_system_tables"; } | eval "$filter_cmd_line" | mysqld_install_cmd_line > /dev/null > +if { echo "use mysql;"; cat "$create_system_tables" "$create_system_tables2" "$fill_system_tables" "$fill_help_tables" "$maria_add_gis_sp"; } | eval "$filter_cmd_line" | mysqld_install_cmd_line > /dev/null > then > s_echo "OK" > else > @@ -462,26 +462,6 @@ else > exit 1 > fi > > -s_echo "Filling help tables..." > -if { echo "use mysql;"; cat "$fill_help_tables"; } | mysqld_install_cmd_line > /dev/null > -then > - s_echo "OK" > -else > - echo > - echo "WARNING: HELP FILES ARE NOT COMPLETELY INSTALLED!" > - echo "The \"HELP\" command might not work properly." > -fi > - > -s_echo "Creating OpenGIS required SP-s..." > -if { echo "use test;"; cat "$maria_add_gis_sp"; } | mysqld_install_cmd_line > /dev/null > -then > - s_echo "OK" > -else > - echo > - echo "WARNING: OPENGIS REQUIRED SP-S WERE NOT COMPLETELY INSTALLED!" > - echo "GIS extentions might not work properly." > -fi Uhm, I don't know. I believe the point was not to treat fill_help_tables failure as fatal. To keep this logic you can merge maria_add_gis_sp and fill_help_tables in one mysqld_install_cmd_line invocation. But that's all. On the other hand, perhaps it's not important to preserve this logic? Then your patch is fine... Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

2 2

Re: [Maria-developers] [Commits] 05c122b: MDEV-8491 - On shutdown, report the user and the host executed that.
by Sergei Golubchik 28 Dec '15

28 Dec '15

Hi, Sergey! On Dec 24, Sergey Vojtovich wrote: > revision-id: 05c122b3ccd0102a3414a7d59313e6662a1fd5e4 (mariadb-10.1.8-101-g05c122b) > parent(s): 7c533a17523d2d6ef380487ab35151da0208090f > committer: Sergey Vojtovich > timestamp: 2015-12-24 19:53:44 +0400 > message: > > MDEV-8491 - On shutdown, report the user and the host executed that. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

1 0

[Maria-developers] Option to _not_ disable semisync on slave timeout?
by Kristian Nielsen 28 Dec '15

28 Dec '15

Hi Jonas (or anyone else who may know), If I understand correctly, semisync replication has a timeout for how long it will wait for a slave to acknowledge transactions. If the timeout is exceeded, semisync is turned off on the master. Do you know if there is a facility to avoid semisync being turned off automatically in this case, maybe instead fail and roll back the master transaction? I was thinking at Google you might already have needed and implemented this (merged or not merged into MariaDB yet), or you may be aware of a patch by someone else for this? Thanks, and merry Christmas! - Kristian.

3 2

Re: [Maria-developers] [Commits] a1ab431: MDEV-7261 - Backport the default autosized value of
by Sergei Golubchik 28 Dec '15

28 Dec '15

Hi, Sergey! On Nov 20, Sergey Vojtovich wrote: > revision-id: a1ab4314d1f88fa954a774c322709822d7b95344 (mariadb-10.1.8-58-ga1ab431) > parent(s): cfc135b645db5fa78b9f63b1959ea3f2be137a26 > committer: Sergey Vojtovich > timestamp: 2015-11-20 17:32:26 +0400 > message: > > MDEV-7261 - Backport the default autosized value of > "table_definition_cache" from MySQL 5.6.8+ > > Fixed table_definition_cache auto-sizing. Note that this patch doesn't > address broken PFS auto-sizing. Do you plan to address it after you've broken it? How? Regards, Sergei

2 9

[Maria-developers] ABI breakage in Connector/C sometime between 2.0.0 and 2.2.2
by Daniel Schepler 27 Dec '15

27 Dec '15

While preparing Debian packaging for MariaDB Connector/C version 2.2.2, I noticed that there's an ABI breakage without a corresponding change of SONAME: version 2.0.0 had a symbol "get_tty_password@Base" whereas version 2.2.2 doesn't. I'm wondering how big of a breakage it is - i.e. is that something that was never meant to be in the API/ABI in the first place, or is an oversight? I'm also wondering whether it's worth renaming the Debian package from libmariadb2 to libmariadb2a because of this. -- Daniel Schepler

1 0

Re: [Maria-developers] [Commits] 2b3bffb: MDEV-8491 - On shutdown, report the user and the host executed that.
by Sergei Golubchik 24 Dec '15

24 Dec '15

Hi, Sergey! On Nov 25, Sergey Vojtovich wrote: > revision-id: 2b3bffb42cc4adbec2d91c1b9c4028374b63a51a (mariadb-10.1.8-75-g2b3bffb) > parent(s): 6019fee7d84e8ec7d64337ad080a04f9c106bb68 > committer: Sergey Vojtovich > timestamp: 2015-11-25 18:12:19 +0400 > message: > > MDEV-8491 - On shutdown, report the user and the host executed that. First, a test would be nice. I suppose you can add it to the main.shutdown test. > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 0502841..0694cb0 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -1806,10 +1806,34 @@ static void close_server_sock() > #endif /*EMBEDDED_LIBRARY*/ > > > -void kill_mysql(void) > +static volatile char *shutdown_user; > +static void set_shutdown_user(THD *thd) > +{ > + char user_host_buff[MAX_USER_HOST_SIZE + 1]; > + char *user, *expected_shutdown_user= 0; > + Security_context *sctx= thd->security_ctx; > + > + strxnmov(user_host_buff, MAX_USER_HOST_SIZE, > + sctx->priv_user ? sctx->priv_user : "", "[", > + sctx->user ? sctx->user : (thd->slave_thread ? "SQL_SLAVE" : ""), > + "] @ ", > + sctx->host ? sctx->host : "", " [", > + sctx->ip ? sctx->ip : "", "]", NullS); this will be the fifth time this expression shows up in the source code. turn it into a function, perhaps? Like, an inline method of THD? also, sometimes it's "SQL_SLAVE", sometimes it's "", sometimes it's "unauthenticated", etc. I hope these differences can be removed and there will be one method that works identically for all occasions. Btw, you can use safe_str() helper here. > + > + if ((user= my_strdup(user_host_buff, MYF(0))) && > + !my_atomic_casptr((void **) &shutdown_user, > + (void **) &expected_shutdown_user, user)) > + my_free(user); > +} Interesting. Why is that? > + > + > +void kill_mysql(THD *thd) > { > DBUG_ENTER("kill_mysql"); > > + if (thd) > + set_shutdown_user(thd); > + > #if defined(SIGNALS_DONT_BREAK_READ) && !defined(EMBEDDED_LIBRARY) > abort_loop=1; // Break connection loops > close_server_sock(); // Force accept to wake up > @@ -1888,7 +1912,13 @@ static void __cdecl kill_server(int sig_ptr) > if (sig != 0) // 0 is not a valid signal number > my_sigset(sig, SIG_IGN); /* purify inspected */ > if (sig == MYSQL_KILL_SIGNAL || sig == 0) > - sql_print_information(ER_DEFAULT(ER_NORMAL_SHUTDOWN),my_progname); > + { > + char *user= (char *) my_atomic_loadptr((void**) &shutdown_user); > + sql_print_information(ER_DEFAULT(ER_NORMAL_SHUTDOWN), my_progname, > + user ? user : "unknown"); can it, really, be "unknown" here? when? > + if (user) > + my_free(user); > + } > else > sql_print_error(ER_DEFAULT(ER_GOT_SIGNAL),my_progname,sig); /* purecov: inspected */ > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 59908dc..f65d466 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -1735,29 +1735,29 @@ ER_WRONG_AUTO_KEY 42000 S1009 > ER_UNUSED_9 > eng "You should never see it" > ER_NORMAL_SHUTDOWN > - cze "%s: normální ukončení\n" > - dan "%s: Normal nedlukning\n" > - nla "%s: Normaal afgesloten \n" > - eng "%s: Normal shutdown\n" > - est "%s: MariaDB lõpetas\n" > - fre "%s: Arrêt normal du serveur\n" > - ger "%s: Normal heruntergefahren\n" > - greek "%s: Φυσιολογική διαδικασία shutdown\n" > - hun "%s: Normal leallitas\n" > - ita "%s: Shutdown normale\n" > - jpn "%s: 通常シャットダウン\n" > - kor "%s: 정상적인 shutdown\n" > - nor "%s: Normal avslutning\n" > - norwegian-ny "%s: Normal nedkopling\n" > - pol "%s: Standardowe zakończenie działania\n" > - por "%s: 'Shutdown' normal\n" > - rum "%s: Terminare normala\n" > - rus "%s: Корректная остановка\n" > - serbian "%s: Normalno gašenje\n" > - slo "%s: normálne ukončenie\n" > - spa "%s: Apagado normal\n" > - swe "%s: Normal avslutning\n" > - ukr "%s: Нормальне завершення\n" > + cze "%s: normální ukončení %s\n" > + dan "%s: Normal nedlukning %s\n" > + nla "%s: Normaal afgesloten %s\n" > + eng "%s: Normal shutdown by %s\n" > + est "%s: MariaDB lõpetas %s\n" > + fre "%s: Arrêt normal du serveur %s\n" > + ger "%s: Normal heruntergefahren %s\n" > + greek "%s: Φυσιολογική διαδικασία shutdown %s\n" > + hun "%s: Normal leallitas %s\n" > + ita "%s: Shutdown normale %s\n" > + jpn "%s: 通常シャットダウン %s\n" > + kor "%s: 정상적인 shutdown %s\n" > + nor "%s: Normal avslutning %s\n" > + norwegian-ny "%s: Normal nedkopling %s\n" > + pol "%s: Standardowe zakończenie działania %s\n" > + por "%s: 'Shutdown' normal %s\n" > + rum "%s: Terminare normala %s\n" > + rus "%s: Корректная остановка пользователем %s\n" > + serbian "%s: Normalno gašenje %s\n" > + slo "%s: normálne ukončenie %s\n" > + spa "%s: Apagado normal %s\n" > + swe "%s: Normal avslutning %s\n" > + ukr "%s: Нормальне завершення %s\n" Here you've fixed only the english and russian messages. Others would look pretty weird. Normal procedure in such a case is to remove all other languages and only keep those that you've fixed. In this case you can, perhaps, fix all of them in some easy way, like ger "%s: Normal heruntergefahren (%s)\n" ger "%s: Normal heruntergefahren / %s\n" ger "%s (%s): Normal heruntergefahren\n" ger "%s / %s: Normal heruntergefahren\n" or something else along these lines. See how it'll look like in the log: mysqld: Normal heruntergefahren foo [bar] @ localhost [127.0.0.1] mysqld: Normal heruntergefahren (foo [bar] @ localhost [127.0.0.1]) mysqld: Normal heruntergefahren / foo [bar] @ localhost [127.0.0.1] mysqld (foo [bar] @ localhost [127.0.0.1]): Normal heruntergefahren mysqld / foo [bar] @ localhost [127.0.0.1]: Normal heruntergefahren Regards, Sergei

2 3

Re: [Maria-developers] [Commits] ed5ec30: MDEV-7780 - Support for faking server version
by Sergei Golubchik 24 Dec '15

24 Dec '15

Hi, Sergey! On Dec 24, Sergey Vojtovich wrote: > revision-id: ed5ec30f9ca448086db5acbfd5f484a534119810 (mariadb-10.1.8-101-ged5ec30) > parent(s): 1f19c6aa93de730cd10c9fcbb337f9ec5c30ef9d > committer: Sergey Vojtovich > timestamp: 2015-12-24 15:46:18 +0400 > message: > > MDEV-7780 - Support for faking server version > > Embedded now supports "--version=<version>", while "--version" is still silently > ignored. Also only run protocol check in non-embedded mode. As I said on irc, I'd simply use not_embedded.inc, but ok, if you want it to work in embedded, sure, do it. One question below. > --- > mysql-test/suite/sys_vars/t/version.test | 3 ++- > sql/mysqld.cc | 4 ++-- > 2 files changed, 4 insertions(+), 3 deletions(-) > > diff --git a/mysql-test/suite/sys_vars/t/version.test b/mysql-test/suite/sys_vars/t/version.test > index 7955bf8..daa9538 100644 > --- a/mysql-test/suite/sys_vars/t/version.test > +++ b/mysql-test/suite/sys_vars/t/version.test > @@ -1,5 +1,6 @@ > SELECT @@version; > perl; > - $cnt= grep /my_favorite_version/, `$ENV{MYSQL} -e status`; > + $cnt= $ENV{MYSQL_TEST} =~ /mysqltest_embedded / ? 1 : > + grep /my_favorite_version/, `$ENV{MYSQL} -e status`; Why wouldn't you use mysql_embedded in this case? Like $mysql= $ENV{MYSQL_TEST} =~ /mysqltest_embedded / ? 'MYSQL_EMBEDDED' : 'MYSQL'; $cnt= grep /my_favorite_version/, `$ENV{$mysql} -e status`; > print "$cnt\n"; > EOF > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 7360204..fa3a70f 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -8972,20 +8972,20 @@ mysqld_get_one_option(int optid, const struct my_option *opt, char *argument) > binlog_format_used= true; > break; > #include <sslopt-case.h> > -#ifndef EMBEDDED_LIBRARY > case 'V': > if (argument) > { > strmake(server_version, argument, sizeof(server_version) - 1); > set_sys_var_value_origin(&server_version_ptr, sys_var::CONFIG); > } > +#ifndef EMBEDDED_LIBRARY > else > { > print_version(); > opt_abort= 1; // Abort after parsing all options > } > - break; > #endif /*EMBEDDED_LIBRARY*/ > + break; > case 'W': > if (!argument) > global_system_variables.log_warnings++; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

2 2

Re: [Maria-developers] [Commits] 370ab48: MDEV-9209 - [PATCH] scripts: Do not prepend the prefix to absolute paths
by Sergei Golubchik 22 Dec '15

22 Dec '15

Hi, Sergey! On Dec 18, Sergey Vojtovich wrote: > revision-id: 370ab48e9fcf4bc15ffa5e313fc34aa97981a832 (mariadb-10.1.9-24-g370ab48) > parent(s): 428e09a789a17211de9b4f0a2c4c1226d5dcf993 > committer: Sergey Vojtovich > timestamp: 2015-12-18 16:50:03 +0400 > message: > > MDEV-9209 - [PATCH] scripts: Do not prepend the prefix to absolute paths > > Allow absolute paths for INSTALL_*DIR. Okay, I agree now. Let's use your approach :) See below, one minor issue in mariadb.pc file. > diff --git a/scripts/CMakeLists.txt b/scripts/CMakeLists.txt > index eb78016..54289c0 100644 > --- a/scripts/CMakeLists.txt > +++ b/scripts/CMakeLists.txt > > IF(UNIX) > CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/mysql_install_db.sh > ${CMAKE_CURRENT_BINARY_DIR}/mysql_install_db ESCAPE_QUOTES @ONLY) > - SET(DEST ${INSTALL_SCRIPTDIR}) > SET(EXT) you can remove EXT as well, it's apparently not used anymore. > EXECUTE_PROCESS( > COMMAND chmod +x ${CMAKE_CURRENT_BINARY_DIR}/mysql_install_db > diff --git a/support-files/mariadb.pc.in b/support-files/mariadb.pc.in > index bc84a5c..0605d86 100644 > --- a/support-files/mariadb.pc.in > +++ b/support-files/mariadb.pc.in > @@ -1,18 +1,18 @@ > # these four variables are present in almost every .pc file > prefix=@CMAKE_INSTALL_PREFIX@ > exec_prefix=${prefix} > -libdir=${prefix}/@INSTALL_LIBDIR@ > -includedir=${prefix}/@INSTALL_INCLUDEDIR@ > +libdir=@INSTALL_LIBDIRABS@ > +includedir=@INSTALL_INCLUDEDIRABS@ This is, strictly speaking, incorrect. In the old version one can change prefix to any other value and all other paths will auto-adjust. See: $ pkg-config --libs openrc -L/lib64 -lrc $ pkg-config --libs openrc --define-variable=prefix=/opt -L/opt/lib64 -lrc Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

2 2

Re: [Maria-developers] [Commits] daad482: MDEV-8615: Assertion `m_cpp_buf <= begin_ptr && begin_ptr <= m_cpp_buf + m_buf_length' failed in Lex_input_stream::body_utf8_start
by Sergei Golubchik 22 Dec '15

22 Dec '15

Hi, Sanja! On Dec 21, OleksandrByelkin wrote: > revision-id: daad4828b078485ee7ac60af4fc165ae35684395 (mariadb-10.1.9-21-gdaad482) > parent(s): 59fcd7ff2315d007045eb987da5f21abbea6f6f1 > committer: Oleksandr Byelkin > timestamp: 2015-12-21 20:13:12 +0100 > message: > > MDEV-8615: Assertion `m_cpp_buf <= begin_ptr && begin_ptr <= m_cpp_buf + m_buf_length' failed in Lex_input_stream::body_utf8_start > > If it is EOF then it may not be start of compound statement. > > --- > mysql-test/r/compound.result | 7 +++++++ > mysql-test/t/compound.test | 7 +++++++ > sql/sql_yacc.yy | 7 ++++++- > 3 files changed, 20 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/compound.result b/mysql-test/r/compound.result > index 92d3226..1d412e6 100644 > --- a/mysql-test/r/compound.result > +++ b/mysql-test/r/compound.result > @@ -162,3 +162,10 @@ a > begin not atomic select a from t1 having a > 1; end| > a > drop table t1| > +# > +# MDEV-8615: Assertion `m_cpp_buf <= begin_ptr && > +# begin_ptr <= m_cpp_buf + m_buf_length' failed in > +# Lex_input_stream::body_utf8_start > +# > +b'| > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'b'' at line 1 > diff --git a/mysql-test/t/compound.test b/mysql-test/t/compound.test > index 1c5d12a..cabdf96 100644 > --- a/mysql-test/t/compound.test > +++ b/mysql-test/t/compound.test > @@ -150,3 +150,10 @@ select a from t1 having a > 1| > begin not atomic select a from t1 having a > 1; end| > drop table t1| > > +--echo # > +--echo # MDEV-8615: Assertion `m_cpp_buf <= begin_ptr && > +--echo # begin_ptr <= m_cpp_buf + m_buf_length' failed in > +--echo # Lex_input_stream::body_utf8_start > +--echo # > +--error ER_PARSE_ERROR > +--query b' > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index a5a62ae..6611411 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -237,6 +237,11 @@ static sp_head *make_sp_head(THD *thd, sp_name *name, > > static bool maybe_start_compound_statement(THD *thd) > { > + if (YYLIP->eof()) > + { > + my_parse_error(thd, ER_SYNTAX_ERROR); > + return 1; > + } no, I don't like that. Syntax errors should be issued by the parser, they should follow from the grammar, not be hard-coded in some explicit checks in the code. The problem in the current grammar - as I see it, that a code block is executed before any keyword is matched. This code block assumes it's the sp_unlabeled_control rule, while it could be anything. The fix would be to remove this code block and put this code after the first keyword is matched. > if (!thd->lex->sphead) > { > if (!make_sp_head(thd, NULL, TYPE_ENUM_PROCEDURE)) > @@ -4807,7 +4812,7 @@ create_body: > | opt_create_table_options opt_create_partitioning opt_create_select {} > /* > the following rule is redundant, but there's a shift/reduce > - conflict that prevents the rule above from parsing a syntax like > + conflict that prevents the rule above from parsing a synax like This is a strange change :) > CREATE TABLE t1 (SELECT 1); > */ Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

2 2

[Maria-developers] MDEV-8844 Unreadable control characters printed as is in warnings
by Alexander Barkov 22 Dec '15

22 Dec '15

Hi Sergei, Please review a patch for MDEV-8844. Thanks!

2 2

Re: [Maria-developers] [Commits] a030a28: MDEV-9095 - [PATCH] systemd capability for --memlock
by Sergei Golubchik 22 Dec '15

22 Dec '15

Hi, Sergey! On Dec 22, Sergey Vojtovich wrote: > revision-id: a030a2856bf39b9b9e743a09519b317e755b2833 (mariadb-10.1.9-26-ga030a28) > parent(s): fdce930ac6de9edfd53b851b542a4d07a95b3139 > committer: Sergey Vojtovich > timestamp: 2015-12-22 15:06:09 +0400 > message: > > MDEV-9095 - [PATCH] systemd capability for --memlock > > Adjust systemd files to enable CAP_IPC_LOCK to allow rootless mlockall > (triggered by memlock option). > > This is amended version of a patch originally submitted by Daniel Black. Ok to push. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

1 0

Re: [Maria-developers] [Commits] 59fcd7f: MDEV-9118 ANALYZE TABLE for Engine independent status fetchs blob/text columns without use
by Sergey Petrunia 22 Dec '15

22 Dec '15

Hi Sanja, So, the patch produces warnings whenever a column is not collected. I think they should be produced only if the columns were explicitly specified in ANALYZE TABLE ... COLUMNS (col1, ...) syntax. The idea is that we should issue a warning when we're clearly not doing what the user has requested. ANALYZE TABLE .. PERSISTENT FOR ALL should not produce a warning. "ALL" here means "ALL that statsistics is can be collected for", so it is reasonable that the server will not collect statistics which it is unable to collect (e.g. blobs). Ok to push after this has been addressed. On Fri, Dec 18, 2015 at 12:23:45PM +0100, OleksandrByelkin wrote: > revision-id: 59fcd7ff2315d007045eb987da5f21abbea6f6f1 (mariadb-10.1.9-20-g59fcd7f) > parent(s): 953d5680a3c050273a8f29253f7386984679f92b > committer: Oleksandr Byelkin > timestamp: 2015-12-18 12:23:45 +0100 > message: > > MDEV-9118 ANALYZE TABLE for Engine independent status fetchs blob/text columns without use > > Do not include BLOB fields by default. > > --- > mysql-test/r/analyze.result | 2 + > mysql-test/r/mysqlcheck.result | 60 ++++++++++++++++++++----- > mysql-test/r/statistics.result | 26 ++++++++++- > mysql-test/r/statistics_index_crash-7362.result | 2 + > mysql-test/t/statistics.test | 11 +++++ > sql/share/errmsg-utf8.txt | 3 ++ > sql/sql_admin.cc | 31 ++++++++++--- > 7 files changed, 117 insertions(+), 18 deletions(-) > > diff --git a/mysql-test/r/analyze.result b/mysql-test/r/analyze.result > index 9dff94a..5a05d1c 100644 > --- a/mysql-test/r/analyze.result > +++ b/mysql-test/r/analyze.result > @@ -34,9 +34,11 @@ create table t1 (a mediumtext, fulltext key key1(a)) charset utf8 collate utf8_g > insert into t1 values ('hello'); > analyze table t1; > Table Op Msg_type Msg_text > +test.t1 analyze Warning Engine-independent statistics are not collected for column 'a' > test.t1 analyze status OK > analyze table t1; > Table Op Msg_type Msg_text > +test.t1 analyze Warning Engine-independent statistics are not collected for column 'a' > test.t1 analyze status Table is already up to date > drop table t1; > CREATE TABLE t1 (a int); > diff --git a/mysql-test/r/mysqlcheck.result b/mysql-test/r/mysqlcheck.result > index 7d646dc..e69653f 100644 > --- a/mysql-test/r/mysqlcheck.result > +++ b/mysql-test/r/mysqlcheck.result > @@ -7,19 +7,34 @@ mtr.test_suppressions OK > mysql.column_stats OK > mysql.columns_priv OK > mysql.db OK > -mysql.event OK > +mysql.event > +Warning : Engine-independent statistics are not collected for column 'body' > +Warning : Engine-independent statistics are not collected for column 'body_utf8' > +status : OK > mysql.func OK > mysql.gtid_slave_pos OK > -mysql.help_category OK > +mysql.help_category > +Warning : Engine-independent statistics are not collected for column 'url' > +status : OK > mysql.help_keyword OK > mysql.help_relation OK > -mysql.help_topic OK > +mysql.help_topic > +Warning : Engine-independent statistics are not collected for column 'description' > +Warning : Engine-independent statistics are not collected for column 'example' > +Warning : Engine-independent statistics are not collected for column 'url' > +status : OK > mysql.host OK > mysql.index_stats OK > mysql.innodb_index_stats OK > mysql.innodb_table_stats OK > mysql.plugin OK > -mysql.proc OK > +mysql.proc > +Warning : Engine-independent statistics are not collected for column 'param_list' > +Warning : Engine-independent statistics are not collected for column 'returns' > +Warning : Engine-independent statistics are not collected for column 'body' > +Warning : Engine-independent statistics are not collected for column 'comment' > +Warning : Engine-independent statistics are not collected for column 'body_utf8' > +status : OK > mysql.procs_priv OK > mysql.proxies_priv OK > mysql.roles_mapping OK > @@ -31,7 +46,12 @@ mysql.time_zone_leap_second OK > mysql.time_zone_name OK > mysql.time_zone_transition OK > mysql.time_zone_transition_type OK > -mysql.user OK > +mysql.user > +Warning : Engine-independent statistics are not collected for column 'ssl_cipher' > +Warning : Engine-independent statistics are not collected for column 'x509_issuer' > +Warning : Engine-independent statistics are not collected for column 'x509_subject' > +Warning : Engine-independent statistics are not collected for column 'authentication_string' > +status : OK > mtr.global_suppressions Table is already up to date > mtr.test_suppressions Table is already up to date > mysql.column_stats OK > @@ -69,19 +89,34 @@ mysql.user OK > mysql.column_stats OK > mysql.columns_priv OK > mysql.db OK > -mysql.event OK > +mysql.event > +Warning : Engine-independent statistics are not collected for column 'body' > +Warning : Engine-independent statistics are not collected for column 'body_utf8' > +status : OK > mysql.func OK > mysql.gtid_slave_pos OK > -mysql.help_category OK > +mysql.help_category > +Warning : Engine-independent statistics are not collected for column 'url' > +status : OK > mysql.help_keyword OK > mysql.help_relation OK > -mysql.help_topic OK > +mysql.help_topic > +Warning : Engine-independent statistics are not collected for column 'description' > +Warning : Engine-independent statistics are not collected for column 'example' > +Warning : Engine-independent statistics are not collected for column 'url' > +status : OK > mysql.host OK > mysql.index_stats OK > mysql.innodb_index_stats OK > mysql.innodb_table_stats OK > mysql.plugin OK > -mysql.proc OK > +mysql.proc > +Warning : Engine-independent statistics are not collected for column 'param_list' > +Warning : Engine-independent statistics are not collected for column 'returns' > +Warning : Engine-independent statistics are not collected for column 'body' > +Warning : Engine-independent statistics are not collected for column 'comment' > +Warning : Engine-independent statistics are not collected for column 'body_utf8' > +status : OK > mysql.procs_priv OK > mysql.proxies_priv OK > mysql.roles_mapping OK > @@ -93,7 +128,12 @@ mysql.time_zone_leap_second OK > mysql.time_zone_name OK > mysql.time_zone_transition OK > mysql.time_zone_transition_type OK > -mysql.user OK > +mysql.user > +Warning : Engine-independent statistics are not collected for column 'ssl_cipher' > +Warning : Engine-independent statistics are not collected for column 'x509_issuer' > +Warning : Engine-independent statistics are not collected for column 'x509_subject' > +Warning : Engine-independent statistics are not collected for column 'authentication_string' > +status : OK > mysql.column_stats Table is already up to date > mysql.columns_priv Table is already up to date > mysql.db Table is already up to date > diff --git a/mysql-test/r/statistics.result b/mysql-test/r/statistics.result > index bd6a084..fdda572 100644 > --- a/mysql-test/r/statistics.result > +++ b/mysql-test/r/statistics.result > @@ -1212,6 +1212,7 @@ test t2 PRIMARY 1 1.0000 > ANALYZE TABLE t1; > Table Op Msg_type Msg_text > test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze Warning Engine-independent statistics are not collected for column 'b' > test.t1 analyze status OK > SELECT * FROM mysql.column_stats; > db_name table_name column_name min_value max_value nulls_ratio avg_length avg_frequency hist_size hist_type histogram > @@ -1225,7 +1226,6 @@ test t1 c aaaa dddddddd 0.1250 6.6571 7.0000 0 NULL NULL > test t1 d 1989-03-12 1999-07-23 0.1500 3.0000 8.5000 0 NULL NULL > test t1 e 0.01 0.112 0.2250 8.0000 6.2000 0 NULL NULL > test t1 f 1 5 0.2000 1.0000 6.4000 0 NULL NULL > -test t1 b NULL NULL 0.2000 17.1250 NULL NULL NULL NULL > SELECT * FROM mysql.index_stats; > db_name table_name index_name prefix_arity avg_frequency > test t1 idx2 1 7.0000 > @@ -1265,6 +1265,7 @@ set use_stat_tables='never'; > ANALYZE TABLE t1 PERSISTENT FOR ALL; > Table Op Msg_type Msg_text > test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze Warning Engine-independent statistics are not collected for column 'b' > test.t1 analyze status Table is already up to date > SELECT * FROM mysql.table_stats; > db_name table_name cardinality > @@ -1276,7 +1277,6 @@ test t1 c aaaa dddddddd 0.1250 6.6571 7.0000 0 NULL NULL > test t1 d 1989-03-12 1999-07-23 0.1500 3.0000 8.5000 0 NULL NULL > test t1 e 0.01 0.112 0.2250 8.0000 6.2000 0 NULL NULL > test t1 f 1 5 0.2000 1.0000 6.4000 0 NULL NULL > -test t1 b NULL NULL 0.2000 17.1250 NULL NULL NULL NULL > SELECT * FROM mysql.index_stats; > db_name table_name index_name prefix_arity avg_frequency > test t1 PRIMARY 1 1.0000 > @@ -1291,6 +1291,28 @@ test t1 idx4 3 NULL > DELETE FROM mysql.table_stats; > DELETE FROM mysql.column_stats; > DELETE FROM mysql.index_stats; > +ANALYZE TABLE t1 PERSISTENT FOR COLUMNS(b) INDEXES(); > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze Warning Engine-independent statistics are not collected for column 'b' > +test.t1 analyze status Table is already up to date > +ANALYZE TABLE t1 PERSISTENT FOR columns(a,b) INDEXES(); > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze Warning Engine-independent statistics are not collected for column 'b' > +test.t1 analyze status Table is already up to date > +ANALYZE TABLE t1 PERSISTENT FOR columns(b) indexes(idx2); > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze Warning Engine-independent statistics are not collected for column 'b' > +test.t1 analyze status Table is already up to date > +ANALYZE TABLE t1 PERSISTENT FOR columns() indexes(idx2); > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status Table is already up to date > +DELETE FROM mysql.table_stats; > +DELETE FROM mysql.column_stats; > +DELETE FROM mysql.index_stats; > DROP TABLE t1,t2; > set names utf8; > CREATE DATABASE world; > diff --git a/mysql-test/r/statistics_index_crash-7362.result b/mysql-test/r/statistics_index_crash-7362.result > index 99f65d7..c213fa4 100644 > --- a/mysql-test/r/statistics_index_crash-7362.result > +++ b/mysql-test/r/statistics_index_crash-7362.result > @@ -3,6 +3,7 @@ INSERT INTO t1 VALUES (unhex('3E0D0A4141414142334E7A6143317963324541414141424977 > ANALYZE TABLE t1 PERSISTENT FOR ALL; > Table Op Msg_type Msg_text > test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze Warning Engine-independent statistics are not collected for column 'a' > test.t1 analyze status OK > SELECT * FROM mysql.index_stats WHERE index_name='a' AND table_name='t1'; > db_name table_name index_name prefix_arity avg_frequency > @@ -13,6 +14,7 @@ INSERT INTO t1 VALUES (unhex('3E0D0A4141414142334E7A6143317963324541414141424977 > ANALYZE TABLE t1 PERSISTENT FOR ALL; > Table Op Msg_type Msg_text > test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze Warning Engine-independent statistics are not collected for column 'a' > test.t1 analyze status OK > SELECT * FROM mysql.index_stats WHERE index_name='a' AND table_name='t1'; > db_name table_name index_name prefix_arity avg_frequency > diff --git a/mysql-test/t/statistics.test b/mysql-test/t/statistics.test > index 2c8dec3..3b57b8f 100644 > --- a/mysql-test/t/statistics.test > +++ b/mysql-test/t/statistics.test > @@ -494,6 +494,17 @@ DELETE FROM mysql.table_stats; > DELETE FROM mysql.column_stats; > DELETE FROM mysql.index_stats; > > + > +ANALYZE TABLE t1 PERSISTENT FOR COLUMNS(b) INDEXES(); > +ANALYZE TABLE t1 PERSISTENT FOR columns(a,b) INDEXES(); > +ANALYZE TABLE t1 PERSISTENT FOR columns(b) indexes(idx2); > +ANALYZE TABLE t1 PERSISTENT FOR columns() indexes(idx2); > + > +DELETE FROM mysql.table_stats; > +DELETE FROM mysql.column_stats; > +DELETE FROM mysql.index_stats; > + > + > DROP TABLE t1,t2; > > set names utf8; > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 59908dc..178d1dd 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7136,3 +7136,6 @@ ER_KILL_QUERY_DENIED_ERROR > eng "You are not owner of query %lu" > ger "Sie sind nicht Eigentümer von Abfrage %lu" > rus "Вы не являетесь владельцем запроса %lu" > +ER_NO_EIS_FOR_FIELD > + eng "Engine-independent statistics are not collected for column '%s'" > + ukr "Незалежна від типу таблиці статистика не збирається для стовбця '%s'" > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index 0787aa9..d8ca863 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -692,10 +692,20 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > } > > if (!lex->column_list) > - { > - uint fields= 0; > - for ( ; *field_ptr; field_ptr++, fields++) ; > - bitmap_set_prefix(tab->read_set, fields); > + { > + bitmap_clear_all(tab->read_set); > + for (uint fields= 0; *field_ptr; field_ptr++, fields++) > + { > + enum enum_field_types type= (*field_ptr)->type(); > + if (type < MYSQL_TYPE_MEDIUM_BLOB || > + type > MYSQL_TYPE_BLOB) > + bitmap_set_bit(tab->read_set, fields); > + else > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_NO_EIS_FOR_FIELD, > + ER_THD(thd, ER_NO_EIS_FOR_FIELD), > + (*field_ptr)->field_name); > + } > } > else > { > @@ -713,8 +723,17 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > compl_result_code= result_code= HA_ADMIN_INVALID; > break; > } > - bitmap_set_bit(tab->read_set, pos-1); > - } > + pos--; > + enum enum_field_types type= tab->field[pos]->type(); > + if (type < MYSQL_TYPE_MEDIUM_BLOB || > + type > MYSQL_TYPE_BLOB) > + bitmap_set_bit(tab->read_set, pos); > + else > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_NO_EIS_FOR_FIELD, > + ER_THD(thd, ER_NO_EIS_FOR_FIELD), > + column_name->str); > + } > tab->file->column_bitmaps_signal(); > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] [Commits] 063967b: MDEV-9081 - Debian: insecure debian-sys-maint password handling
by Sergei Golubchik 21 Dec '15

21 Dec '15

Hi, Sergey! On Dec 21, Sergey Vojtovich wrote: > revision-id: 063967b1e447f0fc908a1ec0224fd7c1d268bf17 (mariadb-10.1.9-25-g063967b) > parent(s): 370ab48e9fcf4bc15ffa5e313fc34aa97981a832 > committer: Sergey Vojtovich > timestamp: 2015-12-21 14:40:41 +0400 > message: > > MDEV-9081 - Debian: insecure debian-sys-maint password handling > > Set file permission before password is actually written. This is a quick fix > to close security gap. To be replaced by MDEV-8375 - passwordless root login. > > diff --git a/debian/mariadb-server-10.1.postinst b/debian/mariadb-server-10.1.postinst > index 0f35802..069c25e 100644 > --- a/debian/mariadb-server-10.1.postinst > +++ b/debian/mariadb-server-10.1.postinst > @@ -158,6 +158,8 @@ EOF > pass=`perl -e 'print map{("a".."z","A".."Z",0..9)[int(rand(62))]}(1..16)'`; > if [ ! -d "$mysql_cfgdir" ]; then install -o 0 -g 0 -m 0755 -d $mysql_cfgdir; fi > cat /dev/null > $dc > + chown 0:0 $dc is that needed? (not that it hurts, but still...) > + chmod 0600 $dc ok. what about other issues you've mentioned in this MDEV-9081? besides, what the plan for moving to unix_socket auth? > echo "# Automatically generated for Debian scripts. DO NOT TOUCH!" >>$dc > echo "[client]" >>$dc > echo "host = localhost" >>$dc Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

2 5

Re: [Maria-developers] [Commits] 6414959: MDEV-7526: TokuDB doesn't build on OS X
by Sergei Golubchik 21 Dec '15

21 Dec '15

Hi, Vicentiu! On Dec 19, Vicentiu Ciorbaru wrote: > revision-id: 64149590c47d1cf6b1b227d8c90bdc23d20a8956 (mariadb-5.5.47-8-g6414959) > parent(s): f89c9fc4b7b5d82c79775cb848225900b45a6b79 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2015-12-19 14:14:10 +0200 > message: > > MDEV-7526: TokuDB doesn't build on OS X > > This patch fixes another compilation error caused by specifying > attribute nonnull for all the parameters of the copyout function. This > is incorrect as the function actually gets called with null parameters > indirectly and thus only the output parameter should be nonnull. > > diff --git a/storage/tokudb/ft-index/util/dmt.h b/storage/tokudb/ft-index/util/dmt.h > index d4b032f..43e44df 100644 > --- a/storage/tokudb/ft-index/util/dmt.h > +++ b/storage/tokudb/ft-index/util/dmt.h > @@ -679,16 +679,16 @@ class dmt { > __attribute__((nonnull)) > void rebalance(subtree *const subtree); > > - __attribute__((nonnull)) > + __attribute__((nonnull(3))) > static void copyout(uint32_t *const outlen, dmtdata_t *const out, const dmt_node *const n); Where is copyout called with the null parameter? In 10.0 they've fixed it differently, it seems E.g. if (value != nullptr) { copyout(value, &this->d.a.values[best_zero]); } May be we'd better use the upstream fix? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

3 3

Re: [Maria-developers] [Commits] ea741d9: MDEV-8627: SHOW GRANTS does not work for a replicated role
by Sergei Golubchik 20 Dec '15

20 Dec '15

Hi, Vicentiu! On Dec 19, Vicentiu Ciorbaru wrote: > revision-id: ea741d98566e95aa07ff94539dfc751c5b42dd44 (mariadb-10.1.9-24-gea741d9) > parent(s): 428e09a789a17211de9b4f0a2c4c1226d5dcf993 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2015-12-19 00:35:14 +0200 > message: > > MDEV-8627: SHOW GRANTS does not work for a replicated role > > The bug was caused by accessing uninitialized fields within the LEX related to > ssl by mysql_show_grants() -> get_current_user() -> has_auth() function. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

1 0

Re: [Maria-developers] [Commits] 591e74c: MDEV-7526: TokuDB doesn't build on OS X
by Sergei Golubchik 20 Dec '15

20 Dec '15

Hi, Vicentiu! On Dec 19, Vicentiu Ciorbaru wrote: > revision-id: 591e74c7e8f682f2eada7496f50aa73f38bd168d (mariadb-5.5.47-11-g591e74c) > parent(s): e386523a41245d8b03f1338934c1aa965530e4fc > author: DevilSatan > committer: Vicențiu Ciorbaru > timestamp: 2015-12-19 14:14:16 +0200 > message: > > MDEV-7526: TokuDB doesn't build on OS X > > A string definition is inconsistent and thus causes a compilation error. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

1 0

Re: [Maria-developers] [Commits] e386523: MDEV-7526: TokuDB doesn't build on OS X
by Sergei Golubchik 20 Dec '15

20 Dec '15

Hi, Vicentiu! On Dec 19, Vicentiu Ciorbaru wrote: > revision-id: e386523a41245d8b03f1338934c1aa965530e4fc (mariadb-5.5.47-10-ge386523) > parent(s): f39b9e04db1e71e72bfc19174f257db5147df440 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2015-12-19 14:14:16 +0200 > message: > > MDEV-7526: TokuDB doesn't build on OS X > > Fixed compile warning related to if statement always being true. The if > statement can not be false, as the address of a member field is always > true. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org -- Vote for my Percona Live 2016 talks: https://www.percona.com/live/data-performance-conference-2016/sessions/mari… https://www.percona.com/live/data-performance-conference-2016/sessions/mari…

1 0