- developers - lists.mariadb.org

[Maria-developers] MDEV-8372 Use helper methods introduced in MDEV-7824 all around the code
by Alexander Barkov 11 Sep '15

11 Sep '15

Hi Sergei, Please review my patch for mdev-8372. I found only a few places to reuse the new methods. Also, had to add some more helper methods to make the code clearer. Thanks.

2 3

Re: [Maria-developers] [Commits] 32b1497: MDEV-7990: ERROR 1526 when procedure executed for second time ALTER TABLE partition ... pMAX values less than MAXVALUE
by Sergei Golubchik 11 Sep '15

11 Sep '15

Hi, Sanja! On Aug 06, sanja(a)mariadb.com wrote: > revision-id: 32b14970a1b88571aa96f604e5c59ab6bf1b298c > parent(s): aa5095627e2619bdad7916d33d1016802a84a9e1 > committer: Oleksandr Byelkin > branch nick: server > timestamp: 2015-08-06 14:00:48 +0200 > message: > > MDEV-7990: ERROR 1526 when procedure executed for second time ALTER > TABLE partition ... pMAX values less than MAXVALUE > > Added cleaning partition data which are saved in LEX structure to make > possible correct re-execution of a SP or PS using the code. This seems to be wrong now. It's an old comment, but now your fix works differently, doesn't it? The patch is ok to push, thanks! Regards, Sergei

1 0

Re: [Maria-developers] [Commits] 0a1b60a: MDEV-8450: PATCH] Wrong macro expansion in Query_cache::send_result_to_client()
by Sergey Vojtovich 06 Sep '15

06 Sep '15

Hi Sanja, ok to push, a few minor notes JFYI inline. I found only one real bug, but generally I agree that we should fix these fragile macros. On Tue, Sep 01, 2015 at 11:47:09AM +0200, sanja(a)mariadb.com wrote: > revision-id: 0a1b60a8aee4bce3235fd5a1d8cb2231b51ff5cc (mariadb-10.0.21-7-g0a1b60a) > parent(s): 4b41e3c7f33714186c97a6cc2e6d3bb93b050c61 > committer: Oleksandr Byelkin > timestamp: 2015-09-01 11:47:06 +0200 > message: > > MDEV-8450: PATCH] Wrong macro expansion in Query_cache::send_result_to_client() > > Expression in macro protected by () > > diff --git a/include/maria.h b/include/maria.h > index 16a9bea..cdaeade 100644 > --- a/include/maria.h > +++ b/include/maria.h > @@ -43,7 +43,7 @@ extern "C" { > #define MARIA_NAME_IEXT ".MAI" > #define MARIA_NAME_DEXT ".MAD" > /* Max extra space to use when sorting keys */ > -#define MARIA_MAX_TEMP_LENGTH 2*1024L*1024L*1024L > +#define MARIA_MAX_TEMP_LENGTH (2*1024L*1024L*1024L) > /* Possible values for maria_block_size (must be power of 2) */ > #define MARIA_KEY_BLOCK_LENGTH 8192 /* default key block length */ > #define MARIA_MIN_KEY_BLOCK_LENGTH 1024 /* Min key block length */ Was this used at all? May be remove it? > diff --git a/sql/log_slow.h b/sql/log_slow.h > index 2ae07da..3ae2060 100644 > --- a/sql/log_slow.h > +++ b/sql/log_slow.h > @@ -16,23 +16,23 @@ > /* Defining what to log to slow log */ > > #define LOG_SLOW_VERBOSITY_INIT 0 > -#define LOG_SLOW_VERBOSITY_INNODB 1 << 0 > -#define LOG_SLOW_VERBOSITY_QUERY_PLAN 1 << 1 > -#define LOG_SLOW_VERBOSITY_EXPLAIN 1 << 2 > +#define LOG_SLOW_VERBOSITY_INNODB (1 << 0) Was LOG_SLOW_VERBOSITY_INNODB used at all? May be remove it? > +#define LOG_SLOW_VERBOSITY_QUERY_PLAN (1 << 1) > +#define LOG_SLOW_VERBOSITY_EXPLAIN (1 << 2) > > #define QPLAN_INIT QPLAN_QC_NO > > -#define QPLAN_ADMIN 1 << 0 > -#define QPLAN_FILESORT 1 << 1 > -#define QPLAN_FILESORT_DISK 1 << 2 > -#define QPLAN_FULL_JOIN 1 << 3 > -#define QPLAN_FULL_SCAN 1 << 4 > -#define QPLAN_QC 1 << 5 > -#define QPLAN_QC_NO 1 << 6 > -#define QPLAN_TMP_DISK 1 << 7 > -#define QPLAN_TMP_TABLE 1 << 8 > -#define QPLAN_FILESORT_PRIORITY_QUEUE 1 << 9 > +#define QPLAN_ADMIN (1 << 0) > +#define QPLAN_FILESORT (1 << 1) > +#define QPLAN_FILESORT_DISK (1 << 2) > +#define QPLAN_FULL_JOIN (1 << 3) > +#define QPLAN_FULL_SCAN (1 << 4) > +#define QPLAN_QC (1 << 5) > +#define QPLAN_QC_NO (1 << 6) > +#define QPLAN_TMP_DISK (1 << 7) > +#define QPLAN_TMP_TABLE (1 << 8) > +#define QPLAN_FILESORT_PRIORITY_QUEUE (1 << 9) > > /* ... */ > -#define QPLAN_MAX ((ulong) 1) << 31 /* reserved as placeholder */ > +#define QPLAN_MAX (((ulong) 1) << 31) /* reserved as placeholder */ > log_slow.h was fixed in rev.203f4d419 in my 10.0. Did you use outdated tree? I may be wrong, but there seem to be no real bug with QPLAN_QC_NO currently. Just because nobody sets it. So in fact it clears QPLAN_ADMIN, which FWICS is never set at this point. But it is a good to have it fixed nevertheless. Btw, QPLAN_QC_NO doesn't seem to be used in fact. May be remove it? > diff --git a/storage/archive/archive_test.c b/storage/archive/archive_test.c > index d01c1e0..bb052b8 100644 > --- a/storage/archive/archive_test.c > +++ b/storage/archive/archive_test.c > @@ -33,7 +33,7 @@ > > #define ARCHIVE_ROW_HEADER_SIZE 4 > > -#define BUFFER_LEN 1024 + ARCHIVE_ROW_HEADER_SIZE > +#define BUFFER_LEN (1024 + ARCHIVE_ROW_HEADER_SIZE) > > char test_string[BUFFER_LEN]; Alright, there was a bug in this line: assert(write_length != count * BUFFER_LEN); > diff --git a/strings/ctype.c b/strings/ctype.c > index 048fbe3..d8a1dd7 100644 > --- a/strings/ctype.c > +++ b/strings/ctype.c > @@ -264,7 +264,7 @@ static const struct my_cs_file_section_st > } > > #define MY_CS_CSDESCR_SIZE 64 > -#define MY_CS_TAILORING_SIZE 32*1024 > +#define MY_CS_TAILORING_SIZE (32*1024) > #define MY_CS_UCA_VERSION_SIZE 64 > #define MY_CS_CONTEXT_SIZE 64 Was this used at all? May be remove it? Regards, Sergey

2 1

Re: [Maria-developers] [Commits] 62fe87c: MDEV-8663: IF Statement returns multiple values erroneously (or Assertion `!null_value' failed in Item::send(Protocol*, String*))
by Alexander Barkov 04 Sep '15

04 Sep '15

Hi Sanja, From my understanding, the patch fixes consequences of the problem, but not the problem itself. It very likely will still have crashes with slightly modified SQL scripts. The source of the problem is that Item_func_hybrid_result_type::val_decimal() does not return NULL when int_op() or real_op() set null_value to "true". By the way, compare Item_func_hybrid_result_type::val_decimal() with a very similar method Item_func_hybrid_result_type::val_str(). The latter already handle the INT_RESULT and REAL_RESULT cases well and return NULL properly in case of null_value. get_date(), val_int(), val_real() also look fine for me. So the are problems only in val_decimal(). I propose a small patch that fixes the source of the problem. This is the small patch: > @@ -1038,17 +1026,21 @@ my_decimal *Item_func_hybrid_result_type::val_decimal(my_decimal *decimal_value) > case INT_RESULT: > { > longlong result= int_op(); > + if (null_value) > + return NULL; > int2my_decimal(E_DEC_FATAL_ERROR, result, unsigned_flag, decimal_value); > break; > } > case REAL_RESULT: > { > double result= (double)real_op(); > + if (null_value) > + return NULL; > double2my_decimal(E_DEC_FATAL_ERROR, result, decimal_value); > break; > } I also propose an extended patch version, to cover the code with more DBUG_ASSERTs, so in case of the null_value related bugs it crashes much earlier than in Item::send(). It will be easier to debug problems this way. To avoid much code duplication, I added three private helper methods. The extended version of the patch is attached. The scripts from the bug report start to return expected results, with no crashes in DEBUG build, and "mtr --suite=main" works fine. I have not run the entire mtr though, to save time. Greetings. On 09/03/2015 08:00 PM, sanja(a)mariadb.com wrote: > revision-id: 62fe87c5a239c7304b245ce5954c87aecb6382ad (mariadb-5.5.45-1-g62fe87c) > parent(s): fa51f70dc68fe2f3afe943e2c81fcbdb34f16cad > committer: Oleksandr Byelkin > timestamp: 2015-09-03 18:00:43 +0200 > message: > > MDEV-8663: IF Statement returns multiple values erroneously (or Assertion `!null_value' failed in Item::send(Protocol*, String*)) > > keeping contract: NULL value mean NULL pointer in val_str and val_deciman. > > --- > mysql-test/r/func_if.result | 17 +++++++++++++++++ > mysql-test/t/func_if.test | 14 ++++++++++++++ > sql/item_cmpfunc.cc | 6 ++++-- > sql/item_func.cc | 5 +++++ > 4 files changed, 40 insertions(+), 2 deletions(-) > > diff --git a/mysql-test/r/func_if.result b/mysql-test/r/func_if.result > index c7f548a..61f63cc 100644 > --- a/mysql-test/r/func_if.result > +++ b/mysql-test/r/func_if.result > @@ -234,3 +234,20 @@ SELECT if(1, NULL, (SELECT min('hello'))); > if(1, NULL, (SELECT min('hello'))) > NULL > End of 5.2 tests > +# > +# MDEV-8663: IF Statement returns multiple values erroneously > +# (or Assertion `!null_value' failed in Item::send(Protocol*, String*) > +# > +CREATE TABLE `t1` ( > +`datas` VARCHAR(25) NOT NULL > +) ENGINE=InnoDB DEFAULT CHARSET=utf8; > +Warnings: > +Warning 1286 Unknown storage engine 'InnoDB' > +Warning 1266 Using storage engine MyISAM for table 't1' > +INSERT INTO `t1` VALUES ('1,2'), ('2,3'), ('3,4'); > +SELECT IF(FIND_IN_SET('1', `datas`), 1.5, IF(FIND_IN_SET('2', `datas`), 2, NULL)) AS `First`, '1' AS `Second`, '2' AS `Third` FROM `t1`; > +First Second Third > +1.5 1 2 > +2.0 1 2 > +NULL 1 2 > +drop table t1; > diff --git a/mysql-test/t/func_if.test b/mysql-test/t/func_if.test > index 2b89a61..8fdba77 100644 > --- a/mysql-test/t/func_if.test > +++ b/mysql-test/t/func_if.test > @@ -206,6 +206,20 @@ SELECT if(1, NULL, (SELECT min('hello'))); > > --echo End of 5.2 tests > > +--echo # > +--echo # MDEV-8663: IF Statement returns multiple values erroneously > +--echo # (or Assertion `!null_value' failed in Item::send(Protocol*, String*) > +--echo # > +CREATE TABLE `t1` ( > +`datas` VARCHAR(25) NOT NULL > +) ENGINE=InnoDB DEFAULT CHARSET=utf8; > + > +INSERT INTO `t1` VALUES ('1,2'), ('2,3'), ('3,4'); > + > +SELECT IF(FIND_IN_SET('1', `datas`), 1.5, IF(FIND_IN_SET('2', `datas`), 2, NULL)) AS `First`, '1' AS `Second`, '2' AS `Third` FROM `t1`; > + > +drop table t1; > + > --disable_query_log > # Restore timezone to default > set time_zone= @@global.time_zone; > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index 998cb1c..4f07318 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -2692,7 +2692,8 @@ Item_func_if::str_op(String *str) > String *res=arg->val_str(str); > if (res) > res->set_charset(collation.collation); > - null_value=arg->null_value; > + if (null_value=arg->null_value) > + res= NULL; > return res; > } > > @@ -2703,7 +2704,8 @@ Item_func_if::decimal_op(my_decimal *decimal_value) > DBUG_ASSERT(fixed == 1); > Item *arg= args[0]->val_bool() ? args[1] : args[2]; > my_decimal *value= arg->val_decimal(decimal_value); > - null_value= arg->null_value; > + if (null_value= arg->null_value) > + value= NULL; > return value; > } > > diff --git a/sql/item_func.cc b/sql/item_func.cc > index 2f51308..cbb8aa0 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -924,6 +924,7 @@ String *Item_func_hybrid_result_type::val_str(String *str) > ltime.time_type= mysql_type_to_time_type(field_type()); > str->length(my_TIME_to_str(&ltime, const_cast<char*>(str->ptr()), decimals)); > str->set_charset(&my_charset_bin); > + DBUG_ASSERT(!null_value); > return str; > } > return str_op(&str_value); > @@ -932,6 +933,7 @@ String *Item_func_hybrid_result_type::val_str(String *str) > case IMPOSSIBLE_RESULT: > DBUG_ASSERT(0); > } > + DBUG_ASSERT(!null_value || (str == NULL)); > return str; > } > > @@ -1071,7 +1073,10 @@ my_decimal *Item_func_hybrid_result_type::val_decimal(my_decimal *decimal_value) > } > String *res; > if (!(res= str_op(&str_value))) > + { > + null_value= 1; > return NULL; > + } > > str2my_decimal(E_DEC_FATAL_ERROR, (char*) res->ptr(), > res->length(), res->charset(), decimal_value); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

1 0

[Maria-developers] [GSoC] MDEV-4674 Port InnoDB memcached interface to MariaDB - Merging the code
by Piotr Jurkiewicz 02 Sep '15

02 Sep '15

Hi again, there are some issues which I think needs to be resolved before merging the code. 1. Bundle libevent or not? Memcached daemon is built with the help of libevent library. MySQL decided to bundle this library inside its sources, in 'libevent' top level directory. This allows to compile and run the plugin on machines without installed system-wide libevent library. The bundled libevent is used by default during the compilation of plugin. Optionally, it can be specified in CMake variable to use system-wide libevent instead. It needs to be decided whether MariaDB should bundle libevent as well inside its sources. Possible outcomes: a) Leave bundled libevent as is, in the 'libevent' top level directory. b) Move bundled libevent into the 'daemon_memcached' plugin directory, in order to not clutter the top level directory. c) Remove bundled libevent. After that, system libevent headers will become compile-time dependency and system libevent shared library runtime dependency of the plugin. 2. How to transfer InnoDB API callback array pointer to the plugin? InnoDB Memcached interface engine needs to obtain the pointer to InnoDB API callback array in some way. MySQL originally implemented that in 'sql/sql_plugin.cc:plugin_initialize()' in the following way: https://github.com/piotrjurkiewicz/mariadb-server/commit/4eb8fd9c211f396694… As you can see, this is not a clean solution. Possible outcomes: a) Leave the MySQL's solution as is. b) Figure out a better way to transfer the pointer. 3. Where should I post the documentation? 4. How should I prepare the pull request? Should I rebase my branch on the top of current branch before that? I looking forward for your suggestions and decisions. Piotr

2 1

Re: [Maria-developers] MDEV-8389 patch review (part#1)
by Sergey Petrunia 27 Aug '15

27 Aug '15

Hi Diwas, On Thu, Aug 27, 2015 at 09:23:55PM +0530, Diwas Joshi wrote: > hello sergei, I am attaching two patches with this mail. > MDEV-8389_first is the original patch with mysql_create_table() method used > to create tables. I have fixed the crash that was occurring with queries > like select name from f21('aaa') where name like '%foo%'; So, this patch is > giving the results. > MDEV-8389_second contains use of create_tmp_table() so far the table is > being created, but the queries can't see it so it is still giving errors > like table not present. I guess we'll have to fix the Name Resolution for > queries to be able to use the created table. > > Also both patches have fix for MDEV-8650. I've looked at the patch. Unfortunately, it gives little confidence that it will work in all cases. There are a lot of weird things. To name a few: > + tmp_table_param->schema_table = 1; why? ... > + switch (field->sql_type) > + { > + case MYSQL_TYPE_VARCHAR: > + case MYSQL_TYPE_VAR_STRING: > + case MYSQL_TYPE_STRING: > + case MYSQL_TYPE_SET: > + field->pack_flag= 0; > + break; ... This big switch statement is copied from Create_field::init_for_tmp_table. Why is it copied (and not factored out?) If there is a need to copy the code from that function into here, why do you not also copy this part: pack_flag|= (maybe_null ? FIELDFLAG_MAYBE_NULL : 0) | (is_unsigned ? 0 : FIELDFLAG_DECIMAL); > + TABLE *vtable= create_virtual_tmp_table(thd, sp->m_cols_list); ... > + if (! (table = create_tmp_table(thd, tmp_table_param, *types, > + (ORDER*) 0, 1, 1, > + TMP_TABLE_ALL_COLUMNS, HA_POS_ERROR, > + sp->m_table_alias.str))) why create_virtual_tmp_table and then create_tmp_table? 'vtable' is created and then not freed. I look at the comment near create_virtual_tmp_table() definition and see: The table is created in THD mem_root, so are the table's fields. Consequently, if you don't BLOB fields, you don't need to free it. But what if the table does have blob fields? > + table->file->extra(HA_EXTRA_IGNORE_DUP_KEY); What's this? I grep for HA_EXTRA_IGNORE_DUP_KEY and find this explanation: HA_EXTRA_IGNORE_DUP_KEY: HA_EXTRA_NO_IGNORE_DUP_KEY: Informs the handler to we will not stop the transaction if we get an duplicate key errors during insert/upate. Do we expect to get duplicate key errors when inserting into the table? > +bool execute_table_function(THD *thd, sp_head *sp, List<Item> *value_list) > +{ > + /* > + We never write CALL statements into binlog: > + - If the mode is non-prelocked, each statement will be logged > + separately. > + - If the mode is prelocked, the invoking statement will care > + about writing into binlog. > + So just execute the statement. How is CALL statement relevant to executing a table function? I think at this point it is clear that this patch needs more work before it can be pushed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-8389 patch review (part#1)
by Sergey Petrunia 27 Aug '15

27 Aug '15

On Sat, Aug 22, 2015 at 02:10:58AM +0530, Diwas Joshi wrote: > Hello Sergei, sorry for late reply, i was busy finishing my office > work yesterday. Please find some of my doubts below (I would come on > irc to discuss but I am travelling today and will be working overnight > in the train, I hope to be on irc tomorrow). > > Regarding use of storage engine-- If one uses SET > default_tmp_storage_engine, to set the default storage engine for > temporary tables in the current session, it is using that default > storage engine. In my function I have used > create_info.use_default_db_type(thd) which does the work. > (Referencing https://dev.mysql.com/doc/refman/5.6/en/storage-engine-setting.html) > > Regarding the name resolution error. I didn't know it but if there is > already a table present in the database with same name, temporary > table is still being created and used int the function. so that is one > problem solved. For the problem of a function being used multiple > times in a query, I was just curious as how would we solve it if we > switch to create_tmp_table(). As I mentioned before, the solution should be: >> - Patch the name resolution code so that the statements inside a Table Function >> see the return table of that Table Function. "Name resolution" is a process of finding which table should be used when a query refers to table named TBL. The process currently is: 1. Look at user-created temporary tables with name TBL. 2. If none found, look at the regular database tables in the current database with the name TBL. Our problem is that tables created with mysql_create_table() are visible to step#1, and all parts of the query can see them. This includes those parts of the query that we don't want to see the table: other stored procedures, the query that invokes the table function, etc. Temporary tables that are created with create_tmp_table() are not visible to step #1. We will not have the problem of temporary table being visible where it should not be visible. We will have the problem though, that temporary table must be visible where it needs to be visible. This is achieved by modifying step#1 in Name Resolution code to also look at the name of the returned table when we are in a table function. > Still looking into crash because of select name from f21('aaa') where > name like '%foo%' and errors MDEV-8649 and MDEV-8650; > Other errors in the patch have been fixed. > BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MDEV-8643 Valgrind error in mroonga/storage.index_multiple_column_unique_datetime_index_read
by Alexander Barkov 27 Aug '15

27 Aug '15

Hi Kentoku and Kouhei, FYI, I just filed this bug report: https://mariadb.atlassian.net/browse/MDEV-8643 Can you please have a look into this? Thanks!

1 1

[Maria-developers] New Hacker
by Aniket Shukla 27 Aug '15

27 Aug '15

Hey MariaDB team,this is Aniket.I'm a final year undegraduate student and would like to contribute to MariaDB.Since this is my first open source project, I have no idea where to begin.I have good knowledge of python,c++ and have benn using MySQL for my previous projects. Any help would be welcomed.

2 1

[Maria-developers] [GSoC] MDEV-4674 Port InnoDB memcached interface to MariaDB - Final report
by Piotr Jurkiewicz 25 Aug '15

25 Aug '15

Hi all, as the pencils down date has passed, I am posting a report on what I have during the whole project. Repository of my project is available here: https://github.com/piotrjurkiewicz/mariadb-server/commits/10.1-memcached Some stats: 33 commits before the midterm 60 commits after the midterm 93 commits total What have been done in details: =============================== BEFORE MIDTERM: --------------- 1. Code of InnoDB Memcached plugin has been imported from MySQL into the MariaDB tree. 2. There were some differences between MariaDB and MySQL, which blocked the plugin from out-of-the-box compilation and correct operation. They have been resolved. 3. Test suite for the plugin has been imported from MySQL. Compatibility changes were introduced into these test cases which required them. The test suite was modified to assign memcached port number dynamically from mtr worker's pool, what allows to run the same test in parallel for many combinations. The plugin fully passes the test suite, both for 'innodb_plugin' and 'xtradb' combinations. 4. Some enhancements and bug fixes have been introduced into the plugin: - detection of memcached daemon initialization has been fixed -- this eliminated unnecessary wait (15 seconds) on plugin shutdown and memory leak (in cases when plugin failed to initialize properly before) - memcached arithmetic commands (incr/decr) handling has been improved -- now numeric columns values are handled in a consistent and predictable way - incorrect handling of unsigned not null integers columns has been fixed AFTER MIDTERM: -------------- 5. Memcached daemon code has been updated to the latest version from upstream memcached/engine-pu branch. 6. Plugin architecture has been changed to "one table per socket", as originally proposed in MDEV-4674 bug report. This allows usage of well known security solutions (firewalls, unix file permissions) to restrict access to different tables for different users. Now user can specify which tables should be accessible on which sockets. Memcached daemon will listen on all these sockets and execute commands on appropriate table, depending on which socket client connected. Supported protocols are TCP, UDP and UNIX stream sockets. Sockets are configured in 'daemon_memcached/containers' table: INSERT INTO containers VALUES ("tcp://*:11211", "db0", "table0", "key_col", "val_col", "c3", "c4", "c5", "PRIMARY", "|"); INSERT INTO containers VALUES ("udp://localhost:11222", "db0", "table1", "key_col", "val_col", "c3", "c4", "c5", "PRIMARY", "|"); INSERT INTO containers VALUES ("unix:/tmp/mc.sock", "db1", "table0", "key_col", "val_col", "c3", "c4", "c5", "PRIMARY", "|"); 7. Plugin has been decoupled from InnoDB. Now plugin is split into two parts: - generic 'daemon_memcached.so' - InnoDB specific 'daemon_memcached_engine_ib.so' The first library is a MariaDB plugin, which should be installed with command: install plugin daemon_memcached soname "daemon_memcached.so"; This library is generic. It provides Memcached protocol handling and it loads the second library, which is API-specific engine for accessing the database. Which engine library should be loaded can be specified in 'daemon_memcached_engine_lib_name' MariaDB server option. Now, the only existing engine library is daemon_memcached_engine_ib.so, which utilizes InnoDB API to access the data. It can operate on InnoDB and XtraDB tables. 8. Some enhancements and bug fixes have been introduced into the plugin: - CMake options and #cmakedefines associated with the plugin have been cleaned up - embedded default Memcached engine has been removed - duplicated code in Memcached source has been removed - fixed MySQL-originating bug with flush_all of non default table -- command flush_all always flushed the default table, not the current one - removed support for in-connection table switching -- after introduction of one table/port architecture it was redundant - plugin shutdown time has been improved -- from constant time of 2 seconds to average time of 0.5 second - logging of plugin has been refactored -- now plugin send log messages to MariaDB log instead of stderr - fixed Memcached-originating bug with get_thread_stats() -- bug resulted in out of bound array accesses and writes, what leaded to SEGFAULT - fixed MySQL-originating crash during 'flush_all' memcached command bug -- crash happened when table was heavily loaded with DML operations and because of that ib_cb_cursor_set_lock() failed to obtain a lock on the table, returning DB_LOCK_WAIT_TIMEOUT error - fixed MySQL-originating multiple memcached_sync_count for the same transaction bug -- the memcached_sync_count was incremented multiple times for the same transaction and after the end of transaction, it was not reset to zero, what blocked TRUNCATE command on any InnoDB table, on which Memcached had operated during program lifetime, even after uninstalling the plugin - support for numeric values has been finished -- now BIGINT overflows in the same way as smaller types, test to check over/under-flowing of all integer types has been added For details of all changes please see commit messages associated with them: https://github.com/piotrjurkiewicz/mariadb-server/commits/10.1-memcached ----------------------------------------------------------------------- Moreover, I planned to implement a new interface inside the Memcached plugin, which will use Handler API for database access. This would make possible to use the plugin to access databases based on storage engines other than InnoDB/XtraDB. However, during testing I discovered many serious bugs, either in Memcached or InnoDB API engine, which originated from upstream Memcached or MySQL and resulted in crashes or deadlocks. I decided that first I should fix them before implementing new functionalities. I lost a lot of time on investigating these bugs and I was unable to finish the implementation of Handler API engine. Nevertheless, I think the whole plugin is ready to be merged into MariaDB. All user-facing changes (like one port/table architecture) have been completed. Eventual later introduction of Handler API engine will not be associated with any user-facing compatibility changes. It will just lift the requirement that all databases accessed via the Memcached plugin must be InnoDB/XtraDB. Now I am going to write a separate email to coordinate the merge of my code into MariaDB repository. Piotr

1 0

Re: [Maria-developers] MDEV-7502 Provisioning of slave - review
by Kristian Nielsen 24 Aug '15

24 Aug '15

Martin Kaluznik <martin.kaluznik(a)gmail.com> writes: > good to do now. And I would also like to ask you for review of > existing code. > > Github link: https://github.com/f4rnham/server/tree/MDEV-7502 > Diff: https://github.com/MariaDB/server/compare/10.1...f4rnham:MDEV-7502.diff Please find my review inline in patch, below. General stuff: > I have created new log event as way to signalize slave successful end > of provisioning, it is last event IO thread writes to relay log before > termination, and last event SQL thread reads from relay log. Test > cases are using Slave_SQL_Running value from SHOW SLAVE STATUS command > to wait until provisioning is completed. It works there, but I am not > sure about its convinience in real world usage. Hm, I am not sure either. Another option would be for LOAD DATA FROM MASTER to block until provisioning is done (failed or successful). Other than review, please check Buildbot results carefully for any failures. Pay particular attention to the valgrind build (for memory leaks). Also pay special attention to random failures in your new test cases that happen only occasionally. Such sporadic / random failures often turn up in replication tests. You need in test cases that use debug_dbug: --source include/have_debug.inc What is the largest data set you tested with? Please do a (manual) test with a large data set. For example, you could run sysbench with a 10GB or 50GB database, let it run for say 10 minutes, and 5 minutes in you start a LOAD DATA FROM MASTER on a slave off the server running the sysbench. Please check comments below on how (and if) the whole approach with provisioning in parallel with active transactions is correct and makes the slave always identical with the master at the end. Is there any issue with foreign keys? An insert to one table could fail if a foreign key constraint against another table is not yet satisfied due to rows not yet provisioned. A CREATE TABLE referencing another table might fail. Maybe look at if mysqldump is doing something to avoid these kinds of issues. A minimal solution is to investigate possible problems, and document them as limitations. Eg. it is reasonable to document that users must disable foreign keys during provisioning, if necessary. What happens if the server that does LOAD DATA FROM MASTER provisioning has --log-bin and --log-slave-updates? What happens if a third-level slave is connected to the server that does LOAD DATA FROM MASTER? I think this needs to be tested. Comments on the patch: > +BOOLEAN _db_keyword_locked_(CODE_STATE *cs, const char *keyword, int strict) Can you explain the race that results without this locking? Is it because _db_keyword_() is scanning the list without any locking, so if the list changes during the scan we can crash? > +#define DBUG_EXECUTE_LOCKED(keyword,a1) \ > + do {if (_db_keyword_locked_(0, (keyword), 0)) { a1 }} while(0) You need empty definition for non-debug build (but I guess you already saw in buildbot run). > +--let $slave_param= Slave_SQL_Running > +--let $slave_param_value= No > +--let $slave_error_param= Last_SQL_Errno > +--source include/wait_for_slave_param.inc Is there a reason not just to use include/wait_for_slave_sql_to_stop.inc here? > +# Ensure that master and slave log names and positions are equal > +--connection master > +--let $binlog_file_master= query_get_value(SHOW MASTER STATUS, File, 1) > +--let $binlog_pos_master= query_get_value(SHOW MASTER STATUS, Position, 1) > +--connection slave > +--let $binlog_file_slave= query_get_value(SHOW SLAVE STATUS, Master_Log_File, 1) > +--let $binlog_pos_slave= query_get_value(SHOW SLAVE STATUS, Read_Master_Log_Pos, 1) Is there a reason you do not use GTIDs here instead (or in addition)? > +# echo Binlog file name master - $binlog_file_master; > +# echo Binlog file name slave - $binlog_file_slave; > +# echo Binlog position master - $binlog_pos_master; > +# echo Binlog position slave - $binlog_pos_slave; Please remove this left-over debugging code. > +++ b/mysql-test/suite/rpl/t/rpl_provisioning_base.inc > +eval CREATE TABLE test.test_1 (a int, PRIMARY KEY (a)) ENGINE=$engine_type; > +eval CREATE TABLE test.test_2 (a int, b int, c int, PRIMARY KEY (a)) ENGINE=$engine_type; Try adding also some foreign key contraints to the table for the test. > +# Start provisioning > +--connection slave > +LOAD DATA FROM MASTER; At this point, master has full binlogs. So just START SLAVE would do the same thing, just by re-playing the binlog. I think the test should use FLUSH LOGS on the master a couple of times during the initial setup of data, and then PURGE BINARY LOGS to delete some (but not all) of that binlog data, to test that the slave can still provision without complete binary logs. > +SET GLOBAL debug_dbug="+d,provisioning_wait"; > +SET GLOBAL debug_dbug="-d,provisioning_wait"; Earlier, when I did this, I ended up with tons of debug output in the error log after setting the "-d,..." which is annoying. Setting an empty "+d" option was enabling debug trace. Maybe this is now fixed, if so then all the better. But if you get this problem (with stray debug trace in error log), you can use the following pattern instead: SET @old_dbug= @@GLOBAL.debug_dbug; ... SET GLOBAL debug_dbug= @old_dbug; > +# > +# Attempt to start provisioning while not in GTID mode > +# Is this an old comment? It seems none of the tests really use any GTID. Does provisioning work in GTID mode? It should be tested. > +# > +# Attempt to start provisioning while SQL thread is running > +# What if IO thread is running? Should probably be tested also. > +# Creation of database `test` depends purely on timing and it has nothing to do with this test case > +--disable_warnings > +DROP DATABASE IF EXISTS test; What is the purpose of this? It sounds like there is a potential race in the test case? Maybe there is a better way to handle it. > +START SLAVE; > + > +# Cleanup after test case > +--connection slave > +STOP SLAVE; > +RESET SLAVE; > +# Creation of database `test` depends purely on timing and it has nothing to do with this test case It is better to use the include/start_slave.inc and include/stop_slave.inc. They prevent some races, which might solve some of the problems that your comment about timing seems to hint at. Also, it is usually best to synchronise the slave with the master before stopping, to avoid it being stopped at a random point. > +# > +# Bug: Deadlock occurs when IO thread fails with error before dump request > +# What do you mean? That the current patch has a bug triggered by this test case? Or that there was a bug, fixed now, and this is a test for it? Please explain better in the comment. > diff --git a/mysql-test/suite/rpl/t/rpl_provisioning_stress.test b/mysql-test/suite/rpl/t/rpl_provisioning_stress.test > new file mode 100644 > index 0000000..8141015 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_provisioning_stress.test > @@ -0,0 +1,48 @@ > +--skip Test requires manual setup > + > +# Stress test for MDEV-7502 Automatic provisioning of slave > +# Using Random Query Generator v2.2.0 release, downloadable from https://launchpad.net/randgen/+download I think it is great that you made this kind of test. But maybe it would be better to put it somewhere else, to not have a test in the normal test suite that is always unconditionally skipped. Eg. it could be put in /tests/, and one would copy it into the test suite, instead of removing the --skip. > +void Provisioning_done_log_event::print(FILE* file, > + print_header(&cache, print_event_info, FALSE); > + my_b_write_string(&cache, "\tProvisioning done\n"); Hm, this can occur in the relay log, but not in the binlog, right? I think it is ok not to have a test of this in mysql-test-run, we generally can only test the "short-form" in there to avoid dependency on time and such. Just be sure to check manually that the output is correct. > @@ -9718,23 +9760,29 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > + // In provisioning mode, silently ignore table not found error > + if (!rgi->thd->slave_thread || !rli->mi->provisioning_mode || > + !provisioning_error_code(actual_error)) Better use thd than rgi->thd here (just for consistency). > + // Ignore certain errors during provisioning What thoughts did you have on whether just ignoring these kinds of errors would always result in the correct result of provisioning? I suppose since we are copying data fresh from the master, it should not be possible that we miss a "real" error in this way. Will things work correctly, if we have multiple rows in a single Rows_log_event, and some of them succeed while others fail? In both MyISAM and InnoDB? Can an argument be made to prove that, if we ignore an insert or update here, we are sure to later write the correct data as part of provisioning? For both InnoDB and MyISAM? Similarly, can it be proven that it is impossible to ignore a delete and still later wrongly provision the deleted row? I suppose it would be something with binlog ordering and isolation modes for the InnoDB case - and some table locking semantics wrt. MyISAM? > + > + /* > + Semaphore used by LOAD DATA FROM MASTER command for test cases, its role is > + to ensure, that slave's IO thread is connected to master before test case > + can continue. > + volatile bool dump_requested_semaphore; Shouldn't this be under #ifndef DBUG_OFF ? > diff --git a/sql/rpl_provisioning.cc b/sql/rpl_provisioning.cc > +void quote_name(char const *name, char *buff) Try to use append_identifier() instead. Or maybe my_snprintf("%`s"). I notice that append_identifier() handles a configuration where "" is used to quote identifiers instead of ``, you should probably test if this causes a bug in your code, btw. > +provisioning_send_info::provisioning_send_info(THD *thd_arg) > + : thd(thd_arg) > + , connection(new Ed_connection(thd)) > + , row_batch_end(NULL) > + , row_buffer(my_malloc(ROW_BUFFER_DEFAULT_SIZE, MYF(0))) > + , row_buffer_size(ROW_BUFFER_DEFAULT_SIZE) > + , phase(PROV_PHASE_INIT) > + , error(0) > + , error_text(NULL) > + , event_conversion_cache(NULL) > +{ > +} So this constructor will probably cause the server to crash in case of out-of-memory, right? Usually, code tries to fail gracefully with ER_OUT_OF_RESOURCES / ER_OUTOFMEMORY if malloc() fails (though its not well tested, probably). > +void provisioning_send_info::close_tables() > +{ > + trans_commit_stmt(thd); I see no call to commit the entire transaction (as opposed to just the statement here). This may be intentional (autocommit?), in any case please add a comment here explaining how the transaction is committed wrt. statement-transaction and full transaction. Maybe add a test where the server is running with auto-commit globally disabled? Though I am not sure if the short mysql-test-run.pl tests would notice it if everything was done as one long transaction (though large production sites undobtedly would ;-). > +bool provisioning_send_info::send_query_log_event(DYNAMIC_STRING const *query, Is there a particular reason that you chose DYNAMIC_STRING over the String class? I think String is more widely used for building queries and such? A common use pattern with String is to allocate a reasonably-sized buffer on the stack and initialise a String object from it. Then in the common case of smallish result, no my_malloc() calls are needed. > + if (cs_info) > + { > + // thd->variables.character_set_client->number > + int2store(evt.charset, cs_info->cs_client); > + // thd->variables.collation_connection->number > + int2store(evt.charset + 2, cs_info->cl_connection); > + // cs_info->cl_server > + int2store(evt.charset + 4, thd->variables.collation_server->number); > + // thd->variables.collation_database->number > + evt.charset_database_number= cs_info->cl_db; Can you explain why you need to consider the charsets/collations of the connection and client? Could you not just always use the system character set for generating queries? > + if (my_strcasecmp(system_charset_info, "VIEW", type->str) == 0) > + { > + char name_buff[NAME_LEN * 2 + 3]; > + DYNAMIC_STRING *str= (DYNAMIC_STRING*)my_malloc(sizeof(DYNAMIC_STRING), > + MYF(0)); > + > + quote_name(database->str, name_buff); > + init_dynamic_string(str, name_buff, > + // Both lengths + dot and quotes, will be reallocated > + // only in special cases > + database->length + name->length + 5, > + 5); > + dynstr_append(str, "."); > + quote_name(name->str, name_buff); > + dynstr_append(str, name_buff); > + > + views.push_back(str); > + } > + else if (my_strcasecmp(system_charset_info, "BASE TABLE", type->str) == 0) > + tables.push_back(my_strdup(name->str, MYF(0))); Why do you need to prepend the database name in VIEW case but not in TABLE case? Please add a comment explaining that. > + else > + DBUG_ASSERT(false && > + "Unexpected type of table returned by 'SHOW FULL TABLES'"); Probably you should add an error return here, that will cause provisioning to fail in the non-debug build, rather than silently press on. > +bool provisioning_send_info::event_to_packet(Log_event &evt, String &packet) > +{ > + // Reset cache for writing > + reinit_io_cache(event_conversion_cache, WRITE_CACHE, 0, false, true); > + > + if (evt.write(event_conversion_cache)) > + { > + error_text= "Failed to write event to conversion cache"; > + return true; > + } > + > + if (reinit_io_cache(event_conversion_cache, READ_CACHE, 0, false, false)) > + { > + error_text= "Failed to switch event conversion cache mode"; > + return true; > + } > + > + packet.set("\0", 1, &my_charset_bin); > + if (packet.append(event_conversion_cache, > + event_conversion_cache->write_pos - event_conversion_cache->buffer)) > + { > + error_text= "Failed to write event data to packet"; > + return true; > + } Hm. So first the event is constructed in-memory. Then it is written to an IO_CACHE. Then it is read back in again into a byte buffer. And then it is written to the slave's client socket. Hm, I see, the client socket we have only as a NET, not an IO_CACHE. But the Log_event subclasses need an IO_CACHE to construct a serialised event. Hm, that's clunky, but I don't see an easy way to avoid it. But please add a comment explaining why this is necessary. Also, doesn't the IO_CACHE spill to disk if the data written becomes large? So wouldn't a very large event (eg. a row containing a large BLOB) fail here? Because the IO_CACHE only contains part of the data in-memory? I am not 100% sure about how this works, please investigate and test with a large row (if you did not already). > + // Set current server as source of event, when slave registers on master, > + // it overwrites thd->variables.server_id with its own server id - and it is > + // then used when writing event > + int4store(&packet[SERVER_ID_OFFSET + 1], global_system_variables.server_id); > + return false; I think this will fail if event checksums are enabled? Or rather, the patch does not seem to handle binlog checksums at all. Try adding a test that enables binlog checksums on master and slave, and check that everything works. And also - why do you need to change this in the first place? Doesn't the server_id get set correctly when the event is initially created? > +bool provisioning_send_info::send_event(Log_event &evt) > +{ > + String packet; > + > + // Slave will accept this event even if it modifies data structures > + evt.flags|= LOG_EVENT_PROVISIONING_F; > + > + if (event_to_packet(evt, packet)) > + return true; > + > + if (my_net_write(&thd->net, (uchar*)packet.ptr(), packet.length())) > + { > + error_text= "Failed to send event packet to slave"; > + return true; > + } Yeah... It would be so much more elegant if we could wrap the thd->net in an IO_CACHE, and write directly to the client. Did you investigate if this would be a possibility, without too much extra work? > +int8 provisioning_send_info::send_table_data() > + if ((error= hdl->prepare_range_scan(row_batch_end, NULL)) != 0) > + if (table->s->primary_key == MAX_KEY) > + { > + error_text= "Table does not contain primary key"; > + close_tables(); > + return 1; > + } It seems more natural to me to check for missing primary key _before_ prepare_range_scan(). (Otherwise one wonders how it is possible to prepare a range scan on a table without _any_ key? I suppose it works to do a table scan instead, but better do the check first to avoid having to wonder). > + while (packed_rows < PROV_ROW_BATCH_SIZE && > + (error= hdl->read_range_next()) == 0); I think there should be two stop criterias. One is max number of rows reached, another is size of rows. At least, you need to protect against exceeding --max-packet-size. Otherwise the slave will be unable to receive the too large packet. Please add a test case that checks large rows and that it correctly splits row events to not exceed max-packet-size (of course at least one row must fit in one Rows_log_event). Check the code for normal binlogging to see how it splits up Row_log_events to avoid creating too large an event. > + table->default_column_bitmaps(); > + close_tables(); So the critical issue here is: how can we be sure that we will not miss rows during provisioning, by racing between provisioning events and normal binlog events? This needs to be clearly thought out, and then explained in a comment somewhere in the code. For example, you write the events to the slave client socket before closing the table and releasing locks. This seems critical (and correct) - because this ensures that if there is a DELETE made just after the provisioning read of the table, that DELETE will be sent only _after_ sending the provisioning rows (so we do not lose the DELETE). This should be explained in a comment. But what about isolation level (for InnoDB)? Don't we need to run with a sufficiently high isolation level to be correct? Intuitively, I thought so, but maybe not, at least you have nothing in the patch about isolation level that I can see. What about this scenario? 1. Provisioning reads rows 20-29, sends them to slave. 2. An UPDATE runs, it updates row 32 to be now row 25, it is written to the binary log and sent to the slave. 3. Provisioning reads rows 30-39, sends them to the slave. On the slave, the UPDATE from 2 should fail - because row 32 does not exist at this point. But then, it seems to me that the row 25 will never appear on the slave, right? So provisioning will be incorrect. Or am I missing something? I wonder if the solution to this particular case is that during provisioning, an UPDATE (at least one that changes primary key value) must be logged as a DELETE (of the old row) + an INSERT (of the new row). Alternatively, maybe the slave can during provisioning execute the UPDATE regardless of the missing before image, still writing the after image (row 25) to the table. More generally, we really need an explanation of how the provisioning ensures that the end result on the slave is identical to the master - otherwise there may be other problems like this hidden. Also, it would be really good if you could make a test case for the above potential issue. You can get some ideas from how I use DBUG_EXECUTE_IF together with DEBUG_SYNC in the rpl_parallel_*.test test cases. But I know it can be a lot of work to get such tricky race-proviking test cases to work reliably. > diff --git a/sql/rpl_provisioning.h b/sql/rpl_provisioning.h > +// FIXME - Farnham > +// Header > + > +#pragma once We do not use this #pragma once in mariadb (maybe that is what you intended to fix with the FIXME, so mentioning just in case). I assume you will take care of other such FIXME's in the patch yourself before pushing. > +/* > + How much rows from table will be sent in one provisioning tick > + FIXME - Farnham > + Dynamic value > +*/ > +#define PROV_ROW_BATCH_SIZE 5 Right. Probably just a matter of adding something to sql/sys_vars.cc > - if (rpl_load_gtid_slave_state(thd)) > + if (!mi->provisioning_mode && rpl_load_gtid_slave_state(thd)) Actually, what should happen with the GTID position for LOAD DATA FROM MASTER? Just clearing it probably is not correct, in case user wants to LOAD DATA FROM MASTER from multiple masters. Maybe the user should clear it herself, if necessary, and then it will be updated by the normal replication stream from master. But why do you skip loading it here for provisioning mode? That does not seem right. First, it will normally be loaded already anyway. And second, if not loaded, then just having the wrong value in-memory (as compared to in the mysql.gtid_slave_pos table) does not seem correct? > + DBUG_EXECUTE_IF("provisioning_test_running", > + if (!slave_errno) > + while (!mi->dump_requested_semaphore) > + /* pass */;); Maybe put a my_sleep(20000) in with the /* pass */. > + case PROVISIONING_DONE_EVENT: > + mi->abort_slave= true; > + /* > + Fall through to default case ... and through debug code, nothing wrong > + should happen > + */ That seems a bit fragile. Better do like in other places: goto default_action; > + // In provisioning mode, master starts sending data from current position > + if (flags & BINLOG_DUMP_PROVISIONING_MODE) So provisioning assumes non-GTID mode? Maybe that is ok... But it should still be tested what happens if provisioning is attempted in GTID mode. Perhaps either GTID or non-GTID mode can work the same, since provisioning does not need to start at any particular position - and at the end of provisioning, both GTID and non-GTID position should be correct. > +bool > +Ed_connection::execute_direct(char *sql_text, size_t length) > +{ > + LEX_STRING tmp_query; > + tmp_query.str= sql_text; > + tmp_query.length= length ? length : strlen(sql_text); > + > + return execute_direct(tmp_query); > +} You are not actually ever calling this without passing length. So please make this an inline method in the .h file, with mandatory length argument, that does not check for zero length or call strlen(). - Kristian.

1 0

[Maria-developers] Fixing Some Trivial Bugs
by Alangi Derick 21 Aug '15

21 Aug '15

Hi, I just found some trivial bugs on jira that i will be fixing for the time being. Once i am done with each one, i will submit a patch or push to my forked repo so it can be merged. My question is, after i fix the bugs, will they accepted, if yes i should continue working on them thanks. I need a respond from the development team. Regards Alangi Derick Ndimnain

3 4

[Maria-developers] [Patch] Fix to mysql.server init script
by Alangi Derick 21 Aug '15

21 Aug '15

This is a fix to the bug in the mysql.server init script which uses a wrong server name (MySQL) instead of MariaDB. It is not fixed and this is the patch below. Regards Alangi Derick Ndimnain

1 0

[Maria-developers] Please review: MDEV-8433 Make field<'broken-string' use indexes
by Alexander Barkov 20 Aug '15

20 Aug '15

Hi Sergei, Please review my patch for MDEV-8433. I had doubts about one detail: how to make Field::store() that we're currently creating a value for range search rather than regular doing a INSERT/UPDATE. I could think of two options: 1. Using sql_mode, similar to how temporal types pass MODE_INVALID_DATES. 2. Using a new parameter to this method: int Field::store(const char *str, uint length, CHARSET_INFO *cs); The former needed less changes, so I chose to use sql_mode. But probably N2 would be better. Please suggest. Thanks.

1 0

Re: [Maria-developers] [JIRA] (MDEV-7063) Split String into logical components
by Alexander Barkov 20 Aug '15

20 Aug '15

Hi Rasmus, I noticed you put this task on Sprint 10.1.7-2. Did I miss some discussion about this? From my understanding, Sergei and Monty were not enthusiastic about this task (but I still think it should be helpful). Thanks. On 08/19/2015 06:26 PM, Rasmus Johansson (JIRA) wrote: > > [ https://mariadb.atlassian.net/browse/MDEV-7063?page=com.atlassian.jira.plug… ] > > Rasmus Johansson updated MDEV-7063: > ----------------------------------- > Sprint: 10.1.7-2 > >> Split String into logical components >> ------------------------------------ >> >> Key: MDEV-7063 >> URL: https://mariadb.atlassian.net/browse/MDEV-7063 >> Project: MariaDB Server >> Issue Type: Task >> Reporter: Alexander Barkov >> Assignee: Michael Widenius >> Fix For: 10.1 >> >> >> The string class currently consists of the following components: >> - memory allocation routines (using mysys) >> - binary string routines >> - character set aware routines. >> Storage engines (e.g. ConnectSE) often need String-alike classes, but cannot use the current String implementation because of their own memory management system. So they have to reimplement the same set of functionality, but with their own memory managers. This is bug prone and makes duplicate code. >> Also, in some cases it is useful to have character-set unaware strings (just array of bytes), and static strings without allocation. >> This task is to split String into logical pieces: >> 1. class Static_binary_string >> 2. class Charset >> 3. class Static_character_string (#1 + #2) >> 4. class Memory_mysys >> 5. template<typename MEM> class String_with_alloc >> 6. class String (#3 + #5) >> ConnectSE will use #5 (templace class String_with_alloc), in combination with their own memory allocator. > > > > -- > This message was sent by Atlassian JIRA > (v7.0.0-OD-01-193#70101) >

1 0

Re: [Maria-developers] MDEV-8389 patch review
by Sergey Petrunia 19 Aug '15

19 Aug '15

Also, I'm getting a crash when trying to debug a slightly complex example: delimiter | CREATE FUNCTION f10(param1 VARCHAR(11)) RETURNS TABLE return_table(name VARCHAR(11)) deterministic BEGIN insert into return_table values('foo'); END| DELIMITER ; select name from f21('aaa') where name like '%foo%'; Causes a crash: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7efa700 (LWP 5142)] 0x0000555555cb6000 in Item_field::fix_fields (this=0x7fff80006db8, thd=0x555558161ca0, reference=0x7fff80007070) at /home/psergey/dev-git/10.1-gsoc-repo/sql/item.cc:4928 (gdb) wher #0 0x0000555555cb6000 in Item_field::fix_fields (this=0x7fff80006db8, thd=0x555558161ca0, reference=0x7fff80007070) at /home/psergey/dev-git/10.1-gsoc-repo/sql/item.cc:4928 #1 0x0000555555cf86e8 in Item_func::fix_fields (this=0x7fff80006fe0, thd=0x555558161ca0, ref=0x7fff8000b060) at /home/psergey/dev-git/10.1-gsoc-repo/sql/item_func.cc:208 #2 0x0000555555cd9747 in Item_func_like::fix_fields (this=0x7fff80006fe0, thd=0x555558161ca0, ref=0x7fff8000b060) at /home/psergey/dev-git/10.1-gsoc-repo/sql/item_cmpfunc.cc:4856 #3 0x00005555559ff1b0 in setup_conds (thd=0x555558161ca0, tables=0x7fff80006780, leaves=..., conds=0x7fff8000b060) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_base.cc:8657 #4 0x0000555555aeb4f7 in setup_without_group (thd=0x555558161ca0, ref_pointer_array=0x7fff8000b198, tables=0x7fff80006780, leaves=..., fields=..., all_fields=..., conds=0x7fff8000b060, order=0x0, group=0x0, hidden_group_fields=0x7fff8000af40, reserved=0x555558165ffc) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_select.cc:631 #5 0x0000555555aa6d99 in JOIN::prepare (this=0x7fff8000ac48, rref_pointer_array=0x555558165fd8, tables_init=0x7fff80006780, wild_num=0, conds_init=0x7fff80006fe0, og_num=0, order_init=0x0, skip_order_by=false, group_init=0x0, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x555558165d60, unit_arg=0x555558165660) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_select.cc:786 #6 0x0000555555aafcab in mysql_select (thd=0x555558161ca0, rref_pointer_array=0x555558165fd8, tables=0x7fff80006780, wild_num=0, fields=..., conds=0x7fff80006fe0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x7fff8000ac28, unit=0x555558165660, select_lex=0x555558165d60) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_select.cc:3301 #7 0x0000555555aa5fd7 in handle_select (thd=0x555558161ca0, lex=0x555558165598, result=0x7fff8000ac28, setup_tables_done_option=0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_select.cc:371 #8 0x0000555555a65fbf in execute_sqlcom_select (thd=0x555558161ca0, all_tables=0x7fff80006780) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:5799 #9 0x0000555555a5c25f in mysql_execute_command (thd=0x555558161ca0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:2939 #10 0x0000555555a69554 in mysql_parse (thd=0x555558161ca0, rawbuf=0x7fff800064a8 "select name from f21('aaa') where name like '%foo%'", length=51, parser_state=0x7ffff7ef90f0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:7175 #11 0x0000555555a584a7 in dispatch_command (command=COM_QUERY, thd=0x555558161ca0, packet=0x555558168ef1 "select name from f21('aaa') where name like '%foo%'", packet_length=51) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:1470 #12 0x0000555555a571c1 in do_command (thd=0x555558161ca0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:1093 #13 0x0000555555b95b3d in do_handle_one_connection (thd_arg=0x555558161ca0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_connect.cc:1347 #14 0x0000555555b95882 in handle_one_connection (arg=0x555558161ca0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_connect.cc:1258 #15 0x00007ffff691de9a in start_thread (arg=0x7ffff7efa700) at pthread_create.c:308 #16 0x00007ffff604e3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #17 0x0000000000000000 in ?? () (gdb) I am nut sure about the cause of this. It needs to be investigated. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-8389 patch review
by Sergey Petrunia 19 Aug '15

19 Aug '15

On Wed, Aug 12, 2015 at 01:58:40AM +0530, Diwas Joshi wrote: > hello sergei, > After I mailed you the last patch, I kept working further. I guess the code > has come to a point where it is giving some results. So, I did the > following things: > Earlier we were storing all the parameters in a select query inside > lex->value_list, but since there can be multiple functions in a query, I > declared a list for parameters inside table_list so we can store params > corresponding to each table function in it. Good. > Then instead of function name i created a table with table alias defined in > the function and change the table name to table alias inside select query > as well. This makes it possible for the queries inside the function > definition to access the temporary table we created. There is a problem > with this approach that there might already be a table present in the > database with that name, I am still thinking of a solution for it but we > can discuss a solution for it later. Yes, you're right. There is a problem here. It is not limited to the already existing tables with the same name. Let's work with this example: delimiter | CREATE FUNCTION f10(param1 VARCHAR(11)) RETURNS TABLE return_table(name VARCHAR(11)) deterministic BEGIN insert into return_table values('foo'); END| DELIMITER ; MariaDB [test]> select name from f10('aaa'); +------+ | name | +------+ | foo | +------+ 1 row in set (0.29 sec) Good. However, attempts to invoke the same function multiple times fail: MariaDB [test]> select name from f10('aaa') as TBL1 , f10('bbb') as TBL2; ERROR 1050 (42S01): Table 'return_table' already exists Your code uses this approach: 1. Create a temporary table in a manner similar to how user-specified CREATE TEMPORARY TABLE works. 2. Rely on regular MySQL's name resolution process to "see" the temporary table. The properties of this solution are: - If a temporary table has the same name as a regular db table, temporary table will "shadow" the regular table (this is ok). - user-created temporary tables are per-thread. Other threads do not see the temp. table (good). However, everything that executes within the thread will see the temp. table. For example, if a query calls two Table Functions, they will both see each others return tables. This is a problem. I think, a better approach would be: - switch to something like create_tmp_table() for creation of temporary tables. Tables that are created this way are normally not visible to the name resolution code. - Patch the name resolution code so that the statements inside a Table Function see the return table of that Table Function. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-8389 patch review
by Sergey Petrunia 19 Aug '15

19 Aug '15

Hi Diwas, While playing around, I've found and filed: https://mariadb.atlassian.net/browse/MDEV-8649 https://mariadb.atlassian.net/browse/MDEV-8650 I've filed these separately because I think they are not related to MDEV-8389. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-8389 patch review (part#1)
by Sergey Petrunia 19 Aug '15

19 Aug '15

Hi Diwas, Please find part #1 of the review below. I'll provide more soon. > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 2998972..a9dc162 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2466,6 +2466,8 @@ static bool do_execute_sp(THD *thd, sp_head *sp) > lex->first_lists_tables_same(); > /* should be assigned after making first tables same */ > all_tables= lex->query_tables; > + > + check_and_process_table_functions(thd, all_tables); > /* set context for commands which do not use setup_tables */ > select_lex-> > context.resolve_in_table_list_only(select_lex-> > @@ -5581,6 +5583,7 @@ static bool do_execute_sp(THD *thd, sp_head *sp) > } > THD_STAGE_INFO(thd, stage_query_end); > thd->update_stats(); > + //clear_table_functions(thd, all_tables); Please remove the comment. > > goto finish; > > @@ -5697,7 +5700,7 @@ static bool do_execute_sp(THD *thd, sp_head *sp) > thd->mdl_context.release_transactional_locks(); > } > #endif /* WITH_WSREP */ > - > + clear_table_functions(thd, all_tables); > DBUG_RETURN(res || thd->is_error()); > } > > @@ -5709,7 +5712,7 @@ static bool execute_sqlcom_select(THD *thd, TABLE_LIST *all_tables) > bool res; > /* assign global limit variable if limit is not given */ > { > - SELECT_LEX *param= lex->unit.global_parameters(); > + SELECT_LEX *param= lex->unit.global_parameters(); No need for this dummy change. > if (!param->explicit_limit) > param->select_limit= > new (thd->mem_root) Item_int((ulonglong) thd->variables.select_limit); > @@ -9222,3 +9225,182 @@ CHARSET_INFO *find_bin_collation(CHARSET_INFO *cs) > } > return cs; > } > + > +/* > + @brief > + Check for table functions and create corresponding tables to be used in query. > + > + @param thd Thread handle > + @param table_list A list of tables being processed in the query. > + > + @return > + False on success > + @retval > + TRUE on error > +*/ > +bool check_and_process_table_functions(THD *thd, TABLE_LIST *table_list) > +{ > + TABLE_LIST *tables = table_list; > + int err= 0; > + while(tables) > + { > + if (!tables->is_table_function || tables->table_name == NULL) > + { > + tables = tables->next_local; > + continue; > + } > + sp_head *sp; > + sp_name *name; > + LEX_STRING lex_db; > + LEX_STRING lex_name; > + lex_db.length= strlen(tables->db); > + lex_name.length= strlen(tables->table_name); > + lex_db.str= thd->strmake(tables->db, lex_db.length); > + lex_name.str= thd->strmake(tables->table_name, lex_name.length); > + name= new sp_name(lex_db, lex_name, true); > + name->init_qname(thd); > + int err=0; Please switch err from int to bool. > + > + if (!err && (sp = sp_find_routine(thd,TYPE_ENUM_FUNCTION, name, Why assign 'err' and then immediately check it? The code looks redundant. > + &thd->sp_proc_cache, FALSE))) //check if the table is a table-valued function. Please move the comment a few lines down, where the check is actually done. > + { > + if (sp->m_type == TYPE_ENUM_FUNCTION) > + my_error(ER_SP_TABLE_VALUE_EXPECTED, MYF(0)); > + LEX *old_lex= thd->lex, newlex; > + thd->lex= &newlex; > + lex_start(thd); > + ulonglong saved_mode= thd->variables.sql_mode; > + Table_specification_st create_info; > + Alter_info alter_info; > + create_info.init(); > + alter_info.reset(); > + create_info.default_table_charset= NULL; > + TABLE_LIST *create_table; > + create_info.options|= HA_LEX_CREATE_TMP_TABLE; > + LEX_STRING tablename,dbname; > + if (strlen(sp->m_table_alias.str) > 0) { > + tablename.str= sp->m_table_alias.str; > + tablename.length= strlen(sp->m_table_alias.str); > + } sp->m_table_alias is the alias of the return table of the table function. When can it have zero length? Please add a comment about this here, and also add a testcase in the .test file > + else { > + tablename.str= tables->table_name; > + tablename.length= strlen(tables->table_name); > + } > + dbname.str= tables->db; > + dbname.length= strlen(tables->db); > + Table_ident *new_ident= new Table_ident(thd, dbname, tablename, 0); > + if (!thd->lex->select_lex.add_table_to_list(thd, new_ident, NULL, > + TL_OPTION_UPDATING, > + TL_WRITE, MDL_EXCLUSIVE)) > + err= 1; > + create_info.use_default_db_type(thd); > + thd->lex->query_tables->open_strategy= TABLE_LIST::OPEN_STUB; > + thd->lex->create_info.default_table_charset= NULL; > + thd->lex->name= null_lex_str; > + create_table= thd->lex->create_last_non_select_table= thd->lex->last_table(); > + thd->lex->current_select= &thd->lex->select_lex; > + create_table_set_open_action_and_adjust_tables(thd->lex); > + alter_info.create_list= sp->m_cols_list; > + err= mysql_create_table(thd, create_table, &create_info, &alter_info); Ok, so now it is using mysql_create_table instead of parsing the table definition. This is a step forward. However, I still think that mysql_create_table() does too much. It is a function that used for regular CREATE TABLE statements. When I trace a query, I still see that the created temporary table uses InnoDB storage engine. I think, more effort is needed to make temporary table creation work in a more similar to how other parts of code create temporary tables. I think I have also found a problem with how name resolution process works. I'll elaborate in a separate email. > + thd->variables.option_bits|= OPTION_KEEP_LOG; What is this for? please add a comment. > + if (!err) > + close_thread_tables(thd); > + execute_table_function(thd,sp,tables->table_func_params); > + thd->variables.sql_mode= saved_mode; > + lex_end(thd->lex); > + thd->lex= old_lex; > + if (strlen(sp->m_table_alias.str) > 0) { > + tables->table_name= sp->m_table_alias.str; > + tables->mdl_request.init(tables->mdl_request.key.mdl_namespace(), > + tables->db,sp->m_table_alias.str, > + tables->mdl_request.type,tables->mdl_request.duration); > + } What does the above statement do? Please add a comment. > + } > + tables = tables->next_local; > + } > + close_thread_tables(thd); > + return err; > +} ... > +bool execute_table_function(THD *thd, sp_head *sp, List<Item> *value_list) > +{ > + /* bits that should be cleared in thd->server_status */ > + uint bits_to_be_cleared= 0; > + if (sp->m_flags & sp_head::MULTI_RESULTS) > + { > + if (!(thd->client_capabilities & CLIENT_MULTI_RESULTS)) > + { > + /* The client does not support multiple result sets being sent back */ > + my_error(ER_SP_BADSELECT, MYF(0), sp->m_qname.str); > + return 1; > + } > + /* > + If SERVER_MORE_RESULTS_EXISTS is not set, > + then remember that it should be cleared > + */ > + bits_to_be_cleared= (~thd->server_status & > + SERVER_MORE_RESULTS_EXISTS); > + thd->server_status|= SERVER_MORE_RESULTS_EXISTS; > + } Does the above make any sense here? This is a piece of logic from code that executes stored *procedures*. A stored procedure may have multiple SELECT statements. Executing this stored procedure with "CALL sp(...)" may produce multiple result sets. So this logic checks that the client is able to receive multiple resultsets. A stored *function* may not emit resultsets. It must return a scalar value, in our case it must return a table. > + ha_rows select_limit= thd->variables.select_limit; > + thd->variables.select_limit= HA_POS_ERROR; > + > + /* > + We never write CALL statements into binlog: > + - If the mode is non-prelocked, each statement will be logged > + separately. > + - If the mode is prelocked, the invoking statement will care > + about writing into binlog. > + So just execute the statement. > + */ > + int res= sp->execute_procedure(thd, value_list); > + > + thd->variables.select_limit= select_limit; > + thd->server_status&= ~bits_to_be_cleared; > + > + if (res) > + { > + DBUG_ASSERT(thd->is_error() || thd->killed); > + return 1; // Substatement should already have sent error > + } > + > + my_ok(thd, (thd->get_row_count_func() < 0) ? 0 : thd->get_row_count_func()); What is the above for? As far as I understand, the code in this function was based on do_execute_sp(). Which is not a problem, however, do_execute_sp() is used to handle "CALL stored_proc" or compound statements, which should produce "Ok" if they have not produced a resultset. There is no such need when using a table function. Upper query will produce a resultset which it will return to the client. > + thd->reset_query_timer(); What is this for? > + thd->lex->unit.cleanup(); > + thd->mdl_context.release_transactional_locks(); > + thd->reset_for_next_command(); > + return 0; > +} > \ No newline at end of file ... > diff --git a/sql/table.h b/sql/table.h > index 622a3b2..5eddd3d 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -2010,6 +2010,7 @@ struct TABLE_LIST > List<Item> used_items; > /* Sublist (tail) of persistent used_items */ > List<Item> persistent_used_items; > + List<Item> *table_func_params; //parameters used for table function. > Item **materialized_items; > > /* View creation context. */ > @@ -2059,7 +2060,7 @@ struct TABLE_LIST > from I_S should be done by create_sort_index() or by JOIN::exec.) > */ > enum enum_schema_table_state schema_table_state; > - > + bool is_table_function; //Check whether the table is actually a table function. Do I get it right that a TABLE_LIST object that denotes a Table Function invocation always has table_func_params != NULL? If yes, please remove 'bool is_table_function' and add a 'bool is_table_function()' function instead which checks for table_func_params!=NULL. > /* Something like a "query plan" for reading INFORMATION_SCHEMA table */ > IS_table_read_plan *is_table_read_plan; > > @@ -2277,6 +2278,18 @@ struct TABLE_LIST > return false; > } > void set_lock_type(THD* thd, enum thr_lock_type lock); > + void fill_table_func_params(List<Item> param_list) > + { > + table_func_params= new List<Item>; > + List_iterator_fast<Item> it(param_list); > + Item *param; > + param= it++; > + while(param) > + { > + table_func_params->push_back(param); > + param= it++; > + } > + } > > private: > bool prep_check_option(THD *thd, uint8 check_opt_type); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-7502 Provisioning of slave - review
by Kristian Nielsen 17 Aug '15

17 Aug '15

Martin Kaluznik <martin.kaluznik(a)gmail.com> writes: > Earlier you mentioned, that you are able to get my code on build > server to try compile and test runs on various platforms, it would be Ok. I have added you as a developer on https://github.com/MariaDB/server. The way it works is that you push your changes to that repository in a branch that starts with "bb-", eg. "bb-10.1-f4rnham". Then Buildbot will pick up your changes and start building (can take some hours or even a day or two in busy periods). Your builds will appear then on this page: https://buildbot.askmonty.org/buildbot/grid?branch=bb-10.1-f4rnham (Please be careful about not accidentally pushing to main branches (or any other branches) in the main repository until we've completed the review process.) Then when the builds are done, you can use the web interface to check for any failures on special platforms or test configuration, and fix them as appropriate. Meanwhile, I will read through all of your patch, and get back to you with comments. > dbug.c,my_dbug.h > When line like SET GLOBAL debug_dbug="+d,flag"; is executed in > test case, it acquires THR_LOCK_dbug, but when it is tested for > (DBUG_EXECUTE_IF) no lock is used - and it causes data races in > replication threads Hm. The way it usually works is that the test cases SET GLOBAL debug_dbug while the slave threads are stopped. That way there should be no races? You might be right that there is a need for a lock, but I would prefer if that was made as a separate change from MDEV-7502, if possible. But it can be tricky to properly control the slave threads for testing, I will keep this point in mind when I check your test cases, and see if I can come up with concrete suggestions where needed. > Edited *.test,*.result > Updated old test cases, most of them because I have added new log > event type, what increased length of format description event and all > positions in test cases have shifted Right, annoying but expected. The Buildbot might catch a few more that are missed by a standard test run. - Kristian.

3 2

[Maria-developers] Unexpected contents of mysql.gtid_slave_pos
by Giuseppe Maxia 13 Aug '15

13 Aug '15

Using MariaDB 10.0.20, when I set a multi-master environment in which one of the masters has log-slave-updated enabled, I get this strange behavior: -- node#1 select @@server_id; +-------------+ | @@server_id | +-------------+ | 101 | +-------------+ use test; create table t1 (i int not null primary key, msg varchar(50), d date, t time, dt datetime, ts timestamp); use mysql; select * from gtid_slave_pos; +-----------+--------+-----------+--------+ | domain_id | sub_id | server_id | seq_no | +-----------+--------+-----------+--------+ | 0 | 1 | 101 | 1 | +-----------+--------+-----------+--------+ Why is this happening? Is there a reason for the server to log its own transactions? The transaction originated from this same server, but it is effectively filtered off when it comes back from the other node (otherwise we would have a conflict). Note: I get the same behavior if I use multi-source setup where one of the nodes has log-slave-updates enabled. Using different domains for each master does not affect the outcome. Thanks Giuseppe

2 4

[Maria-developers] Interested in Contributing Code to Maria DB.
by Alangi Derick 08 Aug '15

08 Aug '15

Hi, I am Alangi Derick and a computer engineering student(specialty: software engineering) from the University of Buea. I am a junior and i code in C. I have found interest in Maria DB. I have used many DBMS but i will like to develop for MariaDB since it is Open source and it is the one that has caught my interest. I have some contributions to make to this organisation in terms of the Maria DB project. I am very proficient in my skills of programming and will first of all start by fixing some bugs in the Maria DB code base before writing a whole module(implementation of my idea) to add to maria db. I will like if the entire community welcomes and support me so as to make Maria DB a better DBMS for everyone. Thanks and hope to participate in the development. Regards Alangi Derick Ndimnain

2 1

Re: [Maria-developers] [Commits] 686761f: MDEV-6066: Merge new defaults from 5.6 and 5.7 (part 1 (no hash serch) v2)
by Sergei Golubchik 07 Aug '15

07 Aug '15

Hi, Sanja! Summary: The implementation looks very much ok, all comments below are about details, nothing big. Well done! On Jul 14, sanja(a)mariadb.com wrote: > revision-id: 686761fbb0fc36e4fa781b68aba24af21599822d > parent(s): 302bf7c4664b904482ecc133476e822d497b114d > committer: Oleksandr Byelkin > branch nick: server could you update your post-commit trigger? thanks. > timestamp: 2015-07-14 00:11:25 +0200 > message: > > MDEV-6066: Merge new defaults from 5.6 and 5.7 (part 1 (no hash serch) v2) > diff --git a/mysql-test/extra/binlog_tests/mysqlbinlog_row_engine.inc b/mysql-test/extra/binlog_tests/mysqlbinlog_row_engine.inc > index 95440ab..c072d50 100644 > --- a/mysql-test/extra/binlog_tests/mysqlbinlog_row_engine.inc > +++ b/mysql-test/extra/binlog_tests/mysqlbinlog_row_engine.inc > @@ -16,6 +16,7 @@ > # > > --source include/have_log_bin.inc > +set sql_mode=""; please split this big commit in 'new defaults' and 'autoset' commits. > > SET NAMES 'utf8'; > #SHOW VARIABLES LIKE 'character_set%'; > diff --git a/mysql-test/r/analyze_stmt_privileges.result b/mysql-test/r/analyze_stmt_privileges.result > index d382b0b..05a35e7 100644 > --- a/mysql-test/r/analyze_stmt_privileges.result > +++ b/mysql-test/r/analyze_stmt_privileges.result > @@ -7,7 +7,7 @@ use db; > create table t1 (i int, c varchar(8)); > insert into t1 values (1,'foo'),(2,'bar'),(3,'baz'),(4,'qux'); > create view v1 as select * from t1 where i > 1; > -grant ALL on db.v1 to u1@localhost; > +set statement sql_mode="" for grant ALL on db.v1 to u1@localhost; heh. I think it would've been less confusing to put here an explicit CREATE USER statement > connect con1,localhost,u1,,; > select * from db.t1; > ERROR 42000: SELECT command denied to user 'u1'@'localhost' for table 't1' > diff --git a/mysql-test/r/blackhole_plugin.result b/mysql-test/r/blackhole_plugin.result > index 4ef9fa0..ab106f8 100644 > --- a/mysql-test/r/blackhole_plugin.result > +++ b/mysql-test/r/blackhole_plugin.result > @@ -1,7 +1,9 @@ > +set sql_mode=""; you could've put --error 1286 instead. > CREATE TABLE t1(a int) ENGINE=BLACKHOLE; > Warnings: > Warning 1286 Unknown storage engine 'BLACKHOLE' > Warning 1266 Using storage engine MyISAM for table 't1' > +set sql_mode=default; > DROP TABLE t1; > INSTALL PLUGIN blackhole SONAME 'ha_blackhole.so'; > INSTALL PLUGIN BLACKHOLE SONAME 'ha_blackhole.so'; > diff --git a/mysql-test/r/change_user.result b/mysql-test/r/change_user.result > index 18c53a5..bd87398 100644 > --- a/mysql-test/r/change_user.result > +++ b/mysql-test/r/change_user.result > @@ -1,3 +1,5 @@ > +set sql_mode=""; > +set global secure_auth=0; > grant select on test.* to test_nopw; > grant select on test.* to test_oldpw identified by password "09301740536db389"; > grant select on test.* to test_newpw identified by "newpw"; you could've used "create user" instead of "grant select on test.* to" and that would work without resetting sql_mode > diff --git a/mysql-test/r/concurrent_innodb_safelog.result b/mysql-test/r/concurrent_innodb_safelog.result > index 24a84af..03d20f4 100644 > --- a/mysql-test/r/concurrent_innodb_safelog.result > +++ b/mysql-test/r/concurrent_innodb_safelog.result > @@ -1,4 +1,5 @@ > SET GLOBAL TRANSACTION ISOLATION LEVEL REPEATABLE READ; > +SET SQL_MODE=""; same. basically I'd suggest to fix tests instead of setting sql_mode="". i suspect it's mostly adding create user or replacing grant with create user. sometimes it's about removing engine=xxx and you're doing that already. > SELECT @@global.tx_isolation; > @@global.tx_isolation > REPEATABLE-READ > diff --git a/mysql-test/r/func_compress.result b/mysql-test/r/func_compress.result > index 9fde006..4763d87 100644 > --- a/mysql-test/r/func_compress.result > +++ b/mysql-test/r/func_compress.result > @@ -1,3 +1,4 @@ > +set global max_allowed_packet=1048576; again, I'd rather fix the test > select @test_compress_string:='string for test compress function aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa '; > @test_compress_string:='string for test compress function aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa ' > string for test compress function aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa > diff --git a/mysql-test/r/host_cache_size_functionality.result b/mysql-test/r/host_cache_size_functionality.result > index e7f9e09..ffa8d71 100644 > --- a/mysql-test/r/host_cache_size_functionality.result > +++ b/mysql-test/r/host_cache_size_functionality.result > @@ -9,7 +9,7 @@ SELECT COUNT(@@GLOBAL.Host_Cache_Size) > set @Default_host_cache_size=128; > select @@global.Host_Cache_Size=@Default_host_cache_size; > @@global.Host_Cache_Size=@Default_host_cache_size > -1 > +0 > 1 Expected Now that looks weird. > '#---------------------WL6372_VAR_6_02----------------------#' > # Restart server with Host_Cache_Size 1 > diff --git a/mysql-test/r/index_intersect.result b/mysql-test/r/index_intersect.result > index 1337c3f..edc52db 100644 > --- a/mysql-test/r/index_intersect.result > +++ b/mysql-test/r/index_intersect.result > @@ -1035,7 +1032,7 @@ EXPLAIN > SELECT * FROM t1 > WHERE (f1 < 535 OR f1 > 985) AND ( f4='r' OR f4 LIKE 'a%' ) ; > id select_type table type possible_keys key key_len ref rows Extra > -1 SIMPLE t1 range PRIMARY,f4 f4 35 NULL # Using index condition; Using where > +1 SIMPLE t1 index_merge PRIMARY,f4 PRIMARY,f4 4,39 NULL # Using sort_intersect(PRIMARY,f4); Using where why? > SELECT * FROM t1 > WHERE (f1 < 535 OR f1 > 985) AND ( f4='r' OR f4 LIKE 'a%' ) ; > f1 f4 f5 > diff --git a/mysql-test/r/mysqld--help.result b/mysql-test/r/mysqld--help.result > index fc22af6..1f2a8cf 100644 > --- a/mysql-test/r/mysqld--help.result > +++ b/mysql-test/r/mysqld--help.result > @@ -890,6 +890,7 @@ The following options may be given as the first argument: > write privileges to the mysql.user table. > --secure-auth Disallow authentication for accounts that have old > (pre-4.1) passwords > + (Defaults to on; use --skip-secure-auth to disable.) You know what, it would be nice if the user would know what options support the --autoset prefix. I think here's a good place for it, the help for back_log can end with gets very many connection requests in a very short time (Automatically configured unless set explicitly) Of course, this line should be automatically generated by my_print_help(), not manually added to the help text of every autoset variable. > --secure-file-priv=name > Limit LOAD DATA, SELECT ... OUTFILE, and LOAD_FILE() to > files within specified directory > diff --git a/mysql-test/r/selectivity.result b/mysql-test/r/selectivity.result > index 620bdc6..4238359 100644 > --- a/mysql-test/r/selectivity.result > +++ b/mysql-test/r/selectivity.result > @@ -1298,10 +1298,10 @@ Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b`,`test`.`t2`.`c` A > select * from t1, t2, t1 as t3 > where t1.b=t2.c and t2.d=t3.a and t3.b<5 and t1.a < 2000; > a b c d a b > -1495 89366 89366 28296 28296 3 > -961 24512 24512 85239 85239 4 > 1063 89366 89366 28296 28296 3 > +1495 89366 89366 28296 28296 3 > 221 56120 56120 28296 28296 3 > +961 24512 24512 85239 85239 4 why? > set optimizer_use_condition_selectivity=3; > explain extended > select * from t1, t2, t1 as t3 > @@ -1315,10 +1315,10 @@ Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b`,`test`.`t2`.`c` A > select * from t1, t2, t1 as t3 > where t1.b=t2.c and t2.d=t3.a and t3.b<5 and t1.a < 2000; > a b c d a b > -961 24512 24512 85239 85239 4 > -1495 89366 89366 28296 28296 3 > 1063 89366 89366 28296 28296 3 > +1495 89366 89366 28296 28296 3 > 221 56120 56120 28296 28296 3 > +961 24512 24512 85239 85239 4 why? > set optimizer_use_condition_selectivity=@save_optimizer_use_condition_selectivity; > drop table t1,t2; > set histogram_type=@save_histogram_type; > diff --git a/mysql-test/r/selectivity_innodb.result b/mysql-test/r/selectivity_innodb.result > index 0acbb46..daf2807 100644 > --- a/mysql-test/r/selectivity_innodb.result > +++ b/mysql-test/r/selectivity_innodb.result > @@ -1308,10 +1308,10 @@ Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b`,`test`.`t2`.`c` A > select * from t1, t2, t1 as t3 > where t1.b=t2.c and t2.d=t3.a and t3.b<5 and t1.a < 2000; > a b c d a b > -1495 89366 89366 28296 28296 3 > -961 24512 24512 85239 85239 4 > 1063 89366 89366 28296 28296 3 > +1495 89366 89366 28296 28296 3 > 221 56120 56120 28296 28296 3 > +961 24512 24512 85239 85239 4 why? > set optimizer_use_condition_selectivity=3; > explain extended > select * from t1, t2, t1 as t3 > @@ -1325,10 +1325,10 @@ Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b`,`test`.`t2`.`c` A > select * from t1, t2, t1 as t3 > where t1.b=t2.c and t2.d=t3.a and t3.b<5 and t1.a < 2000; > a b c d a b > -961 24512 24512 85239 85239 4 > -1495 89366 89366 28296 28296 3 > 1063 89366 89366 28296 28296 3 > +1495 89366 89366 28296 28296 3 > 221 56120 56120 28296 28296 3 > +961 24512 24512 85239 85239 4 why? > set optimizer_use_condition_selectivity=@save_optimizer_use_condition_selectivity; > drop table t1,t2; > set histogram_type=@save_histogram_type; > diff --git a/mysql-test/suite/funcs_1/r/is_columns_is_embedded.result b/mysql-test/suite/funcs_1/r/is_columns_is_embedded.result > index c314165..bd49492 100644 > --- a/mysql-test/suite/funcs_1/r/is_columns_is_embedded.result > +++ b/mysql-test/suite/funcs_1/r/is_columns_is_embedded.result > @@ -167,9 +167,9 @@ def information_schema GEOMETRY_COLUMNS MAX_PPR 12 0 NO tinyint NULL NULL 3 0 NU > def information_schema GEOMETRY_COLUMNS SRID 13 0 NO smallint NULL NULL 5 0 NULL NULL NULL smallint(5) > def information_schema GEOMETRY_COLUMNS STORAGE_TYPE 9 0 NO tinyint NULL NULL 3 0 NULL NULL NULL tinyint(2) > def information_schema GLOBAL_STATUS VARIABLE_NAME 1 NO varchar 64 192 NULL NULL NULL utf8 utf8_general_ci varchar(64) > -def information_schema GLOBAL_STATUS VARIABLE_VALUE 2 NO varchar 1024 3072 NULL NULL NULL utf8 utf8_general_ci varchar(1024) > +def information_schema GLOBAL_STATUS VARIABLE_VALUE 2 NO varchar 2048 6144 NULL NULL NULL utf8 utf8_general_ci varchar(2048) > def information_schema GLOBAL_VARIABLES VARIABLE_NAME 1 NO varchar 64 192 NULL NULL NULL utf8 utf8_general_ci varchar(64) > -def information_schema GLOBAL_VARIABLES VARIABLE_VALUE 2 NO varchar 1024 3072 NULL NULL NULL utf8 utf8_general_ci varchar(1024) > +def information_schema GLOBAL_VARIABLES VARIABLE_VALUE 2 NO varchar 2048 6144 NULL NULL NULL utf8 utf8_general_ci varchar(2048) why? > def information_schema INDEX_STATISTICS INDEX_NAME 3 NO varchar 192 576 NULL NULL NULL utf8 utf8_general_ci varchar(192) > def information_schema INDEX_STATISTICS ROWS_READ 4 0 NO bigint NULL NULL 19 0 NULL NULL NULL bigint(21) > def information_schema INDEX_STATISTICS TABLE_NAME 2 NO varchar 192 576 NULL NULL NULL utf8 utf8_general_ci varchar(192) > diff --git a/mysql-test/suite/perfschema/r/stage_mdl_global.result b/mysql-test/suite/perfschema/r/stage_mdl_global.result > index 1a6f51a..de5df8f 100644 > --- a/mysql-test/suite/perfschema/r/stage_mdl_global.result > +++ b/mysql-test/suite/perfschema/r/stage_mdl_global.result > @@ -6,6 +6,8 @@ user1 statement/sql/flush flush tables with read lock > username event_name nesting_event_type > username event_name nesting_event_type > user1 stage/sql/init STATEMENT > +user1 stage/sql/Waiting for query cache lock STATEMENT > +user1 stage/sql/init STATEMENT why? > user1 stage/sql/query end STATEMENT > user1 stage/sql/closing tables STATEMENT > user1 stage/sql/freeing items STATEMENT > diff --git a/mysys/my_getopt.c b/mysys/my_getopt.c > index 88fa3d4..1d32833 100644 > --- a/mysys/my_getopt.c > +++ b/mysys/my_getopt.c > @@ -47,14 +47,15 @@ static char *check_struct_option(char *cur_arg, char *key_name); > order of their arguments must correspond to each other. > */ > static const char *special_opt_prefix[]= > -{"skip", "disable", "enable", "maximum", "loose", 0}; > +{"skip", "disable", "enable", "maximum", "loose", "autoset", 0}; So, I'm not very happy with the --autoset- prefix. Because it cannot be used from the SQL. In SQL we have to use SET variable=AUTO; or something like that. And it would be good to use the same approach on the command line and from SQL. On the other hand, =AUTO has its problems too - it conflicts with the AUTO value of the enum/set variables. I don't have a good answer to this :( > static const uint special_opt_prefix_lengths[]= > -{ 4, 7, 6, 7, 5, 0}; > +{ 4, 7, 6, 7, 5, 7, 0}; > enum enum_special_opt > -{ OPT_SKIP, OPT_DISABLE, OPT_ENABLE, OPT_MAXIMUM, OPT_LOOSE}; > +{ OPT_SKIP, OPT_DISABLE, OPT_ENABLE, OPT_MAXIMUM, OPT_LOOSE, OPT_AUTOSET}; > > char *disabled_my_option= (char*) "0"; > char *enabled_my_option= (char*) "1"; > +char *autoset_my_option= (char*) ""; > > /* > This is a flag that can be set in client programs. 0 means that > @@ -458,6 +462,36 @@ int handle_options(int *argc, char ***argv, > } > argument= optend; > } > + else if (option_is_autoset) > + { > + if (optend) > + { > + my_getopt_error_reporter(ERROR_LEVEL, > + "%s: automatic setup request of " > + "option '--%s' cannot take an argument", > + my_progname, optp->name); heh, another argument agains --autoset- prefix. With the =AUTO value one cannot possibly have this error. So, perhaps, I'm *slighty* leaning towards the =AUTO vs --autoset. > + > + DBUG_RETURN(EXIT_NO_ARGUMENT_ALLOWED); > + } > + /* > + We support automatic setup only via get_one_option and only for > + marked options. > + */ > + if (!get_one_option || > + !(optp->var_type & GET_AUTO)) > + { > + my_getopt_error_reporter(option_is_loose ? > + WARNING_LEVEL : ERROR_LEVEL, > + "%s: automatic setup request is " > + "unsupported by option '--%s'", > + my_progname, optp->name); > + if (!option_is_loose) > + return EXIT_ARGUMENT_INVALID; > + continue; > + } > + else > + argument= autoset_my_option; > + } > else if (optp->arg_type == REQUIRED_ARG && !optend) > { > /* Check if there are more arguments after this one, > diff --git a/sql/set_var.cc b/sql/set_var.cc > index c65ca3d..2251c36 100644 > --- a/sql/set_var.cc > +++ b/sql/set_var.cc > @@ -1075,7 +1076,7 @@ int fill_sysvars(THD *thd, TABLE_LIST *tables, COND *cond) > // NUMERIC_MAX_VALUE > // NUMERIC_BLOCK_SIZE > bool is_unsigned= true; > - switch (var->option.var_type) > + switch (vartype) ah, yes. thanks. > { > case GET_INT: > case GET_LONG: > @@ -1176,3 +1177,20 @@ void mark_sys_var_value_origin(void *ptr, enum sys_var::where here) > DBUG_ASSERT(found); // variable must have been found > } > > +enum sys_var::where get_sys_var_value_origin(void *ptr) > +{ > + DBUG_ASSERT(!mysqld_server_started); // only to be used during startup > + > + for (uint i= 0; i < system_variable_hash.records; i++) > + { > + sys_var *var= (sys_var*) my_hash_element(&system_variable_hash, i); > + if (var->option.value == ptr) > + { > + return var->value_origin; //first match > + } > + } > + > + DBUG_ASSERT(1); // variable must have been found you mean DBUG_ASSERT(0); here > + return sys_var::CONFIG; > +} > + > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 43a3f0e..21a1198 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -4316,6 +4316,27 @@ static int init_common_variables() > } > #endif /* HAVE_SOLARIS_LARGE_PAGES */ > > + > + /* Fix host_cache_size. */ > + if (IS_SYSVAR_AUTOSIZE(&host_cache_size)) > + { > + if (max_connections <= 628 - 128) > + SYSVAR_AUTOSIZE(host_cache_size, 128 + max_connections); > + else if (max_connections <= ((ulong)(2000 - 628)) * 20 + 500) > + SYSVAR_AUTOSIZE(host_cache_size, 628 + ((max_connections - 500) / 20)); > + else > + SYSVAR_AUTOSIZE(host_cache_size, 2000); > + } > + > + /* Fix back_log */ > + if (back_log == 0 || IS_SYSVAR_AUTOSIZE(&back_log)) why back_log==0 ? > + { > + if ((900 - 50) * 5 >= max_connections) > + SYSVAR_AUTOSIZE(back_log, (50 + max_connections / 5)); > + else > + SYSVAR_AUTOSIZE(back_log, 900); > + } > + > /* connections and databases needs lots of files */ > { > uint files, wanted_files, max_open_files; > diff --git a/sql/sys_vars.ic b/sql/sys_vars.ic > index 7c0359a..b04bdb5 100644 > --- a/sql/sys_vars.ic > +++ b/sql/sys_vars.ic > @@ -226,6 +227,7 @@ class Sys_var_integer: public sys_var > typedef Sys_var_integer<int, GET_INT, SHOW_SINT> Sys_var_int; > typedef Sys_var_integer<uint, GET_UINT, SHOW_UINT> Sys_var_uint; > typedef Sys_var_integer<ulong, GET_ULONG, SHOW_ULONG> Sys_var_ulong; > +typedef Sys_var_integer<ulong, (GET_ULONG|GET_AUTO), SHOW_ULONG> Sys_var_aulong; Hmm, why did you make it a new type? I think the *type* of the variable doesn't depend on the auto-set behavior. The type defines storage size, how to print the value, etc. While auto-set is just a behavior modifier. I'd rather put it in flags: - READ_ONLY GLOBAL_VAR(back_log), CMD_LINE(REQUIRED_ARG), + AUTO_SET READ_ONLY GLOBAL_VAR(back_log), CMD_LINE(REQUIRED_ARG), > typedef Sys_var_integer<ha_rows, GET_HA_ROWS, SHOW_HA_ROWS> Sys_var_harows; > typedef Sys_var_integer<ulonglong, GET_ULL, SHOW_ULONGLONG> Sys_var_ulonglong; > typedef Sys_var_integer<long, GET_LONG, SHOW_SLONG> Sys_var_long; > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 7a7b9d1..6d90305 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -360,12 +360,12 @@ static Sys_var_mybool Sys_automatic_sp_privileges( > GLOBAL_VAR(sp_automatic_privileges), > CMD_LINE(OPT_ARG), DEFAULT(TRUE)); > > -static Sys_var_ulong Sys_back_log( > +static Sys_var_aulong Sys_back_log( > "back_log", "The number of outstanding connection requests " > "MySQL can have. This comes into play when the main MySQL thread " s/MySQL/MariaDB/ I suppose > "gets very many connection requests in a very short time", > READ_ONLY GLOBAL_VAR(back_log), CMD_LINE(REQUIRED_ARG), > - VALID_RANGE(1, 65535), DEFAULT(150), BLOCK_SIZE(1)); > + VALID_RANGE(0, 65535), DEFAULT(150), BLOCK_SIZE(1)); > > static Sys_var_charptr Sys_basedir( > "basedir", "Path to installation directory. All paths are " > @@ -2751,16 +2743,38 @@ static bool check_query_cache_type(sys_var *self, THD *thd, set_var *var) > my_error(ER_QUERY_CACHE_IS_DISABLED, MYF(0)); > return true; > } > - if (var->type != OPT_GLOBAL && > - global_system_variables.query_cache_type == 0 && > - var->value->val_int() != 0) > + > + if (var->type != OPT_GLOBAL && global_system_variables.query_cache_type == 0) > { > - my_error(ER_QUERY_CACHE_IS_GLOBALY_DISABLED, MYF(0)); > - return true; > - } > + uint value= 0 /*default is off*/; > + if (var->value) > + { > + if (var->value->result_type() == INT_RESULT) > + value= var->value->val_int(); > + else > + { > + char buff[STRING_BUFFER_USUAL_SIZE]; > + String str(buff, sizeof(buff), system_charset_info), *res; > + if (!(res=var->value->val_str(&str))) > + return true; > + if (res->length() != 3 || > + my_toupper(res->charset(), res->ptr()[0]) != 'O' || > + my_toupper(res->charset(), res->ptr()[1]) != 'F' || > + my_toupper(res->charset(), res->ptr()[2]) != 'F') eh? query_cache_type is ENUM variable, there is no need to compare string values manually. > + value= 1; // set to something not 0 > + } > + } > > + if (value != 0) > + { > + my_error(ER_QUERY_CACHE_IS_GLOBALY_DISABLED, MYF(0)); > + return true; > + } > + } > return false; > } > + > + > static bool fix_query_cache_type(sys_var *self, THD *thd, enum_var_type type) > { > if (type != OPT_GLOBAL) > diff --git a/unittest/mysys/CMakeLists.txt b/unittest/mysys/CMakeLists.txt > index 8c09a46..6564e09 100644 > --- a/unittest/mysys/CMakeLists.txt > +++ b/unittest/mysys/CMakeLists.txt > @@ -13,7 +13,7 @@ > # along with this program; if not, write to the Free Software > # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA > > -MY_ADD_TESTS(bitmap base64 my_vsnprintf my_atomic my_rdtsc lf my_malloc > +MY_ADD_TESTS(bitmap base64 my_vsnprintf my_atomic my_rdtsc lf my_malloc my_getopt Ah, that's great, thanks! Long overdue. > LINK_LIBRARIES mysys) > > MY_ADD_TESTS(ma_dyncol Regards, Sergei

2 2

Re: [Maria-developers] [Commits] 69704b6: MDEV-6887 Optimize sys_vars.sysvar_* tests
by Sergei Golubchik 06 Aug '15

06 Aug '15

Hi, Elena! On Aug 06, elenst(a)montyprogram.com wrote: > revision-id: 69704b6c6f895fa706f83bcd3bcefe267aaf8e57 > parent(s): afd59b575a75ebbc57f71ce2865fdff85e3e233b > committer: Elena Stepanova > branch nick: bb-10.1-elenst > timestamp: 2015-08-06 02:55:43 +0300 > message: > > MDEV-6887 Optimize sys_vars.sysvar_* tests > > diff --git a/mysql-test/suite/sys_vars/inc/sysvars_server.inc b/mysql-test/suite/sys_vars/inc/sysvars_server.inc > index cb06b40..98bffd4 100644 > --- a/mysql-test/suite/sys_vars/inc/sysvars_server.inc > +++ b/mysql-test/suite/sys_vars/inc/sysvars_server.inc > @@ -11,8 +12,9 @@ set sql_mode=ansi_quotes; > # global_value_origin=SQL > set global div_precision_increment=5; > > ---replace_regex /^\/\S+/PATH/ > +--replace_regex /^\/\S+/PATH/ /^[a-zA-Z]:[\\\/]\S+/PATH/ /^BIGINT\b/<INT or BIGINT>/ /^INT\b/<INT or BIGINT>/ /^(?:18446744073709551615\b|4294967295\b)/<32-bit or 64-bit MAX UNSIGNED>/ /^(?:9223372036854775807\b|2147483647\b)/<32-bit or 64-bit MAX SIGNED>/ /^(?:18446744073709547520\b|4294963200\b)/<32-bit or 64-bit MAX UNSIGNED ADJUSTED>/ /^(?:9223372036853727232\b|2146435072\b)/<32-bit or 64-bit MAX SIGNED ADJUSTED>/ Aren't you removing too much with these replacements? It's kinda good to get rid of 32bit diff files (as they aren't easy to update), but I wouldn't replace every INT or BIGINT with <INT or BIGINT>. May be, move them to a separate select, like let type=BIGINT; if (32-bit) { let type=INT; } --replace $type <INT-or-BIGINT> select * from information_schema.system_variables where variable_name in (list of ulong variables); and something similar for other replacements. In fact, there's obscure feature of let $repl=/foo/bar/ /abc/def; replace_regex $repl; so you can have sysvar32bitrepl.inc that will set up $replace_regexes variable: let type=BIGINT; let max=18446744073709551615; if (32-bit) { let type=INT; let max=4294967295; } let replace_regexes=/$type/INT or BIGINT/ /$max/<32-bit or 64-bit MAX UNSIGNED>/; Then in every sysvar*.test file you can do, like --source sysvar32bitrepl --replace_regex $replace_regexes select * from information_schema.system_variables where variable_name in (list of ulong variables); Regards, Sergei

2 2

Re: [Maria-developers] MDEV-8389 patch review
by Sergey Petrunia 06 Aug '15

06 Aug '15

Hi Diwas, On Wed, Aug 05, 2015 at 08:47:28PM +0530, Diwas Joshi wrote: > I have removed the errors from previous patch, I only got those errors when > i compiled with valgrind. Everything seems to be working fine now. Also I > have fixed the test case for select * from mysql.proc. Also added a > function that removes the tables after they have been used. I have added > the errors that we discussed about and also test cases corresponding to > these errors. Please review. > The big problems in the patch are: 1. check_and_process_table_functions() is at the wrong place. I debug a query, "SELECT * FROM tbl" , where 'tbl' is a regular table. I see check_and_process_table_functions() invoke sp_find_routine() for "test.tbl". Why does this happen? The query makes it clear that the table is not a table function. This is not acceptable. 2. Temporary table 2.1 The temporary table that is created uses InnoDB storage engine. This doesn't follow any other kind of temporary table. When execution engine creates temporary tables (e.g. for VIEWs, or GROUP BY), these temporary tables are either Aria or MyISAM tables, depending on the server settings. We should follow that. 2.2 Temporary table is created by building the "CREATE TEMPORARY TABLE (...)" string and passing it to the parser. Did you try looking into creating the temp. table directly from the data strucutres you have? No other code has to build "CREATE TABLE" statements and then parse them back. I suspect Table Functions don't need to do this either. The rest of the feedback is below: > diff --git a/mysql-test/t/sp-error.test b/mysql-test/t/sp-error.test > index ccf5b98..2ad4144 100644 > --- a/mysql-test/t/sp-error.test > +++ b/mysql-test/t/sp-error.test > @@ -3846,7 +3846,23 @@ DROP FUNCTION f1; > DROP FUNCTION f2; > DROP TABLE t1; > > ---error 1235 > -SELECT * FROM f1('xyz'); > ---error 1064 > -SELECT * FROM t1, f1('xyz',t1.a); > +delimiter |; > +CREATE FUNCTION f1() RETURNS INTEGER > +BEGIN > + RETURN 1; > +END| > +delimiter ;| > +--error 1979 Please use error names, not error numbers. you can write --error ER_SP_SCALAR_VALUE_EXPECTED. > +SELECT * FROM f1(); > +DROP FUNCTION f1; > + > +delimiter |; > +CREATE FUNCTION f1(a INT, b VARCHAR(11)) > +RETURNS TABLE t1(id INT, name VARCHAR(11)) > +BEGIN > +INSERT INTO t1 SELECT id, name FROM t2 WHERE id = a; > +END| > +delimiter ;| > +--error 1978 Same as above. > +SELECT f1(1,'AA'); > +DROP FUNCTION f1; > diff --git a/mysql-test/t/sp.test b/mysql-test/t/sp.test > index 5c42df0..45db6a3 100644 > --- a/mysql-test/t/sp.test > +++ b/mysql-test/t/sp.test > @@ -9398,6 +9398,6 @@ END| > DELIMITER ;| > > --vertical_results > -SELECT * FROM mysql.proc WHERE db = database() and name = 'f1'; > - > +SELECT db, name, type, specific_name, language, sql_data_access, is_deterministic, security_type, param_list, returns, body, definer, sql_mode, comment, character_set_client, collation_connection, db_collation, body_utf8 FROM mysql.proc WHERE db = database() and name = 'f1'; The change is ok. Please also format the query so that its width is limited 80 chars. > +SELECT * FROM f1(10,'XYZ'); > DROP FUNCTION f1; > diff --git a/sql/item_func.cc b/sql/item_func.cc > index 3e78be8..ad18e2e 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -6574,7 +6574,12 @@ void my_missing_function_error(const LEX_STRING &token, const char *func_name) > context->process_error(thd); > DBUG_RETURN(TRUE); > } > - > + > + if (m_sp->m_type == TYPE_ENUM_TABLE){ > + my_error(ER_SP_SCALAR_VALUE_EXPECTED, MYF(0)); > + context->process_error(thd); > + DBUG_RETURN(TRUE); > + } > /* > A Field need to be attached to a Table. > Below we "create" a dummy table by initializing > @@ -6588,7 +6593,7 @@ void my_missing_function_error(const LEX_STRING &token, const char *func_name) > dummy_table->copy_blobs= TRUE; > share->table_cache_key = empty_name; > share->table_name = empty_name; > - > + Please don't make dummy changes like this. > if (!(sp_result_field= m_sp->create_result_field(max_length, name, > dummy_table))) > { > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 2998972..6f94720 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2467,6 +2467,8 @@ static bool do_execute_sp(THD *thd, sp_head *sp) > /* should be assigned after making first tables same */ > all_tables= lex->query_tables; > /* set context for commands which do not use setup_tables */ > + if (lex->sql_command == SQLCOM_SELECT) > + check_and_process_table_functions(thd, all_tables); Why limit table function uses to SELECTs ? A user can write an INSERT.. SELECT statement INSERT INTO table SELECT * FROM table_func('argument'); or a multi-table UPDATE statement: UPDATE regular_table, table_func('arg') SET regular_table.col= ...; In general, there should not be a limit. I think, it's ok to introduce the limit within the scope of this MDEV entry. In that case, you need to add a comment with 'TODO-DiwasJoshi:...' saying that the limitation is temporary. > select_lex-> > context.resolve_in_table_list_only(select_lex-> > table_list.first); > @@ -5697,7 +5699,8 @@ static bool do_execute_sp(THD *thd, sp_head *sp) > thd->mdl_context.release_transactional_locks(); > } > #endif /* WITH_WSREP */ > - > + if (lex->sql_command == SQLCOM_SELECT) > + clear_table_functions(thd, all_tables); This looks like an odd place to do cleanups. Let's get back to this once other issues are resolved. > DBUG_RETURN(res || thd->is_error()); > } > > @@ -9222,3 +9225,185 @@ CHARSET_INFO *find_bin_collation(CHARSET_INFO *cs) > } > return cs; > } > + > +/* > + @brief > + Check for table functions and create corresponding tables to be used in query. > + > + @param thd Thread handle > + @param table_list A list of tables being processed in the query. > + > + @return > + False on success > + @retval > + TRUE on error > +*/ > +bool check_and_process_table_functions(THD *thd, TABLE_LIST *table_list) > +{ > + TABLE_LIST *tables = table_list; > + int err= 0; > + while(tables) > + { > + if (tables->table_name == NULL) > + { > + tables = tables->next_local; > + continue; > + } > + sp_head *sp; > + sp_name *name; > + LEX_STRING lex_db; > + LEX_STRING lex_name; > + lex_db.length= strlen(tables->db); > + lex_name.length= strlen(tables->table_name); > + lex_db.str= thd->strmake(tables->db, lex_db.length); > + lex_name.str= thd->strmake(tables->table_name, lex_name.length); > + name= new sp_name(lex_db, lex_name, true); > + name->init_qname(thd); > + int err=0; > + > + if (!err && (sp = sp_find_routine(thd,TYPE_ENUM_FUNCTION, name, > + &thd->sp_proc_cache, FALSE))) //check if the table is a table-valued function. > + { > + if (sp->m_type == TYPE_ENUM_FUNCTION) > + my_error(ER_SP_TABLE_VALUE_EXPECTED, MYF(0)); > + LEX *old_lex= thd->lex, newlex; > + thd->lex= &newlex; > + ulonglong sql_mode= 0 , saved_mode= thd->variables.sql_mode; > + String defstr; //query in defstr will be parsed to create temporary table. > + defstr.append(STRING_WITH_LEN("CREATE TEMPORARY TABLE ")); > + defstr.append(tables->table_name); > + defstr.append(STRING_WITH_LEN(" (")); > + List_iterator_fast<Create_field> it(sp->m_cols_list); > + > + TABLE table; > + TABLE_SHARE share; > + bzero((char*) &table, sizeof(table)); > + bzero((char*) &share, sizeof(share)); > + table.in_use= thd; > + table.s = &share; > + Create_field *cols_field; > + cols_field= it++; > + while (cols_field) > + { > + uint unused1= 0; > + sp_prepare_create_field(thd, cols_field); > + prepare_create_field(cols_field, &unused1, HA_CAN_GEOMETRY); > + uint field_length; > + Field *field; > + String tmp_result(512); > + field_length= cols_field->length; > + cols_field->flags= 0; > + field= make_field(table.s, /* TABLE_SHARE ptr */ > + (uchar*) 0, /* field ptr */ > + field_length, /* field [max] length */ > + (uchar*) "", /* null ptr */ > + 0, /* null bit */ > + cols_field->pack_flag, > + cols_field->sql_type, > + cols_field->charset, > + cols_field->geom_type, cols_field->srid, > + Field::NONE, /* unreg check */ > + cols_field->interval, > + cols_field->field_name); > + field->vcol_info= cols_field->vcol_info; > + field->stored_in_db= cols_field->stored_in_db; > + if (field) > + field->init(&table); > + defstr.append(cols_field->field_name); > + defstr.append(STRING_WITH_LEN(" ")); > + field->sql_type(tmp_result); > + defstr.append(tmp_result); > + cols_field= it++; > + if(cols_field) > + defstr.append(STRING_WITH_LEN(", ")); > + delete field; > + } > + defstr.append(STRING_WITH_LEN(")")); > + newlex.create_info.DDL_options_st::init(); > + newlex.current_select= NULL; > + Parser_state parser_state; > + thd->variables.sql_mode= sql_mode; > + err= parser_state.init(thd, defstr.c_ptr_safe(), defstr.length()); > + if (!err) { > + lex_start(thd); > + err= parse_sql(thd, &parser_state, NULL, TRUE) || thd->lex == NULL; > + } > + if (!err) > + { > + Table_specification_st create_info(thd->lex->create_info); > + Alter_info alter_info(thd->lex->alter_info, thd->mem_root); > + thd->lex->create_info.default_table_charset= NULL; > + SELECT_LEX *select_lex= &thd->lex->select_lex; > + TABLE_LIST *create_table= select_lex->table_list.first; > + if (append_file_to_dir(thd, &create_info.data_file_name, > + create_table->table_name) || > + append_file_to_dir(thd, &create_info.index_file_name, > + create_table->table_name)) > + { > + err= 1; > + } > + if (!(create_info.used_fields & HA_CREATE_USED_ENGINE)) > + create_info.use_default_db_type(thd); > + > + err= mysql_create_table(thd, create_table, &create_info, &alter_info); > + thd->variables.option_bits|= OPTION_KEEP_LOG; > + } > + thd->variables.sql_mode= saved_mode; > + lex_end(thd->lex); > + thd->lex= old_lex; > + } > + tables = tables->next_local; > + } > + if (!err) > + close_thread_tables(thd); > + return err; > +} > + > +/* > + @brief > + Drop tables created for table functions. > + > + @param thd Thread handle > + @param table_list A list of tables being processed in the query. > + > + @return > + False on success > + @retval > + TRUE on error > +*/ > +bool clear_table_functions(THD *thd, TABLE_LIST *table_list) > +{ > + TABLE_LIST *tables = table_list; > + int err= 0; > + while(tables) > + { > + if (tables->table_name == NULL) > + { > + tables = tables->next_local; > + continue; > + } > + sp_head *sp; > + sp_name *name; > + LEX_STRING lex_db; > + LEX_STRING lex_name; > + lex_db.length= strlen(tables->db); > + lex_name.length= strlen(tables->table_name); > + lex_db.str= thd->strmake(tables->db, lex_db.length); > + lex_name.str= thd->strmake(tables->table_name, lex_name.length); > + name= new sp_name(lex_db, lex_name, true); > + name->init_qname(thd); > + int err=0; > + if (!err && (sp = sp_find_routine(thd,TYPE_ENUM_FUNCTION, name, > + &thd->sp_proc_cache, FALSE)) && This function is supposed to just close the temporary tables. Why does it need to search for SPs? > + sp->m_type== TYPE_ENUM_TABLE) //check if the table is a table-valued function. > + { > + if(tables->table) > + close_temporary_table(thd,tables->table,TRUE,TRUE); > + if (thd->is_error()) > + err= 1; > + tables->table= NULL; > + } > + tables= tables->next_local; > + } > + return err; > +} BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Best way of starting multi-source connections with active slave.
by Giuseppe Maxia 06 Aug '15

06 Aug '15

Hi, I am setting up multi-source replication with four nodes, where nodes 1, 2, and 3 are masters and node 4 is the slave of the first 3. The slave is *active*, meaning that it has a binary log and runs transactions of its own. All is well if I run 'reset master' in all nodes before setting up replication. CHANGE MASTER 'node1' TO master_host='host1', master_port=3306, master_user='some_slave', master_password='some_slave', master_use_gtid=current_pos; CHANGE MASTER 'node2' TO master_host='host2', master_port=3306, master_user='some_slave', master_password='some_slave', master_use_gtid=current_pos; CHANGE MASTER 'node3' TO master_host='host3', master_port=3306, master_user='some_slave', master_password='some_slave', master_use_gtid=current_pos; start all slaves; However, if host 4 has a binary log and has logged one or more transactions before starting replication, we get a conflict. Got fatal error 1236 from master when reading data from binary log: 'Error: connecting slave requested to start from GTID 0-104-1, which is not in the master's binlog. Since the master's binlog contains GTIDs with higher sequence numbers, it probably means that the slave has diverged due to executing extra erroneous transactions' I get the same result if I use 'slave_pos' instead of 'current_pos'. Notice that the changes in the slave do not conflict with anything that happens in the masters. For example: #1 node4 (fan-in slave): create user distribution_user identified by 'something'; #2 node4 CHANGE MASTER ... #3 node4 start all slaves #4 node1 (master): create table t1 #5 node2 (master): create table t2 #6 node3 (master): create table t3 This sequence of events fails. If I execute event #1 after #2 and #3, then all works as expected. Or it all works if I say master_use_gtid=no, but that would defeat the purpose of having GTIDs, or am I missing something? Is there a way of enabling multi-source replication with the slave having non-conflicting transactions already logged? Thanks Giuseppe

3 3

Re: [Maria-developers] 3eceb51: MDEV-7990: ERROR 1526 when procedure executed for second time ALTER TABLE partition ... pMAX values less than MAXVALUE
by Sergei Golubchik 05 Aug '15

05 Aug '15

Hi, Sanja! On May 06, sanja(a)mariadb.com wrote: > revision-id: 3eceb51e697d5a514e70c374a06e7eef6b5a37a0 > parent(s): aa5095627e2619bdad7916d33d1016802a84a9e1 > committer: Oleksandr Byelkin > branch nick: server > timestamp: 2015-05-06 13:19:22 +0200 > message: > > MDEV-7990: ERROR 1526 when procedure executed for second time ALTER > TABLE partition ... pMAX values less than MAXVALUE > > Added cleaning partition data which are saved in LEX structure to make > possible correct re-execution of a SP or PS using the code. No, this is wrong. See prep_alter_part_table(). The first thing it does it to copy thd->lex->part_info to thd->work_part_info. It tries to do a deep copy to be able to modify it freely, but apparently the copy isn't deep enough. I see two options 1. Do a deeper copy. Duplicate all value lists. 2. Make part_column_list_val elements immutable. That would be preferred, of course, but I'm not sure it's possible. Regards, Sergei

2 2

Re: [Maria-developers] MDEV-8389 patch review
by Sergey Petrunia 03 Aug '15

03 Aug '15

On Sun, Aug 02, 2015 at 08:51:44PM +0530, Diwas Joshi wrote: > So what would be the expected result of query select f1(1,'aa'). since it > will be a table shouldn't we use select * from f1(1,'aa'). I know it > shouldn't crash but what should be the output of that query? > My first guess is that since we are storing this is as functions too(as we > decided earlier) it is crashing so we'll have to check for table functions > in the execution cycle of functions Agree. When a table function is invoked in the context where a scalar value is expected, the statement should fail with an error (Please look in sql/share/errmsg.txt and pick an error code. If none works, please add a new error code/message). However, this is not what I wanted to complain about (I pasted the wrong example). Let's consider a testcase where a table function is used in the scalar context (and should return an empty table): == example starts == delimiter | CREATE FUNCTION f1(param1 VARCHAR(11)) RETURNS TABLE t1(name VARCHAR(11)) deterministic BEGIN INSERT INTO t1 SELECT a FROM t2 WHERE id = param1; END| DELIMITER ; select 1 from f1('aaa'); == example ends == This crashes for me, too. The crash is in your function: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7eb8700 (LWP 12653)] 0x0000555555b4c56b in Object_creation_ctx::set_n_backup (this=0xb, thd=0x55555ac4aeb0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/table.cc:83 (gdb) wher #0 0x0000555555b4c56b in Object_creation_ctx::set_n_backup (this=0xb, thd=0x55555ac4aeb0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/table.cc:83 #1 0x0000555555a6d280 in parse_sql (thd=0x55555ac4aeb0, parser_state=0x7ffff7eb4580, creation_ctx=0xb, do_pfs_digest=false) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:9100 #2 0x0000555555a6dcbb in check_and_process_table_functions (thd=0x55555ac4aeb0, table_list=0x7fff5c007630) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:9336 #3 0x0000555555a5ab77 in mysql_execute_command (thd=0x55555ac4aeb0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:2471 #4 0x0000555555a6942e in mysql_parse (thd=0x55555ac4aeb0, rawbuf=0x7fff5c007408 "select 1 from f1('aaa')", length=23, parser_state=0x7ffff7eb7100) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:7174 #5 0x0000555555a5838b in dispatch_command (command=COM_QUERY, thd=0x55555ac4aeb0, packet=0x55555ac51f71 "select 1 from f1('aaa')", packet_length=23) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:1470 #6 0x0000555555a570a5 in do_command (thd=0x55555ac4aeb0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:1093 #7 0x0000555555b95929 in do_handle_one_connection (thd_arg=0x55555ac4aeb0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_connect.cc:1347 #8 0x0000555555b9566e in handle_one_connection (arg=0x55555ac4aeb0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_connect.cc:1258 #9 0x00007ffff691de9a in start_thread (arg=0x7ffff7eb8700) at pthread_create.c:308 #10 0x00007ffff604e3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #11 0x0000000000000000 in ?? () Please check what's going on. (If you don't get the crash, compile and run under valgrind). > Also i removed the test case SELECT * FROM mysql.proc WHERE db = database() > and name = 'f1' because it was failing as it is writing time to result file > so the test passes if we use --record since it writes new file but fails > for ./mysql-test-run sp.test with differences in reject and result file Good point. However, the solution should be not to remove the testcase, but rather make the testcase output not depend on the date/time when the testcase was run. Why did .result file have time/date. Let's look: -SELECT * FROM mysql.proc WHERE db = database() and name = 'f1'; ... -definer root@localhost -created 2015-07-03 16:36:36 -modified 2015-07-03 16:36:36 -sql_mode -comment -character_set_client latin1 ... It did so, because the SELECT selects 'created' and 'modified' columns from mysql.proc. The testcase doesn't really need these columns, please make the SELECT not to select them. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

Re: [Maria-developers] MDEV-8389 patch review
by Sergey Petrunia 02 Aug '15

02 Aug '15

Hi Diwas, I've got a crash when trying the first example. The example was: CREATE FUNCTION f1(a INT, b VARCHAR(11)) RETURNS TABLE t1(id INT, name VARCHAR(11)) DETERMINISTIC BEGIN INSERT INTO t1 SELECT id, name FROM t2 WHERE id = a; END|; delimiter ; select f1(1,'aa'); The crash was: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7eb8700 (LWP 24655)] 0x0000555555c6a82a in Field_str::Field_str (this=0x7fff5c007d30, ptr_arg=0x0, len_arg=4294967295, null_ptr_arg=0x55555644299c "", null_bit_arg=1 '\001', unireg_check_arg=Field::NONE, field_name_arg=0x7fff5c006780 "f1(1,'aa')", charset_arg=0x0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/field.cc:1845 (gdb) wher #0 0x0000555555c6a82a in Field_str::Field_str (this=0x7fff5c007d30, ptr_arg=0x0, len_arg=4294967295, null_ptr_arg=0x55555644299c "", null_bit_arg=1 '\001', unireg_check_arg=Field::NONE, field_name_arg=0x7fff5c006780 "f1(1,'aa')", charset_arg=0x0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/field.cc:1845 #1 0x0000555555ae6820 in Field_longstr::Field_longstr (this=0x7fff5c007d30, ptr_arg=0x0, len_arg=4294967295, null_ptr_arg=0x55555644299c "", null_bit_arg=1 '\001', unireg_check_arg=Field::NONE, field_name_arg=0x7fff5c006780 "f1(1,'aa')", charset_arg=0x0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/field.h:1200 #2 0x0000555555c7d85c in Field_blob::Field_blob (this=0x7fff5c007d30, ptr_arg=0x0, null_ptr_arg=0x55555644299c "", null_bit_arg=1 '\001', unireg_check_arg=Field::NONE, field_name_arg=0x7fff5c006780 "f1(1,'aa')", share=0x7fff5c0062a0, blob_pack_length=4, cs=0x0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/field.cc:7415 #3 0x0000555555c85725 in make_field (share=0x7fff5c0062a0, ptr=0x0, field_length=2779096485, null_pos=0x55555644299c "", null_bit=1 '\001', pack_flag=2779096485, field_type=MYSQL_TYPE_NULL, field_charset=0x0, geom_type=Field::GEOM_GEOMETRY, srid=0, unireg_check=Field::NONE, interval=0x0, field_name=0x7fff5c006780 "f1(1,'aa')") at /home/psergey/dev-git/10.1-gsoc-repo/sql/field.cc:9861 #4 0x0000555555db4d35 in sp_head::create_result_field (this=0x7fff5c01c398, field_max_length=0, field_name=0x7fff5c006780 "f1(1,'aa')", table=0x7fff5c005710) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sp_head.cc:885 #5 0x0000555555d0d2b3 in Item_func_sp::init_result_field (this=0x7fff5c0055e0, thd=0x55555acf2800) at /home/psergey/dev-git/10.1-gsoc-repo/sql/item_func.cc:6593 #6 0x0000555555d0dc99 in Item_func_sp::fix_fields (this=0x7fff5c0055e0, thd=0x55555acf2800, ref=0x7fff5c006778) at /home/psergey/dev-git/10.1-gsoc-repo/sql/item_func.cc:6844 #7 0x00005555559fd1ed in setup_fields (thd=0x55555acf2800, ref_pointer_array=0x7fff5c007c90, fields=..., mark_used_columns=MARK_COLUMNS_READ, sum_func_list=0x7fff5c007a80, allow_sum_func=true) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_base.cc:7925 #8 0x0000555555aa6a83 in JOIN::prepare (this=0x7fff5c007750, rref_pointer_array=0x55555acf6b28, tables_init=0x0, wild_num=0, conds_init=0x0, og_num=0, order_init=0x0, skip_order_by=false, group_init=0x0, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x55555acf68b0, unit_arg=0x55555acf61c0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_select.cc:782 #9 0x0000555555aafa67 in mysql_select (thd=0x55555acf2800, rref_pointer_array=0x55555acf6b28, tables=0x0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x7fff5c007730, unit=0x55555acf61c0, select_lex=0x55555acf68b0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_select.cc:3301 #10 0x0000555555aa5d93 in handle_select (thd=0x55555acf2800, lex=0x55555acf60f8, result=0x7fff5c007730, setup_tables_done_option=0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_select.cc:371 #11 0x0000555555a65e98 in execute_sqlcom_select (thd=0x55555acf2800, all_tables=0x0) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:5798 #12 0x0000555555a5c151 in mysql_execute_command (thd=0x55555acf2800) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:2939 #13 0x0000555555a6942e in mysql_parse (thd=0x55555acf2800, rawbuf=0x7fff5c0051b8 "select f1(1,'aa')", length=17, parser_state=0x7ffff7eb7100) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:7174 #14 0x0000555555a5838b in dispatch_command (command=COM_QUERY, thd=0x55555acf2800, packet=0x55555acf9b41 "select f1(1,'aa')", packet_length=17) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:1470 #15 0x0000555555a570a5 in do_command (thd=0x55555acf2800) at /home/psergey/dev-git/10.1-gsoc-repo/sql/sql_parse.cc:1093 (gdb) list 1840 const char *field_name_arg, CHARSET_INFO *charset_arg) 1841 :Field(ptr_arg, len_arg, null_ptr_arg, null_bit_arg, 1842 unireg_check_arg, field_name_arg) 1843 { 1844 field_charset= charset_arg; 1845 if (charset_arg->state & MY_CS_BINSORT) 1846 flags|=BINARY_FLAG; 1847 field_derivation= DERIVATION_IMPLICIT; 1848 } 1849 (gdb) p charset_arg $10 = (CHARSET_INFO *) 0x0 Another piece of feedback: Please don't change old tests into new tests. Tests that were made for previous MDEVs should stay. New tasks should add new tests. I'll continue to look at the patch. On Sun, Aug 02, 2015 at 06:51:26PM +0530, Diwas Joshi wrote: > I am attaching the patch for MDEV-8389. As discussed earlier, I am creating > tables in the query corresponding to table functions before they are being > accessed. I am checking for each table if its a table-function using > sp_find_routine. If it is then we make a table for it. Going by the same > mechanism used in sp_find_routine to parse the function, i parse a create > table query which initializes create_info and alter_info which i later use > in mysql_create_table() to create temporary tables. I also added a test > case for it. Seems to be working fine. > > I guess we'll also need a function that will delete the tables create after > use. I am working on it right now, but i guess that won't be needed in this > patch. > Please review the patch. > > regards > Diwas Joshi > diff --git a/mysql-test/r/sp-error.result b/mysql-test/r/sp-error.result > index cacfeea..4373925 100644 > --- a/mysql-test/r/sp-error.result > +++ b/mysql-test/r/sp-error.result > @@ -2866,7 +2866,3 @@ SELECT @msg; > DROP FUNCTION f1; > DROP FUNCTION f2; > DROP TABLE t1; > -SELECT * FROM f1('xyz'); > -ERROR 42000: This version of MariaDB doesn't yet support 'Queries using Table Functions' > -SELECT * FROM t1, f1('xyz',t1.a); > -ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '' at line 1 > diff --git a/mysql-test/r/sp.result b/mysql-test/r/sp.result > index f06939b..d0f6578 100644 > --- a/mysql-test/r/sp.result > +++ b/mysql-test/r/sp.result > @@ -7943,29 +7943,7 @@ RETURNS TABLE t1(id INT, name VARCHAR(11)) > BEGIN > INSERT INTO t1 SELECT id, name FROM t2 WHERE id = a; > END| > -SELECT * FROM mysql.proc WHERE db = database() and name = 'f1'; > -db test > -name f1 > -type FUNCTION > -specific_name f1 > -language SQL > -sql_data_access CONTAINS_SQL > -is_deterministic NO > -security_type DEFINER > -param_list a INT, b VARCHAR(11) > -returns TABLE t1(id int(11), name varchar(11)) > -body BEGIN > -INSERT INTO t1 SELECT id, name FROM t2 WHERE id = a; > -END > -definer root@localhost > -created 2015-07-03 16:36:36 > -modified 2015-07-03 16:36:36 > -sql_mode > -comment > -character_set_client latin1 > -collation_connection latin1_swedish_ci > -db_collation latin1_swedish_ci > -body_utf8 BEGIN > -INSERT INTO t1 SELECT id, name FROM t2 WHERE id = a; > -END > +SELECT * FROM f1(10,'XYZ'); > +id name > +DROP TABLE f1; > DROP FUNCTION f1; > diff --git a/mysql-test/t/sp-error.test b/mysql-test/t/sp-error.test > index ccf5b98..c1a21c8 100644 > --- a/mysql-test/t/sp-error.test > +++ b/mysql-test/t/sp-error.test > @@ -3845,8 +3845,3 @@ SELECT @msg; > DROP FUNCTION f1; > DROP FUNCTION f2; > DROP TABLE t1; > - > ---error 1235 > -SELECT * FROM f1('xyz'); > ---error 1064 > -SELECT * FROM t1, f1('xyz',t1.a); > diff --git a/mysql-test/t/sp.test b/mysql-test/t/sp.test > index 5c42df0..906c91c 100644 > --- a/mysql-test/t/sp.test > +++ b/mysql-test/t/sp.test > @@ -9397,7 +9397,6 @@ END| > > DELIMITER ;| > > ---vertical_results > -SELECT * FROM mysql.proc WHERE db = database() and name = 'f1'; > - > +SELECT * FROM f1(10,'XYZ'); > +DROP TABLE f1; > DROP FUNCTION f1; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 2998972..dd16c36 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2467,6 +2467,8 @@ static bool do_execute_sp(THD *thd, sp_head *sp) > /* should be assigned after making first tables same */ > all_tables= lex->query_tables; > /* set context for commands which do not use setup_tables */ > + if (lex->sql_command == SQLCOM_SELECT) > + check_and_process_table_functions(thd, all_tables); > select_lex-> > context.resolve_in_table_list_only(select_lex-> > table_list.first); > @@ -9222,3 +9224,142 @@ CHARSET_INFO *find_bin_collation(CHARSET_INFO *cs) > } > return cs; > } > + > +/* > + @brief > + Check for table functions and create corresponding tables to be used in query. > + > + @param thd Thread handle > + @param table_list A list of tables being processed in the query. > + > + @return > + False on success > + @retval > + TRUE on error > +*/ > +bool check_and_process_table_functions(THD *thd, TABLE_LIST *table_list) > +{ > + TABLE_LIST *tables = table_list; > + int err= 0; > + while(tables) > + { > + if (tables->table_name == NULL) > + { > + tables = tables->next_local; > + continue; > + } > + sp_head *sp; > + sp_name *name; > + LEX_STRING lex_db; > + LEX_STRING lex_name; > + lex_db.length= strlen(tables->db); > + lex_name.length= strlen(tables->table_name); > + lex_db.str= thd->strmake(tables->db, lex_db.length); > + lex_name.str= thd->strmake(tables->table_name, lex_name.length); > + name= new sp_name(lex_db, lex_name, true); > + name->init_qname(thd); > + Object_creation_ctx *creation_ctx; > + int err=0; > + > + if (!err && (sp = sp_find_routine(thd,TYPE_ENUM_FUNCTION, name, > + &thd->sp_proc_cache, FALSE)) && > + sp->m_type== TYPE_ENUM_TABLE) //check if the table is a table-valued function. > + { > + LEX *old_lex= thd->lex, newlex; > + thd->lex= &newlex; > + ulonglong sql_mode= 0 , saved_mode= thd->variables.sql_mode; > + String defstr; //query in defstr will be parsed to create temporary table. > + defstr.append(STRING_WITH_LEN("CREATE TEMPORARY TABLE ")); > + defstr.append(tables->table_name); > + defstr.append(STRING_WITH_LEN(" (")); > + List_iterator_fast<Create_field> it(sp->m_cols_list); > + > + TABLE table; > + TABLE_SHARE share; > + bzero((char*) &table, sizeof(table)); > + bzero((char*) &share, sizeof(share)); > + table.in_use= thd; > + table.s = &share; > + Create_field *cols_field; > + cols_field= it++; > + while (cols_field) > + { > + uint unused1= 0; > + sp_prepare_create_field(thd, cols_field); > + prepare_create_field(cols_field, &unused1, HA_CAN_GEOMETRY); > + uint field_length; > + Field *field; > + String tmp_result(512); > + field_length= cols_field->length; > + cols_field->flags= 0; > + field= make_field(table.s, /* TABLE_SHARE ptr */ > + (uchar*) 0, /* field ptr */ > + field_length, /* field [max] length */ > + (uchar*) "", /* null ptr */ > + 0, /* null bit */ > + cols_field->pack_flag, > + cols_field->sql_type, > + cols_field->charset, > + cols_field->geom_type, cols_field->srid, > + Field::NONE, /* unreg check */ > + cols_field->interval, > + cols_field->field_name); > + field->vcol_info= cols_field->vcol_info; > + field->stored_in_db= cols_field->stored_in_db; > + if (field) > + field->init(&table); > + defstr.append(cols_field->field_name); > + defstr.append(STRING_WITH_LEN(" ")); > + field->sql_type(tmp_result); > + defstr.append(tmp_result); > + cols_field= it++; > + if(cols_field) > + defstr.append(STRING_WITH_LEN(", ")); > + delete field; > + } > + defstr.append(STRING_WITH_LEN(")")); > + char saved_cur_db_name_buf[SAFE_NAME_LEN+1]; > + LEX_STRING saved_cur_db_name= { saved_cur_db_name_buf, > + sizeof(saved_cur_db_name_buf) }; > + newlex.create_info.DDL_options_st::init(); > + Stored_program_creation_ctx *creation_ctx; > + char definer_user_name_holder[USERNAME_LENGTH + 1]; > + LEX_STRING definer_user_name= { definer_user_name_holder, USERNAME_LENGTH }; > + char definer_host_name_holder[HOSTNAME_LENGTH + 1]; > + LEX_STRING definer_host_name= { definer_host_name_holder, HOSTNAME_LENGTH }; > + newlex.current_select= NULL; > + Parser_state parser_state; > + thd->variables.sql_mode= sql_mode; > + err= parser_state.init(thd, defstr.c_ptr_safe(), defstr.length()); > + if (!err) { > + lex_start(thd); > + err= parse_sql(thd, &parser_state, creation_ctx) || thd->lex == NULL; > + } > + if (!err) > + { > + Table_specification_st create_info(thd->lex->create_info); > + Alter_info alter_info(thd->lex->alter_info, thd->mem_root); > + thd->lex->create_info.default_table_charset= NULL; > + SELECT_LEX *select_lex= &thd->lex->select_lex; > + TABLE_LIST *create_table= select_lex->table_list.first; > + if (append_file_to_dir(thd, &create_info.data_file_name, > + create_table->table_name) || > + append_file_to_dir(thd, &create_info.index_file_name, > + create_table->table_name)) > + { > + err= 1; > + } > + if (!(create_info.used_fields & HA_CREATE_USED_ENGINE)) > + create_info.use_default_db_type(thd); > + > + err= mysql_create_table(thd, create_table, &create_info, &alter_info); > + thd->variables.option_bits|= OPTION_KEEP_LOG; > + } > + thd->variables.sql_mode= saved_mode; > + lex_end(thd->lex); > + thd->lex= old_lex; > + } > + tables = tables->next_local; > + } > + return err; > +} > \ No newline at end of file > diff --git a/sql/sql_parse.h b/sql/sql_parse.h > index 76e1b7a..6b89b35 100644 > --- a/sql/sql_parse.h > +++ b/sql/sql_parse.h > @@ -121,7 +121,7 @@ bool push_new_name_resolution_context(THD *thd, > Item *normalize_cond(Item *cond); > Item *negate_expression(THD *thd, Item *expr); > bool check_stack_overrun(THD *thd, long margin, uchar *dummy); > - > +bool check_and_process_table_functions(THD *thd, TABLE_LIST *table_list); > /* Variables */ > > extern const char* any_db; > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index d7d7d7b..dd1f9f6 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -10802,7 +10802,6 @@ table_factor: > Yacc_state *state= & thd->m_parser_state->m_yacc; > if (state->table_factor_part_type== 2) > { > - mysql_init_select(lex); > List_iterator_fast<Item> it(lex->value_list); > Item *item; > while ((item = it++)) > @@ -10817,9 +10816,6 @@ table_factor: > MYSQL_YYABORT; > sel->add_joined_table($$); > lex->pop_context(); > - > - my_error(ER_NOT_SUPPORTED_YET, MYF(0), > - "Queries using Table Functions"); //to be removed later > } > else > { -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 2430d1d: MDEV-7821 - Server crashes in Item_func_group_concat::fix_fields on 2nd
by Oleksandr Byelkin 31 Jul '15

31 Jul '15

Hi! On 30.07.15 16:59, Sergey Vojtovich wrote: > revision-id: 2430d1d7ccfcf429e45dbfe655a24d57a79ebf6c (mariadb-5.5.44-14-g2430d1d) > parent(s): 392df76bc3a40a5dd1956b12628dd6489a37be36 > committer: Sergey Vojtovich > timestamp: 2015-07-30 18:51:44 +0400 > message: > > MDEV-7821 - Server crashes in Item_func_group_concat::fix_fields on 2nd > execution of PS > > GROUP_CONCAT() with ORDER BY column position may crash server on PS reexecution. > > The problem was that arguments array of GROUP_CONCAT() was adjusted to point to > temporary elements (resolved ORDER BY fields) during first execution. > > This patch expands rev. 08763096cb to restore original arguments array as well. OK to push! > > --- > mysql-test/r/func_gconcat.result | 16 ++++++++++++++++ > mysql-test/t/func_gconcat.test | 11 +++++++++++ > sql/item_sum.cc | 3 +++ > 3 files changed, 30 insertions(+) > > diff --git a/mysql-test/r/func_gconcat.result b/mysql-test/r/func_gconcat.result > index f12a0c1..0bc31a5 100644 > --- a/mysql-test/r/func_gconcat.result > +++ b/mysql-test/r/func_gconcat.result > @@ -1103,3 +1103,19 @@ ORDER BY field; > field > c,c > drop table t3, t2, t1; > +# > +# MDEV-7821 - Server crashes in Item_func_group_concat::fix_fields on 2nd > +# execution of PS > +# > +CREATE TABLE t1(a INT); > +INSERT INTO t1 VALUES(1),(2); > +PREPARE stmt FROM "SELECT GROUP_CONCAT(t1a.a ORDER BY 1, t1a.a=0) FROM t1 AS t1a, t1 AS t1b GROUP BY t1a.a"; > +EXECUTE stmt; > +GROUP_CONCAT(t1a.a ORDER BY 1, t1a.a=0) > +1,1 > +2,2 > +EXECUTE stmt; > +GROUP_CONCAT(t1a.a ORDER BY 1, t1a.a=0) > +1,1 > +2,2 > +DROP TABLE t1; > diff --git a/mysql-test/t/func_gconcat.test b/mysql-test/t/func_gconcat.test > index 42a3076..5550eeb 100644 > --- a/mysql-test/t/func_gconcat.test > +++ b/mysql-test/t/func_gconcat.test > @@ -821,3 +821,14 @@ FROM ( SELECT * FROM t2 ) AS sq2, t3 > ORDER BY field; > > drop table t3, t2, t1; > + > +--echo # > +--echo # MDEV-7821 - Server crashes in Item_func_group_concat::fix_fields on 2nd > +--echo # execution of PS > +--echo # > +CREATE TABLE t1(a INT); > +INSERT INTO t1 VALUES(1),(2); > +PREPARE stmt FROM "SELECT GROUP_CONCAT(t1a.a ORDER BY 1, t1a.a=0) FROM t1 AS t1a, t1 AS t1b GROUP BY t1a.a"; > +EXECUTE stmt; > +EXECUTE stmt; > +DROP TABLE t1; > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index d8970ca..a24307b 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -3300,6 +3300,8 @@ void Item_func_group_concat::cleanup() > from Item_func_group_concat::setup() to point to runtime > created objects, we need to reset them back to the original > arguments of the function. > + > + The very same applies to args array. > */ > ORDER **order_ptr= order; > for (uint i= 0; i < arg_count_order; i++) > @@ -3307,6 +3309,7 @@ void Item_func_group_concat::cleanup() > (*order_ptr)->item= &args[arg_count_field + i]; > order_ptr++; > } > + memcpy(args, orig_args, sizeof(Item *) * arg_count); > DBUG_VOID_RETURN; > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] [MDEV-8554] Weird behavior of multi-table UPDATE and THD::save_prep_leaf_list
by Sergey Petrunia 30 Jul '15

30 Jul '15

Hello Igor, I was debugging https://mariadb.atlassian.net/browse/MDEV-8554 The patch is about save_prep_leaf_list is not being cleaned up by a statement. I committed a patch that makes that bug go away. However, the way that statements work with THD::save_prep_leaf_list look weird to me. Since it was you who added save_prep_leaf_list, I wanted to ask you: Consider an example: CREATE TABLE t1 (a INT) ENGINE=MyISAM; INSERT INTO t1 VALUES (1),(2); # Not necessary, the table can be empty CREATE TABLE t2 (b INT) ENGINE=MyISAM; INSERT INTO t2 VALUES (3),(4); # Not necessary, the table can be empty CREATE TABLE t3 (c INT) ENGINE=MyISAM; INSERT INTO t3 VALUES (5),(6); # Not necessary, the table can be empty CREATE OR REPLACE ALGORITHM=MERGE VIEW v3 AS SELECT * FROM t3; Now, let's debug how THD::save_prep_leaf_list changes and is used: UPDATE t1, t2 SET a = 1 WHERE a IN ( SELECT 0 FROM v3 ); We see: 1. Breakpoint 4, THD::THD (this=0x59ff6f0) at /home/psergey/dev-git/10.0/sql/sql_class.cc:1062 save_prep_leaf_list= FALSE; 2. Breakpoint 1, THD::reset_for_next_command (this=0x59ff6f0) at /home/psergey/dev-git/10.0/sql/sql_parse.cc:6272 thd->save_prep_leaf_list= false; ^^^ I have added this line in my fix 3. Breakpoint 5, LEX::save_prep_leaf_tables (this=0x5a02f80) at /home/psergey/dev-git/10.0/sql/sql_lex.cc:4132 bool LEX::save_prep_leaf_tables() { if (!thd->save_prep_leaf_list) return FALSE; ^^^ This is where save_prep_leaf_list is used. 4. Breakpoint 2, mysql_derived_merge (thd=0x59ff6f0, lex=0x5a02f80, derived=0x7fff68008618) at /home/psergey/dev-git/10.0/sql/sql_derived.cc:379 if (thd->lex->sql_command == SQLCOM_UPDATE_MULTI || thd->lex->sql_command == SQLCOM_DELETE_MULTI) thd->save_prep_leaf_list= TRUE; ^^^ this is where save_prep_leaf_list is changed by this statement. 5. (no break). The statement finished. That is, the statement first checks save_prep_leaf_list, and then uses it. This looks weird. Any thoughts? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 9f8a458: MDEV-7846: Server crashes in Item_subselect::fix_fields or fails with Thread stack overrun
by Sergei Golubchik 30 Jul '15

30 Jul '15

Hi, Sanja! On Apr 23, sanja(a)mariadb.com wrote: > revision-id: 9f8a458fb2d07298810bb5d9824ce728017bfb32 > parent(s): e540d023e2ec6f37efc9ab695ccdfd4a6744ad64 > committer: Oleksandr Byelkin > branch nick: server > timestamp: 2015-04-23 20:08:57 +0200 > message: > > MDEV-7846: Server crashes in Item_subselect::fix_fields or fails with > Thread stack overrun > > Substitute into transformed subselects original left expression and > than register its change in case it was substituted. Looks mostly ok to me. But please: 1. Add comments (the patch is attached) 2. Add a test case with row items 3. In your test case - if you remove GROUP BY you get a crash again. This might be a different bug, though. Regards, Sergei

1 0

[Maria-developers] mysql_stmt_prepare_start returns with error code 2006
by András Szabó 29 Jul '15

29 Jul '15

Hi Folks, I am currently working on a c++11 asio based async connector for mariadb. The library takes advantage of mariadb’s async interface (mysql_*_start, mysql_*_cont functions). The whole point would be to prevent blocking in mysql operations and make operations cancellable via a reasonable interface. Some example code: TEST_CASE("tmp3") { init_db(); asio::io_service io; amdb::connector conn(io); bool saved = false; entity e(42, 3.14, "our entity"); amdb::auth_info auth("root"); conn.async_connect(amdb::null_endpoint, "edbtest", auth, amdb::default_flags, [&] (const std::error_code& ec) { REQUIRE(!ec); conn.async_write_entity(e, [&] (const std::error_code &ec) { INFO(ec); REQUIRE(!ec); if (!ec) { saved = true; } io.stop(); }); }); io.run(); REQUIRE(saved); } async_write_entity() is an extra feature of the lib, which is generating an sql “batch” of prepared statements to persist hierarchical objects and executes that batch. Unfortunately I am receiving an error 2006 from my very first mysql_stmt_prepare call. This error roughly translates to "connection to mysql server is gone”. I cannot really spot the issue with my operation, could someone give me a hint about this error (typical causes, what to check, etc etc.)? I am a bit clueless here how to tackle this problem. Since the batch generation/execution code is a bit convoluted and complex, i would not paste it here now, but if someone can point me in the right direction i can share relevant parts. Thanks in advance Andras -- András Szabó Sent with Airmail

2 2

Re: [Maria-developers] 4728581: MDEV-7652 - More explanatory ERROR and WARNING messages when loading plugins
by Sergei Golubchik 29 Jul '15

29 Jul '15

Hi, Sergey! Looks ok, just a couple of comments about the messages. If you're agree - please, fix and push. On Jul 28, Sergey Vojtovich wrote: > revision-id: 4728581813000522d05ab589df8e5f8cb6ad1ee6 > parent(s): cf30074c3ff6815d85bbd864b28adfe7949d340c > committer: Sergey Vojtovich > branch nick: mariadb Please, update the git post-commit hook (bzr-pull from mariadb-tools). I've just changed it to report a branch better than just "mariadb". > timestamp: 2015-07-28 10:18:55 +0400 > message: > > MDEV-7652 - More explanatory ERROR and WARNING messages when loading plugins > with plugin-load-add that are already registered at mysql.plugin > > - issue just one error message, without this extra warning > - don't abuse ER_UDF_EXISTS, instead add a proper error message for plugins > - report started initialization for each plugin source > > --- > sql/share/errmsg-utf8.txt | 3 +++ > sql/sql_plugin.cc | 15 +++++++++++---- > 2 files changed, 14 insertions(+), 4 deletions(-) > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 6954170..374e31f 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7111,3 +7111,6 @@ ER_SLAVE_SKIP_NOT_IN_GTID > eng "When using GTID, @@sql_slave_skip_counter can not be used. Instead, setting @@gtid_slave_pos explicitly can be used to skip to after a given GTID position." > ER_TABLE_DEFINITION_TOO_BIG > eng "The definition for table %`s is too big" > +ER_PLUGIN_EXISTS > + eng "Plugin '%-.192s' already exists" > + rus "Плагин '%-.192s' уже существует" May be "already installed" ? The command is INSTALL PLUGIN. For UDF it was CREATE FUNCTION, that's why "already exists", I suppose. > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index dc40870..21efeb0 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -1546,6 +1546,9 @@ int plugin_init(int *argc, char **argv, int flags) > /* > First we register builtin plugins > */ > + if (global_system_variables.log_warnings >= 9) > + sql_print_information("Initializing mandatory plugins"); > + > for (builtins= mysql_mandatory_plugins; *builtins || mandatory; builtins++) > { > if (!*builtins) Here you should've put "Initializing optional plugins". Actually, a better message would be "Initializing other built-in plugins" or " Initializing remaining built-in plugins". > @@ -1627,11 +1630,17 @@ int plugin_init(int *argc, char **argv, int flags) > { > I_List_iterator<i_string> iter(opt_plugin_load_list); > i_string *item; > + if (global_system_variables.log_warnings >= 9) > + sql_print_information("Initializinig plugins specified by command line"); "Initializinig plugins specified on the command line" > while (NULL != (item= iter++)) > plugin_load_list(&tmp_root, item->ptr); > > if (!(flags & PLUGIN_INIT_SKIP_PLUGIN_TABLE)) > + { > + if (global_system_variables.log_warnings >= 9) > + sql_print_information("Initializing installed plugins"); > plugin_load(&tmp_root); > + } > } Regards, Sergei

2 2

[Maria-developers] Using oqgraph/connect engines in mariadb-galera-server
by Nirbhay Choubey 28 Jul '15

28 Jul '15

Hi Serg, Svoj! I am looking into an issue where accessing oqgraph/connect engines from galera server leads to errors like "Unknown option '<table option>'" and segfaults. The issue being the differences in handlerton structure in galera and non-galera server codes. Since the connect/oqgraph packages are built from non-galera server, when they are loaded into galera server, the engine's handlerton gets improperly initialized. https://mariadb.atlassian.net/browse/MDEV-8240 I can think of the following potential solutions : a) Club connect/oqgraph and mariadb-galera-server into a single package. (this would mean adding new dependencies to mariadb-galera-server) b) New packages for connect/oqgraph out of mariadb-galera build (mariadb-connect-engine-galera-10.0, mariadb-oqgraph-engine-galera-10.0) Do you see any other option? Also, we should add some checks for plugin load to fail in first place in such scenarios. Best, Nirbhay

2 1

[Maria-developers] GTID: On the difference between master_use_gtid = slave_pos or current_pos
by Kristian Nielsen 27 Jul '15

27 Jul '15

I was asked about the difference between master_use_gtid = slave_pos or current_pos. This comes up often, and I thought the below text, which I used to answer the question earlier, might be helpful. Ian, do you think you could incorporate this in some form into the GTID documentation in the knowledgebase? Hope this helps, - Kristian. ----------------------------------------------------------------------- First, consider a slave S2 replicating from a master S1. We stop S2, and then later restart it. Where should it resume replication? Clearly at the point of the last transaction replicated before stopping, which is @@gtid_slave_pos. Now suppose we take down S1 for maintenance and promote S2 as the new master. So we are doing direct transactions on S2. Later we take S1 back up, and now we connect S1 as a slave to S2. Where should S1 start replicating? Clearly at the point of the last transaction in S1's binlog, which is the last transaction made while S1 was the master. This is @@gtid_binlog_pos. Now suppose the user does STOP SLAVE on S1, does a direct transaction on the slave S1, then START SLAVE. That direct transaction was only intended locally, and it is not on the master S2. So when S1 reconnects, it needs to connect at its @@gtid_slave_pos this time, not at the @@gtid_binlog_pos. So we see, only the user knows if a direct transaction made on a server is meant as a cluster-wide transaction, or a local transaction only. And the "correct" behaviour is _different_ depending on this. Normally, direct local transactions in the slave binlog are considered sloppy, because of all the possibilities for errors they introduce. So it is recommended to run slaves in read-only mode, and set @@sql_log_bin=0 if doing local transactions. And gtid_strict_mode=1 is provided to enforce this recommended mode of operation. In this case, user can use current_pos, which will automatically use either @@gtid_slave_pos or @@gtid_binlog_pos, whatever is appropriate. However, some (or many) users like to "be sloppy" and do direct transactions on a slave not intended as master transactions. And for those we provide gtid_strict_mode=0 and master_use_gtid=slave_pos. This makes slave connect always use the position of the last replicated transaction, ignoring any local transactions in the binlog. In this case, when a master is changed to be a slave, it is necessary to manually set @@gtid_slave_pos=@@gtid_binlog_pos. Explicitly marking the updates on the old master to be intended as master transactions, not local transactions. -----------------------------------------------------------------------

2 1

[Maria-developers] PLEASE REVIEW: (MDEV-7574) Security definer views don't work with CONNECT ODBC tables
by Alexander Barkov 26 Jul '15

26 Jul '15

Hello Sergei, Please review a patch that fixes the problem described in MDEV-7574. The idea is that a SELECT from a view over a CONNECT table now checks FILE privileges of the view definer (unless CREATE VIEW states SQL SECURITY INVOKER). It looks like a very good idea and gives more possible security options. The administrator can create a VIEW and give access to it to some user, without giving FILE privilege to this user. I'm not really sure about the patch, but it seems to do the trick :) From what I understood, FILE_ACL is written (among the other privileges) into thd->security_ctx.privilege in TABLE_LIST::prepare_security(). In case of a DEFINER view, thd->security_ctx.privilege is filled exactly with the definer privileges, and to the invoker privileges otherwise. So inside ha_connect::check_privileges() the fact that there is FILE_ACL in thd->security_ctx.privilege means that TABLE_LIST::prepare_security() was previously called and FILE_ACL is set to DEFINER or INVOKER, according to the view definition. This is exactly what we need. I'm not sure about the opposite: if there is no FILE_ACL in thd->security_ctx.privilege, what does it mean? Does it mean that there is no FILE_ACL for the effective user? Or can it also mean that TABLE_LIST::prepare_security() was not called? Thanks. On 02/12/2015 04:04 AM, Olivier Bertrand wrote: > Hi Alexander, > > Can you take care of this issue? I know you wrote the > ha_connect::check_privileges function and I personnally have not the > faintest idea about how this must be checked. > > Thanks, > Olivier > > > -------- Message transféré -------- > Sujet : [JIRA] (MDEV-7574) Security definer views don't work with > CONNECT ODBC tables > Date : Wed, 11 Feb 2015 23:21:00 +0200 (EET) > De : Elena Stepanova (JIRA) <jira(a)mariadb.atlassian.net> > Pour : bertrandop(a)gmail.com > > > > [https://mariadb.atlassian.net/browse/MDEV-7574?page=com.atlassian.jira.plug… ] > > Elena Stepanova reassigned MDEV-7574: > ------------------------------------- > > Assignee: Olivier Bertrand > Fix Version/s: 10.0 > >> Security definer views don't work with CONNECT ODBC tables >> ---------------------------------------------------------- >> >> Key: MDEV-7574 >> URL:https://mariadb.atlassian.net/browse/MDEV-7574 >> Project: MariaDB Server >> Issue Type: Bug >> Components: Storage Engine - Connect >> Affects Versions: 10.0.16 >> Reporter: Geoff Montee >> Assignee: Olivier Bertrand >> Labels: connect-engine >> Fix For: 10.0 >> >> >> One possible way to get around the requirement for having the FILE privilege to access ODBC tables with CONNECT would be to have them called indirectly via a security definer view. However, it does not currently work. >> Create a security definer view to access the ODBC table, then create a new user: >> {code} >> [gmontee@localhost ~]$ mysql -u root tmp >> Reading table information for completion of table and column names >> You can turn off this feature to get a quicker startup with -A >> Welcome to the MariaDB monitor. Commands end with ; or \g. >> Your MariaDB connection id is 16 >> Server version: 10.0.15-MariaDB-log MariaDB Server >> Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others. >> Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. >> MariaDB [tmp]> SHOW CREATE TABLE datetime_table; >> +----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ >> | Table | Create Table | >> +----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ >> | datetime_table | CREATE TABLE `datetime_table` ( >> `id` int(10) NOT NULL, >> `modifiedon` datetime DEFAULT NULL >> ) ENGINE=CONNECT DEFAULT CHARSET=latin1 CONNECTION='DSN=connect_test_azure;UID=connect_test;PWD=Password1' `TABLE_TYPE`='ODBC' `TABNAME`='dbo.datetime_table' | >> +----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ >> 1 row in set (0.00 sec) >> MariaDB [tmp]> DROP USER 'connecttest'@'localhost'; >> Query OK, 0 rows affected (0.00 sec) >> MariaDB [tmp]> CREATE OR REPLACE >> -> DEFINER = CURRENT_USER >> -> SQL SECURITY DEFINER >> -> VIEW datetime_view >> -> AS SELECT * FROM datetime_table; >> Query OK, 0 rows affected (0.00 sec) >> MariaDB [tmp]> CREATE USER 'connecttest'@'localhost'; >> Query OK, 0 rows affected (0.00 sec) >> MariaDB [tmp]> GRANT SELECT ON datetime_view TO 'connecttest'@'localhost'; >> Query OK, 0 rows affected (0.00 sec) >> MariaDB [tmp]> \q >> Bye >> {code} >> Now connect with the new user, and try to use the view: >> {code} >> [gmontee@localhost ~]$ mysql -u connecttest tmp >> Reading table information for completion of table and column names >> You can turn off this feature to get a quicker startup with -A >> Welcome to the MariaDB monitor. Commands end with ; or \g. >> Your MariaDB connection id is 17 >> Server version: 10.0.15-MariaDB-log MariaDB Server >> Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others. >> Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. >> MariaDB [tmp]> SELECT * FROM datetime_view; >> ERROR 1045 (28000): Access denied for user 'connecttest'@'localhost' (using password: NO) >> MariaDB [tmp]> \q >> Bye >> {code} >> It didn't work, so give the user privileges on the underlying ODBC table: >> {code} >> [gmontee@localhost ~]$ mysql -u root tmp >> Reading table information for completion of table and column names >> You can turn off this feature to get a quicker startup with -A >> Welcome to the MariaDB monitor. Commands end with ; or \g. >> Your MariaDB connection id is 18 >> Server version: 10.0.15-MariaDB-log MariaDB Server >> Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others. >> Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. >> MariaDB [tmp]> GRANT FILE ON *.* TO 'connecttest'@'localhost'; >> Query OK, 0 rows affected (0.00 sec) >> MariaDB [tmp]> GRANT SELECT ON datetime_table TO 'connecttest'@'localhost'; >> Query OK, 0 rows affected (0.00 sec) >> MariaDB [tmp]> \q >> Bye >> {code} >> Now try using the view again: >> {code} >> [gmontee@localhost ~]$ mysql -u connecttest tmp >> Reading table information for completion of table and column names >> You can turn off this feature to get a quicker startup with -A >> Welcome to the MariaDB monitor. Commands end with ; or \g. >> Your MariaDB connection id is 19 >> Server version: 10.0.15-MariaDB-log MariaDB Server >> Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others. >> Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. >> MariaDB [tmp]> SELECT * FROM datetime_view; >> +----+---------------------+ >> | id | modifiedon | >> +----+---------------------+ >> | 1 | 2014-01-01 00:00:00 | >> | 2 | 2016-01-01 00:00:00 | >> +----+---------------------+ >> 2 rows in set (0.24 sec) >> {code} > > > > -- > This message was sent by Atlassian JIRA > (v6.4-OD-14-082#64012) > > >

2 9

Re: [Maria-developers] [Commits] 0b82e6d: MDEV-8399 - [PATCH] Missing Sanity Checks for memory allocation in MariaDB
by Sergei Golubchik 23 Jul '15

23 Jul '15

Hi, Sergey! On Jul 16, Sergey Vojtovich wrote: > revision-id: 0b82e6d85a18fb5fa9cc54ad7a854cb2eecfce03 > parent(s): 94bc50650eb490e8d844acfc700f75992857509a > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-07-16 16:20:17 +0400 > message: > > MDEV-8399 - [PATCH] Missing Sanity Checks for memory allocation in MariaDB > > - since param is quite small store it on stack > - fixed a few memory leaks I'm not going to review that, I don't care about missing sanity checks or memory leaks in mysys test programs :) Do whatever you want with it (it = the code inside #ifdef MAIN ... #endif) Regards, Sergei

1 0

Re: [Maria-developers] [Commits] a6ab8ef: MDEV-5997 - MySQL bug#11759114 - '51401: GRANT TREATS NONEXISTENT
by Sergei Golubchik 23 Jul '15

23 Jul '15

Hi, Sergey! On Jul 23, Sergey Vojtovich wrote: > revision-id: a6ab8ef9d7d465a3f0bc1cff6034ca97575629dd > parent(s): d897015d74d82464a21ed154f606adc69503c372 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-07-23 16:45:20 +0400 > message: > > MDEV-5997 - MySQL bug#11759114 - '51401: GRANT TREATS NONEXISTENT > FUNCTIONS/PRIVILEGES DIFFERENTLY' ok to push Regards, Sergei

1 0

Re: [Maria-developers] [Commits] 74b4ead: MDEV-5096 - Wrong error message on attempt to kill somebody else's query ID
by Sergei Golubchik 23 Jul '15

23 Jul '15

Hi, Sergey! On Feb 09, svoj(a)mariadb.org wrote: > revision-id: 74b4ead8ef32eb2bb96b17dafac514ffba28963a > parent(s): f13939061d468a47985dee0268652a6fe2db7862 > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2015-02-09 13:41:24 +0400 > message: > > MDEV-5096 - Wrong error message on attempt to kill somebody else's query ID > > Attempting to kill query owned by different user now returns better error: > "You are not owner of query N" instead of "You are not owner of thread N". Ok to push! Nice test. Regards, Sergei

1 0

[Maria-developers] MariaDB parallel replication
by Kristian Nielsen 18 Jul '15

18 Jul '15

Some interesting work is starting to be done by users to test parallel replication on real workloads. One issue that comes up is the performance on workloads that have a relatively high number of lock conflicts between transactions. I made some patches (against latest 10.1) to help investigate and improve the issues, and I wanted to mention them here in case someone wants to comment on or experiment with them: 1. Adding status variables for parallel replication. http://lists.askmonty.org/pipermail/commits/2015-July/008131.html This patch adds new status variables that measure the time spent by parallel replication worker threads being - Idle (waiting for work from the SQL thread). - Processing relay log events. - Waiting for a prior group commit to finish (measuring the overhead of insufficient parallelism recorded on the master, in conservative mode, or the overhead of serialisation around DDL, in optimistic/aggressive modes). - Waiting for the immediately prior transaction to commit (measuring the overhead of in-order parallel replication). - Rolling back and re-executing events due to deadlocks. It would be interesting to see how the different numbers compare on various workloads and parallel replication modes. 2. More aggressive retry of conflicting transactions. http://lists.askmonty.org/pipermail/commits/2015-July/008133.html When we get a deadlock and a transaction retry, the current MariaDB waits for all prior transactions to commit before retrying. The logic is that since we already got a conflict, there is a high risk that an immediate retry will just give another conflict. So if we have T1 T2 T3 T4, and T4 conflicts with T1, we roll back T4, wait for T3 to commit, and only then retry T4. If we have many such conflicts, we could end up wasting a lot of times on such waits. This patch changes aggressive mode so that T4 will only wait for T1 to commit, then it will retry. This allows T4 to run in parallel with T2 and T3. It will be interesting to see if this improves throughput in aggressive mode in some workloads, and also to see if/how much it increases the number of transaction retries and associated overhead. 3. Debug patch to log all row lock waits. http://lists.askmonty.org/pipermail/commits/2015-July/008132.html This is not a patch suitable for production use. It adds an option --gtid-log-all-lock-conflicts. When enabled, whenever one parallel replication transaction needs to wait on the InnoDB row lock of another, it will log a line to the error log, including the GTIDs of the transactions. This could be used to correlate back with the binlog and study exactly which transactions it is that are conflicting with each other. Such results could again be highly interesting. - Kristian.

1 1

[Maria-developers] MDEV-8441 Bad SHOW CREATE TABLE output for a table with a virtual column
by Alexander Barkov 15 Jul '15

15 Jul '15

Hi Sanja, Please review a patch for MDEV-8441. Thanks.

2 1

[Maria-developers] MDEV-4682 plugin - qc_info - add Query Cache flags - not-GSoc2015 patch for review
by Daniel Black 14 Jul '15

14 Jul '15

This expands out all the query cache info from the flags that wasn't previously available in the information_schema.query_cache_info table by the query_cache_info plugin. https://github.com/MariaDB/server/pull/67 -- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

1 0

Re: [Maria-developers] Likely a dead optimizer related code
by Michael Widenius 13 Jul '15

13 Jul '15

Hi! > Hi Igor, > > I noticed that during Item_cond::fix_fields() and > Item_func_between::fix_fields() update these optimizer > related SELECT_LEX members: > > thd->lex->current_select->cond_count and > thd->lex->current_select->between_count > > The purpose of these members is to allocate optimizer related buffers > in: > - update_ref_and_keys() in sql_select.cc and > - check_func_dependency() in opt_table_elimination,cc > and it seems they have no any other purposes.. Correct. > From my understanding, this is a dead code and the collected values are > never used: > > - cond_count and_between_count are later always initialized to 0 > before the walk(count_sargable_conds) call in update_ref_and_keys(). And the call to check_func_dependency() is done after this call. > - moreover, using the values collected during fix_fields() would not be > correct to allocate optimized related buffers, because the code > in fix_fields() does not distinguish between different query parts, > so the items can be just a part of the SELECT list rather than a > part of WHERE/HAVING/ON. This would note be a problem as the count was supposed to be an upper bound. As it's not normal that you have many IF or BETWEEN in the SELECT part, this was not a problem. I looked at the patch and it's ok to commit. Regards, Monty

2 1

Re: [Maria-developers] Probably in a bug in simple_pred()
by Michael Widenius 09 Jul '15

09 Jul '15

Hi! > From: Alexander Barkov <bar(a)mariadb.org> > Hi Sergey, > > I probably found a bug in simple_pred() in opt_sum.cc. > > In this piece of code: > >> case 3: >> /* field BETWEEN const AND const */ >> item= func_item->arguments()[0]->real_item(); >> if (item->type() == Item::FIELD_ITEM) >> { >> args[0]= item; >> for (int i= 1 ; i <= 2; i++) >> { >> item= func_item->arguments()[i]->real_item(); >> if (!item->const_item()) >> return 0; >> args[i]= item; >> if (check_item1_shorter_item2(args[0], args[1])) > > > Should't it be > > if (check_item1_shorter_item2(args[0], args[i])) > > ? > >> return 0; >> } >> } > > > If this is a bug, can you please help to make an SQL scrip which would > be affected by this bug? yes, it's a bug. Here is a script that shows that: drop table if exists t1; create table t1 (a varchar(10), key (a)); insert into t1 values("bar"),("Cafe"); explain select min(a) from t1 where a between "a" and "Cafe2"; explain select min(a) from t1 where a between "a" and "Cafeeeeeeeeeeeeeeeeeeeeeeeeee"; explain select min(a) from t1 where a between "abbbbbbbbbbbbbbbbbbbb" and "Cafe2"; The output is: 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away | For the two first cases and: | 1 | SIMPLE | t1 | index | a | a | 13 | NULL | 2 | Using where; Using index | for the last The second case should have the same output as the third. Will fix and push into 10.1 Regards, Monty

1 0

[Maria-developers] MDEV-4119 patch comments
by Sergey Petrunia 07 Jul '15

07 Jul '15

Hi Monty, Please find my comments on the patch for MDEV-4119 below. Sorry for the delay. > diff --git a/sql/item.cc b/sql/item.cc > index b4de973..3b1f8b7 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -1705,69 +1705,98 @@ class Item_aggregate_ref : public Item_ref > > @param thd Thread handler > @param ref_pointer_array Pointer to array of reference fields > - @param fields All fields in select > + @param fields All fields in select > @param ref Pointer to item > - @param skip_registered <=> function be must skipped for registered > - SUM items > + @param split_flags Zero or more of the following flags > + SPLIT_FUNC_SKIP_REGISTERED: > + Function be must skipped for registered SUM > + SUM items > + SPLIT_FUNC_SELECT > + We are called on the select level and have to > + register items operated on sum function Typo, it's not SPLIT_FUNC_SELECT, it is SPLIT_SUM_SELECT > > @note > - This is from split_sum_func2() for items that should be split > - > All found SUM items are added FIRST in the fields list and > we replace the item with a reference. > > + If this is an item in the SELECT list then we also have to split out > + all arguments to functions used together with the sum function. > + For example in case of SELECT A*sum(B) we have to split out both > + A and sum(B). > + This is not needed for ORDER BY, GROUP BY or HAVING as all references > + to items in the select list are already of type REF > + > thd->fatal_error() may be called if we are out of memory > */ > > void Item::split_sum_func2(THD *thd, Item **ref_pointer_array, > List<Item> &fields, Item **ref, > - bool skip_registered) > + uint split_flags) > { > - /* An item of type Item_sum is registered <=> ref_by != 0 */ > - if (type() == SUM_FUNC_ITEM && skip_registered && > - ((Item_sum *) this)->ref_by) > - return; > - if ((type() != SUM_FUNC_ITEM && with_sum_func) || > - (type() == FUNC_ITEM && > - (((Item_func *) this)->functype() == Item_func::ISNOTNULLTEST_FUNC || > - ((Item_func *) this)->functype() == Item_func::TRIG_COND_FUNC))) > + if (unlikely(type() == SUM_FUNC_ITEM)) > { > - /* Will split complicated items and ignore simple ones */ > - split_sum_func(thd, ref_pointer_array, fields); > + /* An item of type Item_sum is registered if ref_by != 0 */ > + if ((split_flags & SPLIT_SUM_SKIP_REGISTERED) && > + ((Item_sum *) this)->ref_by) > + return; > } > - else if ((type() == SUM_FUNC_ITEM || (used_tables() & ~PARAM_TABLE_BIT)) && > - type() != SUBSELECT_ITEM && > - (type() != REF_ITEM || > - ((Item_ref*)this)->ref_type() == Item_ref::VIEW_REF)) > + else > { > - /* > - Replace item with a reference so that we can easily calculate > - it (in case of sum functions) or copy it (in case of fields) > - > - The test above is to ensure we don't do a reference for things > - that are constants (PARAM_TABLE_BIT is in effect a constant) > - or already referenced (for example an item in HAVING) > - Exception is Item_direct_view_ref which we need to convert to > - Item_ref to allow fields from view being stored in tmp table. > - */ > - Item_aggregate_ref *item_ref; > - uint el= fields.elements; > - /* > - If this is an item_ref, get the original item > - This is a safety measure if this is called for things that is > - already a reference. > - */ > - Item *real_itm= real_item(); > + /* Not a SUM() function */ > + if (unlikely((!with_sum_func && !(split_flags & SPLIT_SUM_SELECT)))) > + { > + /* > + This is not a SUM function and there are no SUM functions inside. > + Nothing more to do. > + */ > + return; > + } > + if (likely(with_sum_func || > + (type() == FUNC_ITEM && > + (((Item_func *) this)->functype() == > + Item_func::ISNOTNULLTEST_FUNC || > + ((Item_func *) this)->functype() == > + Item_func::TRIG_COND_FUNC)))) > + { > + /* Will call split_sum_func2() for all items */ > + split_sum_func(thd, ref_pointer_array, fields, split_flags); > + return; > + } > > - ref_pointer_array[el]= real_itm; > - if (!(item_ref= new Item_aggregate_ref(&thd->lex->current_select->context, > - ref_pointer_array + el, 0, name))) > - return; // fatal_error is set > - if (type() == SUM_FUNC_ITEM) > - item_ref->depended_from= ((Item_sum *) this)->depended_from(); > - fields.push_front(real_itm); > - thd->change_item_tree(ref, item_ref); > + if (unlikely((!(used_tables() & ~PARAM_TABLE_BIT) || > + type() == SUBSELECT_ITEM || > + (type() == REF_ITEM && > + ((Item_ref*)this)->ref_type() != Item_ref::VIEW_REF)))) > + return; > } > + > + /* > + Replace item with a reference so that we can easily calculate > + it (in case of sum functions) or copy it (in case of fields) > + > + The test above is to ensure we don't do a reference for things > + that are constants (PARAM_TABLE_BIT is in effect a constant) > + or already referenced (for example an item in HAVING) > + Exception is Item_direct_view_ref which we need to convert to > + Item_ref to allow fields from view being stored in tmp table. > + */ > + Item_aggregate_ref *item_ref; > + uint el= fields.elements; > + /* > + If this is an item_ref, get the original item > + This is a safety measure if this is called for things that is > + already a reference. > + */ > + Item *real_itm= real_item(); > + > + ref_pointer_array[el]= real_itm; > + if (!(item_ref= new Item_aggregate_ref(&thd->lex->current_select->context, > + ref_pointer_array + el, 0, name))) > + return; // fatal_error is set > + if (type() == SUM_FUNC_ITEM) > + item_ref->depended_from= ((Item_sum *) this)->depended_from(); > + fields.push_front(real_itm); > + thd->change_item_tree(ref, item_ref); > } > > > diff --git a/sql/item.h b/sql/item.h > index 8254359..44649bf 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -74,6 +74,10 @@ char_to_byte_length_safe(uint32 char_length_arg, uint32 mbmaxlen_arg) > } > > > +/* Bits for the split_sum_func() function */ > +#define SPLIT_SUM_SKIP_REGISTERED 1 /* Skip registered funcs */ > +#define SPLIT_SUM_SELECT 2 /* SELECT item; Split all parts */ > + > /* > "Declared Type Collation" > A combination of collation and its derivation. > @@ -1169,10 +1173,10 @@ class Item: public Type_std_attributes > return false; > } > virtual void split_sum_func(THD *thd, Item **ref_pointer_array, > - List<Item> &fields) {} > + List<Item> &fields, uint flags) {} > /* Called for items that really have to be split */ > void split_sum_func2(THD *thd, Item **ref_pointer_array, List<Item> &fields, > - Item **ref, bool skip_registered); > + Item **ref, uint flags); > virtual bool get_date(MYSQL_TIME *ltime, ulonglong fuzzydate); > bool get_time(MYSQL_TIME *ltime) > { return get_date(ltime, TIME_TIME_ONLY | TIME_INVALID_DATES); } > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index 7b34e64..caa645d 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -4575,12 +4575,13 @@ void Item_cond::traverse_cond(Cond_traverser traverser, > */ > > void Item_cond::split_sum_func(THD *thd, Item **ref_pointer_array, > - List<Item> &fields) > + List<Item> &fields, uint flags) > { > List_iterator<Item> li(list); > Item *item; > while ((item= li++)) > - item->split_sum_func2(thd, ref_pointer_array, fields, li.ref(), TRUE); > + item->split_sum_func2(thd, ref_pointer_array, fields, li.ref(), > + flags | SPLIT_SUM_SKIP_REGISTERED); Why not clear SPLIT_SUM_SELECT flag here? The Item_cond may be at the select list, but its parts are definitely not in the select list? The same applies to Item_row::split_sum_func and Item_func::split_sum_func. > } > > > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index 7d89cf5..ceab597 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -1759,7 +1759,8 @@ class Item_cond :public Item_bool_func > SARGABLE_PARAM **sargables); > SEL_TREE *get_mm_tree(RANGE_OPT_PARAM *param, Item **cond_ptr); > virtual void print(String *str, enum_query_type query_type); > - void split_sum_func(THD *thd, Item **ref_pointer_array, List<Item> &fields); > + void split_sum_func(THD *thd, Item **ref_pointer_array, List<Item> &fields, > + uint flags); > friend int setup_conds(THD *thd, TABLE_LIST *tables, TABLE_LIST *leaves, > COND **conds); > void top_level_item() { abort_on_null=1; } > diff --git a/sql/item_func.cc b/sql/item_func.cc > index 3e78be8..684724b 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -419,11 +419,12 @@ Item *Item_func::compile(Item_analyzer analyzer, uchar **arg_p, > */ > > void Item_func::split_sum_func(THD *thd, Item **ref_pointer_array, > - List<Item> &fields) > + List<Item> &fields, uint flags) > { > Item **arg, **arg_end; > for (arg= args, arg_end= args+arg_count; arg != arg_end ; arg++) > - (*arg)->split_sum_func2(thd, ref_pointer_array, fields, arg, TRUE); > + (*arg)->split_sum_func2(thd, ref_pointer_array, fields, arg, > + flags | SPLIT_SUM_SKIP_REGISTERED); Why not clear SPLIT_SUM_SELECT flag here? (same question as in Item_cond::split_sum_func) > } > > ........ > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 4a50003..ce02484 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -1944,8 +1954,9 @@ int JOIN::init_execution() > /* > Enable LIMIT ROWS EXAMINED during query execution if: > (1) This JOIN is the outermost query (not a subquery or derived table) > - This ensures that the limit is enabled when actual execution begins, and > - not if a subquery is evaluated during optimization of the outer query. > + This ensures that the limit is enabled when actual execution begins, > + and not if a subquery is evaluated during optimization of the outer > + query. > (2) This JOIN is not the result of a UNION. In this case do not apply the > limit in order to produce the partial query result stored in the > UNION temp table. > @@ -2562,8 +2573,12 @@ void JOIN::exec_inner() > skip_sort_order= 0; > } > bool made_call= false; > + SQL_SELECT *select= join_tab[const_tables].select; This line causes a crash in a few tests: main.select_pkeycache main.select main.select_jcl6 main.subselect3 main.subselect3_jcl6 The reason is that execution gets here when the join is degenerate and join_tab==NULL. > if (order && > - (order != group_list || !(select_options & SELECT_BIG_RESULT)) && > + (order != group_list || !(select_options & SELECT_BIG_RESULT) || > + (select && select->quick && > + select->quick->get_type() == QUICK_SELECT_I::QS_TYPE_GROUP_MIN_MAX)) > + && > (const_tables == table_count || > ((simple_order || skip_sort_order) && > (made_call=true) && BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] MDEV-8433 Make field<'broken-string' use indexes
by Alexander Barkov 07 Jul '15

07 Jul '15

Hi Sergei, I created a task according to our discussion a few weeks ago: https://mariadb.atlassian.net/browse/MDEV-8433 Which version should we do this task in? Thanks.

1 0

Re: [Maria-developers] GSoc2015: MDEV-8356 first patch
by Sergey Petrunia 03 Jul '15

03 Jul '15

Hi Diwas, On Fri, Jul 03, 2015 at 02:54:28PM +0530, Diwas Joshi wrote: > hello sergey, I have fixed the issues from the previous patch in this one. > Please find some feedback below. After you have addressed it, please commit the change and push it to your tree. The next task to focus on after this one is MDEV-8389. > diff --git a/mysql-test/r/sp-error.result b/mysql-test/r/sp-error.result > index 897895e..cacfeea 100644 > --- a/mysql-test/r/sp-error.result > +++ b/mysql-test/r/sp-error.result > @@ -2866,12 +2866,6 @@ SELECT @msg; > DROP FUNCTION f1; > DROP FUNCTION f2; > DROP TABLE t1; > -CREATE FUNCTION f1(a INT, b VARCHAR(11)) > -RETURNS TABLE t1(id INT, name VARCHAR(11)) > -BEGIN > -INSERT INTO t1 SELECT id, name FROM t2 WHERE id = a; > -END| > -ERROR 42000: This version of MariaDB doesn't yet support 'Table functions' > SELECT * FROM f1('xyz'); > ERROR 42000: This version of MariaDB doesn't yet support 'Queries using Table Functions' > SELECT * FROM t1, f1('xyz',t1.a); > diff --git a/mysql-test/r/sp.result b/mysql-test/r/sp.result > index 3f28df9..2a931b1 100644 > --- a/mysql-test/r/sp.result > +++ b/mysql-test/r/sp.result > @@ -7937,3 +7937,17 @@ return 1; > end | > ERROR 0A000: Not allowed to return a result set from a function > drop table t1,t2; > +# Tests for Table functions > +CREATE FUNCTION f1(a INT, b VARCHAR(11)) > +RETURNS TABLE t1(id INT, name VARCHAR(11)) > +BEGIN > +INSERT INTO t1 SELECT id, name FROM t2 WHERE id = a; > +END| Please add "--vertical_results" here so that the SELECT results are printed vertically. It will be easier to see what the return value is. > +SELECT * FROM mysql.proc WHERE db = database() and name = 'f1'; > +db name type specific_name language sql_data_access is_deterministic security_type param_list returns body definer created modified sql_mode comment character_set_client collation_connection db_collation body_utf8 > +test f1 FUNCTION f1 SQL CONTAINS_SQL NO DEFINER a INT, b VARCHAR(11) TABLE t1(id int(11), name varchar(11)) BEGIN > +INSERT INTO t1 SELECT id, name FROM t2 WHERE id = a; > +END root@localhost 2015-07-03 12:18:04 2015-07-03 12:18:04 latin1 latin1_swedish_ci latin1_swedish_ci BEGIN > +INSERT INTO t1 SELECT id, name FROM t2 WHERE id = a; > +END > +DROP FUNCTION f1; -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Buildbot branch 10.0-custombld
by Kristian Nielsen 02 Jul '15

02 Jul '15

Hi Rasmus, The branch 10.0-custombld seems to no longer build binaries (after the transition to git?): https://github.com/MariaDB/server/tree/10.0-custombld https://buildbot.askmonty.org/buildbot/grid?branch=10.0-custombld&category=… Can you please fix Buildbot so that this branch works again for producing custom builds of all packages? This is urgent, as it is blocking further progress of MDEV-8302. Thanks, - Kristian.

1 1

Re: [Maria-developers] GSoc2015: MDEV-8356 first patch
by Sergey Petrunia 02 Jul '15

02 Jul '15

Hi Diwas, On Thu, Jul 02, 2015 at 04:26:58PM +0530, Diwas Joshi wrote: > for the return type i am storing a string containing TABLE(<col > definitions>). eg. forCREATE FUNCTION f1(a INT, b VARCHAR(11)) > RETURNS TABLE t1(id INT, name VARCHAR(11)) BEGIN set @a=3; END| > we store > TABLE t1(id INT, name VARCHAR(11)) Ok. > for the type of function we can store it as TYPE_ENUM_FUNCTION only and > later see if it creates a problem, although we can always differentiate > between the functions based on the return type stored above. Yes, let's keep it this way for now. > > I included the table alias with the return type string as you suggested. > There is a warning that i am getting, which i am failing to figure the > reason for but I am still trying. > Please review. > The first thing: please add a testcase that checks what the code code does. I can think of doing this: CREATE table_function .... < whatever here> select * from mysql.proc where name='table_function'; This will show what was stored in the system table. This will also make it clear if/what warning gets produced. Since you've changed CREATE FUNCTION table_function ... not to return error, I guess some of testcases you previously added will need to be modified, too. Overall the patch seems to be ok but I would like to look at the final version of it before giving the approval. > diff --git a/sql/sp.cc b/sql/sp.cc > index 9fd17b0..3413ba1 100644 > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -925,21 +925,66 @@ class Bad_db_error_handler : public Internal_error_handler > bzero((char*) &share, sizeof(share)); > table.in_use= thd; > table.s = &share; > - field= sp->create_result_field(0, 0, &table); > - field->sql_type(result); > - > - if (field->has_charset()) > + if (sp->m_type == TYPE_ENUM_FUNCTION) > { > - result.append(STRING_WITH_LEN(" CHARSET ")); > - result.append(field->charset()->csname); > - if (!(field->charset()->state & MY_CS_PRIMARY)) > + field= sp->create_result_field(0, 0, &table); > + field->sql_type(result); > + > + if (field->has_charset()) > { > - result.append(STRING_WITH_LEN(" COLLATE ")); > - result.append(field->charset()->name); > + result.append(STRING_WITH_LEN(" CHARSET ")); > + result.append(field->charset()->csname); > + if (!(field->charset()->state & MY_CS_PRIMARY)) > + { > + result.append(STRING_WITH_LEN(" COLLATE ")); > + result.append(field->charset()->name); > + } > } > + delete field; > + } > + else if (sp->m_type == TYPE_ENUM_TABLE) > + { > + List_iterator_fast<Create_field> it(sp->m_cols_list); > + Create_field *cols_field; > + cols_field= it++; > + result.append(STRING_WITH_LEN("TABLE ")); > + result.append(sp->m_table_alias.str); > + result.append(STRING_WITH_LEN("(")); > + while (cols_field) > + { > + uint unused1= 0; > + sp_prepare_create_field(thd, cols_field); > + prepare_create_field(cols_field, &unused1, HA_CAN_GEOMETRY); > + uint field_length; > + Field *field; > + String tmp_result(64); > + field_length= cols_field->length; > + cols_field->flags= 0; > + field= make_field(table.s, /* TABLE_SHARE ptr */ > + (uchar*) 0, /* field ptr */ > + field_length, /* field [max] length */ > + (uchar*) "", /* null ptr */ > + 0, /* null bit */ > + cols_field->pack_flag, > + cols_field->sql_type, > + cols_field->charset, > + cols_field->geom_type, cols_field->srid, > + Field::NONE, /* unreg check */ > + cols_field->interval, > + cols_field->field_name); > + field->vcol_info= cols_field->vcol_info; > + field->stored_in_db= cols_field->stored_in_db; > + if (field) > + field->init(&table); > + field->sql_type(tmp_result); > + result.append(tmp_result); > + cols_field= it++; > + if(cols_field) > + result.append(STRING_WITH_LEN(", ")); > + delete field; > + } > + result.append(STRING_WITH_LEN(")")); > } > - > - delete field; > } > > > @@ -1020,7 +1065,8 @@ class Bad_db_error_handler : public Internal_error_handler > char definer_buf[USER_HOST_BUFF_SIZE]; > LEX_STRING definer; > ulonglong saved_mode= thd->variables.sql_mode; > - MDL_key::enum_mdl_namespace mdl_type= type == TYPE_ENUM_FUNCTION ? > + MDL_key::enum_mdl_namespace mdl_type= type == (type == TYPE_ENUM_FUNCTION || > + type == TYPE_ENUM_TABLE) ? > MDL_key::FUNCTION : MDL_key::PROCEDURE; Looks reasonable. > > CHARSET_INFO *db_cs= get_default_db_collation(thd, sp->m_db.str); > @@ -1036,7 +1082,8 @@ class Bad_db_error_handler : public Internal_error_handler > retstr.set_charset(system_charset_info); > > DBUG_ASSERT(type == TYPE_ENUM_PROCEDURE || > - type == TYPE_ENUM_FUNCTION); > + type == TYPE_ENUM_FUNCTION || > + type == TYPE_ENUM_TABLE); > > /* Grab an exclusive MDL lock. */ > if (lock_object_name(thd, mdl_type, sp->m_db.str, sp->m_name.str)) > @@ -1117,7 +1164,8 @@ class Bad_db_error_handler : public Internal_error_handler > > store_failed= store_failed || > table->field[MYSQL_PROC_MYSQL_TYPE]-> > - store((longlong)type, TRUE); > + store((longlong)(type == TYPE_ENUM_TABLE ? > + TYPE_ENUM_FUNCTION : type), TRUE); > > store_failed= store_failed || > table->field[MYSQL_PROC_FIELD_SPECIFIC_NAME]-> > @@ -1145,7 +1193,7 @@ class Bad_db_error_handler : public Internal_error_handler > table->field[MYSQL_PROC_FIELD_PARAM_LIST]-> > store(sp->m_params.str, sp->m_params.length, system_charset_info); > > - if (sp->m_type == TYPE_ENUM_FUNCTION) > + if (sp->m_type == TYPE_ENUM_FUNCTION || sp->m_type == TYPE_ENUM_TABLE) > { > sp_returns_type(thd, retstr, sp); > > @@ -1153,7 +1201,7 @@ class Bad_db_error_handler : public Internal_error_handler > table->field[MYSQL_PROC_FIELD_RETURNS]-> > store(retstr.ptr(), retstr.length(), system_charset_info); > } > - > + String body_with_alias(64); What is this for? It's not used anywhere. > store_failed= store_failed || > table->field[MYSQL_PROC_FIELD_BODY]-> > store(sp->m_body.str, sp->m_body.length, system_charset_info); > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index b53d939..d7d7d7b 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -16392,12 +16392,6 @@ sf_tail: > Lex_input_stream *lip= YYLIP; > > lex->sphead->set_body_start(thd, lip->get_cpp_tok_start()); > - > - if (Lex->sphead->m_type== TYPE_ENUM_TABLE) //to be removed later. > - { > - my_error(ER_NOT_SUPPORTED_YET, MYF(0), "Table functions"); > - MYSQL_YYABORT; > - } > } > sp_proc_stmt /* $15 */ > { BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] GSoC2015: MDEV-8356 patch for review
by Diwas Joshi 02 Jul '15

02 Jul '15

Hello all, I am attaching a patch for MDEV-8356 <https://mariadb.atlassian.net/browse/MDEV-8356> as part of my work under GSoC2015 implementing table valued functions. Please feel free to point out mistakes, if any. regards Diwas Joshi

1 0

[Maria-developers] [GSoC] MDEV-4674 Port InnoDB memcached interface to MariaDB - Midterm report
by Piotr Jurkiewicz 01 Jul '15

01 Jul '15

Hi all, as the midterm is approaching, I am posting a report on what I have done so far. The project is going as scheduled. You can see the timeline in my proposal: http://www.google-melange.com/gsoc/proposal/public/google/gsoc2015/piotrjur… All goals planned before the midterm have been reached. InnoDB Memcached plugin ported from MySQL works now with MariaDB (both with InnoDB and XtraDB engines). That means that MariaDB is now on par with MySQL on this issue. My repository with the code is available here: https://github.com/piotrjurkiewicz/mariadb-server What have been done in details: 1. Code of InnoDB Memcached plugin has been imported from MySQL into the MariaDB tree. 2. There were some differences between MariaDB and MySQL, which blocked the plugin from out-of-the-box compilation and correct operation. They have been resolved. 3. Test suite for the plugin has been imported from MySQL. Compatibility changes were introduced into these test cases which required them. The test suite was modified to assign memcached port number dynamically from mtr worker's pool, what allows to run the same test in parallel for many combinations. The plugin fully passes the test suite, both for 'innodb_plugin' and 'xtradb' combinations. 4. Some enhancements and bug fixes have been introduced into the plugin: - detection of memcached daemon initialization has been fixed -- this eliminated unnecessary wait (15 seconds) on plugin shutdown and memory leak (in cases when plugin failed to initialize properly before) - memcached arithmetic commands (incr/decr) handling has been improved -- now numeric columns values are handled in a consistent and predictable way - incorrect handling of unsigned not null integers columns has been fixed For details of all changes please see commit messages associated with them: https://github.com/piotrjurkiewicz/mariadb-server/commits/10.1-memcached After the midterm, I plan to focus on further enhancements of the plugin, as stated in my proposal: 1. Change plugin architecture to "one table per socket", as originally proposed in MDEV-4674 bug report. This will allow usage of well known security solutions (firewalls, unix file permissions) to restrict access to different tables for different users. 2. Implement a new interface inside the Memcached plugin, which will use Handler API for database access. This will make possible to use the plugin with database engines other than InnoDB/XtraDB. Ultimately, this interface will replace existing InnoDB direct API interface (if there will be no significant performance differences). Therefore, I would like to as you to point me to Handler API documentation (if documentation such exists). I know that HandlerSocket plugin uses the Handler API to access databases. Is HandlerSocket plugin a good example of Handler API usage to follow? Piotr

1 0

[Maria-developers] How to read of innodb_locks table by using low level interface?
by 김경남 01 Jul '15

01 Jul '15

Hi, Sergei I'm developing a mariadb monitoring plugin type agent. We're planning to contribute our monitoring solution for mariadb in one or two months. Our solution is consist of plugin type agent, APM Server and Dashboard. Our plugin type agent collect data by using low level interface instead of using query. I can collect a information_schema.global_status and performance_schema.event_xxxx_table for your help in the past. By referencing the source of feedback.cc, I can collect data from global_status using fill_stauts function. I hava a question. I need to collect data from information_schema.innodb_locks for lock monitoring. But I found that I must use trx_i_s_common_fill_table in i_s.cc for reading of innodb_locks table and that fuction is static. so I only access that function by function pointer. Debugging source for many days, I guess that I must use ST_SCHEMA TABLE's fill_table member variable for pointing to trx_i_s_common_fille_table function. Right? But I don't know how to that. I attemped in various ways for many days. but I failed. If possible. please let me know how to read data of information_schema.innodb_locks using native api. Thanks in advance. Regards, Sergei. _____ 상기 메시지와 첨부화일 내에는 비밀정보가 포함되어 있을 수 있으며, 지정된 수신자에 한하여 조회 및 사용될 수 있습니다. 만약 송신자의 실수로 인하여 상기 메시지를 수신하였다면, 송신자에게 메시지를 반송해 주시고, 원본 메시지와 모든 사본을 폐기해 주시기 바랍니다. 상기 메시지의 전체 또는 일부에 대해 무단 열람, 사용, 공개, 배포하는 것은 금지되어 있습니다.(주)LG CNS. This message and its attachments may contain confidential information, and they are intended to be viewed or used by only the individuals specified in the message. If you have received this message in an error from the sender, please contact the sender immediately to notify the error and delete all of the message and its copies. It is prohibited to view, use, make public and/or distribute part or whole of this message without written permission.

1 0

[Maria-developers] sprint tasks for the next week
by Alexander Barkov 01 Jul '15

01 Jul '15

Hi Sergei, I'd like to do these tasks: - MDEV-8215 Asian MB3 charsets: compare broken bytes as "greater than any non-broken character" - MDEV-8036 Fix all collations to compare broken bytes as "greater than any non-broken character" to close the group of tasks related to handling bad bytes in comparison. Any objections? How do I mark tasks as "sprint"? Thanks.

1 0

Re: [Maria-developers] MDEV-8356 analysis and ideas
by Sergey Petrunia 29 Jun '15

29 Jun '15

Hi Diwas, Nice analysis. Please find my comments below. On Mon, Jun 29, 2015 at 10:01:56AM +0530, Diwas Joshi wrote: > hello sergey, so i was reading further on MDEV-8356 last night. There are > two main things here: type of sf and return type of sf. Right now the > create function... is giving errors at > > store_failed= store_failed || > table->field[MYSQL_PROC_MYSQL_TYPE]-> > store((longlong)type, TRUE); > > in sf_create_routine(), which is right since we have been been using > TYPE_ENUM_TABLE for m_type so far and mysql.proc takes > enum('FUNCTION','PROCEDURE') for type. This can be solved by either > changing this enum in mysql.proc definition and including TABLE, or storing > table_valued functions as FUNCTION as well. I think first option is better, > including TABLE in the enum, since this will help us in distinguishing > these functions while executing later if needed. It is not clear for me which option is better. On one hand, there is your argument that adding 'TABLE' into the enum will make it easier to tell table functions from regular function. On the other hand, try grepping for 'TYPE_ENUM_FUNCTION' in sql/*.{h,cc}. You find a lot of code that checks if m_type == TYPE_ENUM_FUNCTION. You will need to review each check to see what should be done for TYPE_ENUM_FUNCTION. If we can't decide towards one option or another, let's use whatever you prefer right now. If we discover that that works poorly, we will be able to reverse the decision later on. > The second thing is about return type of function. Right now > sp_returns_type() method is being used to store the return type of > functions. But in our case we have a list of create_fields: > List<Create_field> m_cols_list keeping list of attribute names and their > types for the table. We need to store all these, for this we can either > alter mysql.proc definition to make a field appropriate for storing this > list. The type of the field storing return type right now is longblob which > as i read further on it is a binary large object that can hold a variable > amount of data and are treated as binary strings (byte strings). Maybe we > can try to find a way to store all the attribute information in this field > only, maybe putting in some delimiters between different field types or > some other way since after all its a string. I am still not sure about this > problem. I tried creating this function: CREATE FUNCTION f2(a INT, b VARCHAR(11)) RETURNS VARCHAR(32) DETERMINISTIC BEGIN set @a=3; RETURN @a; END| and mysql.proc.returns has: returns: varchar(32) CHARSET latin1 So, it is a text defition of the return type. It seems natural if for table functions we Is it a problem to do similar thing for Table functions? e.g for a function CREATE FUNCTION f1(a INT, b VARCHAR(11)) RETURNS TABLE t1(id INT, name VARCHAR(11)) BEGIN set @a=3; END| store 'TABLE t1(id INT, name VARCHAR(11)' ? I guess it depends on where the value of mysql.proc.returns is used. Please research this. > > Also, another thing is to store the table alias m_table_alias, because > queries inside the function will be using it. There is no field present > specifically for this in mysql.proc. So either we can create one. There is > a field specific_name in it, as i saw in the code is right now storing the > same thing as name field. So maybe we can use this field to store the table > alias. I would prefer that m_table_alias was stored as part of the expression in mysql.proc.returns field. Can we try that option first? If we figure that can't work, we can think of other options. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB multi-source replication testing at Booking.com
by Karoly Nagy 29 Jun '15

29 Jun '15

Dear MariaDB developers, I'm Károly Nagy working for Booking.com currently testing multi-source replication functionality of MariaDB. Kristian suggested I should reach out to you on this mailing list regarding my questions. We're seeing very high and fluctuating mutex contentions while replicating from two sources (Oracle MySQL 5.6) to a single MariaDB slave. You can see that on the graphs below. The spin waits are relatively [1] aligned but the mutex rounds [2] are 5-10 times higher than it is on the two sources combined together and not consistent. The sources have a relatively constant pattern while the target has dips around 2.5k and spikes up to 8k. The os waits are in completely different order of magnitude [3]. The scenario where values were captured: * Multi-source target is replicating the full dataset of `source 2` and a subset of `source 1` (the hot data) - MariaDB 10.0.16 * Both sources are MySQL 5.6 being part of their replication chain as slaves with log_slave_updates * Source 2 is in normal mode - Oracle MySQL 5.6.17 * Source 1 is catching up from a 1 day replication delay - Oracle MySQL 5.6.24 * All the slaves are warm having the buffer pool fully populated Is this behavior expected? Could you give us some insights on why we could see these results? If there is any more information you need please let me know. Thank you for your help in advance! Every metric is on /10 seconds basis. [1] Mutex spin waits [2] Mutex rounds [3] Mutex OS waits Best regards, -- Károly Nagy System engineer Booking.com <http://booking.com/> BV Rembrandt Square Office, Herengracht 597, 1017 CE Amsterdam Direct +31 (0)20 715 8403

3 4

Re: [Maria-developers] [Commits] 132935a: MDEV-8339 - Server crash during table cache eviction
by Sergei Golubchik 27 Jun '15

27 Jun '15

Hi, Sergey! On Jun 25, svoj(a)mariadb.org wrote: > revision-id: 132935a3ef6578f5fabc765ff6ac966cd3798819 > parent(s): 1f4a89b10ca8df61ec51fcc87e1203dc72579759 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-25 14:21:16 +0400 > message: > > MDEV-8339 - Server crash during table cache eviction > > TDC_element::free_tables_back() had pre-lfhash leftover code, which referenced > TDC_element::free_tables via TDC_element::share. This is not correct as share > may be NULL (newly inserted or to be removed), instead access free_tables > directly. ok to push Regards, Sergei

1 0

[Maria-developers] MDEV-8214 Asian MB2 charsets: compare broken bytes as "greater than any non-broken character"
by Alexander Barkov 26 Jun '15

26 Jun '15

Hi Sergei, For simplicity I decided to split: MDEV-8036 Fix all collations to compare broken bytes as "greater than any non-broken character" into sub-tasks. Please review a patch for the first sub-task MDEV-8214. The overall plan is: - MDEV-8214, for the Asian charsets with mbmaxlen=2 - MDEV-8215, for the Asian charsets with mbmaxlen=3, based on MDEV-8214 - MDEV-8036, the rest (Unicode character sets) Thanks.

2 4

[Maria-developers] MDEV-7502 Automatic provisioning of slave - error reporting
by Martin Kaluznik 26 Jun '15

26 Jun '15

Hello Kristian, with general idea working (sending of database / table structures + table data), my next plan is to improve / introduce error handling in existing code along with error handling on slave side (ddl events not from provisioning, ignore errors when updating not yet existing rows, ...). What are general guidelines for introduction of new error codes? Is there anything else I need to do except adding it to /sql/share/errmsg-utf8.txt ? I would create only one new error code - ER_PROVISIONING_FAILED - and place specific messages inside formatted text - same way as ER_MASTER_FATAL_ERROR_READING_BINLOG works. It doesn't go well with translations, but... you can decide. Maybe create different error codes at least for errors, where we are expecting user to fix them - table without index, statement mode. Also, how to tell slave from master that provisioning finished? Is introduction of new error code ER_PROVISIONING_DONE acceptable solution? The error itself would never produce error message in logs. Alternatively I can reuse / create similar event to Stop_log_event, note - provisioning is completed after SQL thread processes whole relay log, not when IO thread receives this event, so even reuse would require additional code. And side question, is there any log event other than Query_log_event which can produce ddl statement? Martin

2 1

Re: [Maria-developers] [Commits] 2cf747e: MDEV-8219: enforce_storage_engine cannot be set globally
by Sergei Golubchik 26 Jun '15

26 Jun '15

Hi, Jan! On Jun 26, Jan Lindström wrote: > revision-id: 2cf747e885603122b2adbefc131516cd12dc953c > parent(s): d3b7eb7b99ef7dae79c20b5beb63acdfdf6d8046 > committer: Jan Lindström > branch nick: 10.1-bugs > timestamp: 2015-06-26 08:56:50 +0300 > message: > > MDEV-8219: enforce_storage_engine cannot be set globally > > Change session only variable enforce_storage_engine to be > session variable and make sure that also global value > is used if session variable is not set. Ok to push, thanks! Regards, Sergei

1 0

[Maria-developers] Please review: MDEV-7824 [Bug #68041] Zero date can be inserted in strict no-zero mode through a default value
by Alexander Barkov 25 Jun '15

25 Jun '15

Hi Sergei, Please review a patch for mdev-7824. It's based on a MySQL patch for http://bugs.mysql.com/bug.php?id=68041 and is a blocker for: MDEV-3929 Add full support for auto-initialized/updated timestamp and datetime Thanks.

2 9

[Maria-developers] MDEV-6877 - binlog_row_image implementation advice
by Vicențiu Ciorbaru 25 Jun '15

25 Jun '15

Hi Sergei! This past week I've done quite a bit of reading regarding the replication code and how the binlog is used by both mysqlbinlog and the slaves in order to do replication. I think I now have a good enough understanding of the flow of the code for both MySQL and MariaDB to actually start coding the feature into MariaDB. I do have a couple of unknowns still. I don't quite understand what's the use for the TABLE_MAP event. My intuition tells me its some sort of id we assign to a table name (test.t1 for example) that gets used afterwards in the binlog row events. We seem to be using a bitmap of columns that we want to log into the binlog. MySQL does extend the TABLE class to flip the bits according to the binlog_row_image variable. That seems like the first place to start with the addition. I could not find any other "hidden" logic that we use to choose how we log row operations in the binlog. What I would like to know is if there are any other pitfalls that I am supposed to watch out for, or if there is anything else I may be missing regarding the logging part. When it comes to actually doing the replication, I concluded that Log_event::do_apply_event is the place where we begin translating the event from the binlog to an SQL statement. Another concern that I have is how to test the implementation. Right now, I run the test, get the binlog file and print it with mysqlbinlog and see if the output is what I expect. Do we have something within mysql-test-run that automates this? Regards, Vicentiu

2 7

Re: [Maria-developers] [Commits] 9f00950: MDEV-7631 : Invalid WSREP_SST rows appear in mysqld-bin.index file
by Sergei Golubchik 25 Jun '15

25 Jun '15

Hi, Nirbhay! On Jun 24, Nirbhay Choubey wrote: > revision-id: 9f00950d2ad9be5533e66a5fa87a795a9574585f > parent(s): 0f4478105f5027ce589a0f08c6019d906d632029 > committer: Nirbhay Choubey > branch nick: 10.0-galera.b7631 > timestamp: 2015-06-24 23:25:22 -0400 > message: > > MDEV-7631 : Invalid WSREP_SST rows appear in mysqld-bin.index file > > Try not to release 0 (STDIN_FILENO) file descriptor as it can > incorrectly get reused by streams opened later. This really deserved a more descriptive comment. There's nothing wrong when fd 0 is reused by other streams. The problem was that because of a bug (arguably) in old glibc in CentOS6/RHEL6 freopen remapped stdout/stderr to fd 0 and 1. Which is, on itself, is not a problem either, but when mysqld forked other processes, these children had no access to stdout and stderr. it's too late to recommit now and, luckily, MDEV-7631 has all the information about this issue. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] aff5606: Add close-on-exec flag to open(), socket(), accept() & fopen().
by Sergei Golubchik 24 Jun '15

24 Jun '15

Hi, Nirbhay! On Jun 24, Nirbhay Choubey wrote: > revision-id: aff560696cac5e081d19f4460ccd9c906ddc9d84 > parent(s): 70714d3597ec9ffa742cc9e173d4ad32f294cc51 > committer: Nirbhay Choubey > branch nick: 10.0-galera.b7631 > timestamp: 2015-06-24 11:58:11 -0400 > message: > > Add close-on-exec flag to open(), socket(), accept() & fopen(). > > @@ -1026,7 +1026,8 @@ static inline void inline_mysql_socket_register( > (&state, socket_listen.m_psi, PSI_SOCKET_CONNECT, (size_t)0, src_file, src_line); > > /* Instrumented code */ > - socket_accept.fd= accept(socket_listen.fd, addr, &addr_length); > + socket_accept.fd= accept4(socket_listen.fd, addr, &addr_length, > + SOCK_CLOEXEC); Note that, according to the manpage, "accept4() is a nonstandard Linux extension". It needs a configure-time check and ifdef. Alternatively, don't use it at all, do fcntl(FD_CLOEXEC) instead. > /* Instrumentation end */ > if (locker != NULL) > @@ -1036,7 +1037,8 @@ static inline void inline_mysql_socket_register( > #endif > { > /* Non instrumented code */ > - socket_accept.fd= accept(socket_listen.fd, addr, &addr_length); > + socket_accept.fd= accept4(socket_listen.fd, addr, &addr_length, > + SOCK_CLOEXEC); > } Regards, Sergei

2 1

Re: [Maria-developers] [Commits] 99ca8b9: MDEV-8219: enforce_storage_engine cannot be set globally
by Sergei Golubchik 24 Jun '15

24 Jun '15

Hi, Jan! On Jun 23, Jan Lindström wrote: > revision-id: 99ca8b9ea8ee1b756fbf1edc567f6d06c9cf63fd > parent(s): d3b7eb7b99ef7dae79c20b5beb63acdfdf6d8046 > committer: Jan Lindström > branch nick: 10.1-bugs > timestamp: 2015-06-23 13:14:46 +0300 > message: > > MDEV-8219: enforce_storage_engine cannot be set globally > > Change session only variable enforce_storage_engine to be > session variable and make sure that also global value > is used if session variable is not set. > > diff --git a/mysql-test/suite/sys_vars/r/sysvars_innodb.result b/mysql-test/suite/sys_vars/r/sysvars_innodb.result > index 927cd05..1c59391 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_innodb.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_innodb.result > @@ -355,6 +355,20 @@ NUMERIC_BLOCK_SIZE 1048576 > ENUM_VALUE_LIST NULL > READ_ONLY YES > COMMAND_LINE_ARGUMENT REQUIRED > +VARIABLE_NAME INNODB_BUF_DUMP_STATUS_FREQUENCY > +SESSION_VALUE NULL > +GLOBAL_VALUE 0 > +GLOBAL_VALUE_ORIGIN COMPILE-TIME > +DEFAULT_VALUE 0 > +VARIABLE_SCOPE GLOBAL > +VARIABLE_TYPE BIGINT UNSIGNED > +VARIABLE_COMMENT A number between [0, 100] that tells how oftern buffer pool dump status in percentages should be printed. E.g. 10 means that buffer pool dump status is printed when every 10% of number of buffer pool pages are dumped. Default is 0 (only start and end status is printed). please, no unrelated changes. If you want to fix forgotten result files, do it in a separate commit. > +NUMERIC_MIN_VALUE 0 > +NUMERIC_MAX_VALUE 100 > +NUMERIC_BLOCK_SIZE 0 > +ENUM_VALUE_LIST NULL > +READ_ONLY NO > +COMMAND_LINE_ARGUMENT REQUIRED > VARIABLE_NAME INNODB_BUF_FLUSH_LIST_NOW > SESSION_VALUE NULL > GLOBAL_VALUE OFF > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 954970d..f18e6a2 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -9772,7 +9772,7 @@ static bool check_engine(THD *thd, const char *db_name, > DBUG_ENTER("check_engine"); > handlerton **new_engine= &create_info->db_type; > handlerton *req_engine= *new_engine; > - handlerton *enf_engine= thd->variables.enforced_table_plugin ? plugin_hton(thd->variables.enforced_table_plugin) : NULL; > + handlerton *enf_engine= thd->variables.enforced_table_plugin ? plugin_hton(thd->variables.enforced_table_plugin) : global_system_variables.enforced_table_plugin ? plugin_hton(global_system_variables.enforced_table_plugin) : NULL; Please, no. Global variable should not be used like that, it simply sets a default value for the session variable for new connections. The way you've done it now, one cannot disable enforced_table_plugin per session at all. > bool no_substitution= thd->variables.sql_mode & MODE_NO_ENGINE_SUBSTITUTION; > *new_engine= ha_checktype(thd, req_engine, no_substitution); Regards, Sergei

1 0

[Maria-developers] GSoc2015
by Diwas Joshi 24 Jun '15

24 Jun '15

hii, please find a patch for MDEV-8343 <https://mariadb.atlassian.net/browse/MDEV-8343> attached to this mail. I have also included a test case in the patch. regards Diwas Joshi

2 2

[Maria-developers] Can't find messagefile '/usr/share/mysql/errmsg.sys'
by Sylvain Raybaud 24 Jun '15

24 Jun '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I built mariadb galera cluster 10.0.17 from source. Starting it with mysql_safe fails with: Can't find messagefile '/usr/share/mysql/errmsg.sys' This file does not exist indeed. It seems to not exist at all in mariadb-galera-10.0.17.tar.gz. Is it supposed to be installed separately? Am I missing something? Regards, - -- Sylvain Raybaud www.green-communications.fr -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJVeAYZAAoJEEkkwl4JtJ9yvfoP/jTnvj7V1Ze+E7sFE7Rz4Hsm +4Ga8f9MKCr5VxBZky3wMb8EgWmfcWDHbKqu4httTMH2RS6CCv7aiBeRfrMHMW5I NHl+k9CPUrDs03n4Vg12G2M33IIRfdEBgYf2xZ3sp3kBJ07MUWZPvlVL6ch+Aybn oOVzQ1rbUpxrkX2gj4XZ/wtXmy8vN6s9L3jy+o5HU4JZJcFyp1IpCwCE0OneI81V EiHs8dbpsUstr/32TaQ7gCBN9Hb25i5aVwRCD7UXE6kFt6OD6ifv8mw5KfmdtjHc i/0OotnOOsfj7tgDXKoPOltauGdSADJm2VSp8/nsfd6MR2nxMGZcw/Xbhj5l4FkY jWZf7uRAiKsYlmZEaJlb0WZrCTGSe6pcO82hObhv/vD0O55gKCdOM2rAExRiuxFn IPAuKGAKhzegxXxmmBTQv09tjSY3WTNZfFbDKGDQe4EYmCJ0K14cq37gC+TYI+xb Jr9SI5WBAvVVrCvhRktloEf03jVqXMH3H03DBjxGDB9KRc3Vr6AGhQmg0ixC2ZHC uTbRcx+R5mEn40TVwzLEnsUfiV+0MuJ5YYM5brYYRmXRg4ipwTcts1byKTHpvy69 PrDF1mzStInfex2ntkmw7puX8CQ4uebCLzHWSPODDa1n66D1vaEYCbZV7hEg25dH ox0Iptm4REWGyxgNeQxB =0Ydk -----END PGP SIGNATURE-----

3 4

[Maria-developers] doubt about query cache and write locks
by Roberto Spadim 24 Jun '15

24 Jun '15

hi guys, i'm with a doubt when i execute LOCK TABLE xxx WRITE any query at query cache could be executed without locking (waiting for table lock state) shouldn't the lock table invalidade the table query cache when using a write lock? -- Roberto Spadim

2 2

[Maria-developers] crash - analyze table with histogram
by Roberto Spadim 24 Jun '15

24 Jun '15

Hi guys, i got a crash, could someone help? --- queries: set histogram_size=255,histogram_type=SINGLE_PREC_HB; ANALYZE TABLE 19_org.estoque_itens PERSISTENT FOR ALL; /* Erro SQL (2013): Lost connection to MySQL server during query */ --- log: 150623 12:31:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.20-MariaDB-log key_buffer_size=2097152000 read_buffer_size=6291456 max_used_connections=22 max_threads=92 thread_count=9 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3180295 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f523b81c008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x4b1410a0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xbcb2ee] sql/signal_handler.cc:153(handle_fatal_signal)[0x71fa8c] /lib/libpthread.so.0[0x7f52ccbe98f0] sql/field.h:703(Field::set_notnull(long long))[0x610c70] sql/sql_statistics.cc:533(Stat_table::update_stat())[0x60dd68] sql/sql_admin.cc:759(mysql_admin_table)[0x67adc1] sql/sql_admin.cc:1183(Sql_cmd_analyze_table::execute(THD*))[0x67c66c] sql/sql_parse.cc:5095(mysql_execute_command(THD*))[0x5986c2] sql/sql_parse.cc:6529(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59fc86] sql/sql_parse.cc:1310(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x5a1bb7] sql/sql_parse.cc:999(do_command(THD*))[0x5a22f9] sql/sql_connect.cc:1378(do_handle_one_connection(THD*))[0x66c7d4] sql/sql_connect.cc:1295(handle_one_connection)[0x66c912] /lib/libpthread.so.0[0x7f52ccbe1fe7] /lib/libc.so.6(clone+0x6d)[0x7f52cbb822bd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f5190884020): is an invalid pointer Connection ID (thread ID): 304 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. -- Roberto Spadim

2 2

Re: [Maria-developers] [Commits] 6050ab6: MDEV-6829 : SELinux/AppArmor policies for Galera server
by Daniel Black 23 Jun '15

23 Jun '15

Nice work. https://mariadb.atlassian.net/browse/MDEV-7637 has some netlink_audit_socket rules that don't appear to be here. Recommend contributing the selinux component to https://github.com/TresysTechnology/refpolicy which distros usually develop their policies from. Does this work for galera multicast? It appears to only allow tcp bind here. note for readme semanage permissive -a mysqld_t - less of a change for enabling just that domain to be permissive. Does any of https://mariadb.com/kb/en/mariadb/what-to-do-if-mariadb-doesnt-start/ need changing? ----- On 18 Jun, 2015, at 11:59 PM, Nirbhay Choubey nirbhay(a)mariadb.com wrote: > revision-id: 6050ab658696925f2a031b901eb398fff65fa92a > parent(s): 9eff9ed5c58e782abf383a52a7e691a55b4798a2 > committer: Nirbhay Choubey > branch nick: 5.5-galera > timestamp: 2015-06-18 09:59:09 -0400 > message: > > MDEV-6829 : SELinux/AppArmor policies for Galera server > > Add SELinux policy and AppArmor profile under policy/. > > --- > policy/apparmor/README | 5 ++ > policy/apparmor/usr.sbin.mysqld | 150 ++++++++++++++++++++++++++++++++++ > policy/apparmor/usr.sbin.mysqld.local | 4 + > policy/selinux/README | 18 ++++ > policy/selinux/mariadb-server.fc | 10 +++ > policy/selinux/mariadb-server.te | 91 +++++++++++++++++++++ > 6 files changed, 278 insertions(+) > > diff --git a/policy/apparmor/README b/policy/apparmor/README > new file mode 100644 > index 0000000..271655f > --- /dev/null > +++ b/policy/apparmor/README > @@ -0,0 +1,5 @@ > +Note: The included AppArmor profiles can be used for MariaDB Galera cluster. > +However, since these profiles had been tested for a limited set of scenarios, > +it is highly recommended to run them in "complain" mode and report any denials > +on mariadb.org/jira. > + > diff --git a/policy/apparmor/usr.sbin.mysqld b/policy/apparmor/usr.sbin.mysqld > new file mode 100644 > index 0000000..307872c > --- /dev/null > +++ b/policy/apparmor/usr.sbin.mysqld > @@ -0,0 +1,150 @@ > +# Last Modified: Fri Mar 1 18:55:47 2013 > +# Based on usr.sbin.mysqld packaged in mysql-server in Ubuntu. > +# This AppArmor profile has been copied under BSD License from > +# Percona XtraDB Cluster, along with some additions. > + > +#include <tunables/global> > + > +/usr/sbin/mysqld flags=(complain) { > + #include <abstractions/base> > + #include <abstractions/mysql> > + #include <abstractions/nameservice> > + #include <abstractions/user-tmp> > + #include <abstractions/winbind> > + > + capability chown, > + capability dac_override, > + capability setgid, > + capability setuid, > + capability sys_rawio, > + capability sys_resource, > + > + network tcp, > + > + /bin/dash rcx, > + /dev/dm-0 r, > + /etc/gai.conf r, > + /etc/group r, > + /etc/hosts.allow r, > + /etc/hosts.deny r, > + /etc/ld.so.cache r, > + /etc/mtab r, > + /etc/my.cnf r, > + /etc/mysql/*.cnf r, > + /etc/mysql/*.pem r, > + /etc/mysql/conf.d/ r, > + /etc/mysql/conf.d/* r, > + /etc/nsswitch.conf r, > + /etc/passwd r, > + /etc/services r, > + /run/mysqld/mysqld.pid w, > + /run/mysqld/mysqld.sock w, > + /sys/devices/system/cpu/ r, > + owner /tmp/** lk, > + /tmp/** rw, > + /usr/lib/mysql/plugin/ r, > + /usr/lib/mysql/plugin/*.so* mr, > + /usr/sbin/mysqld mr, > + /usr/share/mysql/** r, > + /var/lib/mysql/ r, > + /var/lib/mysql/** rwk, > + /var/log/mysql.err rw, > + /var/log/mysql.log rw, > + /var/log/mysql/ r, > + /var/log/mysql/* rw, > + /var/run/mysqld/mysqld.pid w, > + /var/run/mysqld/mysqld.sock w, > + > + > + profile /bin/dash flags=(complain) { > + #include <abstractions/base> > + #include <abstractions/bash> > + #include <abstractions/mysql> > + #include <abstractions/nameservice> > + #include <abstractions/perl> > + > + > + > + /bin/cat rix, > + /bin/dash rix, > + /bin/date rix, > + /bin/grep rix, > + /bin/nc.openbsd rix, > + /bin/netstat rix, > + /bin/ps rix, > + /bin/rm rix, > + /bin/sed rix, > + /bin/sleep rix, > + /bin/tar rix, > + /bin/which rix, > + /dev/tty rw, > + /etc/ld.so.cache r, > + /etc/my.cnf r, > + /proc/ r, > + /proc/*/cmdline r, > + /proc/*/fd/ r, > + /proc/*/net/dev r, > + /proc/*/net/if_inet6 r, > + /proc/*/net/tcp r, > + /proc/*/net/tcp6 r, > + /proc/*/stat r, > + /proc/*/status r, > + /proc/sys/kernel/pid_max r, > + /proc/tty/drivers r, > + /proc/uptime r, > + /proc/version r, > + /sbin/ifconfig rix, > + /sys/devices/system/cpu/ r, > + /tmp/** rw, > + /usr/bin/cut rix, > + /usr/bin/dirname rix, > + /usr/bin/gawk rix, > + /usr/bin/innobackupex rix, > + /usr/bin/mysql rix, > + /usr/bin/perl rix, > + /usr/bin/seq rix, > + /usr/bin/wsrep_sst* rix, > + /usr/bin/wsrep_sst_common r, > + /usr/bin/xtrabackup* rix, > + /var/lib/mysql/ r, > + /var/lib/mysql/** rw, > + /var/lib/mysql/*.log w, > + /var/lib/mysql/*.err w, > + > +# MariaDB additions > + ptrace peer=@{profile_name}, > + > + /bin/hostname rix, > + /bin/ip rix, > + /bin/mktemp rix, > + /bin/ss rix, > + /bin/sync rix, > + /bin/touch rix, > + /bin/uname rix, > + /etc/mysql/*.cnf r, > + /etc/mysql/conf.d/ r, > + /etc/mysql/conf.d/* r, > + /proc/*/attr/current r, > + /proc/*/fdinfo/* r, > + /proc/*/net/* r, > + /proc/locks r, > + /proc/sys/net/ipv4/ip_local_port_range r, > + /run/mysqld/mysqld.sock rw, > + /sbin/ip rix, > + /usr/bin/basename rix, > + /usr/bin/du rix, > + /usr/bin/find rix, > + /usr/bin/lsof rix, > + /usr/bin/my_print_defaults rix, > + /usr/bin/mysqldump rix, > + /usr/bin/pv rix, > + /usr/bin/rsync rix, > + /usr/bin/socat rix, > + /usr/bin/tail rix, > + /usr/bin/timeout rix, > + /usr/bin/xargs rix, > + /usr/bin/xbstream rix, > + } > + # Site-specific additions and overrides. See local/README for details. > + #include <local/usr.sbin.mysqld> > +} > diff --git a/policy/apparmor/usr.sbin.mysqld.local > b/policy/apparmor/usr.sbin.mysqld.local > new file mode 100644 > index 0000000..a0b8a02 > --- /dev/null > +++ b/policy/apparmor/usr.sbin.mysqld.local > @@ -0,0 +1,4 @@ > +# Site-specific additions and overrides for usr.sbin.mysqld.. > +# For more details, please see /etc/apparmor.d/local/README. > +# This AppArmor profile has been copied under BSD License from > +# Percona XtraDB Cluster, along with some additions. > diff --git a/policy/selinux/README b/policy/selinux/README > new file mode 100644 > index 0000000..a8c11c7 > --- /dev/null > +++ b/policy/selinux/README > @@ -0,0 +1,18 @@ > +Note: The included SELinux policy files can be used for MariaDB Galera cluster. > +However, since these policies had been tested for a limited set of scenarios, > +it is highly recommended to run SELinux in "permissive" mode even with these > +policies installed and report any denials on mariadb.org/jira. > + > + > +How to generate and load the policy module of MariaDB Galera cluster ? > + > +* Generate the SELinux policy module. > + # cd <source>/policy/selinux/ > + # make -f /usr/share/selinux/devel/Makefile mariadb-server.pp > + > +* Load the generated policy module. > + # semodule -i /path/to/mariadb-server.pp > + > +* Lastly, run the following command to allow 4568. > + # semanage port -a -t mysqld_port_t -p tcp 4568 > + > diff --git a/policy/selinux/mariadb-server.fc b/policy/selinux/mariadb-server.fc > new file mode 100644 > index 0000000..1a69ecc > --- /dev/null > +++ b/policy/selinux/mariadb-server.fc > @@ -0,0 +1,10 @@ > +# This SELinux file contexts (.fc) file has been copied under BSD License from > +# Percona XtraDB Cluster. > + > +/etc/init\.d/rc\.d/mysql -- > gen_context(system_u:object_r:mysqld_initrc_exec_t,s0) > +/var/lib/mysql/.*\.log -- gen_context(system_u:object_r:mysqld_log_t,s0) > +/var/lib/mysql/.*\.err -- gen_context(system_u:object_r:mysqld_log_t,s0) > +/var/lib/mysql/.*\.pid -- gen_context(system_u:object_r:mysqld_var_run_t,s0) > +/var/lib/mysql/.*\.cnf -- gen_context(system_u:object_r:mysqld_etc_t,s0) > +/usr/bin/xtrabackup.* -- gen_context(system_u:object_r:mysqld_exec_t,s0) > +/usr/bin/wsrep.* -- gen_context(system_u:object_r:mysqld_safe_exec_t,s0) > diff --git a/policy/selinux/mariadb-server.te b/policy/selinux/mariadb-server.te > new file mode 100644 > index 0000000..9c0319c > --- /dev/null > +++ b/policy/selinux/mariadb-server.te > @@ -0,0 +1,91 @@ > +# This SELinux type enforcement (.te) file has been copied under BSD License > +# from Percona XtraDB Cluster, along with some additions. > + > +module mariadb-server 1.0; > + > +require { > + type user_tmp_t; > + type kerberos_port_t; > + type mysqld_safe_t; > + type tmp_t; > + type tmpfs_t; > + type hostname_exec_t; > + type ifconfig_exec_t; > + type sysctl_net_t; > + type proc_net_t; > + type port_t; > + type mysqld_t; > + type var_lib_t; > + type rsync_exec_t; > + type bin_t; > + type shell_exec_t; > + type anon_inodefs_t; > + type fixed_disk_device_t; > + class lnk_file read; > + class process { getattr signull }; > + class unix_stream_socket connectto; > + class capability { sys_resource sys_nice }; > + class tcp_socket { name_bind name_connect }; > + class file { execute setattr read create getattr execute_no_trans write ioctl > open append unlink }; > + class sock_file { create unlink getattr }; > + class blk_file { read write open }; > + class dir { write search getattr add_name read remove_name open }; > + > +# MariaDB additions > + type tram_port_t; > + class process setpgid; > + class netlink_tcpdiag_socket { create nlmsg_read }; > +} > + > + > +#============= mysqld_safe_t ============== > +allow mysqld_safe_t mysqld_t:process signull; > +allow mysqld_safe_t self:capability { sys_resource sys_nice }; > +allow mysqld_safe_t tmp_t:file { create read write open getattr unlink ioctl > setattr }; > +allow mysqld_safe_t tmp_t:dir { write remove_name add_name }; > +allow mysqld_safe_t tmp_t:sock_file { getattr unlink }; > +allow mysqld_safe_t user_tmp_t:sock_file { getattr unlink }; > +allow mysqld_safe_t var_lib_t:dir { write add_name }; > +allow mysqld_safe_t var_lib_t:file { write ioctl setattr create open getattr > append unlink }; > + > +#============= mysqld_t ============== > +allow mysqld_t anon_inodefs_t:file write; > +allow mysqld_t tmp_t:sock_file { create unlink }; > +allow mysqld_t tmpfs_t:dir { write search read remove_name open add_name }; > +allow mysqld_t tmpfs_t:file { write getattr read create unlink open }; > +allow mysqld_t fixed_disk_device_t:blk_file { read write open }; > +allow mysqld_t ifconfig_exec_t:file { read execute open execute_no_trans > getattr }; > + > +#This rule allows connecting on 4444 > +allow mysqld_t kerberos_port_t:tcp_socket { name_bind name_connect }; > + > +allow mysqld_t mysqld_safe_t:dir { getattr search }; > +allow mysqld_t mysqld_safe_t:file { read open }; > +allow mysqld_t self:unix_stream_socket connectto; > +allow mysqld_t port_t:tcp_socket { name_bind name_connect }; > +allow mysqld_t proc_net_t:file { read getattr open }; > +allow mysqld_t sysctl_net_t:dir search; > +allow mysqld_t var_lib_t:file { getattr open append }; > +allow mysqld_t var_lib_t:sock_file { create unlink getattr }; > +allow mysqld_t rsync_exec_t:file { read getattr open execute execute_no_trans > }; > +allow mysqld_t self:process getattr; > +allow mysqld_t hostname_exec_t:file { read getattr execute open > execute_no_trans }; > +allow mysqld_t user_tmp_t:dir { write add_name }; > +allow mysqld_t user_tmp_t:file create; > +allow mysqld_t bin_t:lnk_file read; > +allow mysqld_t tmp_t:file { append create read write open getattr unlink > setattr }; > + > +# Allows too much leeway - the xtrabackup/wsrep rules in fc should fix it, but > +# keep for the moment. > +allow mysqld_t shell_exec_t:file { execute_no_trans getattr read execute open > }; > +allow mysqld_t bin_t:file { getattr read execute open execute_no_trans ioctl }; > + > +# MariaDB additions > +allow mysqld_t self:process setpgid; > +# This rule allows port 4567 > +allow mysqld_t tram_port_t:tcp_socket name_bind; > + > +# Rules related to XtraBackup > +allow mysqld_t self:netlink_tcpdiag_socket { create nlmsg_read }; > +allow mysqld_t sysctl_net_t:file { read getattr open }; > + > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

2 1

[Maria-developers] Likely a dead optimizer related code
by Alexander Barkov 23 Jun '15

23 Jun '15

Hi Igor, I noticed that during Item_cond::fix_fields() and Item_func_between::fix_fields() update these optimizer related SELECT_LEX members: thd->lex->current_select->cond_count and thd->lex->current_select->between_count The purpose of these members is to allocate optimizer related buffers in: - update_ref_and_keys() in sql_select.cc and - check_func_dependency() in opt_table_elimination,cc and it seems they have no any other purposes.. From my understanding, this is a dead code and the collected values are never used: - cond_count and_between_count are later always initialized to 0 before the walk(count_sargable_conds) call in update_ref_and_keys(). - moreover, using the values collected during fix_fields() would not be correct to allocate optimized related buffers, because the code in fix_fields() does not distinguish between different query parts, so the items can be just a part of the SELECT list rather than a part of WHERE/HAVING/ON. Is the attached patch correct? All tests pass. Thanks.

1 1

Re: [Maria-developers] [Commits] d3b7eb7: MDEV-7528 GIS: Functions return NULL instead of specified -1 for NULL arguments.
by Sergei Golubchik 23 Jun '15

23 Jun '15

Hi, Holyfoot! On Jun 23, holyfoot(a)askmonty.org wrote: > revision-id: d3b7eb7b99ef7dae79c20b5beb63acdfdf6d8046 > parent(s): 3e4126e9d6b21d2f330a365b0847999346485863 > committer: Alexey Botchkov > branch nick: server > timestamp: 2015-06-23 11:57:05 +0500 > message: > > MDEV-7528 GIS: Functions return NULL instead of specified -1 for NULL arguments. > The behaviour required by the standard seems too weird to expect. If the function can never return NULL, please set maybe_null accordingly. Regards, Sergei

1 0

[Maria-developers] GSOC [Adding Redis interface to XtraDB]
by Aalekh Nigam 22 Jun '15

22 Jun '15

Hello, I am Aalekh Nigam, working as a GSOC Summer Strudent for the project "Adding Redis interace to XtraDB". Please follow my blog, for weekly update on my project here: http://mimdl.tumblr.com/ Thank you Aalekh

1 0

[Maria-developers] MXS-2: Week 4
by sriram patil 22 Jun '15

22 Jun '15

Hi All, Please find the link to my blog post of fourth week below. <http://srirampatil.tumblr.com/post/122086304290/mxs-2-week-4> MXS-2: Week 4 <http://srirampatil.tumblr.com/post/122086304290/mxs-2-week-4> Feel free to point out mistakes, if any. Also, suggestions ans/or reviews are welcome! Thanks, Sriram

1 0

Re: [Maria-developers] [Commits] 95a99d8: MDEV-8303; Dumping buffer pool noisy in the logs.
by Daniel Black 20 Jun '15

20 Jun '15

Will only get used if documented: https://mariadb.com/kb/en/mariadb/xtradbinnodb-server-system-variables/ + https://mariadb.com/kb/en/mariadb/server-system-variables/ ----- On 18 Jun, 2015, at 6:45 PM, Jan Lindström jan.lindstrom(a)mariadb.com wrote: > revision-id: 95a99d8dd96be43369d5be9cc23735a1ce86674a > parent(s): b94eaff8d90a60b9dea3b5b68fbde3d35563127b > committer: Jan Lindström > branch nick: 10.1-encryption > timestamp: 2015-06-18 08:38:33 +0300 > message: > > MDEV-8303; Dumping buffer pool noisy in the logs. > > Added new dynamic configuration variable innodb_buf_dump_status_frequency > to configure how often buffer pool dump status is printed in the logs. > A number between [0, 100] that tells how oftern buffer pool dump status > in percentages should be printed. E.g. 10 means that buffer pool dump > status is printed when every 10% of number of buffer pool pages are > dumped. Default is 0 (only start and end status is printed). -- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

2 1

Re: [Maria-developers] [Commits] d1f5dfe: MDEV-8030 - Apc_target::disable() locks mutex twice
by Sergey Petrunia 19 Jun '15

19 Jun '15

Hi Sergei, Ok to push. I'm sorry for the delay with this. On Wed, May 20, 2015 at 08:09:36AM +0000, svoj(a)mariadb.org wrote: > revision-id: d1f5dfe67703a22f017eb25dd48b1eb5246a9c54 > parent(s): 80333ad847d9f1708fa02fb47a976e96f014bc50 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-05-20 12:04:32 +0400 > message: > > MDEV-8030 - Apc_target::disable() locks mutex twice > > Moved Apc_target::destroy(), Apc_target::enable() and Apc_targe::disable() > definitions to my_apc.h so that they can be inlined. > > Apc_targe::disable() now calls Apc_target::process_apc_requests() only if > there're APC requests. This saves one pthread_mutex_lock() call. > > Overhead change: > Apc_target::disable 0.04% -> out of radar > Apc_target::enable 0.03% -> out of radar > Apc_target::process_apc_requests 0.02% -> out of radar > pthread_mutex_lock 0.43% -> 0.42% > pthread_mutex_unlock 0.26% -> 0.25% > > --- > sql/my_apc.cc | 39 --------------------------------------- > sql/my_apc.h | 27 +++++++++++++++++++++++---- > 2 files changed, 23 insertions(+), 43 deletions(-) > > diff --git a/sql/my_apc.cc b/sql/my_apc.cc > index 1766068..91f5cd3 100644 > --- a/sql/my_apc.cc > +++ b/sql/my_apc.cc > @@ -41,45 +41,6 @@ void Apc_target::init(mysql_mutex_t *target_mutex) > } > > > -/* > - Destroy the target. The target must be disabled when this call is made. > -*/ > -void Apc_target::destroy() > -{ > - DBUG_ASSERT(!enabled); > -} > - > - > -/* > - Enter ther state where the target is available for serving APC requests > -*/ > -void Apc_target::enable() > -{ > - /* Ok to do without getting/releasing the mutex: */ > - enabled++; > -} > - > - > -/* > - Make the target unavailable for serving APC requests. > - > - @note > - This call will serve all requests that were already enqueued > -*/ > - > -void Apc_target::disable() > -{ > - bool process= FALSE; > - DBUG_ASSERT(enabled); > - mysql_mutex_lock(LOCK_thd_data_ptr); > - if (!(--enabled)) > - process= TRUE; > - mysql_mutex_unlock(LOCK_thd_data_ptr); > - if (process) > - process_apc_requests(); > -} > - > - > /* [internal] Put request qe into the request list */ > > void Apc_target::enqueue_request(Call_request *qe) > diff --git a/sql/my_apc.h b/sql/my_apc.h > index dfeef5e..20b1ee4 100644 > --- a/sql/my_apc.h > +++ b/sql/my_apc.h > @@ -50,10 +50,29 @@ class Apc_target > ~Apc_target() { DBUG_ASSERT(!enabled && !apc_calls);} > > void init(mysql_mutex_t *target_mutex); > - void destroy(); > - void enable(); > - void disable(); > - > + > + /* Destroy the target. The target must be disabled when this call is made. */ > + void destroy() { DBUG_ASSERT(!enabled); } > + > + /* Enter ther state where the target is available for serving APC requests */ > + void enable() { enabled++; } > + > + /* > + Make the target unavailable for serving APC requests. > + > + @note > + This call will serve all requests that were already enqueued > + */ > + void disable() > + { > + DBUG_ASSERT(enabled); > + mysql_mutex_lock(LOCK_thd_data_ptr); > + bool process= !--enabled && have_apc_requests(); > + mysql_mutex_unlock(LOCK_thd_data_ptr); > + if (unlikely(process)) > + process_apc_requests(); > + } > + > void process_apc_requests(); > /* > A lightweight function, intended to be used in frequent checks like this: > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] GSoC2015: facing errors working on udf's for table valued functions.
by Diwas Joshi 19 Jun '15

19 Jun '15

hii i have been stuck on some shift/reduce conflict since past few days. It is giving the error for the code i have added for select query using the udf. I think the rule that I defined under table_factor is conflicting with the first rule defined under it. regards

2 1

[Maria-developers] mariadb 10.1.5 crash
by Roberto Spadim 19 Jun '15

19 Jun '15

hi guys, i was testing mariadb with test servers and i hit a crash could anyone check? server has 8GB of ram, i don't know what query is 2015-06-17 14:01:32 140497706510080 [Note] WSREP: Stop replication 2015-06-17 14:01:32 140497706510080 [Note] Event Scheduler: Purging the queue. 0 events 2015-06-17 14:01:32 140497706510080 [Note] /usr/mariadb/mariadb/bin//mysqld: Shutdown complete 150617 14:01:32 mysqld_safe mysqld from pid file /home/mysql/pid/local.pid ended 150617 14:02:38 mysqld_safe Starting mysqld daemon with databases from /home/mysql/data/local/ 2015-06-17 14:02:38 140260849645440 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release. 2015-06-17 14:02:38 140260849645440 [Note] /usr/mariadb/mariadb/bin//mysqld (mysqld 10.1.5-MariaDB-log) starting as process 2321 ... 2015-06-17 14:02:38 140260849645440 [Note] Plugin 'InnoDB' is disabled. 2015-06-17 14:02:38 140260849645440 [Note] Plugin 'FEEDBACK' is disabled. 2015-06-17 14:02:38 140260849645440 [Note] Server socket created on IP: '::'. 2015-06-17 14:02:38 140260849645440 [Note] Event Scheduler: Loaded 0 events 2015-06-17 14:02:38 140260849645440 [Note] WSREP: Read nil XID from storage engines, skipping position init 2015-06-17 14:02:38 140260849645440 [Note] WSREP: wsrep_load(): loading provider library 'none' 2015-06-17 14:02:38 140260849645440 [Note] /usr/mariadb/mariadb/bin//mysqld: ready for connections. Version: '10.1.5-MariaDB-log' socket: '/tmp/mysql.BH.sock' port: 3307 MariaDB Server 150617 17:02:14 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.1.5-MariaDB-log key_buffer_size=629145600 read_buffer_size=4194304 max_used_connections=23 max_threads=102 thread_count=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1452041 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f90d0353008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f9103539e80 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0x7f9106c50b0b] sql/signal_handler.cc:153(handle_fatal_signal)[0x7f91067eadb5] /lib64/libpthread.so.0(+0x10d00)[0x7f910537dd00] sql/sql_plist.h:181(I_P_List_iterator)[0x7f910679cf03] mysys/lf_hash.c:125(lfind)[0x7f9106c5a1a0] sql/table_cache.cc:1084(tdc_iterate(THD*, char (*)(void*, void*), void*, bool))[0x7f910679e4e3] sql/table_cache.cc:267(tc_add_table(THD*, TABLE*))[0x7f910679e808] sql/sql_base.cc:2572(open_table(THD*, TABLE_LIST*, Open_table_context*))[0x7f9106633272] sql/sql_base.cc:4016(open_and_process_table)[0x7f91066370c8] sql/sql_base.cc:5184(open_and_lock_tables(THD*, DDL_options_st const&, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*))[0x7f9106637394] sql/sql_parse.cc:5718(execute_sqlcom_select)[0x7f910666f083] sql/sql_parse.cc:2938(mysql_execute_command(THD*))[0x7f910667a9ce] sql/sql_parse.cc:7173(mysql_parse)[0x7f910667dc3d] sql/sql_parse.cc:1470(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x7f91066802a8] sql/sql_parse.cc:1095(do_command(THD*))[0x7f910668083b] sql/sql_connect.cc:1347(do_handle_one_connection(THD*))[0x7f91067435ef] sql/sql_connect.cc:1260(handle_one_connection)[0x7f9106743747] /lib64/libpthread.so.0(+0x7654)[0x7f9105374654] /lib64/libc.so.6(clone+0x6d)[0x7f9104a987dd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f90c8424020): is an invalid pointer Connection ID (thread ID): 516 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 150617 17:02:14 mysqld_safe Number of processes running now: 0 150617 17:02:14 mysqld_safe mysqld restarted 2015-06-17 17:02:15 140086665152384 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release. 2015-06-17 17:02:15 140086665152384 [Note] /usr/mariadb/mariadb/bin//mysqld (mysqld 10.1.5-MariaDB-log) starting as process 9785 ... 2015-06-17 17:02:15 140086665152384 [Note] mysqld: Aria engine: starting recovery recovered pages: 0% 10% 21% 31% 41% 51% 78% 90% 100% (0.0 seconds); tables to flush: 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 (0.3 seconds); 2015-06-17 17:02:15 140086665152384 [Note] mysqld: Aria engine: recovery done 2015-06-17 17:02:15 140086665152384 [Note] Plugin 'InnoDB' is disabled. 2015-06-17 17:02:15 140086665152384 [Note] Plugin 'FEEDBACK' is disabled. 2015-06-17 17:02:15 140086665152384 [Note] Recovering after a crash using tc.log 2015-06-17 17:02:15 140086665152384 [Note] Starting crash recovery... 2015-06-17 17:02:15 140086665152384 [Note] Crash recovery finished. 2015-06-17 17:02:15 140086665152384 [Note] Server socket created on IP: '::'. 2015-06-17 17:02:15 140086665152384 [Note] Event Scheduler: Loaded 0 events 2015-06-17 17:02:15 140086665152384 [Note] WSREP: Read nil XID from storage engines, skipping position init 2015-06-17 17:02:15 140086665152384 [Note] WSREP: wsrep_load(): loading provider library 'none' 2015-06-17 17:02:15 140086665152384 [Note] /usr/mariadb/mariadb/bin//mysqld: ready for connections. Version: '10.1.5-MariaDB-log' socket: '/tmp/mysql.BH.sock' port: 3307 MariaDB Server 2015-06-17 17:02:19 140086617798400 [ERROR] mysqld: Table './conquista_comercial/est_atu' is marked as crashed and should be repaired 2015-06-17 17:02:19 140086617798400 [Warning] Checking table: './conquista_comercial/est_atu' 2015-06-17 17:02:19 140086617798400 [ERROR] mysqld: Table './conquista_comercial/est_spa' is marked as crashed and should be repaired 2015-06-17 17:02:19 140086617798400 [Warning] Checking table: './conquista_comercial/est_spa' 2015-06-17 17:02:51 140086617798400 [ERROR] mysqld: Table './conquista_comercial/est_dia' is marked as crashed and should be repaired 2015-06-17 17:02:51 140086617798400 [Warning] Checking table: './conquista_comercial/est_dia' 2015-06-17 17:02:52 140086617798400 [ERROR] mysqld: Table './conquista_comercial/rastreabilidade_impressao' is marked as crashed and should be repaired 2015-06-17 17:02:52 140086617798400 [Warning] Checking table: './conquista_comercial/rastreabilidade_impressao' 2015-06-17 17:29:58 140086663948032 [ERROR] mysqld: Table './conquista_comercial/est_atu_erros' is marked as crashed and should be repaired 2015-06-17 17:29:58 140086663948032 [Warning] Checking table: './conquista_comercial/est_atu_erros' 2015-06-17 19:24:04 140086538364672 [Note] /usr/mariadb/mariadb/bin//mysqld: Normal shutdown 2015-06-17 19:24:04 140086538364672 [Note] WSREP: Stop replication 2015-06-17 19:24:06 140086538364672 [Note] Event Scheduler: Purging the queue. 0 events 2015-06-17 19:24:07 140086538364672 [Note] /usr/mariadb/mariadb/bin//mysqld: Shutdown complete 150617 19:24:07 mysqld_safe mysqld from pid file /home/mysql/pid/local.pid ended -- Roberto Spadim SPAEmpresarial - Software ERP Eng. Automação e Controle

3 4

Re: [Maria-developers] [Commits] 4563e13: MDEV-8156: Assertion failure in file log0crypt.cc line 220 on server restart
by Sergei Golubchik 18 Jun '15

18 Jun '15

Hi, Jan! On Jun 18, Jan Lindström wrote: > revision-id: 4563e1315df6333b280aa441e07d979d8c2ce9f4 > parent(s): eb2c170bb03fb7c98d3bd864bae46ddfb8c3acb5 > committer: Jan Lindström > branch nick: 10.1-encryption > timestamp: 2015-06-18 19:58:57 +0300 > message: > > MDEV-8156: Assertion failure in file log0crypt.cc line 220 on server restart > > Instead of asserting print informative error message to error log > and return failure from innodb_init causing the server to shutdown. A test case would've been nice Regards, Sergei

1 0

Re: [Maria-developers] [Commits] a5799f5: MDEV-7943 - pthread_getspecific() takes 0.76% in OLTP RO
by Sergei Golubchik 18 Jun '15

18 Jun '15

Hi, Sergey! On Jun 18, svoj(a)mariadb.org wrote: > revision-id: a5799f5b087f79a8063cd10242527e9235f87597 > parent(s): 4c336c8ba4fbb4ac1a4fe9680c6899c6a49b6c46 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-18 18:01:56 +0400 > message: > > MDEV-7943 - pthread_getspecific() takes 0.76% in OLTP RO > > Pass THD to Item_field (and all derivatives) constructors. > This reduces number of pthread_getspecific() calls from 322 to 292. ok to push Regards, Sergei

1 0

Re: [Maria-developers] changing some defaults related to replication
by Kristian Nielsen 18 Jun '15

18 Jun '15

Oleksandr Byelkin <sanja(a)montyprogram.com> writes: > Hi, Kristian! > > could you say against some or default changes listed here if they will > be done in 10.1? > > http://dev.mysql.com/doc/refman/5.6/en/server-default-changes.html binlog_checksum NONE CRC32 5.6.6 I am not sure that is a good idea. Binlog checksums are incompatible with earlier slave versions I think (though I guess we supported it all the way back to 5.3?). The implementation is rather bad, and not well tested in MariaDB I think, and could easily have bugs. --binlog-row-event-max-size 1024 8192 5.6.6 Seems like a good idea, 1024 is quite small. Shouldn't it default to something even larger than 8192? Seems safe to change. sync_master_info 0 10000 5.6.6 sync_relay_log 0 10000 5.6.6 sync_relay_log_info 0 10000 5.6.6 Seems safe enough, though I am not sure what the point of this change would be. I think that was all of the replication-related changes. I am not planning myself to change anything about this in 10.1, but I have no strong opinion for or against if someone else wants to do it. - Kristian.

1 0

Re: [Maria-developers] [Commits] 2e08ec2: MDEV-7956 - handler::rebind_psi() takes 0.07% in OLTP RO
by Sergei Golubchik 18 Jun '15

18 Jun '15

Hi, Sergey! On Jun 17, svoj(a)mariadb.org wrote: > revision-id: 2e08ec24623e0b3d3db02333e16658dcfaa0535c > parent(s): b776f8d3a8afff55d5f8208da7bdfce9aa33eb39 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-17 13:09:22 +0400 > message: > > MDEV-7956 - handler::rebind_psi() takes 0.07% in OLTP RO > > Do not call handler::rebind_psi() and handler::unbind_psi() when performance > schema is compiled out. I don't particularly like it, you're just avoding an empty virtual function call, and that in a configuration (no P_S) which no our binaries use. But you can push it if you want. Regards, Sergei

2 2

Re: [Maria-developers] [Commits] 3f0b023: MDEV-7952 - clock_gettime() takes 0.24% in OLTP RO
by Sergei Golubchik 18 Jun '15

18 Jun '15

Hi, Sergey! On Jun 17, svoj(a)mariadb.org wrote: > revision-id: 3f0b02351a53e0fcaee8554c2c852fb3da8fe7da > parent(s): 5d03dd20a4abbfd9777b619dfd51a6927323955a > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-17 18:55:38 +0400 > message: > > MDEV-7952 - clock_gettime() takes 0.24% in OLTP RO > > Initialize abs_timeout when it is about to be used. This saves one my_hrtime() > call on hot path (when we acquire MDL lock without waiting). ok > When filling I_S.PROCESSLIST use THD::start_utime/THD::utime_after_query instead > of THD::start_time. This allows us to save 2 clock_gettime() calls. see below > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 1e4723d..fa3161a 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -2730,7 +2730,7 @@ int fill_schema_processlist(THD* thd, TABLE_LIST* tables, COND* cond) > TABLE *table= tables->table; > CHARSET_INFO *cs= system_charset_info; > char *user; > - my_hrtime_t unow= my_hrtime(); > + ulonglong unow= microsecond_interval_timer(); > DBUG_ENTER("fill_schema_processlist"); > > DEBUG_SYNC(thd,"fill_schema_processlist_after_unow"); > @@ -2793,9 +2793,12 @@ int fill_schema_processlist(THD* thd, TABLE_LIST* tables, COND* cond) > table->field[4]->store(command_name[tmp->get_command()].str, > command_name[tmp->get_command()].length, cs); > /* MYSQL_TIME */ > - ulonglong start_utime= tmp->start_time * HRTIME_RESOLUTION + tmp->start_time_sec_part; > - ulonglong utime= start_utime && start_utime < unow.val > - ? unow.val - start_utime : 0; > + ulonglong utime= tmp->start_utime; > + ulonglong utime_after_query_snapshot= tmp->utime_after_query; > + if (utime < utime_after_query_snapshot) > + utime= utime_after_query_snapshot; // COM_SLEEP > + utime= utime && utime < unow ? unow - utime : 0; 1. Please add an assert here that utime<=unow. It uses microsecond_interval_timer(), which is supposed to be strictly increasing (unlike my_hrtime()). 2. You might need to update mysqld_list_processes() too. > table->field[5]->store(utime / HRTIME_RESOLUTION, TRUE); > /* STATE */ Regards, Sergei

2 1

Re: [Maria-developers] [Commits] 3d1b65b: MDEV-7943 - pthread_getspecific() takes 0.76% in OLTP RO
by Sergei Golubchik 18 Jun '15

18 Jun '15

Hi, Sergey! On Apr 28, svoj(a)mariadb.org wrote: > revision-id: 3d1b65b4923a49f9303501dd546af02982608b2b > parent(s): 8a0f3310275f7b4fa445f907140f677910e18999 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-04-28 15:20:48 +0400 > message: > > MDEV-7943 - pthread_getspecific() takes 0.76% in OLTP RO > > Pass THD to find_all_keys() and Item_equal::Item_equal(). > In MRR use table->in_use instead of current_thd. > > This reduces number of pthread_getspecific() calls from 354 to 320. ok to push. although for find_all_keys() it doesn't make any sense, it's a big, complex, and expensive function. One current_thd doesn't change anything. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] 2e08ec2: MDEV-7956 - handler::rebind_psi() takes 0.07% in OLTP RO
by Sergei Golubchik 18 Jun '15

18 Jun '15

Hi, Sergey! On Jun 17, svoj(a)mariadb.org wrote: > revision-id: 2e08ec24623e0b3d3db02333e16658dcfaa0535c > parent(s): b776f8d3a8afff55d5f8208da7bdfce9aa33eb39 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-17 13:09:22 +0400 > message: > > MDEV-7956 - handler::rebind_psi() takes 0.07% in OLTP RO > > Do not call handler::rebind_psi() and handler::unbind_psi() when performance > schema is compiled out. > > Overhead change: > handler::rebind_psi 0.04% -> out of radar > handler::unbind_psi 0.03% -> out of radar > open_table 0.21% -> 0.18% > close_thread_table 0.05% -> 0.05% Really? I don't believe it. handler::rebind_psi() and handler::unbind_psi() are empty methods, not doing anything. There's only a cost a virtual function call. And you're saying that it took 15% of the open_table() execution time? Regards, Sergei

2 1

[Maria-developers] [GSoC] MXS-2: Week 2 and 3
by sriram patil 17 Jun '15

17 Jun '15

Hi All, Please find link to my first blog post of second and third week below. <http://srirampatil.tumblr.com/post/121594591925/mxs-2-week-2-and-3> MXS-2: Week 2 and 3 <http://srirampatil.tumblr.com/post/121594591925/mxs-2-week-2-and-3> Feel free to point out mistakes, if any. Also, suggestions ans/or reviews are welcome! Thanks, Sriram

2 2

[Maria-developers] GSoC Progress Report [MDEV-5199]
by Blessing Atie 16 Jun '15

16 Jun '15

Hi Sergei, colin! This past week I have studied the sequence engine to understand how it generates tables on the fly. I also looked at how table functions work in other databases. I am currently working on how to implement the simplest table function from our new plugin API(Still to be completed) and which will do exacly what the sequence engine does but this time in a function call like in <select * from udf_seq(1,10)>. Exams are fast approaching in my University. I will acceletrate work as soon as I finish my Exams which will run from 22nd May to 6th June and after which I will be available to the fullest capacity to put enough time on my Work. In the mean while for this week I plan on completing our plugin declaration structure and also look for other candidate functions that can be implemented. Regards, Blessing Atie.

2 1

[Maria-developers] Sprint 10.0: MDEV-8205 timediff returns null when comparing decimal time to time string value
by Alexander Barkov 16 Jun '15

16 Jun '15

Hi Sergei, I'm not sure what to do with this: > MariaDB [test]> select > -> timediff('2014-01-01 00:00:00' , '2014-01-01 01:00:00' ), > -> timediff(20140101000000.000 , 20140101010000.000 ), > -> timediff(20140101000000.000 , '2014-01-01 01:00:00' ), > -> datediff('2014-01-01 00:00:00' , '2014-01-02 01:00:00' ), > -> datediff(20140101000000.000 , 20140102010000.000 ), > -> datediff(20140101000000.000 , '2014-01-02 01:00:00' )\G > *************************** 1. row *************************** > timediff('2014-01-01 00:00:00' , '2014-01-01 01:00:00' ): -01:00:00 > timediff(20140101000000.000 , 20140101010000.000 ): -01:00:00.000 > timediff(20140101000000.000 , '2014-01-01 01:00:00' ): NULL > datediff('2014-01-01 00:00:00' , '2014-01-02 01:00:00' ): -1 > datediff(20140101000000.000 , 20140102010000.000 ): -1 > datediff(20140101000000.000 , '2014-01-02 01:00:00' ): -1 > 1 row in set (0.03 sec) Notice, it returns NULL in one case. The problem is in this piece of the code: > if (args[0]->get_time(&l_time1) || > args[1]->get_time(&l_time2) || > l_time1.time_type != l_time2.time_type) > return (null_value= 1); get_time() forces decimal-to-time conversion to truncate the date part and return MYSQL_TIMESTAMP_TIME. get_time() does not force string-to-time conversion to truncate the date part. It still returns the full MYSQL_TIMESTAMP_DATETIME. I'm afraid a lot of the code rely in this behavior. If we fix string-to-time conversion, mtr will start to fail in other cases. Any ideas what to do?

2 4

[Maria-developers] MDEV-7502 Automatic provisioning of slave – retrieval of data from master
by Martin Kaluznik 16 Jun '15

16 Jun '15

Hello Kristian, after master-slave connection part, I proceeded to retrieval of data from master. There are two types of data which provisioning master will have to send, meta-data and row data. Meta-data For this I found only one method, pass output of SHOW CREATE * to slave in 'Query_log_event'. When I was looking for alternatives, I have found MySQL online backup work log where they are describing similar/same problem under High Level Architecture – Decision ( http://dev.mysql.com/worklog/task/?id=3574 ). Because of this, I find it unlikely, that there currently is better way. Because meta-data retrieval will take only small fraction of provisioning time, I think, that it could be implemented using 'Ed_connection' class, it executes query in string form and returns result set from within replication thread. Disadvantage of this solution is that query has to go through parser and all steps of generic execution, but on the other hand resulting code will be more independent from exact implementations, their pre-conditions and won’t contain any logical mistakes (locking, etc). Row data For this, same method with 'Ed_connection' could work, but it would be probably too big overhead – parser, result processing. So I created low level loop which goes through records of table and fills them in 'Write_rows_log_event'. Could you please check it, if it looks acceptable and I will then rewrite it to generic form, it is provisioning_send_info::send_table_data() function. https://github.com/f4rnham/server/blob/3d36cbfc20dff73d92ce61168dcc9805d68d… Right now it is working as expected on small test case. It goes through whole table at once and creates multiple row events when one would be enough. The same file also contains test of 'Ed_connection' and test of slave being able to receive and process 'Query_log_event'. Thanks for advice, Martin.

2 1

[Maria-developers] Getting the columns from LEX or THD
by sriram patil 16 Jun '15

16 Jun '15

Hi all, I wanted to retrieve a tables columns i.e its schema definition when a query is executed. I figured out LEX and THD structures store these in different variables for different queries. I wanted to get the same when a simple INSERT query is executed, e.g INSERT INTO <table> VALUES(<values>). Where can I find it? Is there any mechanism to get a tables schema that will work for any kind of query? Thanks, Sriram

2 2

Re: [Maria-developers] 8456b23: MDEV-5309 - RENAME TABLE does not check for existence of the table's engine
by Sergei Golubchik 16 Jun '15

16 Jun '15

Hi, Sergey! On Jun 15, svoj(a)mariadb.org wrote: > revision-id: 8456b231e9079be26c77da2a294ea1dac64d9d76 > parent(s): aad8667f8a4e2a26652494b29ee6c12824865ac8 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-15 15:49:00 +0400 > message: > > MDEV-5309 - RENAME TABLE does not check for existence of the table's engine > > When RENAME TABLE is executed, it apparently does not check whether the engine > is available (unlike ALTER TABLE .. RENAME, which does). It means that if the > engine in question was not loaded on some reason, the table might become > unusable, since the engine won't know about the change. > > With this patch RENAME TABLE fails if storage engine is not available. Ok to push. I'm not completely happy with the "table does not exist" error in this case, but I suspect that doing something reasonable might be not very easy. And it's a corner case anyway. Regards, Sergei

2 1

[Maria-developers] unsubscribe
by Abdul Hakeem 14 Jun '15

14 Jun '15

2 1

Re: [Maria-developers] 186e8e9: MDEV-5171: Add support for --innodb-optimize-keys to mysqldump.
by Sergei Golubchik 14 Jun '15

14 Jun '15

Hi, Jan! On May 04, Jan Lindström wrote: > revision-id: 186e8e91f3ce797258389f6d8729082f7d3ea164 > parent(s): aa5095627e2619bdad7916d33d1016802a84a9e1 > committer: Jan Lindström > branch nick: 10.0-git > timestamp: 2015-05-04 10:23:50 +0300 > message: > > MDEV-5171: Add support for --innodb-optimize-keys to mysqldump. > > Merged revision 93 from bzr+ssh://bazaar.launchpad.net/+branch/percona-server/5.6/ > authored by Alexey Kopytov. > > This patch expands the applicability of InnoDB fast index creation to > mysqldump, ALTER TABLE and OPTIMIZE TABLE as follows: > > 1. mysqldump has now a new option, --innodb-optimize-keys, which changes > the way InnoDB tables are dumped so that secondary and foreign keys are > created after loading the data thus taking advantage of fast index > creation. > > This part of the patch is an implementation of the feature request > reported as MySQL bug #49120. > > More specifically: > > - KEY, UNIQUE KEY and CONSTRAINT specifications are omitted from CREATE > TABLE corresponding to InnoDB tables. > > - an additional ALTER TABLE is issued after dumping the data to create > the previously omitted keys. > > Delaying foreign key creation does not introduce any additional risks as > mysqldump always prepends its output with SET FOREIGN_KEY_CHECKS=0 anyway. > > 2. When ALTER TABLE requires a table copy, secondary keys are now dropped > and recreated later after copying the data. The following restrictions > apply: > > - only non-unique keys can be involved in this optimization > > - if the table contains foreign keys, or a foreign key is being added as > a part of the current ALTER TABLE statement, the optimization is > disabled for all keys. > > This part of the patch is an implementation of the feature request > reported as MySQL bug #57583. > > 3. As OPTIMIZE TABLE is mapped to ALTER TABLE ... ENGINE=InnoDB for > InnoDB tables, it now also benefits from fast index creation with the > same restrictions as for ALTER TABLE. > diff --git a/client/client_priv.h b/client/client_priv.h > index 67a6e82..1846926 100644 > --- a/client/client_priv.h > +++ b/client/client_priv.h > @@ -94,6 +94,7 @@ enum options_client > OPT_SSL_CRL, OPT_SSL_CRLPATH, > OPT_USE_GTID, > OPT_GALERA_SST_MODE, > + OPT_INNODB_OPTIMIZE_KEYS, /* mysqldump */ Not needed, please, remove > OPT_MAX_CLIENT_OPTION /* should be always the last */ > }; > > diff --git a/client/mysqldump.c b/client/mysqldump.c > index e118198..0970082 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -375,6 +386,11 @@ static struct my_option my_long_options[] = > "in dump produced with --dump-slave.", &opt_include_master_host_port, > &opt_include_master_host_port, 0, GET_BOOL, NO_ARG, > 0, 0, 0, 0, 0, 0}, > + {"innodb-optimize-keys", OPT_INNODB_OPTIMIZE_KEYS, Here use 0 instead of OPT_INNODB_OPTIMIZE_KEYS > + "Use InnoDB fast index creation by creating secondary indexes after " > + "dumping the data.", > + &opt_innodb_optimize_keys, &opt_innodb_optimize_keys, 0, GET_BOOL, NO_ARG, > + 0, 0, 0, 0, 0, 0}, > {"insert-ignore", OPT_INSERT_IGNORE, "Insert rows with INSERT IGNORE.", > &opt_ignore, &opt_ignore, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, > 0, 0}, > @@ -2581,6 +2597,359 @@ static inline my_bool general_log_or_slow_log_tables(const char *db, > } > > /* > + Find the first occurrence of a quoted identifier in a given string. Returns > + the pointer to the opening quote, and stores the pointer to the closing quote > + to the memory location pointed to by the 'end' argument, > + > + If no quoted identifiers are found, returns NULL (and the value pointed to by > + 'end' is undefined in this case). > +*/ > + > +static const char *parse_quoted_identifier(const char *str, > + const char **end) > +{ > + const char *from; > + const char *to; > + > + if (!(from= strchr(str, '`'))) > + return NULL; This is wrong, see below. Identifiers ar enot necessarily quoted, so this function should parse both quoted and not quoted identifiers (and renamed to, say, parse_next_identifier) > + > + to= from; > + > + while ((to= strchr(to + 1, '`'))) { > + /* > + Double backticks represent a backtick in identifier, rather than a quote > + character. > + */ > + if (to[1] == '`') > + { > + to++; > + continue; > + } > + > + break; > + } > + > + if (to <= from + 1) > + return NULL; /* Empty identifier */ > + > + *end= to; > + > + return from; > +} > + > +/* > + Parse the specified key definition string and check if the key contains an > + AUTO_INCREMENT column as the first key part. We only check for the first key > + part, because unlike MyISAM, InnoDB does not allow the AUTO_INCREMENT column > + as a secondary key column, i.e. the AUTO_INCREMENT column would not be > + considered indexed for such key specification. > +*/ > +static my_bool contains_autoinc_column(const char *autoinc_column, > + const char *keydef, > + key_type_t type) > +{ > + const char *from, *to; > + uint idnum; > + > + DBUG_ASSERT(type != KEY_TYPE_NONE); > + > + if (autoinc_column == NULL) > + return FALSE; > + > + idnum= 0; > + > + /* > + There is only 1 iteration of the following loop for type == KEY_TYPE_PRIMARY > + and 2 iterations for type == KEY_TYPE_UNIQUE / KEY_TYPE_NON_UNIQUE. > + */ > + while ((from= parse_quoted_identifier(keydef, &to))) > + { > + idnum++; > + > + /* > + Skip the check if it's the first identifier and we are processing a > + secondary key. > + */ > + if ((type == KEY_TYPE_PRIMARY || idnum != 1) && > + !strncmp(autoinc_column, from + 1, to - from - 1)) > + return TRUE; > + > + /* > + Check only the first (for PRIMARY KEY) or the second (for secondary keys) > + quoted identifier. > + */ > + if ((idnum == 1 + MY_TEST(type != KEY_TYPE_PRIMARY))) > + break; > + > + keydef= to + 1; > + } > + > + return FALSE; > +} > + > +/* > + Find a node in the skipped keys list whose name matches a quoted > + identifier specified as 'id_from' and 'id_to' arguments. > +*/ > + > +static LIST *find_matching_skipped_key(const char *id_from, > + const char *id_to) > +{ > + LIST *list; > + size_t id_len; > + > + id_len= id_to - id_from + 1; > + DBUG_ASSERT(id_len > 2); > + > + for (list= skipped_keys_list; list; list= list_rest(list)) > + { > + const char *keydef; > + const char *keyname_from; > + const char *keyname_to; > + size_t keyname_len; > + > + keydef= list->data; > + > + if ((keyname_from= parse_quoted_identifier(keydef, &keyname_to))) > + { > + keyname_len= keyname_to - keyname_from + 1; > + > + if (id_len == keyname_len && > + !strncmp(keyname_from, id_from, id_len)) > + return list; > + } > + } > + > + return NULL; > +} > + > +/* > + Remove secondary/foreign key definitions from a given SHOW CREATE TABLE string > + and store them into a temporary list to be used later. > + > + SYNOPSIS > + skip_secondary_keys() > + create_str SHOW CREATE TABLE output > + has_pk TRUE, if the table has PRIMARY KEY > + (or UNIQUE key on non-nullable columns) > + > + > + DESCRIPTION > + > + Stores all lines starting with "KEY" or "UNIQUE KEY" > + into skipped_keys_list and removes them from the input string. > + Ignoring FOREIGN KEYS constraints when creating the table is ok, because > + mysqldump sets foreign_key_checks to 0 anyway. > +*/ > + > +static void skip_secondary_keys(char *create_str, my_bool has_pk) > +{ > + char *ptr, *strend; > + char *last_comma= NULL; > + my_bool pk_processed= FALSE; > + char *autoinc_column= NULL; > + my_bool has_autoinc= FALSE; > + key_type_t type; > + const char *constr_from; > + const char *constr_to; > + LIST *keydef_node; > + my_bool keys_processed= FALSE; > + > + strend= create_str + strlen(create_str); > + > + ptr= create_str; > + while (*ptr && !keys_processed) > + { please, fix the coding style (here and everywhere in the patch) to follow the rest of the file. That is while { char *tmp, *orig_ptr, c; ... > + char *tmp, *orig_ptr, c; > + > + orig_ptr= ptr; > + /* Skip leading whitespace */ > + while (*ptr && my_isspace(charset_info, *ptr)) > + ptr++; > + > + /* Read the next line */ > + for (tmp= ptr; *tmp != '\n' && *tmp != '\0'; tmp++); > + > + c= *tmp; > + *tmp= '\0'; /* so strstr() only processes the current line */ > + > + if (!strncmp(ptr, "CONSTRAINT ", sizeof("CONSTRAINT ") - 1) && > + (constr_from= parse_quoted_identifier(ptr, &constr_to)) && yuck! identifiers do not have to be quoted. this code needs to be fixed to accept non-quoted identifiers too. > + (keydef_node= find_matching_skipped_key(constr_from, constr_to))) > + { > + char *keydef; > + size_t keydef_len; > + > + /* > + There's a skipped key with the same name as the constraint name. Let's > + put it back before the current constraint definition and remove from the > + skipped keys list. 1. fix the formatting, please 2. why not to create the FK constraint afterwards? > + */ > + keydef= keydef_node->data; > + keydef_len= strlen(keydef) + 5; /* ", \n " */ > + > + memmove(orig_ptr + keydef_len, orig_ptr, strend - orig_ptr + 1); > + memcpy(ptr, keydef, keydef_len - 5); > + memcpy(ptr + keydef_len - 5, ", \n ", 5); > + > + skipped_keys_list= list_delete(skipped_keys_list, keydef_node); > + my_free(keydef); > + my_free(keydef_node); > + > + strend+= keydef_len; > + orig_ptr+= keydef_len; > + ptr+= keydef_len; > + tmp+= keydef_len; > + > + type= KEY_TYPE_NONE; > + } > + else if (!strncmp(ptr, "UNIQUE KEY ", sizeof("UNIQUE KEY ") - 1)) > + type= KEY_TYPE_UNIQUE; > + else if (!strncmp(ptr, "KEY ", sizeof("KEY ") - 1)) > + type= KEY_TYPE_NON_UNIQUE; > + else if (!strncmp(ptr, "PRIMARY KEY ", sizeof("PRIMARY KEY ") - 1)) > + type= KEY_TYPE_PRIMARY; > + else > + type= KEY_TYPE_NONE; > + > + has_autoinc= (type != KEY_TYPE_NONE) ? > + contains_autoinc_column(autoinc_column, ptr, type) : FALSE; > + > + /* Is it a secondary index definition? */ > + if (c == '\n' && > + ((type == KEY_TYPE_UNIQUE && (pk_processed || !has_pk)) || > + type == KEY_TYPE_NON_UNIQUE) && !has_autoinc) > + { > + char *data, *end= tmp - 1; > + > + /* Remove the trailing comma */ > + if (*end == ',') > + end--; > + data= my_strndup(ptr, end - ptr + 1, MYF(MY_FAE)); > + skipped_keys_list= list_cons(data, skipped_keys_list); > + > + memmove(orig_ptr, tmp + 1, strend - tmp); > + ptr= orig_ptr; > + strend-= tmp + 1 - ptr; > + > + /* Remove the comma on the previos line */ > + if (last_comma != NULL) > + { > + *last_comma= ' '; > + } > + } > + else > + { > + char *end; > + > + if (last_comma != NULL && *ptr == ')') > + { > + keys_processed= TRUE; > + } > + else if (last_comma != NULL && !keys_processed) > + { > + /* > + It's not the last line of CREATE TABLE, so we have skipped a key > + definition. We have to restore the last removed comma. > + */ > + *last_comma= ','; > + } > + > + /* > + If we are skipping a key which indexes an AUTO_INCREMENT column, it is > + safe to optimize all subsequent keys, i.e. we should not be checking for > + that column anymore. > + */ > + if (type != KEY_TYPE_NONE && has_autoinc) > + { > + DBUG_ASSERT(autoinc_column != NULL); > + > + my_free(autoinc_column); > + autoinc_column= NULL; > + } > + > + if ((has_pk && type == KEY_TYPE_UNIQUE && !pk_processed) || > + type == KEY_TYPE_PRIMARY) > + pk_processed= TRUE; > + > + if (strstr(ptr, "AUTO_INCREMENT") && *ptr == '`') > + { > + /* > + The first secondary key defined on this column later cannot be > + skipped, as CREATE TABLE would fail on import. Unless there is a > + PRIMARY KEY and it indexes that column. > + */ > + for (end= ptr + 1; > + /* Skip double backticks as they are a part of identifier */ > + *end != '\0' && (*end != '`' || end[1] == '`'); > + end++) > + /* empty */; > + > + if (*end == '`' && end > ptr + 1) > + { > + DBUG_ASSERT(autoinc_column == NULL); > + > + autoinc_column= my_strndup(ptr + 1, end - ptr - 1, MYF(MY_FAE)); Eh? Why does it need to strdup? > + } > + } > + > + *tmp= c; > + > + if (tmp[-1] == ',') > + last_comma= tmp - 1; > + ptr= (*tmp == '\0') ? tmp : tmp + 1; > + } > + } > + > + my_free(autoinc_column); > +} > + > +/* > + Check if the table has a primary key defined either explicitly or > + implicitly (i.e. a unique key on non-nullable columns). > + > + SYNOPSIS > + my_bool has_primary_key(const char *table_name) > + > + table_name quoted table name > + > + RETURNS TRUE if the table has a primary key > + > + DESCRIPTION > +*/ > + > +static my_bool has_primary_key(const char *table_name) > +{ > + MYSQL_RES *res= NULL; > + MYSQL_ROW row; > + char query_buff[QUERY_LENGTH]; > + my_bool has_pk= TRUE; > + > + my_snprintf(query_buff, sizeof(query_buff), > + "SELECT COUNT(*) FROM INFORMATION_SCHEMA.COLUMNS WHERE " > + "TABLE_SCHEMA=DATABASE() AND TABLE_NAME='%s' AND " > + "COLUMN_KEY='PRI'", table_name); > + if (mysql_query(mysql, query_buff) || !(res= mysql_store_result(mysql)) || > + !(row= mysql_fetch_row(res))) > + { > + fprintf(stderr, "Warning: Couldn't determine if table %s has a " > + "primary key (%s). " > + "--innodb-optimize-keys may work inefficiently.\n", > + table_name, mysql_error(mysql)); > + goto cleanup; > + } > + > + has_pk= atoi(row[0]) > 0; > + > + cleanup: > + if (res) > + mysql_free_result(res); > + > + return has_pk; > +} > + > +/* > get_table_structure -- retrievs database structure, prints out corresponding > CREATE statement and fills out insert_pat if the table is the type we will > be dumping. > @@ -2659,6 +3029,9 @@ static uint get_table_structure(char *table, char *db, char *table_type, > result_table= quote_name(table, table_buff, 1); > opt_quoted_table= quote_name(table, table_buff2, 0); > > + if (opt_innodb_optimize_keys && !strcmp(table_type, "InnoDB")) > + has_pk= has_primary_key(table); > + see below > if (opt_order_by_primary) > order_by= primary_key_fields(result_table); > > @@ -2838,6 +3211,9 @@ static uint get_table_structure(char *table, char *db, char *table_type, > > row= mysql_fetch_row(result); > > + if (opt_innodb_optimize_keys && !strcmp(table_type, "InnoDB")) > + skip_secondary_keys(row[1], has_pk); > + Not that it makes a lot of difference, but I'd remove the previous hunk, the variable has_pk, and put everything here, as if (opt_innodb_optimize_keys && !strcmp(table_type, "InnoDB")) skip_secondary_keys(row[1], has_primary_key(table)); this removes one if(), one strcmp(), and doesn't invoke has_primary_key() (meaning, one SQL query less) unless all other conditions for skip_secondary_keys() are met. even better - don't do has_primary_key() at all, remove the function altogether, and look for "PRIMARY KEY" in the create string inside skip_secondary_keys(). > is_log_table= general_log_or_slow_log_tables(db, table); > if (is_log_table) > row[1]+= 13; /* strlen("CREATE TABLE ")= 13 */ Regards, Sergei

1 0

Re: [Maria-developers] Features Wish-list in MariaDB 10.1.
by Kristian Nielsen 12 Jun '15

12 Jun '15

So my opinion on these below two points was requested, let's keep it on the public mailing list: > * Crash-Safe replication with filename and position (not depending on > GTIDs) > Something similar as what is described in the following link would > be good: > http://blog.booking.com/better_crash_safe_replication_for_mysql.html > * Have a way to avoid "SELECT binlog_gtid_pos" when a slave connect to > the master in MASTER_USE_GTID = no So my personal opinion on this is that I prefer to focus on improving the MariaDB GTID implementation. Considerable effort was invested into this, and the idea is that future development will focus on GTID mode, keeping non-GTID mode only as a backward compatibility. Significant effort was made to ensure that MariaDB GTID is usable for all existing use-cases of replication. For the first point, GTID replication is already crash-safe. For the second point, for GTID we in any case need a solution for locating GTID position quickly, otherwise GTID replication can have performance problems in case of frequent slave connects (and Jonas Oreland has promised a patch for this any day now ;-). And once this solution is in, the second point should be moot anyway. Of course, others may have a different opinion. For the second case, I suppose the easiest thing is just to remove the code that does SELECT binlog_gtid_pos() from the slave, if no-one cares about that functionality anyway... Hope this helps, - Kristian.

2 2

[Maria-developers] 10.0 sprint: Please advise re MDEV-8109 unexpected CAST result
by Alexander Barkov 11 Jun '15

11 Jun '15

Hi Sergei, I'm thinking about what to do with MDEV-8109 unexpected CAST result. Observations: 1. An empty string in VALUES SET sql_mode='STRICT_ALL_TABLES'; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (a DECIMAL); INSERT INTO t1 VALUES (''); ERROR 1292 (22007): Truncated incorrect DECIMAL value: '' This is OK. In strict mode INSERT with a bad value correctly returns the error. 2. CAST('' AS DECIMAL) SET sql_mode='STRICT_ALL_TABLES'; SELECT CAST('' AS DECIMAL); +---------------------+ | CAST('' AS DECIMAL) | +---------------------+ | 0 | +---------------------+ 1 row in set, 1 warning (0.00 sec) This is also OK. There are no tables involved, and CAST itself does not convert warnings to errors. 3. CAST('' AS DECIMAL) in VALUES: INSERT INTO t1 VALUES(CAST('' AS DECIMAL)); ERROR 1292 (22007): Truncated incorrect DECIMAL value: '' I think this is not OK. I wrote an explicit CAST, so I expect: - CAST to return 0 with a warning, like in #2 - INSERT to write 0 into the table normally, as I'm actually inserting the CAST result, which is 0. 4. COLUMN_GET(AS DECIMAL) in VALUES: SET @aaa = COLUMN_CREATE('price', ''); INSERT INTO t1 VALUES(COLUMN_GET(@aaa, 'price' AS DECIMAL)); ERROR 1918 (22007): Encountered illegal value '' when converting to DECIMAL This is also not OK. I wrote explicit column conversion to DECIMAL. I expect: - COLUMN_GET() to return 0 with a warning, like in #2 - INSERT to write 0 into the table normally, as I'm actually inserting the COLUMN_GET result, which is 0. Do you agree? Thanks.

2 4

Re: [Maria-developers] MariaDB replication event generators?
by Kristian Nielsen 11 Jun '15

11 Jun '15

James Cheng <jcheng(a)tivo.com> writes: > The binlog stream has all the row data, but doesn't have > everything. It doesn't have column names, for example, or primary > keys, or character sets. Using the binlog for replication seems to > assume that you have a copy of the schema beforehand on the receiving > slave. It doesn't seem possible to replicate to a slave using only the > information in the binlogs. It is a common request to be able to do 3rd-party replication from mysqld. However, the replication protocol between master and slave servers was never designed as a public API, it is based on internal data formats and so on. So unfortunately, there does not exist currently a public API for doing external replication, at least not that I am aware of. The binlog stream _does_ have everything of course, eg. in the form of CREATE TABLE statements and so on. However, being internal to the server, they are not very easy to use by an external replication mechanism. - Kristian.

1 0

Re: [Maria-developers] [Commits] 6ff2c0a: MDEV-5309 - RENAME TABLE does not check for existence of the table's engine
by Sergei Golubchik 10 Jun '15

10 Jun '15

Hi, Sergey! On Feb 09, svoj(a)mariadb.org wrote: > revision-id: 6ff2c0a05155f985484e23bd8611eb95f045907f > parent(s): f13939061d468a47985dee0268652a6fe2db7862 > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2015-02-09 16:12:00 +0400 > message: > > MDEV-5309 - RENAME TABLE does not check for existence of the table's engine > > When RENAME TABLE is executed, it apparently does not check whether the engine > is available (unlike ALTER TABLE .. RENAME, which does). It means that if the > engine in question was not loaded on some reason, the table might become > unusable, since the engine won't know about the change. > > With this patch RENAME TABLE fails if storage engine is not available. .... > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index e9a1ae9..f1096da 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -5172,18 +5172,19 @@ bool mysql_create_table(THD *thd, TABLE_LIST *create_table, > error= my_errno; > (void) file->ha_create_partitioning_metadata(to, from, CHF_RENAME_FLAG); > } > - else if (!file || !(error=file->ha_rename_table(from_base, to_base))) > + else if (!file) > { > + error= ER_UNKNOWN_STORAGE_ENGINE; > + } > + else if (!(error= file->ha_rename_table(from_base, to_base))) > + { > if (!(flags & NO_FRM_RENAME) && rename_file_ext(from,to,reg_ext)) > { > error= my_errno; > - if (file) > - { > if (error == ENOENT) > error= 0; // this is ok if file->ha_rename_table() succeeded > else > file->ha_rename_table(to_base, from_base); // Restore old file name > - } > } > } > delete file; May be you'd rather fix it in the caller? Let's state that the first argument of mysql_rename_table() can never be NULL. While mysql_rename_table() is called in many places, the only place where it can actually get a NULL hton is sql_rename.cc, if ha_table_exists() returns hton=NULL. I'd add the check here and simplified mysql_rename_table(). Regards, Sergei

1 0

[Maria-developers] Progress Report on [MDEV5199]: UDF Table Functions.
by Blessing Atie 10 Jun '15

10 Jun '15

Hi Serg, Colin! The following is a rundown on my progress with the above task. I started this project by getting the source of 10.1 from Github, setting my working environment by building and running the tests on ubuntu 14 then proceeded with an analysis by studying plugins and how they interact with the server.(Especially storage engines plugins). With analysis and sources investigations being done, I realized with help from assigned mentor(Serg) that our task is about implementing MYSQL_UDF_PLUGIN with the functionalities of the CONNECT engine which is similar in functionality to table functions in a way. This past week, I worked on the interface for our plugin by declaring a structure with version and some useful init/deinit function pointers to work with our UDFs(table UDF only for now). I am currently studying the Connect engine to see how I can implement its functionalities as a UDF plugin. For this week, I intend to look at other databases for example table functions that will be good candidates that we can implement with our plugin(This was not mentioned in my proposal but I intend to implement a couple of them using our plugin even if we are short of time after the final evaluation). So far, I have had difficulties in communication due to unscheduled powerouts in my city. This has really been an issue to me because my mentor was beginning to loose hope and on the other side, I don't have the chance to ask questions when im faced with a powerout. I really hope my mentors understand with me how difficult it is to code from Cameroon. Please feel free to request for any important details i might have not mentioned which you will want to know. Best Wishes, Blessing Atie

2 1

Re: [Maria-developers] MariaDB replication event generators?
by Kristian Nielsen 10 Jun '15

10 Jun '15

James Cheng <jcheng(a)tivo.com> writes: > On Jun 7, 2015, at 12:05 AM, Kristian Nielsen <knielsen(a)knielsen-hq.org> wrote: > >> James Cheng <jcheng(a)tivo.com> writes: >> >>> I recently read your old blog post from 2010 >>> (http://kristiannielsen.livejournal.com/14305.html) about some ideas >>> for MariaDB replication. In particular, your ideas for replication >>> event generators that would allow plugins to be written to receive >>> replication events. >>> >>> Did this work ever happen in MariaDB? Or anything close/equivalent? If >>> so, could you point me to something so that I can learn more about it? >> >> Unfortunately, nothing like this ever happened. > That's too bad. Are there any interesting details about why it never > happened? Was it simply due to not enough time, not high enough > priority? Was there any technical or organizational resistance to the > actual ideas or functionality? If someone were to build this today and > contribute it back to the community, do you think it would be > accepted? I think there was not enough interest and/or time/priority. For example, Galera cluster would be a prime example for something using this kind of API. But the developers did not prioritise spending time working on such API, rather they hacked in their functionality all over the server. It is the same with other parties in the community, everyone is focused on solving their own narrow problem, not in working together. And MySQL@Oracle, of course, is not interested in anything that involves working with anyone else on the code. So I don't see much chance in the current state of the community of any significant effort being spent neither on developing such an API, nor on using it. - Kristian.

1 0

Re: [Maria-developers] [Commits] 865bbbf: MDEV-3870 - Valgrind warnings on OPTIMIZE MyISAM or Aria TABLE with disabled
by Sergei Golubchik 09 Jun '15

09 Jun '15

Hi, Sergey! On Jun 09, svoj(a)mariadb.org wrote: > revision-id: 865bbbf14d1a8ed500eeeadabe0a7fdfa90420df > parent(s): b8e26b04d4ac22e29a78bca68e101176b6ef507a > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-09 16:08:09 +0400 > message: > > MDEV-3870 - Valgrind warnings on OPTIMIZE MyISAM or Aria TABLE with disabled > keys > > Fixed that OPTIMIZE TABLE against MyISAM/Aria table may write uninitialized key > root position for disabled keys. ok to push, thanks! Regards, Sergei

1 0