- developers - lists.mariadb.org

Re: [Maria-developers] [Commits] 0a1b60a: MDEV-8450: PATCH] Wrong macro expansion in Query_cache::send_result_to_client()
by Sergey Vojtovich 06 Sep '15

06 Sep '15

Hi Sanja, ok to push, a few minor notes JFYI inline. I found only one real bug, but generally I agree that we should fix these fragile macros. On Tue, Sep 01, 2015 at 11:47:09AM +0200, sanja(a)mariadb.com wrote: > revision-id: 0a1b60a8aee4bce3235fd5a1d8cb2231b51ff5cc (mariadb-10.0.21-7-g0a1b60a) > parent(s): 4b41e3c7f33714186c97a6cc2e6d3bb93b050c61 > committer: Oleksandr Byelkin > timestamp: 2015-09-01 11:47:06 +0200 > message: > > MDEV-8450: PATCH] Wrong macro expansion in Query_cache::send_result_to_client() > > Expression in macro protected by () > > diff --git a/include/maria.h b/include/maria.h > index 16a9bea..cdaeade 100644 > --- a/include/maria.h > +++ b/include/maria.h > @@ -43,7 +43,7 @@ extern "C" { > #define MARIA_NAME_IEXT ".MAI" > #define MARIA_NAME_DEXT ".MAD" > /* Max extra space to use when sorting keys */ > -#define MARIA_MAX_TEMP_LENGTH 2*1024L*1024L*1024L > +#define MARIA_MAX_TEMP_LENGTH (2*1024L*1024L*1024L) > /* Possible values for maria_block_size (must be power of 2) */ > #define MARIA_KEY_BLOCK_LENGTH 8192 /* default key block length */ > #define MARIA_MIN_KEY_BLOCK_LENGTH 1024 /* Min key block length */ Was this used at all? May be remove it? > diff --git a/sql/log_slow.h b/sql/log_slow.h > index 2ae07da..3ae2060 100644 > --- a/sql/log_slow.h > +++ b/sql/log_slow.h > @@ -16,23 +16,23 @@ > /* Defining what to log to slow log */ > > #define LOG_SLOW_VERBOSITY_INIT 0 > -#define LOG_SLOW_VERBOSITY_INNODB 1 << 0 > -#define LOG_SLOW_VERBOSITY_QUERY_PLAN 1 << 1 > -#define LOG_SLOW_VERBOSITY_EXPLAIN 1 << 2 > +#define LOG_SLOW_VERBOSITY_INNODB (1 << 0) Was LOG_SLOW_VERBOSITY_INNODB used at all? May be remove it? > +#define LOG_SLOW_VERBOSITY_QUERY_PLAN (1 << 1) > +#define LOG_SLOW_VERBOSITY_EXPLAIN (1 << 2) > > #define QPLAN_INIT QPLAN_QC_NO > > -#define QPLAN_ADMIN 1 << 0 > -#define QPLAN_FILESORT 1 << 1 > -#define QPLAN_FILESORT_DISK 1 << 2 > -#define QPLAN_FULL_JOIN 1 << 3 > -#define QPLAN_FULL_SCAN 1 << 4 > -#define QPLAN_QC 1 << 5 > -#define QPLAN_QC_NO 1 << 6 > -#define QPLAN_TMP_DISK 1 << 7 > -#define QPLAN_TMP_TABLE 1 << 8 > -#define QPLAN_FILESORT_PRIORITY_QUEUE 1 << 9 > +#define QPLAN_ADMIN (1 << 0) > +#define QPLAN_FILESORT (1 << 1) > +#define QPLAN_FILESORT_DISK (1 << 2) > +#define QPLAN_FULL_JOIN (1 << 3) > +#define QPLAN_FULL_SCAN (1 << 4) > +#define QPLAN_QC (1 << 5) > +#define QPLAN_QC_NO (1 << 6) > +#define QPLAN_TMP_DISK (1 << 7) > +#define QPLAN_TMP_TABLE (1 << 8) > +#define QPLAN_FILESORT_PRIORITY_QUEUE (1 << 9) > > /* ... */ > -#define QPLAN_MAX ((ulong) 1) << 31 /* reserved as placeholder */ > +#define QPLAN_MAX (((ulong) 1) << 31) /* reserved as placeholder */ > log_slow.h was fixed in rev.203f4d419 in my 10.0. Did you use outdated tree? I may be wrong, but there seem to be no real bug with QPLAN_QC_NO currently. Just because nobody sets it. So in fact it clears QPLAN_ADMIN, which FWICS is never set at this point. But it is a good to have it fixed nevertheless. Btw, QPLAN_QC_NO doesn't seem to be used in fact. May be remove it? > diff --git a/storage/archive/archive_test.c b/storage/archive/archive_test.c > index d01c1e0..bb052b8 100644 > --- a/storage/archive/archive_test.c > +++ b/storage/archive/archive_test.c > @@ -33,7 +33,7 @@ > > #define ARCHIVE_ROW_HEADER_SIZE 4 > > -#define BUFFER_LEN 1024 + ARCHIVE_ROW_HEADER_SIZE > +#define BUFFER_LEN (1024 + ARCHIVE_ROW_HEADER_SIZE) > > char test_string[BUFFER_LEN]; Alright, there was a bug in this line: assert(write_length != count * BUFFER_LEN); > diff --git a/strings/ctype.c b/strings/ctype.c > index 048fbe3..d8a1dd7 100644 > --- a/strings/ctype.c > +++ b/strings/ctype.c > @@ -264,7 +264,7 @@ static const struct my_cs_file_section_st > } > > #define MY_CS_CSDESCR_SIZE 64 > -#define MY_CS_TAILORING_SIZE 32*1024 > +#define MY_CS_TAILORING_SIZE (32*1024) > #define MY_CS_UCA_VERSION_SIZE 64 > #define MY_CS_CONTEXT_SIZE 64 Was this used at all? May be remove it? Regards, Sergey

2 1

Re: [Maria-developers] [Commits] 62fe87c: MDEV-8663: IF Statement returns multiple values erroneously (or Assertion `!null_value' failed in Item::send(Protocol*, String*))
by Alexander Barkov 04 Sep '15

04 Sep '15

Hi Sanja, From my understanding, the patch fixes consequences of the problem, but not the problem itself. It very likely will still have crashes with slightly modified SQL scripts. The source of the problem is that Item_func_hybrid_result_type::val_decimal() does not return NULL when int_op() or real_op() set null_value to "true". By the way, compare Item_func_hybrid_result_type::val_decimal() with a very similar method Item_func_hybrid_result_type::val_str(). The latter already handle the INT_RESULT and REAL_RESULT cases well and return NULL properly in case of null_value. get_date(), val_int(), val_real() also look fine for me. So the are problems only in val_decimal(). I propose a small patch that fixes the source of the problem. This is the small patch: > @@ -1038,17 +1026,21 @@ my_decimal *Item_func_hybrid_result_type::val_decimal(my_decimal *decimal_value) > case INT_RESULT: > { > longlong result= int_op(); > + if (null_value) > + return NULL; > int2my_decimal(E_DEC_FATAL_ERROR, result, unsigned_flag, decimal_value); > break; > } > case REAL_RESULT: > { > double result= (double)real_op(); > + if (null_value) > + return NULL; > double2my_decimal(E_DEC_FATAL_ERROR, result, decimal_value); > break; > } I also propose an extended patch version, to cover the code with more DBUG_ASSERTs, so in case of the null_value related bugs it crashes much earlier than in Item::send(). It will be easier to debug problems this way. To avoid much code duplication, I added three private helper methods. The extended version of the patch is attached. The scripts from the bug report start to return expected results, with no crashes in DEBUG build, and "mtr --suite=main" works fine. I have not run the entire mtr though, to save time. Greetings. On 09/03/2015 08:00 PM, sanja(a)mariadb.com wrote: > revision-id: 62fe87c5a239c7304b245ce5954c87aecb6382ad (mariadb-5.5.45-1-g62fe87c) > parent(s): fa51f70dc68fe2f3afe943e2c81fcbdb34f16cad > committer: Oleksandr Byelkin > timestamp: 2015-09-03 18:00:43 +0200 > message: > > MDEV-8663: IF Statement returns multiple values erroneously (or Assertion `!null_value' failed in Item::send(Protocol*, String*)) > > keeping contract: NULL value mean NULL pointer in val_str and val_deciman. > > --- > mysql-test/r/func_if.result | 17 +++++++++++++++++ > mysql-test/t/func_if.test | 14 ++++++++++++++ > sql/item_cmpfunc.cc | 6 ++++-- > sql/item_func.cc | 5 +++++ > 4 files changed, 40 insertions(+), 2 deletions(-) > > diff --git a/mysql-test/r/func_if.result b/mysql-test/r/func_if.result > index c7f548a..61f63cc 100644 > --- a/mysql-test/r/func_if.result > +++ b/mysql-test/r/func_if.result > @@ -234,3 +234,20 @@ SELECT if(1, NULL, (SELECT min('hello'))); > if(1, NULL, (SELECT min('hello'))) > NULL > End of 5.2 tests > +# > +# MDEV-8663: IF Statement returns multiple values erroneously > +# (or Assertion `!null_value' failed in Item::send(Protocol*, String*) > +# > +CREATE TABLE `t1` ( > +`datas` VARCHAR(25) NOT NULL > +) ENGINE=InnoDB DEFAULT CHARSET=utf8; > +Warnings: > +Warning 1286 Unknown storage engine 'InnoDB' > +Warning 1266 Using storage engine MyISAM for table 't1' > +INSERT INTO `t1` VALUES ('1,2'), ('2,3'), ('3,4'); > +SELECT IF(FIND_IN_SET('1', `datas`), 1.5, IF(FIND_IN_SET('2', `datas`), 2, NULL)) AS `First`, '1' AS `Second`, '2' AS `Third` FROM `t1`; > +First Second Third > +1.5 1 2 > +2.0 1 2 > +NULL 1 2 > +drop table t1; > diff --git a/mysql-test/t/func_if.test b/mysql-test/t/func_if.test > index 2b89a61..8fdba77 100644 > --- a/mysql-test/t/func_if.test > +++ b/mysql-test/t/func_if.test > @@ -206,6 +206,20 @@ SELECT if(1, NULL, (SELECT min('hello'))); > > --echo End of 5.2 tests > > +--echo # > +--echo # MDEV-8663: IF Statement returns multiple values erroneously > +--echo # (or Assertion `!null_value' failed in Item::send(Protocol*, String*) > +--echo # > +CREATE TABLE `t1` ( > +`datas` VARCHAR(25) NOT NULL > +) ENGINE=InnoDB DEFAULT CHARSET=utf8; > + > +INSERT INTO `t1` VALUES ('1,2'), ('2,3'), ('3,4'); > + > +SELECT IF(FIND_IN_SET('1', `datas`), 1.5, IF(FIND_IN_SET('2', `datas`), 2, NULL)) AS `First`, '1' AS `Second`, '2' AS `Third` FROM `t1`; > + > +drop table t1; > + > --disable_query_log > # Restore timezone to default > set time_zone= @@global.time_zone; > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index 998cb1c..4f07318 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -2692,7 +2692,8 @@ Item_func_if::str_op(String *str) > String *res=arg->val_str(str); > if (res) > res->set_charset(collation.collation); > - null_value=arg->null_value; > + if (null_value=arg->null_value) > + res= NULL; > return res; > } > > @@ -2703,7 +2704,8 @@ Item_func_if::decimal_op(my_decimal *decimal_value) > DBUG_ASSERT(fixed == 1); > Item *arg= args[0]->val_bool() ? args[1] : args[2]; > my_decimal *value= arg->val_decimal(decimal_value); > - null_value= arg->null_value; > + if (null_value= arg->null_value) > + value= NULL; > return value; > } > > diff --git a/sql/item_func.cc b/sql/item_func.cc > index 2f51308..cbb8aa0 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -924,6 +924,7 @@ String *Item_func_hybrid_result_type::val_str(String *str) > ltime.time_type= mysql_type_to_time_type(field_type()); > str->length(my_TIME_to_str(&ltime, const_cast<char*>(str->ptr()), decimals)); > str->set_charset(&my_charset_bin); > + DBUG_ASSERT(!null_value); > return str; > } > return str_op(&str_value); > @@ -932,6 +933,7 @@ String *Item_func_hybrid_result_type::val_str(String *str) > case IMPOSSIBLE_RESULT: > DBUG_ASSERT(0); > } > + DBUG_ASSERT(!null_value || (str == NULL)); > return str; > } > > @@ -1071,7 +1073,10 @@ my_decimal *Item_func_hybrid_result_type::val_decimal(my_decimal *decimal_value) > } > String *res; > if (!(res= str_op(&str_value))) > + { > + null_value= 1; > return NULL; > + } > > str2my_decimal(E_DEC_FATAL_ERROR, (char*) res->ptr(), > res->length(), res->charset(), decimal_value); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

1 0

[Maria-developers] [GSoC] MDEV-4674 Port InnoDB memcached interface to MariaDB - Merging the code
by Piotr Jurkiewicz 02 Sep '15

02 Sep '15

Hi again, there are some issues which I think needs to be resolved before merging the code. 1. Bundle libevent or not? Memcached daemon is built with the help of libevent library. MySQL decided to bundle this library inside its sources, in 'libevent' top level directory. This allows to compile and run the plugin on machines without installed system-wide libevent library. The bundled libevent is used by default during the compilation of plugin. Optionally, it can be specified in CMake variable to use system-wide libevent instead. It needs to be decided whether MariaDB should bundle libevent as well inside its sources. Possible outcomes: a) Leave bundled libevent as is, in the 'libevent' top level directory. b) Move bundled libevent into the 'daemon_memcached' plugin directory, in order to not clutter the top level directory. c) Remove bundled libevent. After that, system libevent headers will become compile-time dependency and system libevent shared library runtime dependency of the plugin. 2. How to transfer InnoDB API callback array pointer to the plugin? InnoDB Memcached interface engine needs to obtain the pointer to InnoDB API callback array in some way. MySQL originally implemented that in 'sql/sql_plugin.cc:plugin_initialize()' in the following way: https://github.com/piotrjurkiewicz/mariadb-server/commit/4eb8fd9c211f396694… As you can see, this is not a clean solution. Possible outcomes: a) Leave the MySQL's solution as is. b) Figure out a better way to transfer the pointer. 3. Where should I post the documentation? 4. How should I prepare the pull request? Should I rebase my branch on the top of current branch before that? I looking forward for your suggestions and decisions. Piotr

2 1

Re: [Maria-developers] MDEV-8389 patch review (part#1)
by Sergey Petrunia 27 Aug '15

27 Aug '15

Hi Diwas, On Thu, Aug 27, 2015 at 09:23:55PM +0530, Diwas Joshi wrote: > hello sergei, I am attaching two patches with this mail. > MDEV-8389_first is the original patch with mysql_create_table() method used > to create tables. I have fixed the crash that was occurring with queries > like select name from f21('aaa') where name like '%foo%'; So, this patch is > giving the results. > MDEV-8389_second contains use of create_tmp_table() so far the table is > being created, but the queries can't see it so it is still giving errors > like table not present. I guess we'll have to fix the Name Resolution for > queries to be able to use the created table. > > Also both patches have fix for MDEV-8650. I've looked at the patch. Unfortunately, it gives little confidence that it will work in all cases. There are a lot of weird things. To name a few: > + tmp_table_param->schema_table = 1; why? ... > + switch (field->sql_type) > + { > + case MYSQL_TYPE_VARCHAR: > + case MYSQL_TYPE_VAR_STRING: > + case MYSQL_TYPE_STRING: > + case MYSQL_TYPE_SET: > + field->pack_flag= 0; > + break; ... This big switch statement is copied from Create_field::init_for_tmp_table. Why is it copied (and not factored out?) If there is a need to copy the code from that function into here, why do you not also copy this part: pack_flag|= (maybe_null ? FIELDFLAG_MAYBE_NULL : 0) | (is_unsigned ? 0 : FIELDFLAG_DECIMAL); > + TABLE *vtable= create_virtual_tmp_table(thd, sp->m_cols_list); ... > + if (! (table = create_tmp_table(thd, tmp_table_param, *types, > + (ORDER*) 0, 1, 1, > + TMP_TABLE_ALL_COLUMNS, HA_POS_ERROR, > + sp->m_table_alias.str))) why create_virtual_tmp_table and then create_tmp_table? 'vtable' is created and then not freed. I look at the comment near create_virtual_tmp_table() definition and see: The table is created in THD mem_root, so are the table's fields. Consequently, if you don't BLOB fields, you don't need to free it. But what if the table does have blob fields? > + table->file->extra(HA_EXTRA_IGNORE_DUP_KEY); What's this? I grep for HA_EXTRA_IGNORE_DUP_KEY and find this explanation: HA_EXTRA_IGNORE_DUP_KEY: HA_EXTRA_NO_IGNORE_DUP_KEY: Informs the handler to we will not stop the transaction if we get an duplicate key errors during insert/upate. Do we expect to get duplicate key errors when inserting into the table? > +bool execute_table_function(THD *thd, sp_head *sp, List<Item> *value_list) > +{ > + /* > + We never write CALL statements into binlog: > + - If the mode is non-prelocked, each statement will be logged > + separately. > + - If the mode is prelocked, the invoking statement will care > + about writing into binlog. > + So just execute the statement. How is CALL statement relevant to executing a table function? I think at this point it is clear that this patch needs more work before it can be pushed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-8389 patch review (part#1)
by Sergey Petrunia 27 Aug '15

27 Aug '15

On Sat, Aug 22, 2015 at 02:10:58AM +0530, Diwas Joshi wrote: > Hello Sergei, sorry for late reply, i was busy finishing my office > work yesterday. Please find some of my doubts below (I would come on > irc to discuss but I am travelling today and will be working overnight > in the train, I hope to be on irc tomorrow). > > Regarding use of storage engine-- If one uses SET > default_tmp_storage_engine, to set the default storage engine for > temporary tables in the current session, it is using that default > storage engine. In my function I have used > create_info.use_default_db_type(thd) which does the work. > (Referencing https://dev.mysql.com/doc/refman/5.6/en/storage-engine-setting.html) > > Regarding the name resolution error. I didn't know it but if there is > already a table present in the database with same name, temporary > table is still being created and used int the function. so that is one > problem solved. For the problem of a function being used multiple > times in a query, I was just curious as how would we solve it if we > switch to create_tmp_table(). As I mentioned before, the solution should be: >> - Patch the name resolution code so that the statements inside a Table Function >> see the return table of that Table Function. "Name resolution" is a process of finding which table should be used when a query refers to table named TBL. The process currently is: 1. Look at user-created temporary tables with name TBL. 2. If none found, look at the regular database tables in the current database with the name TBL. Our problem is that tables created with mysql_create_table() are visible to step#1, and all parts of the query can see them. This includes those parts of the query that we don't want to see the table: other stored procedures, the query that invokes the table function, etc. Temporary tables that are created with create_tmp_table() are not visible to step #1. We will not have the problem of temporary table being visible where it should not be visible. We will have the problem though, that temporary table must be visible where it needs to be visible. This is achieved by modifying step#1 in Name Resolution code to also look at the name of the returned table when we are in a table function. > Still looking into crash because of select name from f21('aaa') where > name like '%foo%' and errors MDEV-8649 and MDEV-8650; > Other errors in the patch have been fixed. > BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MDEV-8643 Valgrind error in mroonga/storage.index_multiple_column_unique_datetime_index_read
by Alexander Barkov 27 Aug '15

27 Aug '15

Hi Kentoku and Kouhei, FYI, I just filed this bug report: https://mariadb.atlassian.net/browse/MDEV-8643 Can you please have a look into this? Thanks!

1 1

[Maria-developers] New Hacker
by Aniket Shukla 27 Aug '15

27 Aug '15

Hey MariaDB team,this is Aniket.I'm a final year undegraduate student and would like to contribute to MariaDB.Since this is my first open source project, I have no idea where to begin.I have good knowledge of python,c++ and have benn using MySQL for my previous projects. Any help would be welcomed.

2 1

[Maria-developers] [GSoC] MDEV-4674 Port InnoDB memcached interface to MariaDB - Final report
by Piotr Jurkiewicz 25 Aug '15

25 Aug '15

Hi all, as the pencils down date has passed, I am posting a report on what I have during the whole project. Repository of my project is available here: https://github.com/piotrjurkiewicz/mariadb-server/commits/10.1-memcached Some stats: 33 commits before the midterm 60 commits after the midterm 93 commits total What have been done in details: =============================== BEFORE MIDTERM: --------------- 1. Code of InnoDB Memcached plugin has been imported from MySQL into the MariaDB tree. 2. There were some differences between MariaDB and MySQL, which blocked the plugin from out-of-the-box compilation and correct operation. They have been resolved. 3. Test suite for the plugin has been imported from MySQL. Compatibility changes were introduced into these test cases which required them. The test suite was modified to assign memcached port number dynamically from mtr worker's pool, what allows to run the same test in parallel for many combinations. The plugin fully passes the test suite, both for 'innodb_plugin' and 'xtradb' combinations. 4. Some enhancements and bug fixes have been introduced into the plugin: - detection of memcached daemon initialization has been fixed -- this eliminated unnecessary wait (15 seconds) on plugin shutdown and memory leak (in cases when plugin failed to initialize properly before) - memcached arithmetic commands (incr/decr) handling has been improved -- now numeric columns values are handled in a consistent and predictable way - incorrect handling of unsigned not null integers columns has been fixed AFTER MIDTERM: -------------- 5. Memcached daemon code has been updated to the latest version from upstream memcached/engine-pu branch. 6. Plugin architecture has been changed to "one table per socket", as originally proposed in MDEV-4674 bug report. This allows usage of well known security solutions (firewalls, unix file permissions) to restrict access to different tables for different users. Now user can specify which tables should be accessible on which sockets. Memcached daemon will listen on all these sockets and execute commands on appropriate table, depending on which socket client connected. Supported protocols are TCP, UDP and UNIX stream sockets. Sockets are configured in 'daemon_memcached/containers' table: INSERT INTO containers VALUES ("tcp://*:11211", "db0", "table0", "key_col", "val_col", "c3", "c4", "c5", "PRIMARY", "|"); INSERT INTO containers VALUES ("udp://localhost:11222", "db0", "table1", "key_col", "val_col", "c3", "c4", "c5", "PRIMARY", "|"); INSERT INTO containers VALUES ("unix:/tmp/mc.sock", "db1", "table0", "key_col", "val_col", "c3", "c4", "c5", "PRIMARY", "|"); 7. Plugin has been decoupled from InnoDB. Now plugin is split into two parts: - generic 'daemon_memcached.so' - InnoDB specific 'daemon_memcached_engine_ib.so' The first library is a MariaDB plugin, which should be installed with command: install plugin daemon_memcached soname "daemon_memcached.so"; This library is generic. It provides Memcached protocol handling and it loads the second library, which is API-specific engine for accessing the database. Which engine library should be loaded can be specified in 'daemon_memcached_engine_lib_name' MariaDB server option. Now, the only existing engine library is daemon_memcached_engine_ib.so, which utilizes InnoDB API to access the data. It can operate on InnoDB and XtraDB tables. 8. Some enhancements and bug fixes have been introduced into the plugin: - CMake options and #cmakedefines associated with the plugin have been cleaned up - embedded default Memcached engine has been removed - duplicated code in Memcached source has been removed - fixed MySQL-originating bug with flush_all of non default table -- command flush_all always flushed the default table, not the current one - removed support for in-connection table switching -- after introduction of one table/port architecture it was redundant - plugin shutdown time has been improved -- from constant time of 2 seconds to average time of 0.5 second - logging of plugin has been refactored -- now plugin send log messages to MariaDB log instead of stderr - fixed Memcached-originating bug with get_thread_stats() -- bug resulted in out of bound array accesses and writes, what leaded to SEGFAULT - fixed MySQL-originating crash during 'flush_all' memcached command bug -- crash happened when table was heavily loaded with DML operations and because of that ib_cb_cursor_set_lock() failed to obtain a lock on the table, returning DB_LOCK_WAIT_TIMEOUT error - fixed MySQL-originating multiple memcached_sync_count for the same transaction bug -- the memcached_sync_count was incremented multiple times for the same transaction and after the end of transaction, it was not reset to zero, what blocked TRUNCATE command on any InnoDB table, on which Memcached had operated during program lifetime, even after uninstalling the plugin - support for numeric values has been finished -- now BIGINT overflows in the same way as smaller types, test to check over/under-flowing of all integer types has been added For details of all changes please see commit messages associated with them: https://github.com/piotrjurkiewicz/mariadb-server/commits/10.1-memcached ----------------------------------------------------------------------- Moreover, I planned to implement a new interface inside the Memcached plugin, which will use Handler API for database access. This would make possible to use the plugin to access databases based on storage engines other than InnoDB/XtraDB. However, during testing I discovered many serious bugs, either in Memcached or InnoDB API engine, which originated from upstream Memcached or MySQL and resulted in crashes or deadlocks. I decided that first I should fix them before implementing new functionalities. I lost a lot of time on investigating these bugs and I was unable to finish the implementation of Handler API engine. Nevertheless, I think the whole plugin is ready to be merged into MariaDB. All user-facing changes (like one port/table architecture) have been completed. Eventual later introduction of Handler API engine will not be associated with any user-facing compatibility changes. It will just lift the requirement that all databases accessed via the Memcached plugin must be InnoDB/XtraDB. Now I am going to write a separate email to coordinate the merge of my code into MariaDB repository. Piotr

1 0

Re: [Maria-developers] MDEV-7502 Provisioning of slave - review
by Kristian Nielsen 24 Aug '15

24 Aug '15

Martin Kaluznik <martin.kaluznik(a)gmail.com> writes: > good to do now. And I would also like to ask you for review of > existing code. > > Github link: https://github.com/f4rnham/server/tree/MDEV-7502 > Diff: https://github.com/MariaDB/server/compare/10.1...f4rnham:MDEV-7502.diff Please find my review inline in patch, below. General stuff: > I have created new log event as way to signalize slave successful end > of provisioning, it is last event IO thread writes to relay log before > termination, and last event SQL thread reads from relay log. Test > cases are using Slave_SQL_Running value from SHOW SLAVE STATUS command > to wait until provisioning is completed. It works there, but I am not > sure about its convinience in real world usage. Hm, I am not sure either. Another option would be for LOAD DATA FROM MASTER to block until provisioning is done (failed or successful). Other than review, please check Buildbot results carefully for any failures. Pay particular attention to the valgrind build (for memory leaks). Also pay special attention to random failures in your new test cases that happen only occasionally. Such sporadic / random failures often turn up in replication tests. You need in test cases that use debug_dbug: --source include/have_debug.inc What is the largest data set you tested with? Please do a (manual) test with a large data set. For example, you could run sysbench with a 10GB or 50GB database, let it run for say 10 minutes, and 5 minutes in you start a LOAD DATA FROM MASTER on a slave off the server running the sysbench. Please check comments below on how (and if) the whole approach with provisioning in parallel with active transactions is correct and makes the slave always identical with the master at the end. Is there any issue with foreign keys? An insert to one table could fail if a foreign key constraint against another table is not yet satisfied due to rows not yet provisioned. A CREATE TABLE referencing another table might fail. Maybe look at if mysqldump is doing something to avoid these kinds of issues. A minimal solution is to investigate possible problems, and document them as limitations. Eg. it is reasonable to document that users must disable foreign keys during provisioning, if necessary. What happens if the server that does LOAD DATA FROM MASTER provisioning has --log-bin and --log-slave-updates? What happens if a third-level slave is connected to the server that does LOAD DATA FROM MASTER? I think this needs to be tested. Comments on the patch: > +BOOLEAN _db_keyword_locked_(CODE_STATE *cs, const char *keyword, int strict) Can you explain the race that results without this locking? Is it because _db_keyword_() is scanning the list without any locking, so if the list changes during the scan we can crash? > +#define DBUG_EXECUTE_LOCKED(keyword,a1) \ > + do {if (_db_keyword_locked_(0, (keyword), 0)) { a1 }} while(0) You need empty definition for non-debug build (but I guess you already saw in buildbot run). > +--let $slave_param= Slave_SQL_Running > +--let $slave_param_value= No > +--let $slave_error_param= Last_SQL_Errno > +--source include/wait_for_slave_param.inc Is there a reason not just to use include/wait_for_slave_sql_to_stop.inc here? > +# Ensure that master and slave log names and positions are equal > +--connection master > +--let $binlog_file_master= query_get_value(SHOW MASTER STATUS, File, 1) > +--let $binlog_pos_master= query_get_value(SHOW MASTER STATUS, Position, 1) > +--connection slave > +--let $binlog_file_slave= query_get_value(SHOW SLAVE STATUS, Master_Log_File, 1) > +--let $binlog_pos_slave= query_get_value(SHOW SLAVE STATUS, Read_Master_Log_Pos, 1) Is there a reason you do not use GTIDs here instead (or in addition)? > +# echo Binlog file name master - $binlog_file_master; > +# echo Binlog file name slave - $binlog_file_slave; > +# echo Binlog position master - $binlog_pos_master; > +# echo Binlog position slave - $binlog_pos_slave; Please remove this left-over debugging code. > +++ b/mysql-test/suite/rpl/t/rpl_provisioning_base.inc > +eval CREATE TABLE test.test_1 (a int, PRIMARY KEY (a)) ENGINE=$engine_type; > +eval CREATE TABLE test.test_2 (a int, b int, c int, PRIMARY KEY (a)) ENGINE=$engine_type; Try adding also some foreign key contraints to the table for the test. > +# Start provisioning > +--connection slave > +LOAD DATA FROM MASTER; At this point, master has full binlogs. So just START SLAVE would do the same thing, just by re-playing the binlog. I think the test should use FLUSH LOGS on the master a couple of times during the initial setup of data, and then PURGE BINARY LOGS to delete some (but not all) of that binlog data, to test that the slave can still provision without complete binary logs. > +SET GLOBAL debug_dbug="+d,provisioning_wait"; > +SET GLOBAL debug_dbug="-d,provisioning_wait"; Earlier, when I did this, I ended up with tons of debug output in the error log after setting the "-d,..." which is annoying. Setting an empty "+d" option was enabling debug trace. Maybe this is now fixed, if so then all the better. But if you get this problem (with stray debug trace in error log), you can use the following pattern instead: SET @old_dbug= @@GLOBAL.debug_dbug; ... SET GLOBAL debug_dbug= @old_dbug; > +# > +# Attempt to start provisioning while not in GTID mode > +# Is this an old comment? It seems none of the tests really use any GTID. Does provisioning work in GTID mode? It should be tested. > +# > +# Attempt to start provisioning while SQL thread is running > +# What if IO thread is running? Should probably be tested also. > +# Creation of database `test` depends purely on timing and it has nothing to do with this test case > +--disable_warnings > +DROP DATABASE IF EXISTS test; What is the purpose of this? It sounds like there is a potential race in the test case? Maybe there is a better way to handle it. > +START SLAVE; > + > +# Cleanup after test case > +--connection slave > +STOP SLAVE; > +RESET SLAVE; > +# Creation of database `test` depends purely on timing and it has nothing to do with this test case It is better to use the include/start_slave.inc and include/stop_slave.inc. They prevent some races, which might solve some of the problems that your comment about timing seems to hint at. Also, it is usually best to synchronise the slave with the master before stopping, to avoid it being stopped at a random point. > +# > +# Bug: Deadlock occurs when IO thread fails with error before dump request > +# What do you mean? That the current patch has a bug triggered by this test case? Or that there was a bug, fixed now, and this is a test for it? Please explain better in the comment. > diff --git a/mysql-test/suite/rpl/t/rpl_provisioning_stress.test b/mysql-test/suite/rpl/t/rpl_provisioning_stress.test > new file mode 100644 > index 0000000..8141015 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_provisioning_stress.test > @@ -0,0 +1,48 @@ > +--skip Test requires manual setup > + > +# Stress test for MDEV-7502 Automatic provisioning of slave > +# Using Random Query Generator v2.2.0 release, downloadable from https://launchpad.net/randgen/+download I think it is great that you made this kind of test. But maybe it would be better to put it somewhere else, to not have a test in the normal test suite that is always unconditionally skipped. Eg. it could be put in /tests/, and one would copy it into the test suite, instead of removing the --skip. > +void Provisioning_done_log_event::print(FILE* file, > + print_header(&cache, print_event_info, FALSE); > + my_b_write_string(&cache, "\tProvisioning done\n"); Hm, this can occur in the relay log, but not in the binlog, right? I think it is ok not to have a test of this in mysql-test-run, we generally can only test the "short-form" in there to avoid dependency on time and such. Just be sure to check manually that the output is correct. > @@ -9718,23 +9760,29 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > + // In provisioning mode, silently ignore table not found error > + if (!rgi->thd->slave_thread || !rli->mi->provisioning_mode || > + !provisioning_error_code(actual_error)) Better use thd than rgi->thd here (just for consistency). > + // Ignore certain errors during provisioning What thoughts did you have on whether just ignoring these kinds of errors would always result in the correct result of provisioning? I suppose since we are copying data fresh from the master, it should not be possible that we miss a "real" error in this way. Will things work correctly, if we have multiple rows in a single Rows_log_event, and some of them succeed while others fail? In both MyISAM and InnoDB? Can an argument be made to prove that, if we ignore an insert or update here, we are sure to later write the correct data as part of provisioning? For both InnoDB and MyISAM? Similarly, can it be proven that it is impossible to ignore a delete and still later wrongly provision the deleted row? I suppose it would be something with binlog ordering and isolation modes for the InnoDB case - and some table locking semantics wrt. MyISAM? > + > + /* > + Semaphore used by LOAD DATA FROM MASTER command for test cases, its role is > + to ensure, that slave's IO thread is connected to master before test case > + can continue. > + volatile bool dump_requested_semaphore; Shouldn't this be under #ifndef DBUG_OFF ? > diff --git a/sql/rpl_provisioning.cc b/sql/rpl_provisioning.cc > +void quote_name(char const *name, char *buff) Try to use append_identifier() instead. Or maybe my_snprintf("%`s"). I notice that append_identifier() handles a configuration where "" is used to quote identifiers instead of ``, you should probably test if this causes a bug in your code, btw. > +provisioning_send_info::provisioning_send_info(THD *thd_arg) > + : thd(thd_arg) > + , connection(new Ed_connection(thd)) > + , row_batch_end(NULL) > + , row_buffer(my_malloc(ROW_BUFFER_DEFAULT_SIZE, MYF(0))) > + , row_buffer_size(ROW_BUFFER_DEFAULT_SIZE) > + , phase(PROV_PHASE_INIT) > + , error(0) > + , error_text(NULL) > + , event_conversion_cache(NULL) > +{ > +} So this constructor will probably cause the server to crash in case of out-of-memory, right? Usually, code tries to fail gracefully with ER_OUT_OF_RESOURCES / ER_OUTOFMEMORY if malloc() fails (though its not well tested, probably). > +void provisioning_send_info::close_tables() > +{ > + trans_commit_stmt(thd); I see no call to commit the entire transaction (as opposed to just the statement here). This may be intentional (autocommit?), in any case please add a comment here explaining how the transaction is committed wrt. statement-transaction and full transaction. Maybe add a test where the server is running with auto-commit globally disabled? Though I am not sure if the short mysql-test-run.pl tests would notice it if everything was done as one long transaction (though large production sites undobtedly would ;-). > +bool provisioning_send_info::send_query_log_event(DYNAMIC_STRING const *query, Is there a particular reason that you chose DYNAMIC_STRING over the String class? I think String is more widely used for building queries and such? A common use pattern with String is to allocate a reasonably-sized buffer on the stack and initialise a String object from it. Then in the common case of smallish result, no my_malloc() calls are needed. > + if (cs_info) > + { > + // thd->variables.character_set_client->number > + int2store(evt.charset, cs_info->cs_client); > + // thd->variables.collation_connection->number > + int2store(evt.charset + 2, cs_info->cl_connection); > + // cs_info->cl_server > + int2store(evt.charset + 4, thd->variables.collation_server->number); > + // thd->variables.collation_database->number > + evt.charset_database_number= cs_info->cl_db; Can you explain why you need to consider the charsets/collations of the connection and client? Could you not just always use the system character set for generating queries? > + if (my_strcasecmp(system_charset_info, "VIEW", type->str) == 0) > + { > + char name_buff[NAME_LEN * 2 + 3]; > + DYNAMIC_STRING *str= (DYNAMIC_STRING*)my_malloc(sizeof(DYNAMIC_STRING), > + MYF(0)); > + > + quote_name(database->str, name_buff); > + init_dynamic_string(str, name_buff, > + // Both lengths + dot and quotes, will be reallocated > + // only in special cases > + database->length + name->length + 5, > + 5); > + dynstr_append(str, "."); > + quote_name(name->str, name_buff); > + dynstr_append(str, name_buff); > + > + views.push_back(str); > + } > + else if (my_strcasecmp(system_charset_info, "BASE TABLE", type->str) == 0) > + tables.push_back(my_strdup(name->str, MYF(0))); Why do you need to prepend the database name in VIEW case but not in TABLE case? Please add a comment explaining that. > + else > + DBUG_ASSERT(false && > + "Unexpected type of table returned by 'SHOW FULL TABLES'"); Probably you should add an error return here, that will cause provisioning to fail in the non-debug build, rather than silently press on. > +bool provisioning_send_info::event_to_packet(Log_event &evt, String &packet) > +{ > + // Reset cache for writing > + reinit_io_cache(event_conversion_cache, WRITE_CACHE, 0, false, true); > + > + if (evt.write(event_conversion_cache)) > + { > + error_text= "Failed to write event to conversion cache"; > + return true; > + } > + > + if (reinit_io_cache(event_conversion_cache, READ_CACHE, 0, false, false)) > + { > + error_text= "Failed to switch event conversion cache mode"; > + return true; > + } > + > + packet.set("\0", 1, &my_charset_bin); > + if (packet.append(event_conversion_cache, > + event_conversion_cache->write_pos - event_conversion_cache->buffer)) > + { > + error_text= "Failed to write event data to packet"; > + return true; > + } Hm. So first the event is constructed in-memory. Then it is written to an IO_CACHE. Then it is read back in again into a byte buffer. And then it is written to the slave's client socket. Hm, I see, the client socket we have only as a NET, not an IO_CACHE. But the Log_event subclasses need an IO_CACHE to construct a serialised event. Hm, that's clunky, but I don't see an easy way to avoid it. But please add a comment explaining why this is necessary. Also, doesn't the IO_CACHE spill to disk if the data written becomes large? So wouldn't a very large event (eg. a row containing a large BLOB) fail here? Because the IO_CACHE only contains part of the data in-memory? I am not 100% sure about how this works, please investigate and test with a large row (if you did not already). > + // Set current server as source of event, when slave registers on master, > + // it overwrites thd->variables.server_id with its own server id - and it is > + // then used when writing event > + int4store(&packet[SERVER_ID_OFFSET + 1], global_system_variables.server_id); > + return false; I think this will fail if event checksums are enabled? Or rather, the patch does not seem to handle binlog checksums at all. Try adding a test that enables binlog checksums on master and slave, and check that everything works. And also - why do you need to change this in the first place? Doesn't the server_id get set correctly when the event is initially created? > +bool provisioning_send_info::send_event(Log_event &evt) > +{ > + String packet; > + > + // Slave will accept this event even if it modifies data structures > + evt.flags|= LOG_EVENT_PROVISIONING_F; > + > + if (event_to_packet(evt, packet)) > + return true; > + > + if (my_net_write(&thd->net, (uchar*)packet.ptr(), packet.length())) > + { > + error_text= "Failed to send event packet to slave"; > + return true; > + } Yeah... It would be so much more elegant if we could wrap the thd->net in an IO_CACHE, and write directly to the client. Did you investigate if this would be a possibility, without too much extra work? > +int8 provisioning_send_info::send_table_data() > + if ((error= hdl->prepare_range_scan(row_batch_end, NULL)) != 0) > + if (table->s->primary_key == MAX_KEY) > + { > + error_text= "Table does not contain primary key"; > + close_tables(); > + return 1; > + } It seems more natural to me to check for missing primary key _before_ prepare_range_scan(). (Otherwise one wonders how it is possible to prepare a range scan on a table without _any_ key? I suppose it works to do a table scan instead, but better do the check first to avoid having to wonder). > + while (packed_rows < PROV_ROW_BATCH_SIZE && > + (error= hdl->read_range_next()) == 0); I think there should be two stop criterias. One is max number of rows reached, another is size of rows. At least, you need to protect against exceeding --max-packet-size. Otherwise the slave will be unable to receive the too large packet. Please add a test case that checks large rows and that it correctly splits row events to not exceed max-packet-size (of course at least one row must fit in one Rows_log_event). Check the code for normal binlogging to see how it splits up Row_log_events to avoid creating too large an event. > + table->default_column_bitmaps(); > + close_tables(); So the critical issue here is: how can we be sure that we will not miss rows during provisioning, by racing between provisioning events and normal binlog events? This needs to be clearly thought out, and then explained in a comment somewhere in the code. For example, you write the events to the slave client socket before closing the table and releasing locks. This seems critical (and correct) - because this ensures that if there is a DELETE made just after the provisioning read of the table, that DELETE will be sent only _after_ sending the provisioning rows (so we do not lose the DELETE). This should be explained in a comment. But what about isolation level (for InnoDB)? Don't we need to run with a sufficiently high isolation level to be correct? Intuitively, I thought so, but maybe not, at least you have nothing in the patch about isolation level that I can see. What about this scenario? 1. Provisioning reads rows 20-29, sends them to slave. 2. An UPDATE runs, it updates row 32 to be now row 25, it is written to the binary log and sent to the slave. 3. Provisioning reads rows 30-39, sends them to the slave. On the slave, the UPDATE from 2 should fail - because row 32 does not exist at this point. But then, it seems to me that the row 25 will never appear on the slave, right? So provisioning will be incorrect. Or am I missing something? I wonder if the solution to this particular case is that during provisioning, an UPDATE (at least one that changes primary key value) must be logged as a DELETE (of the old row) + an INSERT (of the new row). Alternatively, maybe the slave can during provisioning execute the UPDATE regardless of the missing before image, still writing the after image (row 25) to the table. More generally, we really need an explanation of how the provisioning ensures that the end result on the slave is identical to the master - otherwise there may be other problems like this hidden. Also, it would be really good if you could make a test case for the above potential issue. You can get some ideas from how I use DBUG_EXECUTE_IF together with DEBUG_SYNC in the rpl_parallel_*.test test cases. But I know it can be a lot of work to get such tricky race-proviking test cases to work reliably. > diff --git a/sql/rpl_provisioning.h b/sql/rpl_provisioning.h > +// FIXME - Farnham > +// Header > + > +#pragma once We do not use this #pragma once in mariadb (maybe that is what you intended to fix with the FIXME, so mentioning just in case). I assume you will take care of other such FIXME's in the patch yourself before pushing. > +/* > + How much rows from table will be sent in one provisioning tick > + FIXME - Farnham > + Dynamic value > +*/ > +#define PROV_ROW_BATCH_SIZE 5 Right. Probably just a matter of adding something to sql/sys_vars.cc > - if (rpl_load_gtid_slave_state(thd)) > + if (!mi->provisioning_mode && rpl_load_gtid_slave_state(thd)) Actually, what should happen with the GTID position for LOAD DATA FROM MASTER? Just clearing it probably is not correct, in case user wants to LOAD DATA FROM MASTER from multiple masters. Maybe the user should clear it herself, if necessary, and then it will be updated by the normal replication stream from master. But why do you skip loading it here for provisioning mode? That does not seem right. First, it will normally be loaded already anyway. And second, if not loaded, then just having the wrong value in-memory (as compared to in the mysql.gtid_slave_pos table) does not seem correct? > + DBUG_EXECUTE_IF("provisioning_test_running", > + if (!slave_errno) > + while (!mi->dump_requested_semaphore) > + /* pass */;); Maybe put a my_sleep(20000) in with the /* pass */. > + case PROVISIONING_DONE_EVENT: > + mi->abort_slave= true; > + /* > + Fall through to default case ... and through debug code, nothing wrong > + should happen > + */ That seems a bit fragile. Better do like in other places: goto default_action; > + // In provisioning mode, master starts sending data from current position > + if (flags & BINLOG_DUMP_PROVISIONING_MODE) So provisioning assumes non-GTID mode? Maybe that is ok... But it should still be tested what happens if provisioning is attempted in GTID mode. Perhaps either GTID or non-GTID mode can work the same, since provisioning does not need to start at any particular position - and at the end of provisioning, both GTID and non-GTID position should be correct. > +bool > +Ed_connection::execute_direct(char *sql_text, size_t length) > +{ > + LEX_STRING tmp_query; > + tmp_query.str= sql_text; > + tmp_query.length= length ? length : strlen(sql_text); > + > + return execute_direct(tmp_query); > +} You are not actually ever calling this without passing length. So please make this an inline method in the .h file, with mandatory length argument, that does not check for zero length or call strlen(). - Kristian.

1 0

[Maria-developers] Fixing Some Trivial Bugs
by Alangi Derick 21 Aug '15

21 Aug '15

Hi, I just found some trivial bugs on jira that i will be fixing for the time being. Once i am done with each one, i will submit a patch or push to my forked repo so it can be merged. My question is, after i fix the bugs, will they accepted, if yes i should continue working on them thanks. I need a respond from the development team. Regards Alangi Derick Ndimnain

3 4