- developers - lists.mariadb.org

Re: [Maria-developers] e91559c: MDEV-7006 MDEV-7007: SET STATEMENT and slow log
by Sergei Golubchik 16 Feb '15

16 Feb '15

Hi, Oleksandr! On Jan 28, Oleksandr Byelkin wrote: > revision-id: e91559cf7f2b19378e9e5d3f72c3a6794a09d1e0 > parent(s): cbc318fcf3680c01dca1320b87f625124a497528 > committer: Oleksandr Byelkin > branch nick: work-maria-10.1-MDEV-6997 > timestamp: 2014-11-26 11:54:29 +0100 > message: > > MDEV-7006 MDEV-7007: SET STATEMENT and slow log > fixed embedded server tests > > old values of SET STATENENT variables now saved in its own Query_arena and restored later Basically, I have two comments: 1. I think it might be kind of expensive to create MEM_ROOT for every SET STATEMENT. And perhaps a user will never need to set-statement these log variables. I don't see a good solution for that. Either we can drop the support for log related variables in SET STATEMENT, basically not using your patch. I wouldn't like that, from a user point of view this is a pretty artificial limitation, he doesn't care about memroots. Or we can only create memroot for log related variables and put other variables in the statement memroot, which also means restoring values in two steps. Seems like too complex for a solution. A compromise solution would be to do almost like in your patch, but not to free memroot after every statement. That is, a memroot is created once, for the first SET STATEMENT. At the end of the statement you only MY_MARK_BLOCKS_FREE or MY_KEEP_PREALLOC. Then the next SET STATEMENT wouldn't need to allocate that much. Not perfect, but might do as a tradeoff. Or, perhaps you can suggest a better solution. 2. I didn't quite like that you've added restore_set_statement_var, reset_arena_for_set_stmt, free_arena_for_set_stmt, etc everywhere. I'd prefer you to add them to one function which is called from all these places. Like, there must be some cleanup-after-statement function that you should be able to use. > diff --git a/mysql-test/t/set_statement.test b/mysql-test/t/set_statement.test > index a658071..108c1a3 100644 > --- a/mysql-test/t/set_statement.test > +++ b/mysql-test/t/set_statement.test > @@ -995,6 +985,57 @@ set global general_log=0; > set statement lock_wait_timeout=1 for select @@lock_wait_timeout; > set global general_log=@save_general_log; > > +--echo # MDEV-7006 MDEV-7007: SET statement and slow log > + > +set @save_long_query_time= @@long_query_time; > +set @save_slow_query_log= @@slow_query_log; > +set @save_log_output= @@log_output; > + > +set statement long_query_time=default for select @@long_query_time; > +set statement log_slow_filter=default for select @@log_slow_filter; > +set statement log_slow_verbosity=default for select @@log_slow_verbosity; > +set statement log_slow_rate_limit=default for select @@log_slow_rate_limit; > +set statement slow_query_log=default for select @@slow_query_log; > + > +truncate table mysql.slow_log; > +set slow_query_log= 1; > +set global log_output='TABLE'; > + > +select sql_text from mysql.slow_log; > +set @@long_query_time=0.01; > +--echo #should be written > +select sleep(0.1); > +set @@long_query_time=@save_long_query_time; > +select sql_text from mysql.slow_log; > +--echo #--- > +--echo #should be written > +set statement long_query_time=0.01 for select sleep(0.1); > +select sql_text from mysql.slow_log; Okay, I hope it'll work, but please do try it in a separate branch in buildbot. all these timing-related tests have proven to be very fragile, so I've learned to be very cautious about these stuff. even in test that looks safe to me :) > +--echo #--- > +set @@long_query_time=0.01; > +--echo #should NOT be written > +set statement slow_query_log=0 for select sleep(0.1); > +set @@long_query_time=@save_long_query_time; > +select sql_text from mysql.slow_log; > +--echo #--- > +--echo #should be NOT written > +set statement long_query_time=0.01,log_slow_filter='full_scan' for select sleep(0.1); > +select sql_text from mysql.slow_log; > +--echo #--- > +--echo #should be NOT written > +set statement long_query_time=0.01,log_slow_rate_limit=9999 for select sleep(0.1); > +select sql_text from mysql.slow_log; > +--echo #--- > +# > +# log_slow_verbosity is impossible to check because results are not written > +# in TABLE mode > +# > + > +set global log_output= @save_log_output; > +set @@slow_query_log= @save_slow_query_log; > +set @@long_query_time= @save_long_query_time; > + > + > # > # Prohibited Variables > # > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 6432e19..08094ce 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -4235,6 +4236,53 @@ int LEX::print_explain(select_result_sink *output, uint8 explain_flags, > return res; > } > > +void LEX::set_arena_for_set_stmt(Query_arena *backup) > +{ > + DBUG_ENTER("LEX::set_arena_for_set_stmt"); > + DBUG_ASSERT(arena_for_set_stmt== 0); > + MEM_ROOT *mem_root= new MEM_ROOT(); > + if (!(mem_root)) > + DBUG_VOID_RETURN; > + init_sql_alloc(mem_root, ALLOC_ROOT_MIN_BLOCK_SIZE, 0, > + MYF(MY_THREAD_SPECIFIC)); > + if (!(arena_for_set_stmt= new Query_arena(mem_root, > + Query_arena::STMT_INITIALIZED))) > + DBUG_VOID_RETURN; 1. allocate a new memroot for every SET STATEMENT? 2. no error reporting? > + DBUG_PRINT("info", ("mem_root: 0x%lx arena: 0x%lx", > + (ulong) mem_root, (ulong) arena_for_set_stmt)); > + thd->set_n_backup_active_arena(arena_for_set_stmt, backup); > + DBUG_VOID_RETURN; > +} > + > + > +void LEX::reset_arena_for_set_stmt(Query_arena *backup) > +{ > + DBUG_ENTER("LEX::reset_arena_for_set_stmt"); > + DBUG_ASSERT(arena_for_set_stmt); > + thd->restore_active_arena(arena_for_set_stmt, backup); > + DBUG_PRINT("info", ("mem_root: 0x%lx arena: 0x%lx", > + (ulong) arena_for_set_stmt->mem_root, > + (ulong) arena_for_set_stmt)); > + DBUG_VOID_RETURN; > +} > + > + > +void LEX::free_arena_for_set_stmt() > +{ > + DBUG_ENTER("LEX::free_arena_for_set_stmt"); > + if (!arena_for_set_stmt) > + return; > + DBUG_PRINT("info", ("mem_root: 0x%lx arena: 0x%lx", > + (ulong) arena_for_set_stmt->mem_root, > + (ulong) arena_for_set_stmt)); > + arena_for_set_stmt->free_items(); > + free_root(arena_for_set_stmt->mem_root, MYF(0)); > + delete(arena_for_set_stmt->mem_root); > + delete(arena_for_set_stmt); Eh... The arena is allocated in the memroot. I don't think you can free the memroot before the arena. Probably something like this: MEM_ROOT *memroot= arena_for_set_stmt->mem_root; delete arena_for_set_stmt; free_root(memroot, MYF(0)); delete memroot; > + arena_for_set_stmt= 0; > + DBUG_VOID_RETURN; > +} > + > void LEX::restore_set_statement_var() > { > DBUG_ENTER("LEX::restore_set_statement_var"); > @@ -4243,7 +4291,9 @@ void LEX::restore_set_statement_var() > DBUG_PRINT("info", ("vars: %d", old_var_list.elements)); > sql_set_variables(thd, &old_var_list, false); > old_var_list.empty(); > + free_arena_for_set_stmt(); > } > + DBUG_ASSERT(!is_arena_for_set_stmt()); the indentation is broken like this in many parts of the patch. May be that's because you've forwarded it with your MUA... > DBUG_VOID_RETURN; > } > > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 5af700b..ae831d0 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -1956,6 +1958,7 @@ bool dispatch_command(enum enum_server_command command, THD *thd, > > /* Check that some variables are reset properly */ > DBUG_ASSERT(thd->abort_on_warning == 0); > + thd->lex->restore_set_statement_var(); too bad you had to put these restore_set_statement_var, reset_arena_for_set_stmt, free_arena_for_set_stmt, etc everywhere explicitly. > DBUG_RETURN(error); > } > Regards, Sergei

2 2

[Maria-developers] Fwd: Contributing in MariaDB Project
by Rohit Kashyap 13 Feb '15

13 Feb '15

Hi, I am Rohit from NIT Patna, India. A Engineering Student and contributor in MySQL Community, approved by Oracle. You guys are doing great things at Team Maria DB. Kudos for that. I would like to start and contribute in the Maria project, and would like to ask for guiding in the same, with code base and introductory Bugs to prove myself. -- Kind Regards, Rohit Freenode: rkgudboy -- Kind Regards, Rohit

2 5

Re: [Maria-developers] Row-based replication of old DECIMAL to new DECIMAL
by Sergei Golubchik 13 Feb '15

13 Feb '15

Hi, Alexander! On Dec 22, Alexander Barkov wrote: > Hi Sergei, Kristian, > > I'm looking at: > > MDEV-7268 Column of table cannot be converted from type 'decimal(0,?)' > to type ' 'decimal(10,7)' > > This bug cannot be fixed in general case, because the old DECIMAL data > type does not write its metadata into the binary log. > > I suggest we don't fix replication of old DECIMAL > (neither binlogging, nor slave-side). I think that's reasonable. But then CHECK TABLE ... FOR UPGRADE should issue a warning for old DECIMAL columns. > Instead, we fix "ALTER TABLE ... FORCE" (and thus mysql_upgrade) > to force changing old DECIMAL to new DECIMAL. > Currently, "ALTER TABLE...FORCE" preserve old DECIMAL. > > Not sure which version is best to fix in. > The complainer had problems with 5.5. I don't know, I'd say 10.0. For the complainer there's a workaround - I'm sure you can come up with an ALTER TABLE that changes columns from old to new DECIMAL, may be something like ALTER TABLE xxx MODIFY COLUMN yyy DECIMAL(....) Regards, Sergei

7 15

[Maria-developers] Crash during configure without development SSL libraries installed
by Vicențiu Ciorbaru 12 Feb '15

12 Feb '15

Hi Sergei! The following error occurs while trying to configure MariaDB 10.0 using the command "cmake ." without the development SSL libraries installed. I've attached the patch to fix the errors. ERROR LOG: ---------------------- CMake Error at cmake/ssl.cmake:162 (FILE): file STRINGS file "/home/vicentiu/Workspace/MariaDB/10.0/OPENSSL_INCLUDE_DIR-NOTFOUND/openssl/opensslv.h" cannot be read. Call Stack (most recent call first): CMakeLists.txt:356 (MYSQL_CHECK_SSL) Regards, Vicentiu

2 1

[Maria-developers] Fwd: MDEV-6838 Using too big key for internal temp tables
by Vicențiu Ciorbaru 12 Feb '15

12 Feb '15

+ CC maria-developers Hi Sergei! I have attached the diff for MDEV-6838. Let me know your thoughts on it. The test case now fails without the patch and passes with the patch on both 5.5 and 10.0. Sergey Petrunia is alright with the patch. Regards, Vicențiu

2 2

[Maria-developers] Interest in Contributing to MariaDB and participation in GSoC 2015
by Tekang Check 11 Feb '15

11 Feb '15

Hello, I am Tekang Mengwe, a third year Accounting student at the University of Buea. I am interested in contributing code to MariaDB and participating in the 2015 round of the Google Summer of Code. While looking at the projects. I'm interested in Database replication and would love to do some work in this respect. I am proficient in C/C++ and code as a hobby. I will greatly appreciate any assistance in this respect. Regards, Tekang

6 7

[Maria-developers] Debian and reproducible builds errors for MariaDB
by Otto Kekäläinen 06 Feb '15

06 Feb '15

Hello! Can somebody please help me with this, preferrably send me a patch I can add to the Debian MariaDB 10.0 packaging tree? Debian has this new test suite for reproducible builds: https://reproducible.debian.net/reproducible.html MariaDB has notes while MySQL is OK (no notes): https://reproducible.debian.net/index_notes.html MariaDB should be at least as good as MySQL, right? MariaDB notes: https://reproducible.debian.net/rb-pkg/mariadb-10.0.html The errors in the build log https://reproducible.debian.net/rbuild/mariadb-10.0_10.0.16-1.rbuild.log look like this: /tmp/buildd/mariadb-10.0-10.0.16/storage/connect/ha_connect.cc:174:46: error: macro "__DATE__" might prevent reproducible builds [-Werror=date-time] char compver[]= "Version 1.03.0006 " __DATE__ " " __TIME__; ^ /tmp/buildd/mariadb-10.0-10.0.16/storage/connect/ha_connect.cc:174:60: error: macro "__TIME__" might prevent reproducible builds [-Werror=date-time] char compver[]= "Version 1.03.0006 " __DATE__ " " __TIME__; ^ Could somebody look into this and send me a patch that fixes it? (and commit the same patch upstream, ofcourse) PS. While searching the log for errors I also noticed these unrelated ones, how do we fix them?: /tmp/buildd/mariadb-10.0-10.0.16/builddir /tmp/buildd/mariadb-10.0-10.0.16/builddir/storage/perfschema /tmp/buildd/mariadb-10.0-10.0.16/builddir/storage/perfschema/CMakeFiles/perfschema.dir/DependInfo.cmake --color= troff: fatal error: can't find macro file m /tmp/buildd/mariadb-10.0-10.0.16/builddir /tmp/buildd/mariadb-10.0-10.0.16/builddir/storage/federatedx /tmp/buildd/mariadb-10.0-10.0.16/builddir/storage/federatedx/CMakeFiles/federatedx.dir/DependInfo.cmake --color= troff: fatal error: can't find macro file m

2 1

[Maria-developers] storage engine index usage
by Andras Szabo 06 Feb '15

06 Feb '15

Hi Folks, A new question has risen during the development of our storage engine. In short the engine is providing a read-only view to the server filesystem. One can issue (implemented via assisted discovery): CREATE TABLE etc ENGINE=fsview ROOT=‘/etc’; Then it’s possible to tell: SELECT name, fsize FROM etc ORDER BY fsize DESC LIMIT 10 resulting in: +-----------------+--------+ | name | fsize | +-----------------+--------+ | ld.so.cache | 129417 | | mailcap | 44732 | | fb.modes | 24501 | | mime.types | 24066 | | brltty.conf | 22478 | | services | 19558 | | devscripts.conf | 18037 | | ltrace.conf | 14867 | | login.defs | 10551 | | sensors3.conf | 10344 | +-----------------+--------+ So far so good (the listing is not recursive). An additional feature is to access subdirectories of the root (this is working via automated discovery): SELECT name, fsize FROM `etc/init.d` This is not terribly wrong, but we’re thinking about the possiblity of improving it somehow that: SELECT name, fsize FROM etc WHERE path = “init.d”; However we’re in a bit of trouble how we could improve the storage engine to be able to handle this. Of course we would not like to do a "deep enumeration" of the root directory (full table scan) . So somehow we would like to access/use the information from the where clause. Can anybody recommend a way to solve that? -- Andras Szabo Sent with Airmail

3 2

Re: [Maria-developers] ping
by Sergei Golubchik 05 Feb '15

05 Feb '15

Hi, Vicențiu! Next time, please, cc: (or to: ) your commit to commits(a)mariadb.org. Or use the post-commit trigger from http://bazaar.launchpad.net/~maria-captains/mariadb-tools/trunk/files/head:… > commit 4ede7cdbe51332d1b07def8830974877ebcee936 > Author: Vicențiu Ciorbaru <cvicentiu(a)gmail.com> > Date: Wed Feb 4 23:08:05 2015 +0000 > > MDEV-6918 Create a way to see a user's default role. > > Added an extra column to i_s_applicable_roles, named IS_DEFAULT. > The column displays which role is the default role for the user > querying the table. Looks good, thanks! A couple of minor comments, see below. Ok to push after that. > diff --git a/mysql-test/suite/roles/i_s_applicable_roles_is_default.test b/mysql-test/suite/roles/i_s_applicable_roles_is_default.test > new file mode 100644 > index 0000000..1782f7a > --- /dev/null > +++ b/mysql-test/suite/roles/i_s_applicable_roles_is_default.test > @@ -0,0 +1,61 @@ I'd suggest to start the test with --enable_connect_log it'll make the result file a bit easier to follow. > +create user foo; > +create role role1; > +create role role2; > +create role role3; > + > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 96fff6d..45ce43d 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -10349,6 +10349,17 @@ applicable_roles_insert(ACL_USER_BASE *grantee, ACL_ROLE *role, void *ptr) > else > table->field[2]->store(STRING_WITH_LEN("NO"), cs); > > + /* Default role is only valid when looking at a role granted to a user. */ > + if (!is_role) > + { > + if (!is_role && data->user->default_rolename.length && you don't need a second !is_role under if(!is_role). I suppose that's a remnant of the first version of your patch, when you returned NO for roles granted to roles :) > + !strcmp(data->user->default_rolename.str, role->user.str)) > + table->field[3]->store(STRING_WITH_LEN("YES"), cs); > + else > + table->field[3]->store(STRING_WITH_LEN("NO"), cs); > + table->field[3]->set_notnull(); > + } > + > if (schema_table_store_record(table->in_use, table)) > return -1; > return 0; Regards, Sergei

1 0

[Maria-developers] Simplify and stabilize build tests in downstream distributions?
by Otto Kekäläinen 05 Feb '15

05 Feb '15

Hello Elena, I asked you on IRC in December about if the mtr run, as it is now called from debian/rules, is optimal and sensible for in-distro build time testing? Canonical staff did these changes to MySQL 5.6, do you think we should follow suite? https://github.com/basak/debian-mysql/commit/6eb941e0f6fc9de35468dc59605208… - see only line 142 https://github.com/basak/debian-mysql/commit/4710f5a3b429c640176fd45b7c4b70… https://github.com/basak/debian-mysql/commit/78d6a4ac03f2e4d1553cc35574fc78…

2 2

Re: [Maria-developers] [Commits] 7320cbf: MDEV-7177 - Server hangs on shutdown after InnoDB error (main.plugin_loaderr
by Sergei Golubchik 04 Feb '15

04 Feb '15

Hi, Sergey! On Feb 02, svoj(a)mariadb.org wrote: > revision-id: 7320cbf85ddfaf7ef76c8f03718819377fd14318 > parent(s): 11deb416fe12ebfd65a1f089c7eb6087c192e2dd > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2015-02-02 15:41:49 +0400 > message: > > MDEV-7177 - Server hangs on shutdown after InnoDB error (main.plugin_loaderr > fails in buildbot) > > There was a race condition in timer functionality of query timeouts. > > Race was as following: > main thread: init_thr_timers() > timer handler thread: my_thread_init() > main thread: end_thr_timer()/timer_thread_state= ABORTING > timer handler thread: timer_thread_state= RUNNING, continue normal op > main thread: waits indefinitely for timer handler thread to go down > > The original idea of the fix is to set RUNNING state in main thread, before > starting timer handler thread. But it didn't survive further cleanups: > - removed "timer_thread_state" and used "thr_timer_inited" for this purpose > - removed unused "timer_thread_running" > - removed code responisble for "timer handler thread" shutdown synchronization, > use pthread_join() instead. That looks ok to me. The only concern is - in end_thr_timer(), the old code had a timeout waiting for a timer thread to die. pthread_join() doesn't. I don't know if that's a problem, perhaps not. Regards, Sergei

2 1

Re: [Maria-developers] [Commits] 11deb41: MDEV-7499 - System variables have broken default values on big endian
by Sergei Golubchik 04 Feb '15

04 Feb '15

Hi, Sergey! On Jan 30, svoj(a)mariadb.org wrote: > revision-id: 11deb416fe12ebfd65a1f089c7eb6087c192e2dd > parent(s): edf34f38ac4fad7996bf19cd9ac669d2a6825400 > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2015-01-30 16:13:28 +0400 > message: > > MDEV-7499 - System variables have broken default values on big endian > > INFORMATION_SCHEMA.SYSTEM_VARIABLES.DEFAULT_VALUE had broken values on > big endian. > > Default value is internally stored as longlong, while I_S references it's > pointer (longlong *) according to variable type (e.g. int, my_bool, etc). This > works well on little endian, but on big endian we always get 0 for such > variables. Ok to push, thanks! One suggestion below > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 46849e2..1928df5 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -3254,7 +3254,29 @@ static int pluginvar_sysvar_flags(const st_mysql_sys_var *p) > uchar* sys_var_pluginvar::real_value_ptr(THD *thd, enum_var_type type) > { > if (type == OPT_DEFAULT) > - return (uchar*)&option.def_value; > + { > + switch (plugin_var->flags & PLUGIN_VAR_TYPEMASK) { > + case PLUGIN_VAR_BOOL: > + thd->sys_var_tmp.my_bool_value= option.def_value; > + return (uchar*) &thd->sys_var_tmp.my_bool_value; > + case PLUGIN_VAR_INT: > + thd->sys_var_tmp.int_value= option.def_value; > + return (uchar*) &thd->sys_var_tmp.int_value; > + case PLUGIN_VAR_LONG: > + case PLUGIN_VAR_ENUM: > + thd->sys_var_tmp.long_value= option.def_value; > + return (uchar*) &thd->sys_var_tmp.long_value; > + case PLUGIN_VAR_LONGLONG: > + case PLUGIN_VAR_SET: > + case PLUGIN_VAR_DOUBLE: > + return (uchar*) &option.def_value; PLUGIN_VAR_LONGLONG and PLUGIN_VAR_SET - ok, but for PLUGIN_VAR_DOUBLE I'd either copy it to thd->sys_var_tmp too or add an assert, like this case PLUGIN_VAR_DOUBLE: compile_time_assert(sizeof(double) == sizeof(option.def_value)); /* fall through */ case PLUGIN_VAR_LONGLONG: case PLUGIN_VAR_SET: return (uchar*) &option.def_value; More for documentation purposes than for protection agains a wrong sizeof(double). It explains why you don't copy the value. > + case PLUGIN_VAR_STR: > + thd->sys_var_tmp.ptr_value= (void*) option.def_value; > + return (uchar*) &thd->sys_var_tmp.ptr_value; > + default: > + DBUG_ASSERT(0); > + } > + } > Regards, Sergei

2 1

Re: [Maria-developers] [Commits] 101d7eb: MDEV-4987: Sort by domain_id when list of GTIDs are output
by Kristian Nielsen 03 Feb '15

03 Feb '15

nirbhay(a)mariadb.com writes: > revision-id: 101d7eb963816514362da8a98fc7db7135181910 > committer: Nirbhay Choubey > timestamp: 2015-01-31 21:48:14 -0500 > message: > > MDEV-4987: Sort by domain_id when list of GTIDs are output > > Added logic to sort gtid list based on domain_id before > populating them in string. Added a test case. Thanks! The patch looks ok to push after fixing / considering below in-line comments: > diff --git a/mysql-test/suite/rpl/t/rpl_gtid_sort.test b/mysql-test/suite/rpl/t/rpl_gtid_sort.test > new file mode 100644 > index 0000000..d3b8c6d > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_gtid_sort.test > +--connection server_2 > +SHOW VARIABLES LIKE '%GTID%'; > +#SET GLOBAL gtid_slave_pos=""; This commented-out SET looks like some left-over, probably you meant to remove it? > diff --git a/sql/rpl_gtid.cc b/sql/rpl_gtid.cc > index e5620ec..a0471f3 100644 > --- a/sql/rpl_gtid.cc > +++ b/sql/rpl_gtid.cc > @@ -247,11 +247,13 @@ > { > my_hash_init(&hash, &my_charset_bin, 32, offsetof(element, domain_id), > sizeof(uint32), NULL, rpl_slave_state_free_element, HASH_UNIQUE); > + my_init_dynamic_array(&gtid_cache, sizeof(rpl_gtid), 8, 8, MYF(0)); The name 'cache' confused me slightly, as I think of a cache as something that stores some state between invocations. But it is actually just a temporary buffer used for sorting. A suggestion for a better name could be 'gtid_sort_array'. > rpl_slave_state::~rpl_slave_state() > { > + delete_dynamic(&gtid_cache); > } The other parts of rpl_slave_state are freed in rpl_slave_state::deinit(). I think this is because there is a global variable of this class, and destructors in global variables can cause various issues. So better put this delete_dynamic() into deinit() as well (for consistency, if nothing else). > @@ -705,7 +707,13 @@ class Gtid_db_intact : public Table_check_intact > return sub_id; > } > > +/* A callback used in sorting of gtid list based on domain_id. */ > +static int rpl_gtid_cmp_cb(const void *id1, const void *id2) > +{ > + return (((rpl_gtid *)id1)->domain_id - ((rpl_gtid *)id2)->domain_id); > +} I think this is wrong, you can get signed/unsigned integer overflow (eg. it will consider 0xffffffff as being less than 0x00000000). We might be able to do the subtraction in longlong with a cast (though that would still be susceptible to overflow when converting to the int return value, right?), but it's probably clearer to just do: uint32 d1= ((rpl_gtid *)id1)->domain_id; uint32 d2= ((rpl_gtid *)id2)->domain_id; if (d1 < d2) return -1; else if (d1 > f2) return 1; else return 0; You should probably also modify the test case to have a domain id > 0x80000000, just to check. Otherwise looks good, thanks! - Kristian.

2 2

[Maria-developers] custom storage backend rnd_next() implementation and caching questions
by Andras Szabo 03 Feb '15

03 Feb '15

Hi Everyone, I am developing a custom mariadb storage backend which is intended to give filesystem related information via mariadb tables. For example one could tell: CREATE TEMPORARY TABLE IF NOT EXISTS `/var/log` ENGINE=fsview; which would create a table (using the assisted discovery method) with the following schema: MariaDB [test]> describe `/var`; +-----------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-----------+--------------+------+-----+---------+-------+ | file_name | varchar(255) | YES | | NULL | | | contents | longblob | YES | | NULL | | +-----------+--------------+------+-----+---------+———+ However i have some implementation issues: 1. I am having trouble implementing rnd_next() in the backend: int ha_fsview::rnd_next(uchar *buf) { DBUG_ENTER(__PRETTY_FUNCTION__); if (!dirp) { DBUG_RETURN(HA_ERR_INTERNAL_ERROR); } if (struct dirent* entry = readdir(dirp)) { uchar *pos = buf; *pos = 0; ++pos; // file name (1bytes size + str) *pos = strlen(entry->d_name); ++pos; size_t bytes_to_copy = strlen(entry->d_name); memcpy(pos, entry->d_name, bytes_to_copy); pos+=bytes_to_copy; DBUG_PRINT("info", ("fsview filename: %s", entry->d_name)); // content (4 bytes size + content ptr) static const std::string fake_content = "hello world"; int s = fake_content.size(); memcpy(pos, &s, sizeof(s)); pos+=sizeof(s); const char* c = fake_content.c_str(); memcpy(pos, &c, sizeof(c)); pos+=sizeof(c); DBUG_RETURN(0); } else { DBUG_RETURN(HA_ERR_END_OF_FILE); } } My problem is that filenames (first field) is ok, but i cannot really see the expected content “hello world” in the second field. I am assuming that i am doing something wrong, but i cannot really find the reason. Could somebody help me, please? 2. If i am issuing: SELECT * FROM `/var`; twice, then the server is not initiating a table scan again on my backend (most likely results from the first run are stored in some cache). So if filesystem contents are changed between the two statements, then it won’t be reflected in the resultset. Of course if i do a: SELECT SQL_NO_CACHE * FROM `/var`; then everything is fine, however i would not like to put the burden on the user to specify the option to not cache the results. My question is whether it’s possible to specify in the storage backend that the results from the tables should not be cached. Thank for the help in advance -- Andras Szabo Sent with Airmail

3 7

Re: [Maria-developers] [Commits] Rev 4577: MDEV-5719: Wrong result with GROUP BY and LEFT OUTER JOIN in lp:~maria-captains/maria/10.0
by Sergey Petrunia 29 Jan '15

29 Jan '15

Hi Sergei, == TL;DR == I have debugged that LEFT JOIN problem. The fix is not pretty, but correct. I will now push it into the 10.0 tree. == Details == So, the commit changes - return (table->null_row ? TRUE : - null_ptr ? MY_TEST(null_ptr[row_offset] & null_bit) : 0); + return real_maybe_null() ? + MY_TEST(null_ptr[row_offset] & null_bit) : table->null_row; The patch changes the behavior for NULLable fields. Suppose * real_maybe_null()= TRUE * table->null_row==TRUE // the table has a NULL-complemented row * (null_ptr[row_offset] & null_bit) == FALSE // however the value // is not null. the old code will return TRUE the new code will return FALSE The new code may seem wrong at first glance (it's a NULL-complemented row, why we return FALSE from is_null()?) However, if we look where table->null_row is set to true, we will find that it is done here: inline void mark_as_null_row(TABLE *table) { table->null_row=1; table->status|=STATUS_NULL_ROW; bfill(table->null_flags,table->s->null_bytes,255); } The function also sets all NULL-bits, so normally one can assume that (table->null_row=TRUE) => ((null_ptr[..] & null_bit) == TRUE) However, there is a case when the above does not hold. It happens when we run GROUP BY query, and GROUP BY is satisfied by scanning an index which produces rows in group order. In that case, the following can occur: consider the query from the testcase: SELECT ... FROM table_one a LEFT JOIN table_two b ON a.oid=b.oid GROUP BY a.oid * we read from a row (1,3) * we make index lookup in table b and find row (3). * row combination a(1,3)-b(3) passes the WHERE clause, GROUP BY execution code records it as current row. * there are no other matches in table b, so we get back to reading table a. * we read row a(1,4) * we look for matches in table b. there are no matches, so we produce a NULL-complemented row b(NULLs) * row combination a(1,4)-b(NULLs) is passed to the GROUP BY runtime. The runtime sees that we are now in a different group, and figures that it should pass the row combination representing the previous group into query output. * It attempts to pass the row combination a(1,3)-b(3) (*) into query output. However, the current row for table b is already the NULL-complemented row. the row combination (*) is stored in the "result_field" objects (and not in "regular field" objects which refer to table->record[0]). It seems wrong that we look at result_field (which points to a row that's different from table->record[0]), and at the same time we are looking at table->null_row, which refers to the record in table->record[0]. However, "result_field" storage has no place for table->null_row bit. The fix basically means: "for NULL-able fields, field's NULL-bit should override table->null_row". Considering the above, it is correct (albeit ugly). Now, what about GROUP BY and NOT NULL fields? I discovered the following: if a table has table->maybe_null, then Field objects that refer to such table and are pointed by Item_field::result_field, are always NULLable. (code-wise, check out setup_copy_fields(), these calls: item->result_field=field->new_field(thd->mem_root,field->table, 1); item->result_field->move_field(copy->to_ptr,copy->to_null_ptr,1); result_field object of NOT NULL column is null-able. On Fri, Jan 23, 2015 at 08:25:59AM -0800, Nirbhay Choubey wrote: > At lp:~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4577 > revision-id: nirbhay(a)mariadb.com-20150123162551-h88uxn780ivj5k2n > parent: sergii(a)pisem.net-20150119153032-fqrmxjr13z0p1ly0 > committer: Nirbhay Choubey <nirbhay(a)mariadb.com> > branch nick: b5719-10.0 > timestamp: Fri 2015-01-23 11:25:51 -0500 > message: > MDEV-5719: Wrong result with GROUP BY and LEFT OUTER JOIN > > Merged revision 5224 from mysql-5.6 and added a test case. > .. > revno: 5224 > committer: Sergey Glukhov <sergey.glukhov(a)oracle.com> > branch nick: mysql-5.6 > timestamp: Wed 2013-06-19 14:24:08 +0400 > message: > Bug#16620047 INCORRECT QUERY RESULT (AFTER SERVER UPGRADE) > .. > === modified file 'mysql-test/r/group_by_innodb.result' > --- a/mysql-test/r/group_by_innodb.result 2014-01-26 20:49:11 +0000 > +++ b/mysql-test/r/group_by_innodb.result 2015-01-23 16:25:51 +0000 > @@ -57,3 +57,26 @@ > NULL 11.1,22.2 > DROP TABLE t1; > End of 5.5 tests > +# > +# MDEV-5719: Wrong result with GROUP BY and LEFT OUTER JOIN > +# > +CREATE TABLE table_one (oidGroup INT, oid INT PRIMARY KEY)ENGINE=INNODB; > +INSERT INTO table_one VALUES (1,1),(1,2),(1,3),(1,4); > +CREATE TABLE table_two (oid INT PRIMARY KEY)ENGINE=INNODB; > +INSERT INTO table_two VALUES (3); > +SELECT a.oidGroup, a.oid, b.oid FROM table_one a LEFT JOIN table_two b ON > +a.oid=b.oid WHERE a.oidGroup=1; > +oidGroup oid oid > +1 1 NULL > +1 2 NULL > +1 3 3 > +1 4 NULL > +SELECT a.oidGroup, a.oid, b.oid FROM table_one a LEFT JOIN table_two b ON > +a.oid=b.oid WHERE a.oidGroup=1 GROUP BY a.oid; > +oidGroup oid oid > +1 1 NULL > +1 2 NULL > +1 3 3 > +1 4 NULL > +DROP TABLE table_one, table_two; > +# End of tests > > === modified file 'mysql-test/t/group_by_innodb.test' > --- a/mysql-test/t/group_by_innodb.test 2014-01-26 20:49:11 +0000 > +++ b/mysql-test/t/group_by_innodb.test 2015-01-23 16:25:51 +0000 > @@ -67,3 +67,22 @@ > > --echo End of 5.5 tests > > +--echo # > +--echo # MDEV-5719: Wrong result with GROUP BY and LEFT OUTER JOIN > +--echo # > +CREATE TABLE table_one (oidGroup INT, oid INT PRIMARY KEY)ENGINE=INNODB; > +INSERT INTO table_one VALUES (1,1),(1,2),(1,3),(1,4); > + > +CREATE TABLE table_two (oid INT PRIMARY KEY)ENGINE=INNODB; > +INSERT INTO table_two VALUES (3); > + > +# Returns a value > +SELECT a.oidGroup, a.oid, b.oid FROM table_one a LEFT JOIN table_two b ON > +a.oid=b.oid WHERE a.oidGroup=1; > + > +SELECT a.oidGroup, a.oid, b.oid FROM table_one a LEFT JOIN table_two b ON > +a.oid=b.oid WHERE a.oidGroup=1 GROUP BY a.oid; > + > +DROP TABLE table_one, table_two; > + > +--echo # End of tests > > === modified file 'sql/field.h' > --- a/sql/field.h 2014-10-20 12:42:00 +0000 > +++ b/sql/field.h 2015-01-23 16:25:51 +0000 > @@ -656,21 +656,28 @@ > inline bool is_null(my_ptrdiff_t row_offset= 0) const > { > /* > + If the field is NULLable, it returns NULLity based > + on null_ptr[row_offset] value. Otherwise it returns > + NULL flag depending on TABLE::null_row value. > + > The table may have been marked as containing only NULL values > for all fields if it is a NULL-complemented row of an OUTER JOIN > or if the query is an implicitly grouped query (has aggregate > functions but no GROUP BY clause) with no qualifying rows. If > - this is the case (in which TABLE::null_row is true), the field > - is considered to be NULL. > + this is the case (in which TABLE::null_row is true) and the > + field is not nullable, the field is considered to be NULL. > + > + Do not change the order of testing. Fields may be associated > + with a TABLE object without being part of the current row. > + For NULL value check to work for these fields, they must > + have a valid null_ptr, and this pointer must be checked before > + TABLE::null_row. > + > Note that if a table->null_row is set then also all null_bits are > set for the row. > - > - Otherwise, if the field is NULLable, it has a valid null_ptr > - pointer, and its NULLity is recorded in the "null_bit" bit of > - null_ptr[row_offset]. > */ > - return (table->null_row ? TRUE : > - null_ptr ? MY_TEST(null_ptr[row_offset] & null_bit) : 0); > + return real_maybe_null() ? > + MY_TEST(null_ptr[row_offset] & null_bit) : table->null_row; > } > inline bool is_real_null(my_ptrdiff_t row_offset= 0) const > { return null_ptr ? (null_ptr[row_offset] & null_bit ? 1 : 0) : 0; } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 2

[Maria-developers] Dynamic columns support in libmaria client library?
by Peter Laursen 28 Jan '15

28 Jan '15

This page https://mariadb.com/kb/en/mariadb/documentation/nosql/dynamic-columns-api/ does not mention anything about it. it just says *The API is a part of libmysql C client library. In order to use it, one needs to include this header file* *#include <mysql/ma_dyncol.h>* *and link against libmysql.* Now can ma_dyncol.h be linked against libmaria as well? There are 2 questions actually 1) technical: will it work with libmaria? 2) legal: can a client program using libmaria under LGPL license include ma_dyncol.h? -- Peter -- Webyog

3 5

Re: [Maria-developers] [Commits] 51fa027: Improve performance for calculating memory allocation
by Sergei Golubchik 25 Jan '15

25 Jan '15

Hi, Monty! On Jan 23, Michael Widenius wrote: > revision-id: 51fa0272febd782674b3d44828718e805c44c1a8 > parent(s): ab440b0fb7302d707ba0ba41382bf911404db1cb > committer: Michael Widenius > branch nick: maria-10.1 > timestamp: 2015-01-23 14:32:53 +0200 > message: > > Improve performance for calculating memory allocation See my comments below: > diff --git a/include/mysql/plugin.h b/include/mysql/plugin.h > index 16b0c6e..9ae0fe8 100644 > --- a/include/mysql/plugin.h > +++ b/include/mysql/plugin.h > @@ -190,6 +196,7 @@ struct st_mysql_show_var { > > #define SHOW_VAR_FUNC_BUFF_SIZE (256 * sizeof(void*)) > typedef int (*mysql_show_var_func)(MYSQL_THD, struct st_mysql_show_var*, char *); > +typedef int (*mysql_show_var_extended_func)(MYSQL_THD, struct st_mysql_show_var*, enum enum_var_type, char *); I'd put scope the last making mysql_show_var_extended_func backward compatible with mysql_show_var_func. this makes some code simpler, see below. > > /* > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 044b60d..8eed7b9 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -3115,9 +3115,16 @@ static bool show_status_array(THD *thd, const char *wild, > if var->type is SHOW_FUNC or SHOW_SIMPLE_FUNC, call the function. > Repeat as necessary, if new var is again one of the above > */ > - for (var=variables; var->type == SHOW_FUNC || > - var->type == SHOW_SIMPLE_FUNC; var= &tmp) > - ((mysql_show_var_func)(var->value))(thd, &tmp, buff); > + for (var=variables; > + var->type == SHOW_FUNC || var->type == SHOW_SIMPLE_FUNC || > + var->type == SHOW_EXTENDED_FUNC; > + var= &tmp) > + { > + if (var->type == SHOW_EXTENDED_FUNC) > + ((mysql_show_var_extended_func)(var->value))(thd, &tmp, scope, buff); > + else > + ((mysql_show_var_func)(var->value))(thd, &tmp, buff); You won't need this if() when scope is the last argument. In fact I'm still thinking whether it would be a good idea to add scope to mysql_show_var_func and SHOW_FUNC, without introducing SHOW_EXTENDED_FUNC at all. This wouldn't break any existing plugin, as it's perfectly binary compatible extension. Also, see below. > + } > > SHOW_TYPE show_type=var->type; > if (show_type == SHOW_ARRAY) > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 0d49407..d200af6 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1747,6 +1749,16 @@ void add_to_status(STATUS_VAR *to_var, STATUS_VAR *from_var) > to_var->binlog_bytes_written+= from_var->binlog_bytes_written; > to_var->cpu_time+= from_var->cpu_time; > to_var->busy_time+= from_var->busy_time; > + to_var->local_memory_used+= from_var->local_memory_used; why? it's not additive, right? and global_status_var.local_memory_used is never shown anyway. > + > + /* > + Update global_memory_used. We have to do this with atomic_add as the > + global value can change outside of LOCK_status. > + */ > + // workaround for gcc 4.2.4-1ubuntu4 -fPIE (from DEB_BUILD_HARDENING=1) > + int64 volatile * volatile ptr= &to_var->global_memory_used; I'd rather remove this workaround. 4.2.4 is ancient, and quite probably this bug doesn't happen with MY_MEMORY_ORDER_RELAXED at all > + my_atomic_add64_explicit(ptr, from_var->global_memory_used, > + MY_MEMORY_ORDER_RELAXED); > } > > /* > @@ -1784,6 +1796,11 @@ void add_diff_to_status(STATUS_VAR *to_var, STATUS_VAR *from_var, > dec_var->binlog_bytes_written; > to_var->cpu_time+= from_var->cpu_time - dec_var->cpu_time; > to_var->busy_time+= from_var->busy_time - dec_var->busy_time; > + > + /* > + We don't need to accumulate memory_used as these are not after > + this function is called. Sorry, I cannot parse that :( > + */ > } > > #define SECONDS_TO_WAIT_FOR_KILL 2 > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index ea53e47..c4e1dc3 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -3972,19 +3972,24 @@ static void my_malloc_size_cb_func(long long size, my_bool is_thread_specific) > if (thd) > { > DBUG_PRINT("info", ("memory_used: %lld size: %lld", > - (longlong) thd->status_var.memory_used, size)); > + (longlong) thd->status_var.local_memory_used, > + size)); > - thd->status_var.memory_used+= size; > + thd->status_var.local_memory_used+= size; > - DBUG_ASSERT((longlong) thd->status_var.memory_used >= 0); > + DBUG_ASSERT((longlong) thd->status_var.local_memory_used >= 0); > } > } > } > + else if (likely(thd)) > + thd->status_var.global_memory_used+= size; Hmm, ok. So, you think that current_thd is cheaper than updating a global variable? > + else > + { > // workaround for gcc 4.2.4-1ubuntu4 -fPIE (from DEB_BUILD_HARDENING=1) > - int64 volatile * volatile ptr=&global_status_var.memory_used; > + int64 volatile * volatile ptr=&global_status_var.global_memory_used; I would've removed this workaround too > my_atomic_add64_explicit(ptr, size, MY_MEMORY_ORDER_RELAXED); > + } > } > } > > - > static int init_common_variables() > { > umask(((~my_umask) & 0666)); > @@ -8028,6 +8033,21 @@ static int show_default_keycache(THD *thd, SHOW_VAR *var, char *buff) > return 0; > } > > + > +static int show_memory_used(THD *thd, SHOW_VAR *var, enum enum_var_type scope, > + char *buff) > +{ > + var->type= SHOW_LONGLONG; > + var->value= buff; > + if (scope == OPT_GLOBAL) > + *(longlong*) buff= (global_status_var.local_memory_used + > + global_status_var.global_memory_used); > + else > + *(longlong*) buff= thd->status_var.local_memory_used; > + return 0; I'm surprised you haven't added SHOW_EXTENDED_SIMPLE_FUNC :) That would've make a very confusing name though. May be we should indeed add scope as the last argument to mysql_show_var_func and reuse both SHOW_SIMPLE_FUNC and SHOW_FUNC? I'm starting to think it's a good thing to do. Makes the API simpler and the number of choices to understand - smaller. If you do that - please don't forget to increment MARIA_PLUGIN_INTERFACE_VERSION. > +} > + > + > #ifndef DBUG_OFF > static int debug_status_func(THD *thd, SHOW_VAR *var, char *buff) > { Regards, Sergei

1 0

[Maria-developers] Additions to GIS in 10.1.
by Alexey Botchkov 23 Jan '15

23 Jan '15

Hi all. What's new about GIS in 10.1. Related tasks in JIRA are: MDEV-4045 Missing OGC Spatial functions. MDEV-60 Support for Spatial Reference systems for the GIS data. MDEV-12 OpenGIS: create required tables: GeometryColumns, related views. Speaking shortly, the MariaDB GIS part is now OpenGIS compliant, and passes all the OpenGIS conformance tests To get that the next things were done: Missing functions were added: IsRing(ln) The return type is Integer, with a return value of 1 for TRUE, 0 for FALSE, and –1 for UNKNOWN corresponding to a function invocation on NULL arguments; return TRUE if c is a ring, i.e., if c is closed and simple. A simple Curve does not pass through the same Point more than once. PointOnSurface(s) The return type is POINT, returns a Point guaranteed to lie on the Surface. arguments: POLYGON or MYLTIPOLYGON. NumInteriorRing(p) The return type is Integer, returns the number of interiorRings arguments: POLYGON. Relate(g1 Geometry, g2 Geometry, patternMatrix String) The return type is Integer, with a return value of 1 for TRUE, 0 for FALSE, and –1 for UNKNOWN corresponding to a function invocation on NULL arguments; returns TRUE if the spatial relationship specified by the pattern Matrix holds. ConvexHull(g1 Geometry) The return is Geometry, returna a geometric object that is the convex hull of g1 arguments: GEOMETRY. The information about the GEOMETRY column's reference system (SRID) is now stored in the field's metadata. Syntax of the CREATE and ALTER was modified to specify the SRID for the geometry columns. With this information some views required by OpenGIS, were added: GEOMETRY_COLUMNS, SPATIAL_REF_SYS: These are links to the INFORMATION_SCHEMA.GEOMETRY_COLUMNS and INFORMATION_SCHEMA.SPATIAL_REF_SYS respectively. INFORMATION_SCHEMA.GEOMETRY_COLUMNS: describes the available feature tables and their Geometry properties. Fields: F_TABLE_CATALOG CHARACTER VARYING(256) NOT NULL, F_TABLE_SCHEMA CHARACTER VARYING(256) NOT NULL, F_TABLE_NAME CHARACTER VARYING(256) NOT NULL, the above 3 fields are the fully qualified name of the feature table containing the geometry column; F_GEOMETRY_COLUMN CHARACTER VARYING(256) NOT NULL, the name of the column in the feature table that is the Geometry Column. G_TABLE_CATALOG CHARACTER VARYING(256) NOT NULL, G_TABLE_SCHEMA CHARACTER VARYING(256) NOT NULL, G_TABLE_NAME CHARACTER VARYING(256) NOT NULL, the above 3 fields are the name of the geometry table and its schema and catalog. The geometry table implements the geometry column; STORAGE_TYPE INTEGER, always 1 in MariaDB (means binary geometry implementation) GEOMETRY_TYPE INTEGER, the type of geometry values stored in this column. 0 = GEOMETRY 1 = POINT 3 = LINESTRING 5 = POLYGON 7 = MULTIPOINT 9 = MULTILINESTRING 11 = MULTIPOLYGON COORD_DIMENSION INTEGER, the number of dimensions in the spatial reference system. Always 2 in MariaDB. MAX_PPR INTEGER, Always 0 in MariaDB. SRID INTEGER the ID of the Spatial Reference System used for the coordinate geometry in this table. It is a foreign key reference to the SPATIAL_REF_SYS table. INFORMATION_SCHEMA.SPATIAL_REF_SYS: stores information on each spatial reference system used in the database. Fields: SRID INTEGER NOT NULL PRIMARY KEY, an integer value that uniquely identifies each Spatial Reference System within a database. AUTH_NAME CHARACTER VARYING(256), the name of the standard or standards body that is being cited for this reference system. In most cases the value is "EPSG". AUTH_SRID INTEGER TEXT the Well-known Text Representation of the Spatial Reference System. Required stored procedures added: AddGeometryColumn(FEATURE_TABLE_CATALOG, FEATURE_TABLE_SCHEMA, FEATURE_TABLE_NAME, GEOMETRY_COLUMN_NAME, SRID) adds new column of spatial type to the specified table DropGeometryColumn(FEATURE_TABLE_CATALOG, FEATURE_TABLE_SCHEMA, FEATURE_TABLE_NAME, GEOMETRY_COLUMN_NAME) removes the spatial column from the table. That's it for now. Best regards. HF

1 0

[Maria-developers] Renaming Virtual Columns to Computed Columns
by Ian Gilfillan 19 Jan '15

19 Jan '15

There was a suggestion on the docs mailing list that we rename Virtual Columns to Computed Columns. I think it makes sense, but before we implement I want to make sure everyone is happy with the change, so am repeating it here as not everyone follows docs. Currently, we use the term Virtual Columns (see https://mariadb.com/kb/en/mariadb/documentation/sql-structure-and-commands/…) Oracle uses this term, while MS SQL and Firebird use the term Computed Columns to describe similar functionality. MariaDB's virtual column implementation has both persistent and virtual attributes. Talking of "virtual virtual" and "persistent virtual" can be confusing. Oracle's implementation, to my knowledge, only has virtual virtual, while MS SQL's Computed Columns offers both persistent ("persisted" in their case) and virtual. I think "Computed" is a slightly more descriptive term, and avoids the ambiguity of "persistent virtual" and "virtual virtual". The MariaDB error messages already use the term computed, for example "Constant expression in computed column function is not allowed". Unless anyone has a good reason to prefer the term virtual, we're going to rather use the term "computed column" in the docs. The article will still mention virtual column as an alternative etc, but computed will be the preferred term. ian

3 3

Re: [Maria-developers] [Commits] Rev 4562: MDEV-7402 'reset master' hangs, waits for signalled COND_xid_list in lp:~maria-captains/maria/10.0
by Kristian Nielsen 19 Jan '15

19 Jan '15

Sergei Golubchik <serg(a)mariadb.org> writes: > ------------------------------------------------------------ > revno: 4562 > revision-id: sergii(a)pisem.net-20150118130747-rs67yzzk8a7ebwdd > parent: sergii(a)pisem.net-20150118002113-snztyzwvzc500c2m > fixes bug: https://mariadb.atlassian.net/browse/MDEV-7402 > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 10.0 > timestamp: Sun 2015-01-18 14:07:47 +0100 > message: > MDEV-7402 'reset master' hangs, waits for signalled COND_xid_list > > Using a boolean flag for 'there is a RESET MASTER in progress' doesn't > work very well for multiple concurrent RESET MASTER statements. Ok to push with a testcase like the below, as we discussed. - Kristian. ----------------------------------------------------------------------- --echo *** MDEV-7402: 'reset master' hangs, waits for signalled COND_xid_list *** --source include/wait_for_binlog_checkpoint.inc connect(con3,localhost,root,,); # Make the binlog background thread wait before clearing the pending checkpoint. # The bug was that one RESET MASTER would clear the reset_master_pending # flag set by another RESET MASTER; this could cause the wakeup from the # binlog background thread not to be sent, and thus the second RESET MASTER # to wait infinitely. SET debug_sync="reset_logs_after_set_reset_master_pending SIGNAL reset_master_ready WAIT_FOR reset_master_cont"; send RESET MASTER; --connection default SET @old_dbug= @@global.DEBUG_DBUG; SET GLOBAL debug_dbug="+d,inject_binlog_background_thread_before_mark_xid_done"; SET debug_sync="now WAIT_FOR reset_master_ready"; RESET MASTER; SET debug_sync="now WAIT_FOR injected_binlog_background_thread"; SET GLOBAL debug_dbug=@old_dbug; SET debug_sync="now SIGNAL reset_master_cont"; --connection con3 REAP; --connection default SET debug_sync = 'reset';

1 0

Re: [Maria-developers] [MDEV-7362] ANALYZE TABLES crash with table-independent-statistics gathering
by Sergei Golubchik 17 Jan '15

17 Jan '15

Hi, Vicențiu! On Jan 17, Vicențiu Ciorbaru wrote: > > Here's the new diff, let me know if you have any more comments. Shall I > push this as is? Yes, just one small detail (file name and the comment), see below. Then please push, no need to ask for another review :) But... 1. Please don't paste patches into the email anymore. At least not with your mail client that wraps long lines around and sends everything as html. See https://mariadb.com/kb/en/using-git/#commit-emails 2. Why git? This is the patch for 10.0 bug, at the moment 10.0 is still in bzr. 3. This looks like a reverse diff, how did you manage to do that? :) > diff --git a/mysql-test/t/statistics_index_crash.test b/mysql-test/t/statistics_index_crash.test > deleted file mode 100644 > index dbbdca3..0000000 > --- a/mysql-test/t/statistics_index_crash.test if you create a new test file for one bug only (as opposite to adding to an existing test file), mention the bug number in the file name. Like statistics_index_crash-7362.test > +++ /dev/null > @@ -1,26 +0,0 @@ > ---source include/have_stat_tables.inc > ---source include/have_innodb.inc Start a test case for a bug with a comment that says the bug number, the synopsys, and optionally more information, if needed. The bare minimum looks like # # MDEV-7362 ANALYZE TABLES crash with table-independent-statistics gathering # MDEV-7380 engine-independent stats SEGV on ANALYZE TABLE (#2) # > - > -CREATE TABLE t1 (a longtext, FULLTEXT KEY (`a`)) ENGINE=InnoDB; > -INSERT INTO t1 VALUES (unhex('3E0D0A4141414142334E7A6143317963324541414141424977414141674541726D')); > -ANALYZE TABLE t1 PERSISTENT FOR ALL; > ---sorted_result > -SELECT * FROM mysql.index_stats WHERE index_name='a' AND table_name='t1'; > -DROP TABLE t1; > - > -CREATE TABLE t1 (a longtext, FULLTEXT KEY (`a`)) ENGINE=MyISAM; > -INSERT INTO t1 VALUES (unhex('3E0D0A4141414142334E7A6143317963324541414141424977414141674541726D')); > -ANALYZE TABLE t1 PERSISTENT FOR ALL; > ---sorted_result > -SELECT * FROM mysql.index_stats WHERE index_name='a' AND table_name='t1'; > -DROP TABLE t1; > - > -CREATE TABLE geom (g GEOMETRY NOT NULL, SPATIAL INDEX(g)) ENGINE=MyISAM; > -INSERT INTO geom VALUES > - (MultiPolygonFromText('MULTIPOLYGON(((28 26,28 0,84 0,84 42,28 26),(52 18,66 23,73 9,48 6,52 18)),((59 18,67 18,67 13,59 13,59 18)))')), > - (MPolyFromText('MULTIPOLYGON(((28 26,28 0,84 0,84 42,28 26),(52 18,66 23,73 9,48 6,52 18)),((59 18,67 18,67 13,59 13,59 18)))')), > - (MPolyFromWKB(AsWKB(MultiPolygon(Polygon(LineString(Point(0, 3), Point(3, 3), Point(3, 0), Point(0, 3))))))); > -ANALYZE TABLE geom PERSISTENT FOR ALL; > ---sorted_result > -SELECT * FROM mysql.index_stats WHERE index_name='g' AND table_name='geom'; > -DROP TABLE geom; Regards, Sergei

1 0

[Maria-developers] [MDEV-7362] ANALYZE TABLES crash with table-independent-statistics gathering
by Vicențiu Ciorbaru 17 Jan '15

17 Jan '15

Hi Sergei! I've implemented a fix for the bug described in MDEV-7362, that also fixes MDEV-7380. The test case also covers MDEV-7380. With the following fix,the I've spent a bit of time trying to figure out the architecture of things there. I am not confident my fix is complete, or if this is the way we want to go about fixing it. The fix does what we discussed over IRC, that we are not interested in computing statistics for a FULLTEXT index. What I think would be the root cause of the problem is calling ha_index_init() on an index that does not support this. By calling table->file->ha_index_init(), the table->field array gets corrupted and that ultimately leads to a crash in the function using it afterwards. Then again, this module is completely new to me, so any comments are appreciated. Here is the diff of the first changes. diff --git a/mysql-test/t/mdev-7362.test b/mysql-test/t/mdev-7362.test new file mode 100644 index 0000000..6551893 --- /dev/null +++ b/mysql-test/t/mdev-7362.test @@ -0,0 +1,14 @@ +CREATE TABLE t1 (a longtext, FULLTEXT KEY (`a`)) ENGINE=MyISAM; +insert into t1 values (unhex('3E0D0A4141414142334E7A6143317963324541414141424977414141674541726D')); +analyze table t1 persistent for all; +drop table t1; + +CREATE TABLE t1 (a longtext, FULLTEXT KEY (`a`)) ENGINE=MyISAM; +insert into t1 values (unhex('3E0D0A4141414142334E7A6143317963324541414141424977414141674541')); +analyze table t1 persistent for all; +drop table t1; + +CREATE TABLE t1 (f longtext NOT NULL, FULLTEXT KEY f (f)) ENGINE=MyISAM; +INSERT INTO t1 VALUES (REPEAT('a',27)),('foo'); +ANALYZE TABLE t1 PERSISTENT FOR ALL; +drop table t1; diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc index d368145..2c1cccf 100644 --- a/sql/sql_statistics.cc +++ b/sql/sql_statistics.cc @@ -2367,6 +2367,9 @@ int collect_statistics_for_index(THD *thd, TABLE *table, uint index) DBUG_RETURN(rc); } + if (key_info->flags & HA_FULLTEXT) // skip if FULLTEXT index + DBUG_RETURN(rc); + table->key_read= 1; table->file->extra(HA_EXTRA_KEYREAD); Regards, Vicențiu

3 5

[Maria-developers] MDEV-7440 Build fails in libgroonga on Solaris
by Alexander Barkov 16 Jan '15

16 Jan '15

Hello Kentoku, Can you please have a look into this problem? https://mariadb.atlassian.net/browse/MDEV-7440 Can you fix please? Thanks.

1 0

Re: [Maria-developers] [Commits] Rev 4398: MDEV-7368 : SLES: Failed to start mysql.service: Unit
by Sergei Golubchik 14 Jan '15

14 Jan '15

Hi, Nirbhay! On Dec 23, Nirbhay Choubey wrote: > At lp:~maria-captains/maria/5.5 > > ------------------------------------------------------------ > revno: 4398 > revision-id: nirbhay(a)mariadb.com-20141224033207-8lh5i8d84qsjey79 > parent: elenst(a)montyprogram.com-20141223172123-pkfhjnv7oy1suxpa > committer: Nirbhay Choubey <nirbhay(a)mariadb.com> > branch nick: b7368-5.5 > timestamp: Tue 2014-12-23 22:32:07 -0500 > message: > MDEV-7368 : SLES: Failed to start mysql.service: Unit > mysql.service failed to load > > Added 'systemctl daemon-reload' command in postin and > postun rpm scripts. > === modified file 'support-files/rpm/server-postin.sh' > --- a/support-files/rpm/server-postin.sh 2013-11-20 11:18:46 +0000 > +++ b/support-files/rpm/server-postin.sh 2014-12-24 03:32:07 +0000 > @@ -1,6 +1,10 @@ > > # Make MySQL start/shutdown automatically when the machine does it. > if [ $1 = 1 ] ; then > + if [ -x /usr/bin/systemctl ] ; then > + /usr/bin/systemctl daemon-reload >/dev/null 2>&1 > + fi > + looks ok > if [ -x /sbin/chkconfig ] ; then > /sbin/chkconfig --add mysql > fi > @@ -51,30 +55,31 @@ > SETARGETDIR=/etc/selinux/targeted/src/policy > SEDOMPROG=$SETARGETDIR/domains/program > SECONPROG=$SETARGETDIR/file_contexts/program > +if [ -f /etc/redhat-release ] ; then > + if grep -q "Red Hat Enterprise Linux .. release 4" /etc/redhat-release \ > + || grep -q "CentOS release 4" /etc/redhat-release ; then I'd rather written if grep -Eqs '(Red Hat Enterprise Linux ..|CentOS) release 4' /etc/redhat-release; then or (which is more portable, but perhaps unnecessary so for an rpm postin script) if grep '$Red Hat Enterprise Linux ..\|CentOS$ release 4' /etc/redhat-release >/dev/null 2>&1; then but if you'd like you can push your variant. Regards, Sergei

2 1

Re: [Maria-developers] [Commits] Rev 4401: MDEV-7448 - mtr may leave stale mysqld in lp:maria/5.5
by Sergei Golubchik 14 Jan '15

14 Jan '15

Hi, Sergey! On Jan 14, Sergey Vojtovich wrote: > ------------------------------------------------------------ > revno: 4401 > revision-id: svoj(a)mariadb.org-20150114135038-jss2vvpkhihysfmj > parent: sergii(a)pisem.net-20150110130746-xlrofqddt4j8xkw6 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5 > timestamp: Wed 2015-01-14 17:50:38 +0400 > message: > MDEV-7448 - mtr may leave stale mysqld > > Let mtr handle SIGHUP the same way as SIGINT. This solves stale mysqld processes > left after broken/closed ssh connection. Looks ok. But please make sure that all tests pass before pushing it. mysqld uses SIGHUP to print various information to the error log. If there's a test that actually tests that by sending SIGHUP - your change might've broken it. Regards, Sergei

1 0

[Maria-developers] MySQL bug #68557 "sjis changes in 5.5 break date parsing" fixed in MariaDB >= 5.5.38
by Hartmut Holzgraefe 13 Jan '15

13 Jan '15

Any idea how that happened, as I can't find anything that seems to be related in the 5.5.38 changelog? http://bugs.mysql.com/bug.php?id=68557 -- Hartmut Holzgraefe, Principal Support Engineer (EMEA) MariaDB Corporation | http://www.mariadb.com/

2 1

[Maria-developers] Buildbot slow upload
by Kristian Nielsen 13 Jan '15

13 Jan '15

Hi Serg, Daniel, Some time ago there was discussion of slow Buildbot upload speed (sorry, don't have the original mail thread to reply to). I found some new information on this. Yesterday I had to do a custom build, but was not able to because my tarbake upload kept getting stuck. So I had to investigate. Turns out the real problem was that hasky:/media/backup/archive/pack was full. And apparently, the error handling in Buildbot upload step is broken, so the error is not caught properly but just causes the upload to hang. I have noticed this before, that upload gets stuck. I wonder if full disk on hasky has caused this before. Daniel: Do we have something that removes old builds from /archive/pack? I did not see anything in root or buildbot crontabs? With respect to slow uploads, while trying various things, one thing I did was increase the upload blocksize in Buildbot config. Now I see tarbake upload (50MB) taking just 1 minute, which seems fair (probably it can be more if the network or buildbot is busy). So I'm wondering if the slow upload that was seen was in fact a combination of too low default blocksize, and uploads occasionally getting _completely_ stuck? It's of course still fine to move to using rsync, as Serg proposed earlier; just wanted to mention what I found. And in either case, we still need a solution to prevent /archive/pack from running full in the future... - Kristian.

2 1

[Maria-developers] MDEV-7288 USER/ROLE: CREATE OR REPLACE, CREATE IF NOT EXISTS, DROP IF EXISTS
by Alexander Barkov 09 Jan '15

09 Jan '15

Hi Sergei, Please have a look into the patch for MDEV-7288. It consists of the USER/ROLE related part of the big GSoC patch from Sriram that you reviewed earlier, with review suggestions addressed. You earlier had the following suggestions/questions: >> diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc >> index 5f077c2..1e156be 100644 >> --- a/sql/sql_acl.cc >> +++ b/sql/sql_acl.cc >> ... >> @@ -9215,10 +9252,26 @@ bool mysql_create_user(THD *thd, List <LEX_USER> &lis >> */ >> if (handle_grant_data(tables, 0, user_name, NULL)) >> { >> - append_user(thd, &wrong_users, user_name); >> - >> - result= TRUE; >> - continue; >> + if (thd->lex->is_create_or_replace()) >> + { >> + if (handle_grant_data(tables, 1, user_name, NULL) <= 0) >> + { >> + append_user(thd, &wrong_users, user_name); > > I don't understand that at all. If user creation failed, you try to drop > it, but don't create a user again, after the old one was dropped. How > could it work? > Yeah, it looks confusing, because handle_grant_data() can do different things depending on the arguments. The call for handle_grant_data(tables, 0, user_name, NULL) does not create any users. It only checks if the user already exists. The call for handle_grant_data(tables, 1, user_name, NULL) drops the user. And the call for replace_user_table() after this block is the one that actually creates the user. I have added some comments in the new code, to make it clearer. >>> @@ -9268,7 +9321,7 @@ bool mysql_create_user(THD *thd, List <LEX_USER> &list, >>> (handle_as_role) ? "CREATE ROLE" : "CREATE USER", >>> wrong_users.c_ptr_safe()); >>> >>> - if (some_users_created) >>> + if (some_users_created || !result) >> >> What is that for? To binlog CREATE IF NOT EXISTS? Yes, that was the reason. I renamed "some_users_created" to "binlog" and changed the logic slightly to set "binlog" to true under the block checking lex->create_info.if_not_exists(), instead of this additional test for "result". So now it should be easier to read: if (binlog) result |= write_bin_log(thd, FALSE, thd->query(), thd->query_length()); Thanks.

3 3

[Maria-developers] prefix matching for command-line options - how to disable?
by Sergei Golubchik 06 Jan '15

06 Jan '15

Hi, We've got a bit of a problem with command-line option prefixes in the test suite. As you know, in MariaDB a command line option (the same applies to my.cnf too) doesn't need to be specified completely, it's sufficient to use an unambigous prefix of it. For example, 'net-retry-count=5' in my.cnf can be abbreviated as 'net-ret=5'. But not as 'net-re=5', the latter will fail with an error mysqld: ambiguous option '--net-re' (net_read_timeout, net_retry_count) This is the historical behavior of MySQL since about 2002. But it creates problems for plugins. InnoDB is particularly badly affected - it has lots of I_S plugins that are prefixes of other I_S plugins. For example: INNODB_BUFFER_PAGE INNODB_BUFFER_PAGE_LRU ... INNODB_CMP INNODB_CMPMEM INNODB_CMPMEM_RESET INNODB_CMP_PER_INDEX INNODB_CMP_PER_INDEX_RESET INNODB_CMP_RESET ... INNODB_SYS_FOREIGN INNODB_SYS_FOREIGN_COLS ... INNODB_SYS_TABLES INNODB_SYS_TABLESPACES INNODB_SYS_TABLESTATS also, it has plugins that match other plugin variables, like INNODB_LOCKS and innodb_locks_unsafe_for_binlog. As a result one cannot see or predict what these options will do. For example --enable-innodb-buffer-page might enable INNODB_BUFFER_PAGE plugin. Or it might enable INNODB_BUFFER_PAGE_LRU plugin - depending on which plugin will loaded (and will see and consume the option) first. And one cannot possibly disable INNODB_BUFFER_PAGE plugin, if the server will try to load INNODB_BUFFER_PAGE_LRU first. I see few solutions for this issue: 1. support a special suffix to option names that will disable prefix matching, for example: mysqld --enable-innodb-buffer-page! this will not match innodb-buffer-page-lru, so it's unambiguous. Of course, any dedicated character can be used, --enable-innodb-buffer-page$, or even a prefix --enable-exact-innodb-buffer-page, --enable-strict-innodb-buffer-page 2. add a special command-line option to disable prefix matching: mysqld --disable-getopt-prefix-matching --enable-innodb-buffer-page 3. do like mysql-5.7 is doing and completely remove support for prefix matching. In 5.7 one has to specified option names in full. I suspect that this is likely to break many exising setups. Opinions? Anybody cares about this obscure issue at all? Regards, Sergei

3 3

[Maria-developers] Buildslave terrier2 disabled due to KVM problems, needs fixing to be re-enabled
by Kristian Nielsen 05 Jan '15

05 Jan '15

Jani, Daniel, I discovered that our buildslave terrier2 seems to have problems with its KVM. Builds are incredibly slow, and it appears that it is not using KVM at all. So it is apparently using just QEMU, and soft-emulating the CPU. This is too slow to be useful, I think. So for now, I've removed all usage of terrier2 from the Buildbot config. If one of you wants to fix terrier2 to be useful again, then the patch to revert (to re-enable terrier2) is this: revid:knielsen@knielsen-hq.org-20150105182519-7492d95v1blss8jf http://bazaar.launchpad.net/~maria-captains/mariadb-tools/trunk/revision/kn… Before re-enabling, KVM needs to be somehow fixed to work (eg. something like `time perl -le 'for (0..100000000){}'` needs to be tested to run at about the same speed inside a guest VM as directly on the host). Maybe it would be best to upgrade the OS on the machine, maybe to the same as we have on bb01/bb02? We have had a number of problems on terrier2 with various KVM issues over the time. - Kristian.

1 0

[Maria-developers] New buildbot? - debpkg101-sid from Otto's github
by Otto Kekäläinen 02 Jan '15

02 Jan '15

Hello Daniel! Thanks for setting up the current buildbots that build my branches of 5.5 and 10.0. Could you now please also set up one the build from my 10.1 branch as I have started working on it too? The buildbot should be a copy of http://buildbot.askmonty.org/buildbot/builders/debpkg-sid/ but so that it pulls in the branch 10.1 from the repository https://github.com/ottok/mariadb Thanks! 2014-09-01 11:25 GMT+03:00 Otto Kekäläinen <otto(a)seravo.fi>: > Hello! > > 2014-03-14 12:59 GMT+02:00 Kristian Nielsen <knielsen(a)knielsen-hq.org>: >> You can see the two builders here: >> >> http://buildbot.askmonty.org/buildbot/builders/debpkg-trusty >> http://buildbot.askmonty.org/buildbot/builders/debpkg-sid/ > > Thanks Kristian and Daniel for creating these in March. > > The trusty build testing is less relevant nowadays while testing 10.0 > has become relevant. > > Could you please delete the buildbot debpkg-trusty and create a copy > of debpkg-sid called debpkg-10.0-sid which would be exactly like > debpkg-sid, but follow commits in this repo: > https://github.com/ottok/mariadb-10.0 ? > > Thanks! > > PS. I already configured the webhook in Github to be called in the > mariadb-10.0 repo as it is done in the mariadb-5.5 repo.

1 2

Re: [Maria-developers] [Commits] b2f45ba: MDEV-7324 - Lock-free hash for table definition cache
by Sergei Golubchik 28 Dec '14

28 Dec '14

Hi, Sergey! On Dec 22, svoj(a)mariadb.org wrote: > revision-id: b2f45ba8280c97f8dc8b1adb3234cf2e28e48982 > parent(s): 4a469b0ed68bb341e1a1e1ba540a45a3b6ff3ef9 > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2014-12-22 19:56:48 +0400 > message: > > MDEV-7324 - Lock-free hash for table definition cache That looked good to me. I didn't find anything obviously wrong. There are mostly questions below. Some are style suggestions (like, renaming of methods, etc). The only place I'm not sure about is early lf_hash_search_unpin() in tdc_acquire_share(). > diff --git a/mysys/lf_hash.c b/mysys/lf_hash.c > --- a/mysys/lf_hash.c > +++ b/mysys/lf_hash.c > @@ -122,7 +122,7 @@ static int lfind(LF_SLIST * volatile *head, CHARSET_INFO *cs, uint32 hashnr, > { > if (unlikely(callback)) > { > - if (callback(cursor->curr + 1, (void*)key)) > + if (cur_hashnr & 1 && callback(cursor->curr + 1, (void*)key)) Sorry for this. > return 1; > } > else if (cur_hashnr >= hashnr) > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -912,17 +912,17 @@ bool Drop_table_error_handler::handle_condition(THD *thd, > #endif /* defined(ENABLED_DEBUG_SYNC) */ > wait_for_commit_ptr(0), > main_da(0, false, false), > - m_stmt_da(&main_da) > + m_stmt_da(&main_da), > #ifdef WITH_WSREP > - , > wsrep_applier(is_wsrep_applier), > wsrep_applier_closing(false), > wsrep_client_thread(false), > wsrep_apply_toi(false), > wsrep_po_handle(WSREP_PO_INITIALIZER), > wsrep_po_cnt(0), > - wsrep_apply_format(0) > + wsrep_apply_format(0), > #endif > + tdc_hash_pins(0) Eh... I actually wanted wsrep related data to the at the end of the THD. Just to keep the rest of the THD layout identical for wsrep and non-wsrep builds. In case a wsrep-enabled plugin is loaded into non-wsrep server or visa versa. That was an early change, though, I've done more cleanups after that and moved wsrep into a service, so perhaps (even, most probably) this is not needed anymore. But I'd better stay on the safe side and keep wsrep at the end of THD. > { > ulong tmp; > > @@ -1698,6 +1698,8 @@ void THD::cleanup(void) > > free_root(&main_mem_root, MYF(0)); > main_da.free_memory(); > + if (tdc_hash_pins) > + lf_hash_put_pins(tdc_hash_pins); no tdc_hash_pins=0 here? > if (status_var.memory_used != 0) > { > DBUG_PRINT("error", ("memory_used: %lld", status_var.memory_used)); > diff --git a/sql/table.h b/sql/table.h > --- a/sql/table.h > +++ b/sql/table.h > @@ -611,32 +612,7 @@ struct TABLE_SHARE > mysql_mutex_t LOCK_ha_data; /* To protect access to ha_data */ > mysql_mutex_t LOCK_share; /* To protect TABLE_SHARE */ > > - typedef I_P_List <TABLE, TABLE_share> TABLE_list; > - typedef I_P_List <TABLE, All_share_tables> All_share_tables_list; > - struct > - { > - /** > - Protects ref_count, m_flush_tickets, all_tables, free_tables, flushed, > - all_tables_refs. > - */ > - mysql_mutex_t LOCK_table_share; > - mysql_cond_t COND_release; > - TABLE_SHARE *next, **prev; /* Link to unused shares */ > - uint ref_count; /* How many TABLE objects uses this */ > - uint all_tables_refs; /* Number of refs to all_tables */ > - /** > - List of tickets representing threads waiting for the share to be flushed. > - */ > - Wait_for_flush_list m_flush_tickets; > - /* > - Doubly-linked (back-linked) lists of used and unused TABLE objects > - for this share. > - */ > - All_share_tables_list all_tables; > - TABLE_list free_tables; > - ulong version; > - bool flushed; > - } tdc; > + TDC_element *tdc; Hmm. Ok. I think I like this change - TABLE_SHARE now contains table metadata and everything related to tdc is moved to TDC_element. Right? > > LEX_CUSTRING tabledef_version; > > diff --git a/sql/table.cc b/sql/table.cc > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -3845,11 +3844,11 @@ bool TABLE_SHARE::visit_subgraph(Wait_for_flush *wait_for_flush, > because we won't try to acquire tdc.LOCK_table_share while > holding a write-lock on MDL_lock::m_rwlock. > */ > - mysql_mutex_lock(&tdc.LOCK_table_share); > - tdc.all_tables_refs++; > - mysql_mutex_unlock(&tdc.LOCK_table_share); > + mysql_mutex_lock(&tdc->LOCK_table_share); > + tdc->all_tables_refs++; > + mysql_mutex_unlock(&tdc->LOCK_table_share); Do I understand it correctly that a TDC_element can never be deleted from LF_HASH as long as at least one TABLE_SHARE references it? > > - All_share_tables_list::Iterator tables_it(tdc.all_tables); > + TDC_element::All_share_tables_list::Iterator tables_it(tdc->all_tables); > > /* > In case of multiple searches running in parallel, avoid going > @@ -3946,21 +3945,10 @@ bool TABLE_SHARE::wait_for_old_version(THD *thd, struct timespec *abstime, > > mdl_context->done_waiting_for(); > > - mysql_mutex_lock(&tdc.LOCK_table_share); > - > - tdc.m_flush_tickets.remove(&ticket); > - > - if (tdc.m_flush_tickets.is_empty() && tdc.ref_count == 0) > - { > - /* > - If our thread was the last one using the share, > - we must destroy it here. > - */ > - mysql_mutex_unlock(&tdc.LOCK_table_share); > - destroy(); > - } > - else > - mysql_mutex_unlock(&tdc.LOCK_table_share); > + mysql_mutex_lock(&tdc->LOCK_table_share); > + tdc->m_flush_tickets.remove(&ticket); > + mysql_cond_broadcast(&tdc->COND_release); > + mysql_mutex_unlock(&tdc->LOCK_table_share); Why is the logic here different? > > > /* > diff --git a/sql/table_cache.h b/sql/table_cache.h > --- a/sql/table_cache.h > +++ b/sql/table_cache.h > @@ -16,6 +16,171 @@ > Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > > > +extern PSI_mutex_key key_TABLE_SHARE_LOCK_table_share; > +extern PSI_cond_key key_TABLE_SHARE_COND_release; > + > +class TDC_element > +{ > +public: > + uchar m_key[NAME_LEN + 1 + NAME_LEN + 1]; > + uint m_key_length; > + ulong version; > + bool flushed; > + TABLE_SHARE *share; > + > + typedef I_P_List <TABLE, TABLE_share> TABLE_list; > + typedef I_P_List <TABLE, All_share_tables> All_share_tables_list; > + /** > + Protects ref_count, m_flush_tickets, all_tables, free_tables, flushed, > + all_tables_refs. > + */ > + mysql_mutex_t LOCK_table_share; > + mysql_cond_t COND_release; > + TDC_element *next, **prev; /* Link to unused shares */ > + uint ref_count; /* How many TABLE objects uses this */ > + uint all_tables_refs; /* Number of refs to all_tables */ > + /** > + List of tickets representing threads waiting for the share to be flushed. > + */ > + Wait_for_flush_list m_flush_tickets; > + /* > + Doubly-linked (back-linked) lists of used and unused TABLE objects > + for this share. > + */ > + All_share_tables_list all_tables; > + TABLE_list free_tables; > + > + TDC_element() {} > + > + TDC_element(const char *key, uint key_length) : m_key_length(key_length) > + { > + memcpy(m_key, key, key_length); > + } > + > + > + void dbug_release_assertions() Somewhat confusing name. If I hadn't see the code below I'd thought this function releases some assertions (how? why?) may be assert_correctness_at_release() would be less ambiguous? > + { > + DBUG_ASSERT(share == 0); > + DBUG_ASSERT(ref_count == 0); > + DBUG_ASSERT(m_flush_tickets.is_empty()); > + DBUG_ASSERT(all_tables.is_empty()); > + DBUG_ASSERT(free_tables.is_empty()); > + DBUG_ASSERT(all_tables_refs == 0); > + DBUG_ASSERT(next == 0); > + DBUG_ASSERT(prev == 0); > + } > + > + > + /** > + Acquire TABLE object from table cache. > + > + @pre share must be protected against removal. > + > + Acquired object cannot be evicted or acquired again. > + > + While locked: > + - pop object from TABLE_SHARE::tdc.free_tables > + > + While unlocked: > + - mark object used by thd "While locked" / "While unlocked" - what does that mean? > + > + @return TABLE object, or NULL if no unused objects. > + */ > + > + TABLE *acquire_table(THD *thd) > + { > + TABLE *table; > + > + mysql_mutex_lock(&LOCK_table_share); > + table= free_tables.pop_front(); > + if (table) > + { > + DBUG_ASSERT(!table->in_use); lots of asserts - good! > + table->in_use= thd; > + /* The ex-unused table must be fully functional. */ > + DBUG_ASSERT(table->db_stat && table->file); > + /* The children must be detached from the table. */ > + DBUG_ASSERT(!table->file->extra(HA_EXTRA_IS_ATTACHED_CHILDREN)); I don't even think of asking why it's HA_EXTRA_IS_ATTACHED_CHILDREN and not HA_EXTRA_HAS_ATTACHED_CHILDREN or, at least, HA_EXTRA_CHILDREN_ARE_ATTACHED. Ah, whatever... Horrors of the English language hidden in old MySQL code :) > + } > + mysql_mutex_unlock(&LOCK_table_share); > + return table; > + } > + > + > + /** > + Get last element of free_tables. > + */ > + > + TABLE *free_tables_back() Eh, I've thought for a few seconds what could it mean to "free tables back" until I read the function comment. You see, "free" is so often used as a verb that one absolutely doesn't see it that here it means "free_tables' back" even though it's also a common naming pattern (xxx_back for the list xxx). Rename the method, may be? > + { > + TABLE_list::Iterator it(share->tdc->free_tables); > + TABLE *entry, *last= 0; > + while ((entry= it++)) > + last= entry; > + return last; > + } > + > + > + /** > + Wait for MDL deadlock detector to complete traversing tdc.all_tables. > + > + Must be called before updating TABLE_SHARE::tdc.all_tables. > + */ > + > + void wait_for_mdl_deadlock_detector() > + { > + while (all_tables_refs) > + mysql_cond_wait(&COND_release, &LOCK_table_share); > + } > + > + > + /** > + Prepeare table share for use with table definition cache. > + */ > + > + static void lf_alloc_constructor(uchar *arg) > + { > + TDC_element *element= (TDC_element*) (arg + LF_HASH_OVERHEAD); > + DBUG_ENTER("tdc_init_share"); intentionally? or forgot to update DBUG_ENTER? > + mysql_mutex_init(key_TABLE_SHARE_LOCK_table_share, > + &element->LOCK_table_share, MY_MUTEX_INIT_FAST); > + mysql_cond_init(key_TABLE_SHARE_COND_release, &element->COND_release, 0); > + element->m_flush_tickets.empty(); > + element->all_tables.empty(); > + element->free_tables.empty(); > + element->all_tables_refs= 0; > + element->share= 0; > + element->ref_count= 0; > + element->next= 0; > + element->prev= 0; > + DBUG_VOID_RETURN; > + } > + > + > + /** > + Release table definition cache specific resources of table share. > + */ > + > + static void lf_alloc_destructor(uchar *arg) > + { > + TDC_element *element= (TDC_element*) (arg + LF_HASH_OVERHEAD); > + DBUG_ENTER("tdc_deinit_share"); intentionally? or forgot to update DBUG_ENTER? > + element->dbug_release_assertions(); > + mysql_cond_destroy(&element->COND_release); > + mysql_mutex_destroy(&element->LOCK_table_share); > + DBUG_VOID_RETURN; > + } > + > + > + static uchar *key(const TDC_element *element, size_t *length, > + my_bool not_used __attribute__((unused))) > + { > + *length= element->m_key_length; > + return (uchar*) element->m_key; > + } > +}; > + > + > enum enum_tdc_remove_table_type > { > TDC_RT_REMOVE_ALL, > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -270,58 +270,75 @@ uint get_table_def_key(const TABLE_LIST *table_list, const char **key) > # Pointer to list of names of open tables. > */ > > -OPEN_TABLE_LIST *list_open_tables(THD *thd, const char *db, const char *wild) > +struct list_open_tables_arg > { > + THD *thd; > + const char *db; > + const char *wild; > TABLE_LIST table_list; > OPEN_TABLE_LIST **start_list, *open_list; > - TABLE_SHARE *share; > - TDC_iterator tdc_it; > - DBUG_ENTER("list_open_tables"); > +}; > > - bzero((char*) &table_list,sizeof(table_list)); > - start_list= &open_list; > - open_list=0; > > - tdc_it.init(); > - while ((share= tdc_it.next())) > +static my_bool list_open_tables_callback(TDC_element *element, > + list_open_tables_arg *arg) > { > - if (db && my_strcasecmp(system_charset_info, db, share->db.str)) > - continue; > - if (wild && wild_compare(share->table_name.str, wild, 0)) > - continue; > + char *db= (char*) element->m_key; > + char *table_name= (char*) element->m_key + strlen((char*) element->m_key) + 1; > > + if (arg->db && my_strcasecmp(system_charset_info, arg->db, db)) > + return FALSE; > + if (arg->wild && wild_compare(table_name, arg->wild, 0)) > + return FALSE; > + > /* Check if user has SELECT privilege for any column in the table */ > - table_list.db= share->db.str; > - table_list.table_name= share->table_name.str; > - table_list.grant.privilege=0; > + arg->table_list.db= db; > + arg->table_list.table_name= table_name; > + arg->table_list.grant.privilege= 0; > > - if (check_table_access(thd,SELECT_ACL,&table_list, TRUE, 1, TRUE)) > - continue; > + if (check_table_access(arg->thd, SELECT_ACL, &arg->table_list, TRUE, 1, TRUE)) > + return FALSE; > > - if (!(*start_list = (OPEN_TABLE_LIST *) > - sql_alloc(sizeof(**start_list)+share->table_cache_key.length))) > - { > - open_list=0; // Out of memory > - break; > - } > + if (!(*arg->start_list= (OPEN_TABLE_LIST *) sql_alloc( when you know the thd, prefer thd->alloc() over sql_alloc() (which is, basically, current_thd->alloc()) > + sizeof(**arg->start_list) + element->m_key_length))) > + return TRUE; > + > - strmov((*start_list)->table= > - strmov(((*start_list)->db= (char*) ((*start_list)+1)), > - share->db.str)+1, > - share->table_name.str); > + strmov((*arg->start_list)->table= > + strmov(((*arg->start_list)->db= (char*) ((*arg->start_list) + 1)), > + db) + 1, table_name); > - (*start_list)->in_use= 0; > + (*arg->start_list)->in_use= 0; > - mysql_mutex_lock(&share->tdc.LOCK_table_share); > - TABLE_SHARE::All_share_tables_list::Iterator it(share->tdc.all_tables); > + > + mysql_mutex_lock(&element->LOCK_table_share); > + TDC_element::All_share_tables_list::Iterator it(element->all_tables); > TABLE *table; > while ((table= it++)) > if (table->in_use) > - ++(*start_list)->in_use; > + ++(*arg->start_list)->in_use; > - mysql_mutex_unlock(&share->tdc.LOCK_table_share); > + mysql_mutex_unlock(&element->LOCK_table_share); > - (*start_list)->locked= 0; /* Obsolete. */ > + (*arg->start_list)->locked= 0; /* Obsolete. */ > - start_list= &(*start_list)->next; > + arg->start_list= &(*arg->start_list)->next; > - *start_list=0; > + *arg->start_list= 0; > + return FALSE; > } > - tdc_it.deinit(); > - DBUG_RETURN(open_list); > + > + > +OPEN_TABLE_LIST *list_open_tables(THD *thd, const char *db, const char *wild) > +{ > + list_open_tables_arg argument; > + DBUG_ENTER("list_open_tables"); > + > + argument.thd= thd; > + argument.db= db; > + argument.wild= wild; > + bzero((char*) &argument.table_list, sizeof(argument.table_list)); > + argument.start_list= &argument.open_list; > + argument.open_list= 0; > + > + if (tdc_iterate(thd, (my_hash_walk_action) list_open_tables_callback, > + &argument, true)) > + DBUG_RETURN(0); > + > + DBUG_RETURN(argument.open_list); > } > > /***************************************************************************** > diff --git a/sql/table_cache.cc b/sql/table_cache.cc > --- a/sql/table_cache.cc > +++ b/sql/table_cache.cc > @@ -72,32 +76,23 @@ > /** > Protects unused shares list. > > - TABLE_SHARE::tdc.prev > - TABLE_SHARE::tdc.next > - oldest_unused_share > - end_of_unused_share > + TDC_element::prev > + TDC_element::next > + unused_shares > */ > > static mysql_mutex_t LOCK_unused_shares; > -static mysql_rwlock_t LOCK_tdc; /**< Protects tdc_hash. */ > my_atomic_rwlock_t LOCK_tdc_atomics; /**< Protects tdc_version. */ > > #ifdef HAVE_PSI_INTERFACE > -static PSI_mutex_key key_LOCK_unused_shares, key_TABLE_SHARE_LOCK_table_share; > +PSI_mutex_key key_LOCK_unused_shares, key_TABLE_SHARE_LOCK_table_share; really? not static? > static PSI_mutex_info all_tc_mutexes[]= > { > { &key_LOCK_unused_shares, "LOCK_unused_shares", PSI_FLAG_GLOBAL }, > { &key_TABLE_SHARE_LOCK_table_share, "TABLE_SHARE::tdc.LOCK_table_share", 0 } > }; > > -static PSI_rwlock_key key_rwlock_LOCK_tdc; > -static PSI_rwlock_info all_tc_rwlocks[]= > -{ > - { &key_rwlock_LOCK_tdc, "LOCK_tdc", PSI_FLAG_GLOBAL } > -}; > - > - > -static PSI_cond_key key_TABLE_SHARE_COND_release; > +PSI_cond_key key_TABLE_SHARE_COND_release; and here > static PSI_cond_info all_tc_conds[]= > { > { &key_TABLE_SHARE_COND_release, "TABLE_SHARE::tdc.COND_release", 0 } > @@ -265,85 +269,41 @@ void tc_add_table(THD *thd, TABLE *table) > > if (need_purge) > { > - TABLE_SHARE *purge_share= 0; > - TABLE_SHARE *share; > - TABLE *entry; > - ulonglong UNINIT_VAR(purge_time); > - TDC_iterator tdc_it; > + tc_add_table_arg argument; > + argument.purge_time= ULONGLONG_MAX; > + tdc_iterate(thd, (my_hash_walk_action) tc_add_table_callback, &argument); > > - tdc_it.init(); > - while ((share= tdc_it.next())) > + if (argument.purge_time != ULONGLONG_MAX) > { > - mysql_mutex_lock(&share->tdc.LOCK_table_share); > - if ((entry= tc_free_tables_back(share)) && > - (!purge_share || entry->tc_time < purge_time)) > + TDC_element *element= (TDC_element*) lf_hash_search(&tdc_hash, > + thd->tdc_hash_pins, > + argument.key, > + argument.key_length); > + if (element) > { > - purge_share= share; > - purge_time= entry->tc_time; > - } > - mysql_mutex_unlock(&share->tdc.LOCK_table_share); > - } > + TABLE *entry; > + mysql_mutex_lock(&element->LOCK_table_share); > + lf_hash_search_unpin(thd->tdc_hash_pins); > + element->wait_for_mdl_deadlock_detector(); you wait under a mutex? hmm... > > - if (purge_share) > - { > - mysql_mutex_lock(&purge_share->tdc.LOCK_table_share); > - tc_wait_for_mdl_deadlock_detector(purge_share); > - tdc_it.deinit(); > /* > It may happen that oldest table was acquired meanwhile. In this case > just go ahead, number of objects in table cache will normalize > eventually. > */ > - if ((entry= tc_free_tables_back(purge_share)) && > - entry->tc_time == purge_time) > + if ((entry= element->free_tables_back()) && > + entry->tc_time == argument.purge_time) > { > - entry->s->tdc.free_tables.remove(entry); > + element->free_tables.remove(entry); > tc_remove_table(entry); > - mysql_mutex_unlock(&purge_share->tdc.LOCK_table_share); > + mysql_mutex_unlock(&element->LOCK_table_share); > intern_close_table(entry); > } > else > - mysql_mutex_unlock(&purge_share->tdc.LOCK_table_share); > + mysql_mutex_unlock(&element->LOCK_table_share); > } > - else > - tdc_it.deinit(); > } > } > - > - > -/** > - Acquire TABLE object from table cache. > - > - @pre share must be protected against removal. > - > - Acquired object cannot be evicted or acquired again. > - > - While locked: > - - pop object from TABLE_SHARE::tdc.free_tables > - > - While unlocked: > - - mark object used by thd > - > - @return TABLE object, or NULL if no unused objects. > -*/ > - > -static TABLE *tc_acquire_table(THD *thd, TABLE_SHARE *share) > -{ > - TABLE *table; > - > - mysql_mutex_lock(&share->tdc.LOCK_table_share); > - table= share->tdc.free_tables.pop_front(); > - if (table) > - { > - DBUG_ASSERT(!table->in_use); > - table->in_use= thd; > - /* The ex-unused table must be fully functional. */ > - DBUG_ASSERT(table->db_stat && table->file); > - /* The children must be detached from the table. */ > - DBUG_ASSERT(!table->file->extra(HA_EXTRA_IS_ATTACHED_CHILDREN)); > - } ok, you've moved it to table_cache.h... > - mysql_mutex_unlock(&share->tdc.LOCK_table_share); > - return table; > } > > > @@ -397,88 +357,76 @@ bool tc_release_table(TABLE *table) > */ > table->in_use= 0; > /* Add table to the list of unused TABLE objects for this share. */ > - table->s->tdc.free_tables.push_front(table); > - mysql_mutex_unlock(&table->s->tdc.LOCK_table_share); > + table->s->tdc->free_tables.push_front(table); > + mysql_mutex_unlock(&table->s->tdc->LOCK_table_share); > return false; > > purge: > - tc_wait_for_mdl_deadlock_detector(table->s); > + table->s->tdc->wait_for_mdl_deadlock_detector(); > tc_remove_table(table); > - mysql_mutex_unlock(&table->s->tdc.LOCK_table_share); > + mysql_mutex_unlock(&table->s->tdc->LOCK_table_share); > table->in_use= 0; > intern_close_table(table); > return true; > } > > > -extern "C" uchar *tdc_key(const uchar *record, size_t *length, > - my_bool not_used __attribute__((unused))) > -{ > - TABLE_SHARE *entry= (TABLE_SHARE*) record; > - *length= entry->table_cache_key.length; > - return (uchar*) entry->table_cache_key.str; > -} > - > - > /** > Delete share from hash and free share object. > - > - @return > - @retval 0 Success > - @retval 1 Share is referenced > */ > > -static int tdc_delete_share_from_hash(TABLE_SHARE *share) > +static void tdc_delete_share_from_hash(TDC_element *element) > { > + THD *thd= current_thd; > + LF_PINS *pins; > + TABLE_SHARE *share; > DBUG_ENTER("tdc_delete_share_from_hash"); > - mysql_rwlock_wrlock(&LOCK_tdc); > - mysql_mutex_lock(&share->tdc.LOCK_table_share); > - if (--share->tdc.ref_count) > - { > - mysql_cond_broadcast(&share->tdc.COND_release); > - mysql_mutex_unlock(&share->tdc.LOCK_table_share); > - mysql_rwlock_unlock(&LOCK_tdc); > - DBUG_RETURN(1); > - } > - my_hash_delete(&tdc_hash, (uchar*) share); > + > + share= element->share; > + DBUG_ASSERT(share); > + element->share= 0; > /* Notify PFS early, while still locked. */ "while still locked" what? > PSI_CALL_release_table_share(share->m_psi); > share->m_psi= 0; > - mysql_rwlock_unlock(&LOCK_tdc); > > - if (share->tdc.m_flush_tickets.is_empty()) > + if (!element->m_flush_tickets.is_empty()) > { > - /* No threads are waiting for this share to be flushed, destroy it. */ > - mysql_mutex_unlock(&share->tdc.LOCK_table_share); > - free_table_share(share); > - } > - else > - { > - Wait_for_flush_list::Iterator it(share->tdc.m_flush_tickets); > + Wait_for_flush_list::Iterator it(element->m_flush_tickets); > Wait_for_flush *ticket; > while ((ticket= it++)) > (void) ticket->get_ctx()->m_wait.set_status(MDL_wait::GRANTED); > - /* > - If there are threads waiting for this share to be flushed, > - the last one to receive the notification will destroy the > - share. At this point the share is removed from the table > - definition cache, so is OK to proceed here without waiting > - for this thread to do the work. > - */ > - mysql_mutex_unlock(&share->tdc.LOCK_table_share); > + > + do > + { > + mysql_cond_wait(&element->COND_release, &element->LOCK_table_share); > + } while (!element->m_flush_tickets.is_empty()); > } > - DBUG_RETURN(0); > + > + mysql_mutex_unlock(&element->LOCK_table_share); Ok, so the caller must lock LOCK_table_share? please put mysql_mutex_assert_owner() in the beginning of the function > + > + if (thd) When thd is NULL here? On shutdown? > + { > + fix_thd_pins(thd); > + pins= thd->tdc_hash_pins; > } > + else > + pins= lf_hash_get_pins(&tdc_hash); > > + DBUG_ASSERT(pins); // What can we do about it? > + element->dbug_release_assertions(); > + lf_hash_delete(&tdc_hash, pins, element->m_key, element->m_key_length); > + if (!thd) > + lf_hash_put_pins(pins); > + free_table_share(share); > + DBUG_VOID_RETURN; > +} > > + > /** > Initialize table definition cache. > - > - @retval 0 Success > - @retval !0 Error > */ > > -int tdc_init(void) > +void tdc_init(void) > { > DBUG_ENTER("tdc_init"); > #ifdef HAVE_PSI_INTERFACE > @@ -642,26 +548,34 @@ void tdc_deinit_share(TABLE_SHARE *share) > Caller is expected to unlock table share with tdc_unlock_share(). > > @retval 0 Share not found > - @retval !0 Pointer to locked table share > + @retval MY_ERRPTR OOM > + @retval ptr Pointer to locked table share > */ > > -TABLE_SHARE *tdc_lock_share(const char *db, const char *table_name) > +TDC_element *tdc_lock_share(THD *thd, const char *db, const char *table_name) > { > + TDC_element *element; > char key[MAX_DBKEY_LENGTH]; > - uint key_length; > > DBUG_ENTER("tdc_lock_share"); > - key_length= tdc_create_key(key, db, table_name); > + if (fix_thd_pins(thd)) > + DBUG_RETURN((TDC_element*) MY_ERRPTR); I suppose you can do it in THD:init, if you'd like. A THD almost always will need to access table definition cache, so it'll need these pins. Then there will be no need to return MY_ERRPTR here and handle it in the caller > > - mysql_rwlock_rdlock(&LOCK_tdc); > - TABLE_SHARE* share= (TABLE_SHARE*) my_hash_search(&tdc_hash, > - (uchar*) key, key_length); > - if (share && !share->error) > - mysql_mutex_lock(&share->tdc.LOCK_table_share); > - else > - share= 0; > - mysql_rwlock_unlock(&LOCK_tdc); > - DBUG_RETURN(share); > + element= (TDC_element *) lf_hash_search(&tdc_hash, thd->tdc_hash_pins, > + (uchar*) key, > + tdc_create_key(key, db, table_name)); > + if (element) > + { > + mysql_mutex_lock(&element->LOCK_table_share); > + if (!element->share || element->share->error) > + { > + mysql_mutex_unlock(&element->LOCK_table_share); > + element= 0; > + } > + lf_hash_search_unpin(thd->tdc_hash_pins); > + } > + > + DBUG_RETURN(element); Looks good to me > } > > > @@ -702,60 +616,60 @@ TABLE_SHARE *tdc_acquire_share(THD *thd, const char *db, const char *table_name, > TABLE **out_table) > { > TABLE_SHARE *share; > + TDC_element *element; > bool was_unused; > DBUG_ENTER("tdc_acquire_share"); > > - mysql_rwlock_rdlock(&LOCK_tdc); > - share= (TABLE_SHARE*) my_hash_search_using_hash_value(&tdc_hash, hash_value, > - (uchar*) key, > - key_length); > - if (!share) > + if (fix_thd_pins(thd)) > + DBUG_RETURN(0); > + > +retry: > + while (!(element= (TDC_element*) lf_hash_search_using_hash_value(&tdc_hash, Was lf_hash_search_using_hash_value added in a separate patch? > + thd->tdc_hash_pins, hash_value, (uchar*) key, key_length))) > { > - TABLE_SHARE *new_share; > - mysql_rwlock_unlock(&LOCK_tdc); > + TDC_element tmp(key, key_length); > + int res= lf_hash_insert(&tdc_hash, thd->tdc_hash_pins, (uchar*) &tmp); > > - if (!(new_share= alloc_table_share(db, table_name, key, key_length))) > + if (res == -1) > DBUG_RETURN(0); > - new_share->error= OPEN_FRM_OPEN_ERROR; > + else if (res == 1) > + continue; > > - mysql_rwlock_wrlock(&LOCK_tdc); > - share= (TABLE_SHARE*) my_hash_search_using_hash_value(&tdc_hash, hash_value, > - (uchar*) key, > - key_length); > - if (!share) > - { > - bool need_purge; > + element= (TDC_element*) lf_hash_search_using_hash_value(&tdc_hash, > + thd->tdc_hash_pins, hash_value, (uchar*) key, key_length); > + lf_hash_search_unpin(thd->tdc_hash_pins); How can you be sure that nobody frees your element after you unpin it? > + DBUG_ASSERT(element); > + element->dbug_release_assertions(); > > - share= new_share; > - mysql_mutex_lock(&share->tdc.LOCK_table_share); > - if (my_hash_insert(&tdc_hash, (uchar*) share)) > + if (!(share= alloc_table_share(db, table_name, key, key_length))) > { > - mysql_mutex_unlock(&share->tdc.LOCK_table_share); > - mysql_rwlock_unlock(&LOCK_tdc); > - free_table_share(share); > + lf_hash_delete(&tdc_hash, thd->tdc_hash_pins, key, key_length); > DBUG_RETURN(0); > } > - need_purge= tdc_hash.records > tdc_size; > - mysql_rwlock_unlock(&LOCK_tdc); > > /* note that tdc_acquire_share() *always* uses discovery */ > open_table_def(thd, share, flags | GTS_USE_DISCOVERY); > - share->tdc.ref_count++; > - mysql_mutex_unlock(&share->tdc.LOCK_table_share); > > if (share->error) > { > - tdc_delete_share_from_hash(share); > + free_table_share(share); > + lf_hash_delete(&tdc_hash, thd->tdc_hash_pins, key, key_length); > DBUG_RETURN(0); > } > - else if (need_purge) > + > + mysql_mutex_lock(&element->LOCK_table_share); > + element->share= share; > + share->tdc= element; > + element->ref_count++; > + element->version= tdc_refresh_version(); > + element->flushed= false; > + mysql_mutex_unlock(&element->LOCK_table_share); > + > tdc_purge(false); > if (out_table) > *out_table= 0; > share->m_psi= PSI_CALL_get_table_share(false, share); > goto end; > - } > - free_table_share(new_share); > } > > /* cannot force discovery of a cached share */ > @@ -763,18 +677,25 @@ TABLE_SHARE *tdc_acquire_share(THD *thd, const char *db, const char *table_name, > > if (out_table && (flags & GTS_TABLE)) > { > - if ((*out_table= tc_acquire_table(thd, share))) > + if ((*out_table= element->acquire_table(thd))) > { > - mysql_rwlock_unlock(&LOCK_tdc); > + lf_hash_search_unpin(thd->tdc_hash_pins); what this lf_hash_search_unpin corresponds to? > DBUG_ASSERT(!(flags & GTS_NOLOCK)); > - DBUG_ASSERT(!share->error); > - DBUG_ASSERT(!share->is_view); > - DBUG_RETURN(share); > + DBUG_ASSERT(element->share); > + DBUG_ASSERT(!element->share->error); > + DBUG_ASSERT(!element->share->is_view); > + DBUG_RETURN(element->share); > } > } > > - mysql_mutex_lock(&share->tdc.LOCK_table_share); > - mysql_rwlock_unlock(&LOCK_tdc); > + mysql_mutex_lock(&element->LOCK_table_share); > + if (!(share= element->share)) > + { > + mysql_mutex_unlock(&element->LOCK_table_share); > + lf_hash_search_unpin(thd->tdc_hash_pins); and here > + goto retry; > + } > + lf_hash_search_unpin(thd->tdc_hash_pins); may be that first lf_hash_search_unpin was wrong? the one that goes directly after lf_hash_search. > > /* > We found an existing table definition. Return it if we didn't get > @@ -983,13 +864,36 @@ bool tdc_remove_table(THD *thd, enum_tdc_remove_table_type remove_type, > thd->mdl_context.is_lock_owner(MDL_key::TABLE, db, table_name, > MDL_EXCLUSIVE)); > > - if ((share= tdc_delete_share(db, table_name))) > + > + mysql_mutex_lock(&LOCK_unused_shares); > + if (!(element= tdc_lock_share(thd, db, table_name))) > { > - I_P_List <TABLE, TABLE_share> purge_tables; > - uint my_refs= 1; > + mysql_mutex_unlock(&LOCK_unused_shares); > + DBUG_ASSERT(remove_type != TDC_RT_REMOVE_NOT_OWN_KEEP_SHARE); > + DBUG_RETURN(false); > + } > > - mysql_mutex_lock(&share->tdc.LOCK_table_share); > - tc_wait_for_mdl_deadlock_detector(share); > + DBUG_ASSERT(element != MY_ERRPTR); // What can we do about it? allocate thd->tdc_hash_pins in THD::init > + > + if (!element->ref_count) > + { > + if (element->prev) > + { > + unused_shares.remove(element); > + /* Concurrent thread may start using share again, reset prev and next. */ What concurrent thread? element->ref_count is 0, you removed the the share from the unused_shares list. There can be no concurrent threads, as far as I understand. And indeed, you do remove the share from the hash (which does free_table_share) immediately below, which also proves that there can be no concurrent threads. > + element->prev= 0; > + element->next= 0; > + } > + mysql_mutex_unlock(&LOCK_unused_shares); > + > + tdc_delete_share_from_hash(element); > + DBUG_RETURN(true); > + } > + mysql_mutex_unlock(&LOCK_unused_shares); > + > + element->ref_count++; > + > + element->wait_for_mdl_deadlock_detector(); > /* > Mark share flushed in order to ensure that it gets > automatically deleted once it is no longer referenced. Regards, Sergei

2 4

[Maria-developers] ccache with mariadb?
by Otto Kekäläinen 28 Dec '14

28 Dec '14

Hello! Has somebody managed to configure ccache correctly so that it would work with MariaDB compilation? https://ccache.samba.org/ I don't seem to get it working as the build always seems to pick up the real gcc binary instead of any of the ccache symlink wrappers I've manually put in place to replace all of gcc, cc1, cc1plus etc.

2 3

Re: [Maria-developers] MDEV-7273 - 10.1 fails to start up during tc_log initializations on PPC64
by Sergei Golubchik 26 Dec '14

26 Dec '14

Hi, Sergey! On Dec 26, svoj(a)mariadb.org wrote: > revision-id: f65901eef28728212a350f61569dc884c747bffc > parent(s): 8c616cd347aa7b3fcc2d8defb5e172ad316f432c > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2014-12-26 23:38:45 +0400 > message: > > MDEV-7273 - 10.1 fails to start up during tc_log initializations on PPC64 > > log-tc-size is 24K by default. Page size is 64K on PPC64. But log-tc-size > must be at least 3 x page size. This is enforced by TC_LOG_MMAP::open() > with a comment: to guarantee non-empty pool. > > This all makes server not startable in default configuration on PPC64. > > Autosize log-tc-size, so that it's min value= page size * 3, default > value= page size * 6, block size= page size. > > diff --git a/sql/log.cc b/sql/log.cc > index 66e1426..52c85d6 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -8780,6 +8779,7 @@ int TC_LOG_MMAP::sync() > int TC_LOG_MMAP::unlog(ulong cookie, my_xid xid) > { > pending_cookies *full_buffer= NULL; > + uint32 ncookies= tc_log_page_size / sizeof(my_xid); > DBUG_ASSERT(*(my_xid *)(data+cookie) == xid); > > /* > @@ -8793,7 +8793,7 @@ int TC_LOG_MMAP::unlog(ulong cookie, my_xid xid) > mysql_mutex_lock(&LOCK_pending_checkpoint); > if (pending_checkpoint == NULL) > { > - uint32 size= sizeof(*pending_checkpoint); > + uint32 size= sizeof(*pending_checkpoint) + sizeof(ulong) * (ncookies - 1); > if (!(pending_checkpoint= > (pending_cookies *)my_malloc(size, MYF(MY_ZEROFILL)))) > { > @@ -8804,8 +8804,7 @@ int TC_LOG_MMAP::unlog(ulong cookie, my_xid xid) > } > > pending_checkpoint->cookies[pending_checkpoint->count++]= cookie; > - if (pending_checkpoint->count == sizeof(pending_checkpoint->cookies) / > - sizeof(pending_checkpoint->cookies[0])) > + if (pending_checkpoint->count == ncookies) > { > full_buffer= pending_checkpoint; > pending_checkpoint= NULL; > @@ -8839,7 +8838,7 @@ int TC_LOG_MMAP::unlog(ulong cookie, my_xid xid) > if (count == 0) > { > uint i; > - for (i= 0; i < sizeof(pending->cookies)/sizeof(pending->cookies[0]); ++i) > + for (i= 0; i < tc_log_page_size / sizeof(my_xid); ++i) I wonder why you didn't use ncookies here :) > delete_entry(pending->cookies[i]); > my_free(pending); > } Regards, Sergei

2 1

Re: [Maria-developers] MDEV-7273 - 10.1 fails to start up during tc_log initializations on PPC64
by Sergei Golubchik 26 Dec '14

26 Dec '14

Hi, Sergey! Yes, that's what I had in mind, thanks! There are few rough edges, see below. Ok to push after you fix that, there's no need for another review round. On Dec 26, svoj(a)mariadb.org wrote: > revision-id: 72e58dc2b1a0bb859e918de0ff820d5a2a92dc32 > parent(s): db89dd3a8f7b0d868946d25ba98c6f88612d309a > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2014-12-26 17:03:45 +0400 > message: > > MDEV-7273 - 10.1 fails to start up during tc_log initializations on PPC64 > > log-tc-size is 24K by default. Page size is 64K on PPC64. But log-tc-size > must be at least 3 x page size. This is enforced by TC_LOG_MMAP::open() > with a comment: to guarantee non-empty pool. > > This all makes server not startable in default configuration on PPC64. > > Autosize log-tc-size, so that it's min value= page size * 6, default > value= page size * 6, block size= page size. > > --- > mysql-test/r/mysqld--help.result | 1 - > mysql-test/suite/sys_vars/r/log_tc_size_basic.result | 4 ++++ > .../suite/sys_vars/r/sysvars_server_notembedded.result | 14 ++++++++++++++ > mysql-test/suite/sys_vars/t/log_tc_size_basic.test | 5 +++++ > mysql-test/t/mysqld--help.test | 2 +- > sql/log.cc | 5 ++--- > sql/log.h | 3 +-- > sql/mysqld.cc | 7 +------ > sql/sys_vars.cc | 11 +++++++++++ > 9 files changed, 39 insertions(+), 13 deletions(-) > > diff --git a/mysql-test/suite/sys_vars/t/log_tc_size_basic.test b/mysql-test/suite/sys_vars/t/log_tc_size_basic.test > new file mode 100644 > index 0000000..1fce212 > --- /dev/null > +++ b/mysql-test/suite/sys_vars/t/log_tc_size_basic.test > @@ -0,0 +1,5 @@ > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR,ER_UNKNOWN_SYSTEM_VARIABLE > +SET GLOBAL log_tc_size=1; > + > +--error ER_INCORRECT_GLOBAL_LOCAL_VAR,ER_UNKNOWN_SYSTEM_VARIABLE do we ever build on platforms without mmap? if the answer is "no" - you don't need ER_UNKNOWN_SYSTEM_VARIABLE here > +SET SESSION log_tc_size=1; > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > index 006b9a9..5491169 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > @@ -1757,6 +1757,20 @@ NUMERIC_BLOCK_SIZE NULL > ENUM_VALUE_LIST innodb,query_plan,explain > READ_ONLY NO > COMMAND_LINE_ARGUMENT REQUIRED > +VARIABLE_NAME LOG_TC_SIZE > +SESSION_VALUE NULL > +GLOBAL_VALUE 24576 > +GLOBAL_VALUE_ORIGIN AUTO > +DEFAULT_VALUE 24576 > +VARIABLE_SCOPE GLOBAL > +VARIABLE_TYPE BIGINT UNSIGNED > +VARIABLE_COMMENT Size of transaction coordinator log. > +NUMERIC_MIN_VALUE 24576 > +NUMERIC_MAX_VALUE 18446744073709551615 > +NUMERIC_BLOCK_SIZE 4096 > +ENUM_VALUE_LIST NULL > +READ_ONLY YES > +COMMAND_LINE_ARGUMENT REQUIRED if the answer is "yes" - this test will fail there. even if the answer is "no", the variable value is architecture dependent. But it's good to see all stable variable metadata (e.g. that GLOBAL_VALUE_ORIGIN is AUTO). > VARIABLE_NAME LOG_WARNINGS > SESSION_VALUE 1 > GLOBAL_VALUE 1 > diff --git a/sql/log.cc b/sql/log.cc > index 66e1426..6d6512e 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -8793,7 +8792,7 @@ int TC_LOG_MMAP::unlog(ulong cookie, my_xid xid) > mysql_mutex_lock(&LOCK_pending_checkpoint); > if (pending_checkpoint == NULL) > { > - uint32 size= sizeof(*pending_checkpoint); > + uint32 size= sizeof(*pending_checkpoint) + tc_log_page_size / sizeof(my_xid); that's not enough. There are further sizeof's, like if (pending_checkpoint->count == sizeof(pending_checkpoint->cookies) / sizeof(pending_checkpoint->cookies[0])) and for (i= 0; i < sizeof(pending->cookies)/sizeof(pending->cookies[0]); ++i) I've found these two by looking for "sizeof" in the Kristian's patch that introduced pending_checkpoint. There can be more sizeofs added later. > if (!(pending_checkpoint= > (pending_cookies *)my_malloc(size, MYF(MY_ZEROFILL)))) > { > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index fb11377..bfaf7d7 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -5202,3 +5202,14 @@ static Sys_var_mybool Sys_strict_password_validation( > "that cannot be validated (passwords specified as a hash)", > GLOBAL_VAR(strict_password_validation), > CMD_LINE(OPT_ARG), DEFAULT(TRUE), NO_MUTEX_GUARD, NOT_IN_BINLOG); > + > +#ifdef HAVE_MMAP > +static Sys_var_ulong Sys_log_tc_size( > + "log_tc_size", > + "Size of transaction coordinator log.", > + READ_ONLY GLOBAL_VAR(opt_tc_log_size), > + CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(my_getpagesize() * 6, ULONG_MAX), > + DEFAULT(my_getpagesize() * 6), > + BLOCK_SIZE(my_getpagesize())); I think, min value could be my_getpagesize()*3 ? > +#endif Regards, Sergei

2 1

Re: [Maria-developers] MDEV-7273 - 10.1 fails to start up during tc_log initializations on PPC64
by Sergei Golubchik 25 Dec '14

25 Dec '14

Hi, Sergey! On Dec 17, svoj(a)mariadb.org wrote: > revision-id: 4a469b0ed68bb341e1a1e1ba540a45a3b6ff3ef9 > parent(s): ea8862aa9ffa1cfac9a99bbe0fe9b539f12b9a19 > committer: Sergey Vojtovich > branch nick: 10.1 > timestamp: 2014-12-17 13:55:01 +0400 > message: > > MDEV-7273 - 10.1 fails to start up during tc_log initializations on PPC64 > > log-tc-size is 24K by default. Page size is 64K on PPC64. But log-tc-size > must be at least 3 x page size. This is enforced by TC_LOG_MMAP::open() > with a comment: to guarantee non-empty pool. > > This all makes server not startable in default configuration on PPC64. > > When creating tc log, adjust it's size to max(log-tc-size, page size * 6). > > --- > sql/log.cc | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/sql/log.cc b/sql/log.cc > index 24e5283..a2fd70f 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -8316,7 +8316,7 @@ int TC_LOG_MMAP::open(const char *opt_name) > O_RDWR, MYF(MY_WME))) < 0) > goto err; > inited=1; > - file_length= opt_tc_log_size; > + file_length= MY_MAX(tc_log_page_size * 2 * 3, opt_tc_log_size); > if (mysql_file_chsize(fd, file_length, 0, MYF(MY_WME))) > goto err; > } I don't particularly like silent adjustments of user-specified values. And anyway, as you found out, this fix is not sufficient. I'd suggest the following: 1. Create a sysvar for log-tc-size 2. invoke my_getpagesize() not from log.cc, but from mysqld.cc, and set min and default values for log-tc-size appropriately, don't forget to use SYSVAR_AUTOSIZE() 3. change the code not to depend on the TC_LOG_PAGE_SIZE constant. - it's basically used in only two places now: to set a default for TC_LOG_MIN_SIZE (but you'll fix it step 2) and in kristian's code for group commit, cookies[] array. - don't mind TC_LOG_PAGE_SIZE being used in TC_LOG_BINLOG, it means nothing there, any reasonable number will do, I suppose. - so basically you only need to remove TC_LOG_PAGE_SIZE from cookies[] array. That is, you can define it as 'ulong cookies[1]' and take care to fix all sizeof() in the code to add the real size there. 4. Then you can remove that assert. The tc log page will be of my_getpagesize() size and log-tc-size will be N*my_getpagesize() where N>3. Regards, Sergei

1 0

[Maria-developers] JIRA ping
by Jonas Oreland 23 Dec '14

23 Dec '14

Hi Kristian, I light of recent "I did see any of your comments until by accident just now", I now mail you directly that I updated both MDEV-162 and MDEV-7257. I want (review) comments. /Jonas

2 9

[Maria-developers] Review of Debian packaging file listing?
by Otto Kekäläinen 22 Dec '14

22 Dec '14

Hello! I recently updated my build scripts to automatically generate a file listing for each commit in in the Debian packaging "pool" I have. Here is a recent one: http://labs.seravo.fi/~otto/mariadb-repo/mariadb-10.0-sid-amd64/filelist-ec… It is now very easy to review and track what files are in what package. Could somebody please quickly review that file and tell me if you spot: - any files in wrong package and should be moved? - any package that has too few files and maybe something missing? - any file attributes that might be wrong or insecure? - should any of the MariaDB plugins deserve to be shipped as a separate package? (apart from current connect and oqgraph) -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

4 5

[Maria-developers] Report from Debian/Ubuntu MySQL packaging sprint in London in December
by Otto Kekäläinen 21 Dec '14

21 Dec '14

Hello! I thought I'd brief the readers of this mailing list about the packaging sprint in London last week. It was organized by Canonical and in addition to Canonical staff and me there where also people from Oracle and Percona. We mostly worked on syncing the MySQL/MariaDB/Percona server packaging and a little bit also on creating a unified Galera package. We made good progress and all results have been pushed to public repos. I also recently updated the pages https://wiki.debian.org/Teams/MySQL/MariaDB and https://wiki.debian.org/Teams/MySQL/Galera to reflect the current status. One big technical change that was discussed and planned is the decoupling of the config and data directories. The plan is that in the future no MySQL variant in particular would anymore own the /etc/mysql/my.cnf file but instead each variant would register their own config file at installation time by using the Debian update-alternatives mechanism. So in future there would be /etc/mysql/mariadb.conf, mysql.conf, percona.conf etc and some more snippets in /etc/mysql/mariadb.conf.d/.. etc. The plan is not complete nor tested, but there are preliminary changes in the mysql-common package to provide the config file registration hook and in a preliminary Percona Debian package to use the new config file registration facility. As the config file defines the data directory, successful decoupling of config files would make it easy to also decouple the data directories so that if the user installs/removes/upgrades any of the MySQL variants they would not mess up their data accidentally and the installation scripts would help users to dump/migrate/copy/move their data in the common scenarios. -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 1

[Maria-developers] CREATE USER gotcha
by Alexander Barkov 19 Dec '14

19 Dec '14

Hi Ian, I think there is a CREATE USER related gotcha which is not described in the manual. CREATE USER allows to create multiple users at the same time: CREATE USER user1, user2; A reader of https://mariadb.com/kb/en/mariadb/documentation/sql-commands/account-manage… might think that the above query is a short synonym for: CREATE USER user1; CREATE USER user2; But in fact it is not always true. The difference can happen during replication. A "CREATE USER" statement is binlogged as a single shot, but only if at least one user was really created. Suppose (on master) user1 already exists, while user2 does not. "CREATE USER user1, user2" will be binary logged as is, while two independent "CREATE USER user1" and "CREARE USER user2" statements will be logged only as "CREATE USER user2". The CREATE statement for user1 won't be logged, because it failed on master! Now, suppose non of the users exist on slave. In case of "CREATE USER user1, user2;" both will be created on slave. In case of two independent queries, only "user2" will be created on slave.

4 6

[Maria-developers] Syntax for parallel replication
by Kristian Nielsen 17 Dec '14

17 Dec '14

Hi Serg, Can you help me with suggestions/comments for the following proposal for how to do configuration for MDEV-6676, speculative parallel replication? I am not too happy about how things are in 10.0. There, there is a single option --slave-parallel-threads=N. If N>0, then parallel replication is enabled, else not. One problem is that this makes it not configurable per multi-source master connection. Another problem is that there are two possible mechanisms for parallelisation, group-commit based and domain_id based; currently, one can enable none of them or both, but not just one of them. MDEV-6676 will introduce at least one other mechanism, which seems to make it essential to make a better way to configure this. Now, ideally, there would not be any configuration at all. The server would just run things in parallel when possible and to the degree desirable. However, I think there are some reasons that we need to allow the user this configurability: - Parallel replication is still a somewhat experimental feature, so it seems too risky to enable it by default. Also, it doesn't really seem possible for the server to automatically set the best number of threads to use, with current implementation (or possibly any implementation). - When replicating with non-transactional updates, or in non-gtid mode, slave state is not crash safe. This is true in non-parallel replication also, but in parallel replication, the problem seems amplified, as there may be multiple transactions in progress at the time of a crash, complicating possible manual recovery. This also suggests that parallel replication must be configurable. - When using domain-based parallel replication, the user is responsible for ensuring that independent domains are non-conflicting and can be replicated out-of-order wrt. each other. So if replication domains are used, but this property is not guaranteed, then domain-based parallel replication need to be configurable, or parallel replication cannot be used at all. - The new speculative replication feature in MDEV-6676 is not always guaranteed to be a win - in some workloads, where there are many conflicts between successive transactions, excessive rollback could cause it to be less efficient than not using it. Again, this suggests it needs to be configurable. So given this, I came up with the following idea for syntax: CHANGE MASTER TO PARALLEL_MODE=(domain,groupcommit,transactional,waiting) Each of the four keywords in the parenthesis is optional. "domain" enables domain-based parallelisation, where each replication domain is treated independently. "groupcommit" enables the non-speculative mode, where only transactions that group-committed together on the master are applied in parallel on the slave. "transactional" enables the speculative mode, where all transactional DML is optimistically tried in parallel, and then in case of conflict a rollback and retry is done. "groupcommit" and "transactional" are mutually exclusive, at most one of them can be specified. The default would be (domain,groupcommit) to be back-wards compatible with 10.0. If slave_parallel_thread=0, then no parallel apply will happen even if PARALLEL_MODE is non-empty. If slave_parallel_thread>0 but PARALLEL_MODE is empty (PARALLEL_MODE=()), then again no parallel apply will be done. The "waiting" option is not essential to add, we could remove it, I put it in because there were already a number of options so it seemed to cause no harm. The idea is that on the master, we detect if transaction T2 had to do a row lock wait on transaction T1. If so, it seems likely that a similar conflict could occur on the slave, so we will not run T2 in parallel with T1; instead we will let T2 wait for T1 to commit before T2 is started. The "waiting" option could be enabled by a user to disable this check, enabling even more aggressive parallelisation. I am not sure if it is worth it to have this configurable though, comments welcome. I have not checked how hard it will be to implement the new syntax. We do not have any similar multi-option CHANGE MASTER elements, as far as I know, but it is similar to ENUM system variables. And we already have a IGNORE_SERVER_IDS syntax with comma-separated list within parenthesis. So hopefully not too hard to do, and somewhat consistent with existing syntax. So what do you think? Is it a reasonable syntax? Any comments, or suggestions for better way to do it? Thanks, - Kristian.

5 23

Re: [Maria-developers] Rev 4383: MDEV-6916:Upgrade from MySQL to MariaDB breaks already created views
by Sergei Golubchik 17 Dec '14

17 Dec '14

Hi, Sanja! Comments about the code, see below. Comments about the whole approach - see my other email. On Dec 07, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-5.5-MDEV-6916/ > > ------------------------------------------------------------ > revno: 4383 > revision-id: sanja(a)askmonty.org-20141207094825-tx163unk3mm4gkql > parent: sanja(a)askmonty.org-20141203234414-63zkpeq9eb0n9lc1 > committer: sanja(a)askmonty.org > branch nick: work-maria-5.5-MDEV-6916 > timestamp: Sun 2014-12-07 10:48:25 +0100 > message: > MDEV-6916:Upgrade from MySQL to MariaDB breaks already created views > (SQL-level commands to support view update) > === modified file 'mysql-test/r/view.result' > --- a/mysql-test/r/view.result 2014-12-03 22:37:59 +0000 > +++ b/mysql-test/r/view.result 2014-12-07 09:48:25 +0000 > @@ -5398,6 +5398,29 @@ DROP VIEW v1; > DROP TABLE t1, t2; > create view v1 as select 1; > drop view v1; > +# > +#MDEV-6916:Upgrade from MySQL to MariaDB breaks already created views > +# (SQL command for upgrade check) > +# > +create table t1 (a int); > +create algorithm=temptable view v1 as select a from t1; > +show create view v1; > +View Create View character_set_client collation_connection > +v1 CREATE ALGORITHM=TEMPTABLE DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select `t1`.`a` AS `a` from `t1` latin1 latin1_swedish_ci > +check table v1 force mariadb upgrade; > +Table Op Msg_type Msg_text > +test.v1 check note View algorithm swap > +show create view v1; > +View Create View character_set_client collation_connection > +v1 CREATE ALGORITHM=MERGE DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select `t1`.`a` AS `a` from `t1` latin1 latin1_swedish_ci No, I don't like that. FORCE or not, upgrading of a view should not blindly change the alorithm. This functionality is created to solve a specific problem. So when a view has mariadb-version field we know with certainty that the algorithm is correct and it should *not* be changed with or without FORCE. In the test you'll need to copy mysql-created view frm (from std_data). Addendum: ok, I've read till the end. See my comment in view_check() function. > +check table v1 force mariadb upgrade; Also, I don't like that CHECK changes the algorithm of the view. For tables CHECK only performs the *check* (and updates the version in the frm). It's REPAIR that does the change. To be consistent with the behavior for tables, CHECK ... FOR UPGRADE should only report an error, like "view needs upgrading" and a user should then use REPAIR to actually fix the view. > +Table Op Msg_type Msg_text > +test.v1 check note View algorithm swap > +show create view v1; > +View Create View character_set_client collation_connection > +v1 CREATE ALGORITHM=TEMPTABLE DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select `t1`.`a` AS `a` from `t1` latin1 latin1_swedish_ci > +drop view v1; > +drop table t1; > # ----------------------------------------------------------------- > # -- End of 5.5 tests. > # ----------------------------------------------------------------- > > === modified file 'sql/share/errmsg-utf8.txt' > --- a/sql/share/errmsg-utf8.txt 2014-05-27 06:45:01 +0000 > +++ b/sql/share/errmsg-utf8.txt 2014-12-07 09:48:25 +0000 > @@ -6565,3 +6565,5 @@ ER_QUERY_EXCEEDED_ROWS_EXAMINED_LIMIT > ER_NO_SUCH_TABLE_IN_ENGINE 42S02 > eng "Table '%-.192s.%-.192s' doesn't exist in engine" > swe "Det finns ingen tabell som heter '%-.192s.%-.192s' i handlern" > +ER_VIEW_UPGRADE_IS_DONE > + eng "View algorithm swap" No way, you cannot add a new error message in 5.5 because the next major branch (10.0) is already GA. > === modified file 'sql/sql_admin.cc' > --- a/sql/sql_admin.cc 2014-05-03 16:12:17 +0000 > +++ b/sql/sql_admin.cc 2014-12-07 09:48:25 +0000 > @@ -314,7 +314,8 @@ static bool mysql_admin_table(THD* thd, > HA_CHECK_OPT *), > int (handler::*operator_func)(THD *, > HA_CHECK_OPT *), > - int (view_operator_func)(THD *, TABLE_LIST*)) > + int (view_operator_func)(THD *, TABLE_LIST*, > + HA_CHECK_OPT *)) > { > TABLE_LIST *table; > SELECT_LEX *select= &thd->lex->select_lex; > @@ -521,7 +522,7 @@ static bool mysql_admin_table(THD* thd, > ER_CHECK_NO_SUCH_TABLE, ER(ER_CHECK_NO_SUCH_TABLE)); > /* if it was a view will check md5 sum */ > if (table->view && > - view_checksum(thd, table) == HA_ADMIN_WRONG_CHECKSUM) > + view_check(thd, table, check_opt) == HA_ADMIN_WRONG_CHECKSUM) > push_warning(thd, MYSQL_ERROR::WARN_LEVEL_WARN, > ER_VIEW_CHECKSUM, ER(ER_VIEW_CHECKSUM)); > if (thd->stmt_da->is_error() && > @@ -536,7 +537,7 @@ static bool mysql_admin_table(THD* thd, > if (table->view) > { > DBUG_PRINT("admin", ("calling view_operator_func")); > - result_code= (*view_operator_func)(thd, table); > + result_code= (*view_operator_func)(thd, table, check_opt); > goto send_result; > } > > @@ -849,6 +850,14 @@ send_result_message: > system_charset_info); > break; > } > + case HA_ADMIN_VIEW_UPGRADE_IS_DONE: > + { > + protocol->store(STRING_WITH_LEN("note"), system_charset_info); > + protocol->store(ER(ER_VIEW_UPGRADE_IS_DONE), > + strlen(ER(ER_VIEW_UPGRADE_IS_DONE)), > + system_charset_info); > + break; > + } Don't do that. First, you cannot have ER_VIEW_UPGRADE_IS_DONE. Second, it's simply unnecessary. Return HA_ADMIN_OK from the view_operator_func() and use push_warning inside view_operator_func() to say that the algorithm was changed. > case HA_ADMIN_NEEDS_UPGRADE: > case HA_ADMIN_NEEDS_ALTER: > @@ -1071,7 +1080,7 @@ bool Check_table_statement::execute(THD > > res= mysql_admin_table(thd, first_table, &m_lex->check_opt, "check", > lock_type, 0, 0, HA_OPEN_FOR_REPAIR, 0, > - &handler::ha_check, &view_checksum); > + &handler::ha_check, &view_check); > > m_lex->select_lex.table_list.first= first_table; > m_lex->query_tables= first_table; > > === modified file 'sql/sql_view.cc' > --- a/sql/sql_view.cc 2014-12-03 23:44:14 +0000 > +++ b/sql/sql_view.cc 2014-12-07 09:48:25 +0000 > @@ -729,9 +729,29 @@ err: > DBUG_RETURN(res || thd->is_error()); > } > > +static void make_view_filename(LEX_STRING *dir, char *dir_buff, > + size_t dir_buff_len, > + LEX_STRING *path, char *path_buff, > + size_t path_buff_len, > + LEX_STRING *file, > + TABLE_LIST *view) > +{ > + /* print file name */ > + dir->length= build_table_filename(dir_buff, dir_buff_len - 1, > + view->db, "", "", 0); > + dir->str= dir_buff; > + > + path->length= build_table_filename(path_buff, path_buff_len - 1, > + view->db, view->table_name, reg_ext, 0); > + path->str= path_buff; > + > + file->str= path->str + dir->length; > + file->length= path->length - dir->length; > +} > + > > /* number of required parameters for making view */ > -static const int required_view_parameters= 15; > +static const int required_view_parameters= 16; eh? why? > /* > table of VIEW .frm field descriptors > @@ -795,6 +815,65 @@ static File_option view_parameters[]= > static LEX_STRING view_file_type[]= {{(char*) STRING_WITH_LEN("VIEW") }}; > > > +int mariadb_fix_view(THD *thd, TABLE_LIST *view) > +{ > + char dir_buff[FN_REFLEN + 1], path_buff[FN_REFLEN + 1]; > + LEX_STRING dir, file, path; > + DBUG_ENTER("mariadb_fix_view"); > + > + if (view->algorithm == VIEW_ALGORITHM_UNDEFINED) > + DBUG_RETURN(HA_ADMIN_OK); > + > + make_view_filename(&dir, dir_buff, sizeof(dir_buff), > + &path, path_buff, sizeof(path_buff), > + &file, view); > + /* init timestamp */ > + if (!view->timestamp.str) > + view->timestamp.str= view->timestamp_buffer; > + > + /* check old .frm */ > + { > + char path_buff[FN_REFLEN]; > + LEX_STRING path; > + File_parser *parser; > + > + path.str= path_buff; > + fn_format(path_buff, file.str, dir.str, "", MY_UNPACK_FILENAME); > + path.length= strlen(path_buff); > + if (access(path.str, F_OK)) > + DBUG_RETURN(HA_ADMIN_INVALID); > + > + if (!(parser= sql_parse_prepare(&path, thd->mem_root, 0))) > + DBUG_RETURN(HA_ADMIN_INTERNAL_ERROR); > + > + if (!parser->ok() || !is_equal(&view_type, parser->type())) > + DBUG_RETURN(HA_ADMIN_INVALID); > + } > + > + DBUG_ASSERT(view->algorithm == VIEW_ALGORITHM_MERGE || > + view->algorithm == VIEW_ALGORITHM_TMPTABLE); if() instead of an assert, please. while unlikely, it's possible that the file would be changed between two opens. > + if (view->algorithm == VIEW_ALGORITHM_MERGE) > + view->algorithm= VIEW_ALGORITHM_TMPTABLE; > + else > + view->algorithm= VIEW_ALGORITHM_MERGE; > + > + if (sql_create_definition_file(&dir, &file, view_file_type, > + (uchar*)view, view_parameters)) > + { > + sql_print_error("View '%-.192s'.'%-.192s': algorithm swap error.", "View %`s.%`s: cannot change the algorithm from %s to %s" And push_warning_printf() with the same. Or, rather, only push_warning_printf without sql_print_information. > + view->db, view->table_name); > + DBUG_RETURN(HA_ADMIN_INTERNAL_ERROR); > + } > + sql_print_information("View '%-.192s'.'%-.192s': algorithm swapped to '%s'", "View %`s.%`s: algorithm changed from %s to %s" And push_warning_printf() with the same. Or, rather, only push_warning_printf without sql_print_information. > + view->db, view->table_name, > + (view->algorithm == VIEW_ALGORITHM_MERGE)? > + "MERGE":"TEMPTABLE"); > + > + > + DBUG_RETURN(HA_ADMIN_VIEW_UPGRADE_IS_DONE); > +} > + > + > /* > Register VIEW (write .frm & process .frm's history backups) > > @@ -931,16 +1010,9 @@ static int mysql_register_view(THD *thd, > } > loop_out: > /* print file name */ > - dir.length= build_table_filename(dir_buff, sizeof(dir_buff) - 1, > - view->db, "", "", 0); > - dir.str= dir_buff; > - > - path.length= build_table_filename(path_buff, sizeof(path_buff) - 1, > - view->db, view->table_name, reg_ext, 0); > - path.str= path_buff; > - > - file.str= path.str + dir.length; > - file.length= path.length - dir.length; > + make_view_filename(&dir, dir_buff, sizeof(dir_buff), > + &path, path_buff, sizeof(path_buff), > + &file, view); > > /* init timestamp */ > if (!view->timestamp.str) > @@ -1953,6 +2025,33 @@ int view_checksum(THD *thd, TABLE_LIST * > HA_ADMIN_OK); > } you can make view_checksum() static now. > > +/** > + Check view > + > + @param thd thread handle > + @param view view for check > + @param check_opt check options > + > + @retval HA_ADMIN_OK OK > + @retval HA_ADMIN_NOT_IMPLEMENTED it is not VIEW > + @retval HA_ADMIN_WRONG_CHECKSUM check sum is wrong > +*/ > +int view_check(THD *thd, TABLE_LIST *view, HA_CHECK_OPT *check_opt) > +{ > + int res; > + DBUG_ENTER("view_check"); > + if ((res= view_checksum(thd, view)) != HA_ADMIN_OK) > + DBUG_RETURN(res); > + if (((check_opt->sql_flags & TT_MARIADB_UPGRADE) && > + !view->mariadb_version) || > + ((check_opt->sql_flags & (TT_MARIADB_UPGRADE | TT_FORCE)))) What is the point? Why would one ever want to change the algorithm of the provably created-in-mariadb view and pretend that it's an "upgrade"? To change the algorithm one should use ALTER. Fixing the view after an upgrade is not altering it, it's *fixing after an upgrade*. And views that were created in mariadb don't need to be fixed after an upgrade. No FORCE please. > + { > + DBUG_PRINT("XXX", ("Upgrade")); please either remove this DBUG_PRINT or change it to use a reasonable keyword. > + DBUG_RETURN(mariadb_fix_view(thd, view)); > + } > + DBUG_RETURN(HA_ADMIN_OK); > +} > + > === modified file 'sql/sql_yacc.yy' > --- a/sql/sql_yacc.yy 2014-11-08 18:54:42 +0000 > +++ b/sql/sql_yacc.yy 2014-12-07 09:48:25 +0000 > @@ -7299,6 +7300,10 @@ mi_check_type: > | EXTENDED_SYM { Lex->check_opt.flags|= T_EXTEND; } > | CHANGED { Lex->check_opt.flags|= T_CHECK_ONLY_CHANGED; } > | FOR_SYM UPGRADE_SYM { Lex->check_opt.sql_flags|= TT_FOR_UPGRADE; } > + | MARIADB_SYM UPGRADE_SYM > + { Lex->check_opt.sql_flags|= TT_MARIADB_UPGRADE; } > + | FORCE_SYM MARIADB_SYM UPGRADE_SYM > + { Lex->check_opt.sql_flags|= (TT_MARIADB_UPGRADE | TT_FORCE); } > ; > > optimize: Regards, Sergei

1 0

Re: [Maria-developers] Rev 4384: MDEV-6916: Upgrade from MySQL to MariaDB breaks already created views
by Sergei Golubchik 17 Dec '14

17 Dec '14

Hi, Sanja! Ok, I've seen all four patches. Specific comments about the code are below and in other emails. But the main question is - why are you doing it, what problem does it solve. As you said, the use case is - a user upgrades *from MySQL* and *runs mysql_upgrade*. To cover this use case mysql_upgrade needs to 1) detect whether it's an upgrade from MySQL, and to 2) run a command to change the algorithm of all views. You've done a lot of changes that aren't related to that. If this is, indeed, the use case, than you only need to: 1. add MySQL detection in mysql_upgrade. You have it, is_mysql() function. 2. add the code to toggle the algorithm of all views. See? No command line options, no FORCE, no CHECK, nothing. Now, let's consider extensions to this minimal functionality. 3. mariadb-version in the view frm file. this is a good idea, makes views a lot more future-proof. ok. I don't see why other extensions might be needed. Now, how to toggle the algorithm. The most natural solution would be to use ALTER VIEW. Like ALTER VIEW v1 ALGORITHM=MERGE. Unfortunately, ALTER VIEW doesn't work that way, I don't know why it was implemented in that crazy way in MySQL, but it's mostly useless. CHECK should not change its arguments. REPAIR? May be. I'd still prefer ALTER, but I don't know whether it can be extended in 5.5 in a safe way. If not then REPAIR ... AFTER UPGRADE is probably the second best. Or even REPAIR VIEW ... AFTER UPGRADE. It won't conflict with anything, doesn't add new keywords. Drawback - we'll be stuck with that hack for quite a while :( Even better would be to make mysql_upgrade to scan the data directory and rewrite view frms. No hacks or new grammar in the server. No new features in 5.5 GA. Nothing to maintain for years. Almost perfect. But it requires mysql_upgrade to have write access to the datadir. It kind of needs it already, but it's optional. With this approach it'll be less optional than before. So, I don't know... The most reasonable and future-proof solution would be to use ALTER TABLE. But I don't know whether we'll be able to do it in 5.5, looks a bit risky. Then there are reasonably safe hacks like REPAIR VIEW. And most safe and quite clean mysql_upgrade only approach, that needs write access to the datadir. Opinions? Anyway, here are comments about the code, below: On Dec 11, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-5.5-MDEV-6916/ > > ------------------------------------------------------------ > revno: 4384 > revision-id: sanja(a)askmonty.org-20141211180458-j8ajlibnvvydmj40 > parent: sanja(a)askmonty.org-20141207094825-tx163unk3mm4gkql > committer: sanja(a)askmonty.org > branch nick: work-maria-5.5-MDEV-6916 > timestamp: Thu 2014-12-11 19:04:58 +0100 > message: > MDEV-6916: Upgrade from MySQL to MariaDB breaks already created views > > mysql_upgrade now fix views from mysql. > > (+ fix new view detection) > === modified file 'client/mysql_upgrade.c' > --- a/client/mysql_upgrade.c 2014-03-27 21:26:58 +0000 > +++ b/client/mysql_upgrade.c 2014-12-11 18:04:58 +0000 > @@ -40,7 +40,8 @@ static char mysql_path[FN_REFLEN]; > static char mysqlcheck_path[FN_REFLEN]; > > static my_bool opt_force, opt_verbose, debug_info_flag, debug_check_flag, > - opt_systables_only, opt_version_check; > + opt_systables_only, opt_version_check, > + opt_mysql_upgrade= 0, opt_skip_mysql_upgrade= 0; > static my_bool opt_not_used, opt_silent; > static uint my_end_arg= 0; > static char *opt_user= (char*)"root"; > @@ -150,6 +151,14 @@ static struct my_option my_long_options[ > &opt_not_used, &opt_not_used, 0, GET_BOOL, NO_ARG, 1, 0, 0, 0, 0, 0}, > {"version", 'V', "Output version information and exit.", 0, 0, 0, > GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0}, > + {"mysql-upgrade", 'y', > + "Skip automatic detection MySQL and assume that we upgrade it", > + &opt_mysql_upgrade, &opt_mysql_upgrade, 0, GET_BOOL, NO_ARG, > + 0, 0, 0, 0, 0, 0}, > + {"skip-mysql-upgrade", 'Y', > + "Skip view algorithm upgrade from MySQL", > + &opt_skip_mysql_upgrade, &opt_skip_mysql_upgrade, 0, GET_BOOL, NO_ARG, > + 0, 0, 0, 0, 0, 0}, Those two are very bad option names. First, $ mysql_upgrade --mysql-upgrade looks plain weird. $ mysql_upgrade --skip-mysql-upgrade is even worse. And what about $ mysql_upgrade --mysql-upgrade --skip-mysql-upgrade Second, "skip" is the prefix automatically recognized by my_getopt. Normally, --skip-xxx is the same as --xxx=0. But now you're introducing --skip-mysql-upgrade which is very different from --skip-mysql-upgrade. While simultaneously adding support for --skip-skip-mysql-upgrade. A better option would be --fix-view-algorithm = { AUTO | ALWAYS | NEVER } > {"version-check", 'k', "Run this program only if its \'server version\' " > "matches the version of the server to which it's connecting, (enabled by " > "default); use --skip-version-check to avoid this check. Note: the \'server " > @@ -754,6 +771,23 @@ static int run_mysqlcheck_upgrade(void) > NULL); > } > > +static int run_mysqlcheck_views(void) > +{ > + if (!opt_mysql_upgrade) > + return 0; > + verbose("Phase 0: Fixing views"); No "Phase 0" please, renumber all phases to start from 1. > + print_conn_args("mysqlcheck"); > + return run_tool(mysqlcheck_path, > + NULL, /* Send output from mysqlcheck directly to screen */ > + "--no-defaults", > + ds_args.str, > + "--all-databases", > + "--view-upgrade", > + opt_verbose ? "--verbose": "", > + opt_silent ? "--silent": "", > + "2>&1", > + NULL); > +} > > === modified file 'client/mysqlcheck.c' > --- a/client/mysqlcheck.c 2014-02-17 10:00:51 +0000 > +++ b/client/mysqlcheck.c 2014-12-11 18:04:58 +0000 > @@ -42,7 +42,8 @@ static my_bool opt_alldbs = 0, opt_check > opt_medium_check = 0, opt_quick = 0, opt_all_in_1 = 0, > opt_silent = 0, opt_auto_repair = 0, ignore_errors = 0, > tty_password= 0, opt_frm= 0, debug_info_flag= 0, debug_check_flag= 0, > - opt_fix_table_names= 0, opt_fix_db_names= 0, opt_upgrade= 0; > + opt_fix_table_names= 0, opt_fix_db_names= 0, opt_upgrade= 0, > + opt_mariadb_upgrade= 0, opt_f_mariadb_upgrade= 0; > static my_bool opt_write_binlog= 1, opt_flush_tables= 0; > static uint verbose = 0, opt_mysql_port=0; > static int my_end_arg; > @@ -196,6 +197,12 @@ static struct my_option my_long_options[ > NO_ARG, 0, 0, 0, 0, 0, 0}, > {"version", 'V', "Output version information and exit.", 0, 0, 0, GET_NO_ARG, > NO_ARG, 0, 0, 0, 0, 0, 0}, > + {"view-upgrade", 'f', > + "Fix view algorithm view field if it is not new MariaDB view.", "Repair algorithm value for views created by MySQL (or MariaDB before 5.5.41)" > + 0, 0, 0, GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0}, > + {"force-view-upgrade", 'F', > + "Swap view algorithm view field.", "Always change view algorithm" > + 0, 0, 0, GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0}, > {0, 0, 0, 0, 0, 0, GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0} > }; Regards, Sergei

1 0

[Maria-developers] Invisible Indexes and .frm Change?
by James Briggs 16 Dec '14

16 Dec '14

Hi list. I'm looking into adding invisible indexes. There's 3 "scopes" I've thought of: 1) global and persistent (create table, alter table, create index) 2) global temporary (alter table t1 change index invisible global) 3) session temporary (alter table t1 change index invisible session) #1 likely means changing the .frm format, which would cause backwardcompatibility issues. Are there any upcoming changes to .frm planned? Any comments or suggestions? Thanks, James Briggs. -- Cassandra/MySQL DBA. Available in San Jose area or remote. cass_top: https://github.com/jamesbriggs/cassandra-top

2 1

Re: [Maria-developers] Rev 4350: MDEV-6985: MariaDB crashes on stored procedure call
by Sergei Golubchik 16 Dec '14

16 Dec '14

Hi, Sanja! On Nov 17, sanja(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 4350 > revision-id: sanja(a)askmonty.org-20141117132635-gf281ipiomifw1oc > parent: psergey(a)askmonty.org-20141113111559-cuhsyfhor141qo6a > committer: sanja(a)askmonty.org > branch nick: work-maria-5.5-MDEV-6985 > timestamp: Mon 2014-11-17 14:26:35 +0100 > message: > MDEV-6985: MariaDB crashes on stored procedure call > > Item_ident fixed to allow double cleanup(). > === modified file 'sql/item.cc' > --- a/sql/item.cc 2014-10-06 17:53:55 +0000 > +++ b/sql/item.cc 2014-11-17 13:26:35 +0000 > @@ -862,12 +862,20 @@ void Item_ident::cleanup() > field_name ? field_name : "(null)", > orig_field_name ? orig_field_name : "(null)")); > #endif Before pushing, please remove the above #ifdef, in a separate commit. thanks! > + bool was_fixed= fixed; > Item::cleanup(); > db_name= orig_db_name; > table_name= orig_table_name; > field_name= orig_field_name; > /* Store if this Item was depended */ > - can_be_depended= test(depended_from); > + if (was_fixed) > + { > + /* > + We can trust that depended_from set correctly only if this item > + was fixed > + */ > + can_be_depended= test(depended_from); > + } Are double cleanups generally normal? Why this item is cleaned up twice? > DBUG_VOID_RETURN; > } Regards, Sergei

2 2

Re: [Maria-developers] Rev 4382: Added creation time to the view frm
by Sergei Golubchik 16 Dec '14

16 Dec '14

Hi, Sanja! On Dec 03, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-5.5-MDEV-6916/ > > ------------------------------------------------------------ > revno: 4382 > revision-id: sanja(a)askmonty.org-20141203234414-63zkpeq9eb0n9lc1 > parent: sanja(a)askmonty.org-20141203223759-316k8y35qdb4cgyq > committer: sanja(a)askmonty.org > branch nick: work-maria-5.5-MDEV-6916 > timestamp: Thu 2014-12-04 00:44:14 +0100 > message: > Added creation time to the view frm. > ALTER VIEW saves it. CREATE VIEW & CREATE OR REPLACE VIEW reset it. Please no. This has nothing to do with MDEV-6916, it's a separate task and should be reviewed and discussed separately, not sneaked in under the MDEV-6916 cover. Regards, Sergei

1 0

Re: [Maria-developers] Rev 4381: Adding mariadb-version on the view creation to view frm. (MDEV-6916 followup)
by Sergei Golubchik 16 Dec '14

16 Dec '14

Hi, Sanja! On Dec 03, sanja(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 4381 > revision-id: sanja(a)askmonty.org-20141203223759-316k8y35qdb4cgyq > parent: jplindst(a)mariadb.org-20141203113839-xkxdy39mq77vu99y > committer: sanja(a)askmonty.org > branch nick: work-maria-5.5-MDEV-6916 > timestamp: Wed 2014-12-03 23:37:59 +0100 > message: > Adding mariadb-version on the view creation to view frm. (MDEV-6916 followup) Looks ok, thanks Regards, Sergei

1 0

[Maria-developers] Review of MDEV-7112 Split HA_CREATE_INFO
by Sergei Golubchik 04 Dec '14

04 Dec '14

Hi. It think the split is good, I liked it. I didn't like that you've tried to put the refactoring of sql_db.cc and sql_lang.h in the same commit, please move it to a separate patch. MDEV-5359 should not depend on that. Anyway, see all my comments below: > diff --git a/sql/structs.h b/sql/structs.h > index 99561c5..b38685c 100644 > --- a/sql/structs.h > +++ b/sql/structs.h > @@ -471,4 +471,67 @@ class Discrete_intervals_list { > Discrete_interval* get_current() const { return current; }; > }; > > + > +struct DDL_options_st > +{ > +public: > + enum Options > + { > + OPT_NONE= 0, > + OPT_IF_NOT_EXISTS= 2, // CREATE TABLE IF NOT EXISTS > + OPT_LIKE= 4, // CREATE TABLE LIKE > + OPT_OR_REPLACE= 16, // CREATE OR REPLACE TABLE > + OPT_OR_REPLACE_SLAVE_GENERATED= 32,// REPLACE was added on slave, it was > + // not in the original query on master. > + OPT_IF_EXISTS= 64 > + }; > + > +private: > + Options m_options; > + > +protected: > + > +public: > + Options create_like_options() const > + { > + return (DDL_options_st::Options) > + (((uint) m_options) & (OPT_IF_NOT_EXISTS | OPT_OR_REPLACE)); > + } > + void init() { m_options= OPT_NONE; } > + void init(Options options) { m_options= options; } > + void set(Options other) > + { > + m_options= other; > + } > + void set(const DDL_options_st other) > + { > + m_options= other.m_options; > + } > + bool if_not_exists() const { return m_options & OPT_IF_NOT_EXISTS; } > + bool or_replace() const { return m_options & OPT_OR_REPLACE; } > + bool or_replace_slave_generated() const > + { return m_options & OPT_OR_REPLACE_SLAVE_GENERATED; } > + bool like() const { return m_options & OPT_LIKE; } > + bool if_exists() const { return m_options & OPT_IF_EXISTS; } > + void add(const DDL_options_st::Options other) > + { > + m_options= (Options) ((uint) m_options | (uint) other); > + } > + void add(const DDL_options_st &other) > + { > + add(other.m_options); > + } > + DDL_options_st operator|(const DDL_options_st &other) > + { > + add(other.m_options); > + return *this; > + } > + DDL_options_st operator|=(DDL_options_st::Options other) > + { > + add(other); > + return *this; > + } > +}; I don't particularly like this style of C++ing... It's way too verbose. but ok > + > + > #endif /* STRUCTS_INCLUDED */ > diff --git a/sql/handler.h b/sql/handler.h > index 0044556..7b6a5f4 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -376,10 +376,7 @@ enum enum_alter_inplace_result { > #define HA_KEY_BLOB_LENGTH 2 > > #define HA_LEX_CREATE_TMP_TABLE 1 > -#define HA_LEX_CREATE_IF_NOT_EXISTS 2 > -#define HA_LEX_CREATE_TABLE_LIKE 4 > #define HA_CREATE_TMP_ALTER 8 > -#define HA_LEX_CREATE_REPLACE 16 One empty line here, please, HA_MAX_REC_LENGTH is not a flag > #define HA_MAX_REC_LENGTH 65535 > > /* Table caching type */ > @@ -1571,9 +1568,9 @@ enum enum_stats_auto_recalc { HA_STATS_AUTO_RECALC_DEFAULT= 0, > HA_STATS_AUTO_RECALC_ON, > HA_STATS_AUTO_RECALC_OFF }; > Could you add structure comments here? Like what you have in MDEV, "This will be passed to mysql_create_db, mysql_drop_db, mysqld_show_create_db", etc To explain what every structure is for, what kind of data it should store, and expected users (internal DB DDL, internal table DDL, handler, etc) > -struct HA_CREATE_INFO > +struct Table_contents_source_st > { > - CHARSET_INFO *table_charset, *default_table_charset; > + CHARSET_INFO *table_charset; > LEX_CUSTRING tabledef_version; > LEX_STRING connect_string; > const char *password, *tablespace; > @@ -1635,6 +1635,55 @@ struct HA_CREATE_INFO > }; > comments for all new structures, please > +struct Schema_specification_st > +{ > + CHARSET_INFO *default_table_charset; > + void init() > + { > + bzero(this, sizeof(*this)); > + } > +}; > + > + > +struct HA_CREATE_INFO: public Table_contents_source_st, > + public Schema_specification_st > +{ > + void init() > + { > + Table_contents_source_st::init(); > + Schema_specification_st::init(); > + } > +}; > + > + > +struct Table_specification_st: public HA_CREATE_INFO, > + public DDL_options_st > +{ > + // Deep initialization > + void init() > + { > + HA_CREATE_INFO::init(); > + DDL_options_st::init(); > + } > + void init(DDL_options_st::Options options) > + { > + HA_CREATE_INFO::init(); > + DDL_options_st::init(options); > + } > + /* > + Quick initialization, for parser. > + Most of the HA_CREATE_INFO is left uninitialized. > + It gets fully initialized in sql_yacc.yy, only when the parser > + scans a related keyword (e.g. CREATE, ALTER). > + */ > + void lex_start() > + { > + HA_CREATE_INFO::options= 0; > + DDL_options_st::init(); > + } > +}; > + > + > /** > In-place alter handler context. > > diff --git a/sql/sql_lang.h b/sql/sql_lang.h > new file mode 100644 > index 0000000..76a1340 > --- /dev/null > +++ b/sql/sql_lang.h > @@ -0,0 +1,79 @@ > +/* Copyright (c) 2014, MariaDB Foundation > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > +#ifndef SQL_LANG_INCLUDED > +#define SQL_LANG_INCLUDED > + > +namespace SQL { > + > +class Identifier: protected LEX_CSTRING > +{ > + void set(const LEX_STRING &other) > + { > + str= other.str; > + length= other.length; > + DBUG_ASSERT(str[length] == 0); > + } > +public: > + Identifier(const LEX_STRING &other) > + { > + set(other); > + } > + Identifier(const LEX *); > + const char *name() const { return str; } > + const size_t name_length() const { return length; } > +}; Eh? What all this has to do with MDEV-7112 "Split HA_CREATE_INFO" ? it's not mentioned there. If it's unrelated refactoring, please, put it in a separate patch. > + > + > +class Identifier_normalized: public Identifier > +{ > + char m_buf[NAME_LEN + 1 + MYSQL50_TABLE_NAME_PREFIX_LENGTH + 10]; > + void normalize(); > +public: > + Identifier_normalized(const LEX_STRING &other) > + :Identifier(other) { normalize(); } > + Identifier_normalized(const LEX *lex) > + :Identifier(lex) { normalize(); } > +}; > + > + > +namespace Statement { > + > + > +/* > + All SQL statements that can be executed directly. > + Statement instances are created using the data in thd->lex, > + previously populated by the SQL parser. > +*/ > +class Directly_executable > +{ > +public: > + /* > + Perform all stages of a direct SQL execution: > + - Check for validity (e.g. name in "CREATE SCHEMA name" is valid). > + - Check access > + - Apply filters (e.g. Rpl_filters on slave) > + - Perform WSREP_TO_ISOLATION > + - Execute the actual action > + */ > + virtual int exec_direct(THD *thd) const = 0; > +}; > + > +} // End of namescpace Statement > + > + > +} // End of namespace SQL > + > +#endif /* SQL_LANG_INCLUDED */ > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 0bbcca5..0edda65 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -1923,7 +1923,9 @@ bool MDL_deadlock_handler::handle_condition(THD *, > */ > > static bool > -open_table_get_mdl_lock(THD *thd, Open_table_context *ot_ctx, > +open_table_get_mdl_lock(THD *thd, > + const DDL_options_st &options, > + Open_table_context *ot_ctx, What's the point - you don't seem to be using options in this function? And that recursively means that you don't need 'options' in all other functions that only pass 'options' argument down to open_table_get_mdl_lock() but otherwise don't use it. > MDL_request *mdl_request, > uint flags, > MDL_ticket **mdl_ticket) > @@ -4176,8 +4182,8 @@ thr_lock_type read_lock_type_for_table(THD *thd, > DBUG_RETURN(FALSE); > > /* Check if CREATE TABLE without REPLACE was used */ > - create_table= (thd->lex->sql_command == SQLCOM_CREATE_TABLE && > - !(thd->lex->create_info.options & HA_LEX_CREATE_REPLACE)); > + create_table= thd->lex->sql_command == SQLCOM_CREATE_TABLE && > + !options.or_replace(); why, options are still in thd->lex, aren't they? > > if (!(flags & MYSQL_OPEN_SKIP_SCOPED_MDL_LOCK)) > { > diff --git a/sql/sql_db.h b/sql/sql_db.h > index 62d379c..dc62e6b 100644 > --- a/sql/sql_db.h > +++ b/sql/sql_db.h > @@ -35,13 +33,178 @@ bool mysql_opt_change_db(THD *thd, > bool my_dboptions_cache_init(void); > void my_dboptions_cache_free(void); > bool check_db_dir_existence(const char *db_name); > -bool load_db_opt(THD *thd, const char *path, HA_CREATE_INFO *create); > +bool load_db_opt(THD *thd, const char *path, Schema_specification_st *create); > bool load_db_opt_by_name(THD *thd, const char *db_name, > - HA_CREATE_INFO *db_create_info); > + Schema_specification_st *db_create_info); > CHARSET_INFO *get_default_db_collation(THD *thd, const char *db_name); > bool my_dbopt_init(void); > void my_dbopt_cleanup(void); > > + > +namespace SQL { > + > + > +namespace Statement { > + > + > +namespace Schema { > + > + > +// Schema name > +class Name: public Identifier_normalized > +{ > +public: > + Name(const LEX_STRING &other) :Identifier_normalized(other) { } > + Name(const LEX *lex) :Identifier_normalized(lex) { } > +#ifdef HAVE_REPLICATION > + bool check_if_slave_ignored(THD *thd) const; > +#endif > + bool check() const; > + bool check_with_error() const; > +}; > + > + > +// Schema definition options: IF EXISTS, IF NOT EXISTS > +class DDL_options: public DDL_options_st > +{ > +public: > + DDL_options() { init(); } > + DDL_options(Options options) > + { > + set(options); > + } > + DDL_options(const DDL_options_st options) > + { > + set(options); > + } > + DDL_options(const LEX *lex); > +}; > + > + > +// Schema specification, e.g. CHARACTER SET xxx > +class Specification: public Schema_specification_st > +{ > +public: > + Specification(const Schema_specification_st &info) > + :Schema_specification_st(info) > + { } > + Specification(const LEX *lex); > +}; > + > + > +/** > + A common class for all Schema DDL options: > + - CREATE SCHEMA > + - DROP SCHEMA > + - ALTER SCHEMA > + - ALTER SCHEMA UPGRADE DATA DIRECTORY NAME > +*/ > +class DDL: public Directly_executable > +{ > +protected: > + bool rm_db(THD *thd, const Name &name, bool if_exists, bool silent) const; > + > + int create_db(THD *thd, > + const Name &name, > + const DDL_options_st options, > + const Schema_specification_st &info, > + bool silent) const; > +}; > + > + > +class DDL_with_name: public DDL, public Name > +{ > + /* > + Execute a DDL statement. > + Assumes that Name has already been checked for validity. > + */ > + virtual bool exec(THD *thd) const = 0; > + virtual bool check_access(THD *) const = 0; > +protected: > + DDL_with_name(const Name &name): Name(name) { } > + DDL_with_name(const LEX *lex): Name(lex) { } > +public: > + int exec_direct(THD *thd) const; > +}; > + > + > +class DDL_with_name_and_options: public DDL_with_name, public DDL_options > +{ > +protected: > + DDL_with_name_and_options(const LEX *lex) > + :DDL_with_name(lex), DDL_options(lex) { } > + DDL_with_name_and_options(const Name &name) > + :DDL_with_name(name), DDL_options() { } > +}; > + > + > +class Create: public DDL_with_name_and_options, public Specification > +{ > + bool exec(THD *thd) const > + { > + return DDL::create_db(thd, *this, *this, *this, false); > + } > + bool check_access(THD *thd) const > + { > + return ::check_access(thd, CREATE_ACL, name(), NULL, NULL, 1, 0); > + } > +public: > + Create(const LEX *lex) > + :DDL_with_name_and_options(lex), Specification(lex) > + { } > + Create(const Name &name, const Schema_specification_st &info) > + :DDL_with_name_and_options(name), Specification(info) > + { } > +}; > + > + > +class Drop: public DDL_with_name_and_options > +{ > + bool exec(THD *thd) const > + { > + return rm_db(thd, *this, if_exists(), false); > + } > + bool check_access(THD *thd) const > + { > + return ::check_access(thd, DROP_ACL, name(), NULL, NULL, 1, 0); > + } > +public: > + Drop(const Name &name) :DDL_with_name_and_options(name) { } > + Drop(const LEX *lex) :DDL_with_name_and_options(lex) { } > +}; > + > + > +class Alter: public DDL_with_name, public Specification > +{ > + bool exec(THD *thd) const; > + bool check_access(THD *thd) const > + { > + return ::check_access(thd, ALTER_ACL, name(), NULL, NULL, 1, 0); > + } > +public: > + Alter(const LEX *lex) :DDL_with_name(lex), Specification(lex) { } > +}; > + > + > +class Upgrade: public DDL_with_name > +{ > + bool exec(THD *thd) const; > + bool check_access(THD *thd) const > + { > + return ::check_access(thd, ALTER_ACL, name(), NULL, NULL, 1, 0) || > + ::check_access(thd, DROP_ACL, name(), NULL, NULL, 1, 0) || > + ::check_access(thd, CREATE_ACL, name(), NULL, NULL, 1, 0); > + } > +public: > + Upgrade(const LEX *lex) :DDL_with_name(lex) { } > + int exec_direct(THD *thd) const; > +}; > + > + > +} // End of namespace SQL::Statement::Schema > +} // End of namespace SQL::Statement > +} // End of namespace SQL Please, no. This and sql_lang.h - make it a separate patch This one should be *only* about splitting HA_CREATE_INFO > + > #define MY_DB_OPT_FILE "db.opt" > > #endif /* SQL_DB_INCLUDED */ > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index b905d8b..c59332e 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -4181,143 +4176,21 @@ static bool do_execute_sp(THD *thd, sp_head *sp) > } > break; > case SQLCOM_CREATE_DB: > - { > - /* > - As mysql_create_db() may modify HA_CREATE_INFO structure passed to > - it, we need to use a copy of LEX::create_info to make execution > - prepared statement- safe. > - */ > - HA_CREATE_INFO create_info(lex->create_info); > - if (check_db_name(&lex->name)) > - { > - my_error(ER_WRONG_DB_NAME, MYF(0), lex->name.str); > - break; > - } > - /* > - If in a slave thread : > - CREATE DATABASE DB was certainly not preceded by USE DB. > - For that reason, db_ok() in sql/slave.cc did not check the > - do_db/ignore_db. And as this query involves no tables, tables_ok() > - above was not called. So we have to check rules again here. > - */ > -#ifdef HAVE_REPLICATION > - if (thd->slave_thread) > - { > - rpl_filter= thd->system_thread_info.rpl_sql_info->rpl_filter; > - if (!rpl_filter->db_ok(lex->name.str) || > - !rpl_filter->db_ok_with_wild_table(lex->name.str)) > - { > - my_message(ER_SLAVE_IGNORED_TABLE, ER(ER_SLAVE_IGNORED_TABLE), MYF(0)); > - break; > - } > - } > -#endif > - if (check_access(thd, CREATE_ACL, lex->name.str, NULL, NULL, 1, 0)) > - break; > - WSREP_TO_ISOLATION_BEGIN(lex->name.str, NULL, NULL) > - res= mysql_create_db(thd, lex->name.str, &create_info, 0); separate refactoring patch, please > + if ((res= SQL::Statement::Schema::Create(lex).exec_direct(thd)) < 0) > + goto error; > break; > - } > case SQLCOM_DROP_DB: > - { > - if (check_db_name(&lex->name)) > - { > - my_error(ER_WRONG_DB_NAME, MYF(0), lex->name.str); > - break; > - } > - /* > - If in a slave thread : > - DROP DATABASE DB may not be preceded by USE DB. > - For that reason, maybe db_ok() in sql/slave.cc did not check the > - do_db/ignore_db. And as this query involves no tables, tables_ok() > - above was not called. So we have to check rules again here. > - */ > -#ifdef HAVE_REPLICATION > - if (thd->slave_thread) > - { > - rpl_filter= thd->system_thread_info.rpl_sql_info->rpl_filter; > - if (!rpl_filter->db_ok(lex->name.str) || > - !rpl_filter->db_ok_with_wild_table(lex->name.str)) > - { > - my_message(ER_SLAVE_IGNORED_TABLE, ER(ER_SLAVE_IGNORED_TABLE), MYF(0)); > - break; > - } > - } > -#endif > - if (check_access(thd, DROP_ACL, lex->name.str, NULL, NULL, 1, 0)) > - break; > - WSREP_TO_ISOLATION_BEGIN(lex->name.str, NULL, NULL) > - res= mysql_rm_db(thd, lex->name.str, lex->check_exists, 0); > + if ((res= SQL::Statement::Schema::Drop(lex).exec_direct(thd)) < 0) > + goto error; > break; > - } > case SQLCOM_ALTER_DB_UPGRADE: > - { > - LEX_STRING *db= & lex->name; > -#ifdef HAVE_REPLICATION > - if (thd->slave_thread) > - { > - rpl_filter= thd->system_thread_info.rpl_sql_info->rpl_filter; > - if (!rpl_filter->db_ok(db->str) || > - !rpl_filter->db_ok_with_wild_table(db->str)) > - { > - res= 1; > - my_message(ER_SLAVE_IGNORED_TABLE, ER(ER_SLAVE_IGNORED_TABLE), MYF(0)); > - break; > - } > - } > -#endif > - if (check_db_name(db)) > - { > - my_error(ER_WRONG_DB_NAME, MYF(0), db->str); > - break; > - } > - if (check_access(thd, ALTER_ACL, db->str, NULL, NULL, 1, 0) || > - check_access(thd, DROP_ACL, db->str, NULL, NULL, 1, 0) || > - check_access(thd, CREATE_ACL, db->str, NULL, NULL, 1, 0)) > - { > - res= 1; > - break; > - } > - WSREP_TO_ISOLATION_BEGIN(db->str, NULL, NULL) > - res= mysql_upgrade_db(thd, db); > - if (!res) > - my_ok(thd); > + if ((res= SQL::Statement::Schema::Upgrade(lex).exec_direct(thd)) < 0) > + goto error; > break; > - } > case SQLCOM_ALTER_DB: > - { > - LEX_STRING *db= &lex->name; > - HA_CREATE_INFO create_info(lex->create_info); > - if (check_db_name(db)) > - { > - my_error(ER_WRONG_DB_NAME, MYF(0), db->str); > - break; > - } > - /* > - If in a slave thread : > - ALTER DATABASE DB may not be preceded by USE DB. > - For that reason, maybe db_ok() in sql/slave.cc did not check the > - do_db/ignore_db. And as this query involves no tables, tables_ok() > - above was not called. So we have to check rules again here. > - */ > -#ifdef HAVE_REPLICATION > - if (thd->slave_thread) > - { > - rpl_filter= thd->system_thread_info.rpl_sql_info->rpl_filter; > - if (!rpl_filter->db_ok(db->str) || > - !rpl_filter->db_ok_with_wild_table(db->str)) > - { > - my_message(ER_SLAVE_IGNORED_TABLE, ER(ER_SLAVE_IGNORED_TABLE), MYF(0)); > - break; > - } > - } > -#endif > - if (check_access(thd, ALTER_ACL, db->str, NULL, NULL, 1, 0)) > - break; > - WSREP_TO_ISOLATION_BEGIN(db->str, NULL, NULL) > - res= mysql_alter_db(thd, db->str, &create_info); > + if ((res= SQL::Statement::Schema::Alter(lex).exec_direct(thd)) < 0) > + goto error; > break; > - } > case SQLCOM_SHOW_CREATE_DB: > { > char db_name_buff[NAME_LEN+1]; > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 0541cbc..2536255 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -1696,14 +1695,14 @@ int show_create_table(THD *thd, TABLE_LIST *table_list, String *packet, > > packet->append(STRING_WITH_LEN("CREATE ")); > if (create_info_arg && > - (create_info_arg->org_options & HA_LEX_CREATE_REPLACE || > + ((create_info_arg->or_replace() && > + !create_info_arg->or_replace_slave_generated()) || that's a bit strange. Could you explain this? why org_options & HA_LEX_CREATE_REPLACE is translated to or_replace() && !or_replace_slave_generated() > create_info_arg->table_was_deleted)) > packet->append(STRING_WITH_LEN("OR REPLACE ")); > if (share->tmp_table) > packet->append(STRING_WITH_LEN("TEMPORARY ")); > packet->append(STRING_WITH_LEN("TABLE ")); > - if (create_info_arg && > - (create_info_arg->options & HA_LEX_CREATE_IF_NOT_EXISTS)) > + if (create_info_arg && create_info_arg->if_not_exists()) > packet->append(STRING_WITH_LEN("IF NOT EXISTS ")); > if (table_list->schema_table) > alias= table_list->schema_table->table_name; > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index b991215..7bb293b 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -4628,7 +4628,8 @@ int create_table_impl(THD *thd, > const char *orig_db, const char *orig_table_name, > const char *db, const char *table_name, > const char *path, > - HA_CREATE_INFO *create_info, > + const DDL_options_st options, > + HA_CREATE_INFO &create_info, why? I generally prefer pointers over C++ references - one can immediately see what variables can be modified by the function. is there a compelling reason to use references here? > Alter_info *alter_info, > int create_table_mode, > bool *is_trans, > @@ -5273,9 +5275,10 @@ bool mysql_create_like_table(THD* thd, TABLE_LIST* table, > */ > > /* Copy temporarily the statement flags to thd for lock_table_names() */ > + // QQ: is this really needed??? may be not anymore, as you pass flags separately > uint save_thd_create_info_options= thd->lex->create_info.options; > - thd->lex->create_info.options|= create_info->options; > - res= open_tables(thd, &thd->lex->query_tables, &not_used, 0); > + thd->lex->create_info.options|= create_info.options; > + res= open_tables(thd, create_info, &thd->lex->query_tables, &not_used, 0); > thd->lex->create_info.options= save_thd_create_info_options; > > if (res) > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 632bdd2..0f2c0e0 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -1621,7 +1623,8 @@ bool my_yyoverflow(short **a, YYSTYPE **b, ulong *yystacksize); > IDENT_sys TEXT_STRING_sys TEXT_STRING_literal > NCHAR_STRING opt_component key_cache_name > sp_opt_label BIN_NUM label_ident TEXT_STRING_filesystem ident_or_empty > - opt_constraint constraint opt_ident opt_if_not_exists_ident > + opt_constraint constraint opt_ident > + opt_if_not_exists_opt_table_element_name can you please try to come up with shorter identifier names? 40 characters - that's a bit too long :) > > %type <lex_str_ptr> > opt_table_alias > diff --git a/storage/xtradb/handler/ha_innodb.cc b/storage/xtradb/handler/ha_innodb.cc > index f3b1167..6b1942e 100644 > --- a/storage/xtradb/handler/ha_innodb.cc > +++ b/storage/xtradb/handler/ha_innodb.cc > @@ -11349,7 +11349,7 @@ static __attribute__((nonnull, warn_unused_result)) > > /* Do not use DATA DIRECTORY with TEMPORARY TABLE. */ > if (create_info->data_file_name > - && create_info->options & HA_LEX_CREATE_TMP_TABLE) { > + && create_info->tmp_table()) { nope, you should not change storage engine API here. tmp_table() was a convenience wrapper, old way should still work > push_warning( > thd, Sql_condition::WARN_LEVEL_WARN, > ER_ILLEGAL_HA_CREATE_OPTION, Regards, Sergei

2 4

[Maria-developers] better execution plan possible with INNER JOIN of mroonga (or whatever storage engine) ?
by Kazuhiko Shiozaki 03 Dec '14

03 Dec '14

Hello ! I am investigating a slow query and have a question about execution plan with mroonga (possibly any storage engine?). (with MariaDB 10.0.15.) Here are tables : CREATE TABLE catalog ( uid bigint(20) unsigned NOT NULL, path varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', PRIMARY KEY (uid) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; CREATE TABLE catalog_full_text ( uid bigint(20) unsigned NOT NULL, title varchar(255) COLLATE utf8_unicode_ci DEFAULT '', PRIMARY KEY (uid), FULLTEXT KEY title (title) COMMENT 'parser "TokenBigramSplitSymbolAlphaDigit"' ) ENGINE=Mroonga DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; CREATE TABLE category ( uid bigint(20) unsigned NOT NULL, category_uid bigint(20) unsigned NOT NULL DEFAULT '0', base_category_uid bigint(20) unsigned NOT NULL DEFAULT '0', PRIMARY KEY (uid,category_uid,base_category_uid) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; Here is the query : SELECT DISTINCT catalog.path, catalog.uid FROM ( ( catalog AS catalog INNER JOIN ( category AS category INNER JOIN catalog_full_text AS source_title_full_text ON category.base_category_uid = 18482 AND source_title_full_text.uid = category.category_uid ) ON category.uid = catalog.uid ) INNER JOIN catalog_full_text AS title_full_text ON title_full_text.uid = catalog.uid ) WHERE MATCH (source_title_full_text.title) AGAINST ('bar' IN BOOLEAN MODE) AND MATCH (title_full_text.title) AGAINST ('foo' IN BOOLEAN MODE) Here is the output of EXPLAIN : *************************** 1. row *************************** id: 1 select_type: SIMPLE table: source_title_full_text type: fulltext possible_keys: PRIMARY,title key: title key_len: 0 ref: rows: 1 Extra: Using where with pushed condition; Using temporary *************************** 2. row *************************** id: 1 select_type: SIMPLE table: title_full_text type: fulltext possible_keys: PRIMARY,title key: title key_len: 0 ref: rows: 1 Extra: Using where with pushed condition *************************** 3. row *************************** id: 1 select_type: SIMPLE table: catalog type: eq_ref possible_keys: PRIMARY key: PRIMARY key_len: 8 ref: test.title_full_text.uid rows: 1 Extra: *************************** 4. row *************************** id: 1 select_type: SIMPLE table: category type: eq_ref possible_keys: PRIMARY key: PRIMARY key_len: 24 ref: test.title_full_text.uid,test.source_title_full_text.uid,const rows: 1 Extra: Using index; Distinct As you see, we have two full-text queries at the beginning and both have 'rows = 1'. In a slow query case, 25k records mathces with the first full-text query and 1.6k records matches with the second full-text query. When I manually modify the order of INNER JOIN so that the 'lighter' full-text query appears first, EXPLAIN output is different as well and the performance is much improved. Accoding to Kohei, a Mroonga developer, we always have 'rows = 1' for mroonga full-text query because MariaDB does not request such value to mroonga (JOIN::get_examined_rows() in sql/sql_select.cc), and it is possible to modify mroonga to return an estimation of rows if MariaDB has such an API. So here are my questions : * can MariaDB request an 'estimation of rows' to mroonga (or whatever storage engine) ? * can such information help to decide a better execution plan with INNER JOIN like above ? Thanks in advance ! -- Kazuhiko Shiozaki, Nexedi SA Senior Consultant Nexedi: Consulting and Development of Free / Open Source Software http://www.nexedi.com/ ERP5: Full Featured High End Open Source ERP http://www.erp5.com/

3 3

[Maria-developers] Google's patches to MariaDB are available in the opensource repo
by Pavel Ivanov 03 Dec '14

03 Dec '14

Hi all, Effective immediately the full opensource tree of changes made to MariaDB by Google to make it work for our internal applications is available at https://code.google.com/p/google-mysql/. Note: I didn't update the main page on the site to reflect the changes yet, but the sources are there and the master branch points to the tip of MariaDB tree. Note #2: this tree is not an exact copy of our internal tree, and it's not intended to reflect verbatim our internal development process. It's just a set of changes split roughly one feature per commit with a grouping of several commits related to the same feature set into branches. That means that whenever we find a bug in some of our commits we'll squash the fix in the appropriate place rebuilding the whole git tree on top of that. As a consequence of that links to specific commits will not last forever, though links to specific branches likely will. Note #3: unfortunately code.google.com doesn't have a good visualization of branches in the git tree, so it's not easy to see which commits belong to which branch in our structure of branches stacked on top of each other. So the easiest way to see that is to clone the tree on your local disk and run gitk on it (or other tool that you find more convenient). Besides raising a general awareness of existence of our opensource tree, I'd like to draw attention of MariaDB developers especially to the following branches. I think many commits from there can be considered for inclusion into MariaDB proper. 16-encryption https://code.google.com/p/google-mysql/source/list?name=mariadb-10.0.12%2F1… Encryption of different subsystems writing to disk (not all subsystems included yet, but will be soon). 09-rpl https://code.google.com/p/google-mysql/source/list?name=mariadb-10.0.12%2F0… Replication-related fixes and improvements. 01-base https://code.google.com/p/google-mysql/source/list?name=mariadb-10.0.12%2F0… Mostly basic compilation and testing fixes. 03-privileges https://code.google.com/p/google-mysql/source/list?name=mariadb-10.0.12%2F0… Fixes for some quirks in privilege subsystem (aka sql_acl.cc). If you want to discuss any changes that you see in the tree feel free to reply to this email or start a separate email thread. Thank you, Pavel

2 1

[Maria-developers] Some patches for the C client library
by Daniel Schepler 02 Dec '14

02 Dec '14

While I was creating Debian packages for the "MariaDB Client Library for C" I came across several minor issues that I corrected. I thought I'd post the patches here in case you might be interested in integrating them; I'd have no problem with contributing them under the MCA if that's necessary (i.e. if they're even big enough to require such a thing). Each one has a fairly complete description of its purpose at the start. I've also noticed that mariadb_config --libs prints out something like ... -l-lz if WITH_EXTERNAL_ZLIB=ON. However, for the Debian packages I instead patched mariadb_config to remove the indirect dependencies altogether, as we generally do, so I didn't write a fix for the -l-lz issue. Also, it seems that if I try to build with WITH_SQLITE=ON then I get a build error, so I haven't enabled that in the Debian packages. -- Daniel Schepler

2 2

[Maria-developers] crash, reporting bugs
by Roberto Spadim 02 Dec '14

02 Dec '14

hi guys, i have a consistent crash (mdev-7182) but this email is not about it... when i got the crash, i got this info: ... To report this bug, see http://kb.askmonty.org/en/reporting-bugs ... should we update this url to the current one? https://mariadb.com/kb/en/mariadb/development/debugging-mariadb/reporting-b… thanks -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

3 3

Re: [Maria-developers] [Commits] 020e569: MDEV-4412 : SLOW QUERY LOG - add affected rows (UPDATE / DELETE) in slow query log
by Sergei Golubchik 30 Nov '14

30 Nov '14

Hi, Nirbhay! On Nov 29, nirbhay(a)mariadb.com wrote: > revision-id: 020e569dc7f638b6add8a043b1c9ff308f688ffa > parent(s): 7b55b67de5fdfe91283357fe6c7ccc3f9e355925 > committer: Nirbhay Choubey > branch nick: b4412 > timestamp: 2014-11-29 19:05:32 -0500 > message: > > MDEV-4412 : SLOW QUERY LOG - add affected rows (UPDATE / DELETE) in slow query log > > Added Rows_affected to slow query log & mysql.slow_log table. > > diff --git a/scripts/mysql_system_tables_fix.sql b/scripts/mysql_system_tables_fix.sql > index b9bb59c..aeaf919 100644 > --- a/scripts/mysql_system_tables_fix.sql > +++ b/scripts/mysql_system_tables_fix.sql > @@ -249,7 +249,8 @@ SET GLOBAL general_log = @old_log_state; > SET @old_log_state = @@global.slow_query_log; > SET GLOBAL slow_query_log = 'OFF'; > ALTER TABLE slow_log > - ADD COLUMN thread_id BIGINT(21) UNSIGNED NOT NULL AFTER sql_text; > + ADD COLUMN thread_id BIGINT(21) UNSIGNED NOT NULL AFTER sql_text, > + ADD COLUMN rows_affected INTEGER NOT NULL AFTER thread_id; > ALTER TABLE slow_log > MODIFY start_time TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, > MODIFY user_host MEDIUMTEXT NOT NULL, I am not sure it'll work. It would work if you need to upgrade a table that has neither thread_id nor rows_affected column to the latest version. So that ALTER would've been an acceptable solution if both thread_id and rows_affected were added in the same MariaDB version. But they were not and there are quite a few MariaDB releases where slow_log table has thread_id and doesn't have rows_affected. I suspect in that case ALTER will fail. But if you'd like you can try a small test case and see whether such an ALTER will add a second column even if the first one already exists. Better add rows_affected in a separate ALTER TABLE statement. Regards, Sergei

1 0

[Maria-developers] Patch: CREATE [[NO] FORCE] VIEW Options
by James Briggs 30 Nov '14

30 Nov '14

Hi List. Below is my patch that implements the CREATE [[NO] FORCE] VIEW options against MariaDB 10.1.0. Notes: - It's a fairly solid patch, but I think the best thing is to commit it to alpha and let it bake for a while. - it passes t/view.test: # ./mysql-test-run.pl view Logging: ./mysql-test-run.pl view vardir: /usr/local/mariadb-10.1.0/mysql-test/var MariaDB Version 10.1.0-MariaDB-debug TEST RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- main.view [ pass ] 1896 -------------------------------------------------------------------------- The servers were restarted 0 times Spent 1.896 of 7 seconds executing testcases Completed: All 1 tests were successful. - I wrote tests/view.pl which does 8,000 tests. It passes. (I will include it in a separate email.) - Please take a look at the style, and the 411 comments. Thanks, James Briggs. -- Available for MySQL/NoSQL DBA/Programming in San Jose area or remote. $ cat create_force_view.patch --- ../mariadb-10.1.0/sql/sql_view.h 2014-06-27 04:50:36.000000000 -0700 +++ sql/sql_view.h 2014-09-02 02:35:42.000000000 -0700 @@ -29,10 +29,10 @@ /* Function declarations */ bool create_view_precheck(THD *thd, TABLE_LIST *tables, TABLE_LIST *view, - enum_view_create_mode mode); + enum_view_create_mode mode, enum_view_create_force force); bool mysql_create_view(THD *thd, TABLE_LIST *view, - enum_view_create_mode mode); + enum_view_create_mode mode, enum_view_create_force force); bool mysql_make_view(THD *thd, File_parser *parser, TABLE_LIST *table, uint flags); --- ../mariadb-10.1.0/sql/sql_lex.h 2014-06-27 04:50:33.000000000 -0700 +++ sql/sql_lex.h 2014-09-02 01:21:10.000000000 -0700 @@ -170,6 +170,12 @@ VIEW_CREATE_OR_REPLACE // check only that there are not such table }; +enum enum_view_create_force +{ + VIEW_CREATE_NO_FORCE, // default - check that there are not such VIEW/table + VIEW_CREATE_FORCE, // check that there are not such VIEW/table, then ignore table object dependencies +}; + enum enum_drop_mode { DROP_DEFAULT, // mode is not specified @@ -2442,6 +2448,7 @@ }; enum enum_var_type option_type; enum enum_view_create_mode create_view_mode; + enum enum_view_create_force create_view_force; enum enum_drop_mode drop_mode; uint profile_query_id; --- ../mariadb-10.1.0/sql/sql_parse.cc 2014-06-27 04:50:34.000000000 -0700 +++ sql/sql_parse.cc 2014-09-02 02:34:31.000000000 -0700 @@ -4943,7 +4943,7 @@ Note: SQLCOM_CREATE_VIEW also handles 'ALTER VIEW' commands as specified through the thd->lex->create_view_mode flag. */ - res= mysql_create_view(thd, first_table, thd->lex->create_view_mode); + res= mysql_create_view(thd, first_table, thd->lex->create_view_mode, thd->lex->create_view_force); break; } case SQLCOM_DROP_VIEW: --- ../mariadb-10.1.0/sql/sql_yacc.yy 2014-06-27 04:50:37.000000000 -0700 +++ sql/sql_yacc.yy 2014-09-05 17:19:29.000000000 -0700 @@ -1851,7 +1851,7 @@ statement sp_suid sp_c_chistics sp_a_chistics sp_chistic sp_c_chistic xa opt_field_or_var_spec fields_or_vars opt_load_data_set_spec - view_algorithm view_or_trigger_or_sp_or_event + view_algorithm view_or_trigger_or_sp_or_event view_force_option definer_tail no_definer_tail view_suid view_tail view_list_opt view_list view_select view_check_option trigger_tail sp_tail sf_tail udf_tail event_tail @@ -2446,6 +2446,7 @@ VIEW_CREATE_OR_REPLACE); Lex->create_view_algorithm= DTYPE_ALGORITHM_UNDEFINED; Lex->create_view_suid= TRUE; + Lex->create_view_force= VIEW_CREATE_NO_FORCE; /* initialize just in case */ } view_or_trigger_or_sp_or_event { @@ -15887,6 +15888,15 @@ | event_tail ; +view_force_option: + /* empty */ /* 411 - is there a cleaner way of initializing here? */ + { Lex->create_view_force = VIEW_CREATE_NO_FORCE; } + | NO_SYM FORCE_SYM + { Lex->create_view_force = VIEW_CREATE_NO_FORCE; } + | FORCE_SYM + { Lex->create_view_force = VIEW_CREATE_FORCE; } + ; + /************************************************************************** DEFINER clause support. @@ -15944,7 +15954,7 @@ ; view_tail: - view_suid VIEW_SYM table_ident + view_suid view_force_option VIEW_SYM table_ident { LEX *lex= thd->lex; lex->sql_command= SQLCOM_CREATE_VIEW; --- ../mariadb-10.1.0/sql/sql_view.cc 2014-06-27 04:50:36.000000000 -0700 +++ sql/sql_view.cc 2014-09-05 19:33:58.000000000 -0700 @@ -248,7 +248,7 @@ */ bool create_view_precheck(THD *thd, TABLE_LIST *tables, TABLE_LIST *view, - enum_view_create_mode mode) + enum_view_create_mode mode, enum_view_create_force force) { LEX *lex= thd->lex; /* first table in list is target VIEW name => cut off it */ @@ -259,7 +259,7 @@ DBUG_ENTER("create_view_precheck"); /* - Privilege check for view creation: + Privilege check for view creation with default (NO FORCE): - user has CREATE VIEW privilege on view table - user has DROP privilege in case of ALTER VIEW or CREATE OR REPLACE VIEW @@ -272,6 +272,7 @@ checked that we have not more privileges on correspondent column of view table (i.e. user will not get some privileges by view creation) */ + if ((check_access(thd, CREATE_VIEW_ACL, view->db, &view->grant.privilege, &view->grant.m_internal, @@ -285,6 +286,11 @@ check_grant(thd, DROP_ACL, view, FALSE, 1, FALSE)))) goto err; + if (force) { + res = false; + DBUG_RETURN(res || thd->is_error()); + } + for (sl= select_lex; sl; sl= sl->next_select()) { for (tbl= sl->get_table_list(); tbl; tbl= tbl->next_local) @@ -369,7 +375,7 @@ #else bool create_view_precheck(THD *thd, TABLE_LIST *tables, TABLE_LIST *view, - enum_view_create_mode mode) + enum_view_create_mode mode, enum_view_create_force force) { return FALSE; } @@ -391,7 +397,7 @@ */ bool mysql_create_view(THD *thd, TABLE_LIST *views, - enum_view_create_mode mode) + enum_view_create_mode mode, enum_view_create_force force) { LEX *lex= thd->lex; bool link_to_local; @@ -425,14 +431,13 @@ goto err; } - if ((res= create_view_precheck(thd, tables, view, mode))) + if (res= create_view_precheck(thd, tables, view, mode, force)) goto err; lex->link_first_table_back(view, link_to_local); view->open_type= OT_BASE_ONLY; - if (open_temporary_tables(thd, lex->query_tables) || - open_and_lock_tables(thd, lex->query_tables, TRUE, 0)) + if (open_temporary_tables(thd, lex->query_tables) || (!force && open_and_lock_tables(thd, lex->query_tables, TRUE, 0))) { view= lex->unlink_first_table(&link_to_local); res= TRUE; @@ -513,6 +518,7 @@ } } +if (!force) { /* prepare select to resolve all fields */ lex->context_analysis_only|= CONTEXT_ANALYSIS_ONLY_VIEW; if (unit->prepare(thd, 0, 0)) @@ -612,6 +618,7 @@ } } #endif +} res= mysql_register_view(thd, view, mode); @@ -621,7 +628,7 @@ meta-data changes after ALTER VIEW. */ - if (!res) + // if (!res) + if (!res && !force) /* 411 - solves segfault problems with CREATE FORCE VIEW option sometimes */ tdc_remove_table(thd, TDC_RT_REMOVE_ALL, view->db, view->table_name, false); if (mysql_bin_log.is_open()) @@ -908,6 +915,8 @@ fn_format(path_buff, file.str, dir.str, "", MY_UNPACK_FILENAME); path.length= strlen(path_buff); if (ha_table_exists(thd, view->db, view->table_name, NULL)) { if (mode == VIEW_CREATE_NEW) --- ../mariadb-10.1.0/mysql-test/t/view.test 2014-06-27 04:50:30.000000000 -0700 +++ mysql-test/t/view.test 2014-09-06 00:23:32.000000000 -0700 @@ -5263,4 +5263,17 @@ --echo # ----------------------------------------------------------------- --echo # -- End of 10.0 tests. --echo # ----------------------------------------------------------------- + +create no force view v1 as select 1; +drop view if exists v1; + +create force view v1 as select 1; +drop view if exists v1; + +create force view v1 as select * from missing_base_table; +drop view if exists v1; + +--echo # ----------------------------------------------------------------- +--echo # -- End of 10.1 tests. +--echo # ----------------------------------------------------------------- SET optimizer_switch=@save_optimizer_switch;

2 2

Re: [Maria-developers] MDEV-4412 : SLOW QUERY LOG - add affected rows (UPDATE / DELETE) in slow query log
by Sergei Golubchik 30 Nov '14

30 Nov '14

Hi, Nirbhay! On Nov 28, nirbhay(a)mariadb.com wrote: > revision-id: 7a5559421f4f3dda44d04bf946e79661d09b4dc2 > parent(s): 7b55b67de5fdfe91283357fe6c7ccc3f9e355925 > committer: Nirbhay Choubey > branch nick: b4412 > timestamp: 2014-11-28 20:53:02 -0500 > message: > > MDEV-4412 : SLOW QUERY LOG - add affected rows (UPDATE / DELETE) in slow query log > > Added Rows_affected to slow query log & mysql.slow_log table. Just a couple of comments: > diff --git a/scripts/mysql_system_tables_fix.sql b/scripts/mysql_system_tables_fix.sql > index b9bb59c..4973dc9 100644 > --- a/scripts/mysql_system_tables_fix.sql > +++ b/scripts/mysql_system_tables_fix.sql > @@ -262,7 +262,8 @@ ALTER TABLE slow_log > MODIFY insert_id INTEGER NOT NULL, > MODIFY server_id INTEGER UNSIGNED NOT NULL, > MODIFY sql_text MEDIUMTEXT NOT NULL, > - MODIFY thread_id BIGINT(21) UNSIGNED NOT NULL; > + MODIFY thread_id BIGINT(21) UNSIGNED NOT NULL, > + MODIFY rows_affected INTEGER NOT NULL; Eh? That makes no sense. This file is for mysql_upgrade. Your ALTER TABLE would help if there was some MariaDB version where rows_affected column would've been nullable and you wanted to change it to NOT NULL. But there was no such MariaDB version. There was MariaDB version without rows_affected column at all. So, instead of this MODIFY rows_affected, you need to put another ALTER TABLE statement that adds rows_affected INTEGER NOT NULL column. > SET GLOBAL slow_query_log = @old_log_state; > > ALTER TABLE plugin > diff --git a/sql/log.cc b/sql/log.cc > index ff04105..24e5283 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2918,12 +2924,16 @@ bool MYSQL_QUERY_LOG::write(THD *thd, time_t current_time, > sprintf(lock_time_buff, "%.6f", ulonglong2double(lock_utime)/1000000.0); > if (my_b_printf(&log_file, > "# Thread_id: %lu Schema: %s QC_hit: %s\n" \ > - "# Query_time: %s Lock_time: %s Rows_sent: %lu Rows_examined: %lu\n", > + "# Query_time: %s Lock_time: %s Rows_sent: %lu Rows_examined: %lu\n" \ > + "# Rows_affected: %lu\n", I'd rather have Rows_affected on the same line with Rows_sent and Rows_examined > (ulong) thd->thread_id, (thd->db ? thd->db : ""), > ((thd->query_plan_flags & QPLAN_QC) ? "Yes" : "No"), > query_time_buff, lock_time_buff, > (ulong) thd->get_sent_row_count(), > - (ulong) thd->get_examined_row_count()) == (size_t) -1) > + (ulong) thd->get_examined_row_count(), > + thd->get_stmt_da()->is_ok() ? > + (ulong) thd->get_stmt_da()->affected_rows() : > + 0) == (size_t) -1) > tmp_errno= errno; > if ((thd->variables.log_slow_verbosity & LOG_SLOW_VERBOSITY_QUERY_PLAN) && > (thd->query_plan_flags & Regards, Sergei

2 1

Re: [Maria-developers] [Commits] Rev 4484: MDEV-4255 MIXED replication: Unique key updates are not marked unsafe, replication fails in lp:~maria-captains/maria/10.0
by Kristian Nielsen 29 Nov '14

29 Nov '14

Sergei Golubchik <serg(a)mariadb.org> writes: > Kristian Nielsen, could you please review this patch? I guess my main question here is, did you think about why you want to) fix this 1) in a post-GA release, 2) at all? One problem with adding new statements as unsafe for statement-based binlogging is that it introduces new warnings in existing applications. This is especially annoying on a slave with binlogging enabled, as every unsafe statement now causes a warning in the slave error log. Thus, it can happen that a security update for a user leaves them with an error log that constantly fills up on a busy slave. It is my understanding that statement-based binlogging (as opposed to mixed or row) is still quite popular, and there have been complaints about these warnings before. Another point is that this patch seems to fix only a corner case of a bigger problem? This particular patch is about non-transactional updates. But updating a unique key can break replication just as well on transactional tables: CREATE TABLE t1 (a INT PRIMARY KEY); INSERT INTO t1 VALUES (2), (3), (4); UPDATE t1 SET a=a-1; Since we do not have deferred constraints, this UPDATE will fail or succeed depending on the order in which rows are updated. So a statement that succeeds andgets binlogged on the master can break on the slave if the rows are updated in a different order. So any reason that you fix one part of UPDATE of unique key but not the other? (I suppose the same problem can also happen with foreign key constraints, though it might take some creativity to come up with an example where an UPDATE will conflict or not with foreign keys depending on row update order). The bug has fix version 5.5, your patch seems to be against 10.0, where do you intend to push it? I would probably have chosen to fix in 10.1, if at all, but I'm not necessarily against 5.5 or 10.0, I'm mostly just asking what the reason is for whichever version was chosen? > I don't particularly like that it explicitly invokes > thd->decide_logging_format() . But there's no way around it, first time > logging format is decided in lock_tables(), basically when tables are just > opened. At that point we don't know what fields will be updated yet. I am not really familiar with how binlogging is decided on the master, have not worked with that code so far. So unfortunately I am not able to say much about whether this is a problem or not. At least the call is still before the actual update of rows starts, so I suppose it should be ok... > At lp:~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4484 > revision-id: sergii(a)pisem.net-20141116234900-747qdhm9ti9xh070 > parent: sergii(a)pisem.net-20141113161325-mdz5ox7e86iyazss > fixes bug: https://mariadb.atlassian.net/browse/MDEV-4255 > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 10.0 > timestamp: Mon 2014-11-17 00:49:00 +0100 > message: > MDEV-4255 MIXED replication: Unique key updates are not marked unsafe, replication fails I think this needs a better description, especially since this can cause new warnings in slave error logs. Here is a suggestion, but feel free to use something else if you prefer: "MDEV-4255 MIXED replication: Unique key updates are not marked unsafe, replication fails Statement-based replication of UPDATE statements that modify (parts of) unique keys is not safe. If the order of row updates differ between master and slave, the slave can end up modifying a different set of rows before a unique key violation error than what was updated on the master. If the table is non-transactional, then the statement will be partially executed on the master and on the slave, but with different result, causing replication to diverge. This patch marks as unsafe for statement-based replication any UPDATE that modifies a unique key on a non-transactinal table. In mixed-mode replication, this will cause the query to be logged in row mode. In statement-mode replication, this will produce a warning about binlogging of unsafe statement." > === modified file 'sql/share/errmsg-utf8.txt' > --- a/sql/share/errmsg-utf8.txt 2014-11-13 16:13:25 +0000 > +++ b/sql/share/errmsg-utf8.txt 2014-11-16 23:49:00 +0000 > @@ -7111,3 +7111,5 @@ ER_SLAVE_SKIP_NOT_IN_GTID > eng "When using GTID, @@sql_slave_skip_counter can not be used. Instead, setting @@gtid_slave_pos explicitly can be used to skip to after a given GTID position." > ER_TABLE_DEFINITION_TOO_BIG > eng "The definition for table %`s is too big" > +ER_BINLOG_UNSAFE_UPDATE_UNIQUE_NONTRANS > + eng "The statement is unsafe because it updates a UNIQUE key colums in a non-transactional table. This is unsafe because in the case of a conflict a non-predictable part of the table will stay updated." (Adding a new error code in a GA release requires some care, especially if fix-version:5.5 is intended, but I'm sure you're better aware of this than most). > === modified file 'sql/sql_lex.h' > --- a/sql/sql_lex.h 2014-08-07 16:06:56 +0000 > +++ b/sql/sql_lex.h 2014-11-16 23:49:00 +0000 > @@ -1319,6 +1319,12 @@ class Query_tables_list > */ > BINLOG_STMT_UNSAFE_AUTOINC_NOT_FIRST, > > + /** > + INSERT into auto-inc field which is not the first part of composed > + primary key. > + */ > + BINLOG_STMT_UPDATE_UNIQUE_NONTRANS, > + Copy-paste error. Here, you copied the comment of the earlier entry, you surely intended to re-write the comment to describe the newly added entry. Hope this helps, - Kristian.

3 7

Re: [Maria-developers] [Commits] 953dd8d: MDEV-6997 MDEV-6996 MDEV-7002 MDEV-7006 MDEV-7009 MDEV-7012 MDEV-7015 MDEV-6977
by Sergei Golubchik 29 Nov '14

29 Nov '14

Hi, sanja! On Nov 05, sanja(a)mariadb.com wrote: > revision-id: 953dd8d87fea224e26d11452eb458c79ba098f4a > parent(s): d6981391eb8f904135dfdedb5b90a7592b35e648 > committer: Oleksandr Byelkin > branch nick: work-maria-10.1-sysvar4 > timestamp: 2014-11-05 11:05:01 +0100 > message: > > MDEV-6997 MDEV-6996 MDEV-7002 MDEV-7006 MDEV-7009 MDEV-7012 MDEV-7015 MDEV-6977 I'd prefer a slightly more verbose commit comment. And don't forget - first line is the subject. For example: Use NO_SET_STMT flag for more variables Fixes MDEV-6997 MDEV-6996 MDEV-7002 MDEV-7006 MDEV-7009 MDEV-7012 MDEV-7015 MDEV-6977 I won't comment on individual variables here, as you're simply disabling them all. See my comments in Jira. Regards, Sergei

1 0

[Maria-developers] MDEV-7004 review
by Sergei Golubchik 28 Nov '14

28 Nov '14

Hi, Sergey I didn't review MDEV-7200 (InnoDB) and patches that are already in 10.1. Almost everything else was fine, see few comments below. > ------------------------------------------------------------ > revno: 4513 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-power > timestamp: Fri 2014-11-07 22:03:18 +0400 > message: > Multi-rwlock lock for table definition cache hash. > added: > modified: > diff: > === modified file 'sql/table_cache.cc' > --- sql/table_cache.cc 2014-10-21 12:20:39 +0000 > +++ sql/table_cache.cc 2014-11-07 18:03:18 +0000 > @@ -53,6 +53,77 @@ > #include "table.h" > #include "sql_base.h" > > + > +/** > + Table definition hash lock. > + > + Lock consists of multiple rw-locks (8 by default). Lock is write locked > + when all instances are write locked. Lock is read locked when any instance > + is read locked. > + > + This class was implemented to workaround scalability bottleneck of table > + definition hash rw-lock. > + > + To be replaced with lock-free hash when we understand how to solve the > + following problems: > + - lock-free hash iterator (can be workarounded by introducing global > + all_tables list); I don't particularly like that solution. Could you try to switch to lock-free hash now? I'll try to do the iterator asap > + - lf_hash_search() may fail because of OOM. It is rather complex to handle > + it properly. > +*/ > + > +class TDC_lock > +{ > + uint m_instances; > + bool m_write_locked; > + char pad[128]; > + struct st_locks > + { > + mysql_rwlock_t lock; > + char pad[128]; In these cases I usually do: pad[128 - sizeof(mysql_rwlock_t)]; > + } *locks; > +public: > + int init(PSI_rwlock_key key, uint instances) > + { > + m_instances= instances; > + m_write_locked= false; > + locks= (st_locks *) my_malloc(sizeof(struct st_locks) * m_instances, MYF(MY_WME)); > + if (!locks) > + return 1; > + for (uint i= 0; i < m_instances; i++) > + mysql_rwlock_init(key, &locks[i].lock); > + return 0; > + } > + void destroy(void) > + { > + for (uint i= 0; i < m_instances; i++) > + mysql_rwlock_destroy(&locks[i].lock); > + my_free(locks); > + } > + void wrlock(void) > + { > + for (uint i= 0; i < m_instances; i++) > + mysql_rwlock_wrlock(&locks[i].lock); > + m_write_locked= true; > + } > + void rdlock(THD *thd= 0) > + { > + mysql_rwlock_rdlock(&locks[thd ? thd->thread_id % m_instances : 0].lock); > + } > + void unlock(THD *thd= 0) > + { > + if (m_write_locked) > + { > + m_write_locked= false; > + for (uint i= 0; i < m_instances; i++) > + mysql_rwlock_unlock(&locks[i].lock); > + } > + else > + mysql_rwlock_unlock(&locks[thd ? thd->thread_id % m_instances : 0].lock); > + } > +}; > + > + > /** Configuration. */ > ulong tdc_size; /**< Table definition cache threshold for LRU eviction. */ > ulong tc_size; /**< Table cache threshold for LRU eviction. */ > ------------------------------------------------------------ > revno: 4512 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-power > timestamp: Fri 2014-10-31 15:44:36 +0400 > message: > Preallocate locks on THD mem_root to avoid expensive malloc. > modified: > diff: > === modified file 'sql/lock.cc' > --- sql/lock.cc 2014-09-30 17:31:14 +0000 > +++ sql/lock.cc 2014-10-31 11:44:36 +0000 > @@ -700,7 +699,7 @@ MYSQL_LOCK *get_lock_data(THD *thd, TABL > TABLE **to, **table_buf; > DBUG_ENTER("get_lock_data"); > > - DBUG_ASSERT((flags == GET_LOCK_UNLOCK) || (flags == GET_LOCK_STORE_LOCKS)); > + DBUG_ASSERT((flags & GET_LOCK_UNLOCK) || (flags & GET_LOCK_STORE_LOCKS)); Not the same, old assert also verified that these flags aren't set both. New assert doesn't. But I don't think it matters much. > DBUG_PRINT("info", ("count %d", count)); > > for (i=lock_count=table_count=0 ; i < count ; i++) > ------------------------------------------------------------ > revno: 4508 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Tue 2014-10-21 16:20:39 +0400 > message: > modified: > diff: > === modified file 'sql/handler.cc' > --- sql/handler.cc 2014-11-13 10:01:31 +0000 > +++ sql/handler.cc 2014-10-21 12:20:39 +0000 > @@ -410,13 +410,16 @@ static int hton_ext_based_table_discover > static void update_discovery_counters(handlerton *hton, int val) > { > if (hton->discover_table_existence == full_discover_for_existence) > - my_atomic_add32(&need_full_discover_for_existence, val); > + my_atomic_add32_explicit(&need_full_discover_for_existence, val, > + MY_MEMORY_ORDER_RELAXED); > > if (hton->discover_table_names) > - my_atomic_add32(&engines_with_discover_table_names, val); > + my_atomic_add32_explicit(&engines_with_discover_table_names, val, > + MY_MEMORY_ORDER_RELAXED); > > if (hton->discover_table) > - my_atomic_add32(&engines_with_discover, val); > + my_atomic_add32_explicit(&engines_with_discover, val, > + MY_MEMORY_ORDER_RELAXED); I wouldn't do that. Technically there's a dependency between these counters and loaded engines, so the order, kind of, matters. I don't think there is any practical risk of bugs here, but, on the other hand, these counters are only updated when an engine is loaded or unloaded, so there's no need to bother with the relaxed memory ordering here. > } > > int ha_finalize_handlerton(st_plugin_int *plugin) > > === modified file 'sql/mysqld.cc' > --- sql/mysqld.cc 2014-11-18 21:25:47 +0000 > +++ sql/mysqld.cc 2014-10-21 12:20:39 +0000 > @@ -3884,7 +3884,7 @@ static void my_malloc_size_cb_func(long > } > // workaround for gcc 4.2.4-1ubuntu4 -fPIE (from DEB_BUILD_HARDENING=1) > int64 volatile * volatile ptr=&global_status_var.memory_used; > - my_atomic_add64(ptr, size); > + my_atomic_add64_explicit(ptr, size, MY_MEMORY_ORDER_RELAXED); good! > } > } > > > === modified file 'sql/mysqld.h' > --- sql/mysqld.h 2014-11-13 08:56:28 +0000 > +++ sql/mysqld.h 2014-10-21 12:20:39 +0000 > @@ -629,7 +629,7 @@ inline __attribute__((warn_unused_result > { > query_id_t id; > my_atomic_rwlock_wrlock(&global_query_id_lock); > - id= my_atomic_add64(&global_query_id, 1); > + id= my_atomic_add64_explicit(&global_query_id, 1, MY_MEMORY_ORDER_RELAXED); I believe this is fine > my_atomic_rwlock_wrunlock(&global_query_id_lock); > return (id); > } > === modified file 'sql/table_cache.cc' > --- sql/table_cache.cc 2014-06-10 18:20:33 +0000 > +++ sql/table_cache.cc 2014-10-21 12:20:39 +0000 > @@ -139,7 +139,7 @@ uint tc_records(void) > { > uint count; > my_atomic_rwlock_rdlock(&LOCK_tdc_atomics); > - count= my_atomic_load32(&tc_count); > + count= my_atomic_load32_explicit(&tc_count, MY_MEMORY_ORDER_RELAXED); I'm not sure about this file. You should know better whether it's safe to use relaxed memory ordering here. > my_atomic_rwlock_rdunlock(&LOCK_tdc_atomics); > return count; > } Regards, Sergei

2 5

[Maria-developers] add a rotate event at begin of relay log make greate help
by nanyi607rao 28 Nov '14

28 Nov '14

Hi kristian, currently, a relaylog is immediately purged after it be completely replayed (relay_log_purge=1). and if IO thread crashes, we should delete all relay logs and fetch master binlog again from exec_master_log_pos/file (relay_log_purge=ON). I think that make a greap waste of disk IO especially to cloud disk. The critical problem is that there is no method to know the original binlog filename on master of a event in relaylog. Actually we can know a relaylog event's original binlog offset on master, but we can't know the original binlog filename. so why not forcely add a rotate event or any other new type event which contained the original master binlog filename at begin of a relay log. as I know two different master binlogs' events wouldn't contained in one relay log, so the original binlog filename of a relaylog's events is only one. If we want to know a relaylog event's original binlog filename/position, we can get original filename from the begin of relaylog, and get original binlog offset from event's "end log pos". oppositely, if we know a event's binlog filename/end positon on master, wo also can quickly find it in slave's relay log. That can make many benefits, firstly no need to delete all relay logs and fetch master binlogs again when IO thread crashed, because we can get exact last read_master_log_filename/position from last relay log. secondly, there is no need to use GTID in 1 vs n replication failover Scenario (Gtid must set log_slave_updates=ON in mysql 5.6, which increase disk IO load), if master crashes, other slaves can get lost events for the newest slave's relay log, as long as relay log don't be purged, then promotes the newest slave to master. Thanks. 2014-11-26 nanyi607rao

3 3

[Maria-developers] MDEV-6431 review
by Sergey Vojtovich 27 Nov '14

27 Nov '14

Hi Sergei, overall it looks good. I want to double check the clean-up stuff, since I didn't understand everything at the first glance. Some comments inline. > diff --git a/include/mysql/plugin_password_validation.h b/include/mysql/plugin_password_validation.h > new file mode 100644 > index 0000000..0bbb8f4 > --- /dev/null > +++ b/include/mysql/plugin_password_validation.h > @@ -0,0 +1,45 @@ > +/** > + Password validation plugin descriptor > +*/ > +struct st_mysql_password_validation > +{ > + int interface_version; /**< version plugin uses */ > + /** > + Function provided by the plugin which should perform password validation > + and return 0 if the password has passed the validation. > + */ > + int (*validate_password)(MYSQL_LEX_STRING *username, > + MYSQL_LEX_STRING *password); > +}; > +#endif > + This differs from MySQL API, so we can't reuse their plugins. Also they have this cool get_password_strength method, but that's something we can expose via UDF. OTOH we can compare password against username, and that's quite common password misuse. And what I always hated in this validation stuff is that you'll never come up with a password that makes it happy. Of course unless you know requirements. In our case requirements are exposed via system variables. This is more or less acceptable. But I believe it would be nice if this stuff would be able to generate strong passwords. Again, that's something that can be exposed as UDF. > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 529a795..cdf8c7b 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -1051,7 +1055,8 @@ static bool plugin_add(MEM_ROOT *tmp_root, > continue; // invalid plugin type > > if (plugin->type == MYSQL_UDF_PLUGIN || > - (plugin->type == 8 && tmp.plugin_dl->mariaversion == 0)) > + (plugin->type == MariaDB_PASSWORD_VALIDATION_INTERFACE_VERSION && > + tmp.plugin_dl->mariaversion == 0)) > continue; // unsupported plugin type > > if (name->str && my_strnncoll(system_charset_info, You compare type against version here. Why? Regards, Sergey

2 1

Re: [Maria-developers] [Commits] Rev 4369: MDEV-7148 - Recurring: InnoDB: Failing assertion: !lock->recursive in lp:maria/5.5
by Kristian Nielsen 27 Nov '14

27 Nov '14

Sergey Vojtovich <svoj(a)mariadb.org> writes: > revno: 4369 > revision-id: svoj(a)mariadb.org-20141120094823-fozdsrm1kzv46kxi > parent: jplindst(a)mariadb.org-20141119182734-cwbaed0ka90ocj5e > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5 > timestamp: Thu 2014-11-20 13:48:23 +0400 > message: > MDEV-7148 - Recurring: InnoDB: Failing assertion: !lock->recursive > > On PPC64 high-loaded server may crash due to assertion failure in InnoDB > rwlocks code. > > This happened because load order between "recursive" and "writer_thread" > wasn't properly enforced. Looks ok. I failed to understand why the patch would fix the assertion mentioned in the bug report. But the patch itself looks correct. - Kristian.

3 3

Re: [Maria-developers] [Commits] Rev 3902: MDEV-7123: MariaDB 10.0.14 Galera node shutdown with signal 11 in lp:~maria-captains/maria/maria-10.0-galera
by Kristian Nielsen 26 Nov '14

26 Nov '14

Nirbhay Choubey <nirbhay(a)mariadb.com> writes: > revno: 3902 > revision-id: nirbhay(a)mariadb.com-20141123163533-9ov1yirj26cur301 > parent: nirbhay(a)mariadb.com-20141119183337-n5spskf54beqbhif > committer: Nirbhay Choubey <nirbhay(a)mariadb.com> > branch nick: maria-10.0-galera > timestamp: Sun 2014-11-23 11:35:33 -0500 > message: > MDEV-7123: MariaDB 10.0.14 Galera node shutdown with signal 11 > > wsrep-patch uses same connection name for constructing Master_info > objects. As a result all existing wsrep Master_info objects refer > to same rpl_filter object. This could lead to race when multiple > threads try to destruct/delete Master_info object, as they would all > try to delete the same relay_log object. > Fixed by adding a mutex to protect the critical section. The patch is ok. But please consider the following points, and act on them or not as you see fit: 1. You are introducing a mutex lock/unlock in a general data structure used in a couple of places (looks like keycaches and rpl_filters?). Did you consider the performance impact of this? When will the additional locking/unlocking take place? If it only happens in less often used operations (like re-configuring replication filters or key caches?), then extra locking is not an issue. If it happens often, like once per query or something like that, it would probably be best to find a solution with less performance impact. (mutex lock and unlock are quite expensive even if not contended). 2. Consider using explicit mysql_mutex_lock() and mysql_mutex_unlock() calls rather than introducing extra lock()/unlock() methods. The extra methods are needless abtractions that make it harder to understand exactly what is happening wrt. locking. 3. Did you consider if a test case could be made for the originally reported bug? We used to have a rule that any bug fix must include a test case that reproduces the bug fixed (ie. the test case must fail before the patch). Not a hard rule (a few bugs can be extremely hard to write mtr testcases for), but a good rule nevertheless. But maybe this rule has been dropped, now that the focus is on quantity of Jira tickes closed rather than quality. - Kristian.

1 0

[Maria-developers] Windows: git config autocrlf
by Alexander Barkov 26 Nov '14

26 Nov '14

Hi, What would be the correct value for the autocrlf setting in git on Windows? Git suggests autocrlf=true by default at installation time. But then mysql-test do not work with this value. What is a better choice between the other two: "false" or "input"? Thanks.

1 0

[Maria-developers] doubt about read uncommited
by Roberto Spadim 25 Nov '14

25 Nov '14

hi guys, i use read uncommited sometimes with innodb, that's nice and work but now, i'm using a myisam table, and a aria table.... does read uncommited work with this kind of engine? i tested and table stay 'waiting table lock' while a looooong update occurs i don't know what's the internal diference but is possible to "easily" implement read uncommited to myisam/aria? -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

4 10

[Maria-developers] Please review: MDEV-7199 Introduce the concept of data type handlers
by Alexander Barkov 25 Nov '14

25 Nov '14

Hi Sergei, while working on addressing your review suggestions for "MDEV-4912 Add a plugin to field types (column types)", I decided to create a separate tasks for a piece of additional changes: https://mariadb.atlassian.net/browse/MDEV-7199 MDEV-7199 Introduce the concept of data type handlers Please review. Thanks.

1 0

Re: [Maria-developers] [Commits] Rev 4507: MDEV-7157 plugins.server_audit fails sporadically in buildbot. in file:///home/hf/wmar/mdev-7157/
by Kristian Nielsen 24 Nov '14

24 Nov '14

<holyfoot(a)askmonty.org> writes: > ------------------------------------------------------------ > revno: 4507 > revision-id: holyfoot(a)askmonty.org-20141123225345-p2xd7yrklzee10u1 > parent: sergii(a)pisem.net-20141121192039-d0lv6cj96kg5pw02 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-7157 > timestamp: Mon 2014-11-24 02:53:45 +0400 > message: > MDEV-7157 plugins.server_audit fails sporadically in buildbot. > Records can get to the different place in the log when multiple thread > are logged. So the delay added to let the record be saved on the same > place. > === modified file 'mysql-test/suite/plugins/t/server_audit.test' > --- a/mysql-test/suite/plugins/t/server_audit.test 2014-03-26 19:58:27 +0000 > +++ b/mysql-test/suite/plugins/t/server_audit.test 2014-11-23 22:53:45 +0000 > @@ -43,6 +43,7 @@ show variables like 'server_audit%'; > set global server_audit_mode=1; > set global server_audit_events=''; > create database sa_db; > +--sleep 2 The correct way to do this is to use DEBUG_SYNC... - Kristian.

1 0

Re: [Maria-developers] [Commits] Rev 3879: MDEV-6593 : domain_id based replication filters in lp:~maria-captains/maria/maria-10.0-galera
by Kristian Nielsen 24 Nov '14

24 Nov '14

Nirbhay Choubey <nirbhay(a)skysql.com> writes: > ------------------------------------------------------------ > revno: 3879 > revision-id: nirbhay(a)skysql.com-20140912181622-s33e7r0vtgpju95q > parent: nirbhay(a)skysql.com-20140814224304-rtkeal7p9lk06li1 > committer: Nirbhay Choubey <nirbhay(a)skysql.com> > branch nick: maria-10.0-galera_6593 > timestamp: Fri 2014-09-12 14:16:22 -0400 > message: > MDEV-6593 : domain_id based replication filters > > Added support for IGNORE_DOMAIN_IDS and DO_DOMAIN_IDS. Here is my review. Some general comments, followed by some detailed comments in-line in the patch: 1. Did you prepare any documentation? You could either add it to the knowledgebase directly, or write up a text file and ask Daniel to add it in relevant places. 2. Did you run the full test suite? I would have expected at least some tests in the multi_source suite to need modification. The chance touches enough things that you should test it in a feature tree in Buildbot to ensure that it is green before pushing it to a main tree. 3. I think some extra test cases are needed: - You should test (and document) what happens when the same domain_id is in both the DO and the IGNORE list. Is it actually ever sensible to have both DO_DOMAIN_IDS and IGNORE_DOMAIN_IDS ? If not, I think it should be an error to try. - You should also add a test using non-GTID mode. - You should also add a test where parallel replication is used (--slave-parallel-threads > 0). - Add a test for the following scenario when using GTID mode: Some domain id is ignored, some events are received from the master and ignored. Then the slave is stopped, and the ignore rule is removed. What happens when the slave is restarted and reconnects? I think the previously ignored domain id events will now be re-fetched and replicated. I guess that's ok, but it should be documented. Also test what happens when binlog files on the master have been rotated and purged, so that the events are no longer available on the master (an error should occur). - Add a similar test for when not using GTID mode, the last event replicated by the slave is ignored, the slave is stopped, the ignore rule removed and the slave restarted; will the last event now be re-fetched or not (I think not). - Add a test that checks that seconds_behind_master is properly updated to zero even when the last event is ignored due to DO_DOMAIN_IDS/IGNORE_DOMAIN_IDS. It can perhaps be a bit tricky to test, but I think you can use this existing test as a basis: mysql-test/suite/rpl/t/rpl_parallel2.test - It would also be good to check that the code works when used on an old master (which does not send GTID events). You can do this by creating a binlog file with a 5.5 server and storing it in mysql-test/std_data/, and then installing it in a master; there should be several other test cases that already do this. 4. Also see detailed comments for some possible problems with the implementation. The most serious is probably to ensure that events are not skipped after the end of the group, we need a couple of tests for this, see below. Also, there is a small memory leak, see below. > +DO_DOMAIN_IDS (before) : 1 > +IGNORE_DOMAIN_IDS (before) : 2 > +CHANGE MASTER TO DO_DOMAIN_IDS=(4,4,5,1,7,7,7,1,1,2,6,8,1,4,5,5,9,3), MASTER_USE_GTID=current_pos; So if a domain_id is specified multiple times, the redundant ones will be ignored. It would seem preferable to give an error... but as far as I can see the behaviour is the same for IGNORE_SERVER_IDS, so better to be consistent as you did in the code. Just be sure to add this to the documentation. > === added file 'mysql-test/suite/rpl/t/rpl_domain_id_filter.test' > --- a/mysql-test/suite/rpl/t/rpl_domain_id_filter.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/rpl/t/rpl_domain_id_filter.test 2014-09-12 18:16:22 +0000 > +SET @gtid_domain_id_saved = @@session.gtid_domain_id; This is not needed, nor do you use it anywhere, you can remove it. > +##### Case 3: When both DO_DOMAIN_IDS and IGNORE_DOMAIN_IDS are non-empty > + > +source include/stop_slave.inc; > +let $do_domain_ids_before= query_get_value(SHOW SLAVE STATUS, Replicate_Do_Domain_Ids, 1); > +let $ignore_domain_ids_before= query_get_value(SHOW SLAVE STATUS, Replicate_Ignore_Domain_Ids, 1); > +--echo DO_DOMAIN_IDS (before) : $do_domain_ids_before > +--echo IGNORE_DOMAIN_IDS (before) : $ignore_domain_ids_before > + > +# Replicate events belonging to "domain_id 1". (DO_DOMAIN_IDS should win!) > +CHANGE MASTER TO DO_DOMAIN_IDS=(1), IGNORE_DOMAIN_IDS=(2), MASTER_USE_GTID=current_pos; Did you intend here to test overlapping ids in DO_DOMAIN_IDS and IGNORE_DOMAIN_IDS (due to the comment "DO_DOMAIN_IDS should win") ? But if I understand correctly, IGNORE_DOMAIN_IDS will be ignored if DO_DOMAIN_IDS is non-empty. So I think it should be an error to try to set both - do you agree? > +--source include/rpl_end.inc Please add additional testcases, as suggested at the start of review. > === modified file 'sql/rpl_mi.cc' > --- a/sql/rpl_mi.cc 2014-02-19 10:05:15 +0000 > +++ b/sql/rpl_mi.cc 2014-09-12 18:16:22 +0000 > @@ -513,6 +513,25 @@ > else > mi->using_gtid= Master_info::USE_GTID_NO; > } > + > + // DO_DOMAIN_IDS > + if (0 == strncmp(buf, STRING_WITH_LEN("do_domain_ids=")) && > + mi->domain_id_filter.init_ids(buf + sizeof("do_domain_ids"), > + Domain_id_filter::DO_DOMAIN_IDS)) > + { > + sql_print_error("Failed to initialize master info do_domain_ids"); > + goto errwithmsg; > + } > + > + // IGNORE_DOMAIN_IDS > + if (0 == strncmp(buf, STRING_WITH_LEN("ignore_domain_ids=")) && > + mi->domain_id_filter.init_ids(buf + sizeof("ignore_domain_ids"), > + Domain_id_filter::IGNORE_DOMAIN_IDS)) > + { > + sql_print_error("Failed to initialize master info " > + "ignore_domain_ids"); > + goto errwithmsg; > + } So here you use the same style as adjacent code (that I wrote). However, I have since learned that the style (0 == strncmp(...)) is not used in the server. So please change this, as well as my existing code, to use either strncmp(...) == 0 or !strncmp(), as you prefer. Another thing from my old code is the sizeof("do_domain_ids"), which highly confused me for a bit before I realised that while the '=' is not included in the sizeof(), the terminating '\0' _is_ included. So since even I myself is confused by this code, better clarify it ;-) Perhaps it would be clearer if changed to (sizeof("do_domain_ids=")-1) (both in my old code and in your new code). (Normally it's good to follow the style of existing code as you did, I just noticed this in the review and realised that my own "existing code" would benefit from these small improvements). > @@ -634,6 +653,16 @@ > } > } > > + char* do_domain_ids_buf= > + mi->domain_id_filter.as_string(Domain_id_filter::DO_DOMAIN_IDS); > + if (do_domain_ids_buf == NULL) > + DBUG_RETURN(1); > + > + char* ignore_domain_ids_buf= > + mi->domain_id_filter.as_string(Domain_id_filter::IGNORE_DOMAIN_IDS); > + if (ignore_domain_ids_buf == NULL) > + DBUG_RETURN(1); You have a memory leak here, do_domain_ids_buf will not be freed if ignore_domain_ids_buf fails due to out-of-memory. > @@ -1348,4 +1382,218 @@ > DBUG_RETURN(result); > } > > +/* ctor */ > +Domain_id_filter::Domain_id_filter() : m_filter(false) > +{ > + for (int i= DO_DOMAIN_IDS; i <= IGNORE_DOMAIN_IDS; i ++) > + { > + my_init_dynamic_array(&m_domain_ids[i], sizeof(uint32), 16, 16, MYF(0)); > + } > +} > + > +/* dtor */ > +Domain_id_filter::~Domain_id_filter() I don't think comments like /* ctor */ and /* dtor */ are useful, better just omit them. > +/** > + Update the domain id list of the given type with elements from the new list. > + > + @param new_ids [IN] new list of ids > + @param type [IN] domain id list type to update > + @param changed [IN] has the domain id list changed in the > + last CHANGE MASTER command? > + > + @retval void > +*/ > +void Domain_id_filter::update_ids(DYNAMIC_ARRAY *new_ids, enum_list_type type, > + bool changed) > +{ > + DYNAMIC_ARRAY *ids= &m_domain_ids[type]; > + > + if (changed == true) > + reset_dynamic(ids); > + > + /* bsearch requires an ordered list. */ > + sort_dynamic(new_ids, change_master_domain_id_cmp); > + > + for (uint i= 0; i < new_ids->elements; i++) > + { > + uint32 domain_id; > + get_dynamic(new_ids, (void *) &domain_id, i); > + > + if (bsearch((const uint32 *) &domain_id, ids->buffer, ids->elements, > + sizeof(uint32), change_master_domain_id_cmp) == NULL) > + { > + insert_dynamic(ids, (uint32*) &domain_id); > + } > + } > + return; > +} Ok, so I understand that this code is modelled after the existing code for IGNORED_SERVER_IDS. But I found it rather confusing, because it looks like there is a bug: if changed==false, then the code does not ensure that the resulting array will still be sorted. I guess the bug is not possible to trigger, because the list of stuff to add will only be non-empty if changed==true? But that is rather confusing. Please simplify the logic; maybe you only need to call this function when changed==true, so the parameter can be omitted and the array always reset? Also, consider if you can rewrite the existing code for IGNORE_SERVER_IDS so that the same common code can be used as for IGNORE_DOMAIN_IDS and DO_DOMAIN_IDS, that would simplify the code and reduce risk of bugs or inconsistencies. > +/** > + Initialize the given domain id list with the numbers from the specified > + buffer. > + > + @param buf [IN] buffer > + @param type [IN] domain id list type > + > + @retval false Success > + true Error > +*/ > +bool Domain_id_filter::init_ids(char *buf, enum_list_type type) > +{ > + char *token, *last; > + uint num_items; > + DYNAMIC_ARRAY *ids= &m_domain_ids[type]; > + > + token= strtok_r(buf, " ", &last); > + if (token == NULL) > + { > + return true; /* error */ > + } > + > + num_items= atoi(token); > + for (uint i= 0; i < num_items; i++) > + { > + token= strtok_r(NULL, " ", &last); > + if (token == NULL) > + { > + return true; /* error */ > + } > + else > + { > + uint32 val= atoi(token); > + insert_dynamic(ids, (uchar *) &val); > + } > + } > + return false; > +} Again, did you consider using the same code for this and for IGNORE_SERVER_IDS? (I think you did, because you added a comment to init_dynarray_intvar_from_file ;-). Maybe consider it again; it is fine to rewrite/remove the existing code for IGNORE_SERVER_IDS and replace it with your new code. Again, it would reduce risk for bugs or inconsistencies. > + /* getter */ > + bool get_filter() { return m_filter; } > + > + /* setter */ > + void set_filter(uint32 domain_id); These are not really getter and setters, I think, at least I find those names confusing. Maybe something like do_filter(uint32 domain_id) and is_group_filtered() ? And expand the comments, so that they explain that these functions respectively check a GTID for needing to be filtered, and return whether the current group is being filtered. Also, personally I dislike setter/getter, and would prefer just to access the m_filter as a public member. But I suppose that's style and up to you. > + /* > + Initialize the given domain id list with the numbers from the specified > + buffer. > + > + @param buf [IN] buffer > + @param type [IN] domain id list type > + > + @retval false Success > + true Error > + */ > + bool init_ids(char *buf, enum_list_type type); Clarify in the comment that the list is initialized from a string of space-separated numbers, the first of which is the number of following entries. > @@ -182,6 +279,7 @@ > bool flush_relay_log_cache, > bool need_lock_relay_log); > int change_master_server_id_cmp(ulong *id1, ulong *id2); > +int change_master_domain_id_cmp(uint32 *id1, uint32 *id2); I think change_master_domain_id_cmp() is only used in rpl_mi.cc, so should be static there, and not refered here in rpl_mi.h > === modified file 'sql/slave.cc' > --- a/sql/slave.cc 2014-08-12 03:55:41 +0000 > +++ b/sql/slave.cc 2014-09-12 18:16:22 +0000 > @@ -1259,6 +1259,7 @@ > going to ignore (to not log them in the relay log). > Items being read are supposed to be decimal output of values of a > type shorter or equal of @c long and separated by the single space. > + It also used to restore DO_DOMAIN_IDS & IGNORE_DOMAIN_IDS lists. This comment suggests you wanted to share code, then later changed your mind. If you end up keeping the code separate, you still need to delete this comment. (And there is a typo: "It also" -> "It is also", but if you remove it anyway...). > @@ -5601,6 +5610,10 @@ > mi->last_queued_gtid= event_gtid; > mi->last_queued_gtid_standalone= > (gtid_flag & Gtid_log_event::FL_STANDALONE) != 0; > + > + /* Should filter all the subsequent events in the current GTID group? */ > + mi->domain_id_filter.set_filter(event_gtid.domain_id); So, here we will set m_filter to true, if we want to skip the following group. But I don't see any code that will set it to false at the end of the group? That seems a problem, as we could then end up wrongly skipping all kinds of other events, such as Gtid_list, Incident, ... Also note that it is possible for a GTID group to not be terminated by COMMIT/XID. This can happen at least if the master crashes in the middle of writing a group; the slave will receive only the first part of it, followed by a master restart Format_description event. That case needs to be handled, and you should add a test case for it. Also it is possible for eg. a master downgrade to introduce following event groups that do not start with GTID - seems these would also be wrongly skipped? So we also need a test case for this... Hope this helps, - Kristian.

2 8

[Maria-developers] The variable "table_definition_cache" is autosized by default on MySQL 5.6.8+ but not on MariaDB 10
by Jean Weisbuch 20 Nov '14

20 Nov '14

The value of the variable "table_definition_cache" is by default autosized (based on the table_open_cache value) on MySQL >=5.6.8 while it is still static (400) on the latest MariaDB 10.0.14 and 10.1.?, is that by choice or this feature hasn't been backported and should be? From the MySQL doc http://dev.mysql.com/doc/refman/5.6/en/server-system-variables.html#sysvar_… : *Permitted Values (>= 5.6.8)* *Type* |numeric| *Default* |-1 (autosized)| *Min Value* |400| *Max Value* |524288| The default value is based on the following formula, capped to a limit of 2000: 400 + (table_open_cache / 2) Before MySQL 5.6.8, the default is 400.

1 0

Re: [Maria-developers] [Commits] Rev 4367: MDEV-7026: Race in InnoDB/XtraDB mutex implementation can stall or hang the server. in http://bazaar.launchpad.net/~maria-captains/maria/5.5
by Sergey Vojtovich 20 Nov '14

20 Nov '14

Hi Kristian, thanks for taking over this bug! I believe this is the patch that was pushed into 5.5. Unfortunately I think there is still one potential for a hang left (details below). :( Since InnoDB and XtraDB changes are identical I'll comment only XtraDB changes. So you removed os_isync and implemented os_mb instead. That's good! You also removed a bunch of os_wmb/os_rmb calls. That's good too. And you kept RELEASE barrier for mutex_exit() on PPC64. Very good! I will also comment things that I don't really like, but they're just for your information (no action required). On Wed, Nov 19, 2014 at 01:56:46PM +0100, knielsen(a)knielsen-hq.org wrote: > At http://bazaar.launchpad.net/~maria-captains/maria/5.5 > > ------------------------------------------------------------ > revno: 4367 > revision-id: knielsen(a)knielsen-hq.org-20141119125646-njq02h7irmu9s71b > parent: sergii(a)pisem.net-20141118231952-fk3o69075dn3ravv > committer: Kristian Nielsen <knielsen(a)knielsen-hq.org> > branch nick: mariadb-5.5 > timestamp: Wed 2014-11-19 13:56:46 +0100 > message: > MDEV-7026: Race in InnoDB/XtraDB mutex implementation can stall or hang the server. > > The bug was that full memory barrier was missing in the code that ensures that > a waiter on an InnoDB mutex will not go to sleep unless it is guaranteed to be > woken up again by another thread currently holding the mutex. This made > possible a race where a thread could get stuck waiting for a mutex that is in > fact no longer locked. If that thread was also holding other critical locks, > this could stall the entire server. There is an error monitor thread than can > break the stall, it runs about once per second. But if the error monitor > thread itself got stuck or was not running, then the entire server could hang > infinitely. > > This was introduced on i386/amd64 platforms in 5.5.40 and 10.0.13 by an > incorrect patch that tried to fix the similar problem for PowerPC. JFYI: Strictly speaking that "incorrect patch" attempted to fix different problem (read carefully): it added RELEASE barrier between "stores inside mutex" and "lock_word= 0" while this patch adds full barrier between "lock_word= 0" and "if (waiters)". > > This commit reverts the incorrect PowerPC patch, and instead implements a fix > for PowerPC that does not change i386/amd64 behaviour, making PowerPC work > similarly to i386/amd64. ...skip... > === modified file 'storage/xtradb/include/os0sync.h' > --- a/storage/xtradb/include/os0sync.h 2014-09-08 15:10:48 +0000 > +++ b/storage/xtradb/include/os0sync.h 2014-11-19 12:56:46 +0000 > @@ -314,11 +331,28 @@ amount of increment. */ > /**********************************************************//** > Returns the old value of *ptr, atomically sets *ptr to new_val */ > > -# define os_atomic_test_and_set_byte(ptr, new_val) \ > +#ifdef __powerpc__ > +/* > + os_atomic_test_and_set_byte_release() should imply a release barrier before > + setting, and a full barrier after. But __sync_lock_test_and_set() is only > + documented as an aquire barrier. So on PowerPC we need to add the full > + barrier explicitly. */ > +# define os_atomic_test_and_set_byte_release(ptr, new_val) \ > + do { __sync_lock_release(ptr); \ > + __sync_synchronize(); } while (0) > +#else > +/* > + On x86, __sync_lock_test_and_set() happens to be full barrier, due to > + LOCK prefix. > +*/ > +# define os_atomic_test_and_set_byte_release(ptr, new_val) \ > + __sync_lock_test_and_set(ptr, (byte) new_val) > +#endif JFYI: "os_atomic_test_and_set_byte_release" sounds odd to me. Look at gcc manual: https://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Atomic-Builtins.html ... This builtin is not a full barrier, but rather an acquire barrier. This means that references after the builtin cannot move to (or be speculated to) before the builtin, but previous memory stores may not be globally visible yet, and previous memory loads may not yet be satisfied. ... You name RELEASE something that is acqually ACQUIRE. So why didn't you like "os_atomic_lock_release_byte" which is RELASE btw: ... This builtin is not a full barrier, but rather a release barrier. This means that all previous memory stores are globally visible, and all previous memory loads have been satisfied, but following memory reads are not prevented from being speculated to before the barrier. ... > +/* > + os_atomic_test_and_set_byte_acquire() is a full memory barrier on x86. But > + in general, just an aquire barrier should be sufficient. */ > +# define os_atomic_test_and_set_byte_acquire(ptr, new_val) \ > __sync_lock_test_and_set(ptr, (byte) new_val) !!! The problem is here!!! Strictly speaking not here, but in mutex_spin_wait(): ... mutex_set_waiters(mutex, 1); /* Try to reserve still a few times */ for (i = 0; i < 4; i++) { if (mutex_test_and_set(mutex) == 0) { ... We'll get stuck if "waiters= 1" will become visible after "if (mutex_test_and_set(mutex))". I don't see anything preventing this reorder. ...skip... > === modified file 'storage/xtradb/include/sync0sync.ic' > --- a/storage/xtradb/include/sync0sync.ic 2014-08-29 12:02:46 +0000 > +++ b/storage/xtradb/include/sync0sync.ic 2014-11-19 12:56:46 +0000 > @@ -80,11 +80,11 @@ mutex_test_and_set( > mutex_t* mutex) /*!< in: mutex */ > { > #if defined(HAVE_ATOMIC_BUILTINS) > - return(os_atomic_test_and_set_byte(&mutex->lock_word, 1)); > + return(os_atomic_test_and_set_byte_acquire(&mutex->lock_word, 1)); > #else > ibool ret; > > - ret = os_fast_mutex_trylock(&(mutex->os_fast_mutex)); > + ret = os_fast_mutex_trylock_full_barrier(&(mutex->os_fast_mutex)); > > if (ret == 0) { > /* We check that os_fast_mutex_trylock does not leak JFYI: This is probably the most funny thing. Here we're in the "#else" branch of "#if defined(HAVE_ATOMIC_BUILTINS)". E.g. if we compile with gcc and enter this branch it means: sync builtins are not avialble (gcc atomic builtins are very likely not available too). Now you say "os_fast_mutex_trylock_full_barrier" which does os_mb. But since neither sync bultins nor atomic builtins are available, os_mb is basically no-op. This may be different on other platforms, but to my knowledge memory barriers and atomic operations normally come together. Another thing: did you try compiling with this branch enabled? IIRC it didn't compile even berfore we started fixing it. ...skip... Regards, Sergey

2 2

[Maria-developers] constant propagation & LIKE
by Alexander Barkov 20 Nov '14

20 Nov '14

Hello Sergei, The attached patch fixes MDEV-7149 Constant propagation erroneously applied for LIKE All tests pass. But I'm not 100% sure about the patch. Is eq_cmp_result() proper place to fix this? If it's the right place, I can test the collation properties and return: - COND_TRUE for simple collations with one-character-to-one-weight rules, - COND_OK for the complex collations with tricky character to weight mapping. Thanks.

1 0

Re: [Maria-developers] [Commits] Rev 4358: MDEV-7113 difference between check_vcol_func_processor and check_partition_func_processor in lp:~maria-captains/maria/5.5
by Sergey Petrunia 20 Nov '14

20 Nov '14

Hi Sergei, * what about persistent virtual columns? Do we still support environment-dependent virtual functions for those? * the behavior of current solution is not deterministic: thread1> select @@time_zone; +-------------+ | @@time_zone | +-------------+ | SYSTEM | +-------------+ 1 row in set (0.00 sec) thread1> create table t2 ( a varchar(32), b int, v varchar(100) as (from_unixtime(b))); Query OK, 0 rows affected (0.28 sec) thread1> insert into t2 (a,b) values ('local-tz-23-11', unix_timestamp()); Query OK, 1 row affected (0.04 sec) thread1> select * from t2; +----------------+------------+---------------------+ | a | b | v | +----------------+------------+---------------------+ | local-tz-23-11 | 1416427891 | 2014-11-19 23:11:31 | +----------------+------------+---------------------+ 1 row in set (0.00 sec) thread2> set time_zone='Europe/Helsinki'; Query OK, 0 rows affected (0.00 sec) thread2> select * from t2; +----------------+------------+---------------------+ | a | b | v | +----------------+------------+---------------------+ | local-tz-23-11 | 1416427891 | 2014-11-19 23:11:31 | +----------------+------------+---------------------+ 1 row in set (0.00 sec) thread2> flush tables; Query OK, 0 rows affected (0.00 sec) thread2> select * from t2; +----------------+------------+---------------------+ | a | b | v | +----------------+------------+---------------------+ | local-tz-23-11 | 1416427891 | 2014-11-19 22:11:31 | +----------------+------------+---------------------+ 1 row in set (0.00 sec) Is this really ok? Maybe, it makes sense to use some some "default" (e.g. global variables) environment on the non-deterministic virtual column functions? Or, make these functions always fetch the environment from the current thread? On Mon, Nov 17, 2014 at 09:43:46AM +0100, Sergei Golubchik wrote: > At lp:~maria-captains/maria/5.5 > > ------------------------------------------------------------ > revno: 4358 > revision-id: sergii(a)pisem.net-20141115155753-rx5twf6utu1isczh > parent: sergii(a)pisem.net-20141113173729-hirl67og6at3xbni > fixes bugs: https://mariadb.atlassian.net/browse/MDEV-6789 https://mariadb.atlassian.net/browse/MDEV-7113 > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 5.5 > timestamp: Sat 2014-11-15 16:57:53 +0100 > message: > MDEV-7113 difference between check_vcol_func_processor and check_partition_func_processor > MDEV-6789 segfault in Item_func_from_unixtime::get_date on updating table with virtual columns > > * prohibit VALUES in partitioning expression > * prohibit user and system variables in virtual column expressions > * fix Item_func_date_format to cache locale (for %M/%W to return the same as MONTHNAME/DAYNAME) > * fix Item_func_from_unixtime to cache time_zone directly, not THD (and not to crash) > * added tests for other incorrectly allowed (in vcols) functions to see that they don't crash ... BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 2

Re: [Maria-developers] [Commits] Rev 4346: MDEV-7026 - Occasional hang during startup on Power8 in lp:maria/5.5
by Kristian Nielsen 19 Nov '14

19 Nov '14

Sergey Vojtovich <svoj(a)mariadb.org> writes: > revno: 4346 > revision-id: svoj(a)mariadb.org-20141118080742-sdf93kgsgxulenvb > parent: knielsen(a)knielsen-hq.org-20141112101013-t63ayylsk08ncgs3 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5 > timestamp: Tue 2014-11-18 12:07:42 +0400 > message: > MDEV-7026 - Occasional hang during startup on Power8 Right, so you requested second review ASAP, I'm trying to understand what's going on here. If my understanding is correct, this patch goes on top of current 5.5.40, not on top of the patch I reviewed previously. So there are two cases, depending on whether INNODB_RW_LOCKS_USE_ATOMICS is set or not. It is my understanding that this is "usually" set, eg. on our common x86 and powerpc linux binaries. The critical thing for the patch at hand is the memory barrier between mutex_set_waiters() and mutex_test_and_set() in the waiter, and mutex_reset_lock_word() and mutex_get_waiters() in the thread releasing the lock. A full memory barrier is needed in the waiter, to be sure that mutex_set_waiters() becomes visible to the releasing thread before mutex_test_and_set() runs. And a full barrier is needed in the releasing thread, to be sure that mutex_reset_lock_word() becomes visible before mutex_get_waiters() runs. So let's see how this is ensured. First the case of x86 with INNODB_RW_LOCKS_USE_ATOMICS set, HAVE_ATOMIC_BUILTINS, pre-5.5.40: - mutex_set_waiters() uses os_compare_and_swap_ulint() which is __sync_bool_compare_and_swap() which is full barrier. - mutex_reset_lock_word() uses os_atomic_test_and_set_byte() which is __sync_lock_test_and_set(). This is documented only as a release barrier, but is in fact a full barrier on x86 with current GCC. So in this case full barrier is there as needed. If HAVE_ATOMIC_BUILTINS is not set, then mutex_reset_lock_word() uses os_fast_mutex_unlock() which is pthread_mutex_unlock(), which is also full barrier on x86. If INNODB_RW_LOCKS_USE_ATOMICS is not set, then mutex_set_waiters() has no barriers. But then mutex_test_and_set() instead has a full barrier on x86. Either in os_atomic_test_and_set_byte() which is __sync_lock_test_and_set(), if HAVE_ATOMIC_BUILTINS is set, or in os_fast_mutex_trylock() otherwise. Both of these are full barriers on x86. So this is why things work ok on x86, apparently. Very bad code that seriously needs cleanup and proper documentation of memory barrier semantics. But it is what we have to work with in 5.5. So to make things work on PowerPC without risk of breaking x86, we could do this: - Make sure os_atomic_test_and_set_byte() is full barrier on PowerPC: #ifdef __powerpc__ /* os_atomic_test_and_set_byte() should imply a full barrier. But __sync_lock_test_and_set() is only documented as an aquire barrier. So on PowerPC we need to add the full barrier explicitly. */ # define os_atomic_test_and_set_byte(ptr, new_val) \ do { __sync_lock_test_and_set(ptr, (byte) new_val); \ __atomic_thread_fence(__ATOMIC_SEQ_CST); } while(0) #else /* On x86, __sync_lock_test_and_set() happens to be full barrier, due to LOCK prefix. */ # define os_atomic_test_and_set_byte(ptr, new_val) \ __sync_lock_test_and_set(ptr, (byte) new_val) #endif - Make sure that mutex_reset_lock_word() is a full barrier when HAVE_ATOMIC_BUILTINS is not set, by adding a memory barrier: os_fast_mutex_unlock(&(mutex->os_fast_mutex)); #ifdef __powerpc__ /* On PowerPC, pthread_mutex does not imply full barrier, so we need to add it explicitly. */ __atomic_thread_fence(__ATOMIC_SEQ_CST); #endif - For the case of neither HAVE_ATOMIC_BUILTINS nor INNODB_RW_LOCKS_USE_ATOMICS set, similarly add a full barrier in mutex_test_and_set() for PowerPC: #ifdef __powerpc__ /* On PowerPC, pthread_mutex does not imply full barrier, so we need to add it explicitly. */ __atomic_thread_fence(__ATOMIC_SEQ_CST); #endif ret = os_fast_mutex_trylock(&(mutex->os_fast_mutex)); Then in 10.1. if someone is brave enough, we can clean this up properly. Clearly define what the memory barrier semantics is, and implement it using the correct primitives that work as needed on all platforms. So now that I understand (hopefully...) how the code used to work, and how it can be fixed for PowerPC without changing existing code, let me look at the new patch ... > === modified file 'storage/innobase/include/os0sync.h' > --- a/storage/innobase/include/os0sync.h 2014-09-08 15:10:48 +0000 > +++ b/storage/innobase/include/os0sync.h 2014-11-18 08:07:42 +0000 > @@ -310,8 +310,16 @@ Returns the old value of *ptr, atomicall > # define os_atomic_test_and_set_byte(ptr, new_val) \ > __sync_lock_test_and_set(ptr, (byte) new_val) > > +#ifdef __powerpc__ > # define os_atomic_lock_release_byte(ptr) \ > __sync_lock_release(ptr) > +#else > +/* In theory __sync_lock_release should be used to release the lock. > +Unfortunately, it does not work properly alone. The workaround is > +that more conservative __sync_lock_test_and_set is used instead. */ > +# define os_atomic_lock_release_byte(ptr) \ > + __sync_lock_test_and_set(ptr, 0) > +#endif This is very confusing. You define a method with full barrier semantics, but call it something suggesting release semantics. Why not revert the original change so that mutex_reset_lock_word() uses os_atomic_test_and_set_byte()? And then fix the implementation of os_atomic_test_and_set_byte() to be full barrier on PowerPC, as I suggested above? The term "atomic" generally implies full barrier in the MySQL source code... > #elif defined(HAVE_IB_SOLARIS_ATOMICS) > > @@ -428,7 +436,7 @@ clobbered */ > # define os_rmb __atomic_thread_fence(__ATOMIC_ACQUIRE) > # define os_wmb __atomic_thread_fence(__ATOMIC_RELEASE) > #ifdef __powerpc__ > -# define os_isync __asm __volatile ("isync":::"memory") > +# define os_isync __atomic_thread_fence(__ATOMIC_SEQ_CST) > #else > #define os_isync do { } while(0) > #endif As Jonas suggested, don't call a full memory barrier "isync". And do not introduce something that is a barrier on one platform but a no-operation on another platform. But ... > === modified file 'storage/innobase/sync/sync0sync.c' > --- a/storage/innobase/sync/sync0sync.c 2014-11-03 13:43:44 +0000 > +++ b/storage/innobase/sync/sync0sync.c 2014-11-18 08:07:42 +0000 > @@ -474,10 +474,9 @@ mutex_set_waiters( > > ptr = &(mutex->waiters); > > - os_wmb; > - > *ptr = n; /* Here we assume that the write of a single > word in memory is atomic */ > + os_isync; > } With this, the semantics of how things work becomes even more complicated. Now on PowerPC the barrier is in one function, but on x86 the barrier is in another function. Why not instead do as I suggested above? Put in full barriers for PowerPC in the same places that they are in x86? ---- Maybe I should try to say if I think the patch fixes the actual bug, in case you don't want to go with my suggestions and instead keep your approach? Since it goes back to using __sync_lock_test_and_set() on x86, and since it has a full barrier in mutex_exit_func() on PowerPC, I think perhaps it does. But it is really hard to understand exactly what is going on, as you can see above I had to do a lot of reasoning to understand even why the old code worked... My advice is still to revert the original incorrect PowerPC patch and do something that is at least consistent with the existing code... (And then feel free to do a proper cleanup in 10.1). Hope this helps, - Kristian.

2 2

Re: [Maria-developers] [Commits] Rev 4346: MDEV-7026 - Occasional hang during startup on Power8 in lp:maria/5.5
by Jonas Oreland 18 Nov '14

18 Nov '14

> > Since isync is not a memory barrier and lwsync doesn't guarantee > StoreLoad > order, the only option we have is full memory barrier. > === modified file 'storage/innobase/include/os0sync.h' > --- a/storage/innobase/include/os0sync.h 2014-09-08 15:10:48 +0000 > +++ b/storage/innobase/include/os0sync.h 2014-11-18 08:07:42 +0000 > > #ifdef __powerpc__ > -# define os_isync __asm __volatile ("isync":::"memory") > +# define os_isync __atomic_thread_fence(__ATOMIC_SEQ_CST) > you have to rename to the #define too (e.g to os_mb). having a os_isync which is infact a mb is more obfuscation than anyone should have to put up with. /Jonas

2 1

Re: [Maria-developers] [Commits] Rev 4346: MDEV-7026 - Occasional hang during startup on Power8 in lp:maria/5.5
by Kristian Nielsen 17 Nov '14

17 Nov '14

Sergey Vojtovich <svoj(a)mariadb.org> writes: > revno: 4346 > revision-id: svoj(a)mariadb.org-20141112114907-thnro6e3kg2ofdsw > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: Wed 2014-11-12 15:49:07 +0400 > message: > MDEV-7026 - Occasional hang during startup on Power8 > === modified file 'storage/xtradb/sync/sync0sync.c' > --- a/storage/xtradb/sync/sync0sync.c 2014-11-03 13:43:44 +0000 > +++ b/storage/xtradb/sync/sync0sync.c 2014-11-12 11:49:07 +0000 > @@ -467,26 +467,15 @@ mutex_set_waiters( > mutex_t* mutex, /*!< in: mutex */ > ulint n) /*!< in: value to set */ > { > -#ifdef INNODB_RW_LOCKS_USE_ATOMICS > - ut_ad(mutex); > - > - if (n) { > - os_compare_and_swap_ulint(&mutex->waiters, 0, 1); > - } else { > - os_compare_and_swap_ulint(&mutex->waiters, 1, 0); > - } > -#else > volatile ulint* ptr; /* declared volatile to ensure that > the value is stored to memory */ > ut_ad(mutex); > > ptr = &(mutex->waiters); > > - os_wmb; > - > *ptr = n; /* Here we assume that the write of a single > word in memory is atomic */ > -#endif > + os_sync; > } This code is still substantially different from what it was in 5.5.39 and 10.0.12, right? I think we need to instead revert the code to what it was before introducing the problem. The InnoDB mutex code clearly does not work by careful design - mutex_reset_lock_word() needs a full barrier but uses what is documented by GCC as only an acquire barrier. Rather, it works by having been tweaked over the years to just happen to work (eg. GCC in fact uses a full barrier on x86). So changing this to be "correct" is very dangerous in a stable release. It is not possible to ensure correctness by testing, either. But it is not necessary to change it. Instead, leave the code as it was before, and use #ifdef __powerpc__ to ensure that it will work also on PowerPC. For example, something like this (just a sketch, I didn't check exactly how the code should look): #ifdef __powerpc__ /* InnoDB mutex implementation needs full memory barrier for this operation */ # define os_atomic_test_and_set_byte(ptr, new_val) \ do { sync(); __sync_lock_test_and_set(ptr, (byte) new_val); } while (0) #else # define os_atomic_test_and_set_byte(ptr, new_val) \ __sync_lock_test_and_set(ptr, (byte) new_val) #endif Then we leave stable releases 5.5 and 10.0 unchanged, reducing the risk of introducing another serious breakage. And we can still fix PowerPC to work correctly. And then perhaps in 10.1 we can do refactoring of the use of memory barriers in InnoDB, if we really want. (Just in case it's not obvious, yes, I _really_ have the InnoDB mutex code). Remember, the 5.5 and 10.0 minor releases appear as security updates in distros, so it is very important to minimise the risk of introducing serious regressions. > Since isync is not a memory barrier and lwsync doesn't guarantee StoreLoad > order, the only option we have is full memory barrier. It's not a problem to have a full barrier here, I think. Because at this point, the waiter has already gone through a lot of spin loops, and now it is going to take a global mutex on the sync wait array, scan the array, and go to sleep with a context switch. One less full barrier is not going to help a lot. To improve this contended case of the InnoDB mutexes, more radical changes are needed. > InnoDB/XtraDB was getting stuck during server startup. In particular (but not > limited to) while creating doublewrite buffer. The stuck during startup is kind of the minor part of the bug. The major part is stalling the server during operation. (The hang during startup was just how the problem was detected in Buildbot). Better clarify in the commit message that this fixes a bug that could cause the entire server to stall (or even hang completely if error monitor thread itself got stuck). This will help people who experienced the problem and look at the changelog. > It happens because mutex_exit() didn't wake up waiters of affected mutex. This > is kind of acceptable because error monitor thread is supposed to wake up such > waiters. But since error monitor thread hasn't been started that early, nobody > did inform waiting threads that mutex was released. I do not think the error monitor is "supposed to wake up such waiters". In fact, it seems clear that the error monitor wakeup is _not_ used, at least not in commonly used binaries. Because it just never worked - it itself needs to take mutexes, which will prevent it from doing the wakeup. So if it was needed, someone would have noticed this problem. The symptom is _very_ hard to miss, as the whole server gets stuck hard, unable to do anything. It really is not acceptable to have a mutex implementation that can miss wakeups. This can stall the entire server. Even if the error monitor resolves the stall after 1 second, that is still unacceptable for many applications. Hope this helps, - Kristian.

2 3

[Maria-developers] Review: [MDEV-7062] parser refactoring: don't store field properties in LEX
by Alexander Barkov 16 Nov '14

16 Nov '14

Hi Sergei, > Sergei Golubchik edited comment on MDEV-7062 at 11/10/14 11:40 AM: > ------------------------------------------------------------------- > > http://lists.askmonty.org/pipermail/commits/2014-November/006908.html > > {noformat} > From: serg(a)mariadb.org > To: commits(a)mariadb.org > Subject: [Commits] 2c904b9: parser cleanup: don't store field properties > in LEX, use Create_field directly > Date: Mon, 10 Nov 2014 10:26:48 +0100 > {noformat} > Please find my comments below: > > [Commits] 2c904b9: parser cleanup: don't store field properties in LEX, use Create_field directly > serg at mariadb.org serg at mariadb.org > Mon Nov 10 11:26:48 EET 2014 > > Previous message: [Commits] Rev 4344: MDEV-6179: dynamic columns functions/cast()/convert() doesn't play nice with CREATE/ALTER TABLE in lp:~maria-captains/maria/5.5 > Next message: [Commits] Rev 4345: MDEV-6862 "#error <my_config.h>" and third-party libraries in lp:~maria-captains/maria/5.5 > Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] > > revision-id: 2c904b937b5ef7fbec60ebb9cb514a501c5bd8e4 > parent(s): f59588b9112965849986c45fcd150d9abed74063 > committer: Sergei Golubchik > branch nick: maria > timestamp: 2014-11-09 22:33:17 +0100 > message: > > parser cleanup: don't store field properties in LEX, use Create_field directly > > length/dec/charset are still in LEX, because they're also used > for CAST and dynamic columns. > > also > 1. fix "MDEV-7041 COLLATION(CAST('a' AS CHAR BINARY)) returns a wrong result" > 2. allow BINARY modifier in stored function RETURN clause > 3. allow "COLLATION without CHARSET" in SP/SF (parameters, RETURN, DECLARE) > 4. print correct variable name in error messages for stored routine parameters A very nice clean up. I only found it a little bit not easy to track that all Lex and Create_field attributes get initialized and copied to each other properly. I suggest to go a little bit further and get rid from Lex->dec, Lex->length and Lex->charset at all, and write the attributes directly to the destination as soon as they get parsed. See a proposed addon patch attached. > diff --git a/mysql-test/r/cast.result b/mysql-test/r/cast.result > index c81af13..8ad4568 100644 > --- a/mysql-test/r/cast.result > +++ b/mysql-test/r/cast.result > @@ -796,3 +796,32 @@ DATE("foo") > NULL > Warnings: > Warning 1292 Incorrect datetime value: 'foo' > +create table t1 (a int, b char(5) as (cast("a" as char(10) binary) + a) ); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL, > + `b` char(5) AS (cast("a" as char(10) binary) + a) VIRTUAL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +drop table t1; > +select collation(cast("a" as char(10) binary)); > +collation(cast("a" as char(10) binary)) > +latin1_bin > +select collation(cast("a" as char(10) charset utf8 binary)); > +collation(cast("a" as char(10) charset utf8 binary)) > +utf8_bin > +select collation(cast("a" as char(10) ascii binary)); > +collation(cast("a" as char(10) ascii binary)) > +latin1_bin > +select collation(cast("a" as char(10) unicode binary)); > +collation(cast("a" as char(10) unicode binary)) > +ucs2_bin UNICODE is an alias for UCS2, which is not always compiled. Please move ucs2 related tests to ctype_ucs.test. <skip> > diff --git a/mysql-test/t/cast.test b/mysql-test/t/cast.test > index b6c37ca..aa43ee6 100644 > --- a/mysql-test/t/cast.test > +++ b/mysql-test/t/cast.test > @@ -456,3 +456,21 @@ SELECT CAST(TIME('10:20:30') AS DATE) + INTERVAL 1 DAY; > SET SQL_MODE=ALLOW_INVALID_DATES; > SELECT DATE("foo"); > > +# > +# CAST and field definition using same fields in LEX > +# > +create table t1 (a int, b char(5) as (cast("a" as char(10) binary) + a) ); > +show create table t1; > +drop table t1; > + > +# > +# CAST (... BINARY) > +# > +select collation(cast("a" as char(10) binary)); > +select collation(cast("a" as char(10) charset utf8 binary)); > +select collation(cast("a" as char(10) ascii binary)); > +select collation(cast("a" as char(10) unicode binary)); > +select collation(cast("a" as char(10) binary charset utf8)); > +select collation(cast("a" as char(10) binary ascii)); > +select collation(cast("a" as char(10) binary unicode)); More UCS2. You removed this code in field.cc: > - if (fld_length != NULL) > - { > - errno= 0; > - length= strtoul(fld_length, NULL, 10); > - if ((errno != 0) || (length > MAX_FIELD_BLOBLENGTH)) > - { > - my_error(ER_TOO_BIG_DISPLAYWIDTH, MYF(0), fld_name, MAX_FIELD_BLOBLENGTH); > - DBUG_RETURN(TRUE); > - } > - > - if (length == 0) > - fld_length= NULL; /* purecov: inspected */ > - } and added this in sql_yacc.yy: > + if (length) > + { > + int err; > + last_field->length= (ulong)my_strtoll10(length, NULL, &err); > + if (err) > + last_field->length= ~0UL; // safety > + } > + else > + last_field->length= 0; I have a feeling that on a 32bit platforms it will stop sending errors on attempt to create a blob larger that 4Gb. last_field->length is ulong, which is 32bit is a 32 bit platform, and it gets assigned to a 64 bit value.

2 2

[Maria-developers] Please review: sql_yacc.yy and HA_CREATE_INFO refactoring (pre-requisite for MDEV-5359 CREATE OR REPLACE, CREATE IF NOT EXISTS, DROP IF EXISTS)
by Alexander Barkov 14 Nov '14

14 Nov '14

Hi Sergei, The patch is attached. Thanks.

1 1

Re: [Maria-developers] [JIRA] (MDEV-7076) Build fails around libgroonga on some platforms in buildbot
by Alexander Barkov 12 Nov '14

12 Nov '14

Hello Kentoku, Mroonga fails to compile on some platforms. Can you please have a look? Thanks. On 11/12/2014 04:39 PM, Alexander Barkov (JIRA) wrote: > > [ https://mariadb.atlassian.net/browse/MDEV-7076?page=com.atlassian.jira.plug… ] > > Alexander Barkov updated MDEV-7076: > ----------------------------------- > Assignee: Kentoku (was: Alexander Barkov) > >> Build fails around libgroonga on some platforms in buildbot >> ----------------------------------------------------------- >> >> Key: MDEV-7076 >> URL: https://mariadb.atlassian.net/browse/MDEV-7076 >> Project: MariaDB Server >> Issue Type: Bug >> Components: OTHER >> Affects Versions: 10.0 >> Reporter: Elena Stepanova >> Assignee: Kentoku >> Priority: Critical >> Labels: buildbot, compile >> Fix For: 10.0.15 >> >> >> http://buildbot.askmonty.org/buildbot/builders/kvm-bintar-centos5-amd64/bui… >> http://buildbot.askmonty.org/buildbot/builders/kvm-deb-debian6-amd64/builds… >> http://buildbot.askmonty.org/buildbot/builders/kvm-deb-lucid-amd64/builds/4… >> It looks approximately like this, with some variations >> {noformat} >> standard input}: Assembler messages: >> {standard input}:28252: Error: open CFI at the end of file; missing .cfi_endproc directive >> gcc-4.4.real: Internal error: Killed (program cc1) >> Please submit a full bug report. >> See <file:///usr/share/doc/gcc-4.4/README.Bugs> for instructions. >> make[3]: *** [storage/mroonga/vendor/groonga/lib/CMakeFiles/libgroonga.dir/expr.c.o] Error 1 >> make[3]: *** Waiting for unfinished jobs.... >> {noformat} >> Given the list of platforms, I guess they all feature pretty old compilers, maybe that's the distinction. > > > > -- > This message was sent by Atlassian JIRA > (v6.4-OD-09-008#64005) >

1 0

[Maria-developers] Mroonga: wrapper.version_56_or_later_performance_schema is failing on BuildBot
by Alexander Barkov 12 Nov '14

12 Nov '14

Hello Kentoku, This test is failing in the latest builds on BB: mroonga/wrapper.version_56_or_later_performance_schema http://buildbot.askmonty.org/buildbot/builders/bld-dan-release/builds/1793/… Can you please check it? Thanks.

2 1

[Maria-developers] PLEASE REVIEWA: pre-requisite patch for the data type plugins, also fixing a number of optimizer bugs
by Alexander Barkov 12 Nov '14

12 Nov '14

Hello, While working on pluggable data types, I noticed a few problems related to optimizer: > MDEV-6950 Bad results with joins comparing DATE/DATETIME and INT/DECIMAL/DOUBLE/ENUM/VARCHAR columns > MDEV-6971 Bad results with joins comparing TIME and DOUBLE/DECIMAL columns > MDEV-6978 Bad results with joins comparing case insensitive VARCHAR/ENUM/SET expression to a _bin ENUM column > MDEV-6982 LEFT JOIN table elimination is not always used when it could > MDEV-6989 BINARY and COLLATE xxx_bin comparisions are not used for optimization in some cases > MDEV-6990 GROUP_MIN_MAX optimization is not applied in some cases when it could > MDEV-6991 GROUP_MIN_MAX optimization is erroneously applied in some cases In some cases wrong result sets are returned. So it would be nice to have this fixed in 10.0. I made these changes as a standalone patch. Please review. I also noticed some other optimizer bugs but not sure how to fix them properly: > MDEV-6986 Bad results with join comparing INT and VARCHAR columns > MDEV-6969 Bad results with joins comparing DOUBLE to BIGINT/DECIMAL columns > MDEV-6993 Bad results with join comparing DECIMAL and ENUM/SET columns Igor or SergeyP can possibly fix them on top of my changes. Also, I found some other problems (not related to optimizer): > MDEV-6973 XOR aggregates argument collations > MDEV-7005 NULLIF does not work as documented Not sure which version to fix them in. Please suggest. Thanks.

2 2

Re: [Maria-developers] MDEV-6885 - debs do not build on PPC64 in lp:maria/5.5
by Sergei Golubchik 11 Nov '14

11 Nov '14

Hi, Sergey! On Oct 17, Sergey Vojtovich wrote: > At lp:maria/5.5 > > ------------------------------------------------------------ > revno: 4327 > revision-id: svoj(a)mariadb.org-20141017112032-b0p2qvvlbn1xrfec > parent: svoj(a)mariadb.org-20141017110850-5rlt8fm3blxv4wtb > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5 > timestamp: Fri 2014-10-17 15:20:32 +0400 > message: > MDEV-6885 - debs do not build on PPC64 > > Include ha_oqgraph.so into deb package only if it was able to build. Sorry, cannot help much. It looks ok, but I've had so many problems with our debian packaging - all these fixes looks right, and still need twenty iterations to actually be right. So, try it in buildbot, push in some tree that builds deb packages. If it'll work in all deb builders - please, push into 5.5. Regards, Sergei

3 3

Re: [Maria-developers] [Commits] MDEV-60 Support for Spatial Reference systems for the GIS data.
by Sergei Golubchik 11 Nov '14

11 Nov '14

Hi, Alexey! First, please, please, *do not* send patches from Thunderbird. It took me a lot of time to fix it, the patch was badly garbled. May be broken formatting everywhere in the patch was also caused by this. Then, I didn't see any tests in the patch, not even syntax tests, but also not create field, store, read tests. Please add them. Now, the review: > diff --git a/sql/field.cc b/sql/field.cc > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -7808,6 +7808,35 @@ uint Field_blob::is_equal(Create_field *new_field) > > #ifdef HAVE_SPATIAL > > +uint gis_field_options_image(uchar *buff, List<Create_field> &create_fields) > +{ > + uint image_size= 0; > + uint n_field= 0; > + List_iterator<Create_field> it(create_fields); > + Create_field *field; > + while ((field= it++)) > + { > + if (field->sql_type != MYSQL_TYPE_GEOMETRY) > + continue; > + if (buff) > + int2store(buff + image_size, ((uint16) n_field)); > + image_size+= 2; > + if (buff) > + int2store(buff + image_size, ((uint16) field->length)); > + image_size+= 2; > + if (buff) > + int2store(buff + image_size, ((uint16) field->decimals)); > + image_size+= 2; > + if (buff) > + int4store(buff + image_size, ((uint32) field->srid)); > + image_size+= 4; > + n_field++; > + } > + > + return image_size; > +} Hmm... So, you basically, put an array of properties in frm, under EXTRA2_GIS Two comments: 1. Why n_field, length and decimals? I thought you only add srid, other properties are already stored elsewhere, aren't they? 2. Really think about upgrades downgrades. The "extra" section in the frm doesn't handle them well, I know how these problems happen. Think, what happens when you add more properties here? What if you remove some? How new server will parse old frms, how old server will read new frms? > + > + > void Field_geom::sql_type(String &res) const > { > CHARSET_INFO *cs= &my_charset_latin1; > @@ -9181,7 +9210,7 @@ bool Create_field::init(THD *thd, char *fld_name, enum_field_types fld_type, > Item *fld_on_update_value, LEX_STRING *fld_comment, > char *fld_change, List<String> *fld_interval_list, > CHARSET_INFO *fld_charset, uint fld_geom_type, > - Virtual_column_info *fld_vcol_info, > + char *fld_srid, Virtual_column_info *fld_vcol_info, fix the formatting, please > engine_option_value *create_opt, bool check_exists) > { > uint sign_len, allowed_type_modifier= 0; > @@ -9232,6 +9261,7 @@ bool Create_field::init(THD *thd, char *fld_name, enum_field_types fld_type, > pack_length= key_length= 0; > charset= fld_charset; > geom_type= (Field::geometry_type) fld_geom_type; > + srid= fld_srid ? (uint)atoi(fld_srid) : 0; and here > interval_list.empty(); > > comment= *fld_comment; > @@ -9635,7 +9665,7 @@ Field *make_field(TABLE_SHARE *share, uchar *ptr, uint32 field_length, > uint pack_flag, > enum_field_types field_type, > CHARSET_INFO *field_charset, > - Field::geometry_type geom_type, > + Field::geometry_type geom_type, uint srid, and here, and everywhere, I won't comment on that anymore > Field::utype unireg_check, > TYPELIB *interval, > const char *field_name) > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -2358,6 +2358,7 @@ struct LEX: public Query_tables_list > char *backup_dir; /* For RESTORE/BACKUP */ > char* to_log; /* For PURGE MASTER LOGS TO */ > char* x509_subject,*x509_issuer,*ssl_cipher; > + char *srid; really? it can only be used for field declarations, I don't think there's a need to put it in LEX see below. > String *wild; /* Wildcard in SHOW {something} LIKE 'wild'*/ > sql_exchange *exchange; > select_result *result; > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -6359,6 +6360,16 @@ precision: > } > ; > > +srid_option: > + /* empty */ > + { Lex->srid= (char *)0; } > + | > + SRID_SYM EQ NUM > + { > + Lex->srid=$3.str; > + } return the value of srid here, don't store it in lex. Like $$ = $3.str; or even $$ = atoi($3.str); > + ; > + > field_options: > /* empty */ {} > | field_opt_list {} > diff --git a/sql/table.cc b/sql/table.cc > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1451,6 +1464,20 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > #ifdef HAVE_SPATIAL > geom_type= (Field::geometry_type) strpos[14]; > charset= &my_charset_bin; > + if (gis_options && gis_options_len >= 10) > + { > + gis_length= uint2korr(gis_options+2); > + gis_decimals= uint2korr(gis_options+4); > + srid= uint4korr(gis_options+6); > + gis_options_len-= 10; > + gis_options+= 10; If you have the function to generate this data in field.cc, I'd put a small function to read it in field.cc too. Just to have all reading/writing code in the same place. > + } > + else > + { > + srid= 0; > + gis_length= 0; > + gis_decimals= 0; > + } > #else > goto err; > #endif > diff --git a/sql/unireg.h b/sql/unireg.h > --- a/sql/unireg.h > +++ b/sql/unireg.h > @@ -190,6 +190,7 @@ enum extra2_frm_value_type { > #define EXTRA2_ENGINE_IMPORTANT 128 > > EXTRA2_ENGINE_TABLEOPTS=128, > + EXTRA2_GIS=129, better make it 2, not 129 (below EXTRA2_ENGINE_IMPORTANT threshold), because you de facto ignore it anyway when HAVE_SPATIAL is not defined. > }; > > int rea_create_table(THD *thd, LEX_CUSTRING *frm, > diff --git a/unittest/mysys/CMakeLists.txt b/unittest/mysys/CMakeLists.txt > --- a/unittest/mysys/CMakeLists.txt > +++ b/unittest/mysys/CMakeLists.txt > @@ -17,7 +17,7 @@ MY_ADD_TESTS(bitmap base64 my_vsnprintf my_atomic my_rdtsc lf my_malloc > LINK_LIBRARIES mysys) > > MY_ADD_TESTS(ma_dyncol > - LINK_LIBRARIES mysqlclient) > + LINK_LIBRARIES mysqlclient stdc++) Why? > > IF(WIN32) > MY_ADD_TESTS(my_delete LINK_LIBRARIES mysys) > Regards, Sergei

2 2

[Maria-developers] Fwd: [debian-mysql] Bug#766996: libmariadb-client-lgpl-dev-compat: misses a mysql_config symlink
by Daniel Schepler 10 Nov '14

10 Nov '14

I recently got the bug report below for the MariaDB C client library (Debian package). The request to add a compatibility symlink from mysql_config to mariadb_config makes sense to me -- do you see any potential problems with adding it? -- Daniel Schepler ---------- Forwarded message ---------- From: Rene Engelhard <rene(a)debian.org> Date: Mon, Oct 27, 2014 at 6:48 AM Subject: [debian-mysql] Bug#766996: libmariadb-client-lgpl-dev-compat: misses a mysql_config symlink To: Debian Bug Tracking System <submit(a)bugs.debian.org> Package: libmariadb-client-lgpl-dev-compat Version: 2.0.0-1 Severity: important https://packages.debian.org/sid/libmariadb-client-lgpl-dev-compat says: "MariaDB Client Library for C, compatibility symlinks MariaDB is a fast, stable and true multi-user, multi-threaded SQL database server. SQL (Structured Query Language) is the most popular database query language in the world. The main goals of MariaDB are speed, robustness and ease of use. This package includes compatibility symlinks to allow sources expecting the MySQL client libraries to be built against the MariaDB Client Library for C." The full contents are: https://packages.debian.org/sid/amd64/libmariadb-client-lgpl-dev-compat/fil… so it won't work with packages expecting mysql_config to give them the flags -> it should also ship a mysql_config -> mariadb_config link. Regards, Rene _______________________________________________ pkg-mysql-maint mailing list pkg-mysql-maint(a)lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-mysql-maint

2 1

Re: [Maria-developers] [Commits] Rev 4343: MDEV-6179: dynamic columns functions/cast()/convert() doesn't play nice with CREATE/ALTER TABLE in lp:~maria-captains/maria/5.5
by Oleksandr Byelkin 10 Nov '14

10 Nov '14

Hi, Sergei! Thank you! The patch is OK. On 08.11.14 19:54, Sergei Golubchik wrote: > At lp:~maria-captains/maria/5.5 > > ------------------------------------------------------------ > revno: 4343 > revision-id: sergii(a)pisem.net-20141108185442-37i07fh0bvm2atjr > parent: sergii(a)pisem.net-20141102163302-8ukz7pi4mwsx89n0 > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 5.5 > timestamp: Sat 2014-11-08 19:54:42 +0100 > message: > MDEV-6179: dynamic columns functions/cast()/convert() doesn't play nice with CREATE/ALTER TABLE > > When parsing a field declaration, grab type information from LEX before it's overwritten > by further rules. Pass type information through the parser stack to the rule that needs it. > === modified file 'mysql-test/r/create.result' > --- a/mysql-test/r/create.result 2013-05-11 17:31:50 +0000 > +++ b/mysql-test/r/create.result 2014-11-08 18:54:42 +0000 > @@ -2430,3 +2430,50 @@ a b > 1 1 > unlock tables; > drop table t1,t2; > +# > +# MDEV-6179: dynamic columns functions/cast()/convert() doesn't > +# play nice with CREATE/ALTER TABLE > +# > +create table t1 ( > +color char(32) as (COLUMN_GET(dynamic_cols, 1 as char)) persistent, > +cl char(32) as (COLUMN_GET(COLUMN_ADD(COLUMN_CREATE(1 , 'blue' as char), 2, 'ttt'), i as char)) persistent, > +item_name varchar(32) primary key, -- A common attribute for all items > +i int, > +dynamic_cols blob -- Dynamic columns will be stored here > +); > +INSERT INTO t1(item_name, dynamic_cols, i) VALUES > +('MariaDB T-shirt', COLUMN_CREATE(1, 'blue', 2, 'XL'), 1); > +INSERT INTO t1(item_name, dynamic_cols, i) VALUES > +('Thinkpad Laptop', COLUMN_CREATE(1, 'black', 3, 500), 2); > +select item_name, color, cl from t1; > +item_name color cl > +MariaDB T-shirt blue blue > +Thinkpad Laptop black ttt > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `color` char(32) AS (COLUMN_GET(dynamic_cols, 1 as char)) PERSISTENT, > + `cl` char(32) AS (COLUMN_GET(COLUMN_ADD(COLUMN_CREATE(1 , 'blue' as char), 2, 'ttt'), i as char)) PERSISTENT, > + `item_name` varchar(32) NOT NULL, > + `i` int(11) DEFAULT NULL, > + `dynamic_cols` blob, > + PRIMARY KEY (`item_name`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +drop table t1; > +create table t1 ( > +n int, > +c char(32) as (convert(cast(n as char), char)) persistent > +); > +insert into t1(n) values (1),(2),(3); > +select * from t1; > +n c > +1 1 > +2 2 > +3 3 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `n` int(11) DEFAULT NULL, > + `c` char(32) AS (convert(cast(n as char), char)) PERSISTENT > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +drop table t1; > > === modified file 'mysql-test/t/create.test' > --- a/mysql-test/t/create.test 2012-12-17 20:34:56 +0000 > +++ b/mysql-test/t/create.test 2014-11-08 18:54:42 +0000 > @@ -2021,3 +2021,35 @@ connection default; > select * from t1; > unlock tables; > drop table t1,t2; > + > +--echo # > +--echo # MDEV-6179: dynamic columns functions/cast()/convert() doesn't > +--echo # play nice with CREATE/ALTER TABLE > +--echo # > +create table t1 ( > + color char(32) as (COLUMN_GET(dynamic_cols, 1 as char)) persistent, > + cl char(32) as (COLUMN_GET(COLUMN_ADD(COLUMN_CREATE(1 , 'blue' as char), 2, 'ttt'), i as char)) persistent, > + item_name varchar(32) primary key, -- A common attribute for all items > + i int, > + dynamic_cols blob -- Dynamic columns will be stored here > +); > +INSERT INTO t1(item_name, dynamic_cols, i) VALUES > + ('MariaDB T-shirt', COLUMN_CREATE(1, 'blue', 2, 'XL'), 1); > +INSERT INTO t1(item_name, dynamic_cols, i) VALUES > + ('Thinkpad Laptop', COLUMN_CREATE(1, 'black', 3, 500), 2); > + > +select item_name, color, cl from t1; > +show create table t1; > + > +drop table t1; > + > +create table t1 ( > + n int, > + c char(32) as (convert(cast(n as char), char)) persistent > +); > +insert into t1(n) values (1),(2),(3); > + > +select * from t1; > +show create table t1; > + > +drop table t1; > > === modified file 'sql/sql_lex.h' > --- a/sql/sql_lex.h 2014-08-02 19:26:16 +0000 > +++ b/sql/sql_lex.h 2014-11-08 18:54:42 +0000 > @@ -105,6 +105,15 @@ struct sys_var_with_base > LEX_STRING base_name; > }; > > +struct LEX_TYPE > +{ > + enum enum_field_types type; > + char *length, *dec; > + CHARSET_INFO *charset; > + void set(int t, char *l, char *d, CHARSET_INFO *cs) > + { type= (enum_field_types)t; length= l; dec= d; charset= cs; } > +}; > + > #ifdef MYSQL_SERVER > /* > The following hack is needed because mysql_yacc.cc does not define > > === modified file 'sql/sql_yacc.yy' > --- a/sql/sql_yacc.yy 2014-08-02 19:26:16 +0000 > +++ b/sql/sql_yacc.yy 2014-11-08 18:54:42 +0000 > @@ -737,6 +737,7 @@ static bool add_create_index (LEX *lex, > LEX_STRING lex_str; > LEX_STRING *lex_str_ptr; > LEX_SYMBOL symbol; > + LEX_TYPE lex_type; > Table_ident *table; > char *simple_string; > Item *item; > @@ -1471,13 +1472,15 @@ bool my_yyoverflow(short **a, YYSTYPE ** > %type <string> > text_string opt_gconcat_separator > > +%type <lex_type> field_def > + > %type <num> > type type_with_opt_collate int_type real_type order_dir lock_option > udf_type if_exists opt_local opt_table_options table_options > table_option opt_if_not_exists opt_no_write_to_binlog > opt_temporary all_or_any opt_distinct > opt_ignore_leaves fulltext_options spatial_type union_option > - start_transaction_opts field_def > + start_transaction_opts > union_opt select_derived_init option_type2 > opt_natural_language_mode opt_query_expansion > opt_ev_status opt_ev_on_completion ev_on_completion opt_ev_comment > @@ -5483,11 +5486,11 @@ field_spec: > field_def > { > LEX *lex=Lex; > - if (add_field_to_list(lex->thd, &$1, (enum enum_field_types) $3, > - lex->length,lex->dec,lex->type, > + if (add_field_to_list(lex->thd, &$1, $3.type, > + $3.length, $3.dec, lex->type, > lex->default_value, lex->on_update_value, > &lex->comment, > - lex->change,&lex->interval_list,lex->charset, > + lex->change, &lex->interval_list, $3.charset, > lex->uint_geom_type, > lex->vcol_info, lex->option_list)) > MYSQL_YYABORT; > @@ -5495,13 +5498,15 @@ field_spec: > ; > > field_def: > - type opt_attribute {} > - | type opt_generated_always AS '(' virtual_column_func ')' > - vcol_opt_specifier > - vcol_opt_attribute > - { > - $$= (enum enum_field_types)MYSQL_TYPE_VIRTUAL; > - Lex->vcol_info->set_field_type((enum enum_field_types) $1); > + type opt_attribute > + { $$.set($1, Lex->length, Lex->dec, Lex->charset); } > + | type opt_generated_always AS > + { $<lex_type>$.set($1, Lex->length, Lex->dec, Lex->charset); } > + '(' virtual_column_func ')' vcol_opt_specifier vcol_opt_attribute > + { > + $$= $<lex_type>4; > + Lex->vcol_info->set_field_type($$.type); > + $$.type= (enum enum_field_types)MYSQL_TYPE_VIRTUAL; > } > ; > > @@ -6892,11 +6897,11 @@ alter_list_item: > { > LEX *lex=Lex; > if (add_field_to_list(lex->thd,&$3, > - (enum enum_field_types) $5, > - lex->length,lex->dec,lex->type, > + $5.type, > + $5.length, $5.dec, lex->type, > lex->default_value, lex->on_update_value, > &lex->comment, > - $3.str, &lex->interval_list, lex->charset, > + $3.str, &lex->interval_list, $5.charset, > lex->uint_geom_type, > lex->vcol_info, lex->option_list)) > MYSQL_YYABORT; > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] MDEV-6179: dynamic columns functions/cast()/convert() doesn't play nice with CREATE/ALTER TABLE in file:///home/bell/maria/bzr/work-maria-5.5-MDEV-6179/
by Sergei Golubchik 08 Nov '14

08 Nov '14

Hi, Sanja! On Nov 02, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-5.5-MDEV-6179/ > > ------------------------------------------------------------ > revno: 4341 > revision-id: sanja(a)askmonty.org-20141102133114-v46li6r2goyzhhp7 > parent: sanja(a)askmonty.org-20141031130729-dya4hci39x6zrv89 > committer: sanja(a)askmonty.org > branch nick: work-maria-5.5-MDEV-6179 > timestamp: Sun 2014-11-02 14:31:14 +0100 > message: > MDEV-6179: dynamic columns functions/cast()/convert() doesn't play > nice with CREATE/ALTER TABLE > > Type attributes (length, dec, charset) which parsed to global > variables now saved and restored during prsing types which can > appeared in a expression. Why did you create a stack in LEX, when bison generated parser already has a stack of states? Also, there's no need to store that in LEX, it doesn't need to live longer than the field_spec rule. By coincidence I've spent a couple of last days rewriting this very part of the parser. I've also fixed this bug without knowing it :) But in 10.1, and the patch is too big to put it in 10.0, let alone 5.5, so it's useless here. Still, I tried to create a smaller patch for this bug in 5.5. Please, review, it's on the commit list. Regards, Sergei

1 0

Re: [Maria-developers] MariaDB 10.0.14 got rid of replication safety warning without warning in release notes?
by Pavel Ivanov 08 Nov '14

08 Nov '14

And here's response from Monty to my last email. And this response together with the whole thread makes me really worried. It alludes to the fact that all MySQL and MariaDB developers before making any changes should read the whole codebase first and then make sure that their changes work the same way and that they conform to the intentions of original code developers even though there's no way they can talk to those original developers. AFAIK that's not how development on big projects with hundreds of participants works. And it really raises the question for me: what kind of quality the product that is developed with such principles in mind can have? Sergei Golubchik, I believe you are the technical leader of MariaDB project. Could you please tell me what you think about this issue? On Tue, Nov 4, 2014 at 6:32 AM, Michael Widenius <michael.widenius(a)gmail.com> wrote: > > Hi! > >>>>>> "Pavel" == Pavel Ivanov <pivanof(a)google.com> writes: > > Pavel> On Sun, Nov 2, 2014 at 3:27 PM, Michael Widenius <monty(a)mariadb.org> wrote: >>>>> This is the same order that keys are presented in SHOW CREATE TABLE >>> > Pavel> Why is it safe to assume that keys have the same order on master and on slave? >>> >>> Primary and Unique keys should always be ordered the same way on >>> master and slave. This is handled by the 'sort_keys()' function that >>> you can find at: sql/sql_table.cc, line 2770. > > Pavel> This function seem to sort only by "type" of the key. If several > Pavel> unique keys have the same "type" according to its classification then > Pavel> they will be in the "original order" as the comment in the function > Pavel> says. And the original order may be different. > > This function ensures that the table will be identical on slave and > master as long as one of the following requirements are fulfilled: > > - You are using the same create table and alter table statements on > both machines. > - You are using the same CREATE TABLE statement as 'show create' shows > on the master. > - You are adding unique keys in the same order on master and slave. > > This is always the case when you are using the replication that MySQL > and MariaDB uses. > > If you are creating tables differently on the slave than on the > master, then you are on your own. > > So in all normal and most abnormal cases, the table will be created > identically on the master and slave, even if the slave is using a > different storage engine, have different columns or have additional > keys than on the master. > > <cut> > >>> I added a test for this to: >>> mysql-test/t/insert.test >>> >>> --echo # >>> --echo # MDEV-5168: Ensure that we can disable duplicate key warnings >>> --echo # from INSERT IGNORE >>> --echo # > > Pavel> This test doesn't check which duplicate key is asserted to execute ON > Pavel> DUPLICATE KEY action. > > This test verifies that we don't get warnings for the tests. > Elinas test verifies which key is used. > >>> Elena added another test for this in >>> mysql-test/suite/storage_engine/insert_with_keys.test >>> >>> See line 142, that start with: >>> --let $create_definition = a $int_indexed_col UNIQUE KEY, b $int_indexed_col UNIQUE KEY, c $int_col > > Pavel> This test seem to check that UPDATE is executed on the correct > Pavel> duplicate key, but it's not executed during regular mtr run. How and > Pavel> when it's executed in MariaDB? > > The storage engine tests are to be used by people that implements > their own storage engine for MariaDB. > > We run the extended tests from time to time to verify storage engines. > We will also run it on some buildbot servers when running releae builds. > The storage engine tests is not something that has to be run for > normal builds. > >>> What is important is that all storage engine should work the same >>> way. If one returns something different compared to any of the >>> existing engines then it's a bug. >>> >>> In this case the implicit documentation is how MyISAM, Aria and InnoDB >>> works. All of them returns the first conflicting key. > > Pavel> I'm sorry to say, but you've explained exactly the reason why this > Pavel> behavior is not guaranteed and can be changed at any time. > > It can only be changed if someone adds new code that breaks the > storage engine interface, as defined by existing engines and one > doesn't check it with the storage engine test suite. > > Pavel> Yes, I > Pavel> understand that when you wrote the interface you had some particular > Pavel> requirements in mind. But ultimately it's up to people actively > Pavel> working on the code (who I believe are counted by hundreds now) to > Pavel> decide whether to maintain the particular behavior or change > it. > > It's the MariaDB captains that are responsible to maintain the > interface. The only way one can change something is to propose a > change and get it accepted. > > It's not free for anyone to just change old functionality and expect > that to be accepted in the standard version. > > In all cases, including this, it's an absolute requirement that > engines MUST work exactly as existing engines, so that a user can > switch engines and expect them to work identically. > > (The only exceptions are things that are impossible to fix, like > the differences between transactional and not transactional tables and > that there is no defined row order in tables). > > If engine writes don't ensure this, then you will get data corruption > on the server by just running with different engines. > > For case on insert on duplicate key, there is no acceptable reason > for engines to work differently. > > Pavel> And > Pavel> as nobody knows the original requirements. > > The original requirements was well defined. Just that non one bother > to ask doesn't mean that the requirement didn't exist. > > And the fact that all existing engines in MariaDB ensures this shows > that the requirements has been understood. > > In the case when we notice that two engines works differently, it's up > to the MariaDB captains to fix this! > The fix is NOT to extend the API and allow things to work differently. > > > Pavel> and the reasons why they > Pavel> can't be changed, and when the behavior is not tested anywhere > > We add new tests when we notice that they are needed, as we did in > this case. > For example, the case with INSERT ... ON DUPLICATE KEY is now tested. > > Pavel> and > Pavel> doesn't have any documentation even in the weakest form of comments or > Pavel> asserts, the decision is usually very easy -- change at will, nobody > Pavel> should ever depend on implementation details. > > Yes, no one should depend in implementation details, except when the > implementation detail is different from other engines and it results > in different data. > > If that is the case then the engine that does things differently than > the existing engines must be changed. > > Pavel> Even if you promise to > Pavel> fix storage engines that change this behavior, by the time you notice > Pavel> the change you'll likely won't be able to fix it (as e.g. in case of > Pavel> https://mariadb.atlassian.net/browse/MDEV-6916) or someone will > Pavel> already experience a data corruption. > > In most cases we can fix this. What was unfortunate with the views > code was that we didn't store the MariaDB version number in the view > definition. This is to be fixed in 10.1 so that this will never happen > again! > > Pavel> I'd strongly recommend to revert this change. > > Sorry, but I can't agree to revert the change because: > > a) Something that can't ever happen in MariaDB (when used correctly > with the including engines). > b) Allowing a behavior that could for the same command result in > different data sets, depending on the storage engine. > c) I have not seen a real world > > One of my main jobs is to ensure that the storage engine API is > evolved in a resonable manner and if there is a conflict between how > storage engine works ensure that things are resolved so that the user > can expect the same results independent on the storage engine. > > Reverting the change would go directly agains the above and also > against what I belive is the future of the storage engine API. > > Regards, > Monty

2 1

Re: [Maria-developers] MariaDB 10.0.14 got rid of replication safety warning without warning in release notes?
by Pavel Ivanov 07 Nov '14

07 Nov '14

Monty, can you please subscribe all your email address to maria-develoeprs@ ? People reading the list cannot see your emails. I'm forwarding here the unaltered message from Monty that I was answering to in my last email. On Sun, Nov 2, 2014 at 3:27 PM, Michael Widenius <monty(a)mariadb.org> wrote: > > Hi! > > I have again to apologize for a late reply. > After my last reply to your topic I traveled to China and noticed > that the firewall there has got much better; You can't access google, > twitter, facebook, blogspot etc :( I tried to connect over a VPN, but > that didn't work for me either. > > It did feel a bit strange to not be able to do computer work for more > than a week, but the lovely people in China who took care of me did > compensate for that. > >>>>>> "Pavel" == Pavel Ivanov <pivanof(a)google.com> writes: > > Pavel> Which key is "the first duplicate key". >>> >>> The first key that has a duplicate value in the order of keys in the KEY >>> structure that MariaDB provides for the storage engine. > > Pavel> I'm definitely not that familiar with the relevant MariaDB code, but I > Pavel> didn't find KEY structure to be passed to any insert-related function. > Pavel> Can you elaborate (preferably with code references) how this > Pavel> restriction is supposed to work? > > The following code examples are from 10.0. > > All handler functions get access to the TABLE object, which defines > the record format and the keys. > > The keys are stored in TABLE->key_info (see table.h, line 1062) > > When one reads a key, one refers to keys according to the number in > the key_info structure. For example: > > A set of reads by key is done by calling: > > handler->ha_index_init(key_number,..); > read_until_no_more_rows > handler->ha_index_read_map(). > handler->ha->index_end() > > When one calls handler->info() to ask for a which key caused a > duplicate error, it's supposed to store the first duplicate key (key > with smallest number) in handler->errkey. > >>> This is the same order that keys are presented in SHOW CREATE TABLE > > Pavel> Why is it safe to assume that keys have the same order on master and on slave? > > Primary and Unique keys should always be ordered the same way on > master and slave. This is handled by the 'sort_keys()' function that > you can find at: sql/sql_table.cc, line 2770. > > The only case when this is not the case, is if you add an unique key > on the slave that was not on the master and there is conflicts when > you insert things because of this key. > > However, if you do this, I would say that you are on your own. > >>> To help engine writes we are constantly adding new test in the storage >>> engine test suite, to make it easier for storage engine vendors to >>> test that their storage engine follows the interface. >>> We have now added a test to check for the correct behaviour for the >>> case of INSERT ... ON DUPLICATE KEY. > > Pavel> This is very interesting. Can you say which test it is and what > Pavel> exactly does it test? Is it present in 10.0.14 or it's only at the > Pavel> latest 10.0 branch? > > I added a test for this to: > mysql-test/t/insert.test > > --echo # > --echo # MDEV-5168: Ensure that we can disable duplicate key warnings > --echo # from INSERT IGNORE > --echo # > > Elena added another test for this in > mysql-test/suite/storage_engine/insert_with_keys.test > > See line 142, that start with: > --let $create_definition = a $int_indexed_col UNIQUE KEY, b $int_indexed_col UNIQUE KEY, c $int_col > >>> Do you have or use a storage engine that doesn't provide the expected value >>> for the conflicting key ? > > Pavel> I don't have a specific example of storage engine that detects > Pavel> duplicate key in different order, but I'm not convinced and don't have > Pavel> hard proof that such engine cannot exist and that InnoDB itself is not > Pavel> susceptible to this problem. I can easily imagine though that even > Pavel> given the particular order of keys in table definition the key > Pavel> reported as having duplicate can depend on the way storage engine > Pavel> stores those keys. > > As key order is defined on the upper level, above InnoDB, this should > be 100 % safe (at least for InnoDB). > > For other store engines, this is safe for any engine that test for > duplicate key in key order. > > If we ever find a case where it's not safe with any storage engine > that we deliver with MariaDB, we will fix the storage engine. > > It's ok if they do thing differently so that they can detect duplicate > key for other keys faster. > > However, it has been required from all storage engines from start that > if one calls 'handler->info()' to request which key was duplicated, > they must return the first key. > > If not, they are breaking the storage engine interface (both for MySQL > and MariaDB) as there is no way the upper level can guarantee that > things will work identically between storage engines. > > Pavel> BTW, can you point to MySQL documentation and/or comment in the code > Pavel> that indicates this requirement to storage engine implementation? > > Most of the storage engine interface is not 100 % documented. This is > one of the case. > However, I wrote the code related to errkey and this is how it was > supposed to work. > In general if something is not documented, then it's up to the people > that created the interface or are actively working on it to define how > it should work. > > What is important is that all storage engine should work the same > way. If one returns something different compared to any of the > existing engines then it's a bug. > > In this case the implicit documentation is how MyISAM, Aria and InnoDB > works. All of them returns the first conflicting key. > > Pavel> On the related note Release Notes mention the error log flood > Pavel> protection. Why it's not behind flag? How should I disable the feature > Pavel> and get back to the old behavior? > >>> We already had in MariaDB flood protection for some error >>> messages. > > Pavel> Can you say exactly which error messages are suppressed already? Is > Pavel> there a documentation link about that? > > All 'unsafe logging' errors are suppressed. > See enum enum_binlog_stmt_unsafe in sql_lex.h > (This is what I know). > >>> I am happy to look into adding a flag that is usable for any kind of >>> suppressed messages. >>> >>> A couple of ways to do this: >>> >>> a) Not suppress anything if you give --log-warnings a value > 2 >>> b) make LIMIT_UNSAFE_WARNING_ACTIVATION_TIMEOUT a configurable >>> variable, where 0 would mean no suppression. >>> >>> Which of the above, or both, would you prefer? >>> Or do you have any other suggestion of how to do this? > > Pavel> Reading documentation on --log-warnings I don't see how it can be > Pavel> related to the error suppression. So I think making > Pavel> LIMIT_UNSAFE_WARNING_ACTIVATION_TIMEOUT configurable would be much > Pavel> better, though it should apply to all possible error suppressions. > > Currently we only have suppression for a limited set of 'unsafe' > errors, but I agree that when we add a flag we should name this with > a general name to be able to reuse it for all possible future > suppressions. > > In 10.1 we are looking at adding a feature that one can suppress any > error from the error log, but that's a different topic. > > Regards, > Monty

1 0

[Maria-developers] Maybe a new information schema table for replication?
by Roberto Spadim 06 Nov '14

06 Nov '14

Hi guys, i got one question these days and i don't know how to solve... the question is: "How to see the queries executing by replication thread?" i'm thinking about a information schema plugin to see whats is 'queued' in replication thread, at "master" (write to binlog) and "slave" (executing from binlog) side any idea if this is possible? i didn't checked or read any part of binlog/replication code yet thanks guys! :) -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

1 0

[Maria-developers] pcre.h should be installed if MariaDB is built with bundled pcre ?
by Kazuhiko Shiozaki 06 Nov '14

06 Nov '14

Hi, pcre.h is not installed by 'make install' even if MariaDB is built with bundled pcre. But pcre.h is referred in one of installed header files like : include/mysql/private/item_cmpfunc.h:#include "pcre.h" /* pcre header file */ So I think that it is better to install pcre.h as well if MariaDB is built with bundled pcre. What do you think ? -- Kazuhiko Shiozaki, Nexedi SA Senior Consultant Nexedi: Consulting and Development of Free / Open Source Software http://www.nexedi.com/ ERP5: Full Featured High End Open Source ERP http://www.erp5.com/

2 1

[Maria-developers] Fwd: Is it soon 'end of this year'?
by Peter Laursen 04 Nov '14

04 Nov '14

Forwarding to MariaDB developers mailing list. It may be a matter for somebody else than Sergei (Georg, for instance) (and "here" means"Webyog"). Abishek detailed errors in a chat with me "symbols like Snippet mysql_client_find_plugin , Snippet mysql_client_register_plugin are not found" ..exactly like in the MySQL bug report. Do you need a JIRA report with a full test case? -- Peter ---------- Forwarded message ---------- From: Abhishek Kumar Pandey <abhishek.pandey(a)webyog.com> Date: Tue, Nov 4, 2014 at 12:51 PM Subject: Re: [Maria-developers] Is it soon 'end of this year'? To: Peter Laursen <peter_laursen(a)webyog.com> Cc: "team. sqlyog" <team.sqlyog(a)webyog.com>, serg(a)mariadb.org, "team.monyog" <team.monyog(a)webyog.com> I am the developer here working with this. I am facing same problem as reported here- http://bugs.mysql.com/bug.php?id=69204 <http://www.google.com/url?q=http%3A%2F%2Fbugs.mysql.com%2Fbug.php%3Fid%3D69…> . Is the same fix needed in the mariadb library? On Tue, Oct 14, 2014 at 6:58 PM, Peter Laursen <peter_laursen(a)webyog.com> wrote: > You will get to this page: https://edelivery.oracle.com/ > > I think you should use a non-webyog email address. Not sure what Mahesh > did. He downloaded frequently in order to check latest MEM. > > > --Peter > > On Tue, Oct 14, 2014 at 3:26 PM, Peter Laursen <peter_laursen(a)webyog.com> > wrote: > >> You could also try to add the code as described by Sergei. First step is >> if it will compile and connect to servers not usinglain text/LDAP. >> >> -- Peter >> >> On Tue, Oct 14, 2014 at 3:12 PM, Supratik Goswami < >> supratik.goswami(a)webyog.com> wrote: >> >>> On Tue, Oct 14, 2014 at 4:58 PM, Sudhindra Bhat < >>> sudhindra.bhat(a)webyog.com> wrote: >>> >>>> +supratik >>>> >>>> We will check this once we have access to MySQL Enterprise server from >>>> Oracle. >>>> >>>> @supratik, >>>> Do we have the binary for MySQL Enterprise server? A Trial is also >>>> sufficient. >>>> >>> >>> No, I think. You may download it. >>> >>>> >>>> On Tue, Oct 14, 2014 at 2:52 PM, Peter Laursen < >>>> peter_laursen(a)webyog.com> wrote: >>>> >>>>> We need to try this again. I don't rememebr who gave up supporting >>>>> 'clear text plugin' with MariaDB API back in April 2013. Probably it was >>>>> not Sudhindra, Likely Shubhansh or Vishal. @Vishal: do you remember >>>>> anything? >>>>> >>>>> I remember one detail from the old discussion we had with MariaDB. >>>>> They told something like "Once enabled nothing special need to be done to >>>>> support clear text. The client wll automatically detect when the server >>>>> requests the password sent as clear text". This mean as far as I can >>>>> understand that we will - unlke early 11.x version - not need a checkbox in >>>>> the connection manager. >>>>> >>>>> We also need access to a commercial server (TRIAL) from Oracle. Mahesh >>>>> could arrange that. Who can today? >>>>> >>>>> >>>>> -- Peter >>>>> >>>>> >>>>> ---------- Forwarded message ---------- >>>>> From: Sergei Golubchik <serg(a)mariadb.org> >>>>> Date: Mon, Oct 13, 2014 at 11:24 PM >>>>> Subject: Re: [Maria-developers] Is it soon 'end of this year'? >>>>> To: Peter Laursen <peter_laursen(a)webyog.com> >>>>> >>>>> >>>>> Hi, Peter! >>>>> >>>>> On Oct 13, Peter Laursen wrote: >>>>> > Thanks .. you probabaly know that I am not a developer rmyself (I >>>>> basically >>>>> > stopped being so when ALGOL died 35 years ago!). I have shared all >>>>> > information with SQLyog developer team and we will look into this >>>>> again. >>>>> >>>>> Sure! Don't hesitate to tell if something won't work for you. >>>>> >>>>> Regards, >>>>> Sergei >>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Sudhindra Bhat >>>> >>> >>> >> >

2 1

Re: [Maria-developers] MariaDB 10.0.14 got rid of replication safety warning without warning in release notes?
by Pavel Ivanov 03 Nov '14

03 Nov '14

On Sun, Nov 2, 2014 at 3:27 PM, Michael Widenius <monty(a)mariadb.org> wrote: >>> This is the same order that keys are presented in SHOW CREATE TABLE > > Pavel> Why is it safe to assume that keys have the same order on master and on slave? > > Primary and Unique keys should always be ordered the same way on > master and slave. This is handled by the 'sort_keys()' function that > you can find at: sql/sql_table.cc, line 2770. This function seem to sort only by "type" of the key. If several unique keys have the same "type" according to its classification then they will be in the "original order" as the comment in the function says. And the original order may be different. > The only case when this is not the case, is if you add an unique key > on the slave that was not on the master and there is conflicts when > you insert things because of this key. > > However, if you do this, I would say that you are on your own. > >>> To help engine writes we are constantly adding new test in the storage >>> engine test suite, to make it easier for storage engine vendors to >>> test that their storage engine follows the interface. >>> We have now added a test to check for the correct behaviour for the >>> case of INSERT ... ON DUPLICATE KEY. > > Pavel> This is very interesting. Can you say which test it is and what > Pavel> exactly does it test? Is it present in 10.0.14 or it's only at the > Pavel> latest 10.0 branch? > > I added a test for this to: > mysql-test/t/insert.test > > --echo # > --echo # MDEV-5168: Ensure that we can disable duplicate key warnings > --echo # from INSERT IGNORE > --echo # This test doesn't check which duplicate key is asserted to execute ON DUPLICATE KEY action. > Elena added another test for this in > mysql-test/suite/storage_engine/insert_with_keys.test > > See line 142, that start with: > --let $create_definition = a $int_indexed_col UNIQUE KEY, b $int_indexed_col UNIQUE KEY, c $int_col This test seem to check that UPDATE is executed on the correct duplicate key, but it's not executed during regular mtr run. How and when it's executed in MariaDB? >>> Do you have or use a storage engine that doesn't provide the expected value >>> for the conflicting key ? > > Pavel> I don't have a specific example of storage engine that detects > Pavel> duplicate key in different order, but I'm not convinced and don't have > Pavel> hard proof that such engine cannot exist and that InnoDB itself is not > Pavel> susceptible to this problem. I can easily imagine though that even > Pavel> given the particular order of keys in table definition the key > Pavel> reported as having duplicate can depend on the way storage engine > Pavel> stores those keys. > > As key order is defined on the upper level, above InnoDB, this should > be 100 % safe (at least for InnoDB). > > For other store engines, this is safe for any engine that test for > duplicate key in key order. > > If we ever find a case where it's not safe with any storage engine > that we deliver with MariaDB, we will fix the storage engine. > > It's ok if they do thing differently so that they can detect duplicate > key for other keys faster. > > However, it has been required from all storage engines from start that > if one calls 'handler->info()' to request which key was duplicated, > they must return the first key. > > If not, they are breaking the storage engine interface (both for MySQL > and MariaDB) as there is no way the upper level can guarantee that > things will work identically between storage engines. > > Pavel> BTW, can you point to MySQL documentation and/or comment in the code > Pavel> that indicates this requirement to storage engine implementation? > > Most of the storage engine interface is not 100 % documented. This is > one of the case. > However, I wrote the code related to errkey and this is how it was > supposed to work. > In general if something is not documented, then it's up to the people > that created the interface or are actively working on it to define how > it should work. > > What is important is that all storage engine should work the same > way. If one returns something different compared to any of the > existing engines then it's a bug. > > In this case the implicit documentation is how MyISAM, Aria and InnoDB > works. All of them returns the first conflicting key. I'm sorry to say, but you've explained exactly the reason why this behavior is not guaranteed and can be changed at any time. Yes, I understand that when you wrote the interface you had some particular requirements in mind. But ultimately it's up to people actively working on the code (who I believe are counted by hundreds now) to decide whether to maintain the particular behavior or change it. And as nobody knows the original requirements and the reasons why they can't be changed, and when the behavior is not tested anywhere and doesn't have any documentation even in the weakest form of comments or asserts, the decision is usually very easy -- change at will, nobody should ever depend on implementation details. Even if you promise to fix storage engines that change this behavior, by the time you notice the change you'll likely won't be able to fix it (as e.g. in case of https://mariadb.atlassian.net/browse/MDEV-6916) or someone will already experience a data corruption. I'd strongly recommend to revert this change. Thank you, Pavel

1 0

[Maria-developers] MDEV-5528 Command line variable to choose MariaDB-5.3 vs MySQL-5.6 temporal data formats
by Alexander Barkov 03 Nov '14

03 Nov '14

Hello Ian, I've pushed MDEV-5528 into 10.1. I guess it will need some documenting. Can you please have a look? Thanks.

1 0

[Maria-developers] nginx sources in mroonga
by Alexander Barkov 03 Nov '14

03 Nov '14

Hi Kentoku, Sergei Golubchik is asking why we have complete nginx sources inside mroonga? Can you remove them please? Thanks.

3 2

[Maria-developers] Cross-compiling MariaDB galera cluster
by Sylvain Raybaud 30 Oct '14

30 Oct '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear MariaDB community, Recently I've been trying to cross-compile MariaDB and I've been running in some problems. I'd like to have your opinion on them. 1) Determining stack direction 2) Building Percona XtraDB 3) Building jemalloc Let me first tell that I'm trying to build MariaDB galera version 10.0.12. I'm compiling for Arm but may be compiling for something else in the unpredictable future. 1) Determining stack direction I started by passing the following options to cmake (via the buildroot system): - -DCMAKE_CROSSCOMPILING=1 -DWITH_WSREP=1 -DWITH_INNODB_DISALLOW_WRITES=1 The first problem I ran into was cmake complaining: STACK_DIRECTION is not defined. Please specify -DSTACK_DIRECTION=1 or -DSTACK_DIRECTION=-1 when calling cmake. It seems that STACK_DIRECTION is not automatically determined when cross-compiling. I may determine the correct value for a given target. But if I need a portable solution (and I need one in order to create a buildroot package so that I can build for many different target without too much hassle) I'm not sure what my options are. It seems that the toolchain is aware of this value since stack direction (or more precisely STACK_GROWS_DOWNWARD) is set in ./output/host/opt/ext-toolchain/lib/gcc/arm-none-linux-gnueabi/4.8.1/plugin/include/config/arm/arm.h Maybe there is a way to use this somehow? What do you think? 2) Building Percona XtraDB For the sake of trying I explicitly set -DSTACK_DIRECTION=1 just to see what happens. Building goes on until it reaches the xtradb module. Then it fails with: CMake Error at storage/xtradb/CMakeLists.txt:434 (MESSAGE): Percona XtraDB is not supported on this platform (Here I'll just copy-paste something I wrote on maria-discuss list a few days ago) This error is triggered in storage/xtradb/CMakeLists.txt when XTRADB_OK is not set; it seems to me that XTRADB_OK is set in either one of the following cases: a) CMAKE_CROSSCOMPILING is NOT set and some tests are successful (thus setting HAVE_IB_GCC_ATOMIC_BUILTINS) or b) We're compiling for MSVC or CMAKE_SYSTEM_NAME is "SunOS" and some other conditions are met. Since I'm not compiling for MSVC or SunOS and I'm crosscompiling indeed, XTRADB_OK is not set and the error is triggered. Here I'm stuck. I've got the feeling that as such one cannot cross-compile xtradb because XTRADB_OK will never be set; or maybe with proper options passed to cmake the tests would be made elsewhere and HAVE_IB_GCC_ATOMIC_BUILTINS would be set beforehand? Or maybe I should patch this CMakeLists.txt? Or maybe I should just compile without xtradb altogether, if that's possible? Finally I just passed -DXTRADB_OK just to see what would happen, again (obviously there is little chance that it would result in anything usable). And reached problem #3: 3) Building jemalloc This one fails at configure time with: checking whether we are cross compiling... configure: error: in `/home/sylvain/DEPOTS/buildroot/output/build/mariadb-galera-10.0.12/extra/jemalloc/build': configure: error: cannot run C compiled programs. If you meant to cross compile, use `--host'. Somehow I was expecting that having passed -DCMAKE_CROSSCOMPILING=1 and the host being set by buildroot it would propagate down to the modules... Am I missing something? At this point I'm out of dirty hacks, even to "see what happens" ;) Cheers, - -- Sylvain Raybaud www.green-communications.fr -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJT4ke/AAoJEEkkwl4JtJ9yJW0QALBz3wj0VsfMtUBUabP13q74 ++y1AiCLZ0js+UGCqxB9PVz4yvhhnMAz8QnPpH37XJtDm+VaJBQTEZL65CHF3nlr dHe46/q/wswlbdhNsQu6JmMe3JoOe2TFbjDenQMiUvW9mOilt86ioveHVLGYbpnq 5LYMrPlYXUBpDcxEYXXbTpIHsAoHUrOE1Gj5AsxwbUvQuHDDr9M/Ws01iIXLbzT8 0pImhgBPYUwqaj9tJnPkls+9LzoOb2m269n9TCbmWk0ixakpHjeNMn5lnRDhOwNh AKkHrGdMUomSBca4G0kQoeNxwC2rCjO8+bRxjh7v6GTdME/sgsgn3yOv2EznldUr tFaO/JfraHSUgmiRyVo41U3uTESxZKRJYOV9IqUhl4HiS85D5oa/bD6l84rhx95J 52/hUoApoStr8tSk0tzXk1/UWJEQ3fUZ7XoXqgWq+c1uMgbgZftAAqAw7Xvmc88t tf69FqBzw5dpf8Kg0I7iAggBpBLIWGJwnlVvcmLY4UUE1cDFfZstMBSXaS155LQF VClK+ytE2Ef8oAB9/nlMXgg4b3WXlD0PRQDRTNIwCAzdvsuu+S0q64xLtw/JYnLl KPcLWfwXy+kRhT+vtUWtqMjjkSQ3e6rJDybK11vgYIAWm/Xfk5s8i0g7Bgmnf3wp JGYUAR285gY0XMqL6LmU =vBo4 -----END PGP SIGNATURE-----

2 4

[Maria-developers] Valgrind failure in Buildbot
by Kristian Nielsen 30 Oct '14

30 Oct '14

Hi Sergey, We briefly discussed the new valgrind error in buildbot after your push on IRC: https://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/… ==13161== Conditional jump or move depends on uninitialised value(s) ==13161== at 0x4C280AA: memcpy (mc_replace_strmem.c:115) ==13161== by 0x723F6B: Field_varstring::get_key_image(unsigned char*, unsigned int, Field::imagetype) (field.cc:6932) ==13161== by 0x818D5B: key_copy(unsigned char*, unsigned char*, st_key*, unsigned int, bool) (key.cc:146) ==13161== by 0x6DD6FE: Mrr_ordered_index_reader::interrupt_read() (multi_range_read.cc:448) ==13161== by 0x6DE605: Mrr_ordered_rndpos_reader::refill_from_index_reader() (multi_range_read.cc:688) ==13161== by 0x6DE7D2: Mrr_ordered_rndpos_reader::refill_buffer(bool) (multi_range_read.cc:618) ==13161== by 0x6DFB27: DsMrr_impl::dsmrr_init(handler*, st_range_seq_if*, void*, unsigned int, unsigned int, st_handler_buffer*) (multi_range_read.cc:966) ==13161== by 0x91AAE9: ha_innobase::multi_range_read_init(st_range_seq_if*, void*, unsigned int, unsigned int, st_handler_buffer*) (ha_innodb.cc:13961) ==13161== by 0x6DA60A: JOIN_TAB_SCAN_MRR::open() (sql_join_cache.cc:3867) ==13161== by 0x6D7363: JOIN_CACHE::join_matching_records(bool) (sql_join_cache.cc:2249) ==13161== by 0x6D5647: JOIN_CACHE::join_records(bool) (sql_join_cache.cc:2086) ==13161== by 0x5F8259: sub_select_cache(JOIN*, st_join_table*, bool) (sql_select.cc:16717) ==13161== by 0x6D71BA: JOIN_CACHE::generate_full_extensions(unsigned char*) (sql_join_cache.cc:2397) ==13161== by 0x6D74E6: JOIN_CACHE::join_matching_records(bool) (sql_join_cache.cc:2289) ==13161== by 0x6D5647: JOIN_CACHE::join_records(bool) (sql_join_cache.cc:2086) ==13161== by 0x5F8219: sub_select_cache(JOIN*, st_join_table*, bool) (sql_select.cc:16697) I took a quick look. The problem seems to be that the code is copying out the fields of a key from table->record[0] (key_copy()), however, the first length byte of the value is uninitialised. I think it's the field col_varchar_key in the table, testcase is appended below. I checked a bit back up the callstack. The field col_varchar_key is reported as uninitialised already at the start of Mrr_ordered_rndpos_reader::refill_from_index_reader(), but I'm not really familiar with when and how table->record[0] is supposed to be initialised during multi-range read. As you said on IRC, your recent patch is related to interrupt_read(), so probably best if you take a look and see if you can figure out something. Incidentally, I noticed something that looks wrong in key_copy(to_key, from_record, key_info, key_length, with_zerofill) The comments say that the key is copied from `from_record', however actually the function only works if the from_record passed in is table->record[0] (or more precisely, the same record as is installed in key_info->key_part->field). Because it checks the null bit in from_record, but copies the data from table->record[0]. However, this is not the problem in this particular failure (as in this case from_record == table->record[0]), just struck me as something that could easily cause surprises later... Below is a reduced test case that triggers the valgrind error. It did not trigger for me with a debug build, but _did_ trigger (in 5.5) with this build command: EXTRA_FLAGS="-O3 -fno-omit-frame-pointer -Wno-uninitialized" AM_EXTRA_MAKEFLAGS="VERBOSE=1" BUILD/compile-amd64-valgrind-max - Kristian. ----------------------------------------------------------------------- -- source include/have_xtradb.inc set optimizer_switch='mrr=on,mrr_sort_keys=on,index_condition_pushdown=on'; CREATE TABLE t1 ( pk int(11) NOT NULL AUTO_INCREMENT, col_int_key int(11) NOT NULL, col_varchar_key varchar(1) NOT NULL, col_varchar_nokey varchar(1) NOT NULL, PRIMARY KEY (pk), KEY col_int_key (col_int_key), KEY col_varchar_key (col_varchar_key,col_int_key) ) ENGINE=InnoDB; INSERT INTO t1 VALUES (10,8,'v','v'), (11,8,'f','f'), (12,5,'v','v'), (13,8,'s','s'), (14,8,'a','a'), (15,6,'p','p'), (16,7,'z','z'), (17,2,'a','a'), (18,5,'h','h'), (19,7,'h','h'), (20,2,'v','v'), (21,9,'v','v'), (22,142,'b','b'), (23,3,'y','y'), (24,0,'v','v'), (25,3,'m','m'), (26,5,'z','z'), (27,9,'n','n'), (28,1,'d','d'), (29,107,'a','a'); CREATE TABLE t2 ( pk int(11) NOT NULL AUTO_INCREMENT, col_int_key int(11) NOT NULL, col_varchar_key varchar(1) NOT NULL, col_varchar_nokey varchar(1) NOT NULL, PRIMARY KEY (pk), KEY col_int_key (col_int_key), KEY col_varchar_key (col_varchar_key,col_int_key) ) ENGINE=InnoDB; INSERT INTO t2 VALUES (10,3,'i','i'), (11,186,'x','x'), (12,1,'g','g'), (13,8,'q','q'), (14,226,'m','m'), (15,133,'p','p'), (16,6,'e','e'), (17,3,'t','t'), (18,8,'j','j'), (19,5,'h','h'), (20,7,'w','w'); set join_cache_level=6; set join_buffer_size=1536; SELECT count(*), sum(table1.col_int_key*table2.pk) FROM t2 AS table1, t1 AS table2, t2 AS table3 WHERE table3.col_varchar_nokey = table2.col_varchar_key AND table3.pk > table2.col_varchar_nokey ; -----------------------------------------------------------------------

1 0

Re: [Maria-developers] Rev 4323: Set proper alternative names on RHEL7: they should match Fedora names
by Sergei Golubchik 30 Oct '14

30 Oct '14

Hi, Sergey! I cannot verify whether rhel7 alternative names should match Fedora names or they should be something different. Assuming they should match, your patch is ok. On Oct 09, Sergey Vojtovich wrote: > ------------------------------------------------------------ > revno: 4323 > revision-id: svoj(a)mariadb.org-20141009112748-eq3xwoyobgnz0cxj > parent: svoj(a)mariadb.org-20141008141031-xv2x7vrq269202vf > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5 > timestamp: Thu 2014-10-09 15:27:48 +0400 > message: > Set proper alternative names on RHEL7: they should match Fedora names. > === modified file 'cmake/cpack_rpm.cmake' > --- a/cmake/cpack_rpm.cmake 2014-10-08 07:35:00 +0000 > +++ b/cmake/cpack_rpm.cmake 2014-10-09 11:27:48 +0000 > @@ -182,7 +182,7 @@ IF(RPM MATCHES "(rhel|centos)5") > ELSEIF(RPM MATCHES "(rhel|centos)6") > ALTERNATIVE_NAME("client" "mysql") > ALTERNATIVE_NAME("shared" "mysql-libs") > -ELSEIF(RPM MATCHES "fedora") > +ELSEIF(RPM MATCHES "fedora" OR RPM MATCHES "(rhel|centos)7") > SET(epoch 1) # this is fedora > ALTERNATIVE_NAME("client" "mariadb") > ALTERNATIVE_NAME("client" "mysql") Regards, Sergei

2 2

[Maria-developers] Fwd: [debian-mysql] Backporting the mysql_no_login plugin
by Colin Charles 30 Oct '14

30 Oct '14

This seems to get a +1 for backporting by at least Honza (RH), so I am wondering if we do this in the 5.5 branch too, that is shipping in many distributions. Begin forwarded message: > From: "Norvald H. Ryeng" <norvald.ryeng(a)oracle.com> > Subject: [debian-mysql] Backporting the mysql_no_login plugin > Date: 24 October 2014 15:49:34 GMT+8 > To: "pkg-mysql-maint(a)lists.alioth.debian.org" <pkg-mysql-maint(a)lists.alioth.debian.org>, "Honza Horak" <hhorak(a)redhat.com>, "Roman Drahtmueller" <draht(a)suse.de> > > Hi package maintainers, > > We have a new plugin in MySQL 5.7 that makes it possible to have > accounts that can't log in: > > CREATE USER foo@localhost IDENTIFIED WITH 'mysql_no_login'; > > The mysql_no_login plugin simply denies all login attempts. This is > useful for users that are created, e.g., to serve as proxy users, or > as owners of stored programs/functions, views or events. > > This new plugin doesn't fix known security defects in the server, but > does provide new and better means to harden security. Best practices > for security include application of least-required privileges, and in > some cases, that means no client connections for privileged > accounts. This new plugin provides means to implement such > restrictions in a standard way. > > Because of the security benefits, we'd like to discuss backporting it > to 5.6. Like you, we don't like big changes to GA releases, but this > time we think it has a good use case, it's safe and has a very low > risk of regressions: > > - Since this is a plugin, it doesn't touch server code > - All new code is in a plugin that must be enabled explicitly by the > DBA > - The code itself is very simple. It's only one line of "real" code > (unconditionally return authentication failure), plus necessary > plugin plumbing to fill out the plugin API. > > If we backport this to 5.6, there are multiple ways to avoid it: > > - Apply a patch from us to remove the plugin > - Don't build it > - Build it, but don't ship it > - Build and ship it, but don't use it (in any case, the DBA has to > enable it and alter the user accounts to use it) > > So what do you think about backporting this? The only thing you'll > notice is one more file in the plugins directory. > > Regards, > > Norvald > > _______________________________________________ > pkg-mysql-maint mailing list > pkg-mysql-maint(a)lists.alioth.debian.org > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-mysql-maint -- Colin Charles, Chief Evangelist, MariaDB Corporation blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

2 1

Re: [Maria-developers] [Commits] b28e6bf: MDEV-6945 MDEV-6948 MDEV-6956 MDEV-6957 MDEV-6953 MDEV-6954 MDEV-6955
by Sergei Golubchik 29 Oct '14

29 Oct '14

Hi, Sanja! On Oct 29, sanja(a)mariadb.com wrote: > --- > mysql-test/r/set_statement.result | 14 ++++++++++++++ > mysql-test/r/set_statement_profiling.result | 4 ++++ > mysql-test/t/set_statement.test | 18 ++++++++++++++++++ > mysql-test/t/set_statement_profiling.test | 9 +++++++++ Did you put set_statement_debug.test in a separate commit? That's good. > sql/set_var.h | 4 +++- > sql/share/errmsg-utf8.txt | 2 ++ > sql/sql_parse.cc | 5 +++++ > sql/sys_vars.cc | 20 +++++++++++--------- > sql/sys_vars.h | 1 + > 9 files changed, 67 insertions(+), 10 deletions(-) > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index bb42807..1633169 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7113,3 +7113,5 @@ ER_STATEMENT_TIMEOUT 70100 > eng "Query execution was interrupted (max_statement_time exceeded)" > ER_SUBQUERIES_NOT_SUPPORTED 42000 > eng "%s does not support subqueries or stored functions." > +ER_SET_STATEMENT_NOT_SUPPORTED 42000 > + eng "Variable '%s' is not supported in SET STATEMENT." Use the text eng "The system variable %.200s cannot be set in SET STATEMENT." > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 6252e89..fd77e5f 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -997,7 +997,7 @@ static bool check_master_connection(sys_var *self, THD *thd, set_var *var) > static Sys_var_session_lexstring Sys_default_master_connection( > "default_master_connection", > "Master connection to use for all slave variables and slave commands", > - SESSION_ONLY(default_master_connection), > + NO_SET_STMT SESSION_ONLY(default_master_connection), Why should this be prohibited? I'd rather have it supported. Use case: setting many connection variables for a single master, like in SET STATEMENT default_master_connection=foobar SET max_relay_log_size=10000000, sql_slave_skip_counter=5; Admittedly, this use case, while valid, isn't a strong argument at the moment - we don't have many per-master variables yet. When we'll have more, this will become more important. So, if fixing default_master_connection is not easy - keep it NO_SET_STMT, it can be enabled later. Why, exactly, does it fail? > NO_CMD_LINE, IN_SYSTEM_CHARSET, > DEFAULT(""), MAX_CONNECTION_NAME, ON_CHECK(check_master_connection)); > #endif > @@ -3527,12 +3528,12 @@ static Sys_var_bit Sys_unique_checks( > #ifdef ENABLED_PROFILING > static Sys_var_bit Sys_profiling( > "profiling", "profiling", > - SESSION_VAR(option_bits), NO_CMD_LINE, OPTION_PROFILING, > + NO_SET_STMT SESSION_VAR(option_bits), NO_CMD_LINE, OPTION_PROFILING, Why should this be prohibited? I'd rather have it supported. Use case: one usually needs to profile a single statement, not everything. > DEFAULT(FALSE)); > > static Sys_var_ulong Sys_profiling_history_size( > "profiling_history_size", "Limit of query profiling memory", > - SESSION_VAR(profiling_history_size), CMD_LINE(REQUIRED_ARG), > + NO_SET_STMT SESSION_VAR(profiling_history_size), CMD_LINE(REQUIRED_ARG), > VALID_RANGE(0, 100), DEFAULT(15), BLOCK_SIZE(1)); > #endif > > @@ -3566,7 +3567,8 @@ static bool check_skip_replication(sys_var *self, THD *thd, set_var *var) > > static Sys_var_bit Sys_skip_replication( > "skip_replication", "skip_replication", > - SESSION_ONLY(option_bits), NO_CMD_LINE, OPTION_SKIP_REPLICATION, > + NO_SET_STMT SESSION_ONLY(option_bits), > + NO_CMD_LINE, OPTION_SKIP_REPLICATION, Why should this be prohibited? I'd rather have it supported. Use case: skip replication of a single statement. A generic version of FLUSH NO_WRITE_TO_BINLOG TABLE REPAIR NO_WRITE_TO_BINLOG TABLE ANALYZE NO_WRITE_TO_BINLOG TABLE OPTIMIZE NO_WRITE_TO_BINLOG TABLE and all ALTER TABLE ... PARTITION variants: ALTER TABLE ... REBUILD PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... OPTIMIZE PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... ANALYZE PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... REPAIR PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... COALESCE PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... ADD PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... REORGANIZE PARTITION NO_WRITE_TO_BINLOG > DEFAULT(FALSE), NO_MUTEX_GUARD, NOT_IN_BINLOG, > ON_CHECK(check_skip_replication)); > > @@ -3688,7 +3690,7 @@ static ulonglong read_rand_seed(THD *thd) > static Sys_var_session_special Sys_rand_seed1( > "rand_seed1", "Sets the internal state of the RAND() " > "generator for replication purposes", > - sys_var::ONLY_SESSION, NO_CMD_LINE, > + NO_SET_STMT sys_var::ONLY_SESSION, NO_CMD_LINE, > VALID_RANGE(0, ULONG_MAX), BLOCK_SIZE(1), > NO_MUTEX_GUARD, IN_BINLOG, ON_CHECK(0), > ON_UPDATE(update_rand_seed1), ON_READ(read_rand_seed)); > @@ -3706,7 +3708,7 @@ static bool update_rand_seed2(THD *thd, set_var *var) > static Sys_var_session_special Sys_rand_seed2( > "rand_seed2", "Sets the internal state of the RAND() " > "generator for replication purposes", > - sys_var::ONLY_SESSION, NO_CMD_LINE, > + NO_SET_STMT sys_var::ONLY_SESSION, NO_CMD_LINE, > VALID_RANGE(0, ULONG_MAX), BLOCK_SIZE(1), > NO_MUTEX_GUARD, IN_BINLOG, ON_CHECK(0), > ON_UPDATE(update_rand_seed2), ON_READ(read_rand_seed)); I'd rather fix rand_seed1/rand_seed2 variables not to be special anymore. Like, move them to SV structure, remove special behavior and so on. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] b18ddd2: Added 'mariadb-galera-server' metapackage to install all
by Sergei Golubchik 28 Oct '14

28 Oct '14

Hi, Nirbhay! On Oct 23, nirbhay(a)skysql.com wrote: > > Added 'mariadb-galera-server' metapackage to install all > necessary Galera-related dependencies. > > --- > debian/dist/Debian/control | 13 ++++++++++--- > debian/dist/Ubuntu/control | 13 ++++++++++--- > 2 files changed, 20 insertions(+), 6 deletions(-) > > diff --git a/debian/dist/Debian/control b/debian/dist/Debian/control > index e68f027..bc48dbc 100644 > --- a/debian/dist/Debian/control > +++ b/debian/dist/Debian/control > @@ -194,9 +194,7 @@ Recommends: libhtml-template-perl > Pre-Depends: mariadb-common, adduser (>= 3.40), debconf > Depends: mariadb-client-10.1 (>= ${source:Version}), libdbi-perl, > perl (>= 5.6), ${shlibs:Depends}, ${misc:Depends}, psmisc, passwd, > - lsb-base (>= 3.0-10), mariadb-server-core-10.1 (>= ${binary:Version}), > - galera (>=25.2), rsync, lsof, socat | netcat, grep, gawk, iproute, > - coreutils, findutils > + lsb-base (>= 3.0-10), mariadb-server-core-10.1 (>= ${binary:Version}) > Provides: mariadb-server, mysql-server, virtual-mysql-server > Conflicts: mariadb-server (<< ${source:Version}), mysql-server (<< ${source:Version}), > mysql-server-4.1, mysql-server-5.0, mysql-server-5.1, mysql-server-5.5, Looks ok. Few comments: 1. What about RPM? 2. Did you compare with, say, non-galera 10.0 - just to make sure all dependencies you removed are, indeed, only for galera and the server doesn't need any of that? 3. I'd also add /etc/mysql/conf.d/galera.cnf (/etc/my.cnf.d/galera.cnf in rpms) that enables wsrep, and may be sets other reasonable defaults for galera. Regards, Sergei

1 0

[Maria-developers] DisconnectEx() problem
by Delveri chick 28 Oct '14

28 Oct '14

Hi i was tracing why mysql_socket_shutdown(...) won't call DisconnectEx(...) as reported in the TODO list and realised DisconnectEx(...) cannot be called in include/mysql/psi/mysql_socket.h. Possible solution would be to add it here or include the *.h where it it defined. Where is it defined please? Regards Delveri

2 1

Re: [Maria-developers] [Commits] 18a7470: MDEV-6939 : Dots in file names of configuration files
by Sergei Golubchik 28 Oct '14

28 Oct '14

Hi, Nirbhay! 1. Why 10.1? 1.0 would be fine too, may be even 5.5 2. In cases like this I usually make two commits, one for a cleanup and one for a bug fix. Anyway, the fix is ok to push, of course. On Oct 24, nirbhay(a)mariadb.com wrote: > revision-id: 18a7470a563c30bffbfe787e4e5406b687597a44 > parent(s): cc656e4d8538326d85e84adbe48d0a3739d55674 > committer: Nirbhay Choubey > branch nick: server > timestamp: 2014-10-24 16:34:54 -0400 > message: > > MDEV-6939 : Dots in file names of configuration files > > Use fn_ext2() to get the file extension from last occurrence > of FN_EXTCHAR ('.') instead. Also made some cosmetic changes > in mysys/mf_fn_ext.c. > > --- > mysys/mf_fn_ext.c | 11 ++++++----- > mysys/my_default.c | 2 +- > 2 files changed, 7 insertions(+), 6 deletions(-) > Regards, Sergei

1 0