- developers - lists.mariadb.org

[Maria-developers] KB Question: Storage engine plugin that "proxies" another engine?
by Daniel Bartholomew 08 Apr '13

08 Apr '13

The following question has been posted to the AskMonty Knowledgebase: https://kb.askmonty.org/en/storage-engine-plugin-that-proxies-another-engin… - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Storage engine plugin that "proxies" another engine? I would like to write a plugin that adds transparent versioning support for table rows. The idea is one would create a table and specify something like "Engine=Versioned", which would create two regular tables in InnoDB/MyISAM. One table to hold the current data, and one to hold a couple of previous versions. Then, when running UPDATE/DELETE statements, the "versioned" engine would first copy the current data to the "history" table and then forward the statement to the underlying storage engine. Is this possible to do in a nice way using the plugin architecture? Thanks Thomas - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

[Maria-developers] mysql test question
by Rich Prohaska 05 Apr '13

05 Apr '13

Hello, I was wondering if there was a way to run a mysql-test on with a prebuilt data directory. If so, can you point me to some example? Thanks Rich Prohaska

3 4

[Maria-developers] Rev 3712: MDEV-4338 : Support atomic option on directFS/FusionIO
by Sergei Golubchik 05 Apr '13

05 Apr '13

Hi, Vladislav! On Mar 29, Vladislav Vaintroub wrote: > ------------------------------------------------------------ > revno: 3712 > revision-id: wlad(a)montyprogram.com-20130328203348-1qe9u9nlq6ol9b8l > parent: igor(a)askmonty.org-20130329021836-x6swah73a59im9tn > fixes bug: https://mariadb.atlassian.net/browse/MDEV-4338 > committer: Vladislav Vaintroub <wlad(a)montyprogram.com> > branch nick: 5.5 > timestamp: Thu 2013-03-28 21:33:48 +0100 > message: > MDEV-4338 : Support atomic option on directFS/FusionIO. > === modified file 'storage/xtradb/handler/ha_innodb.cc' > --- a/storage/xtradb/handler/ha_innodb.cc 2013-03-08 18:08:45 +0000 > +++ b/storage/xtradb/handler/ha_innodb.cc 2013-03-28 20:33:48 +0000 > @@ -185,6 +185,7 @@ static my_bool innobase_file_format_chec > static my_bool innobase_log_archive = FALSE; > static char* innobase_log_arch_dir = NULL; > #endif /* UNIV_LOG_ARCHIVE */ > +static my_bool innobase_use_atomic_writes = TRUE; > static my_bool innobase_use_doublewrite = TRUE; > static my_bool innobase_use_checksums = TRUE; > static my_bool innobase_fast_checksum = FALSE; > @@ -3057,6 +3058,35 @@ innobase_init( > srv_kill_idle_transaction = 0; > #endif > > + srv_use_atomic_writes = (ibool) innobase_use_atomic_writes; Is that the typical usage pattern in the innodb code? Because it looks a bit silly to me, why couldn't you create a sysvar directly on srv_use_atomic_writes, why an intermediate innobase_use_atomic_writes variable? > + if (innobase_use_atomic_writes) { > + fprintf(stderr, "InnoDB: using atomic writes.\n"); > + > + /* Force doublewrite buffer off, atomic writes replace it. */ > + if (srv_use_doublewrite_buf) { > + fprintf(stderr, "InnoDB: Switching off doublewrite buffer " > + "because of atomic writes.\n"); > + innobase_use_doublewrite = srv_use_doublewrite_buf = FALSE; > + } > + > + /* Force O_DIRECT on Unixes (on Windows writes are always unbuffered)*/ > +#ifndef _WIN32 > + if(!innobase_file_flush_method || > + !strstr(innobase_file_flush_method, "O_DIRECT")) { > + innobase_file_flush_method = > + srv_file_flush_method_str = (char*)"O_DIRECT"; > + fprintf(stderr, "InnoDB: using O_DIRECT due to atomic writes.\n"); > + } > +#endif > +#ifdef HAVE_POSIX_FALLOCATE > + /* Due to a bug in directFS, using atomics needs > + * posix_fallocate to extend the file > + * pwrite() past end of the file won't work > + */ > + srv_use_posix_fallocate = TRUE; > +#endif > + } it'd be nice if you could try os_file_set_atomic_writes() here to see if that works. This function creates and opens quite a few files, you could use one of them to mark it for atomic writes, and if that would fail, you'd disable atomic writes, and wouldn't change innobase_file_flush_method and innobase_use_doublewrite. Btw, why not to use posix_fallocate whenever it's available? Or, at least, with its own --innodb-use-fallocate option? Regards, Sergei

3 6

Re: [Maria-developers] MDEV-3917 multiple use locks (GET_LOCK) in one connection
by Sergei Golubchik 04 Apr '13

04 Apr '13

Hi, Holyfoot! (Kostja - there're a couple of questions at the end that you might want to reply to) On Apr 02, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-3917/ > > ------------------------------------------------------------ > revno: 3510 > revision-id: holyfoot(a)askmonty.org-20130402085219-1sxuzssq7segeig4 > parent: igor(a)askmonty.org-20130331221855-1n7hagklb3soplxq > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-3917 > timestamp: Tue 2013-04-02 13:52:19 +0500 > message: > MDEV-3917 multiple use locks (GET_LOCK) in one connection. > The patch contributed by Konstantin Osipov applied. > Native comments: > Implement multiple user-level locks per connection. > > GET_LOCK() function in MySQL allows a connection to hold at most > one user level lock. Taking a new lock automatically releases the > old lock, if any. > > The limit of one lock per session existed since early versions > of MySQL didn't have a deadlock detector for SQL locks. > MDL patches in MySQL 5.5 added a deadlock detector, > so starting from 5.5 it became possible to take multiple locks > in any order -- a deadlock, should it occur, would be detected > and an error returned to the client which closed the wait chain. > > This is exactly what is done in this patch: ULLs are moved > to use MDL subsystem. > > also: > MDL_key::USER renamed to MDL::USER_LOCK What does that mean? > boundary checks for user lock name added. > === modified file 'sql/item_func.cc' > --- a/sql/item_func.cc 2013-03-27 22:41:02 +0000 > +++ b/sql/item_func.cc 2013-04-02 08:52:19 +0000 > @@ -4010,7 +3896,121 @@ int Interruptible_wait::wait(mysql_cond_ > > > /** > - Get a user level lock. If the thread has an old lock this is first released. > + For locks with EXPLICIT duration, MDL returns a new ticket > + every time a lock is granted. This allows to implement recursive > + locks without extra allocation or additional data structures, such > + as below. However, if there are too many tickets in the same > + MDL_context, MDL_context::find_ticket() is getting too slow, > + since it's using a linear search. > + This is why a separate structure is allocated for a user > + level lock, and before requesting a new lock from MDL, > + GET_LOCK() checks thd->ull_hash if such lock is already granted, > + and if so, simply increments a reference counter. > +*/ > + > +class User_level_lock > +{ > +public: > + MDL_ticket *lock; > + int refs; > +}; > + > + > +/** Extract a hash key from User_level_lock. */ > + > +uchar *ull_get_key(const uchar *ptr, size_t *length, > + my_bool not_used __attribute__((unused))) > +{ > + User_level_lock *ull = (User_level_lock*) ptr; > + MDL_key *key = ull->lock->get_key(); > + *length= key->length(); > + return (uchar*) key->ptr(); > +} > + > + > +/** > + Release all user level locks for this THD. > +*/ > + > +void mysql_ull_cleanup(THD *thd) > +{ > + User_level_lock *ull; > + DBUG_ENTER("mysql_ull_cleanup"); > + > + for (uint i= 0; i < thd->ull_hash.records; i++) > + { > + ull = (User_level_lock*) my_hash_element(&thd->ull_hash, i); > + thd->mdl_context.release_lock(ull->lock); > + free(ull); my_free > + } > + > + my_hash_free(&thd->ull_hash); > + > + DBUG_VOID_RETURN; > +} > + > + > +/** > + Set explicit duration for metadata locks corresponding to > + user level locks to protect them from being released at the end > + of transaction. > +*/ > + > +void mysql_ull_set_explicit_lock_duration(THD *thd) > +{ > + User_level_lock *ull; > + DBUG_ENTER("mysql_ull_set_explicit_lock_duration"); > + > + for (uint i= 0; i < thd->ull_hash.records; i++) > + { > + ull= (User_level_lock*) my_hash_element(&thd->ull_hash, i); > + thd->mdl_context.set_lock_duration(ull->lock, MDL_EXPLICIT); > + } > + DBUG_VOID_RETURN; > +} > + > + > +/** > + When MDL detects a lock wait timeout, it pushes > + an error into the statement diagnostics area. > + For GET_LOCK(), lock wait timeout is not an error, > + but a special return value (0). NULL is returned in > + case of error. > + Capture and suppress lock wait timeout. > +*/ > + > +class Lock_wait_timeout_handler: public Internal_error_handler > +{ > +public: > + Lock_wait_timeout_handler() :m_lock_wait_timeout(false) {} > + > + bool m_lock_wait_timeout; > + > + bool handle_condition(THD * /* thd */, uint sql_errno, > + const char * /* sqlstate */, > + MYSQL_ERROR::enum_warning_level /* level */, > + const char *message, > + MYSQL_ERROR ** /* cond_hdl */); > +}; > + > +bool > +Lock_wait_timeout_handler:: > +handle_condition(THD * /* thd */, uint sql_errno, > + const char * /* sqlstate */, > + MYSQL_ERROR::enum_warning_level /* level */, > + const char *message, > + MYSQL_ERROR ** /* cond_hdl */) > +{ > + if (sql_errno == ER_LOCK_WAIT_TIMEOUT) > + { > + m_lock_wait_timeout= true; > + return true; /* condition handled */ > + } > + return false; > +} > + > +/** > + Get a user level lock. > > @retval > 1 : Got lock > @@ -4039,104 +4038,75 @@ longlong Item_func_get_lock::val_int() > it's not guaranteed to be same as on master. > */ > if (thd->slave_thread) > + { > + null_value= 0; > DBUG_RETURN(1); > + } > > if (!res || !res->length()) > + DBUG_RETURN(0); > + > + if (res->length() > NAME_LEN) > { > + my_error(ER_TOO_LONG_IDENT, MYF(0), res->c_ptr_safe()); > DBUG_RETURN(0); > } > + > DBUG_PRINT("info", ("lock %.*s, thd=%ld", res->length(), res->ptr(), > (long) thd->real_id)); > + /* HASH entries are of type User_level_lock. */ > + if (! my_hash_inited(&thd->ull_hash) && > + my_hash_init(&thd->ull_hash, &my_charset_bin, > + 16 /* small hash */, 0, 0, ull_get_key, NULL, 0)) > { > + DBUG_RETURN(0); > } > > + MDL_request ull_request; > + ull_request.init(MDL_key::USER_LOCK, res->c_ptr_safe(), "", > + MDL_SHARED_NO_WRITE, MDL_EXPLICIT); > + MDL_key *ull_key = &ull_request.key; > > > + if ((ull= (User_level_lock*) > + my_hash_search(&thd->ull_hash, ull_key->ptr(), ull_key->length()))) > + { > + /* Recursive lock */ > + ull->refs++; > + null_value = 0; > + DBUG_RETURN(1); > + } > > + Lock_wait_timeout_handler lock_wait_timeout_handler; > + thd->push_internal_handler(&lock_wait_timeout_handler); > + bool error= thd->mdl_context.acquire_lock(&ull_request, timeout); > + (void) thd->pop_internal_handler(); > + if (error) > { > + if (lock_wait_timeout_handler.m_lock_wait_timeout) > + null_value= 0; > + DBUG_RETURN(0); > } > > + ull= (User_level_lock*) malloc(sizeof(User_level_lock)); my_malloc() with MYF(MY_WME|MY_THREAD_SPECIFIC) > + if (ull == NULL) > { > + thd->mdl_context.release_lock(ull_request.ticket); > + DBUG_RETURN(0); > } > + > + ull->lock= ull_request.ticket; > + ull->refs= 1; > + > + if (my_hash_insert(&thd->ull_hash, (uchar*) ull)) > { > + thd->mdl_context.release_lock(ull->lock); > + free(ull); my_free > + DBUG_RETURN(0); > } > + null_value= 0; > > + DBUG_RETURN(1); > } > > > @@ -4151,43 +4121,104 @@ longlong Item_func_get_lock::val_int() > longlong Item_func_release_lock::val_int() > { > DBUG_ASSERT(fixed == 1); > + String *res= args[0]->val_str(&value); > + THD *thd= current_thd; > DBUG_ENTER("Item_func_release_lock::val_int"); > + null_value= 1; > + > if (!res || !res->length()) > + DBUG_RETURN(0); > + > + if (res->length() > NAME_LEN) > { > + my_error(ER_TOO_LONG_IDENT, MYF(0), res->c_ptr_safe()); > DBUG_RETURN(0); > } > + > DBUG_PRINT("info", ("lock %.*s", res->length(), res->ptr())); > > + MDL_key ull_key; > + ull_key.mdl_key_init(MDL_key::USER_LOCK, res->c_ptr_safe(), ""); > + > + User_level_lock *ull; > + > + if (!(ull= > + (User_level_lock*) my_hash_search(&thd->ull_hash, > + ull_key.ptr(), ull_key.length()))) > { > + null_value= thd->mdl_context.get_lock_owner(&ull_key) == 0; > + DBUG_RETURN(0); > } > + null_value= 0; > + if (--ull->refs == 0) > { > + my_hash_delete(&thd->ull_hash, (uchar*) ull); > + thd->mdl_context.release_lock(ull->lock); > + free(ull); my_free > } > + DBUG_RETURN(1); > +} > + > + > +/** > + Check a user level lock. > + > + Sets null_value=TRUE on error. > + > + @retval > + 1 Available > + @retval > + 0 Already taken, or error > +*/ > + > +longlong Item_func_is_free_lock::val_int() > +{ > + DBUG_ASSERT(fixed == 1); > + String *res= args[0]->val_str(&value); > + THD *thd= current_thd; > + null_value= 1; > + this block ----- from here > + if (!res || !res->length()) > + return 0; > + > + if (res->length() > NAME_LEN) > + { > + my_error(ER_TOO_LONG_IDENT, MYF(0), res->c_ptr_safe()); > + return 0; > + } and up ---- to here is repeated four times. please move it to a separate function. Like if (!ull_name_ok(res)) return 0; > + > + MDL_key ull_key; > + ull_key.mdl_key_init(MDL_key::USER_LOCK, res->c_ptr_safe(), ""); > + > + null_value= 0; > + return thd->mdl_context.get_lock_owner(&ull_key) == 0; > +} > + > + > +longlong Item_func_is_used_lock::val_int() > +{ > + DBUG_ASSERT(fixed == 1); > + String *res= args[0]->val_str(&value); > + THD *thd= current_thd; > + null_value= 1; > + > + if (!res || !res->length()) > + return 0; > + > + if (res->length() > NAME_LEN) > + { > + my_error(ER_TOO_LONG_IDENT, MYF(0), res->c_ptr_safe()); > + return 0; > + } > + > + MDL_key ull_key; > + ull_key.mdl_key_init(MDL_key::USER_LOCK, res->c_ptr_safe(), ""); > + ulong thread_id = thd->mdl_context.get_lock_owner(&ull_key); > + if (thread_id == 0) > + return 0; > + > + null_value= 0; > + return thread_id; > } > > === modified file 'sql/mdl.cc' > --- a/sql/mdl.cc 2013-01-29 14:10:47 +0000 > +++ b/sql/mdl.cc 2013-04-02 08:52:19 +0000 > @@ -2094,8 +2151,6 @@ MDL_context::acquire_lock(MDL_request *m > > find_deadlock(); > > - if (lock->needs_notification(ticket)) > - { okay, so before the patch only locks that needs_notification were using the loop with short waits. Now all locks do. Presumably to check whether thd_is_connected still. Before the patch, how were client disconnects handled? Like, one could have started to wait on the mdl lock, and then disconnect. Was the wait aborted? > struct timespec abs_shortwait; > set_timespec(abs_shortwait, 1); > wait_status= MDL_wait::EMPTY; > @@ -2106,17 +2163,25 @@ MDL_context::acquire_lock(MDL_request *m > > if (wait_status != MDL_wait::EMPTY) > break; > + /* Check if the client is gone while we were waiting. */ > + if (! thd_is_connected(m_thd)) > + { > + /* > + * The client is disconnected. Don't wait forever: > + * assume it's the same as a wait timeout, this > + * ensures all error handling is correct. > + */ > + wait_status= MDL_wait::TIMEOUT; > + break; > + } > > mysql_prlock_wrlock(&lock->m_rwlock); > + if (lock->needs_notification(ticket)) why this if() is under the prlock, and not the other way around? > lock->notify_conflicting_locks(this); > mysql_prlock_unlock(&lock->m_rwlock); > set_timespec(abs_shortwait, 1); > } > if (wait_status == MDL_wait::EMPTY) > - wait_status= m_wait.timed_wait(m_thd, &abs_timeout, TRUE, > - mdl_request->key.get_wait_state_name()); > - } > - else > wait_status= m_wait.timed_wait(m_thd, &abs_timeout, TRUE, > mdl_request->key.get_wait_state_name()); Regards, Sergei

1 0

[Maria-developers] MDEV-4145/MWL#253 review feedback (unfinished)
by Sergei Petrunia 03 Apr '13

03 Apr '13

Hello Igor, Please find some review feedback below. ===== sql_statistics.cc has this line with comment: > If the value the variable 'use_stat_tables' is set to > "preferably" the optimizer uses a particular statistical data only if > it can't be found in the statistical data. the part > uses a particular statistical data > only if it can't be found > in the statistical data. doesn't seem to make any sense. Is this a typo? ===== sql_statistics.cc: collect_statistics_for_table() has this code: > table_field->collected_stats->cleanup(); > } > bitmap_clear_all(table->write_set); Why suddenly no indentation on the line with the bitmap_clear_all() call? This looks as if a merge has gone wrong, please fix. ===== sql_statistics.cc has this : > class Table_stat: public Stat_table > { > > private: > > Field *db_name_field; /* Field for the column table_stats.db_name */ Why do all class definitions start with 'private:'? It is active by default in C++. The code looks as if it was auto-translated from some other language. ===== I see that my progress reporting patch from November is not part of the tree yet. ===== I don't any code that would check if table DDLs of statistical tables are what the server expects them to be. I think the check should be made, just like it is made for mysql.proc and other tables. ==== Then I started to play with the code and have filed MDEV-4350, MDEV-4357, MDEV-4359, MDEV-4360, MDEV-4362, MDEV-4363, MDEV-4364. I find it difficult to proceed when there are so many simple counterexamples. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] read_buffer_size query
by Ian Gilfillan 02 Apr '13

02 Apr '13

Hi I'm working on adding complete descriptions to the knowledgebase for all the server variables right now, and have a query on the read_buffer_size variable. The default in my MariaDB 10.0.1 seems to be 2097152, but it's listed on the MySQL docs with a default of 131072. I can't find a mention in the changelogs of this having changed - can anyone tell me in which version the change in default happened? Also, in the MySQL documentation, this variable is described as applying to MyISAM tables, while in the only short description on the Knowledgebase I can find, at https://kb.askmonty.org/en/mysqld-options-full-list/, it's described more generally, not specifically mentioning MyISAM. Does this variable apply to other storage engines as well now (Aria?), or still only MyISAM? thanks, ian gilfillan

3 4

[Maria-developers] Cassandra disabled due to excessive memory usage causing test failures
by Kristian Nielsen 02 Apr '13

02 Apr '13

Hi, When I added my 10.0-mdev26 tree to buildbot, I got some strange errors on the kvm-fulltest2 builder. I finally tracked it down to be caused by the OOM killer killing random processes. The root issue is Cassandra. At some point it seems cassandra was enabled in the VM used by the fulltest2 builder. However, just starting up Cassandra creates a jsvc process that uses 3.4 _gigabyte_ of memory, 1.3 GB of which is resident. That is too much... I also noticed that cassandra tests are not even run, plugins.cassandra is skipped due to cassandra not running. Possibly this is because the OOM killer has killed the process ;-). In any case, for now I have disabled cassandra on the -fulltest VMs: sudo update-rc.d cassandra disable If someone wants to re-enable it, feel free, but please first make sure that it does not cause unrelated test failures due to excessive memory usage! I would suggest not to run cassandra by default, instead add a separate buildbot step that explicitly starts up cassandra when needed. ---- In general, I am disappointed about the state of our trees. This problem seems to have been there for more than a month without anyone doing anything, and there are tons of other test failures in 10.0 and 10.0-base. We need to tighten up our processes here. Remember, the rules are that anything major should be green in a feature tree before pushing to a main tree. Clearly this is not what is happening at the moment. It's ridiculous to constantly hear people pressing for release of 10.0.2-beta when our trees are not even alpha state ... - Kristian.

1 0

[Maria-developers] I have a friend. Her name is Maria
by Кукушкин Александр 01 Apr '13

01 Apr '13

Hello! excuse me for off topic. I have a friend, her name is Maria and she has a birthday today. I thought maybe someone of you could write a commit message mentioning her. Wish her good performance and persistence or something like that. Then I could just show her a page like that one http://bazaar.launchpad.net/~maria-captains/maria/10.0/revision/3505 What do you think about that? is it possible? (sorry if this message is buplicate, I tried to post it yesterday to your mailing list but it seems like I failed)

1 1

Re: [Maria-developers] Rev 3497: MDEV-318 IF (NOT) EXIST clauses for ALTER TABLE (MWL #252)
by Sergei Golubchik 29 Mar '13

29 Mar '13

Hi, Holyfoot! Please push, with one little change: On Feb 19, holyfoot(a)askmonty.org wrote: > > message: > MDEV-318 IF (NOT) EXIST clauses for ALTER TABLE (MWL #252). > Syntax modified to allow statements: > ALTER TABLE ADD/DROP COLUMN > ALTER TABLE ADD/DROP INDEX > ALTER TABLE ADD/DROP FOREIGN KEY > ALTER TABLE ADD/DROP PARTITION > ALTER TABLE CHANGE COLUMN > ALTER TABLE MODIFY COLUMN > DROP INDEX > to have IF (NOT) EXISTS options. > Appropriate implementations added to mysql_alter_table(). ... > key_def: > - normal_key_type opt_ident key_alg '(' key_list ')' > + normal_key_type opt_if_not_exists_ident key_alg '(' key_list ')' > { Lex->option_list= NULL; } remove your new opt_if_not_exists_ident rule, write it like - normal_key_type opt_ident key_alg '(' key_list ')' + normal_key_type opt_if_not_exists opt_ident key_alg '(' key_list ')' Regards, Sergei

2 2

[Maria-developers] New problem with GTID patch
by Pavel Ivanov 29 Mar '13

29 Mar '13

Kristian, Unfortunately with your recent commit (revision 3546) test rpl.rpl_gtid_startpos doesn't work when GTID patch is applied on top of 10.0.1 tarball. It works okay in 10.0-mdev26 though, so I'm not sure if you'll want to look at the problem now. In particular the problematic addition is if (!is_relay_log && read_state_from_file()) DBUG_RETURN(1); to function MYSQL_BIN_LOG::open(). Without it test works. With it test breaks in the section "Test that master gives error when slave asks for empty gtid pos and binlog files have been purged". The problem is when get_gtid_list_event() is called from gtid_find_binlog_file() it returns event with count == 0. It causes contains_all_slave_gtid() to return true and master doesn't send error to slave as expected... I didn't look in any further details because I'm not familiar with everything that should happen here that much. If you want me to look at something else to better understand the problem let me know. Pavel

2 3