September 2013 - developers - lists.mariadb.org

[Maria-developers] Status and future of spatial extensions
by Mateusz Loskot 22 Sep '13

22 Sep '13

Hi, I haven't monitored MariaDB development closely, neither I tracked what's happening in the MySQL sandbox, so I'm hoping to catch up by asking questions here. I'm interested in the Spatial Extensions. So, I looked around the KB and the documentation and I found a few pages: general overview [1], 5.3 features [2] and page about plans [3]. I've looked into JIRA issues and recent commits too. Still, I don't see a clear big picture about the current status, roadmap and developers of the Spatial Extensions. Could anyone shed more light on the Spatial part of the MariaDB project, where it is now, where it is going, who is behind the wheel, etc.? What is (if any) the relation with geo-spatial development in MySQL? Is that correct understanding that MySQL and MariaDB follow separate paths, thus the GIS implementations go separately and in future they may seriously diverge and become no longer equivalent drop-in replacements? [1] https://mariadb.com/kb/en/gis-functionality/ [2] https://mariadb.com/kb/en/the-maria53-gis-tree-on-launchpad/ [3] https://mariadb.com/kb/en/mariadb-plans-gis/ Best regards, -- Mateusz Loskot, http://mateusz.loskot.net "Participation in this whole process is a form of torture" ~~ Szalony

1 0

Re: [Maria-developers] Rev 3818: Check for duplicate index (port from mysql)
by Sergei Golubchik 22 Sep '13

22 Sep '13

Hi, Sanja! On Sep 20, sanja(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3818 > revision-id: sanja(a)askmonty.org-20130920193019-vr2q24c8g255aofu > parent: sanja(a)askmonty.org-20130920141533-nwk5jt07ojwmmctr > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-fts2 > timestamp: Fri 2013-09-20 22:30:19 +0300 > message: > Check for duplicate index (port from mysql) > (pre fts) ok to push Regards, Sergei

1 0

[Maria-developers] Review of MDEV-4506, parallel replication, part 2
by Michael Widenius 22 Sep '13

22 Sep '13

Hi! Part 2 of review of parallel replication === added file 'sql/rpl_parallel.cc' <cut> +pthread_handler_t +handle_rpl_parallel_thread(void *arg) +{ <cut> + if (end_of_group) + { + in_event_group= false; + Add comment: /* All events for this group has now been executed and logged (but not necessaerly synced). Inform the other event groups that are waiting for this thread that we are done. */ + rgi->commit_orderer.unregister_wait_for_prior_commit(); + thd->wait_for_commit_ptr= NULL; + + /* + Record that we have finished, so other event groups will no + longer attempt to wait for us to commit. + + We can race here with the next transactions, but that is fine, as + long as we check that we do not decrease last_committed_sub_id. If + this commit is done, then any prior commits will also have been + done and also no longer need waiting for. + */ + mysql_mutex_lock(&entry->LOCK_parallel_entry); + if (entry->last_committed_sub_id < event_gtid_sub_id) + { + entry->last_committed_sub_id= event_gtid_sub_id; + mysql_cond_broadcast(&entry->COND_parallel_entry); + } + mysql_mutex_unlock(&entry->LOCK_parallel_entry); + Add: /* Tell storage engines that they can stop waiting. See comments in sql_class.cc::wakeup_subsequent_commits2() for why we can't do this part of the wakeup in unregister_wait_for_prior_commit(). */ + rgi->commit_orderer.wakeup_subsequent_commits(); + delete rgi; + } + + events= next; + } + + mysql_mutex_lock(&rpt->LOCK_rpl_thread); + if ((events= rpt->event_queue) != NULL) + { + rpt->event_queue= rpt->last_in_queue= NULL; + mysql_mutex_unlock(&rpt->LOCK_rpl_thread); + goto more_events; + } + + if (!in_event_group) + { + rpt->current_entry= NULL; As far as I can see, rpt->free is always 0 here. - It's set to 0 at the top of this loop and only set to 1 here. Can we remove the rpt->free variable? + if (!rpt->stop && !rpt->free) + { Add comment. /* Take next group of events from the replication pool. This is faster than having to wakeup the pool manager thread to give us a new event. */ + mysql_mutex_lock(&rpt->pool->LOCK_rpl_thread_pool); + list= rpt->pool->free_list; + rpt->next= list; + rpt->pool->free_list= rpt; + if (!list) + mysql_cond_broadcast(&rpt->pool->COND_rpl_thread_pool); + mysql_mutex_unlock(&rpt->pool->LOCK_rpl_thread_pool); + rpt->free= true; + } + } + } <cut> +int +rpl_parallel_change_thread_count(rpl_parallel_thread_pool *pool, + uint32 new_count, bool skip_check) +{ + uint32 i; + rpl_parallel_thread **new_list= NULL; + rpl_parallel_thread *new_free_list= NULL; + + /* + Allocate the new list of threads up-front. + That way, if we fail half-way, we only need to free whatever we managed + to allocate, and will not be left with a half-functional thread pool. + */ + if (new_count && + !(new_list= (rpl_parallel_thread **)my_malloc(new_count*sizeof(*new_list), + MYF(MY_WME)))) + { + my_error(ER_OUTOFMEMORY, MYF(0), (int(new_count*sizeof(*new_list)))); + goto err;; + } Why not allocate new_list[] elements at the same time? Faster, less memory not allocation to check and freeing is also esier. if (!my_multi_malloc(MYF(MY_WME), &new_list, (uint) new_count*sizeof(*new_list), &list_part, new_count*sizeof(**new_list), NULL)) ... + + for (i= 0; i < new_count; ++i) + { + pthread_t th; + + if (!(new_list[i]= (rpl_parallel_thread *)my_malloc(sizeof(*(new_list[i])), + MYF(MY_WME)))) + { + my_error(ER_OUTOFMEMORY, MYF(0), sizeof(*(new_list[i]))); + goto err; + } The above would then simple be: new_list[i]= list_part++; Instead of setting everything to false and NULL, why not simple bzero() the structure? That way we can be more sure that we don't forget to initialize any new element in the future. + new_list[i]->delay_start= true; + new_list[i]->running= false; + new_list[i]->stop= false; + new_list[i]->free= false; + mysql_mutex_init(key_LOCK_rpl_thread, &new_list[i]->LOCK_rpl_thread, + MY_MUTEX_INIT_SLOW); + mysql_cond_init(key_COND_rpl_thread, &new_list[i]->COND_rpl_thread, NULL); + new_list[i]->pool= pool; + new_list[i]->current_entry= NULL; + new_list[i]->event_queue= NULL; + new_list[i]->last_in_queue= NULL; <cut> Add: /* The following loops wait until each thread is done it's work for each event group and then stops it when the thread has nothing to do. Note that this may take some time as rpl_paralell:do_event() may request threads at the same time. */ To make this faster and ensuring we don't start new request, wouldn't it be better to go through pool->threads and signal each of them? As long as rpt->stop is defined as 'stop when you are done processing one group' then this is fine (which is how it looks it's defined now). + for (i= 0; i < pool->count; ++i) + { + rpl_parallel_thread *rpt= pool->get_thread(NULL); + rpt->stop= true; + mysql_cond_signal(&rpt->COND_rpl_thread); + mysql_mutex_unlock(&rpt->LOCK_rpl_thread); + } + + for (i= 0; i < pool->count; ++i) + { + rpl_parallel_thread *rpt= pool->threads[i]; + mysql_mutex_lock(&rpt->LOCK_rpl_thread); + while (rpt->running) + mysql_cond_wait(&rpt->COND_rpl_thread, &rpt->LOCK_rpl_thread); + mysql_mutex_unlock(&rpt->LOCK_rpl_thread); + mysql_mutex_destroy(&rpt->LOCK_rpl_thread); + mysql_cond_destroy(&rpt->COND_rpl_thread); + my_free(rpt); By using my_multi_malloc() you can remove the above free + } + + my_free(pool->threads); + pool->threads= new_list; + pool->free_list= new_free_list; + pool->count= new_count; + for (i= 0; i < pool->count; ++i) + { + mysql_mutex_lock(&pool->threads[i]->LOCK_rpl_thread); + pool->threads[i]->delay_start= false; + mysql_cond_signal(&pool->threads[i]->COND_rpl_thread); + mysql_mutex_unlock(&pool->threads[i]->LOCK_rpl_thread); + } + + if (!skip_check) + { + mysql_mutex_lock(&LOCK_active_mi); + pool->changing= false; + mysql_mutex_unlock(&LOCK_active_mi); + } + return 0; + +err: + if (new_list) + { + while (new_free_list) + { + rpl_parallel_thread *next= new_free_list->next; + mysql_mutex_lock(&new_free_list->LOCK_rpl_thread); + new_free_list->delay_start= false; + new_free_list->stop= true; + mysql_cond_signal(&new_free_list->COND_rpl_thread); + while (!new_free_list->running) + mysql_cond_wait(&new_free_list->COND_rpl_thread, + &new_free_list->LOCK_rpl_thread); + while (new_free_list->running) + mysql_cond_wait(&new_free_list->COND_rpl_thread, + &new_free_list->LOCK_rpl_thread); + mysql_mutex_unlock(&new_free_list->LOCK_rpl_thread); + my_free(new_free_list); By using my_multi_malloc() you can remove the above free and even the 'next' variable + new_free_list= next; + } + my_free(new_list); + } <cut> <cut> +void +rpl_parallel::wait_for_done() +{ + struct rpl_parallel_entry *e; + uint32 i; + + for (i= 0; i < domain_hash.records; ++i) + { + e= (struct rpl_parallel_entry *)my_hash_element(&domain_hash, i); + mysql_mutex_lock(&e->LOCK_parallel_entry); + while (e->current_sub_id > e->last_committed_sub_id) + mysql_cond_wait(&e->COND_parallel_entry, &e->LOCK_parallel_entry); + mysql_mutex_unlock(&e->LOCK_parallel_entry); + } +} How to handle errors in the above case? For example when we get an error and we had to abort things? I assume we should also abort for: e->rpl_thread->thd->killed() but this is hard as thd and even rpl_thread may be deleted. One way to fix this would be to add an entry 'killed' to rpl_parell_entry that we can set. However it will be hard to keep this up to date. The other way would be to, on error, set last_committed_sub_id back to it's last value. We could also have some kind of shutdown flag that we check to avoid hang on shutdown. What do you suggest? Assuming setting the value back would be the easiest solution.. + for (i= 0; i < domain_hash.records; ++i) + { + e= (struct rpl_parallel_entry *)my_hash_element(&domain_hash, i); + mysql_mutex_lock(&e->LOCK_parallel_entry); + while (e->current_sub_id > e->last_committed_sub_id) + mysql_cond_wait(&e->COND_parallel_entry, &e->LOCK_parallel_entry); + mysql_mutex_unlock(&e->LOCK_parallel_entry); + } +bool +rpl_parallel::do_event(struct rpl_group_info *serial_rgi, Log_event *ev) +{ + rpl_parallel_entry *e; + rpl_parallel_thread *cur_thread; + rpl_parallel_thread::queued_event *qev; + struct rpl_group_info *rgi; + Relay_log_info *rli= serial_rgi->rli; + enum Log_event_type typ; + + /* ToDo: what to do with this lock?!? */ + mysql_mutex_unlock(&rli->data_lock); + + if (!(qev= (rpl_parallel_thread::queued_event *)my_malloc(sizeof(*qev), + MYF(0)))) + { + my_error(ER_OUT_OF_RESOURCES, MYF(0)); + return true; + } + qev->ev= ev; + qev->next= NULL; + + if ((typ= ev->get_type_code()) == GTID_EVENT) + { + Gtid_log_event *gtid_ev= static_cast<Gtid_log_event *>(ev); + + if (!(e= find(gtid_ev->domain_id)) || + !(rgi= new rpl_group_info(rli)) || + event_group_new_gtid(rgi, gtid_ev)) + { Here add delete of rgi. Also set rgi to 0 before. + my_error(ER_OUT_OF_RESOURCES, MYF(MY_WME)); + return true; + } + + if ((gtid_ev->flags2 & Gtid_log_event::FL_GROUP_COMMIT_ID) && + e->last_commit_id == gtid_ev->commit_id) + { + /* + We are already executing something else in this domain. But the two + event groups were committed together in the same group commit on the + master, so we can still do them in parallel here on the slave. + + However, the commit of this event must wait for the commit of the prior + event, to preserve binlog commit order and visibility across all + servers in the replication hierarchy. + */ + rpl_parallel_thread *rpt= global_rpl_thread_pool.get_thread(e); Here we could add a test if get_thread() is 0, which could happen on shutdown or when killing a replication thread. I think it's relatively safe to use get_thread() as a end-synchronize point as we should never have anything half-done when we call get_thread()). As get_thread() can take a very long time and is not called in many places, it's also sounds like a resonable thing to do. + rgi->wait_commit_sub_id= e->current_sub_id; + rgi->wait_commit_group_info= e->current_group_info; + rgi->wait_start_sub_id= e->prev_groupcommit_sub_id; + e->rpl_thread= cur_thread= rpt; + /* get_thread() returns with the LOCK_rpl_thread locked. */ Add the abvove comments also where get_thread() is defined. + } + else + { + /* Check if we already have a worker thread for this entry. */ I had a hard time to understand what the above meant. Is this new suggested comments correct? /* Get the last or current thread for the domain */ + cur_thread= e->rpl_thread; + if (cur_thread) + { /* Active thread exists. The thread may still be executing queries for this domain or it may have been rescheduled to work for another domain. */ + mysql_mutex_lock(&cur_thread->LOCK_rpl_thread); + if (cur_thread->current_entry != e) + { + /* Not ours anymore, we need to grab a new one. */ Suggested new comment: /* The thread has been reused for another domain. (e->rpl_thread was not reset or was changed during the mutex_lock) As there is no threads working on this domain, it's safe to assign a new thread to work on it */ + mysql_mutex_unlock(&cur_thread->LOCK_rpl_thread); + e->rpl_thread= cur_thread= NULL; + } + } <cut> Add here: /* Current is not set for the case where the master did not have GTID, like a MariaDB 5.5 or MySQL master. */ + else if (!Log_event::is_group_event(typ) || !current) + { + /* + Events like ROTATE and FORMAT_DESCRIPTION. Do not run in worker thread. + Same for events not preceeded by GTID (we should not see those normally, + but they might be from an old master). + */ + qev->rgi= serial_rgi; + rpt_handle_event(qev, NULL); + delete_or_keep_event_post_apply(rli, typ, qev->ev); Why is delete_or_keep_event_post_apply() not part of rpt_handle_event()? You always call these two together... + + return false; + } + else + { The following code is identical (except one line) to the code earlier in this function. Better to create a function for this to reuse code and to clarify the differences. + cur_thread= current->rpl_thread; + if (cur_thread) + { + mysql_mutex_lock(&cur_thread->LOCK_rpl_thread); + if (cur_thread->current_entry != current) + { + /* Not ours anymore, we need to grab a new one. */ + mysql_mutex_unlock(&cur_thread->LOCK_rpl_thread); + cur_thread= NULL; + } + } + if (!cur_thread) + { + cur_thread= current->rpl_thread= + global_rpl_thread_pool.get_thread(current); + } + } + qev->rgi= current->current_group_info; + + /* + Queue the event for processing. + */ + if (cur_thread->last_in_queue) + cur_thread->last_in_queue->next= qev; + else + cur_thread->event_queue= qev; + cur_thread->last_in_queue= qev; + mysql_cond_signal(&cur_thread->COND_rpl_thread); + mysql_mutex_unlock(&cur_thread->LOCK_rpl_thread); In this case it's safe to move the cond_signal outside of the mutex_unlock. (Should make things notable faster as we win a sleep-wakup in other thread). <cut> === added file 'sql/rpl_parallel.h' --- sql/rpl_parallel.h 1970-01-01 00:00:00 +0000 +++ sql/rpl_parallel.h 2013-08-29 08:18:22 +0000 @@ -0,0 +1,92 @@ +#ifndef RPL_PARALLEL_H +#define RPL_PARALLEL_H + +#include "log_event.h" + + +struct rpl_parallel; +struct rpl_parallel_entry; +struct rpl_parallel_thread_pool; + +class Relay_log_info; +struct rpl_parallel_thread { + bool delay_start; I think we can remove deley_start. Se comments from previous review. + bool running; + bool stop; We need to define 'stop' properly. Like 'stop after we have executed the current event group' (So that we don't have to do rollback). + bool free; I think we can remove free. Se comments above. Otherwise it would be good to have a good descripition for each struct in this file. Will now start reviewing your new commits to the parallel replication tree. Regards, Monty

1 0

[Maria-developers] MDEV-5019 - THREADPOOL - Create Information Schema Table for Threadpool
by Roberto Spadim 22 Sep '13

22 Sep '13

Hi guys! I openned a new MDEV https://mariadb.atlassian.net/browse/MDEV-5019 the idea is expose the threadpool group information like oracle thread pool (more information about oracle thread pool here: http://mikaelronstrom.blogspot.com.br/2011_10_01_archive.html) I can help, but i don't know how access the threadpool at scheduler.h/scheduler.c from a information schema plugin 1) lock important mutex and others mdl 2) acess variables 3) write information schema based on variables 1 and 2 i don't know how to do :( can anyone help ? :) thanks guys :) -- Roberto Spadim

2 9

[Maria-developers] Perf tuning for hot key updating with high concurrency
by Hui Liu 22 Sep '13

22 Sep '13

It's an old issue which discussed in Percona-google-group: https://groups.google.com/forum/?fromgroups#!topic/percona-discussion/z3r4-… The patch provided in that discuss group is a rough idea, based on Percona-Server, and later I almost re-wrote the patch for real-case testing. I think it's more simple to implement in the InnoDB layer, compared to the the idea post by Monty, that introduce 'Incremental lock' or 'Relative update lock' to solve this problem. I backported them to Mariadb-10.0.4, and use simple simulate test with oltp_cc.lua, which is provided as an attachment. These notes migth be helpful to understand of this patch. 1) Why InnoDB's concurrency control is BAD for high concurrency hot key updating case? It's described in the percona-disscuss-group, please refer that. 2) What's the basic logic for high concurrency hot key updating? The idea is still the same as before, but the implement is different now. With a simple case that 1000 threads updating the same record id=1 and threshold=2 to describe the basic idea for this patch. conn (1000) ... (6) -> (5) -> (4) (1) (running) (2) (suspended) (3) (suspended) That is, thread 1 is running, and thread 2 and 3 are suspended, waiting for thread 1 relase the record lock for id=1. For thread 4, it detected they are already two threads suspended, and it would be rescheduled in hash queue. conn (1000) .. (6) -> (5) (1) (running) (2) (suspended) (3) (suspended) bucket1: (4) bucket2: And for 5 and 6 ... 1000, they are also rescheduled in that queue (1) (running) (2) (suspended) (3) (suspended) bucket1: (1000)...(6) -> (5) -> (4) bucket2: After 1 is committed, 2 is waked up and running, then 4 detects it would entry now, so it enters. (1) (commit/exit) (2) (running) (3) (suspended) (4) (suspended) bucket1: (1000) ... (6) -> (5) bucket2: The same logic for 5 and 6...1000. While for the old logic, a snapshot might be: (1) (commit/exit) (2) (running) (3) (suspended) (4) (suspended) (5) (suspended) .... (1000) (suspended) Hope the demo is not too naive. You can see, in such case, there are always threshold threads running inside innodb, while not all are suspended inside. 3) Why not use the new concurrency control with AOTMIC add/sub for active conc? The result is bad for new concurrency control in 5.6/10.0, that no FIFO any more. So I changed to default (5.1/5.5) control strategy again. Note the changes in CMakeLists.txt that make -DUSE_DEFAULT_CONC=ON as default. 4) How about the concurrency counter for rescheduled thread in bucket queue? If thread number waiting for the record lock is over the configured threshold, it would be rescheduled into queue in one hash bucket, hashed by (space,space_no, heap_no) a) for the first one in the queue, that is the queue header, it would timed-wait retrying whether it would enter into innodb layer, by checking the n_ref_count for that record stored by HASH. b) for the non-head threads, they would be FAKE released the concurrent slot, that means threads in FIFO would be waked up, enter/exit innodb with faked inside and tickets. Until they are force to exit innodb, the active slot number is equal to real case without rescheduling. 5) What about the basic test result? $ ./sysbench/sysbench --test=sysbench/tests/db/oltp_cc.lua --num-threads=1000 --mysql-socket=/u01/md1004/run/mysql.sock --mysql-db=bmw --oltp-tables-count=4 --mysql-user=root --oltp-table-size=4 --max-requests=0 --max-time=120000 --oltp-read-only=off run 1000conn, oltp-cc.lua, 4x4(16 hotkey): patched (innodb_lock_wait_threshold=12 as default) : 3.1K QPS, 9.1K TPS unpatched (innodb_lock_wait_threshold=0 as disabled) : 1.2K QPS, 3.5K TPS $ ./sysbench/sysbench --test=sysbench/tests/db/oltp_cc.lua --num-threads=1000 --mysql-socket=/u01/md1004/run/mysql.sock --mysql-db=bmw --oltp-tables-count=1 --mysql-user=root --oltp-table-size=4 --max-requests=0 --max-time=120000 --oltp-read-only=off run 1000conn, oltp-cc.lua, 1x4(4 hotkey): patched (innodb_lock_wait_threshold=12 as default) : 1.3K QPS, 3.5K TPS unpatched (innodb_lock_wait_threshold=0 as disabled) : 200 QPS, 720 TPS 6) Any rest issues? The rescheduling of the waiting thread breaks the normal mechanism: each lock request must be registered in lock-table, then deadlock could be found if exists. As I am trying to reduce n in lock relaetd O(n) and O(n^2) operations, for the deadlock case, it does not work very well. For example, the update sequance: t1 updates 1, and t2 updates 1, then t3/t4/t5/t6 update 1, and then t1 updates 2, after that t2 updates 1: t1 1 2 t2 2 1 t3 1 t4 1 t5 1 t6 1 As we can see, in the new schedule model, t3/t4/t5 are suspended, and might be rescheduled in queue. Then t1 is suspended as it updates 2 which record lock is hold by t2. t2 is rescheduled as it detected too many lock waiting for rec 1. The snapshot might be: (t1[2]) (suspended) (t3) (suspended) (t4) (suspended) bucket1: (t2[1]) -> (t6) -> (t5) bucket2: The deadlock can be break until t2 enters with 1. Then for t5/t6, they would wait until timeout, and then force enter into innodb side, suspended as it should be. It's same for t2. Until t2 enters, the deadlock is found. So, the cost for deadlock detecting might be much large than it should be, depends on the timeout and retry counts. Hope Jan or Monty would have a look for this patch, or other ideas for solve this problem, if you are going.

1 0

[Maria-developers] Prep for MariaDB 5.5.33a
by Daniel Bartholomew 20 Sep '13

20 Sep '13

I've begun the prep for the special 5.5.33a release. Draft changelog and release notes are here: https://mariadb.com/kb/en/mariadb-5533a-changelog/ https://mariadb.com/kb/en/mariadb-5533a-release-notes/ As always, fixes, additions, and such are welcome. I'm currently just waiting for buildbot to finish building and then I'll package up the release so that it can get mirrored over the next few hours and we can release tomorrow after some final testing of the finished repos. Assuming things go well, my plan is to release around noon US Eastern time. Thanks. -- Daniel Bartholomew, Sr Tech Writer / Sys Admin MariaDB

2 1

Re: [Maria-developers] [Commits] Rev 3816: MDEV-4883 - Not all host_cache tests have been merged in lp:maria/10.0
by Roberto Spadim 20 Sep '13

20 Sep '13

Hi guys, what's the "MySQL variables auto-tuning code" means? i didn't saw that in mysql source and at mysql docs, any idea what this means? 2013/9/18 Sergey Vojtovich <svoj(a)mariadb.org> > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 3816 > revision-id: svoj(a)mariadb.org-20130918071716-yu2a5489uldzfuy9 > parent: sanja(a)askmonty.org-20130917190515-g3xncb1slqwy4cix > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-mdev4883 > timestamp: Wed 2013-09-18 11:17:16 +0400 > message: > MDEV-4883 - Not all host_cache tests have been merged > > Merged main.host_cache_size_functionality test. > > Test had to be adjusted because MariaDB doesn't have > MySQL variables auto-tuning code. > === added file 'mysql-test/r/host_cache_size_functionality.result' > --- a/mysql-test/r/host_cache_size_functionality.result 1970-01-01 > 00:00:00 +0000 > +++ b/mysql-test/r/host_cache_size_functionality.result 2013-09-18 > 07:17:16 +0000 > @@ -0,0 +1,78 @@ > +'#________________________VAR_06_Host_Cache_Size__________________#' > +echo '##' > +--echo '#---------------------WL6372_VAR_6_01----------------------#' > +#################################################################### > +# Checking default value # > +#################################################################### > +SELECT COUNT(@@GLOBAL.Host_Cache_Size) > +1 Expected > +set @Default_host_cache_size=128; > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +@@global.Host_Cache_Size=@Default_host_cache_size > +1 > +1 Expected > +'#---------------------WL6372_VAR_6_02----------------------#' > +# Restart server with Host_Cache_Size 1 > +SELECT @@GLOBAL.Host_Cache_Size; > +@@GLOBAL.Host_Cache_Size > +1 > +1 Expected > +set @Default_host_cache_size=128; > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +@@global.Host_Cache_Size=@Default_host_cache_size > +1 > +1 Expected > +'#---------------------WL6372_VAR_6_03----------------------#' > +SET @@local.Host_Cache_Size=1; > +ERROR HY000: Variable 'host_cache_size' is a GLOBAL variable and should > be set with SET GLOBAL > +Expected error 'Global variable' > +SET @@session.Host_Cache_Size=1; > +ERROR HY000: Variable 'host_cache_size' is a GLOBAL variable and should > be set with SET GLOBAL > +Expected error 'Global variable' > +SET @@GLOBAL.Host_Cache_Size=1; > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +COUNT(@@GLOBAL.Host_Cache_Size) > +1 > +1 Expected > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +@@global.Host_Cache_Size=@Default_host_cache_size > +1 > +1 Expected > +'#---------------------WL6372_VAR_6_04----------------------#' > +SELECT @@GLOBAL.Host_Cache_Size = VARIABLE_VALUE > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='Host_Cache_Size'; > +@@GLOBAL.Host_Cache_Size = VARIABLE_VALUE > +1 > +1 Expected > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +COUNT(@@GLOBAL.Host_Cache_Size) > +1 > +1 Expected > +SELECT COUNT(VARIABLE_VALUE) > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='Host_Cache_Size'; > +COUNT(VARIABLE_VALUE) > +1 > +1 Expected > +'#---------------------WL6372_VAR_6_05----------------------#' > +SELECT @@Host_Cache_Size = @@GLOBAL.Host_Cache_Size; > +@@Host_Cache_Size = @@GLOBAL.Host_Cache_Size > +1 > +1 Expected > +SELECT COUNT(@@local.Host_Cache_Size); > +ERROR HY000: Variable 'host_cache_size' is a GLOBAL variable > +Expected error 'Variable is a GLOBAL variable' > +SELECT COUNT(@@SESSION.Host_Cache_Size); > +ERROR HY000: Variable 'host_cache_size' is a GLOBAL variable > +Expected error 'Variable is a GLOBAL variable' > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +COUNT(@@GLOBAL.Host_Cache_Size) > +1 > +1 Expected > +SELECT Host_Cache_Size = @@SESSION.Host_Cache_Size; > +ERROR 42S22: Unknown column 'Host_Cache_Size' in 'field list' > +Expected error 'Unknown column Host_Cache_Size in field list' > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > > === added file 'mysql-test/t/host_cache_size_functionality.test' > --- a/mysql-test/t/host_cache_size_functionality.test 1970-01-01 > 00:00:00 +0000 > +++ b/mysql-test/t/host_cache_size_functionality.test 2013-09-18 > 07:17:16 +0000 > @@ -0,0 +1,185 @@ > > +############################################################################### > +# > # > +# Variable Name: Host_Cache_Size > # > +# Scope: Global > # > +# Access Type: Dynamic > # > +# Data Type: numeric > # > +# > # > +# > # > +# Creation Date: 2012-08-31 > # > +# Author : Tanjot Singh Uppal > # > +# > # > +# > # > +# Description:Test Cases of Dynamic System Variable Host_Cache_Size > # > +# that checks the behavior of this variable in the following > ways # > +# * Value Check > # > +# * Scope Check > # > +# * Functionality Check > # > +# * Accessability Check > # > +# > # > +# This test does not perform the crash recovery on this variable > # > +# For crash recovery test on default change please run the ibtest > # > > +############################################################################### > + > +-- source include/have_innodb.inc > +-- source include/not_embedded.inc > +-- source include/have_innodb_16k.inc > +--disable_warnings > +echo '#________________________VAR_06_Host_Cache_Size__________________#' > +echo '##' > +--echo '#---------------------WL6372_VAR_6_01----------------------#' > +#################################################################### > +# Checking default value # > +#################################################################### > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +--echo 1 Expected > + > +#set @Default_host_cache_size=(select > if(if(@@global.max_connections<500,128+@@global.max_connections,128+@ > @global.max_connections+floor((@@global.max_connections-500)/20))>2000,2000,if(@@global.max_connections<500,128+@ > @global.max_connections,128+@ > @global.max_connections+floor((@@global.max_connections-500)/20)))); > +set @Default_host_cache_size=128; > + > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +--echo 1 Expected > + > + > +--echo '#---------------------WL6372_VAR_6_02----------------------#' > > +################################################################################# > +# Checking the Default value post starting the server with other value > # > > +################################################################################# > +--echo # Restart server with Host_Cache_Size 1 > + > +let $restart_file= $MYSQLTEST_VARDIR/tmp/mysqld.1.expect; > +--exec echo "wait" > $restart_file > +--shutdown_server 10 > +--source include/wait_until_disconnected.inc > +-- exec echo "restart:--host_cache_size=1 " > > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +-- enable_reconnect > +-- source include/wait_until_connected_again.inc > + > +--disable_warnings > + > +SELECT @@GLOBAL.Host_Cache_Size; > +--echo 1 Expected > + > +#set @Default_host_cache_size=(select > if(if(@@global.max_connections<500,128+@@global.max_connections,128+@ > @global.max_connections+floor((@@global.max_connections-500)/20))>2000,2000,if(@@global.max_connections<500,128+@ > @global.max_connections,128+@ > @global.max_connections+floor((@@global.max_connections-500)/20)))); > +set @Default_host_cache_size=128; > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +--echo 1 Expected > + > + > +--echo '#---------------------WL6372_VAR_6_03----------------------#' > +#################################################################### > +# Checking Value can be set - Dynamic # > +#################################################################### > +--error ER_GLOBAL_VARIABLE > +SET @@local.Host_Cache_Size=1; > +--echo Expected error 'Global variable' > + > +--error ER_GLOBAL_VARIABLE > +SET @@session.Host_Cache_Size=1; > +--echo Expected error 'Global variable' > + > +SET @@GLOBAL.Host_Cache_Size=1; > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > + > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +--echo 1 Expected > + > +select @@global.Host_Cache_Size=@Default_host_cache_size; > +--echo 1 Expected > + > +--echo '#---------------------WL6372_VAR_6_04----------------------#' > +################################################################# > +# Check if the value in GLOBAL Table matches value in variable # > +################################################################# > +SELECT @@GLOBAL.Host_Cache_Size = VARIABLE_VALUE > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='Host_Cache_Size'; > +--echo 1 Expected > + > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +--echo 1 Expected > + > +SELECT COUNT(VARIABLE_VALUE) > +FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES > +WHERE VARIABLE_NAME='Host_Cache_Size'; > +--echo 1 Expected > + > +--echo '#---------------------WL6372_VAR_6_05----------------------#' > > +################################################################################ > +# Checking Variable Scope > # > > +################################################################################ > +SELECT @@Host_Cache_Size = @@GLOBAL.Host_Cache_Size; > +--echo 1 Expected > + > +--Error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SELECT COUNT(@@local.Host_Cache_Size); > +--echo Expected error 'Variable is a GLOBAL variable' > + > +--Error ER_INCORRECT_GLOBAL_LOCAL_VAR > +SELECT COUNT(@@SESSION.Host_Cache_Size); > +--echo Expected error 'Variable is a GLOBAL variable' > + > +SELECT COUNT(@@GLOBAL.Host_Cache_Size); > +--echo 1 Expected > + > +--Error ER_BAD_FIELD_ERROR > +SELECT Host_Cache_Size = @@SESSION.Host_Cache_Size; > +--echo Expected error 'Unknown column Host_Cache_Size in field list' > + > +#The below check has been commented out as the IP fetch is different in a > P2P connection than BroadBand connection > +#--echo '#---------------------WL6372_VAR_6_06----------------------#' > > +############################################################################### > +# Checking the Host cahce functionality > # > > +############################################################################### > + > +#SET @@GLOBAL.Host_Cache_Size=2; > +#--disable_warnings > + > +#--perl > +#my $ip=`ifconfig | egrep "inet addr|inet" | sed -e 's/^.*inet addr://' > -e 's/^.*inet//'| sed 's/ .*\$//'|egrep -i "broadcast|bcast"|head -1|awk > '{print $1}'`; > +#open (LOGFH, ">" . $ENV{'MYSQL_TMP_DIR'} . "/bind_ip"); > +#print LOGFH "let \$bind_ip = $ip;\n"; > +#close LOGFH; > +#EOF > + > +#--source $MYSQL_TMP_DIR/bind_ip > +#--remove_file $MYSQL_TMP_DIR/bind_ip > + > +#let $restart_file= $MYSQLTEST_VARDIR/tmp/mysqld.1.expect; > +#--exec echo "wait" > $restart_file > +#--shutdown_server 10 > +#--source include/wait_until_disconnected.inc > +#-- exec echo "restart:--bind-address=$bind_ip " > > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +#-- enable_reconnect > +#-- source include/wait_until_connected_again.inc > + > +#connection default; > +#--disable_warnings > + > +#create user binduser; > +#grant all on *.* to binduser; > + > +#select count(IP) from performance_schema.host_cache; > +#--echo 0 Expected > + > +#connect (con1,$bind_ip,binduser,,); > +#select count(IP) from performance_schema.host_cache; > +#--echo 1 Expected > + > +#disconnect con1; > +#connection default; > + > +#--disable_warnings > + > +# The below check is hashed until the BUG14689561# is fixed > +# > +#flush hosts; > + > +#connect (con2,$bind_ip,binduser,,); > +#select count(IP) from performance_schema.host_cache; > +#--echo 1 Expected > + > +#disconnect con2; > + > +SET @@GLOBAL.Host_Cache_Size=DEFAULT; > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits > -- Roberto Spadim SPAEmpresarial

2 1

Re: [Maria-developers] Rev 3814: Client attributes
by Sergei Golubchik 19 Sep '13

19 Sep '13

Hi, Sanja! On Sep 19, sanja(a)askmonty.org wrote: > === modified file 'sql/sql_acl.cc' > --- a/sql/sql_acl.cc 2013-07-21 14:39:19 +0000 > +++ b/sql/sql_acl.cc 2013-09-19 10:19:36 +0000 > @@ -4615,7 +4615,8 @@ bool check_grant(THD *thd, ulong want_ac > case ACL_INTERNAL_ACCESS_GRANTED: > /* > Currently, > - - the information_schema does not subclass ACL_internal_table_access, > + - the information_schema does not subclass ACL_internal_table_access > +, Eh? What was that? > there are no per table privilege checks for I_S, > - the performance schema does use per tables checks, but at most > returns 'CHECK_GRANT', and never 'ACCESS_GRANTED'. > @@ -8222,6 +8223,43 @@ static bool find_mpvio_user(MPVIO_EXT *m > mpvio->acl_user->plugin.str)); > DBUG_RETURN(0); > } > + > +static bool > +read_client_connect_attrs(char **ptr, size_t *max_bytes_available, > + const CHARSET_INFO *from_cs) > +{ > + size_t length, length_length; > + char *ptr_save; > + /* not enough bytes to hold the length */ > + if (*max_bytes_available < 1) > + return true; > + > + /* read the length */ > + ptr_save= *ptr; > + length= net_field_length_ll((uchar **) ptr); > + length_length= *ptr - ptr_save; > + if (*max_bytes_available < length_length) > + return true; This is incorrect, it might read past the end of the buffer. it's too late to check the length *after* you've read the data. Correct code might look like if (*max_bytes_available >= 9) { ptr_save= *ptr; length= net_field_length_ll((uchar **) ptr); length_length= *ptr - ptr_save; DBUG_ASSERT(length_length <= 9); *max_bytes_available-= length_length; } else { char buf[10], *len_ptr= buf; strncpy(buf, *ptr, 9) length= net_field_length_ll((uchar **) &len_ptr); length_length= len_ptr - buf; *ptr+= length_length; if (*max_bytes_available < length_length) return true; *max_bytes_available-= length_length; } > + > + *max_bytes_available-= length_length; > + > + /* length says there're more data than can fit into the packet */ > + if (length > *max_bytes_available) > + return true; > + > + /* impose an artificial length limit of 64k */ > + if (length > 65535) > + return true; > + > +#ifdef HAVE_PSI_THREAD_INTERFACE > + if (PSI_THREAD_CALL(set_thread_connect_attrs)(*ptr, length, from_cs) && > + current_thd->variables.log_warnings) > + sql_print_warning("Connection attributes of length %lu were truncated", > + (unsigned long) length); > +#endif > + return false; > +} > + > #endif > > /* the packet format is described in send_change_user_packet() */ > @@ -8349,6 +8387,13 @@ static bool parse_com_change_user_packet > } > } > > + size_t bytes_remaining_in_packet= end - ptr; 1. This is very wrong, did you ever test it? You need a pointer to *after* the plugin name. But 'ptr' is two bytes *before* the plugin. 2. remove bytes_remaining_in_packet, pass 'end' instead to read_client_connect_attrs(). > + > + if ((mpvio->thd->client_capabilities & CLIENT_CONNECT_ATTRS) && > + read_client_connect_attrs(&ptr, &bytes_remaining_in_packet, > + mpvio->thd->charset())) > + return packet_error; > + > DBUG_PRINT("info", ("client_plugin=%s, restart", client_plugin)); > /* > Remember the data part of the packet, to present it to plugin in > @@ -8487,7 +8532,21 @@ static ulong parse_client_handshake_pack > /* strlen() can't be easily deleted without changing protocol */ > db_len= db ? strlen(db) : 0; > > - char *client_plugin= passwd + passwd_len + (db ? db_len + 1 : 0); > + char *client_plugin= end= passwd + passwd_len + (db ? db_len + 1 : 0); > + > + if ((thd->client_capabilities & CLIENT_PLUGIN_AUTH) && > + (client_plugin < (char *)net->read_pos + pkt_len)) > + { > + end+= strlen(end) + 1; > + } > + > + size_t bytes_remaining_in_packet= (((char *)net->read_pos) + pkt_len) - end; This is very-very wrong. if there're no connection attributes, then your 'end' will be after the end of the packet, it'll be after net->read_pos + pkt_len. But then you subtract, get -1 and cast it to unsigned size_t ! Don't worry, when you remove bytes_remaining_in_packet as I wrote above, this bug will go away. > + > + if (db && > + (thd->client_capabilities & CLIENT_CONNECT_ATTRS) && > + read_client_connect_attrs(&end, &bytes_remaining_in_packet, > + mpvio->thd->charset())) > + return packet_error; Put it in the same place as in parse_com_change_user_packet, that is after reading the plugin name (and fix_plugin_ptr). And rename 'end' to 'connection_attributes' > > /* Since 4.1 all database names are stored in utf8 */ > if (db) > > === modified file 'sql/sys_vars.cc' > --- a/sql/sys_vars.cc 2013-08-14 08:48:50 +0000 > +++ b/sql/sys_vars.cc 2013-09-19 10:19:36 +0000 > @@ -71,6 +71,10 @@ > #define export /* not static */ > > #ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > +#ifndef EMBEDDED_LIBRARY > +#define PFS_TRAILING_PROPERTIES \ > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(NULL), ON_UPDATE(NULL), \ > + NULL, sys_var::PARSE_EARLY Remove PFS_TRAILING_PROPERTIES define (not used in MariaDB) > > static Sys_var_mybool Sys_pfs_enabled( > "performance_schema", > @@ -331,8 +335,10 @@ static Sys_var_long Sys_pfs_connect_attr > GLOBAL_VAR(pfs_param.m_session_connect_attrs_sizing), > CMD_LINE(REQUIRED_ARG), VALID_RANGE(-1, 1024 * 1024), > DEFAULT(-1), > - BLOCK_SIZE(1)); > + BLOCK_SIZE(1), > + NO_MUTEX_GUARD, NOT_IN_BINLOG); Remove this change too (sysvars are NO_MUTEX_GUARD and NOT_IN_BINLOG by default) > > +#endif /* EMBEDDED_LIBRARY */ > #endif /* WITH_PERFSCHEMA_STORAGE_ENGINE */ > > static Sys_var_ulong Sys_auto_increment_increment( > > === modified file 'libmysqld/lib_sql.cc' > --- a/libmysqld/lib_sql.cc 2013-08-01 12:09:26 +0000 > +++ b/libmysqld/lib_sql.cc 2013-09-19 10:19:36 +0000 > @@ -698,12 +698,38 @@ err: > } > > > +static void > +emb_transfer_connect_attrs(MYSQL *mysql) > +{ > +#ifdef HAVE_PSI_THREAD_INTERFACE > + if (mysql->options.extension && > + mysql->options.extension->connection_attributes_length) > + { > + uchar *buf, *ptr; > + THD *thd= (THD*)mysql->thd; > + size_t length= mysql->options.extension->connection_attributes_length; > + > + /* 9 = max length of the serialized length */ > + ptr= buf= (uchar *) my_alloca(length + 9); > + send_client_connect_attrs(mysql, buf); > + net_field_length_ll(&ptr); > + PSI_THREAD_CALL(set_thread_connect_attrs)((char *) ptr, length, thd->charset()); > + my_afree(buf); > + } > +#endif > +} > + > + > #ifdef NO_EMBEDDED_ACCESS_CHECKS > int check_embedded_connection(MYSQL *mysql, const char *db) > { > int result; > LEX_STRING db_str = { (char*)db, db ? strlen(db) : 0 }; > THD *thd= (THD*)mysql->thd; > + > + /* the server does the same as the client */ > + mysql->server_capabilities= mysql->client_flag; > + Remove the similar line in init_embedded_mysql() > thd_init_client_charset(thd, mysql->charset->number); > thd->update_charset(); > Security_context *sctx= thd->security_ctx; > @@ -728,11 +755,17 @@ int check_embedded_connection(MYSQL *mys > we emulate a COM_CHANGE_USER user here, > it's easier than to emulate the complete 3-way handshake > */ > - char buf[USERNAME_LENGTH + SCRAMBLE_LENGTH + 1 + 2*NAME_LEN + 2], *end; > + char *buf, *end; > NET *net= &mysql->net; > THD *thd= (THD*)mysql->thd; > Security_context *sctx= thd->security_ctx; > + size_t connect_attrs_len= > + (mysql->server_capabilities & CLIENT_CONNECT_ATTRS && This doesn't seem to make any sense. Why does it check for CLIENT_CONNECT_ATTRS if it's part of the server (embedded). If a client connects to the server, then the client needs to check whether the server supports CLIENT_CONNECT_ATTRS, and the server needs to check whether the client supports it. But a server doesn't need to check whether it itself support it, it knows it already :) > + mysql->options.extension) ? > + mysql->options.extension->connection_attributes_length : 0; > > + buf= my_alloca(USERNAME_LENGTH + SCRAMBLE_LENGTH + 1 + 2*NAME_LEN + 2 + > + connect_attrs_len + 2); > if (mysql->options.client_ip) > { > sctx->host= my_strdup(mysql->options.client_ip, MYF(0)); > @@ -766,6 +799,13 @@ int check_embedded_connection(MYSQL *mys > int2store(end, (ushort) mysql->charset->number); > end+= 2; > > + //end= strmake(end, "mysql_native_password", NAME_LEN) + 1; Why is it commented out? > + > + /* the server does the same as the client */ > + mysql->server_capabilities= mysql->client_flag; > + > + end= (char *) send_client_connect_attrs(mysql, (uchar *) end); > + > /* acl_authenticate() takes the data from thd->net->read_pos */ > thd->net.read_pos= (uchar*)buf; Regards, Sergei

1 1

[Maria-developers] Please review: EXPLAIN UPDATE/DELETE, post-review changes.
by Sergei Petrunia 17 Sep '13

17 Sep '13

Hello Igor, I think I'm ready to submit the EXPLAIN UPDATE/DELETE code refactoring patch for another round of review. The code is pushed to lp:~~maria-captains/maria/10.0-base-explain-update-r2/ . I am also attaching the combined patch (against 10.0-base). BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] BUG? at windows versino of mariadb 10.0.4
by Roberto Spadim 16 Sep '13

16 Sep '13

Hi guys, i was trying the mariadb with lower value of thread_pool_max_threads to 3 and after my fourth connection i got a error and server crashed, it's a new server, i downloaded the 32bits version of mariadb, unziped and run this command line: C:\Program Files\mariadb-10.0.4-win32\bin>mysqld --datadir=..\data\ --log_error=..\log --port=3306 --thread_pool_max_threads=3 the error output at log was: 130915 21:43:37 [ERROR] mysqld got exception 0x80000003 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.4-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=4 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 141750 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... mysqld.exe!os_aio_windows_handle()[os0file.cc:4786] mysqld.exe!fil_aio_wait()[fil0fil.cc:5396] mysqld.exe!io_handler_thread()[srv0start.cc:477] kernel32.dll!BaseThreadInitThunk() ntdll.dll!RtlInitializeExceptionChain() ntdll.dll!RtlInitializeExceptionChain() The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. any help is wellcome, maybe the max_threads is bigger than thread_pool_max_threads? that's could be the problem? i didn't checked the source yet -- Roberto Spadim SPAEmpresarial

2 2