- developers - lists.mariadb.org

Re: [Maria-developers] [Commits] Rev 3710: MariaDB made be compiled by gcc 4.8.1
by Sergei Golubchik 24 Oct '13

24 Oct '13

Hi, Sanja! On Oct 24, sanja(a)montyprogram.com wrote: > At file:///home/bell/maria/bzr/work-maria-5.3-test/ > > ------------------------------------------------------------ > revno: 3710 > revision-id: sanja(a)montyprogram.com-20131024170011-r5r6fd84y4x1yr09 > parent: bar(a)mnogosearch.org-20131021093629-0nyk7osjssern1sa > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.3-test > timestamp: Thu 2013-10-24 20:00:11 +0300 > message: > MariaDB made be compiled by gcc 4.8.1 Okay, that looks much better. Please push it in a stage tree, just to check that it doesn't break on all other compilers that we use, and then push into main. Thanks! Regards, Sergei

1 0

Re: [Maria-developers] MariaDB made be compiled by gcc 4.8.1
by Sergei Golubchik 24 Oct '13

24 Oct '13

Hi, Sanja! > >>> === modified file 'storage/maria/ma_checksum.c' > >>> --- a/storage/maria/ma_checksum.c 2008-04-03 13:40:25 +0000 > >>> +++ b/storage/maria/ma_checksum.c 2013-10-22 07:56:05 +0000 > >>> @@ -58,7 +58,7 @@ ha_checksum _ma_checksum(MARIA_HA *info, > >>> length= _ma_calc_blob_length(blob_size_length, pos); > >>> if (length) > >>> { > >>> - memcpy((char*) &pos, pos + blob_size_length, sizeof(char*)); > >>> + memcpy((char*) &pos, pos + blob_size_length, (sizeof(char*))); > >> Why? > > ma_checksum.c: In function '_ma_checksum': > > ma_checksum.c:61:60: error: argument to 'sizeof' in 'memcpy' call is the > > same pointer type 'char *' as the destination; expected 'char' or an > > explicit length [-Werror=sizeof-pointer-memaccess] > > memcpy((char*) &pos, pos + blob_size_length, sizeof(char*)); > > ^ > > shouldn't it be: > memcpy((char**) &pos, pos + blob_size_length, sizeof(char*)); Sanja ^^^ That would be a better fix. The result looks more logical than with meaningless parentheses. Or, I suppose, you can try to drop the cast completely: memcpy(&pos, pos + blob_size_length, sizeof(pos)); > > BTW all other such fixes was done because this check. Yes, I've figured it out. Regards, Sergei

2 2

[Maria-developers] hash + like
by Roberto Spadim 24 Oct '13

24 Oct '13

Hi guys, how could i optimize a hash index using like? i was thinking about something like a index over first left bytes using hash, example: field LIKE "abc%" create index hash_index( field(3) ) using hash any idea if this could be possible? this could boost some queries here that i'm using btree today i'm using the LIKE, but i'm considering a virtual column + a index over virtual column + a ( AND virtual_column='abc') any other idea? or maybe optimize the LIKE function -- Roberto Spadim

2 1

Re: [Maria-developers] MariaDB made be compiled by gcc 4.8.1
by Sergei Golubchik 23 Oct '13

23 Oct '13

Hi, Sanja! See below On Oct 22, sanja(a)montyprogram.com wrote: > message: > MariaDB made be compiled by gcc 4.8.1 > === modified file 'extra/yassl/include/yassl_int.hpp' > --- a/extra/yassl/include/yassl_int.hpp 2012-04-05 08:49:38 +0000 > +++ b/extra/yassl/include/yassl_int.hpp 2013-10-22 07:56:05 +0000 > @@ -440,7 +440,7 @@ public: > const Ciphers& GetCiphers() const; > const DH_Parms& GetDH_Parms() const; > const Stats& GetStats() const; > - const VerifyCallback getVerifyCallback() const; > + VerifyCallback getVerifyCallback() const; Why? > === modified file 'storage/maria/ma_checksum.c' > --- a/storage/maria/ma_checksum.c 2008-04-03 13:40:25 +0000 > +++ b/storage/maria/ma_checksum.c 2013-10-22 07:56:05 +0000 > @@ -58,7 +58,7 @@ ha_checksum _ma_checksum(MARIA_HA *info, > length= _ma_calc_blob_length(blob_size_length, pos); > if (length) > { > - memcpy((char*) &pos, pos + blob_size_length, sizeof(char*)); > + memcpy((char*) &pos, pos + blob_size_length, (sizeof(char*))); Why? > crc= my_checksum(crc, pos, length); > } > continue; Regards, Sergei

2 1

[Maria-developers] gcc warning
by Roberto Spadim 20 Oct '13

20 Oct '13

hi guys! just a warning , bye :) [ 99%] Building C object tests/CMakeFiles/bug25714.dir/bug25714.c.o /home/src/mariadb/mariadb-5.5.33a/tests/bug25714.c: In function 'main': /home/src/mariadb/mariadb-5.5.33a/tests/bug25714.c:26:7: warning: variable 'OK' set but not used [-Wunused-but-set-variable] int OK; ^ -- Roberto Spadim

1 0

Re: [Maria-developers] Comments for parallel replication patch (MDEV-4506)
by Kristian Nielsen 18 Oct '13

18 Oct '13

Michael Widenius <michael.widenius(a)gmail.com> writes: >> What about Rotate_log_event::do_update_pos() and >> Stop_log_event::do_update_pos()? >> >> They currently access the flag in rli. I am not sure that is right, as > this >> could be far ahead of the event they are actually executing? > > Aren't the above events handled by the sql driver thread? No, by the worker threads. I do not understand how it could be done in the driver thread? do_update_pos updates the current position of the last event executed, it cannot happen until after events have been executed? And this happens in the worker threads. > My thought was that the driver thread was in charge of all reading of the > relay log and would thus hide any stop or rotate events from the sql > execution threads. The Rotate event is what updates the group_master_log_name to the name of the next binlog file on the master. This must not happen until previous events have been executed (otherwise they will be wrongly skipped if we stop and restart the slave). Or did I misunderstand? My best suggestion at the moment is to copy the required information from m_flags (and a couple of other fields from rli) into rgi, so that the right value gets associated with the right event in each worker thread. But let's discuss when you are back from travelling if you have a better idea. > The purgeing of log should be changed to check the position of the st > committed event. It's safe to delay any older logs. > The other I am not sure about as I can't look at the code just know. > However we need a solution that works also for masters that don't support > GTID. Yes, the non-GTID case is the main problem. (Using GTID, everything is updated automatically in parallel in the mysql.gtid_slave_pos and there is a sub_id field that ensures we can figure out which GTIDs are the most recent ones). - Kristian.

1 0

[Maria-developers] Parallel replication MDEV-4506
by Michael Widenius 14 Oct '13

14 Oct '13

Hi! Just a quick status update of parallel replication in MariaDB 10.0 I yesterday pushed to the 10.0-knielsen tree changes that makes many of the replication variables thread safe. I will today push a patch that should fix the rest of the variables. Here is the changelog entry for this: Fixes for parallel slave: - Made slaves temporary table multi-thread slave safe by adding mutex around save_temporary_table usage. - rli->save_temporary_tables is the active list of all used temporary tables - This is copied to THD->temporary_tables when temporary tables are opened and updated when temporary tables are closed - Added THD->lock_temporary_tables() and THD->unlock_temporary_tables() to simplify this. - Relay_log_info->sql_thd renamed to Relay_log_info->sql_driver_thd to avoid wrong usage for merged code. - Added is_part_of_group() to mark functions that are part of the next function. This replaces setting IN_STMT when events are executed. - Added is_begin(), is_commit() and is_rollback() functions to Query_log_event to simplify code. - If slave_skip_counter is set run things in single threaded mode. This simplifies code for skipping events. - Updating state of relay log (IN_STMT and IN_TRANSACTION) is moved to one single function: update_state_of_relay_log() We can't use OPTION_BEGIN to check for the state anymore as the sql_driver and sql execution threads may be different. Clear IN_STMT and IN_TRANSACTION in init_relay_log_pos() and Relay_log_info::cleanup_context() to ensure the flags doesn't survive slave restarts is_in_group() is now independent of state of executed transaction. - Reset thd->transaction.all.modified_non_trans_table() if we did set it for single table row events. This was mainly for keeping the flag as documented. - Changed slave_open_temp_tables to uint32 to be able to use atomic operators on it. - Relay_log_info::sleep_lock -> rpl_group_info::sleep_lock - Relay_log_info::sleep_cond -> rpl_group_info::sleep_cond - Changed some functions to take rpl_group_info instead of Relay_log_info to make them multi-slave safe and to simplify usage - do_shall_skip() - continue_group() - sql_slave_killed() - next_event() - Simplifed arguments to io_slave_killed(), check_io_slave_killed() and sql_slave_killed(); No reason to supply THD as this is part of the given structure. - set_thd_in_use_temporary_tables() removed as in_use is set on usage - Added information to thd_proc_info() which thread is waiting for slave mutex to exit. - In open_table() reuse code from find_temporary_table() Other things: - More DBUG statements - Fixed the rpl_incident.test can be run with --debug - More comments - Disabled not used function rpl_connect_master() The TODO for parallel replication is documented at top of rpl_parallel.cc. Here is the comment: ------- - Error handling. If we fail in one of multiple parallel executions, we need to make a best effort to complete prior transactions and roll back following transactions, so slave binlog position will be correct. And all the retry logic for temporary errors like deadlock. - Stopping the slave needs to handle stopping all parallel executions. And the logic in sql_slave_killed() that waits for current event group to complete needs to be extended appropriately... - Audit the use of Relay_log_info::data_lock. Make sure it is held correctly in all needed places also when using parallel replication. - We need some user-configurable limit on how far ahead the SQL thread will fetch and queue events for parallel execution (otherwise if slave gets behind we will fill up memory with pending malloc()'ed events). - Fix update of relay-log.info and master.info. In non-GTID replication, they must be serialised to preserve correctness. In GTID replication, we should not update them at all except at slave thread stop. - All the waits (eg. in struct wait_for_commit and in rpl_parallel_thread_pool::get_thread()) need to be killable. And on kill, everything needs to be correctly rolled back and stopped in all threads, to ensure a consistent slave replication state. - Handle the case of a partial event group. This occurs when the master crashes in the middle of writing the event group to the binlog. The slave rolls back the transaction; parallel execution needs to be able to deal with this wrt. commit_orderer and such. - We should notice if the master doesn't support GTID, and then run in single threaded mode against that master. This is needed to be able to support multi-master-replication with old and new masters. - Retry of failed transactions is not yet implemented for the parallel case. ---------- What this means is: In theory things should work, as long as there is no problems in the binary or relay log and we don't fill up memory with too many events. We will merge this branch to 10.0-base and then to 10.0 withing the next few days and start testing it. We also plan to release a beta of 10.0 with this code ASAP as 10.0 is now feature complete. During the beta phase we will fix the above outstanding issues. Kristian will this week start working on the error handling. After that we have to look at the memory consumption (ie, not give the sql execution threads more work than they can handle). I will shortly update the following article with information about how to use parallel replication. https://mariadb.com/kb/en/parallel-replication/ The task itself is documented at: https://mariadb.atlassian.net/browse/MDEV-4506 Regards, Monty

1 0

[Maria-developers] Wrong SQL thread position reporting after IO thread restart
by Pavel Ivanov 11 Oct '13

11 Oct '13

Krisitan, I found what I think is a bug in IO and SQL thread accounting. How to reproduce: 1) Set up two servers S1 and S2. S1 is a master, S2 is slave with master_using_gtid = current_pos. 2) Execute some transactions on the master, e.g. create database d; create table d.t (n int); insert into d.t values (1); 3) Both servers are at 0-1-3 now, SHOW SLAVE STATUS on S2 shows Read_Master_Log_Pos equals to Exec_Master_Log_Pos. 4) Execute STOP SLAVE IO_THREAD on S2. 5) S2 reports in the logs: "Slave I/O thread exiting, ... GTID position 0-1-2". So IO thread didn't realize that it received full transaction for 0-1-3 even though it didn't receive next GTID event. 6) Execute START SLAVE IO_THREAD on S2. 7) At this point SHOW SLAVE STATUS on S2 shows Read_Master_Log_Pos the same as in step 3, but Exec_Master_Log_Pos is now less than in step 3 as if SQL thread didn't catch up with IO thread yet. But despite both threads running and no more transactions executed on master Exec_Master_Log_Pos doesn't change and doesn't become equal to Read_Master_Log_Pos. This apparently happens because IO thread restarts from one transaction behind, adds to relay log Rotate event that master sends with the position of that transaction, but then doesn't add any events for the transaction because it knows they already were added into relay log. I think both problems are bugs. And although after fixing the first it would be really hard (if possible) to reproduce second, I'd think the reporting of SQL thread's position still should be fixed. Thank you, Pavel

2 2

Re: [Maria-developers] [Commits] Rev 3702: Fixed bug mdev-5105. in file:///home/igor/maria/maria-5.3-mdev5105/
by Oleksandr Byelkin 10 Oct '13

10 Oct '13

Hi! 10.10.2013 20:08, Igor Babaev пишет: > At file:///home/igor/maria/maria-5.3-mdev5105/ > > ------------------------------------------------------------ > revno: 3702 > revision-id: igor(a)askmonty.org-20131010170826-dl2xwc048hvnxmau > parent: igor(a)askmonty.org-20131004165107-0hdknrn9111ojupx > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-mdev5105 > timestamp: Thu 2013-10-10 10:08:26 -0700 > message: > Fixed bug mdev-5105. > The bug caused a memory overwrite in the function update_ref_and_keys() > It happened due to a wrong value of SELECT_LEX::cond_count. This value > historically was calculated by the fix_fields method. Now the logic of > calling this method became too complicated and, as a result, this value > is calculated not always correctly. > The patch changes the way how and when the values of SELECT_LEX::cond_count > and of SELECT_LEX::between_count are calculated. The new code does it just at > the beginning of update_ref_and_keys(). > > OK to push.

1 0

[Maria-developers] On predicate pushdown in presense of join buffering.
by Sergei Petrunia 10 Oct '13

10 Oct '13

Hello, Here is the result of discussion on Join Buffering + Static Predicate Pushdown. == Problem setting == Consider a query using join buffering +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 2 | Using where; Using join buffer (flat, BNL join) | | 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | 2 | Using join buffer (flat, BNL join) | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ Consider a predicate "P(t2)". Join buffering works as follows: read records from table t1 and put them into buffer; for each record in table t2 { if (cond(t2)) { for each record in the join buffer { if (cond(t1, t2) { pass record for further processing } } } } cond(2) is stored in "join_tab->cache_select->cond" cond(t1, t2) is stored in join_tab->select_cond". The question is, how should Static Predicate Pushdown work with join buffering? Should the condition be put into cache_select->cond or into select_cond of some later join_tab? Currently, the server will put the predicate P(t2) into both join_tab[t2]->select_cond and join_tab[t2]->cache_select->cond. (And subquery cache will wrap them into different caches) == Solution == First observation: if the expensive subquery predicate depends on several tables (not counting the const tables) then it will never be put into cache_select->cond. join_tab[$TBL]->cache_select->cond can only have conditions that depend only on table $TBL. If a predicate P(T) depends only on table $T, it can be put into 1. join_tab[$T]->cache_select->cond 2. join_tab[$T]->select_cond 3. join_tab[$T+1]->select_cond 4. join_tab[$T+2]->select_cond ... Putting the predicate into both #1 and #2 does not make sense. We have decided that Timour will remove duplicate predicate pushdown from cache_select and select_cond. After that feature, the predicates in cache_select->cond will not have duplicates in cache_select. That way, the options for Static Predicate Pushdown will be: 1. join_tab[$T]->cache_select->cond 3. join_tab[$T+1]->select_cond 4. join_tab[$T+2]->select_cond ... === Needed adjustments in optimizer === Consider the EXPLAIN again: +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 2 | Using where; Using join buffer (flat, BNL join) | | 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | 2 | Using join buffer (flat, BNL join) | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ and a predicate P(t2). Currently, Static Predicate Pushdown will consider putting the predicate at: t2->select_cond: fanout(t1) * fanout(t2) predicate invocations t3->select_cond: fanout(t1) * fanout(t2)* fanout(t3) predicate invocations (here, fanout takes into account selectivity, so it can be that fanout <1) If P(t2) is not attached to t2->select_cond but to t2->cache_select->cond instead, the number of invocations will change to: t2->cache_select->cond: fanout(t1) / number_of_t1_records_in_join_buffer * fanout(t2). The above is obvious for non-BKA case. ==== A note about subquery predicate cache ==== [A theoretical comment] We're not going to do this, but: if P(t2) was attached [ONLY] to t2->select_cond, execution of the query will cause these invocations: P(t2.row1) ---+ P(t2.row1) |-- number_of_t1_records_in_join_buffer times ... | P(t2.row1) ----+ P(t2.row2) ----+ P(t2.row2) |-- number_of_t1_records_in_join_buffer times ... | P(t2.row2) ----+ ... etc. That is, subquery predicate cache will reduce the number of times the predicate is executed. == Applicability for join algorithms == Basic idea: when join buffer is using BKA, join_tab->cache_select==NULL, there is no cache_select. Details: Join algorithms where join_tab->cache_selct->cond is used: - JOIN_CACHE_BNL - JOIN_CACHE_BNLH - JOIN_CACHE_BKAH BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0