- developers - lists.mariadb.org

[Maria-developers] problem with the linux thread pool in mariadb 5.5
by Rich Prohaska 16 Jan '14

16 Jan '14

we have a customer that has observed problems related to the interaction of the thread pools in mariadb 5.5.30 and tokudb 7.1.0. the problem is that connections that stuck for 100's of seconds in the Killed state when a big tokudb transaction commit is in progress. the customer claims that this problem was resolved by turning the thread pool OFF. have there been any fixes to the thread pool implementation post the 5.5.30 release? here is some information that may be useful. the processlist showed 2838 client connections of which: 851 Killed 483 Sleep 1467 Connect 1 show processlist 1 processing a commit on a big delete from a tokudb table 35 blocked on a row lock held by the big delete the Killed connections look like: 84847752 sfi_mysql 10.0.0.60:1814 sfi Killed 93 NULL 0.000 with the connection address and time slightly different a gdb snapshot of the system at the time of the failure: 291 total threads 130 threads waiting on fil_aio_wait 52 threads waiting on tokudb's work_on_kibbutz 16 thread tokudb deserialization thread waiting 36 mysql update thread blocked on the tokudb lock tree 1thread executing a big tokudb delete transaction 40 threads in the thread pool get_event function 1 tokudb checkpoint thread blocked waiting for the LP MO lock, which is held by the big txn commit thread that leaves 15 misc worker threads that are not doing anything interesting the gdb stacks for the thread pool threads are: Thread 78 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 71 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 70 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 69 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 68 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 62 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 57 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 52 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 51 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 49 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 45 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 44 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 43 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 42 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 41 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 39 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 37 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 36 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 35 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 33 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 32 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 31 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 26 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 24 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 22 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 21 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 20 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 19 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 18 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 17 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 16 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 13 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 12 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 11 : epoll_wait , io_poll_wait , listener , get_event , worker_main , start_thread , clone Thread 8 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 7 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 5 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 4 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 3 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone Thread 2 : pthread_cond_timedwait@@GLIBC_2.3.2 , inline_mysql_cond_timedwait , get_event , worker_main , start_thread , clone thanks rich prohaska

1 0

[Maria-developers] libmysqlclient.so and Fedora-specific symbol versioning
by Honza Horak 14 Jan '14

14 Jan '14

Hi guys, I'm writing this as a maintainer of MariaDB package in Fedora and RHEL. I've got some complaints about libmysqlclient symbols versioning in Fedora and future RHEL-7: https://bugzilla.redhat.com/show_bug.cgi?id=1045013 Simply put, the issue there is that RPMs (both made by MariaDB upstream or Fedora community) uses different symbol versioning than it is used in other distributions (deb packages, binary tar balls) or by other MySQL providers (Oracle, Percona). Fedora does it because it used to be done that way before and it was not changed at a time upstream changed that, since Fedora started to handle symbols list on its own. MariaDB upstream adopted Fedora's behavior as a resolution of the bug: https://mariadb.atlassian.net/browse/MDEV-3923 As a result, we have different versioning only based on packaging format now, which seems to be a problem. So, I'd like to propose to sync this gap between "RPM-based packages" and "rest of the world" in the future Fedora/RHEL releases, which basically means I'd like to ask: In case Fedora 21 and RHEL-7 adopt non-versioned symbols in libmysqlclient, would MariaDB upstream be able to do the same in RPMs for these releases? There is also second difference between RPM and non-RPM builds, which is that only limited subset of symbols (those that are documented as API) is exported in the client library in Fedora RPMs. This is because we believe exporting all symbols is wrong thing. Regarding this issue, we'd like to keep the limited set of symbols in RPMs, so, the only change I'm proposing to change versioning of these symbols (as described above). Primarily, I'd like to hear what MariaDB upstream's position on that is, if the sync can happen, but any ideas will be welcome. Honza

2 6

[Maria-developers] TokuDB INSERT NOAR / UPDATE NOAR
by Laurynas Biveinis 09 Jan '14

09 Jan '14

Hello - https://mariadb.com/kb/en/tokudb-differences/ says, for version 10: "No INSERT NOAR or UPDATE NOAR commands. We are working with Tokutek to improve this feature before adding it to MariaDB. " What improvements you are looking into? Are there any JIRA tasks to tracks? Thanks, -- Laurynas

1 0

[Maria-developers] Need Review: MDEV-5509: Seconds_behind_master incorrect in parallel replication in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Kristian Nielsen 08 Jan '14

08 Jan '14

Could someone review this patch for MDEV-5509? I am in particular worried that this might change some subtle behaviour of Seconds_Behind_Master in unexpected ways. Of course I tried my best to make sure such could not occur, but the semantics is really tricky and poorly documented, and I do not think we have good coverage in the test suite. So I would like to have one other developer check it carefully as well and see if I've missed something. - Kristian. knielsen(a)knielsen-hq.org writes: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 3959 > revision-id: knielsen(a)knielsen-hq.org-20140108100044-iszrbhddvxkfne71 > parent: knielsen(a)knielsen-hq.org-20140107105703-sn31yvf80batk1yx > committer: knielsen(a)knielsen-hq.org > branch nick: mariadb-10.0-base > timestamp: Wed 2014-01-08 11:00:44 +0100 > message: > MDEV-5509: Seconds_behind_master incorrect in parallel replication > > The problem was a race between the SQL driver thread and the worker threads. > The SQL driver thread would set rli->last_master_timestamp to zero to > mark that it has caught up with the master, while the worker threads would > set it to the timestamp of the executed event. This can happen out-of-order > in parallel replication, causing the "caught up" status to be overwritten > and Seconds_Behind_Master to wrongly grow when the slave is idle. > > To fix, introduce a separate flag rli->sql_thread_caught_up to mark that the > SQL driver thread is caught up. This avoids issues with worker threads > overwriting the SQL driver thread status. In parallel replication, we then > make SHOW SLAVE STATUS check in addition that all worker threads are idle > before showing Seconds_Behind_Master as 0 due to slave idle. > === added file 'mysql-test/suite/rpl/r/rpl_parallel2.result' > --- a/mysql-test/suite/rpl/r/rpl_parallel2.result 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/rpl/r/rpl_parallel2.result 2014-01-08 10:00:44 +0000 > @@ -0,0 +1,18 @@ > +include/rpl_init.inc [topology=1->2] > +*** MDEV-5509: Incorrect value for Seconds_Behind_Master if parallel replication *** > +SET @old_parallel_threads=@@GLOBAL.slave_parallel_threads; > +include/stop_slave.inc > +SET GLOBAL slave_parallel_threads=5; > +include/start_slave.inc > +CREATE TABLE t1 (a INT PRIMARY KEY, b INT); > +CALL mtr.add_suppression("Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statement is unsafe because it uses a system function that may return a different value on the slave"); > +INSERT INTO t1 VALUES (1,sleep(2)); > +Warnings: > +Note 1592 Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statement is unsafe because it uses a system function that may return a different value on the slave. > +Seconds_Behind_Master should be zero here because the slave is fully caught up and idle. > +Seconds_Behind_Master = '0' > +include/stop_slave.inc > +SET GLOBAL slave_parallel_threads=@old_parallel_threads; > +include/start_slave.inc > +DROP TABLE t1; > +include/rpl_end.inc > > === added file 'mysql-test/suite/rpl/t/rpl_parallel2.test' > --- a/mysql-test/suite/rpl/t/rpl_parallel2.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/suite/rpl/t/rpl_parallel2.test 2014-01-08 10:00:44 +0000 > @@ -0,0 +1,41 @@ > +--source include/have_binlog_format_statement.inc > +--let $rpl_topology=1->2 > +--source include/rpl_init.inc > + > +--echo *** MDEV-5509: Incorrect value for Seconds_Behind_Master if parallel replication *** > + > +--connection server_2 > +SET @old_parallel_threads=@@GLOBAL.slave_parallel_threads; > +--source include/stop_slave.inc > +SET GLOBAL slave_parallel_threads=5; > +--source include/start_slave.inc > + > +--connection server_1 > +CREATE TABLE t1 (a INT PRIMARY KEY, b INT); > +CALL mtr.add_suppression("Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statement is unsafe because it uses a system function that may return a different value on the slave"); > +--save_master_pos > + > +--connection server_2 > +--sync_with_master > + > +--connection server_1 > +INSERT INTO t1 VALUES (1,sleep(2)); > +--save_master_pos > + > +--connection server_2 > +--sync_with_master > + > +--echo Seconds_Behind_Master should be zero here because the slave is fully caught up and idle. > +--let $status_items= Seconds_Behind_Master > +--source include/show_slave_status.inc > + > + > +--connection server_2 > +--source include/stop_slave.inc > +SET GLOBAL slave_parallel_threads=@old_parallel_threads; > +--source include/start_slave.inc > + > +--connection server_1 > +DROP TABLE t1; > + > +--source include/rpl_end.inc > > === modified file 'sql/rpl_parallel.cc' > --- a/sql/rpl_parallel.cc 2014-01-03 11:20:53 +0000 > +++ b/sql/rpl_parallel.cc 2014-01-08 10:00:44 +0000 > @@ -766,6 +766,27 @@ rpl_parallel::wait_for_done() > } > > > +bool > +rpl_parallel::workers_idle() > +{ > + struct rpl_parallel_entry *e; > + uint32 i, max_i; > + > + max_i= domain_hash.records; > + for (i= 0; i < max_i; ++i) > + { > + bool active; > + e= (struct rpl_parallel_entry *)my_hash_element(&domain_hash, i); > + mysql_mutex_lock(&e->LOCK_parallel_entry); > + active= e->current_sub_id > e->last_committed_sub_id; > + mysql_mutex_unlock(&e->LOCK_parallel_entry); > + if (active) > + break; > + } > + return (i == max_i); > +} > + > + > /* > do_event() is executed by the sql_driver_thd thread. > It's main purpose is to find a thread that can execute the query. > > === modified file 'sql/rpl_parallel.h' > --- a/sql/rpl_parallel.h 2013-11-05 11:01:26 +0000 > +++ b/sql/rpl_parallel.h 2014-01-08 10:00:44 +0000 > @@ -117,6 +117,7 @@ struct rpl_parallel { > void reset(); > rpl_parallel_entry *find(uint32 domain_id); > void wait_for_done(); > + bool workers_idle(); > bool do_event(rpl_group_info *serial_rgi, Log_event *ev, > ulonglong event_size); > }; > > === modified file 'sql/rpl_rli.cc' > --- a/sql/rpl_rli.cc 2013-12-17 09:50:34 +0000 > +++ b/sql/rpl_rli.cc 2014-01-08 10:00:44 +0000 > @@ -56,7 +56,7 @@ Relay_log_info::Relay_log_info(bool is_s > is_fake(FALSE), > #endif > group_master_log_pos(0), log_space_total(0), ignore_log_space_limit(0), > - last_master_timestamp(0), slave_skip_counter(0), > + last_master_timestamp(0), sql_thread_caught_up(true), slave_skip_counter(0), > abort_pos_wait(0), slave_run_id(0), sql_driver_thd(), > inited(0), abort_slave(0), slave_running(0), until_condition(UNTIL_NONE), > until_log_pos(0), retried_trans(0), executed_entries(0), > @@ -1287,9 +1287,14 @@ void Relay_log_info::stmt_done(my_off_t > (probably ok - except in some very rare cases, only consequence > is that value may take some time to display in > Seconds_Behind_Master - not critical). > + > + In parallel replication, we take care to not set last_master_timestamp > + backwards, in case of out-of-order calls here. > */ > if (!(event_creation_time == 0 && > - IF_DBUG(debug_not_change_ts_if_art_event > 0, 1))) > + IF_DBUG(debug_not_change_ts_if_art_event > 0, 1)) && > + !(rgi->is_parallel_exec && event_creation_time <= last_master_timestamp) > + ) > last_master_timestamp= event_creation_time; > } > DBUG_VOID_RETURN; > > === modified file 'sql/rpl_rli.h' > --- a/sql/rpl_rli.h 2013-11-08 14:14:18 +0000 > +++ b/sql/rpl_rli.h 2014-01-08 10:00:44 +0000 > @@ -221,6 +221,12 @@ class Relay_log_info : public Slave_repo > bool sql_force_rotate_relay; > > time_t last_master_timestamp; > + /* > + The SQL driver thread sets this true while it is waiting at the end of the > + relay log for more events to arrive. SHOW SLAVE STATUS uses this to report > + Seconds_Behind_Master as zero while the SQL thread is so waiting. > + */ > + bool sql_thread_caught_up; > > void clear_until_condition(); > > > === modified file 'sql/slave.cc' > --- a/sql/slave.cc 2013-12-16 12:48:32 +0000 > +++ b/sql/slave.cc 2014-01-08 10:00:44 +0000 > @@ -2615,8 +2615,24 @@ static bool send_show_master_info_data(T > if ((mi->slave_running == MYSQL_SLAVE_RUN_CONNECT) && > mi->rli.slave_running) > { > - long time_diff= ((long)(time(0) - mi->rli.last_master_timestamp) > - - mi->clock_diff_with_master); > + long time_diff; > + bool idle; > + time_t stamp= mi->rli.last_master_timestamp; > + > + if (!stamp) > + idle= true; > + else > + { > + idle= mi->rli.sql_thread_caught_up; > + if (opt_slave_parallel_threads > 0 && idle && > + !mi->rli.parallel.workers_idle()) > + idle= false; > + } > + if (idle) > + time_diff= 0; > + else > + { > + time_diff= ((long)(time(0) - stamp) - mi->clock_diff_with_master); > /* > Apparently on some systems time_diff can be <0. Here are possible > reasons related to MySQL: > @@ -2632,13 +2648,15 @@ static bool send_show_master_info_data(T > slave is 2. At SHOW SLAVE STATUS time, assume that the difference > between timestamp of slave and rli->last_master_timestamp is 0 > (i.e. they are in the same second), then we get 0-(2-1)=-1 as a result. > - This confuses users, so we don't go below 0: hence the max(). > + This confuses users, so we don't go below 0. > > last_master_timestamp == 0 (an "impossible" timestamp 1970) is a > special marker to say "consider we have caught up". > */ > - protocol->store((longlong)(mi->rli.last_master_timestamp ? > - max(0, time_diff) : 0)); > + if (time_diff < 0) > + time_diff= 0; > + } > + protocol->store((longlong)time_diff); > } > else > { > @@ -6096,6 +6114,7 @@ static Log_event* next_event(rpl_group_i > > if (hot_log) > mysql_mutex_unlock(log_lock); > + rli->sql_thread_caught_up= false; > DBUG_RETURN(ev); > } > if (opt_reckless_slave) // For mysql-test > @@ -6133,12 +6152,10 @@ static Log_event* next_event(rpl_group_i > Seconds_Behind_Master would be zero only when master has no > more updates in binlog for slave. The heartbeat can be sent > in a (small) fraction of slave_net_timeout. Until it's done > - rli->last_master_timestamp is temporarely (for time of > - waiting for the following event) reset whenever EOF is > - reached. > + rli->sql_thread_caught_up is temporarely (for time of waiting for > + the following event) set whenever EOF is reached. > */ > - time_t save_timestamp= rli->last_master_timestamp; > - rli->last_master_timestamp= 0; > + rli->sql_thread_caught_up= true; > > DBUG_ASSERT(rli->relay_log.get_open_count() == > rli->cur_log_old_open_count); > @@ -6264,7 +6281,7 @@ static Log_event* next_event(rpl_group_i > rli->relay_log.wait_for_update_relay_log(rli->sql_driver_thd); > // re-acquire data lock since we released it earlier > mysql_mutex_lock(&rli->data_lock); > - rli->last_master_timestamp= save_timestamp; > + rli->sql_thread_caught_up= false; > continue; > } > /* > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Correct way to determine the ThreadPool usage
by Jean Weisbuch 07 Jan '14

07 Jan '14

Hello, I am counter-checking the mysqlreport "fork" i am maintening (https://mariadb.atlassian.net/browse/MDEV-573) so it could be included in Maria 10 before GA, i am actually unsure that i am checking correctly the Threadpool usage. For these stats (with thread_pool_max_threads=1200 and thread_handling=pool-of-threads) : +------------------------------------------+---------+ | Variable_name | Value | +------------------------------------------+---------+ | Threadpool_idle_threads | 47 | | Threadpool_threads | 52 | | Threads_cached | 0 | | Threads_connected | 19 | | Threads_created | 599771 | | Threads_running | 6 | +------------------------------------------+---------+ The actual output is : __ Threads ___________________________________________ Running 6 of 24 Created 599.77k 0.2/s Slow 0 0/s Threadpool 98 of 1200 %Used: 8.17 Running 52 of 1200 %Running: 4.33 Idle 46 of 1200 %Idle: 3.83 The formulae used are : Threadpool $stats{'Threadpool_threads'} + $stats{'Threadpool_idle_threads'}) of $vars{'thread_pool_max_threads'} %Used: percentage( ($stats{'Threadpool_threads'} + $stats{'Threadpool_idle_threads'}) / $vars{'thread_pool_max_threads'} Running $stats{'Threadpool_threads'} of $vars{'thread_pool_max_threads'} %Running: percentage($stats{'Threadpool_threads'} / $vars{'thread_pool_max_threads'}) Idle $stats{'Threadpool_idle_threads'} of $vars{'thread_pool_max_threads'} %Idle: percentage($stats{'Threadpool_idle_threads'} / $vars{'thread_pool_max_threads'}) Is the Threadpool usage calculation right or should only the Threadpool_threads status be used? What is the meaning of the Threads_connected value when the threadpool is used? Regards.

1 0

[Maria-developers] Elena please test: (MDEV-4974) memory leak in 5.5.32-MariaDB-1~wheezy-log
by Sergei Petrunia 03 Jan '14

03 Jan '14

Hi Elena, Could you please test the below? The changed code may affect - queries that use GROUP BY/ORDER BY/DISTINCT, or a combination of those. - correlated/uncorrelated subqueries. The subquery should remain a separate SELECT, i.e. it should not be converted into semi-join or optimized away. - combination of the above. The code adds a cleanup, so I expect a possible problem to be where we cleaned up something we shouldn't have. Most likely effects are either crash or wrong result. I don't think this patch has a chance of introducing a memory leak. ----- Forwarded message from "Sergei Petrunia (JIRA)" <jira(a)mariadb.atlassian.net> ----- Date: Fri, 3 Jan 2014 21:26:24 +0200 (EET) From: "Sergei Petrunia (JIRA)" <jira(a)mariadb.atlassian.net> To: psergey(a)montyprogram.com Subject: [JIRA] (MDEV-4974) memory leak in 5.5.32-MariaDB-1~wheezy-log [ https://mariadb.atlassian.net/browse/MDEV-4974?page=com.atlassian.jira.plug… ] Sergei Petrunia commented on MDEV-4974: ---------------------------------------- Elena, could you please test 5.5 tree, patched with psergey-fix-mdev4954.diff ? > memory leak in 5.5.32-MariaDB-1~wheezy-log > ------------------------------------------ > > Key: MDEV-4974 > URL: https://mariadb.atlassian.net/browse/MDEV-4974 > Project: MariaDB Development > Issue Type: Bug > Affects Versions: 5.5.32 > Environment: Debian Wheezy x86_64 > Linux greeneggs.lentz.com.au 3.9.3-x86_64-linode33 #1 SMP Mon May 20 10:22:57 EDT 2013 x86_64 GNU/Linux > Reporter: Daniel Black > Assignee: Sergei Petrunia > Priority: Critical > Fix For: 5.5.35 > > Attachments: allqueries.sql, catinthehat_memory-day_no_indexmerge.png, drupal.sql, green-eggs-mysql_connections-day.png, greeneggs-memory-day.png, greeneggs-memory-week.png, greeneggs-mysql_bin_relay_log-day.png, greeneggs-mysql_commands-day.png, greeneggs-mysql_files_tables-day.png, greeneggs-mysql_innodb_bpool-day.png, greeneggs-mysql_innodb_semaphores-day.png, greeneggs-mysql_innodb_tnx-day.png, greeneggs-mysql_myisam_indexes-day.png, greeneggs-mysql_qcache-day.png, greeneggs-mysql_select_types-day.png, greeneggs-mysql_sorts-day.png, greeneggs-mysql_table_locks-day.png, greeneggs-mysql_tmp_tables-day.png, leaks-track-allqueries.sql, leaks-track.sql, psergey-fix-mdev4954.diff, psergey-mdev4974-xpl1.diff, valgrind.mysqld.27336 > > > After running mariadb-5.5.32 in a multimaster for a few days its almost out of memory on the active master (the one getting all the reads). > The replication slave (same version) doesn't suffer the memory leak (with or without the replication filters defined). > Disabling the query cache on the active master may (was slightly off peak) have slowed the memory leak however it wasn't stopped. In the graph attached the query cache was disabled from Wed 5:30 to Thursday 03:00 > For greeneggs-mysql_commands-day.png the first drop is when query cache was turned back on. At the end I moved the active master to the other server. Other graphs are for this same time interval. > Memory usage calculation: > From http://dev.mysql.com/doc/refman/5.5/en/memory-use.html > per connection: > @read_buffer_size + @read_rnd_buffer_size + @sort_buffer_size + @join_buffer_size + @binlog_cache_size + @thread_stack + @@tmp_table_size = 19070976 > Max_used_connections 15 > Static component: > @key_buffer_size + @query_cache_size + @innodb_buffer_pool_size + @innodb_additional_mem_pool_size + @@innodb_log_buffer_size > 322961408 > select 15 * 19070976 + 322961408; = 609026048 > 609M max > From top: > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 4532 mysql 20 0 1999m 1.4g 6072 S 4.0 71.9 1017:21 mysqld > I've still got the server running if more status is required. > show engine innodb status > ===================================== > 130830 0:46:23 INNODB MONITOR OUTPUT > ===================================== > Per second averages calculated from the last 17 seconds > ----------------- > BACKGROUND THREAD > ----------------- > srv_master_thread loops: 372124 1_second, 372076 sleeps, 37113 10_second, 1716 background, 1715 flush > srv_master_thread log flush and writes: 350597 > ---------- > SEMAPHORES > ---------- > OS WAIT ARRAY INFO: reservation count 999301, signal count 1149136 > Mutex spin waits 3647275, rounds 12020660, OS waits 198769 > RW-shared spins 896419, rounds 19893071, OS waits 574516 > RW-excl spins 68006, rounds 6887165, OS waits 204958 > Spin rounds per wait: 3.30 mutex, 22.19 RW-shared, 101.27 RW-excl > -------- > FILE I/O > -------- > I/O thread 0 state: waiting for completed aio requests (insert buffer thread) > I/O thread 1 state: waiting for completed aio requests (log thread) > I/O thread 2 state: waiting for completed aio requests (read thread) > I/O thread 3 state: waiting for completed aio requests (read thread) > I/O thread 4 state: waiting for completed aio requests (write thread) > I/O thread 5 state: waiting for completed aio requests (write thread) > Pending normal aio reads: 0 [0, 0] , aio writes: 0 [0, 0] , > ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 > Pending flushes (fsync) log: 0; buffer pool: 0 > 7906118 OS file reads, 39186717 OS file writes, 23493355 OS fsyncs > 0.65 reads/s, 16384 avg bytes/read, 66.29 writes/s, 29.06 fsyncs/s > ------------------------------------- > INSERT BUFFER AND ADAPTIVE HASH INDEX > ------------------------------------- > Ibuf: size 1, free list len 287, seg size 289, 111478 merges > merged operations: > insert 106175, delete mark 100854, delete 13481 > discarded operations: > insert 0, delete mark 0, delete 0 > Hash table size 553229, node heap has 584 buffer(s) > 244.87 hash searches/s, 22.53 non-hash searches/s > --- > LOG > --- > Log sequence number 1981376581482 > Log flushed up to 1981376581482 > Last checkpoint at 1981376562791 > Max checkpoint age 84223550 > Checkpoint age target 81591565 > Modified age 18691 > Checkpoint age 18691 > 0 pending log writes, 0 pending chkp writes > 22419510 log i/o's done, 26.00 log i/o's/second > ---------------------- > BUFFER POOL AND MEMORY > ---------------------- > Total memory allocated 275644416; in additional pool allocated 0 > Total memory allocated by read views 136 > Internal hash tables (constant factor + variable factor) > Adaptive hash index 13998304 (4425832 + 9572472) > Page hash 277432 (buffer pool 0 only) > Dictionary cache 10287074 (1107952 + 9179122) > File system 648160 (82672 + 565488) > Lock system 665688 (664936 + 752) > Recovery system 0 (0 + 0) > Dictionary memory allocated 9179122 > Buffer pool size 16383 > Buffer pool size, bytes 268419072 > Free buffers 1 > Database pages 15798 > Old database pages 5811 > Modified db pages 97 > Pending reads 0 > Pending writes: LRU 0, flush list 0, single page 0 > Pages made young 10760065, not young 0 > 0.53 youngs/s, 0.00 non-youngs/s > Pages read 7903370, created 882751, written 16452888 > 0.65 reads/s, 0.00 creates/s, 39.47 writes/s > Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 > Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s > LRU len: 15798, unzip_LRU len: 0 > I/O sum[1550]:cur[322], unzip sum[0]:cur[0] > -------------- > ROW OPERATIONS > -------------- > 0 queries inside InnoDB, 0 queries in queue > 1 read views open inside InnoDB > 0 transactions active inside InnoDB > 0 out of 1000 descriptors used > ---OLDEST VIEW--- > Normal read view > Read view low limit trx n:o 7D2D2F07 > Read view up limit trx id 7D2D2F07 > Read view low limit trx id 7D2D2F07 > Read view individually stored trx ids: > ----------------- > Main thread process no. 4532, id 140201824495360, state: sleeping > Number of rows inserted 1308400, updated 9736429, deleted 1227755, read 34888786828 > 1.00 inserts/s, 11.71 updates/s, 0.00 deletes/s, 249.22 reads/s > ------------------------ > LATEST DETECTED DEADLOCK > ------------------------ > 130830 0:27:15 > *** (1) TRANSACTION: > TRANSACTION 7D2C63D3, ACTIVE 0 sec starting index read > | Variable_name | Value | > | Aborted_clients | 57 | > | Aborted_connects | 0 | > | Access_denied_errors | 0 | > | Aria_pagecache_blocks_not_flushed | 0 | > | Aria_pagecache_blocks_unused | 15737 | > | Aria_pagecache_blocks_used | 3127 | > | Aria_pagecache_read_requests | 262055553 | > | Aria_pagecache_reads | 107163 | > | Aria_pagecache_write_requests | 69866678 | > | Aria_pagecache_writes | 0 | > | Aria_transaction_log_syncs | 0 | > | Binlog_commits | 12529081 | > | Binlog_group_commits | 12486299 | > | Binlog_snapshot_file | mariadb-bin.001408 | > | Binlog_snapshot_position | 57228334 | > | Binlog_bytes_written | 43273655871 | > | Binlog_cache_disk_use | 768128 | > | Binlog_cache_use | 12510967 | > | Binlog_stmt_cache_disk_use | 24 | > | Binlog_stmt_cache_use | 18077 | > | Busy_time | 0.000000 | > | Bytes_received | 60129853267 | > | Bytes_sent | 691695773018 | > | Com_admin_commands | 3809 | > | Com_assign_to_keycache | 0 | > | Com_alter_db | 0 | > | Com_alter_db_upgrade | 0 | > | Com_alter_event | 0 | > | Com_alter_function | 0 | > | Com_alter_procedure | 0 | > | Com_alter_server | 0 | > | Com_alter_table | 114 | > | Com_alter_tablespace | 0 | > | Com_analyze | 0 | > | Com_begin | 62278 | > | Com_binlog | 0 | > | Com_call_procedure | 0 | > | Com_change_db | 69728 | > | Com_change_master | 0 | > | Com_check | 1426 | > | Com_checksum | 0 | > | Com_commit | 45717 | > | Com_create_db | 1 | > | Com_create_event | 0 | > | Com_create_function | 0 | > | Com_create_index | 0 | > | Com_create_procedure | 1 | > | Com_create_server | 0 | > | Com_create_table | 42 | > | Com_create_trigger | 0 | > | Com_create_udf | 0 | > | Com_create_user | 0 | > | Com_create_view | 0 | > | Com_dealloc_sql | 24 | > | Com_delete | 388136 | > | Com_delete_multi | 5 | > | Com_do | 0 | > | Com_drop_db | 1 | > | Com_drop_event | 0 | > | Com_drop_function | 0 | > | Com_drop_index | 0 | > | Com_drop_procedure | 2 | > | Com_drop_server | 0 | > | Com_drop_table | 1 | > | Com_drop_trigger | 0 | > | Com_drop_user | 0 | > | Com_drop_view | 0 | > | Com_empty_query | 0 | > | Com_execute_sql | 24 | > | Com_flush | 6 | > | Com_grant | 0 | > | Com_ha_close | 0 | > | Com_ha_open | 0 | > | Com_ha_read | 0 | > | Com_help | 2 | > | Com_insert | 847948 | > | Com_insert_select | 551 | > | Com_install_plugin | 0 | > | Com_kill | 50 | > | Com_load | 0 | > | Com_lock_tables | 0 | > | Com_optimize | 0 | > | Com_preload_keys | 0 | > | Com_prepare_sql | 24 | > | Com_purge | 0 | > | Com_purge_before_date | 0 | > | Com_release_savepoint | 0 | > | Com_rename_table | 0 | > | Com_rename_user | 0 | > | Com_repair | 0 | > | Com_replace | 0 | > | Com_replace_select | 0 | > | Com_reset | 0 | > | Com_resignal | 0 | > | Com_revoke | 0 | > | Com_revoke_all | 0 | > | Com_rollback | 3 | > | Com_rollback_to_savepoint | 0 | > | Com_savepoint | 0 | > | Com_select | 47174988 | > | Com_set_option | 1189075 | > | Com_signal | 0 | > | Com_show_authors | 0 | > | Com_show_binlog_events | 3 | > | Com_show_binlogs | 1166 | > | Com_show_charsets | 0 | > | Com_show_client_statistics | 0 | > | Com_show_collations | 0 | > | Com_show_contributors | 0 | > | Com_show_create_db | 115 | > | Com_show_create_event | 0 | > | Com_show_create_func | 0 | > | Com_show_create_proc | 0 | > | Com_show_create_table | 4387 | > | Com_show_create_trigger | 0 | > | Com_show_databases | 7 | > | Com_show_engine_logs | 0 | > | Com_show_engine_mutex | 0 | > | Com_show_engine_status | 1167 | > | Com_show_events | 0 | > | Com_show_errors | 0 | > | Com_show_fields | 5721 | > | Com_show_function_status | 0 | > | Com_show_grants | 0 | > | Com_show_keys | 0 | > | Com_show_index_statistics | 0 | > | Com_show_master_status | 1 | > | Com_show_open_tables | 0 | > | Com_show_plugins | 0 | > | Com_show_privileges | 0 | > | Com_show_procedure_status | 0 | > | Com_show_processlist | 1039 | > | Com_show_profile | 0 | > | Com_show_profiles | 0 | > | Com_show_relaylog_events | 0 | > | Com_show_slave_hosts | 0 | > | Com_show_slave_status | 140780 | > | Com_show_status | 12670 | > | Com_show_storage_engines | 0 | > | Com_show_table_statistics | 0 | > | Com_show_table_status | 4287 | > | Com_show_tables | 6794 | > | Com_show_triggers | 4280 | > | Com_show_user_statistics | 0 | > | Com_show_variables | 2175 | > | Com_show_warnings | 0 | > | Com_slave_start | 10 | > | Com_slave_stop | 4 | > | Com_stmt_close | 24 | > | Com_stmt_execute | 24 | > | Com_stmt_fetch | 0 | > | Com_stmt_prepare | 24 | > | Com_stmt_reprepare | 0 | > | Com_stmt_reset | 0 | > | Com_stmt_send_long_data | 0 | > | Com_truncate | 0 | > | Com_uninstall_plugin | 0 | > | Com_unlock_tables | 2 | > | Com_update | 7082015 | > | Com_update_multi | 1 | > | Com_xa_commit | 0 | > | Com_xa_end | 0 | > | Com_xa_prepare | 0 | > | Com_xa_recover | 0 | > | Com_xa_rollback | 0 | > | Com_xa_start | 0 | > | Compression | OFF | > | Connections | 1164212 | > | Cpu_time | 0.000000 | > | Created_tmp_disk_tables | 2202858 | > | Created_tmp_files | 420678 | > | Created_tmp_tables | 5264753 | > | Delayed_errors | 0 | > | Delayed_insert_threads | 0 | > | Delayed_writes | 0 | > | Empty_queries | 14900340 | > | Executed_events | 0 | > | Executed_triggers | 0 | > | Feature_dynamic_columns | 0 | > | Feature_fulltext | 2 | > | Feature_gis | 28 | > | Feature_locale | 0 | > | Feature_subquery | 135848 | > | Feature_timezone | 6850 | > | Feature_trigger | 586 | > | Feature_xml | 0 | > | Flush_commands | 4 | > | Handler_commit | 63318523 | > | Handler_delete | 748245 | > | Handler_discover | 0 | > | Handler_icp_attempts | 71118314 | > | Handler_icp_match | 71108704 | > | Handler_mrr_init | 0 | > | Handler_mrr_key_refills | 0 | > | Handler_mrr_rowid_refills | 0 | > | Handler_prepare | 14968970 | > | Handler_read_first | 1155850 | > | Handler_read_key | 466199920 | > | Handler_read_last | 55346 | > | Handler_read_next | 27258307498 | > | Handler_read_prev | 8491322 | > | Handler_read_rnd | 66364460 | > | Handler_read_rnd_deleted | 1194 | > | Handler_read_rnd_next | 8174479457 | > | Handler_rollback | 21990 | > | Handler_savepoint | 0 | > | Handler_savepoint_rollback | 0 | > | Handler_tmp_update | 15018397 | > | Handler_tmp_write | 515819305 | > | Handler_update | 6481729 | > | Handler_write | 849123 | > | Innodb_adaptive_hash_cells | 553229 | > | Innodb_adaptive_hash_heap_buffers | 581 | > | Innodb_adaptive_hash_hash_searches | 1107671046 | > | Innodb_adaptive_hash_non_hash_searches | 249488632 | > | Innodb_background_log_sync | 350697 | > | Innodb_buffer_pool_pages_data | 15800 | > | Innodb_buffer_pool_bytes_data | 258867200 | > | Innodb_buffer_pool_pages_dirty | 295 | > | Innodb_buffer_pool_bytes_dirty | 4833280 | > | Innodb_buffer_pool_pages_flushed | 16456234 | > | Innodb_buffer_pool_pages_LRU_flushed | 48422 | > | Innodb_buffer_pool_pages_free | 1 | > | Innodb_buffer_pool_pages_made_not_young | 0 | > | Innodb_buffer_pool_pages_made_young | 10760117 | > | Innodb_buffer_pool_pages_misc | 582 | > | Innodb_buffer_pool_pages_old | 5812 | > | Innodb_buffer_pool_pages_total | 16383 | > | Innodb_buffer_pool_read_ahead_rnd | 0 | > | Innodb_buffer_pool_read_ahead | 2072088 | > | Innodb_buffer_pool_read_ahead_evicted | 91126 | > | Innodb_buffer_pool_read_requests | 10973288017 | > | Innodb_buffer_pool_reads | 5715842 | > | Innodb_buffer_pool_wait_free | 9 | > | Innodb_buffer_pool_write_requests | 132620870 | > | Innodb_checkpoint_age | 88015 | > | Innodb_checkpoint_max_age | 84223550 | > | Innodb_checkpoint_target_age | 81591565 | > | Innodb_data_fsyncs | 23497491 | > | Innodb_data_pending_fsyncs | 0 | > | Innodb_data_pending_reads | 0 | > | Innodb_data_pending_writes | 0 | > | Innodb_data_read | 129493962752 | > | Innodb_data_reads | 7906173 | > | Innodb_data_writes | 39194030 | > | Innodb_data_written | 586483447808 | > | Innodb_dblwr_pages_written | 16456234 | > | Innodb_dblwr_writes | 182242 | > | Innodb_deadlocks | 432 | > | Innodb_dict_tables | 1315 | > | Innodb_have_atomic_builtins | ON | > | Innodb_history_list_length | 3523 | > | Innodb_ibuf_discarded_delete_marks | 0 | > | Innodb_ibuf_discarded_deletes | 0 | > | Innodb_ibuf_discarded_inserts | 0 | > | Innodb_ibuf_free_list | 287 | > | Innodb_ibuf_merged_delete_marks | 100855 | > | Innodb_ibuf_merged_deletes | 13481 | > | Innodb_ibuf_merged_inserts | 106192 | > | Innodb_ibuf_merges | 111495 | > | Innodb_ibuf_segment_size | 289 | > | Innodb_ibuf_size | 1 | > | Innodb_log_waits | 0 | > | Innodb_log_write_requests | 73751900 | > | Innodb_log_writes | 22385577 | > | Innodb_lsn_current | 1981377626218 | > | Innodb_lsn_flushed | 1981377626218 | > | Innodb_lsn_last_checkpoint | 1981377538203 | > | Innodb_master_thread_1_second_loops | 372239 | > | Innodb_master_thread_10_second_loops | 37124 | > | Innodb_master_thread_background_loops | 1716 | > | Innodb_master_thread_main_flush_loops | 1715 | > | Innodb_master_thread_sleeps | 372191 | > | Innodb_max_trx_id | 2100117045 | > | Innodb_mem_adaptive_hash | 13965536 | > | Innodb_mem_dictionary | 10287074 | > | Innodb_mem_total | 275644416 | > | Innodb_mutex_os_waits | 198788 | > | Innodb_mutex_spin_rounds | 12021403 | > | Innodb_mutex_spin_waits | 3647342 | > | Innodb_oldest_view_low_limit_trx_id | 2100116933 | > | Innodb_os_log_fsyncs | 22425791 | > | Innodb_os_log_pending_fsyncs | 0 | > | Innodb_os_log_pending_writes | 0 | > | Innodb_os_log_written | 47227117568 | > | Innodb_page_size | 16384 | > | Innodb_pages_created | 882752 | > | Innodb_pages_read | 7903425 | > | Innodb_pages_written | 16456234 | > | Innodb_purge_trx_id | 2100116933 | > | Innodb_purge_undo_no | 0 | > | Innodb_row_lock_current_waits | 0 | > | Innodb_current_row_locks | 0 | > | Innodb_row_lock_time | 3209772 | > | Innodb_row_lock_time_avg | 74 | > | Innodb_row_lock_time_max | 31935 | > | Innodb_row_lock_waits | 43208 | > | Innodb_rows_deleted | 1227755 | > | Innodb_rows_inserted | 1308502 | > | Innodb_rows_read | 34888819968 | > | Innodb_rows_updated | 9738262 | > | Innodb_read_views_memory | 136 | > | Innodb_descriptors_memory | 8000 | > | Innodb_s_lock_os_waits | 574527 | > | Innodb_s_lock_spin_rounds | 19893402 | > | Innodb_s_lock_spin_waits | 896432 | > | Innodb_truncated_status_writes | 0 | > | Innodb_x_lock_os_waits | 204987 | > | Innodb_x_lock_spin_rounds | 6888035 | > | Innodb_x_lock_spin_waits | 68006 | > | Key_blocks_not_flushed | 0 | > | Key_blocks_unused | 5353 | > | Key_blocks_used | 4170 | > | Key_blocks_warm | 141 | > | Key_read_requests | 6295929 | > | Key_reads | 9626 | > | Key_write_requests | 21159 | > | Key_writes | 10962 | > | Last_query_cost | 0.000000 | > | Max_used_connections | 15 | > | Not_flushed_delayed_rows | 0 | > | Open_files | 142 | > | Open_streams | 0 | > | Open_table_definitions | 397 | > | Open_tables | 595 | > | Opened_files | 9756380 | > | Opened_table_definitions | 6425 | > | Opened_tables | 8666 | > | Opened_views | 0 | > | Performance_schema_cond_classes_lost | 0 | > | Performance_schema_cond_instances_lost | 0 | > | Performance_schema_file_classes_lost | 0 | > | Performance_schema_file_handles_lost | 0 | > | Performance_schema_file_instances_lost | 0 | > | Performance_schema_locker_lost | 0 | > | Performance_schema_mutex_classes_lost | 0 | > | Performance_schema_mutex_instances_lost | 0 | > | Performance_schema_rwlock_classes_lost | 0 | > | Performance_schema_rwlock_instances_lost | 0 | > | Performance_schema_table_handles_lost | 0 | > | Performance_schema_table_instances_lost | 0 | > | Performance_schema_thread_classes_lost | 0 | > | Performance_schema_thread_instances_lost | 0 | > | Prepared_stmt_count | 0 | > | Qcache_free_blocks | 6616 | > | Qcache_free_memory | 18486664 | > | Qcache_hits | 75724207 | > | Qcache_inserts | 8304484 | > | Qcache_lowmem_prunes | 1719071 | > | Qcache_not_cached | 2443705 | > | Qcache_queries_in_cache | 8992 | > | Qcache_total_blocks | 24978 | > | Queries | 142821923 | > | Questions | 133827961 | > | Rows_read | 35275818740 | > | Rows_sent | 1787826753 | > | Rows_tmp_read | 592687087 | > | Rpl_status | AUTH_MASTER | > | Select_full_join | 268533 | > | Select_full_range_join | 6393 | > | Select_range | 6218451 | > | Select_range_check | 0 | > | Select_scan | 3587397 | > | Slave_heartbeat_period | 1800.000 | > | Slave_open_temp_tables | 0 | > | Slave_received_heartbeats | 0 | > | Slave_retried_transactions | 0 | > | Slave_running | ON | > | Slow_launch_threads | 0 | > | Slow_queries | 3796864 | > | Sort_merge_passes | 269513 | > | Sort_range | 170798 | > | Sort_rows | 1870364969 | > | Sort_scan | 4917255 | > | Sphinx_error | | > | Sphinx_time | | > | Sphinx_total | | > | Sphinx_total_found | | > | Sphinx_word_count | | > | Sphinx_words | | > | Ssl_accept_renegotiates | 0 | > | Ssl_accepts | 0 | > | Ssl_callback_cache_hits | 0 | > | Ssl_cipher | | > | Ssl_cipher_list | | > | Ssl_client_connects | 0 | > | Ssl_connect_renegotiates | 0 | > | Ssl_ctx_verify_depth | 0 | > | Ssl_ctx_verify_mode | 0 | > | Ssl_default_timeout | 0 | > | Ssl_finished_accepts | 0 | > | Ssl_finished_connects | 0 | > | Ssl_session_cache_hits | 0 | > | Ssl_session_cache_misses | 0 | > | Ssl_session_cache_mode | NONE | > | Ssl_session_cache_overflows | 0 | > | Ssl_session_cache_size | 0 | > | Ssl_session_cache_timeouts | 0 | > | Ssl_sessions_reused | 0 | > | Ssl_used_session_cache_entries | 0 | > | Ssl_verify_depth | 0 | > | Ssl_verify_mode | 0 | > | Ssl_version | | > | Subquery_cache_hit | 4512 | > | Subquery_cache_miss | 3930 | > | Syncs | 4171361 | > | Table_locks_immediate | 85865730 | > | Table_locks_waited | 15 | > | Tc_log_max_pages_used | 0 | > | Tc_log_page_size | 0 | > | Tc_log_page_waits | 46 | > | Threadpool_idle_threads | 0 | > | Threadpool_threads | 0 | > | Threads_cached | 13 | > | Threads_connected | 2 | > | Threads_created | 15 | > | Threads_running | 2 | > | Uptime | 349494 | > | Uptime_since_flush_status | 349494 | > | Variable_name | Value | > | aria_block_size | 8192 | > | aria_checkpoint_interval | 30 | > | aria_checkpoint_log_activity | 1048576 | > | aria_force_start_after_recovery_failures | 0 | > | aria_group_commit | none | > | aria_group_commit_interval | 0 | > | aria_log_file_size | 1073741824 | > | aria_log_purge_type | immediate | > | aria_max_sort_file_size | 9223372036853727232 | > | aria_page_checksum | ON | > | aria_pagecache_age_threshold | 300 | > | aria_pagecache_buffer_size | 134217728 | > | aria_pagecache_division_limit | 100 | > | aria_recover | NORMAL | > | aria_repair_threads | 1 | > | aria_sort_buffer_size | 134217728 | > | aria_stats_method | nulls_unequal | > | aria_sync_log_dir | NEWFILE | > | aria_used_for_temp_tables | ON | > | auto_increment_increment | 2 | > | auto_increment_offset | 2 | > | autocommit | ON | > | automatic_sp_privileges | ON | > | back_log | 50 | > | basedir | /usr | > | big_tables | OFF | > | binlog_annotate_row_events | OFF | > | binlog_cache_size | 32768 | > | binlog_checksum | NONE | > | binlog_direct_non_transactional_updates | OFF | > | binlog_format | MIXED | > | binlog_optimize_thread_scheduling | ON | > | binlog_stmt_cache_size | 32768 | > | bulk_insert_buffer_size | 1048576 | > | character_set_client | latin1 | > | character_set_connection | latin1 | > | character_set_database | latin1 | > | character_set_filesystem | binary | > | character_set_results | latin1 | > | character_set_server | latin1 | > | character_set_system | utf8 | > | character_sets_dir | /usr/share/mysql/charsets/ | > | collation_connection | latin1_swedish_ci | > | collation_database | latin1_swedish_ci | > | collation_server | latin1_swedish_ci | > | completion_type | NO_CHAIN | > | concurrent_insert | ALWAYS | > | connect_timeout | 5 | > | datadir | /var/lib/mysql/ | > | date_format | %Y-%m-%d | > | datetime_format | %Y-%m-%d %H:%i:%s | > | deadlock_search_depth_long | 15 | > | deadlock_search_depth_short | 4 | > | deadlock_timeout_long | 50000000 | > | deadlock_timeout_short | 10000 | > | debug_no_thread_alarm | OFF | > | default_storage_engine | InnoDB | > | default_week_format | 0 | > | delay_key_write | ON | > | delayed_insert_limit | 100 | > | delayed_insert_timeout | 300 | > | delayed_queue_size | 1000 | > | div_precision_increment | 4 | > | engine_condition_pushdown | OFF | > | event_scheduler | OFF | > | expensive_subquery_limit | 100 | > | expire_logs_days | 3 | > | extra_max_connections | 1 | > | extra_port | 0 | > | flush | OFF | > | flush_time | 0 | > | foreign_key_checks | ON | > | ft_boolean_syntax | + -><()~*:""&| | > | ft_max_word_len | 84 | > | ft_min_word_len | 4 | > | ft_query_expansion_limit | 20 | > | ft_stopword_file | (built-in) | > | general_log | OFF | > | general_log_file | greeneggs.log | > | group_concat_max_len | 1024 | > | have_compress | YES | > | have_crypt | YES | > | have_csv | YES | > | have_dynamic_loading | YES | > | have_geometry | YES | > | have_innodb | YES | > | have_ndbcluster | NO | > | have_openssl | DISABLED | > | have_partitioning | YES | > | have_profiling | YES | > | have_query_cache | YES | > | have_rtree_keys | YES | > | have_ssl | DISABLED | > | have_symlink | YES | > | hostname | greeneggs.lentz.com.au | > | ignore_builtin_innodb | OFF | > | ignore_db_dirs | | > | init_connect | | > | init_file | | > | init_slave | | > | innodb_adaptive_flushing | ON | > | innodb_adaptive_flushing_method | estimate | > | innodb_adaptive_hash_index | ON | > | innodb_adaptive_hash_index_partitions | 1 | > | innodb_additional_mem_pool_size | 8388608 | > | innodb_autoextend_increment | 8 | > | innodb_autoinc_lock_mode | 1 | > | innodb_blocking_buffer_pool_restore | OFF | > | innodb_buffer_pool_instances | 1 | > | innodb_buffer_pool_populate | OFF | > | innodb_buffer_pool_restore_at_startup | 600 | > | innodb_buffer_pool_shm_checksum | ON | > | innodb_buffer_pool_shm_key | 0 | > | innodb_buffer_pool_size | 268435456 | > | innodb_change_buffering | all | > | innodb_checkpoint_age_target | 0 | > | innodb_checksums | ON | > | innodb_commit_concurrency | 0 | > | innodb_concurrency_tickets | 500 | > | innodb_corrupt_table_action | assert | > | innodb_data_file_path | ibdata1:10M:autoextend | > | innodb_data_home_dir | | > | innodb_dict_size_limit | 0 | > | innodb_doublewrite | ON | > | innodb_doublewrite_file | | > | innodb_fake_changes | OFF | > | innodb_fast_checksum | OFF | > | innodb_fast_shutdown | 1 | > | innodb_file_format | Antelope | > | innodb_file_format_check | ON | > | innodb_file_format_max | Antelope | > | innodb_file_per_table | ON | > | innodb_flush_log_at_trx_commit | 1 | > | innodb_flush_method | O_DIRECT | > | innodb_flush_neighbor_pages | area | > | innodb_force_load_corrupted | OFF | > | innodb_force_recovery | 0 | > | innodb_ibuf_accel_rate | 100 | > | innodb_ibuf_active_contract | 1 | > | innodb_ibuf_max_size | 134201344 | > | innodb_import_table_from_xtrabackup | 0 | > | innodb_io_capacity | 1000 | > | innodb_kill_idle_transaction | 0 | > | innodb_large_prefix | OFF | > | innodb_lazy_drop_table | 0 | > | innodb_lock_wait_timeout | 50 | > | innodb_locking_fake_changes | ON | > | innodb_locks_unsafe_for_binlog | OFF | > | innodb_log_block_size | 512 | > | innodb_log_buffer_size | 4194304 | > | innodb_log_file_size | 52428800 | > | innodb_log_files_in_group | 2 | > | innodb_log_group_home_dir | ./ | > | innodb_max_bitmap_file_size | 104857600 | > | innodb_max_changed_pages | 1000000 | > | innodb_max_dirty_pages_pct | 75 | > | innodb_max_purge_lag | 0 | > | innodb_merge_sort_block_size | 1048576 | > | innodb_mirrored_log_groups | 1 | > | innodb_old_blocks_pct | 37 | > | innodb_old_blocks_time | 0 | > | innodb_open_files | 400 | > | innodb_page_size | 16384 | > | innodb_print_all_deadlocks | OFF | > | innodb_purge_batch_size | 20 | > | innodb_purge_threads | 1 | > | innodb_random_read_ahead | OFF | > | innodb_read_ahead | linear | > | innodb_read_ahead_threshold | 56 | > | innodb_read_io_threads | 2 | > | innodb_recovery_stats | OFF | > | innodb_recovery_update_relay_log | OFF | > | innodb_replication_delay | 0 | > | innodb_rollback_on_timeout | OFF | > | innodb_rollback_segments | 128 | > | innodb_show_locks_held | 10 | > | innodb_show_verbose_locks | 0 | > | innodb_spin_wait_delay | 6 | > | innodb_stats_auto_update | 1 | > | innodb_stats_method | nulls_equal | > | innodb_stats_on_metadata | ON | > | innodb_stats_sample_pages | 8 | > | innodb_stats_update_need_lock | 1 | > | innodb_strict_mode | OFF | > | innodb_support_xa | ON | > | innodb_sync_spin_loops | 30 | > | innodb_table_locks | ON | > | innodb_thread_concurrency | 0 | > | innodb_thread_concurrency_timer_based | OFF | > | innodb_thread_sleep_delay | 10000 | > | innodb_track_changed_pages | OFF | > | innodb_use_atomic_writes | OFF | > | innodb_use_fallocate | OFF | > | innodb_use_global_flush_log_at_trx_commit | ON | > | innodb_use_native_aio | ON | > | innodb_use_sys_malloc | ON | > | innodb_use_sys_stats_table | OFF | > | innodb_version | 5.5.32-MariaDB-30.2 | > | innodb_write_io_threads | 2 | > | interactive_timeout | 28800 | > | join_buffer_size | 131072 | > | join_buffer_space_limit | 2097152 | > | join_cache_level | 2 | > | keep_files_on_create | OFF | > | key_buffer_size | 8388608 | > | key_cache_age_threshold | 300 | > | key_cache_block_size | 1024 | > | key_cache_division_limit | 100 | > | key_cache_segments | 0 | > | large_files_support | ON | > | large_page_size | 0 | > | large_pages | OFF | > | lc_messages | en_US | > | lc_messages_dir | /usr/share/mysql | > | lc_time_names | en_US | > | license | GPL | > | local_infile | ON | > | lock_wait_timeout | 31536000 | > | locked_in_memory | OFF | > | log | OFF | > | log_bin | ON | > | log_bin_trust_function_creators | OFF | > | log_error | | > | log_output | FILE | > | log_queries_not_using_indexes | ON | > | log_slave_updates | ON | > | log_slow_filter | admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk | > | log_slow_queries | ON | > | log_slow_rate_limit | 1 | > | log_slow_verbosity | query_plan | > | log_warnings | 2 | > | long_query_time | 3.000000 | > | low_priority_updates | OFF | > | lower_case_file_system | OFF | > | lower_case_table_names | 0 | > | master_verify_checksum | OFF | > | max_allowed_packet | 16777216 | > | max_binlog_cache_size | 18446744073709547520 | > | max_binlog_size | 104857600 | > | max_binlog_stmt_cache_size | 18446744073709547520 | > | max_connect_errors | 10 | > | max_connections | 100 | > | max_delayed_threads | 20 | > | max_error_count | 64 | > | max_heap_table_size | 16777216 | > | max_insert_delayed_threads | 20 | > | max_join_size | 18446744073709551615 | > | max_length_for_sort_data | 1024 | > | max_long_data_size | 16777216 | > | max_prepared_stmt_count | 16382 | > | max_relay_log_size | 0 | > | max_seeks_for_key | 4294967295 | > | max_sort_length | 1024 | > | max_sp_recursion_depth | 0 | > | max_tmp_tables | 32 | > | max_user_connections | 0 | > | max_write_lock_count | 4294967295 | > | metadata_locks_cache_size | 1024 | > | min_examined_row_limit | 0 | > | mrr_buffer_size | 262144 | > | multi_range_count | 256 | > | myisam_block_size | 1024 | > | myisam_data_pointer_size | 6 | > | myisam_max_sort_file_size | 9223372036853727232 | > | myisam_mmap_size | 18446744073709551615 | > | myisam_recover_options | BACKUP,QUICK | > | myisam_repair_threads | 1 | > | myisam_sort_buffer_size | 536870912 | > | myisam_stats_method | nulls_unequal | > | myisam_use_mmap | OFF | > | net_buffer_length | 16384 | > | net_read_timeout | 30 | > | net_retry_count | 10 | > | net_write_timeout | 60 | > | old | OFF | > | old_alter_table | OFF | > | old_passwords | OFF | > | open_files_limit | 2159 | > | optimizer_prune_level | 1 | > | optimizer_search_depth | 62 | > | optimizer_switch | index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off | > | performance_schema | OFF | > | performance_schema_events_waits_history_long_size | 10000 | > | performance_schema_events_waits_history_size | 10 | > | performance_schema_max_cond_classes | 80 | > | performance_schema_max_cond_instances | 1000 | > | performance_schema_max_file_classes | 50 | > | performance_schema_max_file_handles | 32768 | > | performance_schema_max_file_instances | 10000 | > | performance_schema_max_mutex_classes | 200 | > | performance_schema_max_mutex_instances | 1000000 | > | performance_schema_max_rwlock_classes | 30 | > | performance_schema_max_rwlock_instances | 1000000 | > | performance_schema_max_table_handles | 100000 | > | performance_schema_max_table_instances | 50000 | > | performance_schema_max_thread_classes | 50 | > | performance_schema_max_thread_instances | 1000 | > | pid_file | /var/run/mysqld/mysqld.pid | > | plugin_dir | /usr/lib/mysql/plugin/ | > | plugin_maturity | unknown | > | port | 3306 | > | preload_buffer_size | 32768 | > | profiling | OFF | > | profiling_history_size | 15 | > | progress_report_time | 56 | > | protocol_version | 10 | > | query_alloc_block_size | 8192 | > | query_cache_limit | 131072 | > | query_cache_min_res_unit | 4096 | > | query_cache_size | 33554432 | > | query_cache_strip_comments | OFF | > | query_cache_type | ON | > | query_cache_wlock_invalidate | OFF | > | query_prealloc_size | 8192 | > | range_alloc_block_size | 4096 | > | read_buffer_size | 1048576 | > | read_only | ON | > | read_rnd_buffer_size | 524288 | > | relay_log | | > | relay_log_index | | > | relay_log_info_file | relay-log.info | > | relay_log_purge | ON | > | relay_log_recovery | OFF | > | relay_log_space_limit | 0 | > | replicate_annotate_row_events | OFF | > | replicate_do_db | | > | replicate_do_table | | > | replicate_events_marked_for_skip | replicate | > | replicate_ignore_db | | > | replicate_ignore_table | peoplesforum.cache | > | replicate_wild_do_table | | > | replicate_wild_ignore_table | peoplesforum.cache% | > | report_host | greeneggs | > | report_password | | > | report_port | 3306 | > | report_user | | > | rowid_merge_buff_size | 8388608 | > | rpl_recovery_rank | 0 | > | secure_auth | OFF | > | secure_file_priv | | > | server_id | 12302 | > | skip_external_locking | ON | > | skip_name_resolve | ON | > | skip_networking | OFF | > | skip_show_database | OFF | > | slave_compressed_protocol | OFF | > | slave_exec_mode | STRICT | > | slave_load_tmpdir | /tmp | > | slave_max_allowed_packet | 1073741824 | > | slave_net_timeout | 3600 | > | slave_skip_errors | 1062 | > | slave_sql_verify_checksum | ON | > | slave_transaction_retries | 10 | > | slave_type_conversions | | > | slow_launch_time | 2 | > | slow_query_log | ON | > | slow_query_log_file | /var/log/mysql/mariadb-slow.log | > | socket | /var/run/mysqld/mysqld.sock | > | sort_buffer_size | 262144 | > | sql_auto_is_null | OFF | > | sql_big_selects | ON | > | sql_big_tables | OFF | > | sql_buffer_result | OFF | > | sql_log_bin | ON | > | sql_log_off | OFF | > | sql_low_priority_updates | OFF | > | sql_max_join_size | 18446744073709551615 | > | sql_mode | NO_ENGINE_SUBSTITUTION | > | sql_notes | ON | > | sql_quote_show_create | ON | > | sql_safe_updates | OFF | > | sql_select_limit | 18446744073709551615 | > | sql_slave_skip_counter | 0 | > | sql_warnings | OFF | > | ssl_ca | | > | ssl_capath | | > | ssl_cert | | > | ssl_cipher | | > | ssl_key | | > | storage_engine | InnoDB | > | stored_program_cache | 256 | > | sync_binlog | 3 | > | sync_frm | ON | > | sync_master_info | 0 | > | sync_relay_log | 0 | > | sync_relay_log_info | 0 | > | system_time_zone | UTC | > | table_definition_cache | 400 | > | table_open_cache | 1024 | > | thread_cache_size | 128 | > | thread_concurrency | 10 | > | thread_handling | one-thread-per-connection | > | thread_pool_idle_timeout | 60 | > | thread_pool_max_threads | 500 | > | thread_pool_oversubscribe | 3 | > | thread_pool_size | 8 | > | thread_pool_stall_limit | 500 | > | thread_stack | 294912 | > | time_format | %H:%i:%s | > | time_zone | SYSTEM | > | timed_mutexes | OFF | > | tmp_table_size | 16777216 | > | tmpdir | /tmp | > | transaction_alloc_block_size | 8192 | > | transaction_prealloc_size | 4096 | > | tx_isolation | REPEATABLE-READ | > | unique_checks | ON | > | updatable_views_with_limit | YES | > | userstat | OFF | > | version | 5.5.32-MariaDB-1~wheezy-log | > | version_comment | mariadb.org binary distribution | > | version_compile_machine | x86_64 | > | version_compile_os | debian-linux-gnu | > | wait_timeout | 600 | > I'm planning on doing a debug build from MDEV-572 and maybe try to get valgrind to narrow it down (if that doesn't bring the server to a total halt). Better suggestions welcome. -- This message was sent by Atlassian JIRA (v6.2-OD-05-4#6207) ----- End forwarded message -----

1 0

[Maria-developers] MDEV-5115 review
by Kristian Nielsen 02 Jan '14

02 Jan '14

Hi Serg, Here is my review of your MDEV-5115 patch. First, a general comment: Another way to fix this would be a smaller patch. It would not attempt to implement anything related to the new event types. Instead, it would just have a small bit of code that allows to read the binary format of the new event types, but it would represent them internally as the old type, just ignoring any of the extra information, which appears to be currently unused outside of NDB. Probably then it would also make sense to not implement the renaming of the old event types. There would just be three extra cases in Log_event::read_log_event(), and a bit of extra code in the *_rows_log_event construstructors to be able to ignore any extra info. However, I do not really have much against your approach of doing a partial merge to get code that is closer to what is in MySQL, if you prefer that. I mention it because we discussed briefly already on IRC whether a smaller patch could be enough. A few detailed comments below. Otherwise, while I would have probably done the minimal patch myself, I am ok with this way also. So ok to push after fixing the few commit and code comments mentioned below. - Kristian. > ------------------------------------------------------------ > revno: 3921 > revision-id: sergii(a)pisem.net-20131203192301-afokgpslfq0f3xg8 > parent: sergii(a)pisem.net-20131201111624-xe3f1ul6otcyvyom > fixes bug: https://mariadb.atlassian.net/browse/MDEV-5115 > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 10.0 > timestamp: Tue 2013-12-03 20:23:01 +0100 > message: > MDEV-5115 RBR from MySQL 5.6 to MariaDB 10.0 does not work > > Patially merge WL#5917, to understand v2 row events "Partially merge WL#5917, which introduces new event types for row-based binlogging. This is needed to be able to replicate from a >=5.6 MySQL master to a MariaDB slave." > === modified file 'mysql-test/suite/binlog/t/binlog_old_versions.test' > --- mysql-test/suite/binlog/t/binlog_old_versions.test 2013-04-09 21:27:41 +0000 > +++ mysql-test/suite/binlog/t/binlog_old_versions.test 2013-12-03 19:23:01 +0000 > @@ -24,6 +24,17 @@ > > source include/not_embedded.inc; > > +--echo ==== Read binlog with v2 row events ==== > + > +# Read binlog. > +--exec $MYSQL_BINLOG --local-load=$MYSQLTEST_VARDIR/tmp/ suite/binlog/std_data/ver_trunk_row_v2.001 | $MYSQL --local-infile=1 > +# Show result. > +SELECT * FROM t1 ORDER BY a; > +SELECT * FROM t2 ORDER BY a; > +SELECT COUNT(*) FROM t3; > +# Reset. > +DROP TABLE t1, t2, t3; > + > > --echo ==== Read modern binlog (version 5.1.23) ==== > Is this the only test we have of reading the v2 row event types? I think it is not a particularly good test, as it tests mysqlbinlog reading them, while the interesting thing is a slave receiving them from the master. Even though some code is shared between the two there are lots of differences as well. The better test would be to install the pre-generated binlog file manually on a master and have the slave replicate it. However, it is easy to end up spending too much time on setting up these replication tests and getting them to work reliably on all platforms, so maybe not worth it. Maybe a good compromise is to instead ask Elena to test it manually against a real MySQL 5.6 server - after all, that is the actual case we want to work. > === modified file 'sql/log_event.h' > --- sql/log_event.h 2013-11-01 11:00:11 +0000 > +++ sql/log_event.h 2013-12-03 19:23:01 +0000 > @@ -659,11 +663,11 @@ enum Log_event_type > PRE_GA_DELETE_ROWS_EVENT = 22, > > /* > - These event numbers are used from 5.1.16 and forward > + These event numbers are used from 5.1.16 until mysql-trunk-xx Please change this comment to explain the actual situation. Ie. that they are used in MariaDB, but not in MySQL >= 5.6. > @@ -677,6 +681,20 @@ enum Log_event_type > HEARTBEAT_LOG_EVENT= 27, > > /* > + In some situations, it is necessary to send over ignorable > + data to the slave: data that a slave can handle in case there > + is code for handling it, but which can be ignored if it is not > + recognized. > + */ > + IGNORABLE_LOG_EVENT= 28, > + ROWS_QUERY_LOG_EVENT= 29, Add a comment that these are MySQL 5.6 events that are unimplemented in MariaDB. (Do we need another MDEV to be able to replicate against a MySQL master that produces ROWS_QUERY_LOG_EVENT? It is quite similar to our ANNOTATE_ROWS_EVENT, IIRC). > + /* Version 2 of the Row events */ > + WRITE_ROWS_EVENT = 30, > + UPDATE_ROWS_EVENT = 31, > + DELETE_ROWS_EVENT = 32, Add comment that these are only generated by MySQL 5.6.

2 1

[Maria-developers] review: [Commits] Rev 3917: MDEV-9095: Executing triggers on slave in row-based replication in file:///home/bell/maria/bzr/work-maria-10.0-MDEV-5095/
by Michael Widenius 19 Dec '13

19 Dec '13

Hi! > === modified file 'sql/log_event.cc' > === modified file 'sql/sql_delete.cc' > --- a/sql/sql_delete.cc 2013-10-16 09:38:42 +0000 > +++ b/sql/sql_delete.cc 2013-12-19 08:10:00 +0000 > @@ -525,12 +525,7 @@ bool mysql_delete(THD *thd, TABLE_LIST * > table->triggers->has_triggers(TRG_EVENT_DELETE, > TRG_ACTION_AFTER)) > { > - /* > - The table has AFTER DELETE triggers that might access to subject table > - and therefore might need delete to be done immediately. So we turn-off > - the batching. > - */ > - (void) table->file->extra(HA_EXTRA_DELETE_CANNOT_BATCH); > + table->prepare_triggers_for_delete_stmt_or_event(); > will_batch= FALSE; > } In the above code, you check for this twice: table->triggers->has_triggers(TRG_EVENT_DELETE, TRG_ACTION_AFTER)) Would be better to have prepare_triggers_for_delete_stmt_or_event() return 1 if we have delete triggers and do: if (table->prepare_triggers_for_delete_stmt_or_event()) will_batch= FALSE; else will_batch= ... <cut> > === modified file 'sql/sql_update.cc' > --- a/sql/sql_update.cc 2013-10-16 09:38:42 +0000 > +++ b/sql/sql_update.cc 2013-12-19 08:10:00 +0000 > @@ -707,12 +707,7 @@ int mysql_update(THD *thd, > table->triggers->has_triggers(TRG_EVENT_UPDATE, > TRG_ACTION_AFTER)) > { > - /* > - The table has AFTER UPDATE triggers that might access to subject > - table and therefore might need update to be done immediately. > - So we turn-off the batching. > - */ > - (void) table->file->extra(HA_EXTRA_UPDATE_CANNOT_BATCH); > + table->prepare_triggers_for_update_stmt_or_event(); > will_batch= FALSE; > } Same here. Regards, Monty

1 0

[Maria-developers] review of mdev-5095 (executing triggers on slave)
by Michael Widenius 19 Dec '13

19 Dec '13

Hi! Here is the review of mdev-5095 I got the diff from: bzr diff -r3942.. > === modified file 'mysql-test/r/mysqld--help.result' > --- mysql-test/r/mysqld--help.result 2012-10-18 21:33:06 +0000 > +++ mysql-test/r/mysqld--help.result 2013-12-03 10:53:01 +0000 > @@ -747,6 +747,14 @@ > --slave-net-timeout=# > Number of seconds to wait for more data from a > master/slave connection before aborting the read > + --slave-run-triggers-for-rbr=name > + Modes for how triggers in row-base replication on slave > + side will be executed. Legal values are NO (default), YES > + and LOGGING. NO means that trigger for RBR will not be > + running on slave YES and LOGGING means that triggers will > + be running on slave (if there was not triggers running on > + the naster), LOGGING also mens that flag about executed > + triggers will be written to binlog. Suggested new text: --slave-run-triggers-for-rbr=name Modes for how triggers in row-base replication on slave side will be executed. Legal values are NO (default), YES and LOGGING. NO means that trigger for RBR will not be running on slave. YES and LOGGING means that triggers will be running on slave, if there was not triggers running on the master for the statement. LOGGING also means that the executed triggers will be written to binlog. <cut> > === modified file 'sql/log_event.cc' <cut> > -#if !defined(MYSQL_CLIENT) && defined(HAVE_REPLICATION) > +#if defined(MYSQL_SERVER) && defined(HAVE_REPLICATION) Was the above needed or mostly an optimization? > + > +static void restore_empty_query_table_list(LEX *lex) > +{ > + if (lex->first_not_own_table()) > + (*lex->first_not_own_table()->prev_global)= NULL; > + lex->query_tables= NULL; > + lex->query_tables_last= &lex->query_tables; > +} Add a comment of what is the purpose of the above function. > @@ -8340,6 +8351,22 @@ int Rows_log_event::do_apply_event(Relay > /* A small test to verify that objects have consistent types */ > DBUG_ASSERT(sizeof(thd->variables.option_bits) == sizeof(OPTION_RELAXED_UNIQUE_CHECKS)); > > + if (slave_run_triggers_for_rbr) > + { > + LEX *lex= thd->lex; > + uint8 new_trg_event_map= get_trg_event_map(); > + > + DBUG_ASSERT(lex->query_tables == NULL); > + if ((lex->query_tables= rli->tables_to_lock)) > + rli->tables_to_lock->prev_global= &lex->query_tables; Add a comment why we have to set prev_global (not obvious). It's also not obvious why restore_empty_tables() is not resetting rli->tables_to_lock->prev_global as it's reseting the the rest of the changed things. > @@ -8429,8 +8462,11 @@ int Rows_log_event::do_apply_event(Relay > */ > TABLE_LIST *ptr= rli->tables_to_lock; > for (uint i=0 ; ptr && (i < rli->tables_to_lock_count); ptr= ptr->next_global, i++) > + { > const_cast<Relay_log_info*>(rli)->m_table_map.set_table(ptr->table_id, ptr->table); > - What does this test really do ? (I was quickly checking the code to understand what m_table_id stands for, but it was not obvious). > + if (m_table_id == ptr->table_id) > + master_had_triggers= ((RPL_TABLE_LIST*)ptr)->master_had_triggers; <cut> > if (table) > { > bool transactional_table= table->file->has_transactions(); > @@ -8618,6 +8656,9 @@ int Rows_log_event::do_apply_event(Relay > */ > thd->reset_current_stmt_binlog_format_row(); > thd->is_slave_error= 1; > + /* remove trigger's tables */ > + if (slave_run_triggers_for_rbr) > + restore_empty_query_table_list(thd->lex); > DBUG_RETURN(error); > } You do the above 'restore_empty_query_table_list(thd->lex);' before every return in this function, except the last return. - Isn't it better to do a 'error:' tag at the end of the function to ensure we don't miss any cases ? - Add a comment why we don't need to do this at the very end of Rows_log_event::do_apply_event(). Something like: /* We don't have to call restore_empty_query_table_list() here as it's called in rows_event_stmt_cleanup(). */ When is lex->query_tables used? Is it only used by 'open' ? If yes, isn't it enough to always reset it after the 'open' call? > @@ -9773,6 +9831,29 @@ Write_rows_log_event::do_before_row_oper > from the start. > */ > } > + if (slave_run_triggers_for_rbr && !master_had_triggers && m_table->triggers ) > + { > + if (m_table->triggers->has_triggers(TRG_EVENT_DELETE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER DELETE triggers that might access to > + subject table and therefore might need delete to be done > + immediately. So we turn-off the batching. > + */ > + (void) m_table->file->extra(HA_EXTRA_DELETE_CANNOT_BATCH); > + } > + if (m_table->triggers->has_triggers(TRG_EVENT_UPDATE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER UPDATE triggers that might access to subject > + table and therefore might need update to be done immediately. > + So we turn-off the batching. > + */ > + (void) m_table->file->extra(HA_EXTRA_UPDATE_CANNOT_BATCH); > + } > + } Good that you noticed and fixed the above. However, why not just call the function prepare_triggers_for_insert_stmt() that does exactly the same thing? <cut> > @@ -10749,6 +10894,17 @@ Delete_rows_log_event::do_before_row_ope > */ > return 0; > } > + if (slave_run_triggers_for_rbr && !master_had_triggers && m_table->triggers && > + m_table->triggers->has_triggers(TRG_EVENT_DELETE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER DELETE triggers that might access to subject table > + and therefore might need delete to be done immediately. So we turn-off > + the batching. > + */ > + (void) m_table->file->extra(HA_EXTRA_DELETE_CANNOT_BATCH); > + } Would be better if we could do a function 'prepare_triggers_for_delete_stmt()' and use the same code here and in sql_delete.cc <cut> > @@ -10877,6 +11049,18 @@ Update_rows_log_event::do_before_row_ope > > m_table->timestamp_field_type= TIMESTAMP_NO_AUTO_SET; > > + if (slave_run_triggers_for_rbr && !master_had_triggers && m_table->triggers && > + m_table->triggers->has_triggers(TRG_EVENT_UPDATE, > + TRG_ACTION_AFTER)) > + { > + /* > + The table has AFTER UPDATE triggers that might access to subject > + table and therefore might need update to be done immediately. > + So we turn-off the batching. > + */ > + (void) m_table->file->extra(HA_EXTRA_UPDATE_CANNOT_BATCH); > + } > + Would be better if we could do a function 'prepare_triggers_for_update_stmt()' and use the same code here and in sql_update.cc <cut> Regards, Monty

2 4

[Maria-developers] FW: Rev 3758: MDEV-5470: Cost-based subquery item pushdown must take into account semi-joins
by Sergei Petrunia 18 Dec '13

18 Dec '13

Hi Timour, Please find below a patch that should implement execution-time protection against malfunction between condition moving and semi-joins. There needs also to be an optimization-time counterpart, which should have the same logic. Will you code it yourself or want me to? ----- Forwarded message from Sergei Petrunia <psergey(a)askmonty.org> ----- Date: Thu, 19 Dec 2013 02:49:22 +0400 From: Sergei Petrunia <psergey(a)askmonty.org> To: commits(a)mariadb.org Subject: Rev 3758: MDEV-5470: Cost-based subquery item pushdown must take into account semi-joins At http://bazaar.launchpad.net/~maria-captains/maria/10.0-mdev83 ------------------------------------------------------------ revno: 3758 committer: Sergey Petrunya <psergey(a)askmonty.org> branch nick: 10.0-mdev83 timestamp: Thu 2013-12-19 06:50:28 +0400 message: MDEV-5470: Cost-based subquery item pushdown must take into account semi-joins - When moving expensive conditions, take into account that semi-joins limit where they can be moved. - This is only execution part in JOIN::setup_dynamic_conditions(). Optimization part is not yet present. - No testcases yet. - Moving inside SJ-Materialization nests doesn't seem to be supported (either before this patch or after. This patch itself does take SJ-Materialization into account) === modified file 'sql/item_cmpfunc.cc' --- sql/item_cmpfunc.cc 2013-11-01 11:10:38 +0000 +++ sql/item_cmpfunc.cc 2013-12-19 02:50:28 +0000 @@ -6591,6 +6591,8 @@ void Item_func_collect_stats::dbug_print } #endif /*DBUG_OFF*/ +JOIN_TAB *find_last_tab_for_dynamic_cond(table_map cond_tables, + JOIN *join, JOIN_TAB *join_tab); Item_func_dynamic_cond::Item_func_dynamic_cond(Item *cond, JOIN_TAB *atab, JOIN_TAB **ctab) : Item_func_collect_stats(cond) @@ -6624,13 +6626,15 @@ Item_func_dynamic_cond::Item_func_dynami if (active_tab->bush_root_tab) { first_tab= active_tab->bush_root_tab->bush_children->start; - last_tab= active_tab->bush_root_tab->bush_children->end; + //last_tab= active_tab->bush_root_tab->bush_children->end; } else { first_tab= join->join_tab + join->const_tables; - last_tab= join->join_tab + join->top_join_tab_count; + //last_tab= join->join_tab + join->top_join_tab_count; } + last_tab= find_last_tab_for_dynamic_cond(cond->used_tables(), + active_tab->join, active_tab); DBUG_ASSERT(active_tab >= first_tab && active_tab < last_tab); === modified file 'sql/item_cmpfunc.h' --- sql/item_cmpfunc.h 2013-10-18 19:56:55 +0000 +++ sql/item_cmpfunc.h 2013-12-19 02:50:28 +0000 @@ -561,14 +561,20 @@ class Item_func_collect_stats : public I class Item_func_dynamic_cond : public Item_func_collect_stats { -protected: +protected: // as if there were any ancestor classes? List<Item_subselect> subqueries; /* The join_tab where the condition is currently 'pushed'.*/ st_join_table *active_tab; /* The join_tab currently being executed. */ st_join_table **exec_tab; /* The boundaries of the sub-plan where this predicate can be pushed. */ - st_join_table *first_tab, *last_tab; + st_join_table *first_tab; + st_join_table *last_tab; //psergey-todo: this should point to the last tab. + // conditions depending on semi-join tables have + // limits re. how far they can be moved. +public: + inline st_join_table * get_last_tab() { return last_tab; } +protected: /* The relative access cost based on optimizer estimates. This is the cost of all access methods per one record from the first table in the plan. === modified file 'sql/opt_subselect.cc' --- sql/opt_subselect.cc 2013-11-04 15:56:09 +0000 +++ sql/opt_subselect.cc 2013-12-19 02:50:28 +0000 @@ -1480,6 +1480,7 @@ static bool convert_subq_to_sj(JOIN *par sj_nest->sj_subq_pred= subq_pred; sj_nest->original_subq_pred_used_tables= subq_pred->used_tables() | subq_pred->left_expr->used_tables(); + sj_nest->sj_strategys_join_tab= uint(-1); /* Nests do not participate in those 'chains', so: */ /* sj_nest->next_leaf= sj_nest->next_local= sj_nest->next_global == NULL*/ emb_join_list->push_back(sj_nest); @@ -4321,6 +4322,21 @@ int init_dups_weedout(JOIN *join, uint f /* + For each SJ-inner table in [start_tab; end_tab], set emb_sj_nest to idx +*/ +static void update_sj_strategys_join_tab(JOIN_TAB *start_tab, + JOIN_TAB *end_tab, + uint idx) +{ + for (JOIN_TAB *j= start_tab; j < end_tab; j++) + { + TABLE_LIST *emb_nest; + if ((emb_nest= j->emb_sj_nest)) + emb_nest->sj_strategys_join_tab= idx; + } +} + +/* Setup the strategies to eliminate semi-join duplicates. SYNOPSIS @@ -4448,6 +4464,10 @@ int setup_semijoin_dups_elimination(JOIN for (uint j= i; j < i + pos->n_sj_tables; j++) join->join_tab[j].inside_loosescan_range= TRUE; + + update_sj_strategys_join_tab(join->join_tab + i, + join->join_tab + i + pos->n_sj_tables, + i + pos->n_sj_tables - 1); /* Calculate key length */ keylen= 0; @@ -4461,6 +4481,7 @@ int setup_semijoin_dups_elimination(JOIN tab[pos->n_sj_tables - 1].do_firstmatch= tab; i+= pos->n_sj_tables; pos+= pos->n_sj_tables; + break; } case SJ_OPT_DUPS_WEEDOUT: @@ -4503,6 +4524,9 @@ int setup_semijoin_dups_elimination(JOIN break; } } + update_sj_strategys_join_tab(join->join_tab + i, + join->join_tab + i + pos->n_sj_tables, + i + pos->n_sj_tables - 1); init_dups_weedout(join, first_table, i, i + pos->n_sj_tables - first_table); i+= pos->n_sj_tables; @@ -4558,6 +4582,9 @@ int setup_semijoin_dups_elimination(JOIN } } j[-1].do_firstmatch= jump_to; + update_sj_strategys_join_tab(join->join_tab + i, + join->join_tab + i + pos->n_sj_tables, + i + pos->n_sj_tables - 1); i+= pos->n_sj_tables; pos+= pos->n_sj_tables; === modified file 'sql/sql_select.cc' --- sql/sql_select.cc 2013-11-08 12:36:02 +0000 +++ sql/sql_select.cc 2013-12-19 02:50:28 +0000 @@ -24608,8 +24608,8 @@ double JOIN::static_pushdown_cost(uint i @param pred the predicate to be wrapped in Item_func_dynamic_cond @param init_tab the initial 'active' JOIN_TAB where the dynamic predicate is pushed to - @param cur_tab pointer to the JOIN_TAB* that stores the current JOIN_TAB - during execution + @param active_tab_ptr pointer to the JOIN_TAB* that stores the current JOIN_TAB + during execution */ static Item_func_collect_stats * wrap_with_dynamic_conds(Item *pred, JOIN_TAB *init_tab, JOIN_TAB **active_tab_ptr, @@ -24674,6 +24674,81 @@ wrap_with_dynamic_conds(Item *pred, JOIN } +/* + @brief Find the last JOIN_TAB that a condition can be attached. + + + @param cond_tables used_tables() of the condition that we intend to move. + @param join_tab The left-most join tab where the condition can be + attached (in other words, default attachment point) + + @detail + In a regular inner join, one can take a condition that is attached to a + table $T, and move it to any table that comes after $T. + + When semi-joins are present, this is no longer true. Detailed explanation + can be found in maria-developers@, email "Movable conditions and semi-joins". + In short: + + - Inside an SJ-Materialization nest, a condition can be moved to any table to + the right, but it must stay within the nest. (We don't support nested + semi-joins, so one will not find semi-joins inside an SJ-M nest) + + - All other Semi-Join strategies have a "range" where they apply. If a + condition depends on a table that is within a semi-join, it must not be + moved over the right edge of the range. +*/ + +JOIN_TAB *find_last_tab_for_dynamic_cond(table_map cond_tables, + JOIN *join, JOIN_TAB *join_tab) +{ + if (join_tab->bush_root_tab) + { + /* + We are inside SJ-Materialization nest. There are no semi-joins here, + return the last tab in the nest. + */ + return join_tab->bush_root_tab->bush_children->end; + } + + /* Start from the right-most bound */ + uint last_tab = join->top_join_tab_count; + + cond_tables= cond_tables & ~(PSEUDO_TABLE_BITS | join->const_table_map); + /* + Walk through semi-joins that this table depends on, and check their last + join tabs. The smallest last join tab sets the limit about how far right + the condition can be moved. + */ + Table_map_iterator tm_it(cond_tables); + int tableno; + while ((tableno = tm_it.next_bit()) != Table_map_iterator::BITMAP_END) + { + TABLE_LIST *sj_nest; + if ((sj_nest= join->map2table[tableno]->emb_sj_nest)) + { + if (sj_nest->is_active_sjm()) + { + /* + Whoa this condition also depends on a table from within + SJ-Materialization nest. (Can happen possible because of equality + propagation/substitution). If we didn't return earlier in this + function, we're not inside a semi-join nest. A value that depends + on a table within a SJ-Materialization nest means that it depends on + the materialized table columns. + */ + continue; + } + + if (sj_nest->sj_strategys_join_tab != uint(-1) && + sj_nest->sj_strategys_join_tab < last_tab) + last_tab= sj_nest->sj_strategys_join_tab; + } + } + return join->join_tab + last_tab; +} + + /** Make expensive subqueries dynamically pushdownable. */ @@ -24703,7 +24778,12 @@ bool JOIN::setup_dynamic_conditions() Update the least cardinality join_tab for each tab for the chosen join order. Check if there are any conditions at all that need to be made dynamic. */ - min_tab= NULL; + min_tab= NULL; + /* + psergey-todo: the following loop should walk inside SJ-Materialization + nests, too. Maybe, this whole function should be invoked for each + SJ-Materilization nest. + */ for (tab= last_tab - 1; tab >= first_tab; tab--) { if (optimizer_flag(thd, OPTIMIZER_SWITCH_STATIC_COND_PUSHDOWN)) @@ -24746,7 +24826,8 @@ bool JOIN::setup_dynamic_conditions() */ for (tab= first_tab; tab < last_tab; tab++) { - Item_func_collect_stats *wrapped_select_cond= NULL, *cur_wrapped_cond; + Item_func_collect_stats *wrapped_select_cond= NULL; + Item_func_dynamic_cond *cur_wrapped_cond; if (tab->select_cond && !(wrapped_select_cond= @@ -24800,14 +24881,29 @@ bool JOIN::setup_dynamic_conditions() Add all dynamic conditions pushed to previous JOIN_TABs also to the current JOIN_TAB. This allows to "move" a dynamic condition from one tab to another by enabling it for the corrseponding tab. + + psergey-todo: conditions have limits on how far to the right they can be + moved (Item_func_dynamic_cond::last_tab). Don't attach conditions to + where they can't be moved. */ if (!prev_dynamic_conds.is_empty()) { - List_iterator_fast<Item_func_dynamic_cond> dyn_cond_it(prev_dynamic_conds); - if (!wrapped_select_cond) - wrapped_select_cond= dyn_cond_it++; + List_iterator<Item_func_dynamic_cond> dyn_cond_it(prev_dynamic_conds); while ((cur_wrapped_cond= dyn_cond_it++)) - wrapped_select_cond->add_pred(cur_wrapped_cond); + { + if (!wrapped_select_cond) + wrapped_select_cond= cur_wrapped_cond; + else + wrapped_select_cond->add_pred(cur_wrapped_cond); + + /* + psergey-todo: conditions have limits on how far to the right they can be + moved (Item_func_dynamic_cond::last_tab). Don't attach conditions to + where they can't be moved. + */ + if (cur_wrapped_cond->get_last_tab() == tab) + dyn_cond_it.remove(); + } } prev_dynamic_conds.concat(&cur_dynamic_conds); cur_dynamic_conds.empty(); === modified file 'sql/table.h' --- sql/table.h 2013-09-25 18:07:06 +0000 +++ sql/table.h 2013-12-19 02:50:28 +0000 @@ -1721,6 +1721,9 @@ struct TABLE_LIST table_map sj_inner_tables; /* Number of IN-compared expressions */ uint sj_in_exprs; + + //psergey-todo: add the index of last table in the join order + uint sj_strategys_join_tab; /* If this is a non-jtbm semi-join nest: corresponding subselect predicate */ Item_in_subselect *sj_subq_pred; ----- End forwarded message ----- -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB r3955 make fail; " error: ‘isfinite’ was not declared in this scope"
by darx＠sent.com 18 Dec '13

18 Dec '13

hi, upgrading a MariaDB 10/git src build from r3911 -> r3955, on x86_64 after configure, make fails with a warning & an error @, ... [ 19%] Building CXX object storage/connect/CMakeFiles/connect.dir/ha_connect.cc.o cd /usr/local/src/mariadb/bld/storage/connect && /usr/bin/g++-4.8 -DFORCE_INIT_OF_VARS -DHAVE_CONFIG_H -DHUGE_SUPPORT -DLIBXML2_SUPPORT -DLINUX -DMARIADB -DMYSQL_DYNAMIC_PLUGIN -DMYSQL_SUPPORT -DODBC_SUPPORT -DPIVOT_SUPPORT -DUBUNTU -DUNIX -DZIP_SUPPORT -Dconnect_EXPORTS -Wall -O2 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -march=amdfam10 -mtune=amdfam10 -std=c++11 -felide-constructors -fno-exceptions -fno-rtti -Wall -Wno-unused-parameter -fno-exceptions -fno-rtti -fpermissive -fexceptions -fPIC -O2 -g -DNDEBUG -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1 -fPIC -I/usr/local/src/mariadb/bld/include -I/usr/include/libxml2 -I/usr/local/src/mariadb/include -I/usr/local/src/mariadb/sql -I/usr/local/src/mariadb/bld/pcre -I/usr/local/src/mariadb/pcre -I/usr/local/ssl/include -O2 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -march=amdfam10 -mtune=amdfam10 -Wall -Wmissing-declarations -Wno-write-strings -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-parentheses -o CMakeFiles/connect.dir/ha_connect.cc.o -c /usr/local/src/mariadb/storage/connect/ha_connect.cc In file included from /usr/local/src/mariadb/sql/gcalc_tools.h:21:0, from /usr/local/src/mariadb/sql/spatial.h:28, from /usr/local/src/mariadb/sql/item.h:3667, from /usr/local/src/mariadb/sql/sql_lex.h:26, from /usr/local/src/mariadb/sql/sql_class.h:463, from /usr/local/src/mariadb/storage/connect/ha_connect.cc:104: /usr/local/src/mariadb/sql/gcalc_slicescan.h:29:40: warning: invalid suffix on literal; C++11 requires a space between literal and identifier [-Wliteral-suffix] #define GCALC_DBUG_ENTER(a) DBUG_ENTER("Gcalc "a) ^ In file included from /usr/local/src/mariadb/sql/item.h:3669:0, from /usr/local/src/mariadb/sql/sql_lex.h:26, from /usr/local/src/mariadb/sql/sql_class.h:463, from /usr/local/src/mariadb/storage/connect/ha_connect.cc:104: /usr/local/src/mariadb/sql/item_func.h: In member function ‘double Item_func::check_float_overflow(double)’: /usr/local/src/mariadb/sql/item_func.h:281:26: error: ‘isfinite’ was not declared in this scope return isfinite(value) ? value : raise_float_overflow(); ^ /usr/local/src/mariadb/sql/item_func.h:281:26: note: suggested alternative: In file included from /usr/include/c++/4.8/random:38:0, from /usr/include/c++/4.8/bits/stl_algo.h:65, from /usr/include/c++/4.8/algorithm:62, from /usr/local/src/mariadb/sql/mdl.h:32, from /usr/local/src/mariadb/sql/table.h:22, from /usr/local/src/mariadb/sql/field.h:29, from /usr/local/src/mariadb/sql/unireg.h:172, from /usr/local/src/mariadb/sql/sql_class.h:24, from /usr/local/src/mariadb/storage/connect/ha_connect.cc:104: /usr/include/c++/4.8/cmath:596:5: note: ‘std::isfinite’ isfinite(_Tp __x) ^ make[2]: *** [storage/connect/CMakeFiles/connect.dir/ha_connect.cc.o] Error 1 make[2]: Leaving directory `/usr/local/src/mariadb/bld' make[1]: *** [storage/connect/CMakeFiles/connect.dir/all] Error 2 make[1]: Leaving directory `/usr/local/src/mariadb/bld' make: *** [all] Error 2

1 0

Re: [Maria-developers] metadata_lock_info plugin
by Sergey Vojtovich 13 Dec '13

13 Dec '13

Hi Kentoku, thanks for your contribution and sorry for this long delay. I finally managed to go through your code. The most important thing I'm worried about is that there is no protection of ticket lists: lists may be updated while we iterate them and it may cause server crash in concurrent environment. From what I can see MDL_context is mostly supposed to be accessed only by thread which owns this context. As such it is not protected by any locks. I believe we should use global lists instead, something like: mysql_mutex_lock(&mdl_locks.m_mutex); iterate mdl_locks.m_locks (hash of MDL_lock) { mysql_prlock_rdlock(&lock->m_rwlock); iterate lock->m_granted (list of MDL_ticket) { // store data } mysql_prlock_unlock(&lock->m_rwlock); } mysql_mutex_lock(&mdl_locks.m_mutex); A few more suggestions in no particular order: - instead of making m_tickets public I'd suggest to try to make i_s_metadata_lock_info_fill_table() a friend function (shouldn't be relevant anymore) - why check for thd->killed? (shouldn't be relevant anymore) - metadata_lock_info_lock_mode_length[] includes ending zero, and then we store e.g. "MDL_SHARED\0" whereas we should store "MDL_SHARED" I'd suggest to declare these arrays as following: static const LEX_STRING metadata_lock_info_lock_mode_str[]= { STRING_WITH_LEN("MDL_INTENTION_EXCLUSIVE"), // or C_STRING_WITH_LEN() ... }; - Roberto suggests to rename ID to THREAD_ID and I agree - Roberto suggests to add QUERY_ID column and I agree - Roberto suggests to add LOCK_DURATION column and I agree - Roberto suggests to use m_namespace_to_wait_state_name instead of metadata_lock_info_lock_names, but I tend to disagree because the former says "Waiting for...". Though I'd suggest to make array of LEX_STRING's to avoid calling strdup(). - Roberto suggests to replace DB with KEY and NAME with SUB_KEY, but I tend to disagree since in most cases it is db and table name. I'd vote for TABLE_SCHEMA and TABLE_NAME to stay in line with the rest I_S tables. Regards, Sergey On Mon, Jul 01, 2013 at 03:46:22AM +0900, kentoku wrote: > Hi Sergey, > Sorry. I think this plugin needs more locks "mdl_locks.m_mutex", > "mdl_locks.m_global_lock" and "mdl_locks.m_commit_lock". Is it right? Would > you please teach me how can I use it? > Thanks, > Kentoku > > > 2013/7/1 kentoku <kentokushiba(a)gmail.com> > > > Hi Sergey, > > > > I made new information schema plugin which named "metadata_lock_info" > > plugin. This plugin makes it possible knowing "who has metadata locks". In > > development stage, sometimes DBAs meet metadata locks when using alter > > table statement. This metadata locks are sometimes caused by GUI tools. In > > service stage, sometimes DBAs meet metadata locks when using alter table > > statement. This metadata locks are sometimes caused by long batch > > processing. In many cases, the DBAs need to judge immediately. So I made it > > for all DBAs. > > Would you please review this plugin? > > lp:~kentokushiba/maria/10.0.3-metadata_lock_info > > Note: This plugin needs a change of sql/mdl.h > > > > Thanks, > > Kentoku > >

3 15

Re: [Maria-developers] Rev 3915: MDEV-5388 - Reduce usage of LOCK_open: unused_tables
by Sergei Golubchik 12 Dec '13

12 Dec '13

Hi, Sergey! Did you benchmark the improvement you get with this patch? On Dec 05, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 3915 > revision-id: svoj(a)mariadb.org-20131205084430-7rt735tbanw0i6uq > parent: svoj(a)mariadb.org-20130827121233-b9mguuxctnjlq5wd > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Thu 2013-12-05 12:44:30 +0400 > message: > MDEV-5388 - Reduce usage of LOCK_open: unused_tables > > Removed unused_tables, find LRU object by timestamp instead. > === modified file 'sql/table_cache.cc' > --- a/sql/table_cache.cc 2013-08-27 12:12:33 +0000 > +++ b/sql/table_cache.cc 2013-12-05 08:44:30 +0000 > @@ -324,20 +248,44 @@ void tc_add_table(THD *thd, TABLE *table > DBUG_ASSERT(table->in_use == thd); > mysql_mutex_lock(&LOCK_open); > table->s->tdc.all_tables.push_front(table); > - tc_count++; > /* If we have too many TABLE instances around, try to get rid of them */ > - if (tc_count > tc_size && unused_tables) > + if (tc_count == tc_size) I'd rather put (tc_count >= tc_size), looks safer. > { > - TABLE *purge_table= unused_tables; > - tc_remove_table(purge_table); > - mysql_rwlock_rdlock(&LOCK_flush); > + TDC_iterator tdc_it; > mysql_mutex_unlock(&LOCK_open); > - intern_close_table(purge_table); > - mysql_rwlock_unlock(&LOCK_flush); > - check_unused(thd); > + > + tdc_it.init(); > + mysql_mutex_lock(&LOCK_open); > + if (tc_count == tc_size) and here > + { > + TABLE *purge_table= 0; > + TABLE_SHARE *share; I think a status variable for this would be nice. So that one could make an informed decision about the desired size of the cache. > + while ((share= tdc_it.next())) > + { > + TABLE_SHARE::TABLE_list::Iterator it(share->tdc.free_tables); > + TABLE *entry; > + while ((entry= it++)) > + if (!purge_table || entry->tc_time < purge_table->tc_time) > + purge_table= entry; > + } > + if (purge_table) > + { > + tdc_it.deinit(); > + purge_table->s->tdc.free_tables.remove(purge_table); > + purge_table->s->tdc.all_tables.remove(purge_table); > + mysql_rwlock_rdlock(&LOCK_flush); > + mysql_mutex_unlock(&LOCK_open); > + intern_close_table(purge_table); > + mysql_rwlock_unlock(&LOCK_flush); > + check_unused(thd); > + return; > + } > + } > + tdc_it.deinit(); Okay, so you only purge one table at time? Since purge is kind of an expensive operation, it's better to amortize the cost of it by purging many tables at once. > } > - else > - mysql_mutex_unlock(&LOCK_open); > + /* Nothing to evict, increment tc_count. */ > + tc_count++; > + mysql_mutex_unlock(&LOCK_open); > } > 2 Regards, Sergei

2 6

Re: [Maria-developers] Rev 3914: MDEV-4956 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.used_tables
by Sergei Golubchik 12 Dec '13

12 Dec '13

Hi, Sergey! On Dec 10, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 3914 > revision-id: svoj(a)mariadb.org-20131210150036-0hwh1gy4z3jalwsg > parent: bar(a)mnogosearch.org-20131203101253-mbz05d9jtkb3iycp > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-mdev4956 > timestamp: Tue 2013-12-10 19:00:36 +0400 > message: > MDEV-4956 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.used_tables > > - tc_acquire_table and tc_release_table do not access > TABLE_SHARE::tdc.used_tables anymore > - in tc_acquire_table(): release LOCK_tdc after we relase LOCK_open > (saves a few CPU cycles in critical section) > - in tc_release_table(): if we reached table cache threshold, evict > to-be-released table without moving it to unused_tables. unused_tables > must be empty at this point. Looks ok to me Regards, Sergei

1 0

[Maria-developers] Issue with index condition pushdown being used with no end_range set
by Zardosht Kasheff 12 Dec '13

12 Dec '13

Hello all, I've noticed a scenario with index condition pushdown (ICP) in MariaDB that leads to really bad performance in TokuDB. I don't know if it is a bug in ICP or if TokuDB is misusing/misunderstanding the API. Here is the problem. Suppose we run the following query on the following schema: SELECT * FROM foo WHERE col1 = '7' ORDER BY b desc | foo | CREATE TABLE `foo` ( `id` int(11) NOT NULL AUTO_INCREMENT, `col1` varchar(100) COLLATE utf8_unicode_ci NOT NULL, `b` int(11) DEFAULT NULL, `c` int(11) DEFAULT NULL, `d` int(11) DEFAULT NULL, PRIMARY KEY (`id`), CLUSTERING KEY `col1_2` (`col1`,`b`) ) ENGINE=TokuDB AUTO_INCREMENT=8001 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ROW_FORMAT=TOKUDB_ZLIB | The output of explain is: MariaDB [test]> explain SELECT * FROM foo WHERE col1 = '7' ORDER BY b desc; +------+-------------+-------+------+---------------+--------+---------+-------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+--------+---------+-------+------+-------------+ | 1 | SIMPLE | foo | ref | col1_2 | col1_2 | 302 | const | 1320 | Using where | +------+-------------+-------+------+---------------+--------+---------+-------+------+-------------+ 1 row in set (2.32 sec) The query is doing a reverse range scan, as it should. We get handler::idx_cond_push called, but end_range is not set. As a result, TokuDB thinks it can use index condition pushdown to filter rows. As we do this and get to the end, because end_range is not set, we never get a result of ICP_OUT_OF_RANGE, and always get a result of ICP_NO_MATCH. So, when we go to retrieve that first row past the end of the range, the row that will tell MySQL it should stop searching, TokuDB never finds a row and never gets ICP_NO_MATCH. It scans to the beginning of the index (because it is running in reverse order), getting ICP_NO_MATCH for every row it encounters. Although my index is clustering, I've seen this with a normal key as well. If col1 is an int instead of the funky varchar, handler::idx_cond_push is never called, so this problem does not exist. It seems to me that if end_range is not set and we cannot reliably learn when we go out of range, we should not have handler::idx_cond_push called, otherwise we can get bad performance such as the example above. Thoughts? Thanks --Zardosht

1 1

Re: [Maria-developers] [Commits] Rev 3916: MDEV-5403 - Reduce usage of LOCK_open: tc_count
by Sergei Golubchik 10 Dec '13

10 Dec '13

Hi, Sergey! Not a review, only a comment on TDC_atomic part: 1. This is perfectly generally useful, not TDC specific at all. I'd consider moving it to my_atomic.h under #ifdef __cplusplus 2. These are supposed to be trivial inlined functions. I don't think you should use inheritance and virtual methods here. Everything should be in the template. I understand that function names (add64/add32/etc) are a bit tricky. Two approaches: pass them as template parameters (like in template <typename T, ADD, CAS, ...>) or - perhaps, if you put this into my_atomic.h you'll be able to generate the necessary code without using my_atomic_add32/etc ? Regards, Sergei

2 1

Re: [Maria-developers] [svoj@mariadb.org: [Commits] Rev 3807: MDEV-4956 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.used_tables in lp:maria/10.0]
by Sergei Golubchik 10 Dec '13

10 Dec '13

Hi, Sergey! Looks pretty much ok. See few comments/questions below. On Oct 24, Sergey Vojtovich wrote: > ------------------------------------------------------------ > revno: 3807 > revision-id: svoj(a)mariadb.org-20130827121233-xh1uyhgfwbhedqyf > parent: jplindst(a)mariadb.org-20130823060357-pww92qxla7o8iir7 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Tue 2013-08-27 16:12:33 +0400 > message: > MDEV-4956 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.used_tables > > - tc_acquire_table and tc_release_table do not access > TABLE_SHARE::tdc.used_tables anymore ok > - in tc_acquire_table(): release LOCK_tdc after we relase LOCK_open > (saves a few CPU cycles in critical section) This is a bit counter-intuitive. Your change is to hold the lock more than necessary. Does it give a measurable performance improvement? > - in tc_release_table(): if we reached table cache threshold, evict > to-be-released table without moving it to unused_tables. unused_tables > must be empty at this point (added assertion). Why unused_tables must be empty at this point? > === modified file 'sql/sql_base.cc' > --- a/sql/sql_base.cc 2013-08-14 08:48:50 +0000 > +++ b/sql/sql_base.cc 2013-08-27 12:12:33 +0000 > @@ -370,7 +372,7 @@ void free_io_cache(TABLE *table) > > void kill_delayed_threads_for_table(TABLE_SHARE *share) > { > - TABLE_SHARE::TABLE_list::Iterator it(share->tdc.used_tables); > + TABLE_SHARE::All_share_tables_list::Iterator it(share->tdc.all_tables); In this function, I'd suggest to add some flag to be able to skip the mutex and the loop altogether - as in the absolute majority of cases there will be no delayed insert thread to kill. Either use a global flag to know if there's a delayed thread running, or create a per-share flag. > TABLE *tab; > Regards, Sergei

2 4

[Maria-developers] Reviews of parallel replications
by Michael Widenius 10 Dec '13

10 Dec '13

Hi! Now continuing with reviews of the parallel replication code. Will take one patch at a time with a target to have everything reviewed this week. Review of revno 3678: Parallel replication: error handling. ------------------------------------------------------------ revno: 3678 committer: knielsen(a)knielsen-hq.org branch nick: work-10.0-mdev4506 timestamp: Mon 2013-10-14 15:28:16 +0200 message: MDEV-4506: Parallel replication: error handling. Add an error code to the wait_for_commit facility. Now, when a transaction fails, it can signal the error to any subsequent transaction that is waiting for it to commit. The waiting transactions then receive the error code back from wait_for_prior_commit() and can handle the error appropriately. Also fix one race that could cause crash if @@slave_parallel_threads were changed several times quickly in succession. <cut> === modified file 'sql/handler.cc' --- sql/handler.cc 2013-10-13 21:24:05 +0000 +++ sql/handler.cc 2013-10-14 13:28:16 +0000 @@ -1458,10 +1458,11 @@ int ha_commit_one_phase(THD *thd, bool a transaction.all.ha_list, see why in trans_register_ha()). */ bool is_real_trans=all || thd->transaction.all.ha_list == 0; + int res; DBUG_ENTER("ha_commit_one_phase"); - if (is_real_trans) - thd->wait_for_prior_commit(); - int res= commit_one_phase_2(thd, all, trans, is_real_trans); + if (is_real_trans && (res= thd->wait_for_prior_commit())) + DBUG_RETURN(res); + res= commit_one_phase_2(thd, all, trans, is_real_trans); DBUG_RETURN(res); Another way to write the above is: if (is_real_trans && !(res= thd->wait_for_prior_commit())) res= commit_one_phase_2(thd, all, trans, is_real_trans); DBUG_RETURN(res); <cut> I was checking how ha_commit_one_phase() was used. In ha_commit_trans() or trans_xa_commit(), if ha_commit_one_phase() fails, it will not run rollback. Is this right? I would have assumed that on failure, ha_commit_one_phase() should run ha_rollback_trans() before returning. === modified file 'sql/log.cc' @@ -7844,7 +7844,8 @@ int TC_LOG_MMAP::log_and_order(THD *thd, mysql_mutex_unlock(&LOCK_prepare_ordered); } - thd->wait_for_prior_commit(); + if (thd->wait_for_prior_commit()) + return 0; What does the return 0 mean here? Is it an error? Is thd->is_fatal_error assumed to be set if this happens? A comment here would not be totally wrong... @@ -5633,7 +5634,7 @@ wait_for_commit::~wait_for_commit() void -wait_for_commit::wakeup() +wait_for_commit::wakeup(int wakeup_error) { /* We signal each waiter on their own condition and mutex (rather than using @@ -5649,6 +5650,7 @@ wait_for_commit::wakeup() */ mysql_mutex_lock(&LOCK_wait_commit); waiting_for_commit= false; + this->wakeup_error= wakeup_error; Consider adding an assert that 'this->wakeup_error' is 0. Good way to ensure we don't accidently call this function twice... (as things would be bad if we first would call it with an error and then with 0) mysql_mutex_unlock(&LOCK_wait_commit); mysql_cond_signal(&COND_wait_commit); } Regards, Monty

1 0

[Maria-developers] (MDEV-4403) Attempting to use cassandra storage engine causes "service 'my_snprintf_service' interface version mismatch"
by Sergei Golubchik 10 Dec '13

10 Dec '13

Hi. I don't see my commit emails on the list, so here you are: first patch is the bug fix itself. second - cleanup of the existing code after the bug fix. Regards, Sergei

2 1

[Maria-developers] Coverity
by Christian Convey 09 Dec '13

09 Dec '13

Hi guys, I'm afraid I won't have the time available that I'd hope to, for analyzing MariaDB with Coverity Prevent. Would anyone like to take over from me the Coverity MariaDB account? Thanks, Christian

2 1

[Maria-developers] plugin doesn't load
by AugustQ 06 Dec '13

06 Dec '13

Hi, just one question: I have a simple plugin (a minimal storage engine) that does not load. When I try to load it I get this error-message: MariaDB [(none)]> install plugin dummy soname 'ha_dummy'; ERROR 1126 (HY000): Can't open shared library '/home/august/MariaDB/pgm/lib/plugin/ha_dummy.so' (errno: 2 undefined symbol: _ZN9ha_myisam5cloneEPKcP11st_mem_root) Can you please give me a tip. Thanks. Environment is MariaDB 10.0.4. The same code works without any problems in MySQL (5.5.8 - just for a test). The code is a minimal engine that just does nothing, I want to use it as a template for further work. this is the code in the form that it is working (the relevant parts): in the header-file: class ha_dummy: public handler in the code: ha_dummy::ha_dummy(handlerton *hton, TABLE_SHARE *table_arg) :handler(hton, table_arg) now I change this by modifying the inheritance-order: in the header file: added: #include "../myisam/ha_myisam.h" #include "../myisam/myisamdef.h" modified: class ha_dummy: public ha_myisam in the code: /* Constructor */ ha_dummy::ha_dummy(handlerton *hton, TABLE_SHARE *table_arg) :ha_myisam(hton, table_arg) It compiles but I cannot load it. What's the problem with st_mem_root? Thanks AugustQ PS: as written before: both versions work with MySQL.

3 4

Re: [Maria-developers] [Commits] Rev 3732: Fixed bug mdev-5382 in file:///home/igor/maria/maria-5.3-mdev5382/
by Oleksandr Byelkin 05 Dec '13

05 Dec '13

Hi! 05.12.2013 21:13, Igor Babaev пишет: > At file:///home/igor/maria/maria-5.3-mdev5382/ > > ------------------------------------------------------------ > revno: 3732 > revision-id: igor(a)askmonty.org-20131205191320-pfja761po2x9p9ri > parent: bar(a)mnogosearch.org-20131203110843-1bsdu0hcrpzhie8w > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-mdev5382 > timestamp: Thu 2013-12-05 11:13:20 -0800 > message: > Fixed bug mdev-5382 > When marking used columns the function find_field_in_table_ref() erroneously > called the walk method for the real item behind a view/derived table field > with the second parameter set to TRUE. > This erroneous code was introduced in 2006. OK to push! > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] RQG crashes tokudb due to unexpected storage engine API call sequence
by Rich Prohaska 05 Dec '13

05 Dec '13

We are debugging a crash induced by RQG in the tokudb storage engine. The cause of the crash is a mismatch in assumptions about the sequence of storage engine API calls. We see: tokudb::external_lock creates a txn tokudb::index_init creates a cursor on a index db with the txn tokudb::index_read use the cursor to try to read a row, and returns an error tokudb_commit commits the txn (called from sql_parse.cc:4584) tokudb::index_end tries to close the cursor and crashes since the txn has been committed. (called from the unit.cleanup() call at sql_parse.cc:4592) The current tokudb software does not expect the commit to occur before index_end is called. Is this assumption correct? Since mariadb (and i suspect mysql) does this, it appears that the storage engine should expect commit before index_end. Any comments? Thanks Rich Prohaska

2 2

[Maria-developers] MDEV-5345 - Deadlock between mysql_change_user(), SHOW VARIABLES and INSTALL PLUGIN
by Sergey Vojtovich 04 Dec '13

04 Dec '13

Hi Sergei, I'm afraid I couldn't find easy fix for this deadlock. The question is if it is worth to spend time for more complex (and possibly less stable) solution. Details below. Locking profile --------------- INSTALL PLUGIN: LOCK_plugin -> LOCK_system_variables_hash (w) SHOW VARIABLES: LOCK_system_variables_hash (r) -> LOCK_global_system_variables change_user(): LOCK_global_system_variables -> LOCK_plugin INSTALL PLUGIN -------------- Needs LOCK_plugin to register new plugin. Needs LOCK_system_variables_hash to register plugin variables. Doesn't seem to need both locks at the same time. Similar lock order is used in a few places. SHOW VARIABLES -------------- Needs LOCK_system_variables_hash to iterate system_variable_hash. Needs LOCK_global_system_variables to read variable value. Does seem to need both locks at the same time. This lock order is not used in other places. change_user() ------------- Needs LOCK_global_system_variables to read global_system_variable.table_plugin. Needs LOCK_plugin to my_plugin_lock(table_plugin). Does seem to need both locks at the same time. Similar lock order is used in a few places. "SHOW VARIABLES" looks correct. So I have only two ideas: - Fix INSTALL PLUGIN so it releases LOCK_plugin before registering variables. Sounds like the best solution, but there are a few more things to fix, e.g. UNINTALL PLUGIN. - Store default storage engine as a string, not Sys_var_plugin. Thanks, Sergey

2 3

[Maria-developers] Build fail of MariaDB10 with TokuDB enabled
by darx＠sent.com 03 Dec '13

03 Dec '13

On the (eventual) road to successfully building MariaDB10 + TokuDB, Building cd src/mariadb bzr log | head ------------------------------------------------------------ revno: 3911 committer: Alexander Barkov <bar(a)mnogosearch.org> branch nick: maria-10.0 timestamp: Tue 2013-11-26 10:53:21 +0400 message: Fixing malformed data in mysql-test/std_data/Index.xml ------------------------------------------------------------ revno: 3910 committer: Sergey Vojtovich <svoj(a)mariadb.org> on uname -a Linux dev1 3.7.10-1.16-desktop #1 SMP PREEMPT Fri May 31 20:21:23 UTC 2013 (97c14ba) x86_64 x86_64 x86_64 GNU/Linux lsb_release -ar LSB Version: core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch Distributor ID: openSUSE project Description: openSUSE 12.3 (x86_64) Release: 12.3 Codename: Dartmouth with gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib64/gcc/x86_64-suse-linux/4.8/lto-wrapper Target: x86_64-suse-linux Configured with: ../configure --prefix=/usr --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.8 --enable-ssp --disable-libssp --disable-plugin --with-bugurl=http://bugs.opensuse.org/ --with-pkgversion='SUSE Linux' --disable-libgcj --disable-libmudflap --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --enable-version-specific-runtime-libs --enable-linker-build-id --enable-linux-futex --program-suffix=-4.8 --without-system-libunwind --with-arch-32=i586 --with-tune=generic --build=x86_64-suse-linux --host=x86_64-suse-linux Thread model: posix gcc version 4.8.2 20131016 [gcc-4_8-branch revision 203692] (SUSE Linux) cmake --version cmake version 2.8.12.1 applying a needed src patch cd /usr/local/src/mariadb # BUG @ http://www.linuxfromscratch.org/blfs/view/svn/server/mariadb.html sed -i "s/srv_buf_size/srv_sort_buf_size/" storage/innobase/row/row0log.cc after cmake, with gcc c++11 support NOT enabled, mkdir bld cd bld cmake .. \ ... \ -DWITHOUT_TOKUDB=1 build completes successfully, make ... ldd sql/mysqld linux-vdso.so.1 (0x00007fffb9dff000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9031e86000) libaio.so.1 => /lib64/libaio.so.1 (0x00007f9031c84000) libz.so.1 => /lib64/libz.so.1 (0x00007f9031a6e000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f9031832000) libssl.so.1.0.0 => /usr/local/ssl/lib64/libssl.so.1.0.0 (0x00007f90315c6000) libcrypto.so.1.0.0 => /usr/local/ssl/lib64/libcrypto.so.1.0.0 (0x00007f90311da000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f9030fd6000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f9030cce000) libm.so.6 => /lib64/libm.so.6 (0x00007f90309d0000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f90307b9000) libc.so.6 => /lib64/libc.so.6 (0x00007f903040b000) /lib64/ld-linux-x86-64.so.2 (0x00007f90320a2000) repeating, but with TokuDB build enabled, cd bld cmake .. \ ... \ - -DWITHOUT_TOKUDB=1 + -DWITHOUT_TOKUDB=0 build now fails, make ... [ 29%] Built target perfschema Scanning dependencies of target tokuportability_static_conv [ 29%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/huge_page_detection.cc.o [ 29%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/file.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/memory.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/os_malloc.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/portability.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_assert.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_crash.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_fair_rwlock.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_path.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_pthread.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static_conv.dir/toku_time.cc.o Linking CXX static library libtokuportability_static_conv.a [ 30%] Built target tokuportability_static_conv [ 30%] Generating tokuportability_static_depends.cc Scanning dependencies of target tokuportability_static [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability_static.dir/tokuportability_static_depends.cc.o Linking CXX static library libtokuportability_static.a /usr/bin/ar: creating /usr/local/src/mariadb/bld/storage/tokudb/ft-index/portability/libtokuportability_static.a [ 30%] Built target tokuportability_static Scanning dependencies of target tokuportability [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/huge_page_detection.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/file.cc.o [ 30%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/memory.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/os_malloc.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/portability.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_assert.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_crash.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_fair_rwlock.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_path.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_pthread.cc.o [ 31%] Building CXX object storage/tokudb/ft-index/portability/CMakeFiles/tokuportability.dir/toku_time.cc.o Linking CXX shared library libtokuportability.so [ 31%] Built target tokuportability Scanning dependencies of target logformat [ 31%] Building CXX object storage/tokudb/ft-index/ft/CMakeFiles/logformat.dir/logformat.cc.o Linking CXX executable logformat [ 31%] Built target logformat Scanning dependencies of target generate_log_code [ 31%] Generating log_code.cc, log_print.cc, log_header.h ./logformat: error while loading shared libraries: libtokuportability.so: cannot open shared object file: No such file or directory make[2]: *** [storage/tokudb/ft-index/ft/log_code.cc] Error 127 make[1]: *** [storage/tokudb/ft-index/ft/CMakeFiles/generate_log_code.dir/all] Error 2 make: *** [all] Error 2 yes, I'm aware that opensuse is not a currently "supported platform" for TokuDB. If that's a dealbreaker, I'd like to know MariaDB project's position, if any, on getting that support in place. If it's not, I'd like to keep trying to get it working, here. darx

4 15

Re: [Maria-developers] MDEV-4956 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.used_tables
by Sergei Golubchik 03 Dec '13

03 Dec '13

Hi, Sergey! On Aug 27, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 3807 > revision-id: svoj(a)mariadb.org-20130827121233-xh1uyhgfwbhedqyf > parent: jplindst(a)mariadb.org-20130823060357-pww92qxla7o8iir7 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Tue 2013-08-27 16:12:33 +0400 > message: > MDEV-4956 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.used_tables > > - tc_acquire_table and tc_release_table do not access > TABLE_SHARE::tdc.used_tables anymore > - in tc_acquire_table(): release LOCK_tdc after we relase LOCK_open > (saves a few CPU cycles in critical section) > - in tc_release_table(): if we reached table cache threshold, evict > to-be-released table without moving it to unused_tables. unused_tables > must be empty at this point (added assertion). I don't understand what you're doing here, could you elaborate? E.g. explain in a changeset comment what you've done, why you introduced a new list for tables (all_share), what is its semantics, etc. Regards, Sergei

3 12

[Maria-developers] Building TOKUDB into 5.5.34 from sources
by md＠rpzdesign.com 02 Dec '13

02 Dec '13

I am trying to build 5.5.34 from source on Linux x64 (Fedora 14). I was looking at the readme files and could not find anything about building 5.5.34 from source and including TokuDB (which has the source code in the /storage/TokuDB/ directory) into the build. I was looking for something like: -DWITH_TOKUDB_STORAGE_ENGINE=ON but did not stumble across anything. Anybody got some ideas? Thanks, Marco Here is my build script: rm -f CMakeCache.txt make clean cmake \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/mysql \ -DENABLED_PROFILING=OFF \ -DENABLE_DEBUG_SYNC=ON \ -DENABLE_GCOV=OFF \ -DINSTALL_LAYOUT=STANDALONE \ -DMYSQL_DATADIR=/mysql/data \ -DINSTALL_INCLUDEDIR=include/mysql \ -DINSTALL_UNIX_ADDRDIR=/mysql/mysql.sock \ -DUSE_ARIA_FOR_TMP_TABLES=ON \ -DWITHOUT_PBXT_STORAGE_ENGINE=ON \ -DWITHOUT_SERVER=OFF \ -DWITH_ARCHIVE_STORAGE_ENGINE=OFF \ -DWITH_EMBEDDED_SERVER=OFF \ -DWITH_FAST_MUTEXES=OFF \ -DWITH_FEDERATEDX_STORAGE_ENGINE=OFF \ -DWITH_FEEDBACK=OFF \ -DWITH_LIBWRAP=OFF \ -DWITH_PIC=ON \ -DWITH_PARTITION_STORAGE_ENGINE=OFF \ -DWITH_PERFSCHEMA_STORAGE_ENGINE=ON \ -DWITH_SPHINX_STORAGE_ENGINE=OFF \ -DWITH_SSL=OFF \ -DWITH_UNIT_TESTS=OFF \ -DWITH_VALGRIND=OFF \ -DWITH_XTRADB_STORAGE_ENGINE=ON \ -DWITHOUT_OQGRAPH=ON \ -DWITH_ZLIB=bundled make (Sorry, I posted this question on maria.com knowledge base open questions also) https://mariadb.com/kb/en/cmake-settings-to-include-tokudb-in-5534/

3 2

[Maria-developers] Movable conditions and semi-joins
by Sergei Petrunia 27 Nov '13

27 Nov '13

Hi! The below is an extension of the skype discussion. I wrote down the limitation imposed by FirstMatch, and also covered other Semi-join strategies. <contents> 1. SJ-Materialization 2. FirstMatch 3. Duplicate Elimination 4. Loose Scan </contents> == 1. SJ-Materialization == The general rule is: If the condition is originally attached inside an SJM nest, one should not move it outside. Conditions that are originally outside should not be moved inisde the SJM nest. The reason for the rule is that field values for inner tables are only available inside the SJM nest. Correspondingly, values of fields of outer tables are not available at materialization step. The only possible exception is when the condition depends only on columns that are in the IN-equality. Then, it could be moved. I don't think this is worth handling for the first milestone. == 2. FirstMatch == Consider a query select ... from ot1, nt1, nt2 where expr(ot1) in (select ... from it1, it2, ... where ...) and a join order ot1 FirstMatch(it1, it2) nt1 nt2 It is possible to move cond(otX) to table itX or ntX. There is a problem with moving condition attached to itX (will call it SUBQ_COND. this ocndition depends on table itX and maybe on ot1) to table ntX. Consider join execution execution: ot1.row0 ot1.row1 -+- it1.row1 -- it2.row1 | +- it1.row2 -- it2.row2 ->- (F1)-->-- nt1.row1 (F2)--<-- FirstMatch check is at point (F2). At that point, we know that ot1.row1 has had a match in the subquery, and the execution returns to table ot1. If SUBQ_COND is moved to table ntX, then there may be a situation where we will get to point (F2) and assume that the subquery had a match, while actually it hasn't. A suggestion by Igor: at point (F2) we should check the last return value of SUBQ_COND (actually, of all conditions that were moved like SUBQ_COND was) It is actually more complicated: what if there is COND(ot1, nt1) such that COND(ot1.ro1, nt1.row1) = FALSE? Then, we will never evaluate SUBQ_COND(ot1.row1, ...). What if never evaluate SUBQ_COND(ot1.row1, ...) but before we have evaluated SUBQ_COND(ot1.row0, ...)? We will have to track what we have evaluated for the current row of ot1.row1. == 3. Duplicate Elimination == Duplicate Elimination removes duplicate matches using a temporary table. In order to work correctly, filtering needs to be performed before the weedout is done. Let's again use an example. select ... from ot1, nt1, nt2 where expr(ot1) in (select ... from it1, it2, ... where ...) and a join order (DUPS-W-START) it1 ot1 it2 (DUPS-W-WEEDOUT) nt1 nt2 at DUPS-W-START, all rows are deleted from the temp. table. at DUPS-W-WEEDOUT, we take ot1.rowid and attempt to write it into the temp table. Basic idea: If a condition SUBQ_COND(itX, ...) is moved to the right across the (DUPS-W-WEEDOUT) point, wrong query results will occur. Example: Suppose, SUBQ_COND(it2) is moved to nt1, and join execution proceeds as follows: it1.row1 --- ot1.row1 --- it2.row1 -- DUPS-W-WEEDOUT -- nt1 | - We reach DUPS-W-WEEDOUT with (it1.row1, ot1.row1, it2.row1). - The temp. table is empty, so rowid(ot1.row1) is written into temp.table, and Weedout doesn't weed it out. - At nt1, SUBQ_COND(it2.row1) is evaluated and found to be false. join execution returns back. - Suppose, tables it2 and ot1 have no more matching rows, so execution returns to it1. (the same diagram with a few more steps) it1.row1 --- ot1.row1 --- it2.row1 -- DUPS-W-WEEDOUT -- nt1 | it1.row2 --- ot1.row1 --- it2.row2 -- DUPS-W-WEEDOUT - We reach DUPS-W-WEEDOUT with (it1.row2, ot1.tow1, it2.rowX). - The temptable already has rowid(ot1.row1). The record combination is filtered out. if SUBQ_COND(it2.row2)=TRUE, then the result will miss a row. == 4. Loose Scan == Loose Scan is similar to First Match - it does short-cutting (but also makes forward jumps on the driving table). Short-cutting imposes a limitation similar to FirstMatch - conditions that depend on subquery's tables cannot be moved out of LooseScan's range. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] maria-tokudb-engine-10.0 deb package (raring) - if files are empty where are sources?
by Paul Hewson 27 Nov '13

27 Nov '13

Hello, I'm trying to follow the instructions on https://mariadb.com/kb/en/how-to-enable-tokudb-in-mariadb/ which imply there is more tuned version of tokudb for mariadb than I might get by following instructions at tokutek. However, I have ubuntu 13.04 (raring), and despite some mismatch in numbers (I have maria 10.0 rather than 5.5) I am getting an empty file when I try to apt-get install mariadb-tokudb-engine-10.0 I found a bit of discussion about this point, but got the impression it had been fixed. Does it mean it was fixed and the debs were removed? If so, do you mind if I ask where I get the mariadb tokutek sources. Again the webpage https://mariadb.com/kb/en/how-to-enable-tokudb-in-mariadb/ implies only some parts (most parts) of totudb have been merged into mariadb. Thanks, sorry for troubling you with naive issues like this. Paul ________________________________ [http://www.plymouth.ac.uk/images/email_footer.gif]<http://www.plymouth.ac.uk/worldclass> This email and any files with it are confidential and intended solely for the use of the recipient to whom it is addressed. If you are not the intended recipient then copying, distribution or other use of the information contained is strictly prohibited and you should not rely on it. If you have received this email in error please let the sender know immediately and delete it from your system(s). Internet emails are not necessarily secure. While we take every care, Plymouth University accepts no responsibility for viruses and it is your responsibility to scan emails and their attachments. Plymouth University does not accept responsibility for any changes made after it was sent. Nothing in this email or its attachments constitutes an order for goods or services unless accompanied by an official order form.

2 2

[Maria-developers] Malformed XML in mysql-test/std_data/Index.xml added in revision 3873
by Jeremy Cole 26 Nov '13

26 Nov '13

Bar, Regarding your change: http://bazaar.launchpad.net/~maria-captains/maria/10.0/revision/3873/mysql-… I was wondering if it's intentional that mysql-test/std_data/Index.xml now contains malformed XML. It wasn't mentioned in your commit message and the test nor XML files do not mention it specifically. Although utf8_5624_5_bad is supposed to be "bad" I didn't think it was supposed to be *that* "bad" (mismatched "p" vs "a" tags): $ xmllint --noout mysql-test/std_data/Index.xml mysql-test/std_data/Index.xml:126: parser error : Opening and ending tag mismatch: p line 126 and a <reset>a-a4</reset><p>xxx04</a> ^ mysql-test/std_data/Index.xml:127: parser error : Opening and ending tag mismatch: p line 127 and a <reset>a-aa5</reset><p>xxx05</a> ^ mysql-test/std_data/Index.xml:128: parser error : Opening and ending tag mismatch: p line 128 and a <reset>a-aaa6</reset><p>xxx06</a> ^ mysql-test/std_data/Index.xml:129: parser error : Opening and ending tag mismatch: p line 129 and a <reset>a-aaaa7</reset><p>xxx07</a> ^ mysql-test/std_data/Index.xml:130: parser error : Opening and ending tag mismatch: p line 130 and a <reset>a-aaaaa8</reset><p>xxx08</a> ^ mysql-test/std_data/Index.xml:131: parser error : Opening and ending tag mismatch: p line 131 and a <reset>a-aaaaaa9</reset><p>xxx09</a> ^ mysql-test/std_data/Index.xml:132: parser error : Opening and ending tag mismatch: p line 132 and a <reset>a-aaaaaa10</reset><p>xxx10</a> ^ Regards, Jeremy

2 1

[Maria-developers] Force PK option and global lock for DDLs
by erkan yanar 26 Nov '13

26 Nov '13

Ahoi, I would like to propagate two ideas (did it on #irc some times but still some love is missing) 1. Force PK option. I would like to have an option to force PK for (innodb) tables. Forcing is quite easy (a poc from 2011): http://linsenraum.de/erkules/2011/03/this-table-type-requires-a-primary-key… Why? RBR and so Galera would benefit from such an option. 2. Global DDL Lock Without Global DDL Lock i.e. ALTER TABLE, DROP TABLE etc. brings you in trouble while doing a --single-transaction dump. You can use flush tables with read lock, but then you need not to do a single-transaction dump anyway. Even xtrabackup got troubles: https://bugs.launchpad.net/percona-xtrabackup/+bug/722638 I got to confess I think ignoring tablespace ids is the right soltution. Even I think this are simple options. I believe they would be quite welcome to a lot of dba's out there. Regards Erkan -- über den grenzen muß die freiheit wohl wolkenlos sein

1 1

Re: [Maria-developers] [Commits] Rev 3908: MDEV-5277 - Ensure that all MySQL 5.6 options are supported
by Sergei Golubchik 25 Nov '13

25 Nov '13

Hi, Sergey! On Nov 25, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 3908 > revision-id: svoj(a)mariadb.org-20131125144940-nhdqjt2eisqaa7pb > parent: svoj(a)mariadb.org-20131114140000-5eq8gopqu5giwvqq > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-mdev5277 > timestamp: Mon 2013-11-25 18:49:40 +0400 > message: > MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the > MariaDB 10.0 server Looks ok to me, thanks! Regards, Sergei

1 0

Re: [Maria-developers] Rev 3985: MDEV-5321 Calling mysql_library_end accesses freed memory; dumps memory to display
by Sergei Golubchik 25 Nov '13

25 Nov '13

Hi, Holyfoot! On Nov 24, holyfoot(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3985 > revision-id: holyfoot(a)askmonty.org-20131124184915-spkpy6qt8vdbluaz > parent: igor(a)askmonty.org-20131123023813-0mgl217lyyyysv5b > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5321 > timestamp: Sun 2013-11-24 22:49:15 +0400 > message: > MDEV-5321 Calling mysql_library_end accesses freed memory; dumps memory to display. > Don't call the vio_end() in the clean_up() in EMBEDDED mode. > Call vio_end() before the end_embedded_server(). ok to push Regards, Sergei

1 0

Re: [Maria-developers] Rev 3908: MDEV-5277 - Ensure that all MySQL 5.6 options are supported
by Sergei Golubchik 25 Nov '13

25 Nov '13

Hi, Sergey! On Nov 20, Sergey Vojtovich wrote: > revno: 3908 > revision-id: svoj(a)mariadb.org-20131120113753-vuxj3v4jt9cvccdy > parent: svoj(a)mariadb.org-20131114140000-5eq8gopqu5giwvqq > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-mdev5277 > timestamp: Wed 2013-11-20 15:37:53 +0400 > message: > MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the > MariaDB 10.0 server > === modified file 'mysql-test/r/mysqld--help.result' > --- a/mysql-test/r/mysqld--help.result 2013-11-03 20:26:44 +0000 > +++ b/mysql-test/r/mysqld--help.result 2013-11-20 11:37:53 +0000 > @@ -4,6 +4,9 @@ > --defaults-file=# Only read default options from the given file #. > --defaults-extra-file=# Read this file after the global files are read. > > + --abort-slave-event-count[=name] > + MySQL 5.6 compatibility option. Use > + '--debug-abort-slave-event-count' instead. > --allow-suspicious-udfs Do we really want all these do-nothing compatibility options to show up in --help? May be not? The goal of this MDEV was not to fail with an error when starting on mysqld's my.cnf file, no one needs --help for that. > Allows use of UDFs consisting of only one symbol xxx() > without corresponding xxx_init() or xxx_deinit(). That > @@ -136,10 +151,16 @@ > --deadlock-timeout-short=# > Short timeout for the two-step deadlock detection (in > microseconds) > + --default-authentication-plugin[=name] > + MySQL 5.6 compatibility option. To be implemented in > + later 10.x versions. Remove "10.x" please, just "in later versions". We don't want to promise that all these options will be implemented in 10.x series. > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- a/storage/innobase/handler/ha_innodb.cc 2013-11-13 21:58:19 +0000 > +++ b/storage/innobase/handler/ha_innodb.cc 2013-11-20 11:37:53 +0000 > @@ -15750,6 +15750,12 @@ static MYSQL_SYSVAR_BOOL(purge_stop_now, > PLUGIN_VAR_OPCMDARG, > "Set purge state to STOP", > NULL, purge_stop_now_set, FALSE); > + > +static my_bool innodb_log_checkpoint_now; > +static MYSQL_SYSVAR_BOOL(log_checkpoint_now, innodb_log_checkpoint_now, > + PLUGIN_VAR_OPCMDARG | PLUGIN_VAR_NOSYSVAR, > + "MySQL 5.6 compatibility option. To be implemented in later 10.x versions.", > + NULL, NULL, FALSE); > #endif /* UNIV_DEBUG */ Why is that (and everything else in ha_innodb.cc) necessary? We'll merge with the latest InnoDB, we will have all its variables. Regards, Sergei

2 3

[Maria-developers] MariaDB-10.0-beta sysbench results
by Axel Schwenke 22 Nov '13

22 Nov '13

Hi, I have run another sysbench with in-memory workload to check where we are with respect to mutex contention. The benchmark is sysbench OLTP (using the bzr trunk of sysbench with LUA support). 10 mio rows in a single table (about 2.5G .ibd file). InnoDB buffer pool set to 32G and warmed up. I have run the test on two different machines. First on our old trusty facebook2 machine with 8 cores/16 threads in two NUMA zones. Second on the fat lizard2 machine with 32 cores/64 threads in 4 NUMA zones. As expected, the heavy contention of the LOCK_open mutex is more visible on the bigger machine (I have not done any detailed analysis in this benchmark, but before while doing benchmarks for MDEV-5081) I notice a visible performance gain in MariaDB-10.0 over MariaDB-5.5 at 32 threads. This is probably due to the work on the InnoDB kernel mutex (impossible to prove since 5.5 doesn't have mutex instrumentation). At higher concurrency however the LOCK_open mutex kicks in and performance is clearly dominated by that mutex alone (it flattens out for all 5.5 and 10.0 versions of MariaDB) The good news is, that 10.0.4, 10.0.5 and 10.0.6 are quite close. There is a small regression from 10.0.4 to 10.0.5, but IMHO this isn't worth investigation while the LOCK_open issue is not fixed. Attached: diagrams. OpenOffice spreadsheets. XL

6 7

[Maria-developers] get is now a reserved word in mariadb 10 ?
by Jocelyn Fournier 21 Nov '13

21 Nov '13

Hi, I have a query with a column named "get". i.e. SELECT get FROM my_table... It works perfectly in 5.5, but in 10.0.6 it fails : ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'get FROM my_table' at line 1 I assume "get" is a new reserved word, probably inherited from MySQL 5.6 memcached support, although memcached is not supported in MariaDB ? (it's mentionned here : http://dev.mysql.com/doc/mysqld-version-reference/en/mysqld-version-referen… ) Is there a specific KB entry for MariaDB 10.0.6 reserved words ? (I only see an entry for 5.5 here : https://mariadb.com/kb/en/reserved-words/ ) Thanks and regards, Jocelyn Fournier

4 4

[Maria-developers] Prep for MariaDB 5.5.34
by Daniel Bartholomew 20 Nov '13

20 Nov '13

I'm currently in the process of preparing for the MariaDB 5.5.34 release. Draft changelog and release notes are here: - https://mariadb.com/kb/en/mariadb-5534-release-notes/ - https://mariadb.com/kb/en/mariadb-5534-changelog/ As always, fixes, additions, and such are welcome. Buildbot is still working on building the packages and when it's finished I'll get them prepped, signed, and uploaded to the mirrors with the plan to release tomorrow morning US Eastern Time. Thanks. -- Daniel Bartholomew MariaDB

1 0

[Maria-developers] MariaDB 10.0.6 using more file descriptors than 5.5 ?
by Jocelyn Fournier 20 Nov '13

20 Nov '13

Hi, I've just migrated MariaDB (+tokudb) from 5.5.33a to 10.0.6, and I've encountered a '[ERROR] Error in accept: Too many open files' error during a backup. As this was not occuring with MariaDB 5.5, is this expected to have 10.0.6 using much more file descriptors ? (open_files_limit was set to about 8100 with 5.5, I raised it to 65535 now to fix this issue). Thanks, Jocelyn Fournier

3 3

[Maria-developers] MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the MariaDB 10.0 server
by Sergey Vojtovich 20 Nov '13

20 Nov '13

Hi! I'm gathering 5.6 options that are not present in 10.0 and the way we want to handle them. Could you check options listed below and if you're aware of their destiny indicate if options: * not used or needed in MariaDB * to be implemented in later 10.x versions * tuned via different 10.x specific options The following options exist in 5.6 but not in 10.0: * default-tmp-storage-engine * log-raw * default_authentication_plugin * ignore-db-dir * binlog-max-flush-queue-time * binlog-row-image * explicit-defaults-for-timestamp * master-info-repository * relay-log-info-repository * binlog-rows-query-log-events * binlog-order-commits * log-throttle-queries-not-using-indexes * end-markers-in-json * optimizer-trace * optimizer-trace-features * optimizer-trace-offset * optimizer-trace-limit * optimizer-trace-max-mem-size * eq-range-index-dive-limit * server-uuid * server-id-bits * slave-rows-search-algorithms * table-open-cache-instances * slave-allow-batching * slave-checkpoint-period * slave-checkpoint-group * slave-parallel-workers * slave-pending-jobs-size-max * enforce-gtid-consistency * gtid-mode * disconnect-on-expired-password * sha256-password-private-key-path (sha256_password plugin) * sha256-password-public-key-path (sha256_password plugin) The following options exist in 5.5 and 5.6 but not in 10.0: * abort-slave-event-count => debug-abort-slave-event-count * disconnect-slave-event-count => debug-disconnect-slave-event-count * exit-info => debug-exit-info * max-binlog-dump-events => debug-max-binlog-dump-events * sporadic-binlog-dump-fail => debug-sporadic-binlog-dump-fail * new * skip-new * skip-stack-trace The following options were added after 5.6.10: * innodb-buffer-pool-evict // UNIV_DEBUG * innodb-ft-total-cache-size * innodb-ft-result-cache-limit * innodb-log-compressed-pages * innodb-log-checkpoint-now // UNIV_DEBUG * rpl-stop-slave-timeout * validate-user-plugins The following NO_CMD_LINE options exist in 5.6 but not in 10.0 (won't add them): * gtid-executed * gtid-next * gtid-next-list * gtid-owned * gtid-purged * log-bin-basename * relay-log-basename * transaction-allow-batching Thanks, Sergey

4 12

Re: [Maria-developers] Rev 3974: MDEV-5182 - build of 10.0.4/r3863 fails @ 'cmake'
by Sergei Golubchik 20 Nov '13

20 Nov '13

Hi, Sergey! On Nov 18, Sergey Vojtovich wrote: > ------------------------------------------------------------ > revno: 3974 > revision-id: svoj(a)mariadb.org-20131118114801-1kny2cvx8kpjsw7t > parent: jplindst(a)mariadb.org-20131115132442-5d4hqg5idjta2dni > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5-mdev5182 > timestamp: Mon 2013-11-18 15:48:01 +0400 > message: > MDEV-5182 - build of 10.0.4/r3863 fails @ 'cmake' with > -DINSTALL_SYSCONFDIR/-DDEFAULT_SYSCONFDIR specified > > The reason for a bug is in support-files/CMakeLists.txt that tries to > install files in INSTALL_SYSCONF2DIR, without checking whether this > variable is actually defined. > > Normally INSTALL_SYSCONF2DIR is always defined if INSTALL_SYSCONFDIR is > defined, and support-files/CMakeLists.txt assumes that. But when one > specifies INSTALL_SYSCONFDIR manually on the command line, this > assumption might be broken. Looks ok to me. An alternative solution would be to define INSTALL_SYSCONF2DIR if INSTALL_SYSCONFDIR is defined. Regards, Sergei

2 1

[Maria-developers] Open Questions in the Knowledge Base
by Ian Gilfillan 19 Nov '13

19 Nov '13

There are quite a few unanswered questions in the Knowledge Base https://mariadb.com/kb/en/+questions/ If anyone knowledgeable in those areas can help answer some of these, even just a request for more info, that would be great :) thanks, ian

2 1

Re: [Maria-developers] Accessing rpl_global_gtid_binlog_state is not consistently protected
by Kristian Nielsen 19 Nov '13

19 Nov '13

Pavel Ivanov <pivanof(a)google.com> writes: > sql/log.cc has highly inconsistent pattern of accessing > rpl_global_gtid_binlog_state. In some places it's protected by mutex > rpl_global_gtid_binlog_state.LOCK_binlog_state, in some places it's > protected by global mutex LOCK_rpl_gtid_state, and in some places it's > not protected by any mutex at all. This is a bug that should be fixed, > right? Yes, agree, thanks for finding this. > I can file this in JIRA if you want, just wanted to let you know faster. I've filed it as MDEV-5306. > I've noticed this problem after one of our AddressSanitizer testing > bots reported that rpl_binlog_state::bump_seq_no_if_needed (called > from MYSQL_BIN_LOG::bump_seq_no_counter_if_needed) was using memory > from hash after it was already freed by rpl_binlog_state::reset > (called by MYSQL_BIN_LOG::reset_logs). Can you test again with the latest patch I've just pushed to 10.0-base? I looked briefly into making an mtr test case to catch this, but I did not see any easy way to do that. I went through the code to try and check all uses of rpl_binlog_state for proper locking, but it would be nice to get a confirmation that I did not miss anything. - Kristian.

3 2

Re: [Maria-developers] MariaDB Galera replication
by Pavel Ivanov 18 Nov '13

18 Nov '13

I'm starting a new thread as this is already doesn't have anything to do with the original topic. On Fri, Nov 15, 2013 at 10:46 AM, Alex Yurchenko <alexey.yurchenko(a)codership.com> wrote: >>> Please pardon this arrogant interruption of your discussion and shameless >>> self-promotion, but I just could not help noticing that Galera >>> replication >>> was designed specifically with these goals in mind. And it does seem to >>> achieve them better than semi-sync plugin. Have you considered Galera? >>> What >>> makes you prefer semi-sync over Galera, if I may ask? >> >> >> To be honest I never looked at how Galera works before. I've looked at >> it now and I don't see how it can fit with us. The major disadvantages >> I immediately see: >> 1. Synchronous replication. That means client must wait while >> transaction is applied on all nodes which is unacceptably big latency >> of each transaction. And what if there's a network blip and some node >> becomes inaccessible? All writes will just freeze? I see the statement >> that "failed nodes automatically excluded from the cluster", but to do >> that cluster must wait for some timeout in case it's indeed a network >> blip and node will "quickly" reconnect. And every client must wait for >> cluster to decide what happened with that one node. >> 2. Let's say node fell out of the cluster for 5 minutes and then >> reconnected. I guess it will be treated as "new node", it will >> generate state transfer and the node will start downloading the whole >> database? And while it's trying to download say 500GB of data files >> all other nodes (or maybe just donor?) won't be able to change those >> files locally and thus will blow up its memory consumption. That means >> they could quickly run out of memory and "new node" won't be able to >> finish its "initialization"... >> 3. It looks like there's strong asymmetry in starting cluster nodes -- >> the first one should be started with empty wsrep_cluster_address and >> all others should be started with the address of the first node. So I >> can't start all nodes uniformly and then issue some commands to >> connect them to each other. That's bad. >> 4. What's the transition path? How do I upgrade MySQL/MariaDB >> replicating using usual replication to Galera? It looks like there's >> no such path and the solution is stop the world using regular >> replication and restart it using Galera. Sorry I can't do that with >> our production systems. >> >> I believe these problems are severe enough for us, so that we can't >> work with Galera. > > > Pavel, you seem to be terribly mistaken on almost all accounts: > > 1. *Replication* (i.e. data buffer copying) is indeed synchronous. But > nobody said that commit is. What Galera does is very similar to semi-sync, > except that it does it technically better. I would not dare to suggest > Galera replication if I didn't believe it to be superior to semi-sync in > every respect. Well, apparently we have a different understanding of what the term "synchronous replication" means. This term is all over the Galera doc, but I didn't find the detailed description of how actually Galera replication work. So I assumed that my understanding of the term (which actually seem to be in line with wiki's definitions http://en.wikipedia.org/wiki/Replication_(computing) ) is what was implied there. So I hope you'll be able to describe in detail how Galera replication works. > As an example here's an independent comparison of Galera vs. > semi-sync performance: > http://linsenraum.de/erkules/2011/06/momentum-galera.html. This is a nice blog post written in German and posted in 2011. And while Google Translate gave me an idea what post was about it would be nice to see something more recent and with better description of what was the actual testing set up. > In fact, majority > of Galera users migrated from the regular *asynchronous* MySQL replication, > which I think is a testimony to Galera performance. I don't mean to troll, but this can also mean that everyone who migrated didn't care much about performance and Galera's performance was within sane boundaries... BTW, just found here https://mariadb.com/kb/en/mariadb-galera-cluster-known-limitations/ : "by design performance of the cluster cannot be higher than performance of the slowest node; however, even if you have only one node, its performance can be considerably lower comparing to running the same server in a standalone mode". That contradicts your words. > 2. Node reconnecting to cluster will normally receive only events that it > missed while being disconnected. This seem to contradict to the docs. Again from https://mariadb.com/kb/en/mariadb-galera-cluster-known-limitations/ : "After a temporary split, if the 'good' part of the cluster was still reachable and its state was modified, resynchronization occurs". > 3. You are partially right about it, but isn't it much different from > regular MySQL replication where you first need to set up master and then > connect slaves (even if you have physically launched the servers at the same > time). Operation of setting up master and then connecting slaves consists of mostly only executing CHANGE MASTER TO and then START SLAVE on all slaves after all MySQL instances (including master) were started with the same set of command line flags. This is fundamentally different from starting instances with different arguments, especially when these arguments should be different depending on whether the replica is starting first or there's already some other replica running. > Yet, Galera nodes can be started simultaneously and then joined > together by setting wsrep_cluster_address from mysql client connection. This > is not advertised method, because in that case state snapshot transfer can > be done only by mysqldump. If you set the address in advance, rsync or > xtrabackup can be used to provision the fresh node. This is of course better because I can start all instances with the same command line arguments. But transferring snapshot of a very big database using mysqldump, and causing the node that creates mysqldump to blow up memory consumption during the process, that is still a big problem. > 4. Every Galera node can perfectly work as either master or slave to native > MySQL replication. So migration path is quite clear. Nope, not clear yet. So I'll be able to upgrade all my MySQL instances to a Galera-supporting binary while they are replicating using standard MySQL replication. That's good. Now, how the Galera replication is turned on after that? What will happen if I just set wsrep_cluster_address address on all replicas? What will replicas do, and what will happen with the standard MySQL replication? > It is very sad that you happen to have such gross misconceptions about > Galera. If those were true, how would MariaDB Galera Cluster get paying > customers? Care to share some numbers? Like what's the rough amount of those paying customers? What size is the biggest installation -- number of clusters, replicas, highest QPS load? I'm not asking to share any confidential information, but the rough ballpark of the numbers would be helpful. > May be my reply will convince you to have a second look at it. > (In addition to the above Galera is fully multi-master, does parallel > applying and works great in WAN) I hope your explanation of how Galera replication work will help me understand how great it works over WAN and how you could make full multi-master work without fully synchronous replication in my understanding of that term. Pavel

3 9

[Maria-developers] What is bundled libevent library for?
by Pavel Ivanov 18 Nov '13

18 Nov '13

Sergei, Do you know why MariaDB sources have libevent library bundled with them? I don't see it being used anywhere. I see it was added in this commit by Monty http://bazaar.launchpad.net/~maria-captains/maria/10.0/revision/3492.3.1, but even in that commit I can't find any usages of the library. So should it be removed from MariaDB? Thank you, Pavel

4 4

[Maria-developers] Why local modifications to PCRE library?
by Pavel Ivanov 16 Nov '13

16 Nov '13

Alexander, I see you've included the full source of PCRE library into MariaDB tree and even made some local modifications which are not included into PCRE upstream. Could you please explain why did you do that? Why not allow to build with system standard PCRE library? I understand that you could have needed some modifications and thus could want to include sources from PCRE trunk to allow users to build with modifications when their system-wide library is too old. But as I don't see your modifications on PCRE trunk that means you probably had something different in mind, and decided to incur the pain of constantly updating the included sources and merging security fixes. So why did you decide to do that? Thank you, Pavel

3 7

Re: [Maria-developers] Semisync plugin incompatibility
by Kristian Nielsen 15 Nov '13

15 Nov '13

Pavel Ivanov <pivanof(a)google.com> writes: > We've noticed recently that semisync_master plugin in MariaDB (which > apparently was fully inherited from MySQL) is seriously incompatible > with our understanding of the purpose of semi-sync replication. This > incompatibility was apparently introduced as a fix for > http://bugs.mysql.com/bug.php?id=45672. The "major no-no" that bug So as I understand it, this bug is about what should happen when semisync is enabled, but no slaves are connected. Apparently before the fix of Bug#45672, an error was thrown late during COMMIT. So the transaction was committed (locally on the master), but the client still got an error back. And if I understand correctly, after the fix of Bug#45672, no error is thrown in the case where no slave is connected. > talks about is in our opinion the whole purpose of semi-sync > replication -- if transaction is not replicated to at least one slave > client shouldn't get OK even if transaction is committed locally on > the master. Also master shouldn't just turn off semi-sync replication > whenever it wants. So with "just turn off semi-sync replication whenever it wants" - what are you refering to here? I seem to remember that semisync has a timeout, and it gets disabled if that timeout triggers? My guess is that this is what you have in mind, but I wanted to ask to make sure ... > We will fix this problem for us, but first I wanted to understand > what's your view of the purpose of semi-sync replication and how you > think it should work? I need to know your opinion to understand how I > should fix this issue... Well, personally, I never was much interested in semi-sync. But it is my understanding that there is some interest, so I will answer with what small opinion I have. I suppose the general idea is that when client sees its COMMIT complete, it can know that its transaction exists in at least two places (master binlog + at least one slave relay log). So there is no longer any single point of failure that can cause loss of the transaction. Another point of view I is that semi-sync provides some sort of throttle on how fast the master can generate events compared to how fast the slaves can receive them: http://www.mysqlperformanceblog.com/2012/01/19/how-does-semisynchronous-mys… There was also a suggestion (and a patch is floating around somewhere) for "enhanced semisync replication": https://mariadb.atlassian.net/browse/MDEV-162 This delays not only client acknowledge but also InnoDB commit until the ack from at least one slave, which means that transactions are not visible to other clients until they exist on at least one slave in addition to on the master. Since this is _semi_-sync, not real two-phase commit synchronous replication, the main problem is that there is way to ensure consistency in the general error case. The transaction is already fully committed on the master, it cannot be rolled back. So we are left with the choice of one of two evils: 1. Report an error to the client. Most clients would then probably wrongly assume that the transaction was _not_ committed. There also does not seem to be much the client can do about the error except perhaps log an incident to the monitoring system. On the other hand, then at least the problem is not silently ignored. 2. Report success to the client but complain loudly in the error log (I assume this is what happens in current code). This leaves the client unaware that there is a problem (but presumably the monitoring system will catch the message in the error log). >From this summary, I think I can see the logic of the current behaviour: - It preserves protection against single-point-of-failure. If all slaves are gone, then we already have one failure, and unless we experience a double failure (master also failing before slave recovers), the transaction will eventually be sent to a slave and no overall failure happens. - If the client can anyway not do anything about the problem except notify the monitoring system, the server may as well do the notification itself. But the opposite point of view also has merit. The client asked for semi-sync behavior, but did not get it, and it does not even have a way to know about the problem. That is not good. Does the client currently at least get a warning for the COMMIT? I think it should (eg. the fix for Bug#45672 should at least have been to turn the error into a warning, not remove the error completely). What I think could make sense is if the client got an error during the prepare phase if no slaves are connected. In this case we _can_ roll back the transaction and give an error to the client without any issue of consistency. But it still leaves a small window where the last slave can disappear between the prepare and the commit phase and leave us with the original problem. I hope this helps you ... Maybe you can describe your use-case, and how you need things to work for that case? Personally I have nothing against changing this behaviour to something more logical, I am just not sure what the most logical behaviour is ... - Kristian.

4 8

[Maria-developers] parallel replication monitoring
by Giuseppe Maxia 15 Nov '13

15 Nov '13

What can I use to see the status of parallel replication? I am looking for: * how many workers are used; * in which schema they are working; * what query they are running. So far, I only see mysql.gtid_slave_pos and information_schema.processlist. Anything else available? Thanks in advance Giuseppe

4 6

[Maria-developers] Slave can take a very long time to start replication
by Pavel Ivanov 15 Nov '13

15 Nov '13

Kristian, As I understand currently when slave connects to master and wants to start replicating it passes GTID to start from, master finds binlog file where the earliest GTID is located and then scans through that file to find the exact binlog position to start sending binlog events from. If this binlog file is pretty big then scanning can take a very long time. I guess especially long when several slaves try to start replicating roughly at the same time. We observed 60-90 seconds between initial connection by the slave and the first real binlog events starting to flow. In this period of time slave doesn't receive anything from master and thus it's very easy to confuse such situation with connection loss, hit slave_net_timeout, reopen connection to the master again and force it to start searching through binlog file from the very beginning... Putting aside the argument of what value is good enough for slave_net_timeout, I'd say in any case slave taking 60 seconds to just start receiving binlog events from master is unacceptable. Did you think about this problem before? Maybe you've even planned already to implement some solution for this? Thank you, Pavel

2 1

[Maria-developers] Review for maria/10.0-serg/revision/3899 (MDEV-5248 Serious incompatibility and data corruption of DATETIME and DATE types due to get_innobase_type_from_mysql_type refactor combined with InnoDB Online DDL)
by Jeremy Cole 14 Nov '13

14 Nov '13

Serg, Thanks for looking into this. General comments: - The approach overall looks sound. I am very much in favor of reverting back to Oracle MySQL's type codes as you have, by reverting the refactor of get_innobase_type_from_mysql_type as you did. - You've mixed tabs and spaces in a bunch of places. InnoDB code should use tabs to indent. - It would probably be worth visually comparing to MySQL 5.6's get_innobase_type_from_mysql_type to ensure there is nothing new there aside from my comments. - It is almost certainly worth creating test cases with data files from older MariaDB and newer MySQL versions to exercise the usage of the code paths that only occur during upgrade; that doesn't need to happen here necessarily. Comments on the diff inline below. Regards, Jeremy (I removed the .result file differences as any issues there will be resolved by fixes to the .test.) === renamed file 'mysql-test/suite/innodb/t/1byte_data_int.opt' => > 'mysql-test/suite/innodb/t/data_types.opt' > === renamed file 'mysql-test/suite/innodb/t/1byte_data_int.test' => > 'mysql-test/suite/innodb/t/data_types.test' > --- mysql-test/suite/innodb/t/1byte_data_int.test 2013-10-31 22:20:05 > +0000 > +++ mysql-test/suite/innodb/t/data_types.test 2013-11-11 20:54:09 +0000 > @@ -1,25 +1,76 @@ > --source include/have_innodb.inc > > I would strongly suggest a detailed explanatory comment and warning added with "--echo #" so that it appears in the result file as well, explaining the repercussions changing any of the results here. > ---echo # Create a table with a 1-byte ENUM, 1-byte SET, and TINYINT > UNSIGNED. > - > CREATE TABLE t1 > ( > - t1_enum ENUM("a", "b", "c"), > - t1_set SET("a", "b", "c"), > - t1_tinyint_s TINYINT, > - t1_tinyint_u TINYINT UNSIGNED > + t1_BIGINT BIGINT, > + t1_BIGINT_UNSIGNED BIGINT UNSIGNED, > + t1_BINARY_100 BINARY(100), > + t1_BIT_2 BIT(2), > + t1_BIT_20 BIT(20), > + t1_BLOB BLOB, > + t1_CHAR_100 CHAR(100), > + t1_CHAR_100_BINARY CHAR(100) BINARY, > + t1_DATE DATE, > + t1_DATETIME DATETIME, > + t1_DATETIME_6 DATETIME(6), > + t1_DECIMAL_10_3 DECIMAL(10,3), > + t1_DECIMAL_10_3_UNSIGNED DECIMAL(10,3) UNSIGNED, > + t1_DOUBLE DOUBLE, > + t1_DOUBLE_UNSIGNED DOUBLE UNSIGNED, > + t1_ENUM ENUM("a", "b", "c"), > + t1_ENUM_BINARY ENUM('a','b') BINARY, > Why the mix of quote styles? For completeness this should also use an ENUM with > 254 elements to create a 2-byte ENUM. > + t1_FLOAT FLOAT, > + t1_FLOAT_UNSIGNED FLOAT UNSIGNED, > + t1_INT INT, > + t1_INT_UNSIGNED INT UNSIGNED, > + t1_LONGBLOB LONGBLOB, > + t1_LONGTEXT LONGTEXT, > + t1_MEDIUMBLOB MEDIUMBLOB, > + t1_MEDIUMINT MEDIUMINT, > + t1_MEDIUMINT_UNSIGNED MEDIUMINT UNSIGNED, > + t1_MEDIUMTEXT MEDIUMTEXT, > + t1_SET SET("a", "b", "c"), > + t1_SET_BINARY SET('a','b') BINARY, > Same mix of quote styles. Also should create a SET with > 8 elements to create a 2-byte SET. > + t1_SMALLINT SMALLINT, > + t1_SMALLINT_UNSIGNED SMALLINT UNSIGNED, > + t1_TEXT TEXT, > + t1_TIME TIME, > + t1_TIMESTAMP TIMESTAMP, > + t1_TIMESTAMP_5 TIMESTAMP(5), > + t1_TIME_4 TIME(4), > Maybe move up to TIME above (I guess these are all alphabetically sorted, but it may be better to sort only by the type [or add e.g. _0 to the base one]). Why 4 and not 3 or 6 or some other value? > + t1_TINYBLOB TINYBLOB, > + t1_TINYINT TINYINT, > + t1_TINYINT_UNSIGNED TINYINT UNSIGNED, > + t1_TINYTEXT TINYTEXT, > + t1_VARBINARY_100 VARBINARY(100), > + t1_VARCHAR_10 VARCHAR(10), > + t1_VARCHAR_10_BINARY VARCHAR(10) BINARY, > + t1_VARCHAR_500 VARCHAR(500), > + t1_VARCHAR_500_BINARY VARCHAR(500) BINARY, > + t1_YEAR_2 YEAR(2), > + t1_YEAR_4 YEAR(4) > ) ENGINE=InnoDB; > > ---echo # All t1 fields' mtypes should be 6 (DATA_INT). > - > SELECT > - name, > - mtype, > - (prtype & 512) = 512 AS is_unsigned > + name, CASE mtype WHEN 1 THEN "DATA_VARCHAR" > Put CASE on its own line (distinct from "name") and move the first WHEN to its own line. Re-indent other WHEN's to match. > + WHEN 2 THEN "DATA_CHAR" > + WHEN 3 THEN "DATA_FIXBINARY" > + WHEN 4 THEN "DATA_BINARY" > + WHEN 5 THEN "DATA_BLOB" > + WHEN 6 THEN "DATA_INT" > + WHEN 7 THEN "DATA_SYS_CHILD" > + WHEN 8 THEN "DATA_SYS" > + WHEN 9 THEN "DATA_FLOAT" > + WHEN 10 THEN "DATA_DOUBLE" > + WHEN 11 THEN "DATA_DECIMAL" > + WHEN 12 THEN "DATA_VARMYSQL" > + WHEN 13 THEN "DATA_MYSQL" > + WHEN 63 THEN "DATA_MTYPE_MAX" > + ELSE mtype > + END, > Add an alias e.g. AS base_type so that the result's column name isn't a truncated version of this CASE. > + IF((prtype & 512) = 512,"UNSIGNED","") AS is_unsigned > It's a bit weird to have is_unsigned be "UNSIGNED" or empty. Maybe YES/NO instead? Not a big deal I suppose. > FROM information_schema.INNODB_SYS_COLUMNS > WHERE name LIKE "t1\_%" > ORDER BY name; > > ---echo # Cleanup > - > DROP TABLE t1; > === modified file 'storage/innobase/dict/dict0stats.cc' > --- storage/innobase/dict/dict0stats.cc 2013-06-22 15:47:12 +0000 > +++ storage/innobase/dict/dict0stats.cc 2013-11-11 20:54:09 +0000 > @@ -176,7 +176,7 @@ > DATA_NOT_NULL, 192}, > > {"last_update", DATA_INT, > - DATA_NOT_NULL | DATA_UNSIGNED, 4}, > + DATA_NOT_NULL, 4}, > > {"n_rows", DATA_INT, > DATA_NOT_NULL | DATA_UNSIGNED, 8}, > @@ -207,7 +207,7 @@ > DATA_NOT_NULL, 192}, > > {"last_update", DATA_INT, > - DATA_NOT_NULL | DATA_UNSIGNED, 4}, > + DATA_NOT_NULL, 4}, > > {"stat_name", DATA_VARMYSQL, > DATA_NOT_NULL, 64*3}, > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- storage/innobase/handler/ha_innodb.cc 2013-11-03 23:45:27 +0000 > +++ storage/innobase/handler/ha_innodb.cc 2013-11-11 20:54:09 +0000 > @@ -5745,67 +5745,94 @@ > 8 bits: this is used in ibuf and also when DATA_NOT_NULL is ORed to > the type */ > > - compile_time_assert((ulint)MYSQL_TYPE_STRING < 256); > - compile_time_assert((ulint)MYSQL_TYPE_VAR_STRING < 256); > - compile_time_assert((ulint)MYSQL_TYPE_DOUBLE < 256); > - compile_time_assert((ulint)MYSQL_TYPE_FLOAT < 256); > - compile_time_assert((ulint)MYSQL_TYPE_DECIMAL < 256); > - > - *unsigned_flag = 0; > - > - switch (field->key_type()) { > - case HA_KEYTYPE_USHORT_INT: > - case HA_KEYTYPE_ULONG_INT: > - case HA_KEYTYPE_UINT24: > - case HA_KEYTYPE_ULONGLONG: > + DBUG_ASSERT((ulint)MYSQL_TYPE_STRING < 256); > + DBUG_ASSERT((ulint)MYSQL_TYPE_VAR_STRING < 256); > + DBUG_ASSERT((ulint)MYSQL_TYPE_DOUBLE < 256); > + DBUG_ASSERT((ulint)MYSQL_TYPE_FLOAT < 256); > + DBUG_ASSERT((ulint)MYSQL_TYPE_DECIMAL < 256); > + > + if (field->flags & UNSIGNED_FLAG) { > + > *unsigned_flag = DATA_UNSIGNED; > - /* fall through */ > - case HA_KEYTYPE_SHORT_INT: > - case HA_KEYTYPE_LONG_INT: > - case HA_KEYTYPE_INT24: > - case HA_KEYTYPE_INT8: > - case HA_KEYTYPE_LONGLONG: > + } else { > + *unsigned_flag = 0; > + } > + > + if (field->real_type() == MYSQL_TYPE_ENUM > + || field->real_type() == MYSQL_TYPE_SET) { > + > + /* MySQL has field->type() a string type for these, but the > + data is actually internally stored as an unsigned integer > + code! */ > + > + *unsigned_flag = DATA_UNSIGNED; /* MySQL has its own unsigned > + flag set to zero, even though > + internally this is an unsigned > + integer type */ > return(DATA_INT); > - case HA_KEYTYPE_FLOAT: > - return(DATA_FLOAT); > - case HA_KEYTYPE_DOUBLE: > - return(DATA_DOUBLE); > - case HA_KEYTYPE_BINARY: > - if (field->type() == MYSQL_TYPE_TINY || > - field->real_type() == MYSQL_TYPE_ENUM || > - field->real_type() == MYSQL_TYPE_SET > - ) > - { // compatibility workaround > - *unsigned_flag= DATA_UNSIGNED; > - return DATA_INT; > - } > - return(DATA_FIXBINARY); > - case HA_KEYTYPE_VARBINARY2: > - if (field->type() != MYSQL_TYPE_VARCHAR) > - return(DATA_BLOB); > - /* fall through */ > - case HA_KEYTYPE_VARBINARY1: > - return(DATA_BINARY); > - case HA_KEYTYPE_VARTEXT2: > - if (field->type() != MYSQL_TYPE_VARCHAR) > - return(DATA_BLOB); > - /* fall through */ > - case HA_KEYTYPE_VARTEXT1: > - if (field->charset() == &my_charset_latin1) { > + } > + > + switch (field->type()) { > + /* NOTE that we only allow string types in DATA_MYSQL and > + DATA_VARMYSQL */ > + case MYSQL_TYPE_VAR_STRING: /* old <= 4.1 VARCHAR */ > + case MYSQL_TYPE_VARCHAR: /* new >= 5.0.3 true VARCHAR */ > + if (field->binary()) { > + return(DATA_BINARY); > + } else if (strcmp(field->charset()->name, > + "latin1_swedish_ci") == 0) { > return(DATA_VARCHAR); > } else { > return(DATA_VARMYSQL); > } > - case HA_KEYTYPE_TEXT: > - if (field->charset() == &my_charset_latin1) { > + case MYSQL_TYPE_BIT: > + case MYSQL_TYPE_STRING: if (field->binary()) { > + > + return(DATA_FIXBINARY); > + } else if (strcmp(field->charset()->name, > + "latin1_swedish_ci") == 0) { > return(DATA_CHAR); > } else { > return(DATA_MYSQL); > } > - case HA_KEYTYPE_NUM: > + case MYSQL_TYPE_NEWDECIMAL: > + return(DATA_FIXBINARY); > + case MYSQL_TYPE_LONG: > + case MYSQL_TYPE_LONGLONG: > + case MYSQL_TYPE_TINY: > + case MYSQL_TYPE_SHORT: > + case MYSQL_TYPE_INT24: > + case MYSQL_TYPE_DATE: > + case MYSQL_TYPE_YEAR: > + case MYSQL_TYPE_NEWDATE: > + return(DATA_INT); > + case MYSQL_TYPE_TIMESTAMP: > + *unsigned_flag = 0; > + /* fall through */ > + case MYSQL_TYPE_TIME: > + case MYSQL_TYPE_DATETIME: > + if (field->key_type() == HA_KEYTYPE_BINARY) > + return(DATA_FIXBINARY); > + else > + return(DATA_INT); > You also need to support MYSQL_TYPE_{TIMESTAMP,TIME,DATETIME}2 for MySQL 5.6 here. > + case MYSQL_TYPE_FLOAT: > + return(DATA_FLOAT); > + case MYSQL_TYPE_DOUBLE: > + return(DATA_DOUBLE); > + case MYSQL_TYPE_DECIMAL: > return(DATA_DECIMAL); > - case HA_KEYTYPE_BIT: > - case HA_KEYTYPE_END: > + case MYSQL_TYPE_GEOMETRY: > + case MYSQL_TYPE_TINY_BLOB: > + case MYSQL_TYPE_MEDIUM_BLOB: > + case MYSQL_TYPE_BLOB: > + case MYSQL_TYPE_LONG_BLOB: > + return(DATA_BLOB); > + case MYSQL_TYPE_NULL: > + /* MySQL currently accepts "NULL" datatype, but will > + reject such datatype in the next release. We will cope > + with it and not trigger assertion failure in 5.1 */ > + break; > + default: > ut_error; > } > > === modified file 'storage/innobase/handler/handler0alter.cc' > --- storage/innobase/handler/handler0alter.cc 2013-09-21 08:14:42 +0000 > +++ storage/innobase/handler/handler0alter.cc 2013-11-11 20:54:09 +0000 > @@ -347,6 +347,32 @@ > } > } > > + /* > + InnoDB in different MariaDB versions was generating different > mtype > + codes for certain types. Also signed/unsigned bit was generated > + differently too. > It may be more clear to merge these two sentences into one. > + > + Online ALTER would change the mtype/unsigned_flag (to what the > + current code generates) without changing the underlying data > + represenation, and it might result in data corruption. > + > + Don't do online ALTER if mtype/unsigned_flag are wrong. > + */ > + for (ulint i = 0; i < table->s->fields; i++) { > + const Field* field = table->field[i]; > + const dict_col_t* col = dict_table_get_nth_col(prebuilt->table, i); > + ulint unsigned_flag; > + if (col->mtype != get_innobase_type_from_mysql_type(&unsigned_flag, > field)) { > + > + DBUG_RETURN(HA_ALTER_INPLACE_NOT_SUPPORTED); > + } > + > + if ((col->prtype & DATA_UNSIGNED) != unsigned_flag) { > + > + DBUG_RETURN(HA_ALTER_INPLACE_NOT_SUPPORTED); > + } > + } > + > This looks reasonable to me. > /* We should be able to do the operation in-place. > See if we can do it online (LOCK=NONE). */ > bool online = true;

2 4

Re: [Maria-developers] maria-tokudb-engine-10.0 deb package (raring) - if files are empty where are sources?
by Sergei Golubchik 14 Nov '13

14 Nov '13

Hi, Jocelyn! On Nov 14, Jocelyn Fournier wrote: > Hi, > > Perhaps it would be great to mention in the 10.0.5 download page it's > not yet compatible with TokuDB. > I've tried myself to upgrade my server using MariaDB 5.5.33a with tokudb > to 10.0.5 and it failed because of this issue. I believe, only deb packages are affected. TokuDB is present in rpms, in the source and in the binary tarballs. But I see you point, we'll probably add some warning, thanks! Regards, Sergei

2 1

Re: [Maria-developers] MDEV-5220 - [PATCH] MariaDB 10.0.4 doesn't compile without perfschema
by Sergei Golubchik 14 Nov '13

14 Nov '13

Hi, Sergey! Generally, it's ok, of course. But as a matter of style (up to you to fix it here, but keep in mind for the future) - see below On Nov 13, Sergey Vojtovich wrote: > === modified file 'sql/mysqld.cc' > --- a/sql/mysqld.cc 2013-11-01 11:00:11 +0000 > +++ b/sql/mysqld.cc 2013-11-13 07:07:55 +0000 > @@ -760,6 +760,7 @@ char **orig_argv; > > static struct my_option pfs_early_options[] __attribute__((unused)) = I suppose, we don't need __attribute__((unused)) here, if the array is used unconditionally. > { > +#ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > {"performance_schema_instrument", OPT_PFS_INSTRUMENT, > "Default startup value for a performance schema instrument.", > &pfs_param.m_pfs_instrument, &pfs_param.m_pfs_instrument, 0, GET_STR, > @@ -824,6 +825,7 @@ static struct my_option pfs_early_option > &pfs_param.m_consumer_statement_digest_enabled, > &pfs_param.m_consumer_statement_digest_enabled, 0, > GET_BOOL, OPT_ARG, TRUE, 0, 0, 0, 0, 0} > +#endif /* WITH_PERFSCHEMA_STORAGE_ENGINE */ > }; > > #ifdef HAVE_PSI_INTERFACE > @@ -1146,7 +1148,6 @@ void net_after_header_psi(struct st_net > > void init_net_server_extension(THD *thd) > { > -#ifdef HAVE_PSI_INTERFACE > /* Start with a clean state for connection events. */ > thd->m_idle_psi= NULL; > thd->m_statement_psi= NULL; > @@ -1157,9 +1158,6 @@ void init_net_server_extension(THD *thd) > thd->m_net_server_extension.m_after_header= net_after_header_psi; > /* Activate this private extension for the mysqld server. */ > thd->net.extension= & thd->m_net_server_extension; > -#else > - thd->net.extension= NULL; > -#endif > } > #endif /* EMBEDDED_LIBRARY */ > > @@ -4397,7 +4395,9 @@ static int init_thread_environment() > #ifdef HAVE_EVENT_SCHEDULER > Events::init_mutexes(); > #endif > +#ifdef HAVE_PSI_INTERFACE > init_show_explain_psi_keys(); > +#endif I usually define an no-op function and use it unconditionally, to avoid scattering #ifdefs in the code. > /* Parameter for threads created for connections */ > (void) pthread_attr_init(&connection_attrib); > (void) pthread_attr_setdetachstate(&connection_attrib, > @@ -5137,9 +5137,12 @@ int mysqld_main(int argc, char **argv) > buffered_logs.init(); > my_getopt_error_reporter= buffered_option_error_reporter; > my_charset_error_reporter= buffered_option_error_reporter; > +#ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > pfs_param.m_pfs_instrument= const_cast<char*>(""); > > - int ho_error= handle_early_options(); > + int ho_error= > +#endif /* WITH_PERFSCHEMA_STORAGE_ENGINE */ > + handle_early_options(); Please, don't #ifdef part of a statement. Here I'd use __attribute__((unused)) to suppress a compiler warning. > #ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > if (ho_error == 0) > Regards, Sergei

2 1

[Maria-developers] Thoughts on a variable to control high resolution temporal type format and rounding behavior?
by Jeremy Cole 13 Nov '13

13 Nov '13

Sergei, In the process of working through the bugs I've reported re. time formats and InnoDB storage formats we noted that there isn't currently any way to cause MariaDB to create a *new* Oracle MySQL 5.6-format TIMESTAMP/TIME/DATETIME field, and thus there's not really very good test coverage of them. Additionally the existence of this difference relative to Oracle MySQL is not really desirable. I would propose adding a new sysvar to control the temporal time format e.g. temporal_microsecond_format={mysql|mariadb} which controls whether newly created columns are created as Field_*_hires or Field_*f. This would especially allow: 1. Tests to be run in both Oracle MySQL and MariaDB modes. 2. A single test to create a column in one format and request conversion to the other. 3. A user to convert the format of their existing tables for compatibility. Additionally I would suggest a new sysvar to control the rounding behavior of the same temporal types, e.g. temporal_microsecond_rounding={round|truncate} which controls whether to round (as Oracle MySQL 5.6 does) or to truncate (as MariaDB does) when faced with fitting a larger number of decimals in a value into a smaller number decimals in a field. (This of course assumes that the current MariaDB behavior and divergence from Oracle MySQL behavior is intentional, and not a bug. Even if it is considered a bug it could probably not be changed outright at this point and the sysvar would still be needed.) Any thoughts on either of these? We would of course be willing to do the work and provide the patches for them. I just wanted to get your general feelings on the ideas. Regards, Jeremy

3 6

[Maria-developers] EXPLAIN in the slow query log
by Sergei Petrunia 12 Nov '13

12 Nov '13

Hello, After recent changes in the [SHOW] EXPLAIN architecture in MariaDB, it became really easy to add an option to print EXPLAIN(*) output into the slow query log, so I went ahead and implemented it. @@log_slow_verbosity has got a 'explain' flag, and if it is on, the slow query log entries look like this: # Time: 131017 21:26:02 # User@Host: root[root] @ localhost [] # Thread_id: 2 Schema: dbt3sf1 QC_hit: No # Query_time: 5.535819 Lock_time: 0.000350 Rows_sent: 1 Rows_examined: 65633 ## <explain> ## id select_type table type possible_keys key key_len ref rows Extra ## 1 SIMPLE nation ref PRIMARY,n_name n_name 26 const 1 Using where; Using index ## 1 SIMPLE customer ref PRIMARY,i_c_nationkey i_c_nationkey 5 dbt3sf1.nation.n_nationkey 25124 Using index ## 1 SIMPLE orders ref i_o_custkey i_o_custkey 5 dbt3sf1.customer.c_custkey 7 Using index ## </explain> SET timestamp=1382030762; select count(*) from customer, orders, nation where c_custkey=o_custkey and c_nationkey=n_nationkey and n_name='GERMANY'; It seems, pt-slow-query-log has no problem parsing this. Open questions: - Is this syntax OK? - Currently, some statements (like SHOW KEYS) print a dumb explain that shows that full table scan is used. There is a suggestion to not print EXPLAIN for any SHOW statements. Is this ok? (*) something like EXPLAIN ANALYZE, with numbers from actual execution, would be better but we are not there yet. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

5 6

[Maria-developers] A newer InnoDB in 10.0 series?
by Colin Charles 12 Nov '13

12 Nov '13

Hi! I see that we are shipping InnoDB from 5.6.5 (at least that's what the plugin version tells me - 1.2.5). Are we going to pop XtraDB or a modern InnoDB into 10.0.6? Thanks -- Colin Charles, Chief Evangelist MariaDB | t: +6-012-204-3201 | Skype: colincharles

3 3

Re: [Maria-developers] [Commits] Rev 3886: MDEV-4439 ALTER TABLE .. [ADD|DROP] FOREIGN KEY IF [NOT] EXISTS does not work if constraint name is not used. in file:///home/hf/wmar/10exp/
by Jan Lindström 11 Nov '13

11 Nov '13

Hi, MySQL changes look good, InnoDB changes need some work that I will do, please do not push those. R: Jan > At file:///home/hf/wmar/10exp/ > > ------------------------------------------------------------ > revno: 3886 > revision-id: holyfoot(a)askmonty.org-20131108110551-j0rxpxrnfertay2q > parent: knielsen(a)knielsen-hq.org-20131107065240-gxxg4ilcs8o2v0q2 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: 10exp > timestamp: Fri 2013-11-08 15:05:51 +0400 > message: > MDEV-4439 ALTER TABLE .. [ADD|DROP] FOREIGN KEY IF [NOT] EXISTS does not work if constraint name is not used. > Patches for server and the Innodb engine. > Server is fixed so it does nothing if no indexes left to alter. > Innodb parser is fixed so it looks for the IF [NOT] EXISTS option, > another change is that it uses the index name for the internal dictionary. > Prior to that it only used the CONSTRAINT name for it. > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- -- Jan Lindström Principal Engineer MariaDB | MaxScale | skype: jan_p_lindstrom www.skysql.com <http://www.skysql.com/> Twitter <http://twitter.com/skysql> Blog <http://www.skysql.com/blog/> Facebook <http://www.facebook.com/skysql> LinkedIn <http://www.linkedin.com/company/1214250> Google+ <https://plus.google.com/117544963211695643458/posts>

1 0

Re: [Maria-developers] patch to upgrade collations
by Michael Widenius 08 Nov '13

08 Nov '13

Hi! >>>>> "Alexander" == Alexander Barkov <bar(a)mariadb.org> writes: Alexander> Hi Monty, Alexander> Please find the full patch attached. Alexander> Greetings. Here is a modified patch that also fixes that REPAIR works. It doesn't include the myisam files that you had in your patch, but it will of course need these. I have verified that all test cases you changed works. Regards, Monty

1 0

[Maria-developers] MDEV-4954
by Sergei Golubchik 08 Nov '13

08 Nov '13

Hi, Oden could you please take a look at MDEV-4954? https://mariadb.atlassian.net/browse/MDEV-4954 "RPM Packaging Removes Config File" The complain is that when one uninstalls MariaDB, /etc/my.cnf gets "removed" (actually, renamed to /etc/my.cnf.rpmsave). And MySQL packages don't do that (because they don't include my.cnf at all). Anything we can do to make the user happier? Regards, Sergei

1 0

[Maria-developers] Debian/Ubuntu packages
by Matt Coleman 07 Nov '13

07 Nov '13

Hello, I’m working on fixing the dependency issues with MariaDB 5.5.33a vs MySQL 5.5.34. When I’m ready to publish my fix, what version number should I give it? The current packages have the version 5.5.33a+maria-1~CODENAME, where CODENAME is the code name for the Debian or Ubuntu release (ie. “sid” or “precise”, etc). Since I am not changing any code, it seems like like 5.5.33a should remain the same. I expected that the fixed package would be 5.5.33a+maria-2~CODENAME, but autobake-deb.sh assembles that version string and the 1 is hardcoded, not set by a variable. The two variables RELEASE_NAME and RELEASE_EXTRA can insert text between “+maria-“ and “1”. What is this project’s versioning convention in regards to changes that only affect the packaging? Matt

2 2

Re: [Maria-developers] [Commits] Rev 3942: MDEV-5205 - MariaDB does not start if more than 128 cpu's are available
by Sergei Golubchik 06 Nov '13

06 Nov '13

Hi, Sergey! On Nov 06, Sergey Vojtovich wrote: > ------------------------------------------------------------ > revno: 3942 > revision-id: svoj(a)mariadb.org-20131106185339-wjz3xuru8ojnz00l > parent: monty(a)askmonty.org-20131105183036-051tak9a2fq9ii05 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5-mdev5205 > timestamp: Wed 2013-11-06 22:53:39 +0400 > message: > MDEV-5205 - MariaDB does not start if more than 128 cpu's are available > > An addition to fix for MDEV-5205, fixes server crash on shutdown. > > Thread groups are destroyed asynchronously, that is kill server > thread sends shutdown request to all thread groups without waiting > for compeltion. > > It means all_groups array must not be freed until all thread groups > are down. This patch suggests that all_groups is freed when last > thread group is destroyed. Looks reasonable. ok to push Regards, Sergei

1 0

[Maria-developers] MariaDB 10.0.5 Beta Release Prep
by Daniel Bartholomew 05 Nov '13

05 Nov '13

I'm currently in the process of preparing for the MariaDB 10.0.5 Beta release. Draft changelog and release notes are here: - https://mariadb.com/kb/en/mariadb-1005-release-notes/ - https://mariadb.com/kb/en/mariadb-1005-changelog/ The release notes page is not much more than a template at present and needs to be be filled in before the release. Please add any items that should be there ASAP. Current plans are to activate and announce the release tomorrow (6 Nov). Thanks. -- Daniel Bartholomew MariaDB

1 0

[Maria-developers] Opened_plugin_libraries status variable
by Ian Gilfillan 05 Nov '13

05 Nov '13

Hi I would like to document the Opened_plugin_libraries status variable, introduced in MariaDB 10.0, but can't understand what it's all about. I have looked at: http://lists.askmonty.org/pipermail/commits/2013-April/004595.html but it's not really making sense what it's testing. In my own setup, each time I query the INFORMATION_SCHEMA.ALL_PLUGINS table, Opened_plugin_libraries increments by 19 (no matter what conditions I apply). The only correlation I can find is that there are 19 records with the LOAD_OPTION = 'OFF'. Is this what Opened_plugin_libraries is recording? Why does it increment each time I query the table (19, 38, 57 etc) - how is this useful? Can someone clarify exactly what this variable records, and how it's useful? thanks, ian

2 1

[Maria-developers] On how to fix indentation and history in sql_join_cache.cc
by Sergei Petrunia 05 Nov '13

05 Nov '13

Hi Sanja, My experiment was successfull. It seems, one can fix identation in sql_join_cache.cc so in the following way: # Create a branch in a state "right after the problem" bzr branch -rsanja(a)askmonty.org-20121223213711-ac5vz2fspwapjvtr 10.0 work-maria-10.0-base-cassandra-look1 # The -r3457 branch shows the state "right before the merge that borked it" bzr branch -r3457 work-maria-10.0-base-cassandra-look1 work-maria-10.0-base-cassandra-look1-r3457 cd work-maria-10.0-base-cassandra-look1-r3457 # Do some dummy change to force a merge gvim sql/sql_join_cache.cc bzr gcommit # Prepare the branch where we will fix it cd .. bzr branch work-maria-10.0-base-cassandra-look1 work-maria-10.0-base-cassandra-look1-cp cd work-maria-10.0-base-cassandra-look1-cp # This should show that you're going to merge the dummy change bzr missing --theirs-only ../work-maria-10.0-base-cassandra-look1-r3457/ # # This will will succeed without conflicts. We will use this merge as an # excuse to fix the problem # bzr merge --pull ../work-maria-10.0-base-cassandra-look1-r3457/ # # Now, edit the "broken" file and the original file until sql/sql_join_cache.cc # is ok # gvimdiff ~/home/psergey/dev2/work-maria-10.0-base-cassandra-look1-r3457/sql/sql_join_cache.cc sql/sql_join_cache.cc # Commit bzr gcommit # # Merge the change into a recent 10.0-base. # cd 10.0-base bzr merge --pull ../work-maria-10.0-base-cassandra-look1-cp/ bzr gcommit # # Check the history. It MUST NOT show any of today's merges. # the code in bkah_skip_index_tuple() should show that lines belong to # # igor(a)askmonty.org-20100831163421-iiddbral4jr3q75q # The main patch for MWL#128: Implement Block Nested Loop Hash Join. # # sanja(a)montyprogram.com-20120216064910-k0ud8ja7ef1jnpx4 # Counters for Index Condition Pushdown added (MDEV-130). bzr gannotate sql/sql_join_cache.cc BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 2

[Maria-developers] Preparing MDEV-4506, Parallel replication, for 10.0.5 release
by Kristian Nielsen 04 Nov '13

04 Nov '13

Hi Monty, I was asked to prepare MDEV-4506, parallel replication, for release of 10.0.5 on Monday. For this, I did merge to 10.0-base and then to 10.0. I have pushed the results here: lp:~maria-captains/maria/mdev-4506-base # for 10.0-base merge lp:~maria-captains/maria/mdev-4506 # for 10.0 merge However, I had problems with a couple of your changes, to temporary tables and innodb log file size. The code that you modified in 10.0-base does not seem to exist in 10.0 (or at least I couldn't find it easily). Can you take a look at this and see if you can complete the merge? The missing bits are just the few hunks appended at the end of this mail, so I assume it should be easy for someone who knows what the patches are supposed to do. The tree lp:~maria-captains/maria/mdev-4506 currently has a lot of test failures, but it looks like most of them will be fixed by completing the last bits of the merge. - Kristian. ----------------------------------------------------------------------- sql/sql_base.cc: <<<<<<< TREE if (table_list->mdl_request.type >= MDL_SHARED_WRITE && thd->tx_read_only && !(flags & (MYSQL_LOCK_LOG_TABLE | MYSQL_OPEN_HAS_MDL_LOCK))) ||||||| BASE-REVISION if (table_list->open_type != OT_BASE_ONLY && ! (flags & MYSQL_OPEN_SKIP_TEMPORARY)) ======= if (table_list->open_type != OT_BASE_ONLY && ! (flags & MYSQL_OPEN_SKIP_TEMPORARY) && thd->have_temporary_tables()) >>>>>>> MERGE-SOURCE { <<<<<<< TREE my_error(ER_CANT_EXECUTE_IN_READ_ONLY_TRANSACTION, MYF(0)); DBUG_RETURN(true); ||||||| BASE-REVISION for (table= thd->temporary_tables; table ; table=table->next) { if (table->s->table_cache_key.length == key_length + TMP_TABLE_KEY_EXTRA && !memcmp(table->s->table_cache_key.str, key, key_length + TMP_TABLE_KEY_EXTRA)) { /* We're trying to use the same temporary table twice in a query. Right now we don't support this because a temporary table is always represented by only one TABLE object in THD, and it can not be cloned. Emit an error for an unsupported behaviour. */ if (table->query_id) { DBUG_PRINT("error", ("query_id: %lu server_id: %u pseudo_thread_id: %lu", (ulong) table->query_id, (uint) thd->variables.server_id, (ulong) thd->variables.pseudo_thread_id)); my_error(ER_CANT_REOPEN_TABLE, MYF(0), table->alias.c_ptr()); DBUG_RETURN(TRUE); } table->query_id= thd->query_id; thd->thread_specific_used= TRUE; DBUG_PRINT("info",("Using temporary table")); goto reset; } } ======= if ((table= find_temporary_table(thd, key, key_length + TMP_TABLE_KEY_EXTRA))) { /* Check if we're trying to use the same temporary table twice in a query. Right now we don't support this because a temporary table is always represented by only one TABLE object in THD, and it can not be cloned. Emit an error for an unsupported behaviour. */ if (table->query_id) { DBUG_PRINT("error", ("query_id: %lu server_id: %u pseudo_thread_id: %lu", (ulong) table->query_id, (uint) thd->variables.server_id, (ulong) thd->variables.pseudo_thread_id)); my_error(ER_CANT_REOPEN_TABLE, MYF(0), table->alias.c_ptr()); DBUG_RETURN(TRUE); } table->query_id= thd->query_id; thd->thread_specific_used= TRUE; DBUG_PRINT("info",("Using temporary table")); goto reset; } >>>>>>> MERGE-SOURCE sql/slave.cc: <<<<<<< TREE static void set_thd_in_use_temporary_tables(Relay_log_info *rli) { TABLE *table; for (table= rli->save_temporary_tables ; table ; table= table->next) { table->in_use= rli->sql_thd; if (table->file != NULL) { /* Since we are stealing opened temporary tables from one thread to another, we need to let the performance schema know that, for aggregates per thread to work properly. */ table->file->unbind_psi(); table->file->rebind_psi(); } } } ||||||| BASE-REVISION static void set_thd_in_use_temporary_tables(Relay_log_info *rli) { TABLE *table; for (table= rli->save_temporary_tables ; table ; table= table->next) table->in_use= rli->sql_thd; } ======= >>>>>>> MERGE-SOURCE === modified file 'storage/innobase/srv/srv0start.c' --- storage/innobase/srv/srv0start.c 2013-09-06 20:31:30 +0000 +++ storage/innobase/srv/srv0start.c 2013-11-01 08:17:06 +0000 @@ -621,7 +621,7 @@ open_or_create_log_file( || size_high != srv_calc_high32(srv_log_file_size)) { fprintf(stderr, - "InnoDB: Error: log file %s is" + "InnoDB: Warning: log file %s is" " of different size %lu %lu bytes\n" "InnoDB: than specified in the .cnf" " file %lu %lu bytes!\n", @@ -629,7 +629,8 @@ open_or_create_log_file( (ulong) srv_calc_high32(srv_log_file_size), (ulong) srv_calc_low32(srv_log_file_size)); - return(DB_ERROR); + srv_log_file_size= size + + (((longlong) size_high) << 32); } } else { *log_file_created = TRUE;

2 3

Re: [Maria-developers] MDEV-5205 - MariaDB does not start if more than 128 cpu's are available
by Sergei Golubchik 01 Nov '13

01 Nov '13

Hi, Sergey! On Nov 01, Sergey Vojtovich wrote: > revno: 3939 > revision-id: svoj(a)mariadb.org-20131101154841-egjkqc4e00mxc3ty > parent: sanja(a)montyprogram.com-20131029165036-f4wqcr2nrxzuwuuv > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5-mdev5205 > timestamp: Fri 2013-11-01 19:48:41 +0400 > message: > MDEV-5205 - MariaDB does not start if more than 128 cpu's are available > > === modified file 'sql/threadpool_unix.cc' > --- a/sql/threadpool_unix.cc 2013-06-22 12:02:03 +0000 > +++ b/sql/threadpool_unix.cc 2013-11-01 15:48:41 +0000 > @@ -907,6 +907,7 @@ int thread_group_init(thread_group_t *th > thread_group->pollfd= -1; > thread_group->shutdown_pipe[0]= -1; > thread_group->shutdown_pipe[1]= -1; > + thread_group->queue.empty(); ok, but please add to the definition of I_P_List::empty() a comment that it's used in many places in the code instead of a constructor, to initialize a bzero-ed I_P_List instance. > DBUG_RETURN(0); > } > Regards, Sergei

1 0

[Maria-developers] doubt, Got a packet bigger than 'max_allowed_packet' bytes - 1153
by Roberto Spadim 31 Oct '13

31 Oct '13

Hi guys, i'm with a doubt now... using the php extension of mysql i got this error Got a packet bigger than 'max_allowed_packet' bytes - 1153 the point is... should the client api change the packet to a size smaller than max limit? or should i set the max_allowed_packet at server side? this variable have some memory problem if changed to a big value? today i'm using 1MB, but i don't know if i change for example to 1024MB i could break the server, does the max_allowed_packet just a limit, or a malloc parameter to alloc memory at each packet? -- Roberto Spadim SPAEmpresarial

1 1

Re: [Maria-developers] MWL83: review
by Sergei Petrunia 31 Oct '13

31 Oct '13

Hi Timour, Found another possible issue: The code in best_extension_by_limited_search() calls join->static_pushdown_cost(). static_pushdown_cost() may modify join->position[$I].read_time for any $I. Now, consider an example: SELECT * FROM t1,t2,t3 WHERE subquery_conditions The join optimizer will consider join orders: S1: t1 S2: t1 t2 S3: t1 t2 t3 S4: t1 t3 t2 On step S3, we will get a complete join order, and the new code in best_extension_by_limited_search() is invoked. It will call join->static_pushdown_cost(...) which may change the value of join->position[0].read_time On step S4, we again will get a complete join order. The new code in best_extension_by_limited_search() is invoked, it will call join->static_pushdown_cost(...), which again may change join->position[0].read_time Now, that value will be meaningless. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] Rev 3711: MDEV-5104 crash in Item_field::used_tables with broken order by in file:///home/tsk/mprog/src/5.3/
by Oleksandr Byelkin 29 Oct '13

29 Oct '13

Hi! OK to push! Thank you! 29.10.2013 12:39, timour(a)askmonty.org пишет: > At file:///home/tsk/mprog/src/5.3/ > > ------------------------------------------------------------ > revno: 3711 > revision-id: timour(a)askmonty.org-20131029103903-bvjnf3b2aqnoyut5 > parent: sanja(a)askmonty.org-20131021104549-857lwynmy813qyd4 > fixes bug: https://mariadb.atlassian.net/browse/MDEV-5104 > committer: timour(a)askmonty.org <timour(a)askmonty.org> > branch nick: 5.3 > timestamp: Tue 2013-10-29 12:39:03 +0200 > message: > MDEV-5104 crash in Item_field::used_tables with broken order by > > Analysis: > st_select_lex_unit::prepare() computes can_skip_order_by as TRUE. > As a result join->prepare() gets called with order == NULL, and > doesn't do name resolution for the inner ORDER clause. Due to this > the prepare phase doesn't detect that the query references non-exiting > function and field. > > Later join->optimize() calls update_used_tables() for a non-resolved > Item_field, which understandably has no Field object. This call results > in a crash. > > Solution: > Resolve unnecessary ORDER BY clauses to detect if they reference non-exising > objects. Then remove such clauses from the JOIN object. > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] MWL83: review
by Sergei Petrunia 29 Oct '13

29 Oct '13

Hi Timour, This is the first portion of the feedback. A general note about EXPLAIN output: "Subqueries: 2" looks ugly. A human will read this as " there are two subqueries somewhere", which is clearly not the case. All other lists (indexes for index merge, ref components) are comma-separated, while list of subquery ids uses spaces. As far as I understand, @@optimizer_switch flags and other controls for dynamic condition are present, but the feature is not working? We can't push it in this state, because this will be very confusing for anybody trying to use the new features. Further comments below marked with 'psergey:' > === modified file 'sql/item.cc' > --- sql/item.cc 2013-08-20 11:48:29 +0000 > +++ sql/item.cc 2013-09-27 08:25:14 +0000 > @@ -457,7 +457,8 @@ int Item::save_str_value_in_field(Field > Item::Item(): > is_expensive_cache(-1), rsize(0), name(0), orig_name(0), name_length(0), > fixed(0), is_autogenerated_name(TRUE), > - collation(&my_charset_bin, DERIVATION_COERCIBLE) > + collation(&my_charset_bin, DERIVATION_COERCIBLE)/*TODO, > + pushdown_tables(0)*/ > { > marker= 0; > maybe_null=null_value=with_sum_func=with_field=unsigned_flag=0; > @@ -466,7 +467,7 @@ int Item::save_str_value_in_field(Field > with_subselect= 0; > cmp_context= IMPOSSIBLE_RESULT; > /* Initially this item is not attached to any JOIN_TAB. */ > - join_tab_idx= MAX_TABLES; > + join_tab_idx= -1; > > /* Put item in free list so that we can free all items at end */ > THD *thd= current_thd; > @@ -515,7 +516,8 @@ int Item::save_str_value_in_field(Field > is_autogenerated_name(item->is_autogenerated_name), > with_subselect(item->has_subquery()), > collation(item->collation), > - cmp_context(item->cmp_context) > + cmp_context(item->cmp_context)/*TODO, > + pushdown_tables(0)*/ > { > next= thd->free_list; // Put in free list > thd->free_list= this; > @@ -583,7 +585,7 @@ void Item::cleanup() > DBUG_PRINT("enter", ("this: %p", this)); > fixed=0; > marker= 0; > - join_tab_idx= MAX_TABLES; > + join_tab_idx= -1; > if (orig_name) > name= orig_name; > DBUG_VOID_RETURN; > @@ -605,6 +607,19 @@ bool Item::cleanup_processor(uchar *arg) > > > /** > + Add the cost of an item to the cost of previous items in the > + containing expression tree. > + > + @param arg the cost of the previously visited items in the exression tree > +*/ > + > +bool Item::sum_cost_processor(uchar *arg) > +{ > + *((double*)arg)= *((double*)arg) + get_cost(); > + return false; > +} > + > +/** > rename item (used for views, cleanup() return original name). > > @param new_name new name of item; > @@ -8719,6 +8734,17 @@ void Item_cache::set_null() > } > > > +void Item_cache::update_used_tables() > +{ > + example->update_used_tables(); > + if (example->const_item()) > + { > + used_table_map= 0; > + maybe_null= example->maybe_null; > + } > +} > + > + > bool Item_cache_int::cache_value() > { > if (!example) > @@ -9649,7 +9675,7 @@ table_map Item_ref_null_helper::used_tab > #ifndef DBUG_OFF > > /* Debugger help function */ > -static char dbug_item_print_buf[256]; > +static char dbug_item_print_buf[2048]; > > const char *dbug_print_item(Item *item) > { > > === modified file 'sql/item.h' > --- sql/item.h 2013-07-17 19:24:29 +0000 > +++ sql/item.h 2013-09-27 13:22:36 +0000 > @@ -578,10 +578,12 @@ class Item { > make_cond_for_table procedure. During query execution, this item is > evaluated when the join loop reaches the corresponding JOIN_TAB. > > - If the value of join_tab_idx >= MAX_TABLES, this means that there is no > - corresponding JOIN_TAB. > + If join_tab_idx < 0, this means that there is no corresponding JOIN_TAB. > + > + cur_join_tab_idx is used during join optimization to determine the best > + position where to push an expression. > */ > - uint join_tab_idx; > + int join_tab_idx; > > public: > static void *operator new(size_t size) throw () > @@ -658,6 +660,9 @@ class Item { > subselect */ > DTCollation collation; > Item_result cmp_context; /* Comparison context */ > + /* tables in the QEP where this predicate should be pushed to */ > + //table_map pushdown_tables; > + psergey: It seems, you've introduced the above member, but then decided not to use it? Please remove all leftover comments, then. > // alloc & destruct is done as start of select using sql_alloc > Item(); > /* > @@ -1006,6 +1011,7 @@ class Item { > have to be updated in update_used_tables() > */ > virtual table_map not_null_tables() const { return used_tables(); } > + > /* > Returns true if this is a simple constant item like an integer, not > a constant expression. Used in the optimizer to propagate basic constants. > @@ -1172,12 +1178,14 @@ class Item { > virtual bool remove_fixed(uchar * arg) { fixed= 0; return 0; } > virtual bool cleanup_processor(uchar *arg); > virtual bool collect_item_field_processor(uchar * arg) { return 0; } > + virtual bool collect_item_subselect_processor(uchar * arg) { return 0; } > virtual bool add_field_to_set_processor(uchar * arg) { return 0; } > virtual bool find_item_in_field_list_processor(uchar *arg) { return 0; } > virtual bool find_item_processor(uchar *arg); > virtual bool change_context_processor(uchar *context) { return 0; } > virtual bool reset_query_id_processor(uchar *query_id_arg) { return 0; } > virtual bool is_expensive_processor(uchar *arg) { return 0; } > + virtual bool sum_cost_processor(uchar *arg); > virtual bool register_field_in_read_map(uchar *arg) { return 0; } > virtual bool register_field_in_write_map(uchar *arg) { return 0; } > virtual bool enumerate_field_refs_processor(uchar *arg) { return 0; } > @@ -1196,6 +1204,22 @@ class Item { > virtual bool find_selective_predicates_list_processor(uchar *opt_arg) > { return 0; } > > + /** > + Set the join tab index to the minimal (left-most) JOIN_TAB to which this > + Item is attached. The number is an index in depth_first_tab() traversal > + order. > + */ > + virtual bool set_join_tab_idx_processor(uchar *join_tab_idx_arg) > + { > + /* > + Items can participate in several pushdown conditions attached to > + different join tabs. The join tab index property identifies the > + left-most join tab in the join plan. > + */ > + set_join_tab_idx(*((int*) join_tab_idx_arg)); > + return false; > + } > + > /* To call bool function for all arguments */ > struct bool_func_call_args > { > @@ -1450,6 +1474,32 @@ class Item { > is_expensive_cache= walk(&Item::is_expensive_processor, 0, (uchar*)0); > return test(is_expensive_cache); > } > + /** > + The default cost to evaluate a 'cheap' predicate. > + TODO: > + Notice that this also adds the cost of accessing Fields (via Item_field). > + Perhaps this cost should be smaller or 0. > + */ > + virtual double get_cost() { return (1 / (double)TIME_FOR_COMPARE); } > + /* The default execution-time cost of an Item is its compile-time cost. */ > + virtual double exec_cost() { return get_cost(); } psergey: I'm reading the above and I don't understand get_cost() and exec_cost(). > + > + > + /** > + Return true if this Item must be wrapped in a dynamic condition. > + */ > + bool needs_dynamic_cond() > + { > + /* > + @todo Do not check if a subquery is expensive, because even subqueries > + that the optimizer reports to be cheap, are actually quite expensive to > + re-evaluate multiple times. For instance the scalar SQ of DBT3-Q20 > + reports only ~7 examined rows, but still it is very beneficial to move > + it to later JOIN_TAB in the query plan. > + */ > + return (with_subselect && !const_item() /* && is_expensive() */); > + } > + > virtual Field::geometry_type get_geometry_type() const > { return Field::GEOM_GEOMETRY; }; > String *check_well_formed_result(String *str, bool send_error= 0); > @@ -1488,12 +1538,12 @@ class Item { > Item is attached. The number is an index is depth_first_tab() traversal > order. > */ > - virtual void set_join_tab_idx(uint join_tab_idx_arg) > + virtual void set_join_tab_idx(int join_tab_idx_arg) > { > - if (join_tab_idx_arg < join_tab_idx) > + if (join_tab_idx < 0 || join_tab_idx_arg < join_tab_idx) > join_tab_idx= join_tab_idx_arg; > } > - virtual uint get_join_tab_idx() { return join_tab_idx; } > + virtual int get_join_tab_idx() { return join_tab_idx; } > > table_map view_used_tables(TABLE_LIST *view) > { > @@ -4065,6 +4115,7 @@ class Item_cache: public Item_basic_cons > } > > void set_used_tables(table_map map) { used_table_map= map; } > + void update_used_tables(); > > virtual bool allocate(uint i) { return 0; } > virtual bool setup(Item *item) > @@ -4443,4 +4494,8 @@ class Item_iterator_row: public Item_ite > void close() {} > }; > > +#ifndef DBUG_OFF > +const char *dbug_print_item(Item *item); > +#endif /*DBUG_OFF*/ > + > #endif /* SQL_ITEM_INCLUDED */ > > === modified file 'sql/item_cmpfunc.cc' > --- sql/item_cmpfunc.cc 2013-08-20 11:48:29 +0000 > +++ sql/item_cmpfunc.cc 2013-09-27 08:25:14 +0000 > @@ -461,7 +461,17 @@ static bool convert_const_to_int(THD *th > field_item->field_type() != MYSQL_TYPE_YEAR) > return 1; > > - if ((*item)->const_item() && !(*item)->is_expensive()) > + /* > + An item can be converted to an int if: > + - is constant, and > + - it is not a subquery (or contains a subquery), and > + - it is cheap to compute. > + The reason for the second requirement is to prevent subquery > + optimization/execution during the JOIN::prepare phase, because this > + function is called during the prepare phase as well. > + */ > + if ((*item)->const_item() && > + !(*item)->with_subselect && !(*item)->is_expensive()) > { > TABLE *table= field->table; > ulonglong orig_sql_mode= thd->variables.sql_mode; > @@ -1478,7 +1488,8 @@ bool Item_in_optimizer::fix_left(THD *th > eval_not_null_tables(NULL); > with_sum_func= args[0]->with_sum_func; > with_field= args[0]->with_field; > - if ((const_item_cache= args[0]->const_item())) > + if (!(args[0]->with_subselect || args[0]->is_expensive()) && > + (const_item_cache= args[0]->const_item())) > { > cache->store(args[0]); > cache->cache_value(); > @@ -4685,6 +4696,17 @@ void Item_cond::neg_arguments(THD *thd) > } > > > +double Item_cond::exec_cost() > +{ > + List_iterator_fast<Item> li(list); > + Item *item; > + double cost= 0; > + while ((item= li++)) > + cost+= item->exec_cost(); > + return cost; > +} > + > + > void Item_cond_and::mark_as_condition_AND_part(TABLE_LIST *embedding) > { > List_iterator<Item> li(list); > @@ -6386,3 +6408,295 @@ longlong Item_func_dyncol_exists::val_in > null_value= TRUE; > return 0; > } > + > + > +double Item_func_collect_stats::exec_cost() > +{ > + return args[0]->exec_cost(); > +} > + > + > +double Item_func_collect_stats::exec_selectivity() > +{ > + return count_evals ? count_true / count_evals : 0; > +} > + > + > +#ifndef DBUG_OFF > +void Item_func_collect_stats::dbug_print_stats() > +{ > + char buff[1024]; > + String str(buff,(uint32) sizeof(buff), system_charset_info); > + str.length(0); > + str.extra_allocation(1024); > + print(&str, QT_ORDINARY); > + > + (void) fprintf(DBUG_FILE, " sel: %2.2f, cond_evals: %f, cond_true: %f (", > + exec_selectivity(), count_evals, count_true); > + (void) fputs(str.c_ptr_safe(),DBUG_FILE); > + (void) fprintf(DBUG_FILE, ")\n"); > + > + if (is_cond_and(args[0])) > + { > + List_iterator<Item> and_parts_it(*((Item_cond*) args[0])->argument_list()); > + Item *and_part; > + (void) fprintf(DBUG_FILE, " Individual conjunct selectivity: {\n"); > + while ((and_part= and_parts_it++)) > + { > + if (and_part->type() == Item::FUNC_ITEM && > + (((Item_func*)and_part)->functype() == Item_func::DYNAMIC_COND_FUNC || > + ((Item_func*)and_part)->functype() == Item_func::EXEC_STATS_FUNC)) > + { > + Item_func_collect_stats *stat_cond= (Item_func_collect_stats*) and_part; > + stat_cond->dbug_print_stats(); > + } > + } > + (void) fprintf(DBUG_FILE, "}\n"); > + } > +} > +#endif /*DBUG_OFF*/ > + > + > +Item_func_dynamic_cond::Item_func_dynamic_cond(Item *cond, JOIN_TAB *atab, JOIN_TAB **ctab) > + : Item_func_collect_stats(cond) > +{ > + JOIN *join= atab->join; > + JOIN_TAB *cur_tab; > + THD *thd= atab->table->in_use; > + > + active_tab= atab; > + exec_tab= ctab; > + max_evals= thd->variables.dynamic_pushdown_max_evals; > + cost_ratio= (double) thd->variables.dynamic_pushdown_cost_ratio / 100.0; > + sampling_probability= (double) thd->variables.dynamic_pushdown_sampling_probability / 100.0; > + max_selectivity= (double) thd->variables.dynamic_pushdown_max_selectivity / 100.0; > + > + /* Collect the set of all subqueries referenced inside 'cond'. */ > + cond->walk(&Item::collect_item_subselect_processor, 0, > + (uchar*) &subqueries); > + > + /* Find the boundaries between which the predicate can be moved. */ > + if (active_tab->bush_root_tab) > + { > + first_tab= active_tab->bush_root_tab->bush_children->start; > + last_tab= active_tab->bush_root_tab->bush_children->end; > + } > + else > + { > + first_tab= join->join_tab + join->const_tables; > + last_tab= join->join_tab + join->top_join_tab_count; > + } > + > + DBUG_ASSERT(active_tab >= first_tab && active_tab < last_tab); > + > + /* Precompute some cost information that is static at execution time. */ > + first_tab->exec_fanout= 1; > + for (cur_tab= first_tab; cur_tab < last_tab; cur_tab++) > + rel_access_cost+= cur_tab->read_time; > + rel_access_cost/= first_tab->records_read; > + > + /* Copied from Item_func_rand::seed_random, what are the right args? */ > + my_rnd_init(&rand, (uint32) (0x10001L+55555555L), > + (uint32) (0x10000001L)); > + last_res= -1; > +} > + > + > +longlong Item_func_dynamic_cond::val_int() > +{ > + longlong res= 1; > + THD *thd= active_tab->join->thd; > + > + DBUG_ASSERT(active_tab >= first_tab && active_tab < last_tab); > + DBUG_ASSERT(!(*exec_tab) || (active_tab->join == (*exec_tab)->join)); > + DBUG_ASSERT(!(*exec_tab) || (*exec_tab >= first_tab && *exec_tab < last_tab)); > + > + if (is_active()) > + { > + /* > + If the active tab is the same as the first tab, this method must > + re-evaluate on each call (that is, not use the cached last_res). > + */ > + DBUG_ASSERT(*exec_tab > first_tab || last_res == -1); > + /* > + If this dynamic condition is attached to the JOIN_TAB currently being > + executed, evaluate the wrapper condition. > + */ > + if (last_res == -1) > + { > + last_res= Item_func_collect_stats::val_int(); > + /* Choose the best JOIN_TAB where to attach this dynamic condition. */ > + if (optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_DYNAMIC_PUSHDOWN) && > + ((ulonglong) count_evals % max_evals == 0)) > + choose_best_join_tab(); > + } > + res= last_res; > + last_res= -1; > + } > + else if (optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_DYNAMIC_PUSHDOWN) && > + (*exec_tab == first_tab) && my_rnd(&rand) <= sampling_probability) > + { > + /* > + If dynamic pushdown is used, evaluate the condition to collect statistics > + about its selectivity. The result is returned so it can be used to > + filter join records during sampling. > + */ > + DBUG_ASSERT(last_res == -1); > + res= last_res= Item_func_collect_stats::val_int(); > + } > + return res; > +} > + > + > +/** > +*/ > + > +bool Item_func_dynamic_cond::walk(Item_processor processor, bool walk_subquery, > + uchar *argument) > +{ > + return is_active() ? > + Item_func::walk(processor, walk_subquery, argument) : false; > +} > + > + > +/** > + Compute the execution-time selectivity of the condition based on counters. > +*/ > + > +double Item_func_dynamic_cond::exec_selectivity() > +{ > + return is_active() ? Item_func_collect_stats::exec_selectivity() : 1; > +} > + > + > +/** > + Compute the total cost of all subqueries wrapped in this dynamic condition. > + > + @todo > + Currently the method returns the compile-time cost. It should instead > + recompute the cost in the same way as choose_best_join_tab() using execution > + time statistics. > +*/ > + > +double Item_func_dynamic_cond::exec_cost() > +{ > + if (is_active()) > + { > + double cost= 0; > + Item_subselect *subs; > + List_iterator_fast<Item_subselect> subs_it(subqueries); > + while ((subs= subs_it++)) > + cost+= subs->get_cost(); > + return cost; > + } > + else > + return 0; > +} > + > + > +/** > + Choose the JOIN_TAB in the query plan, where to attach this condition so that > + that the total plan cost is minimal. > +*/ > + > +void Item_func_dynamic_cond::choose_best_join_tab() > +{ > + JOIN_TAB *cur_tab, *best_tab; > + > + /* The current JOI_TAB where this dynamic cond is attached. */ > + JOIN_TAB *active_tab_save= active_tab; > + /* The JOIN_TAB that is being executed. */ > + JOIN_TAB *exec_tab_save= *exec_tab; > + double join_fanout; > + double cond_cost; > + double cond_sel; > + double rel_plan_cost; > + double best_plan_cost= DBL_MAX; > + > + /* > + Update the execution-time approximations of partial join fanouts. > + > + TODO: this is very inexact approach, because even if there is a fanout > + estimate for join_tab J_i, it will be incomparable to the estimate of the > + previous join tabs if not all rows of J_[i-1] were joined with J_i. > + */ > + first_tab->exec_fanout= first_tab->exec_partial_join_records ? > + /* use the execution-time counter */ > + first_tab->exec_partial_join_records : > + /* use the optimize-time estimate */ > + first_tab->records_read; > + for (cur_tab= first_tab + 1; cur_tab < last_tab; cur_tab++) > + { > + /* > + If an execution time counter is present, use it, otherwise use the > + optimize-time estimate. > + */ > + if (cur_tab->exec_partial_join_records) > + cur_tab->exec_fanout= cur_tab->exec_partial_join_records / > + (cur_tab-1)->exec_partial_join_records; > + else > + cur_tab->exec_fanout= cur_tab->records_read; > + } > + > + /* > + Compute the cost of each alternative plan where this dynamic condition is > + pushed to each possible JOIN_TAB. Currently we only consider moving the > + condition later in the join plan, that is, to any JOIN_TAB after active_tab. > + > + This condition is 'moved' from one JOIN_TAB to the next by looping through > + all possible 'active_tab' values, starting from the current one. > + */ > + for (active_tab= active_tab_save; active_tab < last_tab; active_tab++) > + { > + rel_plan_cost= 0; > + > + /* Recompute the plan cost for the current active_tab. */ > + for (cur_tab= first_tab; cur_tab < last_tab; cur_tab++) > + { > + Item_func_collect_stats *pushed_cond; > + cond_cost= 0; > + cond_sel= 0; > + join_fanout= cur_tab->exec_fanout; > + > + /* > + Set the current execution tab. This affects all dynamic conditions of > + the current JOIN. > + */ > + *exec_tab= cur_tab; > + > + if (cur_tab->select_cond) > + { > + /* > + If a top-level AND condition contains an expensive non-constant > + subquery, each of the AND terms is wrapped in an item that collectes > + execution statistics. > + */ > + DBUG_ASSERT(cur_tab->select_cond->type() == Item::FUNC_ITEM && > + (((Item_func*)cur_tab->select_cond)->functype() == Item_func::DYNAMIC_COND_FUNC || > + ((Item_func*)cur_tab->select_cond)->functype() == Item_func::EXEC_STATS_FUNC)); > + > + pushed_cond= (Item_func_collect_stats*) cur_tab->select_cond; > + cond_cost+= pushed_cond->exec_cost(); > + cond_sel*= pushed_cond->exec_selectivity(); > + } > + > + rel_plan_cost+= rel_access_cost + join_fanout * cond_sel * cond_cost; > + } > + > + if (rel_plan_cost / best_plan_cost < cost_ratio) > + { > + best_plan_cost= rel_plan_cost; > + best_tab= active_tab; > + } > + } > + > + active_tab= best_tab; > + *exec_tab= exec_tab_save; > +} > + > + > +void Item_func_dynamic_cond::print(String *str, enum_query_type query_type) > +{ > + if (is_active()) > + Item_bool_func::print(str, query_type); > +} > > === modified file 'sql/item_cmpfunc.h' > --- sql/item_cmpfunc.h 2013-07-17 19:24:29 +0000 > +++ sql/item_cmpfunc.h 2013-09-27 08:25:14 +0000 > @@ -125,6 +125,12 @@ class Item_bool_func :public Item_int_fu > bool is_bool_func() { return 1; } > void fix_length_and_dec() { decimals=0; max_length=1; } > uint decimal_precision() const { return 1; } > + /* > + The execution logic must guarantee that these methods are called only > + for classes that provide real implementations. In this way we can use > + polymorphism without implementing the methods for all subclasses. > + */ > + virtual double exec_selectivity() { DBUG_ASSERT(FALSE); return 0; } > }; > > > @@ -258,13 +264,12 @@ class Item_in_optimizer: public Item_boo > void cleanup(); > const char *func_name() const { return "<in_optimizer>"; } > Item_cache **get_cache() { return &cache; } > + Item *left_expr() { return args[0]; } > void keep_top_level_cache(); > Item *transform(Item_transformer transformer, uchar *arg); > virtual Item *expr_cache_insert_transformer(uchar *thd_arg); > bool is_expensive_processor(uchar *arg); > bool is_expensive(); > - void set_join_tab_idx(uint join_tab_idx_arg) > - { args[1]->set_join_tab_idx(join_tab_idx_arg); } > virtual void get_cache_parameters(List<Item> &parameters); > bool is_top_level_item(); > bool eval_not_null_tables(uchar *opt_arg); > @@ -492,8 +497,139 @@ class Item_func_trig_cond: public Item_b > const char *func_name() const { return "trigcond"; }; > bool const_item() const { return FALSE; } > bool *get_trig_var() { return trig_var; } > + void update_used_tables() > + { > + Item_bool_func::update_used_tables(); > + const_item_cache= false; > + used_tables_cache|= OUTER_REF_TABLE_BIT; > + } > +}; > + > + > +/** > + Item wrapper that collects execution-time statistics about its child Item. > + > + This class can be used only during execution. > +*/ > + > +class Item_func_collect_stats : public Item_bool_func > +{ > +protected: > + double count_true; /* The number of times the predicate was true. */ > + double count_evals; /* The number of times the predicate was evaluated. */ > +public: > + Item_func_collect_stats(Item *cond) > + : Item_bool_func(cond), count_true(0), count_evals(0) > + { > + DBUG_ASSERT(cond && cond->fixed); > + /* > + It doesn't make sense to wrap an item that is already capable of > + collecting execution-time statistitcs. > + */ > + DBUG_ASSERT(cond->type() != Item::FUNC_ITEM || > + (((Item_func*)cond)->functype() != Item_func::EXEC_STATS_FUNC && > + ((Item_func*)cond)->functype() != Item_func::DYNAMIC_COND_FUNC)); > + fixed= true; > + } > + enum Functype functype() const { return EXEC_STATS_FUNC; }; > + const char *func_name() const { return "exec_stats"; }; > + bool const_item() const { return FALSE; } > + longlong val_int() > + { > + longlong res= args[0]->val_int(); > + ++count_evals; > + if (res) > + ++count_true; > + return res; > + } > + virtual double exec_selectivity(); > + virtual double exec_cost(); > + > +#ifndef DBUG_OFF > + void dbug_print_stats(); > +#endif > +}; > + > + > +/** > + A wrapper class that allows to change dynamically (during execution) the > + JOIN_TAB that a condition is attached to. > +*/ > + > +class Item_func_dynamic_cond : public Item_func_collect_stats > +{ > +protected: > + List<Item_subselect> subqueries; > + /* The join_tab where the condition is currently 'pushed'.*/ > + st_join_table *active_tab; > + /* The join_tab currently being executed. */ > + st_join_table **exec_tab; > + /* The boundaries of the sub-plan where this predicate can be pushed. */ > + st_join_table *first_tab, *last_tab; > + /* > + The relative access cost based on optimizer estimates. This is the cost > + of all access methods per one record from the first table in the plan. > + */ > + double rel_access_cost; > + /* > + The number of times a dynamic condition must be evaluated in > + order to trigger dynamic pushdown". Cached copy of system variable > + dynamic_pushdown_max_evals. > + */ > + ha_rows max_evals; > + /* > + The ratio by which a new placement of the predicate in the plan > + must improve the plan cost in order to trigger dynamic pushdown. > + Derived from system variable dynamic_pushdown_cost_ratio. > + */ > + double cost_ratio; > + /* > + The probability with which to evaluate this condition in order to > + estimate its selectivity. > + */ > + double sampling_probability; > + /* > + If this predicate's selectivity is below this limit, it is considered > + sufficiently selective to re-filter all records in a partial join. > + The selectivity itself is estimated by sampling the predicate with > + 'sampling_probability'. > + */ > + double max_selectivity; > + /* > + Random number generator state. > + @todo Consider making it static. > + */ > + struct my_rnd_struct rand; > + /* > + The result of the last evaluation, -1 denotes an invalid result which > + requires a re-evaluation. This member is needed because the same Item > + may be evaluated more than once for the same partial join record due to: > + - sampling - then the item is "active" both for the left-most pushdown > + join_tab, and for the "active" join_tab. > + - moving the item to a later join_tab. > + */ > + longlong last_res; > + > +protected: > + bool is_active() > + { > + DBUG_ASSERT(*exec_tab); > + return (active_tab == *exec_tab); > + } > +public: > + Item_func_dynamic_cond(Item *cond, st_join_table *atab, st_join_table **ctab); > + longlong val_int(); > + enum Functype functype() const { return DYNAMIC_COND_FUNC; }; > + const char *func_name() const { return "dyncond"; }; > + bool walk(Item_processor processor, bool walk_subquery, uchar *argument); > + virtual double exec_selectivity(); > + virtual double exec_cost(); > + void choose_best_join_tab(); > + > + virtual void print(String *str, enum_query_type query_type); > }; > > + > class Item_func_not_all :public Item_func_not > { > /* allow to check presence of values in max/min optimization */ > @@ -1609,6 +1745,7 @@ class Item_cond :public Item_bool_func > Item *compile(Item_analyzer analyzer, uchar **arg_p, > Item_transformer transformer, uchar *arg_t); > bool eval_not_null_tables(uchar *opt_arg); > + virtual double exec_cost(); > }; > > template <template<class> class LI, class T> class Item_equal_iterator; > @@ -1922,6 +2059,9 @@ class Item_cond_and :public Item_cond > > inline bool is_cond_and(Item *item) > { > + if (!item) > + return FALSE; > + > if (item->type() != Item::COND_ITEM) > return FALSE; > > > === modified file 'sql/item_func.h' > --- sql/item_func.h 2013-08-20 11:48:29 +0000 > +++ sql/item_func.h 2013-09-27 08:25:14 +0000 > @@ -64,7 +64,7 @@ class Item_func :public Item_result_fiel > SP_STARTPOINT,SP_ENDPOINT,SP_EXTERIORRING, > SP_POINTN,SP_GEOMETRYN,SP_INTERIORRINGN, > NOT_FUNC, NOT_ALL_FUNC, > - NOW_FUNC, TRIG_COND_FUNC, > + NOW_FUNC, TRIG_COND_FUNC, DYNAMIC_COND_FUNC, EXEC_STATS_FUNC, > SUSERVAR_FUNC, GUSERVAR_FUNC, COLLATE_FUNC, > EXTRACT_FUNC, CHAR_TYPECAST_FUNC, FUNC_SP, UDF_FUNC, > NEG_FUNC, GSYSVAR_FUNC, DYNCOL_FUNC }; > > === modified file 'sql/item_subselect.cc' > --- sql/item_subselect.cc 2013-07-17 19:24:29 +0000 > +++ sql/item_subselect.cc 2013-09-23 15:47:28 +0000 > @@ -50,8 +50,8 @@ int check_and_do_in_subquery_rewrites(JO > Item_subselect::Item_subselect(): > Item_result_field(), value_assigned(0), own_engine(0), thd(0), old_engine(0), > used_tables_cache(0), have_to_be_excluded(0), const_item_cache(1), > - inside_first_fix_fields(0), done_first_fix_fields(FALSE), > - expr_cache(0), forced_const(FALSE), substitution(0), engine(0), eliminated(FALSE), > + inside_first_fix_fields(0), done_first_fix_fields(FALSE), expr_cache(0), > + forced_const(FALSE), substitution(0), engine(0), eliminated(FALSE), > changed(0), is_correlated(FALSE) > { > DBUG_ENTER("Item_subselect::Item_subselect"); > @@ -400,6 +400,21 @@ bool Item_subselect::set_fake_select_as_ > } > > > +bool Item_subselect::collect_item_subselect_processor(uchar * arg) > +{ > + List<Item_subselect> *subs_list= (List<Item_subselect>*) arg; > + List_iterator<Item_subselect> subs_list_it(*subs_list); > + Item_subselect *cur_subs; > + while ((cur_subs= subs_list_it++)) > + { > + if (cur_subs->eq(this, 1)) > + return false; /* Already in the set. */ > + } > + subs_list->push_back(this); > + return false; > +} > + > + > bool Item_subselect::mark_as_dependent(THD *thd, st_select_lex *select, > Item *item) > { > @@ -551,7 +566,7 @@ void Item_subselect::recalc_used_tables( > */ > bool Item_subselect::is_expensive() > { > - double examined_rows= 0; > + double examined_rows= 0, cur_examined_rows= 0; > > for (SELECT_LEX *sl= unit->first_select(); sl; sl= sl->next_select()) > { > @@ -560,19 +575,20 @@ bool Item_subselect::is_expensive() > continue; > > /* > - Subqueries whose result is known after optimization are not expensive. > - Such subqueries have all tables optimized away, thus have no join plan. > + The cost of a subquery is known if it was optimized, and it is either > + executable, or was already executed and cleaned up. > */ > - if (cur_join->optimized && > - (cur_join->zero_result_cause || !cur_join->tables_list)) > - return false; > + if (cur_join->have_query_plan != JOIN::QEP_EXECUTABLE && > + cur_join->have_query_plan != JOIN::QEP_DELETED) > + return true; > > /* > - If a subquery is not optimized we cannot estimate its cost. A subquery is > - considered optimized if it has a join plan. > + If JOIN::optimize determined that the subquery has an empty result, or > + this is a table-less subquery, then then subquery is cheap. > */ > - if (!(cur_join->optimized && cur_join->join_tab)) > - return true; > + if (!cur_join->table_count || !cur_join->tables_list || > + cur_join->zero_result_cause) > + return false; > > if (sl->first_inner_unit()) > { > @@ -583,13 +599,64 @@ bool Item_subselect::is_expensive() > return true; > } > > - examined_rows+= cur_join->get_examined_rows(); > + if ((cur_examined_rows= cur_join->get_examined_rows_estimate()) < 0) > + return true; psergey: This looks very weird. Perhaphs, it is better to add a constant: const ha_rows NO_EXAMINED_ROWS_ESTIMATE_YET=-1; > + examined_rows+= cur_examined_rows; > } > > return (examined_rows > thd->variables.expensive_subquery_limit); > } > > > +/** > + Compute the cost of a subquery and all its subqueries. > + > + @return total subquery cost in terms of JOIN::best_read > +*/ > + > +double Item_subselect::get_cost() > +{ > + double total_cost= 0; > + for (SELECT_LEX *sl= unit->first_select(); sl; sl= sl->next_select()) > + { > + JOIN *inner_join= sl->join; > + total_cost+= > + inner_join->best_read + /* The cost of this subquery. */ > + /* The cost of the subqueries of the subquery. */ > + inner_join->total_subquery_cost(); > + } > + return total_cost; > +} > + > + > +/** > + Check if the subquery predicate has only one possible execution strategy. > + > + @details > + If there is only one possible execution strategy for a subquery predicate, > + the optimizer doesn't need to know the number of times the subquery will be > + executed in order to choose the better strategy. In this case the underlying > + subquery can be optimized before optimizing the outer query. > + > + @retval FALSE more than one strategies are possible > + @retval TRUE only one strategy is possible > +*/ > + > +bool Item_subselect::has_single_strategy() const > +{ > + if (!is_in_predicate()) > + return true; > + else > + { > + Item_in_subselect *in_subs= (Item_in_subselect*) this; > + if (!in_subs->test_strategy(SUBS_MATERIALIZATION) || > + !in_subs->test_strategy(SUBS_IN_TO_EXISTS)) > + return true; > + } > + return false; > +} > + > + > bool Item_subselect::walk(Item_processor processor, bool walk_subquery, > uchar *argument) > { > @@ -835,6 +902,10 @@ bool Item_in_subselect::exec() > test_if_item_cache_changed(*left_expr_cache) < 0) > DBUG_RETURN(FALSE); > > + if (first_execution && test_strategy(SUBS_MATERIALIZATION) && > + engine->engine_type() == subselect_engine::SINGLE_SELECT_ENGINE) > + setup_mat_engine(); > + > /* > The exec() method below updates item::value, and item::null_value, thus if > we don't call it, the next call to item::val_int() will return whatever > @@ -888,6 +959,12 @@ void Item_subselect::update_used_tables( > if (!(used_tables_cache & ~engine->upper_select_const_tables())) > const_item_cache= 1; > } > + /* > + TODO: when merged with dynamic pushdown, pushdown_tables should not be > + added to the used tables. Instead, dynamic pushdown could use this table > + as the initial dynamic position, or it may ignore this estimate altogether. > + used_tables_cache|= pushdown_tables; > + */ > } > } > > @@ -896,6 +973,16 @@ void Item_subselect::print(String *str, > { > if (engine) > { > + /* > + For non-SJ materialization the engine is created during exec(), therefore > + it is not available during EXPLAIN. Create the materialization engine in > + order to print it. > + */ > + if (is_in_predicate() && > + ((Item_in_subselect*)this)->test_strategy(SUBS_MATERIALIZATION) && > + engine->engine_type() == subselect_engine::SINGLE_SELECT_ENGINE) > + ((Item_in_subselect*)this)->setup_mat_engine(); > + > str->append('('); > engine->print(str, query_type); > str->append(')'); > @@ -1921,7 +2008,7 @@ bool Item_allany_subselect::transform_in > The swap is needed for expressions of type 'f1 < ALL ( SELECT ....)' > where we want to evaluate the sub query even if f1 would be null. > */ > - subs= func->create_swap(*(optimizer->get_cache()), subs); > + subs= func->create_swap(optimizer->left_expr(), subs); > thd->change_item_tree(place, subs); > if (subs->fix_fields(thd, &subs)) > DBUG_RETURN(true); > @@ -3174,8 +3261,13 @@ void Item_in_subselect::update_used_tabl > { > Item_subselect::update_used_tables(); > left_expr->update_used_tables(); > - //used_tables_cache |= left_expr->used_tables(); > used_tables_cache= Item_subselect::used_tables() | left_expr->used_tables(); > + /* > + Constant optimization of the outer query may transform the left IN operand > + into a constant. The subquery is no longer uncacheable. > + */ > + if (!used_tables_cache) > + engine->make_cacheable(); > } > > > @@ -4128,12 +4220,24 @@ uint8 subselect_single_select_engine::un > } > > > +void subselect_single_select_engine::make_cacheable() > +{ > + select_lex->uncacheable= 0; > +} > + > + > uint8 subselect_union_engine::uncacheable() > { > return unit->uncacheable; > } > > > +void subselect_union_engine::make_cacheable() > +{ > + unit->uncacheable= 0; > +} > + > + > void subselect_single_select_engine::exclude() > { > select_lex->master_unit()->exclude_level(); > @@ -5045,7 +5149,7 @@ double get_fanout_with_deps(JOIN *join, > !tab->emb_sj_nest && > tab->records_read != 0) > { > - fanout *= rows2double(tab->records_read); > + fanout *= tab->records_read; > } > } > return fanout; > @@ -5234,7 +5338,7 @@ int subselect_hash_sj_engine::exec() > materialize_join->exec(); > if ((res= test(materialize_join->error || thd->is_fatal_error || > thd->is_error()))) > - goto err; > + goto end; > > /* > TODO: > @@ -5259,7 +5363,8 @@ int subselect_hash_sj_engine::exec() > item_in->reset(); > item_in->make_const(); > item_in->set_first_execution(); > - DBUG_RETURN(FALSE); > + res= 0; > + goto end; > } > > /* > @@ -5302,7 +5407,8 @@ int subselect_hash_sj_engine::exec() > item_in->null_value= 1; > item_in->make_const(); > item_in->set_first_execution(); > - DBUG_RETURN(FALSE); > + res= 0; > + goto end; > } > > if (has_covering_null_row) > @@ -5360,7 +5466,7 @@ int subselect_hash_sj_engine::exec() > { > /* This is an irrecoverable error. */ > res= 1; > - goto err; > + goto end; > } > } > } > @@ -5369,7 +5475,7 @@ int subselect_hash_sj_engine::exec() > lookup_engine= pm_engine; > item_in->change_engine(lookup_engine); > > -err: > +end: > thd->lex->current_select= save_select; > DBUG_RETURN(res); > } > > === modified file 'sql/item_subselect.h' > --- sql/item_subselect.h 2013-03-27 22:41:02 +0000 > +++ sql/item_subselect.h 2013-09-23 15:47:28 +0000 > @@ -133,13 +133,14 @@ class Item_subselect :public Item_result > > Item_subselect(); > > - virtual subs_type substype() { return UNKNOWN_SUBS; } > - bool is_in_predicate() > + virtual subs_type substype() const { return UNKNOWN_SUBS; } > + bool is_in_predicate() const > { > return (substype() == Item_subselect::IN_SUBS || > substype() == Item_subselect::ALL_SUBS || > substype() == Item_subselect::ANY_SUBS); > } > + bool has_single_strategy() const; > > /* > We need this method, because some compilers do not allow 'this' > @@ -182,6 +183,7 @@ class Item_subselect :public Item_result > void make_const() > { > used_tables_cache= 0; > + //pushdown_tables= 0; > const_item_cache= 0; > forced_const= TRUE; > } > @@ -234,6 +236,8 @@ class Item_subselect :public Item_result > @retval FALSE otherwise > */ > bool is_expensive_processor(uchar *arg) { return is_expensive(); } > + virtual double get_cost(); > + bool collect_item_subselect_processor(uchar * arg); > > /** > Get the SELECT_LEX structure associated with this Item. > @@ -261,6 +265,8 @@ class Item_subselect :public Item_result > friend bool convert_join_subqueries_to_semijoins(JOIN *join); > }; > > +typedef bool (Item_subselect::*subselect_filter)() const; > + > /* single value subselect */ > > class Item_cache; > @@ -274,7 +280,7 @@ class Item_singlerow_subselect :public I > {} > > void cleanup(); > - subs_type substype() { return SINGLEROW_SUBS; } > + subs_type substype() const { return SINGLEROW_SUBS; } > > void reset(); > void no_rows_in_result(); > @@ -370,7 +376,7 @@ class Item_exists_subselect :public Item > emb_on_expr_nest(NULL), optimizer(0), exists_transformed(0) > {} > > - subs_type substype() { return EXISTS_SUBS; } > + subs_type substype() const { return EXISTS_SUBS; } > void reset() > { > eliminated= FALSE; > @@ -577,7 +583,7 @@ class Item_in_subselect :public Item_exi > pushed_cond_guards(NULL), func(NULL), is_jtbm_merged(FALSE), > is_jtbm_const_tab(FALSE), upper_item(0) {} > void cleanup(); > - subs_type substype() { return IN_SUBS; } > + subs_type substype() const { return IN_SUBS; } > void reset() > { > eliminated= FALSE; > @@ -717,7 +723,7 @@ class Item_allany_subselect :public Item > > void cleanup(); > // only ALL subquery has upper not > - subs_type substype() { return all?ALL_SUBS:ANY_SUBS; } > + subs_type substype() const { return all?ALL_SUBS:ANY_SUBS; } > bool select_transformer(JOIN *join); > void create_comp_func(bool invert) { func= func_creator(invert); } > virtual void print(String *str, enum_query_type query_type); > @@ -788,6 +794,7 @@ class subselect_engine: public Sql_alloc > virtual int exec()= 0; > virtual uint cols()= 0; /* return number of columns in select */ > virtual uint8 uncacheable()= 0; /* query is uncacheable */ > + virtual void make_cacheable() {} > enum Item_result type() { return res_type; } > enum Item_result cmptype() { return cmp_type; } > enum_field_types field_type() { return res_field_type; } > @@ -828,6 +835,7 @@ class subselect_single_select_engine: pu > int exec(); > uint cols(); > uint8 uncacheable(); > + void make_cacheable(); > void exclude(); > table_map upper_select_const_tables(); > virtual void print (String *str, enum_query_type query_type); > @@ -862,6 +870,7 @@ class subselect_union_engine: public sub > int exec(); > uint cols(); > uint8 uncacheable(); > + void make_cacheable(); > void exclude(); > table_map upper_select_const_tables(); > virtual void print (String *str, enum_query_type query_type); > > === modified file 'sql/opt_subselect.cc' > --- sql/opt_subselect.cc 2013-07-17 19:24:29 +0000 > +++ sql/opt_subselect.cc 2013-05-28 12:22:32 +0000 > @@ -3620,7 +3620,10 @@ bool setup_sj_materialization_part2(JOIN > emb_sj_nest->sj_subq_pred))) > DBUG_RETURN(TRUE); /* purecov: inspected */ > sjm_tab->type= JT_EQ_REF; > - sjm_tab->select_cond= sjm->in_equality; > + remove_sj_conds(&sjm_tab->select_cond); > + sjm_tab->select_cond= and_items(sjm_tab->select_cond, sjm->in_equality); > + if (!sjm_tab->select_cond->fixed) > + sjm_tab->select_cond->fix_fields(thd, &sjm_tab->select_cond); > } > else > { > @@ -4907,7 +4910,7 @@ static void remove_subq_pushed_predicate > > > /** > - Optimize all subqueries of a query that were not flattened into a semijoin. > + Optimize all non-IN subqueries that were not flattened into a semijoin. > > @details > Optimize all immediate children subqueries of a query. > @@ -4924,7 +4927,12 @@ static void remove_subq_pushed_predicate > > bool JOIN::optimize_unflattened_subqueries() > { > - return select_lex->optimize_unflattened_subqueries(false); > + return select_lex->optimize_subqueries(&Item_subselect::has_single_strategy); > +} > + > +bool JOIN::optimize_in_subqueries() > +{ > + return select_lex->optimize_subqueries(NULL); > } > > /** > @@ -4958,7 +4966,7 @@ bool JOIN::optimize_constant_subqueries( > not for EXPLAIN. > */ > select_lex->options&= ~SELECT_DESCRIBE; > - res= select_lex->optimize_unflattened_subqueries(true); > + res= select_lex->optimize_subqueries(&Item_subselect::const_item); > select_lex->options= save_options; > return res; > } > @@ -5242,9 +5250,13 @@ bool setup_jtbm_semi_joins(JOIN *join, L > { > DBUG_ASSERT(subq_pred->test_set_strategy(SUBS_MATERIALIZATION)); > subq_pred->is_jtbm_const_tab= FALSE; > + > + DBUG_ASSERT(subq_pred->engine->engine_type() == subselect_engine::SINGLE_SELECT_ENGINE); > + subq_pred->setup_mat_engine(); > + DBUG_ASSERT(subq_pred->engine->engine_type() == subselect_engine::HASH_SJ_ENGINE); > subselect_hash_sj_engine *hash_sj_engine= > - ((subselect_hash_sj_engine*)item->engine); > - > + ((subselect_hash_sj_engine*) subq_pred->engine); > + > table->table= hash_sj_engine->tmp_table; > table->table->pos_in_table_list= table; > > @@ -5464,27 +5476,6 @@ bool JOIN::choose_subquery_plan(table_ma > DBUG_PRINT("info",("outer_lookup_keys: %.2f", outer_lookup_keys)); > } > > - /* > - If (1) materialization is a possible strategy based on semantic analysis > - during the prepare phase, then if > - (2) it is more expensive than the IN->EXISTS transformation, and > - (3) it is not possible to create usable indexes for the materialization > - strategy, > - fall back to IN->EXISTS. > - otherwise > - use materialization. > - */ > - if (in_subs->test_strategy(SUBS_MATERIALIZATION) && > - in_subs->setup_mat_engine()) > - { > - /* > - If materialization was the cheaper or the only user-selected strategy, > - but it is not possible to execute it due to limitations in the > - implementation, fall back to IN-TO-EXISTS. > - */ > - in_subs->set_strategy(SUBS_IN_TO_EXISTS); > - } > - > if (in_subs->test_strategy(SUBS_MATERIALIZATION)) > { > /* Restore the original query plan used for materialization. */ > > === modified file 'sql/sql_class.h' > --- sql/sql_class.h 2013-09-13 16:14:56 +0000 > +++ sql/sql_class.h 2013-09-27 08:25:14 +0000 > @@ -508,6 +508,10 @@ typedef struct system_variables > ulong optimizer_search_depth; > ulong optimizer_selectivity_sampling_limit; > ulong optimizer_use_condition_selectivity; > + ha_rows dynamic_pushdown_max_evals; > + uint dynamic_pushdown_cost_ratio; > + uint dynamic_pushdown_sampling_probability; > + uint dynamic_pushdown_max_selectivity; > ulong use_stat_tables; > ulong histogram_size; > ulong histogram_type; > > === modified file 'sql/sql_delete.cc' > --- sql/sql_delete.cc 2013-08-06 20:33:18 +0000 > +++ sql/sql_delete.cc 2013-04-01 10:36:05 +0000 > @@ -126,7 +126,7 @@ bool mysql_delete(THD *thd, TABLE_LIST * > } > > /* Apply the IN=>EXISTS transformation to all subqueries and optimize them. */ > - if (select_lex->optimize_unflattened_subqueries(false)) > + if (select_lex->optimize_subqueries(NULL)) > DBUG_RETURN(TRUE); > > const_cond= (!conds || conds->const_item()); > > === modified file 'sql/sql_join_cache.cc' > --- sql/sql_join_cache.cc 2013-05-05 18:39:31 +0000 > +++ sql/sql_join_cache.cc 2013-09-27 08:25:14 +0000 > @@ -2080,6 +2080,7 @@ enum_nested_loop_state JOIN_CACHE::join_ > if (outer_join_first_inner && !join_tab->first_unmatched) > join_tab->not_null_compl= TRUE; > > + *(join->cur_exec_tab)= join_tab; psergey: Do you still remember why you had to add this? It would be nice to put in a comment. > if (!join_tab->first_unmatched) > { > /* Find all records from join_tab that match records from join buffer */ > > === modified file 'sql/sql_lex.cc' > --- sql/sql_lex.cc 2013-08-18 19:29:06 +0000 > +++ sql/sql_lex.cc 2013-04-01 10:36:05 +0000 > @@ -3461,7 +3461,7 @@ bool st_select_lex::add_index_hint (THD > @retval TRUE error occurred. > */ > > -bool st_select_lex::optimize_unflattened_subqueries(bool const_only) > +bool st_select_lex::optimize_subqueries(subselect_filter filter) > { > for (SELECT_LEX_UNIT *un= first_inner_unit(); un; un= un->next_unit()) > { > @@ -3476,11 +3476,8 @@ bool st_select_lex::optimize_unflattened > continue; > } > > - if (const_only && !subquery_predicate->const_item()) > - { > - /* Skip non-constant subqueries if the caller asked so. */ > + if (filter && !((subquery_predicate->*filter)())) > continue; > - } > > bool empty_union_result= true; > bool is_correlated_unit= false; > @@ -4204,7 +4201,9 @@ int st_select_lex::print_explain(select_ > bool *printed_anything) > { > int res; > - if (join && join->have_query_plan == JOIN::QEP_AVAILABLE) > + if (join && > + (join->have_query_plan == JOIN::QEP_EXPLAINABLE || > + join->have_query_plan == JOIN::QEP_EXECUTABLE)) > { > /* > There is a number of reasons join can be marked as degenerate, so all > > === modified file 'sql/sql_lex.h' > --- sql/sql_lex.h 2013-07-17 19:24:29 +0000 > +++ sql/sql_lex.h 2013-04-01 10:36:05 +0000 > @@ -1026,7 +1026,7 @@ class st_select_lex: public st_select_le > > void clear_index_hints(void) { index_hints= NULL; } > bool is_part_of_union() { return master_unit()->is_union(); } > - bool optimize_unflattened_subqueries(bool const_only); > + bool optimize_subqueries(subselect_filter filter); > /* Set the EXPLAIN type for this subquery. */ > void set_explain_type(bool on_the_fly); > bool handle_derived(LEX *lex, uint phases); > > === modified file 'sql/sql_priv.h' > --- sql/sql_priv.h 2013-07-17 19:24:29 +0000 > +++ sql/sql_priv.h 2013-09-27 08:25:14 +0000 > @@ -228,7 +228,8 @@ template <class T> bool valid_buffer_ran > #define OPTIMIZER_SWITCH_TABLE_ELIMINATION (1ULL << 26) > #define OPTIMIZER_SWITCH_EXTENDED_KEYS (1ULL << 27) > #define OPTIMIZER_SWITCH_EXISTS_TO_IN (1ULL << 28) > -#define OPTIMIZER_SWITCH_USE_CONDITION_SELECTIVITY (1ULL << 29) > +#define OPTIMIZER_SWITCH_EXPENSIVE_PRED_STATIC_PUSHDOWN (1ULL << 29) > +#define OPTIMIZER_SWITCH_EXPENSIVE_PRED_DYNAMIC_PUSHDOWN (1ULL << 30) > > #define OPTIMIZER_SWITCH_DEFAULT (OPTIMIZER_SWITCH_INDEX_MERGE | \ > OPTIMIZER_SWITCH_INDEX_MERGE_UNION | \ > @@ -250,7 +251,7 @@ template <class T> bool valid_buffer_ran > OPTIMIZER_SWITCH_SUBQUERY_CACHE | \ > OPTIMIZER_SWITCH_SEMIJOIN | \ > OPTIMIZER_SWITCH_FIRSTMATCH | \ > - OPTIMIZER_SWITCH_LOOSE_SCAN ) > + OPTIMIZER_SWITCH_LOOSE_SCAN) > /* > Replication uses 8 bytes to store SQL_MODE in the binary log. The day you > use strictly more than 64 bits by adding one more define above, you should > > === modified file 'sql/sql_select.cc' > --- sql/sql_select.cc 2013-08-20 11:48:29 +0000 > +++ sql/sql_select.cc 2013-09-27 13:22:36 +0000 > @@ -200,14 +200,12 @@ int join_read_always_key_or_null(JOIN_TA > int join_read_next_same_or_null(READ_RECORD *info); > static COND *make_cond_for_table(THD *thd, Item *cond,table_map table, > table_map used_table, > - int join_tab_idx_arg, > bool exclude_expensive_cond, > bool retain_ref_cond); > static COND *make_cond_for_table_from_pred(THD *thd, Item *root_cond, > Item *cond, > table_map tables, > table_map used_table, > - int join_tab_idx_arg, > bool exclude_expensive_cond, > bool retain_ref_cond); > > @@ -334,6 +332,50 @@ bool dbug_user_var_equals_int(THD *thd, > } > return FALSE; > } > + > + > +/** > + Print query execution statistics. > +*/ > + > +void print_execution_stats(JOIN *join) > +{ > + JOIN_TAB *first_tab, *tab; > + > + DBUG_LOCK_FILE; > + > + fprintf(DBUG_FILE, "\nexecution_statistics {\n"); > + > + first_tab= first_breadth_first_tab(join, WALK_OPTIMIZATION_TABS); > + for (tab= first_tab; tab; > + tab= next_breadth_first_tab(join, WALK_OPTIMIZATION_TABS, tab)) > + { > + if (tab->bush_root_tab) > + continue; /* TODO: how to process nested tabs?*/ > + fprintf(DBUG_FILE, "%s: actual_card: %f, est_card: %g, est_sel: %f\n", > + tab->table->alias.c_ptr(), > + tab->exec_partial_join_records, > + tab->partial_join_cardinality, > + tab->cond_selectivity); > + > + if (!tab->select_cond) > + continue; > + > + fprintf(DBUG_FILE, " dynamic_cond_stats:\n"); > + if (tab->select_cond->type() == Item::FUNC_ITEM && > + (((Item_func*)tab->select_cond)->functype() == Item_func::DYNAMIC_COND_FUNC || > + ((Item_func*)tab->select_cond)->functype() == Item_func::EXEC_STATS_FUNC)) > + { > + Item_func_collect_stats *stat_cond= (Item_func_collect_stats*) tab->select_cond; > + stat_cond->dbug_print_stats(); > + } > + } > + fprintf(DBUG_FILE, "}\n"); > + fprintf(DBUG_FILE, "\n"); > + > + DBUG_UNLOCK_FILE; > +} > + > #endif > > > @@ -1013,9 +1055,16 @@ int JOIN::optimize() > short-circuit because optimized==TRUE. > */ > if (!res && have_query_plan != QEP_DELETED) > - have_query_plan= QEP_AVAILABLE; > + { > + if (have_query_plan == QEP_INCOMPLETE) > + have_query_plan= QEP_EXPLAINABLE; > + else > + have_query_plan= QEP_EXECUTABLE; > + } > return res; > } > + > + > /** > global select optimisation. > > @@ -1046,6 +1095,7 @@ JOIN::optimize_inner() > > set_allowed_join_cache_types(); > need_distinct= TRUE; > + examined_rows_estimate= -1; > > /* Run optimize phase for all derived tables/views used in this SELECT. */ > if (select_lex->handle_derived(thd->lex, DT_OPTIMIZE)) > @@ -1282,8 +1332,7 @@ TODO: make view to decide if it is possi > if (conds && !(thd->lex->describe & DESCRIBE_EXTENDED)) > { > COND *table_independent_conds= > - make_cond_for_table(thd, conds, PSEUDO_TABLE_BITS, 0, -1, > - FALSE, FALSE); > + make_cond_for_table(thd, conds, PSEUDO_TABLE_BITS, 0, FALSE, FALSE); > DBUG_EXECUTE("where", > print_where(table_independent_conds, > "where after opt_sum_query()", > @@ -1296,6 +1345,11 @@ TODO: make view to decide if it is possi > { > DBUG_PRINT("info",("No tables")); > error= 0; > + /* > + Even if there are no tables the query has some small cost. Use the default > + cost of all 'cheap' items. > + */ > + best_read= (1 / (double)TIME_FOR_COMPARE); > goto setup_subq_exit; > } > error= -1; // Error is sent to client > @@ -1730,12 +1784,18 @@ TODO: make view to decide if it is possi > if (!(select_options & SELECT_DESCRIBE)) > init_ftfuncs(thd, select_lex, test(order)); > > - if (optimize_unflattened_subqueries()) > + if (optimize_in_subqueries()) > DBUG_RETURN(1); > > int res; > if ((res= rewrite_to_index_subquery_engine(this)) != -1) > + { > + /* > + This subquery has been rewritten to direct index access via handler calls > + instead of going through JOIN::exec. > + */ > DBUG_RETURN(res); > + } > if (setup_subquery_caches()) > DBUG_RETURN(-1); > > @@ -1863,7 +1923,7 @@ TODO: make view to decide if it is possi > Even with zero matching rows, subqueries in the HAVING clause may > need to be evaluated if there are aggregate functions in the query. > */ > - if (optimize_unflattened_subqueries()) > + if (optimize_unflattened_subqueries() || optimize_in_subqueries()) > DBUG_RETURN(1); > error= 0; > > @@ -2015,6 +2075,10 @@ int JOIN::init_execution() > DBUG_RETURN(-1); /* purecov: inspected */ > } > > + > + if (setup_dynamic_conditions()) > + DBUG_RETURN(1); > + > DBUG_RETURN(0); > } > > @@ -2845,8 +2909,7 @@ void JOIN::exec_inner() > curr_table->table->map); > > Item* sort_table_cond= make_cond_for_table(thd, curr_join->tmp_having, > - used_tables, > - (table_map)0, -1, > + used_tables, (table_map)0, > FALSE, FALSE); > if (sort_table_cond) > { > @@ -2886,7 +2949,7 @@ void JOIN::exec_inner() > QT_ORDINARY);); > curr_join->tmp_having= make_cond_for_table(thd, curr_join->tmp_having, > ~ (table_map) 0, > - ~used_tables, -1, > + ~used_tables, > FALSE, FALSE); > DBUG_EXECUTE("where",print_where(curr_join->tmp_having, > "having after sort", > @@ -3331,6 +3394,7 @@ make_join_statistics(JOIN *join, List<TA > */ > join->best_positions= (POSITION*) join->thd->alloc(sizeof(POSITION)* > (table_count +1)); > + join->cur_exec_tab= (JOIN_TAB**) join->thd->calloc(sizeof(JOIN_TAB*)); > > if (join->thd->is_fatal_error) > DBUG_RETURN(1); // Eom /* purecov: inspected */ > @@ -3884,6 +3948,9 @@ make_join_statistics(JOIN *join, List<TA > > if (join->const_tables != join->table_count) > optimize_keyuse(join, keyuse_array); > + > + if (join->optimize_unflattened_subqueries()) > + DBUG_RETURN(1); > > if (optimize_semijoin_nests(join, all_table_map)) > DBUG_RETURN(TRUE); /* purecov: inspected */ > @@ -6879,28 +6946,57 @@ void JOIN::get_prefix_cost_and_fanout(ui > > > /** > + Compute the total cost of all subqueries of a (sub)query. > + > + @return the cost of all subqueries > +*/ > + > +double JOIN::total_subquery_cost() psergey: I think this function needs to be called get_total_subquery() cost, to follow the pattern set by other similar functions. When I read this function, I don't understand why it doesn't account for the fact that a subquery might be evaluated multiple times. Please add a comment about that. > +{ > + double total_cost= 0; > + for (SELECT_LEX_UNIT *un= select_lex->first_inner_unit(); > + un; un= un->next_unit()) > + { > + Item_subselect *subquery_predicate= un->item; > + if (!subquery_predicate) > + continue; > + > + total_cost+= subquery_predicate->get_cost(); > + } > + return total_cost; > +} > + > + > +/** > Estimate the number of rows that query execution will read. > > @todo This is a very pessimistic upper bound. Use join selectivity > when available to produce a more realistic number. > */ > > -double JOIN::get_examined_rows() > +double JOIN::get_examined_rows_estimate() > { > - ha_rows examined_rows; > - double prev_fanout= 1; > - JOIN_TAB *tab= first_breadth_first_tab(this, WALK_OPTIMIZATION_TABS); > - JOIN_TAB *prev_tab= tab; > + if (examined_rows_estimate >= 0) > + return examined_rows_estimate; > + else if (!(have_query_plan == QEP_EXPLAINABLE || > + have_query_plan == QEP_EXECUTABLE)) > + return -1; > + else > + { > + double prev_fanout= 1; > + JOIN_TAB *tab= first_breadth_first_tab(this, WALK_OPTIMIZATION_TABS); > + JOIN_TAB *prev_tab= tab; > > - examined_rows= tab->get_examined_rows(); > + examined_rows_estimate= tab->get_examined_rows_estimate(); > > - while ((tab= next_breadth_first_tab(this, WALK_OPTIMIZATION_TABS, tab))) > - { > - prev_fanout *= prev_tab->records_read; > - examined_rows+= (ha_rows) (tab->get_examined_rows() * prev_fanout); > - prev_tab= tab; > + while ((tab= next_breadth_first_tab(this, WALK_OPTIMIZATION_TABS, tab))) > + { > + prev_fanout *= prev_tab->records_read; > + examined_rows_estimate+= tab->get_examined_rows_estimate() * prev_fanout; > + prev_tab= tab; > + } > + return examined_rows_estimate; > } > - return examined_rows; > } > > > @@ -7374,15 +7470,15 @@ best_extension_by_limited_search(JOIN > remaining_tables & > ~real_table_bit); > join->positions[idx].cond_selectivity= pushdown_cond_selectivity; > - double partial_join_cardinality= current_record_count * > - pushdown_cond_selectivity; > + join->positions[idx].partial_join_cardinality= current_record_count * > + pushdown_cond_selectivity; > if ( (search_depth > 1) && (remaining_tables & ~real_table_bit) & allowed_tables ) > { /* Recursively expand the current partial plan */ > swap_variables(JOIN_TAB*, join->best_ref[idx], *pos); > if (best_extension_by_limited_search(join, > remaining_tables & ~real_table_bit, > idx + 1, > - partial_join_cardinality, > + join->positions[idx].partial_join_cardinality, > current_read_time, > search_depth - 1, > prune_level, > @@ -7400,13 +7496,68 @@ best_extension_by_limited_search(JOIN > join->positions[join->const_tables].table->table) > /* We have to make a temp table */ > current_read_time+= current_record_count; > + > + /* > + For each subquery find the partial join with the least cardinality > + in order to minimize the number of subquery evaluations. > + */ psergey: Please move this new piece of code into a separate function. It does one thing, its action can be easily separated from what the rest of the code in this function does, so it should be moved away. > + if (optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_STATIC_PUSHDOWN) && > + join->conds && join->conds->with_subselect > + /* @todo This is not a reliable estimate: > + && join->conds->is_expensive()*/) > + { > + bool may_benefit_from_static_pushdown= false; > + /* > + Find and store the POSITION with the least partial result cardinality > + between the current and last POSITION in the query plan. > + */ psergey: Does the loop below do what the comment says? AFAIU, the code sets "least_card_pos" for every position in the join->positions[] array. > + POSITION *min_pos= NULL; > + for (POSITION *p_pos= join->positions + idx, > + *p_end= join->positions + join->const_tables; > + p_pos >= p_end ; > + p_pos--) > + { > + if (!min_pos || > + p_pos->partial_join_cardinality <= min_pos->partial_join_cardinality) > + min_pos= p_pos; > + p_pos->least_card_pos= min_pos; > + if (p_pos->least_card_pos != p_pos) > + may_benefit_from_static_pushdown= true; > + } > + > + /* > + Make expensive predicates dependent on the table that is in the > + partial join with the least cardinality after the left-most join > + where the predicate can be pushed. > + */ psergey: ^ out of date comment? > + if (may_benefit_from_static_pushdown) > + { > + if (is_cond_and(join->conds)) > + { psergey: I find it weird that every time you need to analyze the WHERE condition for expensive items. I would have expected that you would - collect a list of AND-parts of the WHERE that can be moved - do some pre-processing (like, fetch their cost) and after that run join optimization that would use this data. > + List_iterator<Item> and_parts_it(*((Item_cond*) join->conds)->argument_list()); > + Item *and_part; > + while ((and_part= and_parts_it++)) > + { > + if (!and_part->with_subselect > + /* @todo This is not a reliable estimate: > + !and_part->is_expensive()*/) psergey: There are a lot of places in the code that check a condition whether a given Item object should be processed by cost-based condition pushdown. This means, the check should be factored out into a function. > + continue; > + current_read_time+= join->static_pushdown_cost(and_part, idx); > + } > + } > + else > + current_read_time+= join->static_pushdown_cost(join->conds, idx); > + } > + } > + > if (current_read_time < join->best_read) > { > memcpy((uchar*) join->best_positions, (uchar*) join->positions, > sizeof(POSITION) * (idx + 1)); > - join->record_count= partial_join_cardinality; > + join->record_count= join->positions[idx].partial_join_cardinality; > join->best_read= current_read_time - 0.001; > } > + > DBUG_EXECUTE("opt", print_plan(join, idx+1, > current_record_count, > read_time, > @@ -7605,7 +7756,7 @@ int JOIN_TAB::make_scan_filter() > if (cond && > (tmp= make_cond_for_table(join->thd, cond, > join->const_table_map | table->map, > - table->map, -1, FALSE, TRUE))) > + table->map, FALSE, TRUE))) > { > DBUG_EXECUTE("where",print_where(tmp,"cache", QT_ORDINARY);); > if (!(cache_select= > @@ -8048,7 +8199,8 @@ get_best_combination(JOIN *join) > sub-order > */ > SJ_MATERIALIZATION_INFO *sjm= cur_pos->table->emb_sj_nest->sj_mat_info; > - j->records= j->records_read= (ha_rows)(sjm->is_sj_scan? sjm->rows : 1); > + j->records_read= sjm->is_sj_scan? sjm->rows : 1; > + j->records= (ha_rows) j->records_read; > j->cond_selectivity= 1.0; > JOIN_TAB *jt; > JOIN_TAB_RANGE *jt_range; > @@ -8112,7 +8264,7 @@ get_best_combination(JOIN *join) > Save records_read in JOIN_TAB so that select_describe()/etc don't have > to access join->best_positions[]. > */ > - j->records_read= (ha_rows)join->best_positions[tablenr].records_read; > + j->records_read= join->best_positions[tablenr].records_read; > j->cond_selectivity= join->best_positions[tablenr].cond_selectivity; > join->map2table[j->table->tablenr]= j; > > @@ -8614,6 +8766,7 @@ JOIN::make_simple_join(JOIN *parent, TAB > bzero((char*) &join_tab->read_record,sizeof(join_tab->read_record)); > temp_table->status=0; > temp_table->null_row=0; > + *cur_exec_tab= *(parent->cur_exec_tab); > DBUG_RETURN(FALSE); > } > > @@ -8969,7 +9122,7 @@ make_join_select(JOIN *join,SQL_SELECT * > join->exec_const_cond= > make_cond_for_table(thd, cond, > join->const_table_map, > - (table_map) 0, -1, FALSE, FALSE); > + (table_map) 0, FALSE, FALSE); > /* Add conditions added by add_not_null_conds(). */ > for (uint i= 0 ; i < join->const_tables ; i++) > add_cond_and_fix(thd, &join->exec_const_cond, > @@ -8991,7 +9144,7 @@ make_join_select(JOIN *join,SQL_SELECT * > join->const_table_map | > OUTER_REF_TABLE_BIT, > OUTER_REF_TABLE_BIT, > - -1, FALSE, FALSE); > + FALSE, FALSE); > if (outer_ref_cond) > { > add_cond_and_fix(thd, &outer_ref_cond, join->outer_ref_cond); > @@ -9005,7 +9158,7 @@ make_join_select(JOIN *join,SQL_SELECT * > join->const_table_map | > PSEUDO_TABLE_BITS, > PSEUDO_TABLE_BITS, > - -1, FALSE, FALSE); > + FALSE, FALSE); > if (pseudo_bits_cond) > { > add_cond_and_fix(thd, &pseudo_bits_cond, > @@ -9102,8 +9255,10 @@ make_join_select(JOIN *join,SQL_SELECT * > } > else > { > - tmp= make_cond_for_table(thd, cond, used_tables, current_map, i, > + tmp= make_cond_for_table(thd, cond, used_tables, current_map, > FALSE, FALSE); > + if (tmp) > + tmp->walk(&Item::set_join_tab_idx_processor, 0, (uchar*) &i); > } > /* Add conditions added by add_not_null_conds(). */ > if (tab->select_cond) > @@ -9184,7 +9339,7 @@ make_join_select(JOIN *join,SQL_SELECT * > { > COND *push_cond= > make_cond_for_table(thd, tmp, current_map, current_map, > - -1, FALSE, FALSE); > + FALSE, FALSE); > if (push_cond) > { > /* Push condition to handler */ > @@ -9357,7 +9512,7 @@ make_join_select(JOIN *join,SQL_SELECT * > JOIN_TAB *cond_tab= join_tab->first_inner; > COND *tmp= make_cond_for_table(thd, *join_tab->on_expr_ref, > join->const_table_map, > - (table_map) 0, -1, FALSE, FALSE); > + (table_map) 0, FALSE, FALSE); > if (!tmp) > continue; > tmp= new Item_func_trig_cond(tmp, &cond_tab->not_null_compl); > @@ -9403,11 +9558,11 @@ make_join_select(JOIN *join,SQL_SELECT * > current_map= tab->table->map; > used_tables2|= current_map; > /* > - psergey: have put the -1 below. It's bad, will need to fix it. > + psergey: After pushdown, set the join_tab_idx of tmp_cond to the > + right join tab (tab - first_tab)? > */ > COND *tmp_cond= make_cond_for_table(thd, on_expr, used_tables2, > - current_map, /*(tab - first_tab)*/ -1, > - FALSE, FALSE); > + current_map, FALSE, FALSE); > bool is_sjm_lookup_tab= FALSE; > if (tab->bush_children) > { > @@ -10198,6 +10353,8 @@ uint check_join_cache_usage(JOIN_TAB *ta > if ((tab->cache= new JOIN_CACHE_BNL(join, tab, prev_cache)) && > ((options & SELECT_DESCRIBE) || !tab->cache->init())) > { > + if (options & SELECT_DESCRIBE) > + join->have_query_plan= JOIN::QEP_INCOMPLETE; > tab->icp_other_tables_ok= FALSE; > return (2-test(!prev_cache)); > } > @@ -10233,6 +10390,8 @@ uint check_join_cache_usage(JOIN_TAB *ta > if ((tab->cache= new JOIN_CACHE_BNLH(join, tab, prev_cache)) && > ((options & SELECT_DESCRIBE) || !tab->cache->init())) > { > + if (options & SELECT_DESCRIBE) > + join->have_query_plan= JOIN::QEP_INCOMPLETE; > tab->icp_other_tables_ok= FALSE; > return (4-test(!prev_cache)); > } > @@ -10253,7 +10412,11 @@ uint check_join_cache_usage(JOIN_TAB *ta > prev_cache= 0; > if ((tab->cache= new JOIN_CACHE_BKA(join, tab, flags, prev_cache)) && > ((options & SELECT_DESCRIBE) || !tab->cache->init())) > + { > + if (options & SELECT_DESCRIBE) > + join->have_query_plan= JOIN::QEP_INCOMPLETE; > return (6-test(!prev_cache)); > + } > goto no_join_cache; > } > else > @@ -10263,7 +10426,9 @@ uint check_join_cache_usage(JOIN_TAB *ta > if ((tab->cache= new JOIN_CACHE_BKAH(join, tab, flags, prev_cache)) && > ((options & SELECT_DESCRIBE) || !tab->cache->init())) > { > - tab->idx_cond_fact_out= FALSE; > + if (options & SELECT_DESCRIBE) > + join->have_query_plan= JOIN::QEP_INCOMPLETE; > + tab->idx_cond_fact_out= FALSE; > return (8-test(!prev_cache)); > } > goto no_join_cache; > @@ -10422,10 +10587,9 @@ make_join_readinfo(JOIN *join, ulonglong > for (tab= join->join_tab; tab != join->join_tab + join->const_tables; tab++) > tab->partial_join_cardinality= 1; > > - JOIN_TAB *prev_tab= NULL; > for (tab= first_linear_tab(join, WITHOUT_CONST_TABLES), i= join->const_tables; > tab; > - prev_tab=tab, tab= next_linear_tab(join, tab, WITH_BUSH_ROOTS)) > + tab= next_linear_tab(join, tab, WITH_BUSH_ROOTS)) > { > /* > The approximation below for partial join cardinality is not good because > @@ -10435,12 +10599,9 @@ make_join_readinfo(JOIN *join, ulonglong > Later it should be improved. > */ > > - if (tab->bush_root_tab && tab->bush_root_tab->bush_children->start == tab) > - prev_tab= NULL; > DBUG_ASSERT(tab->bush_children || tab->table == join->best_positions[i].table->table); > > - tab->partial_join_cardinality= join->best_positions[i].records_read * > - (prev_tab? prev_tab->partial_join_cardinality : 1); > + tab->partial_join_cardinality= join->best_positions[i].partial_join_cardinality; > if (!tab->bush_children) > i++; > } > @@ -10807,9 +10968,9 @@ double JOIN_TAB::scan_time() > @todo: why not use JOIN_TAB::found_records > */ > > -ha_rows JOIN_TAB::get_examined_rows() > +double JOIN_TAB::get_examined_rows_estimate() > { > - ha_rows examined_rows; > + double examined_rows; > > if (select && select->quick && use_quick != 2) > examined_rows= select->quick->records; > @@ -10839,7 +11000,7 @@ ha_rows JOIN_TAB::get_examined_rows() > } > } > else > - examined_rows= (ha_rows) records_read; > + examined_rows= records_read; > > return examined_rows; > } > @@ -16457,6 +16618,13 @@ do_select(JOIN *join,List<Item> *fields, > sufficient to check only the condition pseudo_bits_cond. > */ > DBUG_ASSERT(join->outer_ref_cond == NULL); > + DBUG_EXECUTE_IF("show_explain_probe_do_select_const", > + if (dbug_user_var_equals_int(join->thd, > + "show_explain_probe_select_id", > + join->select_lex->select_number)) > + dbug_serve_apcs(join->thd, 1); > + ); > + > if (!join->pseudo_bits_cond || join->pseudo_bits_cond->val_int()) > { > error= (*end_select)(join, 0, 0); > @@ -16512,6 +16680,23 @@ do_select(JOIN *join,List<Item> *fields, > if (error == NESTED_LOOP_NO_MORE_ROWS || join->thd->killed == ABORT_QUERY) > error= NESTED_LOOP_OK; > > + DBUG_EXECUTE("counts", > + join->is_top_level_join() ? > + print_execution_stats(join) : (void)0;); > + /* Reset execution-time counters. */ > + if (join->table_count) > + { > + JOIN_TAB *first_tab, *tab; > + first_tab= first_breadth_first_tab(join, WALK_OPTIMIZATION_TABS); > + for (tab= first_tab; tab; > + tab= next_breadth_first_tab(join, WALK_OPTIMIZATION_TABS, tab)) > + { > + if (tab->bush_root_tab) > + continue; /* TODO: how to process nested tabs?*/ > + tab->exec_partial_join_records= 0; > + } > + } > + > if (table) > { > int tmp, new_errno= 0; > @@ -16927,6 +17112,8 @@ evaluate_join_record(JOIN *join, JOIN_TA > if (join_tab->table->vfield) > update_virtual_fields(join->thd, join_tab->table); > > + *(join->cur_exec_tab)= join_tab; > + > if (select_cond) > { > select_cond_result= test(select_cond->val_int()); > @@ -17053,6 +17240,7 @@ evaluate_join_record(JOIN *join, JOIN_TA > if (found) > { > enum enum_nested_loop_state rc; > + join_tab->exec_partial_join_records++; > /* A match from join_tab is found for the current partial join. */ > rc= (*join_tab->next_select)(join, join_tab+1, 0); > join->thd->warning_info->inc_current_row_for_warning(); > @@ -17106,6 +17294,7 @@ evaluate_null_complemented_join_record(J > and no matches has been found for the current outer row. > */ > JOIN_TAB *last_inner_tab= join_tab->last_inner; > + *(join->cur_exec_tab)= join_tab; > /* Cache variables for faster loop */ > COND *select_cond; > for ( ; join_tab <= last_inner_tab ; join_tab++) > @@ -17169,6 +17358,8 @@ evaluate_null_complemented_join_record(J > join->return_tab= join_tab->do_firstmatch; > } > > + join_tab->exec_partial_join_records++; > + > /* > Send the row complemented by nulls to be joined with the > remaining tables. > @@ -18536,21 +18727,34 @@ bool test_if_ref(Item *root_cond, Item_f > static Item * > make_cond_for_table(THD *thd, Item *cond, table_map tables, > table_map used_table, > - int join_tab_idx_arg, > bool exclude_expensive_cond __attribute__((unused)), > bool retain_ref_cond) > { > return make_cond_for_table_from_pred(thd, cond, cond, tables, used_table, > - join_tab_idx_arg, > exclude_expensive_cond, > retain_ref_cond); > } > > > +static int compare_items_by_cost(Item* it1, Item* it2, void *arg) > +{ > + double cost1= 0, cost2= 0; > + > + it1->walk(&Item::sum_cost_processor, false, (uchar*) &cost1); > + it2->walk(&Item::sum_cost_processor, false, (uchar*) &cost2); > + > + if (cost1 == cost2) > + return 0; > + if (cost1 > cost2) > + return -1; > + else > + return 1; > +} > + > + > static Item * > make_cond_for_table_from_pred(THD *thd, Item *root_cond, Item *cond, > table_map tables, table_map used_table, > - int join_tab_idx_arg, > bool exclude_expensive_cond __attribute__ > ((unused)), > bool retain_ref_cond) > @@ -18573,7 +18777,6 @@ make_cond_for_table_from_pred(THD *thd, > { > Item *fix=make_cond_for_table_from_pred(thd, root_cond, item, > tables, used_table, > - join_tab_idx_arg, > exclude_expensive_cond, > retain_ref_cond); > if (fix) > @@ -18594,6 +18797,11 @@ make_cond_for_table_from_pred(THD *thd, > new_cond->used_tables_cache= > ((Item_cond_and*) cond)->used_tables_cache & > tables; > + /* > + Move expensive conditions to the end of the AND list to reduce the > + number of times they are evaluated. > + */ > + bubble_sort<Item>(new_cond->argument_list(), compare_items_by_cost, NULL); > return new_cond; > } > } > @@ -18608,7 +18816,6 @@ make_cond_for_table_from_pred(THD *thd, > { > Item *fix=make_cond_for_table_from_pred(thd, root_cond, item, > tables, 0L, > - join_tab_idx_arg, > exclude_expensive_cond, > retain_ref_cond); > if (!fix) > @@ -18623,6 +18830,11 @@ make_cond_for_table_from_pred(THD *thd, > new_cond->fix_fields(thd, 0); > new_cond->used_tables_cache= ((Item_cond_or*) cond)->used_tables_cache; > new_cond->top_level_item(); > + /* > + Move expensive conditions to the end of the AND list to reduce the > + number of times they are evaluated. > + */ > + bubble_sort<Item>(new_cond->argument_list(), compare_items_by_cost, NULL); > return new_cond; > } > } > @@ -18638,7 +18850,6 @@ make_cond_for_table_from_pred(THD *thd, > > if (cond->marker == 2 || cond->eq_cmp_result() == Item::COND_OK) > { > - cond->set_join_tab_idx(join_tab_idx_arg); > return cond; // Not boolean op > } > > @@ -18661,7 +18872,6 @@ make_cond_for_table_from_pred(THD *thd, > } > } > cond->marker=2; > - cond->set_join_tab_idx(join_tab_idx_arg); > return cond; > } > > @@ -19688,6 +19898,7 @@ create_sort_index(THD *thd, JOIN *join, > tab= join->join_tab + join->const_tables; > table= tab->table; > select= tab->select; > + *(join->cur_exec_tab)= tab; > > JOIN_TAB *save_pre_sort_join_tab= NULL; > if (join->pre_sort_join_tab) > @@ -22287,7 +22498,8 @@ int JOIN::print_explain(select_result_si > DBUG_PRINT("info", ("Select 0x%lx, type %s, message %s", > (ulong)join->select_lex, join->select_lex->type, > message ? message : "NULL")); > - DBUG_ASSERT(on_the_fly? have_query_plan == QEP_AVAILABLE: TRUE); > + DBUG_ASSERT(on_the_fly? (have_query_plan == QEP_EXPLAINABLE || > + have_query_plan == QEP_EXECUTABLE) : TRUE); > /* Don't log this into the slow query log */ > > if (!on_the_fly) > @@ -22326,6 +22538,9 @@ int JOIN::print_explain(select_result_si > uint select_id= join->select_lex->select_number; > JOIN_TAB* const first_top_tab= first_breadth_first_tab(join, WALK_OPTIMIZATION_TABS); > > + if (!join->initialized && join->setup_dynamic_conditions()) > + return 1; > + > for (JOIN_TAB *tab= first_breadth_first_tab(join, WALK_OPTIMIZATION_TABS); tab; > tab= next_breadth_first_tab(join, WALK_OPTIMIZATION_TABS, tab)) > { > @@ -22583,7 +22798,7 @@ int JOIN::print_explain(select_result_si > } > else > { > - ha_rows examined_rows= tab->get_examined_rows(); > + double examined_rows= tab->get_examined_rows_estimate(); > > item_list.push_back(new Item_int((longlong) (ulonglong) examined_rows, > MY_INT64_NUM_DECIMAL_DIGITS)); > @@ -22596,7 +22811,7 @@ int JOIN::print_explain(select_result_si > { > double pushdown_cond_selectivity= tab->cond_selectivity; > if (pushdown_cond_selectivity == 1.0) > - f= (float) (100.0 * tab->records_read / examined_rows); > + f= (100.0 * (float)tab->records_read) / examined_rows; > else > f= (float) (100.0 * pushdown_cond_selectivity); > } > @@ -22687,6 +22902,25 @@ int JOIN::print_explain(select_result_si > } > else > extra.append(STRING_WITH_LEN("; Using where")); > + /* Add the id's of the subselects to the table they are pushed to. */ > + if (explain_flags & DESCRIBE_EXTENDED) > + { > + List<Item_subselect> subs_list; > + *cur_exec_tab= tab; > + tab->select_cond->walk(&Item::collect_item_subselect_processor, 0, > + (uchar*) &subs_list); > + if (subs_list.elements) > + { > + List_iterator_fast<Item_subselect> subs_it(subs_list); > + Item_subselect *subs; > + extra.append(STRING_WITH_LEN("; Subqueries:")); > + while ((subs= subs_it++)) > + { > + extra.append(STRING_WITH_LEN(" ")); > + extra.append_ulonglong(subs->get_select_lex()->select_number); > + } > + } > + } > } > } > if (table_list /* SJM bushes don't have table_list */ && > @@ -23652,6 +23886,204 @@ void JOIN::cache_const_exprs() > } > > > +double JOIN::static_pushdown_cost(Item *pushed_cond, uint idx) > +{ psergey: This function needs a comment. I see that it modifies position opt_pos->read_time. What is the function that undoes this change? > + table_map cond_tables= pushed_cond->used_tables(); > + POSITION *last_pos= NULL; > + POSITION *opt_pos; > + double pushed_cond_cost; > + > + DBUG_ASSERT(pushed_cond->with_subselect > + /*pushed_cond->is_expensive()*/); > + > + if (!cond_tables || cond_tables == OUTER_REF_TABLE_BIT || idx <= const_tables) > + return 0; > + > + /* > + Find the last join_tab on which the subquery depends. This is the > + earliest join_tab where the condition can be pushed. > + */ > + for (POSITION *p_pos= positions + idx, *p_end= positions + const_tables; > + p_pos >= p_end ; p_pos--) > + { > + if (p_pos->table->table->map & cond_tables) > + { > + last_pos= p_pos; > + break; > + } > + } > + if (!last_pos) > + return 0; // pushed_cond depends only on constant tables > + > + opt_pos= last_pos->least_card_pos; > + > + /* > + Update the cost of the POSITION where the subquery is pushed to, > + and the total cost of the query plan. > + */ > + pushed_cond_cost= opt_pos->partial_join_cardinality * pushed_cond->get_cost(); > + opt_pos->read_time+= pushed_cond_cost; > + > + return pushed_cond_cost; > +} > + > + > +/** > + Make expensive subqueries dynamically pushdownable. > +*/ > + > +bool JOIN::setup_dynamic_conditions() > +{ > + List<Item_func_dynamic_cond> prev_dynamic_conds; > + List<Item_func_dynamic_cond> cur_dynamic_conds; > + bool do_static_pushdown= false, do_dynamic_pushdown= false; > + JOIN_TAB *first_tab, *last_tab, *min_tab, *tab; > + > + if (!(optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_STATIC_PUSHDOWN) || > + optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_DYNAMIC_PUSHDOWN)) || !conds || > + (table_count - const_tables < 2)) > + return false; > + > + first_tab= join_tab + const_tables; > + last_tab= join_tab + top_join_tab_count; > + > + /* > + Update the least cardinality join_tab for each tab for the chosen join order. > + Check if there are any conditions at all that need to be made dynamic. > + */ > + min_tab= NULL; > + for (tab= last_tab - 1; tab >= first_tab; tab--) > + { > + if (optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_STATIC_PUSHDOWN)) psergey: Is there any reason to have this check inside the loop? Please move it outside. > + { > + if (!min_tab || > + tab->partial_join_cardinality <= min_tab->partial_join_cardinality) > + min_tab= tab; > + tab->least_card_tab= min_tab; > + /* > + At least one join_tab must have lower cardinality than its preceeding > + join_tabs in order to use static pushdown. > + */ > + if (tab != tab->least_card_tab) > + do_static_pushdown= true; > + } > + else > + { > + /* Only dynamic pushdown, push conditions to the left-most possible tab. */ > + tab->least_card_tab= tab; > + } > + > + if (optimizer_flag(thd, OPTIMIZER_SWITCH_EXPENSIVE_PRED_DYNAMIC_PUSHDOWN) && > + tab->select_cond && tab->select_cond->needs_dynamic_cond()) > + do_dynamic_pushdown= true; > + } > + > + if (!do_static_pushdown && !do_dynamic_pushdown) > + return false; > + > + /* > + TODO: the following loop processes only the top-level JOIN_TABs. The loop > + skips semi-join nests. Each semi-join nest must be processed in the same > + way as the top-level joins. > + */ > + for (tab= first_tab; tab < last_tab; tab++) > + { > + Item_func_collect_stats *wrapped_cond; > + > + if (tab->select_cond) > + { > + if (is_cond_and(tab->select_cond)) > + { > + List_iterator<Item> and_parts_it(*((Item_cond*) tab->select_cond)->argument_list()); > + Item *and_part; > + while ((and_part= and_parts_it++)) > + { > + if (and_part->needs_dynamic_cond()) > + { > + /* > + TODO: > + If the leftmost join_tab where the dynamic condion can be pushed > + is in fact the last join table, do not wrap the condition since > + it has nowhere to move. > + */ > + if (!(wrapped_cond= new Item_func_dynamic_cond(and_part, > + tab->least_card_tab, > + cur_exec_tab))) > + return true; > + cur_dynamic_conds.push_back((Item_func_dynamic_cond*)wrapped_cond); > + } > + else > + { > + if (!(wrapped_cond= new Item_func_collect_stats(and_part))) > + return true; > + } > + and_parts_it.replace(wrapped_cond); > + } > + } > + else if (tab->select_cond->needs_dynamic_cond()) > + { > + /* > + TODO: > + If the leftmost join_tab where the dynamic condion can be pushed > + is in fact the last join table, do not wrap the condition since > + it has nowhere to move. > + */ > + if (!(wrapped_cond= new Item_func_dynamic_cond(tab->select_cond, > + tab->least_card_tab, > + cur_exec_tab))) > + return true; > + tab->set_select_cond(wrapped_cond, __LINE__); > + cur_dynamic_conds.push_back((Item_func_dynamic_cond*)wrapped_cond); > + } > + else > + { > + if (!(wrapped_cond= new Item_func_collect_stats(tab->select_cond))) > + return true; > + tab->set_select_cond(wrapped_cond, __LINE__); > + } > + } > + > + /* > + Add all dynamic conditions pushed to previous JOIN_TABs also to the > + current JOIN_TAB. This allows to "move" a dynamic condition > + from one tab to another by enabling it for the corrseponding tab. > + */ > + List_iterator_fast<Item_func_dynamic_cond> dyn_cond_it(prev_dynamic_conds); > + while ((wrapped_cond= dyn_cond_it++)) > + { > + if (is_cond_and(tab->select_cond)) > + ((Item_cond_and*)tab->select_cond)->add(wrapped_cond); > + else > + { > + tab->set_select_cond(and_items(tab->select_cond, wrapped_cond), __LINE__); > + tab->select_cond->quick_fix_field(); > + } > + } > + prev_dynamic_conds.concat(&cur_dynamic_conds); > + cur_dynamic_conds.empty(); > + > + /* > + If the top-level condition attached to this JOIN_TAB is not wrapped in a > + condition that can collect statistics, wrap it into one. > + */ > + if (tab->select_cond && > + !(tab->select_cond->type() == Item::FUNC_ITEM && > + (((Item_func*)tab->select_cond)->functype() == Item_func::DYNAMIC_COND_FUNC || > + ((Item_func*)tab->select_cond)->functype() == Item_func::EXEC_STATS_FUNC))) > + { > + if (!(wrapped_cond= new Item_func_collect_stats(tab->select_cond))) > + return true; > + tab->set_select_cond(wrapped_cond, __LINE__); > + } > + > + if (tab->select_cond) > + tab->select_cond->quick_fix_field(); > + } > + > + return false; > +} > + > + > /** > Find a cheaper access key than a given @a key > > > === modified file 'sql/sql_select.h' > --- sql/sql_select.h 2013-07-17 19:24:29 +0000 > +++ sql/sql_select.h 2013-09-27 08:25:14 +0000 > @@ -289,8 +289,8 @@ typedef struct st_join_table { > */ > double read_time; > > - /* psergey-todo: make the below have type double, like POSITION::records_read? */ > - ha_rows records_read; > + /* Copy of POSITION::records_read, set by get_best_combination() */ > + double records_read; > > /* The selectivity of the conditions that can be pushed to the table */ > double cond_selectivity; > @@ -300,6 +300,12 @@ typedef struct st_join_table { > > double partial_join_cardinality; > > + /* Execution-time counters of actual cardinality and selectivity. */ > + /* The cardinality of the partial join represented by this JOIN_TAB. */ > + double exec_partial_join_records; psergey: I understand this is a counter? Why use floating point type for it, then? > + /* Estimate of the fanout of this join based on execution-time counters. */ > + double exec_fanout; > + > table_map dependent,key_dependent; > /* > 1 - use quick select > @@ -377,6 +383,12 @@ typedef struct st_join_table { > NULL - Not doing a loose scan on this join tab. > */ > struct st_join_table *loosescan_match_tab; > + > + /* > + The JOIN_TAB after this one with the least partial join cardinality. > + Valid only after the join optimization phase. > + */ > + st_join_table *least_card_tab; > > /* TRUE <=> we are inside LooseScan range */ > bool inside_loosescan_range; > @@ -522,7 +534,7 @@ typedef struct st_join_table { > return (is_hash_join_key_no(key) ? hj_key : table->key_info+key); > } > double scan_time(); > - ha_rows get_examined_rows(); > + double get_examined_rows_estimate(); > bool preread_init(); > > bool is_sjm_nest() { return test(bush_children); } > @@ -767,6 +779,8 @@ typedef struct st_position :public Sql_a > { > /* The table that's put into join order */ > JOIN_TAB *table; > + /* The JOIN_TAB after this one with the least partial join cardinality */ > + st_position *least_card_pos; > > /* > The "fanout": number of output rows that will be produced (after > @@ -777,6 +791,7 @@ typedef struct st_position :public Sql_a > > /* The selectivity of the pushed down conditions */ > double cond_selectivity; > + double partial_join_cardinality; > > /* > Cost accessing the table in course of the entire complete join execution, > @@ -917,6 +932,7 @@ class JOIN :public Sql_alloc > void restore_query_plan(Join_plan_state *restore_from); > /* Choose a subquery plan for a table-less subquery. */ > bool choose_tableless_subquery_plan(); > + double examined_rows_estimate; > > public: > JOIN_TAB *join_tab, **best_ref; > @@ -1214,13 +1230,20 @@ class JOIN :public Sql_alloc > > bool union_part; ///< this subselect is part of union > > - enum join_optimization_state { NOT_OPTIMIZED=0, > - OPTIMIZATION_IN_PROGRESS=1, > - OPTIMIZATION_DONE=2}; > bool optimized; ///< flag to avoid double optimization in EXPLAIN > bool initialized; ///< flag to avoid double init_execution calls > > - enum { QEP_NOT_PRESENT_YET, QEP_AVAILABLE, QEP_DELETED} have_query_plan; > + enum {QEP_NOT_PRESENT_YET, > + /* A query plan exists. It is being optimized, but is still incomplete. */ > + QEP_INCOMPLETE, > + /* > + A query plan exists, but it is usable only for EXPLAIN because > + some of its structures are incomplete. > + */ > + QEP_EXPLAINABLE, > + /* A query plan exists, and is ready for execution. */ > + QEP_EXECUTABLE, > + QEP_DELETED} have_query_plan; > > /* > Additional WHERE and HAVING predicates to be considered for IN=>EXISTS > @@ -1248,6 +1271,10 @@ class JOIN :public Sql_alloc > TABLE *table_reexec[1]; // make_simple_join() > JOIN_TAB *join_tab_reexec; // make_simple_join() > /* end of allocation caching storage */ > + /* > + The index of the current partial join being processed during query execution. > + */ > + JOIN_TAB **cur_exec_tab; > > JOIN(THD *thd_arg, List<Item> &fields_arg, ulonglong select_options_arg, > select_result *result_arg) > @@ -1259,7 +1286,7 @@ class JOIN :public Sql_alloc > void init(THD *thd_arg, List<Item> &fields_arg, ulonglong select_options_arg, > select_result *result_arg) > { > - join_tab= join_tab_save= 0; > + join_tab= join_tab_save= table_access_tabs= 0; > table= 0; > table_count= 0; > top_join_tab_count= 0; > @@ -1274,6 +1301,7 @@ class JOIN :public Sql_alloc > found_records= 0; > fetch_limit= HA_POS_ERROR; > examined_rows= 0; > + examined_rows_estimate= -1; > exec_tmp_table1= 0; > exec_tmp_table2= 0; > sortorder= 0; > @@ -1334,6 +1362,7 @@ class JOIN :public Sql_alloc > query plan was produced > */ > table_access_tabs= NULL; > + cur_exec_tab= NULL; > } > > int prepare(Item ***rref_pointer_array, TABLE_LIST *tables, uint wind_num, > @@ -1352,6 +1381,7 @@ class JOIN :public Sql_alloc > bool alloc_func_list(); > bool flatten_subqueries(); > bool optimize_unflattened_subqueries(); > + bool optimize_in_subqueries(); > bool optimize_constant_subqueries(); > bool make_sum_func_list(List<Item> &all_fields, List<Item> &send_fields, > bool before_group_by, bool recompute= FALSE); > @@ -1452,9 +1482,13 @@ class JOIN :public Sql_alloc > void get_prefix_cost_and_fanout(uint n_tables, > double *read_time_arg, > double *record_count_arg); > - double get_examined_rows(); > + double get_examined_rows_estimate(); > + double total_subquery_cost(); > + > /* defined in opt_subselect.cc */ > bool transform_max_min_subquery(); > + double static_pushdown_cost(Item *cond, uint idx); > + bool setup_dynamic_conditions(); > /* True if this JOIN is a subquery under an IN predicate. */ > bool is_in_subquery() > { > > === modified file 'sql/sql_test.cc' > --- sql/sql_test.cc 2013-06-06 15:51:28 +0000 > +++ sql/sql_test.cc 2013-09-27 08:25:14 +0000 > @@ -365,7 +365,11 @@ print_plan(JOIN* join, uint idx, double > pos= join->best_positions[i]; > table= pos.table->table; > if (table) > + { > fputs(table->s->table_name.str, DBUG_FILE); > + fprintf(DBUG_FILE, "(records_read: %.2f, read_time: %.2f)", > + pos.records_read, pos.read_time); > + } > fputc(' ', DBUG_FILE); > } > } > @@ -377,10 +381,11 @@ print_plan(JOIN* join, uint idx, double > { > join_table= (*plan_nodes); > fputs(join_table->table->s->table_name.str, DBUG_FILE); > - fprintf(DBUG_FILE, "(%lu,%lu,%lu)", > - (ulong) join_table->found_records, > - (ulong) join_table->records, > - (ulong) join_table->read_time); > + fprintf(DBUG_FILE, "(found_records: %llu, records_read: %llu, records: %llu, read_time: %.2f)", > + (ulonglong) join_table->found_records, > + (ulonglong) join_table->records_read, > + (ulonglong) join_table->records, > + join_table->read_time); > fputc(' ', DBUG_FILE); > } > fputc('\n', DBUG_FILE); > > === modified file 'sql/sql_update.cc' > --- sql/sql_update.cc 2013-07-17 19:24:29 +0000 > +++ sql/sql_update.cc 2013-04-01 10:36:05 +0000 > @@ -358,7 +358,7 @@ int mysql_update(THD *thd, > } > > /* Apply the IN=>EXISTS transformation to all subqueries and optimize them. */ > - if (select_lex->optimize_unflattened_subqueries(false)) > + if (select_lex->optimize_subqueries(NULL)) > DBUG_RETURN(TRUE); > > if (select_lex->inner_refs_list.elements && > > === modified file 'sql/sys_vars.cc' > --- sql/sys_vars.cc 2013-08-26 11:26:21 +0000 > +++ sql/sys_vars.cc 2013-09-27 08:25:14 +0000 > @@ -1836,6 +1836,8 @@ export const char *optimizer_switch_name > "table_elimination", > "extended_keys", > "exists_to_in", > + "expensive_pred_static_pushdown", > + "expensive_pred_dynamic_pushdown", > "default", NullS > }; > /** propagates changes to @@engine_condition_pushdown */ > @@ -1878,7 +1880,9 @@ static Sys_var_flagset Sys_optimizer_swi > "subquery_cache, " > "table_elimination, " > "extended_keys, " > - "exists_to_in " > + "exists_to_in, " > + "expensive_pred_static_pushdown, " > + "expensive_pred_dynamic_pushdown" > "} and val is one of {on, off, default}", > SESSION_VAR(optimizer_switch), CMD_LINE(REQUIRED_ARG), > optimizer_switch_names, DEFAULT(OPTIMIZER_SWITCH_DEFAULT), > @@ -4243,6 +4247,36 @@ static Sys_var_harows Sys_expensive_subq > SESSION_VAR(expensive_subquery_limit), CMD_LINE(REQUIRED_ARG), > VALID_RANGE(0, HA_POS_ERROR), DEFAULT(100), BLOCK_SIZE(1)); > > +static Sys_var_harows Sys_dynamic_pushdown_max_evals( > + "dynamic_pushdown_max_evals", > + "The number of times a dynamic condition must be evaluated in " > + "order to trigger dynamic pushdown", > + SESSION_VAR(dynamic_pushdown_max_evals), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, HA_POS_ERROR), DEFAULT(100), BLOCK_SIZE(1)); > + > +static Sys_var_uint Sys_dynamic_pushdown_cost_ratio( > + "dynamic_pushdown_cost_ratio", > + "The ratio (in %) by which a new placement of the predicate in the plan" > + "must improve the plan cost in order to trigger dynamic pushdown.", > + SESSION_VAR(dynamic_pushdown_cost_ratio), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, 100), DEFAULT(25), BLOCK_SIZE(1)); > + > +static Sys_var_uint Sys_dynamic_pushdown_sampling_probability( > + "dynamic_pushdown_sampling_probability", > + "The probability (in %) with which to evaluate this condition in order" > + "to estimate its selectivity.", > + SESSION_VAR(dynamic_pushdown_sampling_probability), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, 100), DEFAULT(3), BLOCK_SIZE(1)); > + > +static Sys_var_uint Sys_dynamic_pushdown_max_selectivity( > + "dynamic_pushdown_max_selectivity", > + "If a predicate's selectivity is below this limit, it is considered" > + "sufficiently selective to re-filter all records in a partial blocked join." > + "The selectivity itself is estimated by sampling the predicate with" > + "probability 'dynamic_pushdown_sampling_probability'.", > + SESSION_VAR(dynamic_pushdown_max_selectivity), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, 100), DEFAULT(25), BLOCK_SIZE(1)); > + > static bool check_pseudo_slave_mode(sys_var *self, THD *thd, set_var *var) > { > longlong previous_val= thd->variables.pseudo_slave_mode; > BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

[Maria-developers] @@optimizer_search_depth default value - why 62 and not 0?
by Sergei Petrunia 25 Oct '13

25 Oct '13

Hi Timour, Me and Stephane were discussing a performance issue, and are wondering - why does the @@optimizer_search_depth has a value of 62? There is a vale of optimizer_search_depth=0 which tells the optimizer to chose depth automatically (exhaustive if less than 7 tables, greedy search if more than that). This value looks like it was intended to be the default. Yet, currently the default (in both MariaDB and MySQL) is 62, which tells the optimizer use the "new" (as in "new in 5.1") greedy search code, but sets the depth so high that it uses exhaustive search. Is there any reason behind this? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] Rev 3710: MariaDB made be compiled by gcc 4.8.1
by Sergei Golubchik 24 Oct '13

24 Oct '13

Hi, Sanja! On Oct 24, sanja(a)montyprogram.com wrote: > At file:///home/bell/maria/bzr/work-maria-5.3-test/ > > ------------------------------------------------------------ > revno: 3710 > revision-id: sanja(a)montyprogram.com-20131024170011-r5r6fd84y4x1yr09 > parent: bar(a)mnogosearch.org-20131021093629-0nyk7osjssern1sa > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.3-test > timestamp: Thu 2013-10-24 20:00:11 +0300 > message: > MariaDB made be compiled by gcc 4.8.1 Okay, that looks much better. Please push it in a stage tree, just to check that it doesn't break on all other compilers that we use, and then push into main. Thanks! Regards, Sergei

1 0

Re: [Maria-developers] MariaDB made be compiled by gcc 4.8.1
by Sergei Golubchik 24 Oct '13

24 Oct '13

Hi, Sanja! > >>> === modified file 'storage/maria/ma_checksum.c' > >>> --- a/storage/maria/ma_checksum.c 2008-04-03 13:40:25 +0000 > >>> +++ b/storage/maria/ma_checksum.c 2013-10-22 07:56:05 +0000 > >>> @@ -58,7 +58,7 @@ ha_checksum _ma_checksum(MARIA_HA *info, > >>> length= _ma_calc_blob_length(blob_size_length, pos); > >>> if (length) > >>> { > >>> - memcpy((char*) &pos, pos + blob_size_length, sizeof(char*)); > >>> + memcpy((char*) &pos, pos + blob_size_length, (sizeof(char*))); > >> Why? > > ma_checksum.c: In function '_ma_checksum': > > ma_checksum.c:61:60: error: argument to 'sizeof' in 'memcpy' call is the > > same pointer type 'char *' as the destination; expected 'char' or an > > explicit length [-Werror=sizeof-pointer-memaccess] > > memcpy((char*) &pos, pos + blob_size_length, sizeof(char*)); > > ^ > > shouldn't it be: > memcpy((char**) &pos, pos + blob_size_length, sizeof(char*)); Sanja ^^^ That would be a better fix. The result looks more logical than with meaningless parentheses. Or, I suppose, you can try to drop the cast completely: memcpy(&pos, pos + blob_size_length, sizeof(pos)); > > BTW all other such fixes was done because this check. Yes, I've figured it out. Regards, Sergei

2 2

[Maria-developers] hash + like
by Roberto Spadim 24 Oct '13

24 Oct '13

Hi guys, how could i optimize a hash index using like? i was thinking about something like a index over first left bytes using hash, example: field LIKE "abc%" create index hash_index( field(3) ) using hash any idea if this could be possible? this could boost some queries here that i'm using btree today i'm using the LIKE, but i'm considering a virtual column + a index over virtual column + a ( AND virtual_column='abc') any other idea? or maybe optimize the LIKE function -- Roberto Spadim

2 1

Re: [Maria-developers] MariaDB made be compiled by gcc 4.8.1
by Sergei Golubchik 23 Oct '13

23 Oct '13

Hi, Sanja! See below On Oct 22, sanja(a)montyprogram.com wrote: > message: > MariaDB made be compiled by gcc 4.8.1 > === modified file 'extra/yassl/include/yassl_int.hpp' > --- a/extra/yassl/include/yassl_int.hpp 2012-04-05 08:49:38 +0000 > +++ b/extra/yassl/include/yassl_int.hpp 2013-10-22 07:56:05 +0000 > @@ -440,7 +440,7 @@ public: > const Ciphers& GetCiphers() const; > const DH_Parms& GetDH_Parms() const; > const Stats& GetStats() const; > - const VerifyCallback getVerifyCallback() const; > + VerifyCallback getVerifyCallback() const; Why? > === modified file 'storage/maria/ma_checksum.c' > --- a/storage/maria/ma_checksum.c 2008-04-03 13:40:25 +0000 > +++ b/storage/maria/ma_checksum.c 2013-10-22 07:56:05 +0000 > @@ -58,7 +58,7 @@ ha_checksum _ma_checksum(MARIA_HA *info, > length= _ma_calc_blob_length(blob_size_length, pos); > if (length) > { > - memcpy((char*) &pos, pos + blob_size_length, sizeof(char*)); > + memcpy((char*) &pos, pos + blob_size_length, (sizeof(char*))); Why? > crc= my_checksum(crc, pos, length); > } > continue; Regards, Sergei

2 1

[Maria-developers] gcc warning
by Roberto Spadim 20 Oct '13

20 Oct '13

hi guys! just a warning , bye :) [ 99%] Building C object tests/CMakeFiles/bug25714.dir/bug25714.c.o /home/src/mariadb/mariadb-5.5.33a/tests/bug25714.c: In function 'main': /home/src/mariadb/mariadb-5.5.33a/tests/bug25714.c:26:7: warning: variable 'OK' set but not used [-Wunused-but-set-variable] int OK; ^ -- Roberto Spadim

1 0

Re: [Maria-developers] Comments for parallel replication patch (MDEV-4506)
by Kristian Nielsen 18 Oct '13

18 Oct '13

Michael Widenius <michael.widenius(a)gmail.com> writes: >> What about Rotate_log_event::do_update_pos() and >> Stop_log_event::do_update_pos()? >> >> They currently access the flag in rli. I am not sure that is right, as > this >> could be far ahead of the event they are actually executing? > > Aren't the above events handled by the sql driver thread? No, by the worker threads. I do not understand how it could be done in the driver thread? do_update_pos updates the current position of the last event executed, it cannot happen until after events have been executed? And this happens in the worker threads. > My thought was that the driver thread was in charge of all reading of the > relay log and would thus hide any stop or rotate events from the sql > execution threads. The Rotate event is what updates the group_master_log_name to the name of the next binlog file on the master. This must not happen until previous events have been executed (otherwise they will be wrongly skipped if we stop and restart the slave). Or did I misunderstand? My best suggestion at the moment is to copy the required information from m_flags (and a couple of other fields from rli) into rgi, so that the right value gets associated with the right event in each worker thread. But let's discuss when you are back from travelling if you have a better idea. > The purgeing of log should be changed to check the position of the st > committed event. It's safe to delay any older logs. > The other I am not sure about as I can't look at the code just know. > However we need a solution that works also for masters that don't support > GTID. Yes, the non-GTID case is the main problem. (Using GTID, everything is updated automatically in parallel in the mysql.gtid_slave_pos and there is a sub_id field that ensures we can figure out which GTIDs are the most recent ones). - Kristian.

1 0

[Maria-developers] Parallel replication MDEV-4506
by Michael Widenius 14 Oct '13

14 Oct '13

Hi! Just a quick status update of parallel replication in MariaDB 10.0 I yesterday pushed to the 10.0-knielsen tree changes that makes many of the replication variables thread safe. I will today push a patch that should fix the rest of the variables. Here is the changelog entry for this: Fixes for parallel slave: - Made slaves temporary table multi-thread slave safe by adding mutex around save_temporary_table usage. - rli->save_temporary_tables is the active list of all used temporary tables - This is copied to THD->temporary_tables when temporary tables are opened and updated when temporary tables are closed - Added THD->lock_temporary_tables() and THD->unlock_temporary_tables() to simplify this. - Relay_log_info->sql_thd renamed to Relay_log_info->sql_driver_thd to avoid wrong usage for merged code. - Added is_part_of_group() to mark functions that are part of the next function. This replaces setting IN_STMT when events are executed. - Added is_begin(), is_commit() and is_rollback() functions to Query_log_event to simplify code. - If slave_skip_counter is set run things in single threaded mode. This simplifies code for skipping events. - Updating state of relay log (IN_STMT and IN_TRANSACTION) is moved to one single function: update_state_of_relay_log() We can't use OPTION_BEGIN to check for the state anymore as the sql_driver and sql execution threads may be different. Clear IN_STMT and IN_TRANSACTION in init_relay_log_pos() and Relay_log_info::cleanup_context() to ensure the flags doesn't survive slave restarts is_in_group() is now independent of state of executed transaction. - Reset thd->transaction.all.modified_non_trans_table() if we did set it for single table row events. This was mainly for keeping the flag as documented. - Changed slave_open_temp_tables to uint32 to be able to use atomic operators on it. - Relay_log_info::sleep_lock -> rpl_group_info::sleep_lock - Relay_log_info::sleep_cond -> rpl_group_info::sleep_cond - Changed some functions to take rpl_group_info instead of Relay_log_info to make them multi-slave safe and to simplify usage - do_shall_skip() - continue_group() - sql_slave_killed() - next_event() - Simplifed arguments to io_slave_killed(), check_io_slave_killed() and sql_slave_killed(); No reason to supply THD as this is part of the given structure. - set_thd_in_use_temporary_tables() removed as in_use is set on usage - Added information to thd_proc_info() which thread is waiting for slave mutex to exit. - In open_table() reuse code from find_temporary_table() Other things: - More DBUG statements - Fixed the rpl_incident.test can be run with --debug - More comments - Disabled not used function rpl_connect_master() The TODO for parallel replication is documented at top of rpl_parallel.cc. Here is the comment: ------- - Error handling. If we fail in one of multiple parallel executions, we need to make a best effort to complete prior transactions and roll back following transactions, so slave binlog position will be correct. And all the retry logic for temporary errors like deadlock. - Stopping the slave needs to handle stopping all parallel executions. And the logic in sql_slave_killed() that waits for current event group to complete needs to be extended appropriately... - Audit the use of Relay_log_info::data_lock. Make sure it is held correctly in all needed places also when using parallel replication. - We need some user-configurable limit on how far ahead the SQL thread will fetch and queue events for parallel execution (otherwise if slave gets behind we will fill up memory with pending malloc()'ed events). - Fix update of relay-log.info and master.info. In non-GTID replication, they must be serialised to preserve correctness. In GTID replication, we should not update them at all except at slave thread stop. - All the waits (eg. in struct wait_for_commit and in rpl_parallel_thread_pool::get_thread()) need to be killable. And on kill, everything needs to be correctly rolled back and stopped in all threads, to ensure a consistent slave replication state. - Handle the case of a partial event group. This occurs when the master crashes in the middle of writing the event group to the binlog. The slave rolls back the transaction; parallel execution needs to be able to deal with this wrt. commit_orderer and such. - We should notice if the master doesn't support GTID, and then run in single threaded mode against that master. This is needed to be able to support multi-master-replication with old and new masters. - Retry of failed transactions is not yet implemented for the parallel case. ---------- What this means is: In theory things should work, as long as there is no problems in the binary or relay log and we don't fill up memory with too many events. We will merge this branch to 10.0-base and then to 10.0 withing the next few days and start testing it. We also plan to release a beta of 10.0 with this code ASAP as 10.0 is now feature complete. During the beta phase we will fix the above outstanding issues. Kristian will this week start working on the error handling. After that we have to look at the memory consumption (ie, not give the sql execution threads more work than they can handle). I will shortly update the following article with information about how to use parallel replication. https://mariadb.com/kb/en/parallel-replication/ The task itself is documented at: https://mariadb.atlassian.net/browse/MDEV-4506 Regards, Monty

1 0

[Maria-developers] Wrong SQL thread position reporting after IO thread restart
by Pavel Ivanov 11 Oct '13

11 Oct '13

Krisitan, I found what I think is a bug in IO and SQL thread accounting. How to reproduce: 1) Set up two servers S1 and S2. S1 is a master, S2 is slave with master_using_gtid = current_pos. 2) Execute some transactions on the master, e.g. create database d; create table d.t (n int); insert into d.t values (1); 3) Both servers are at 0-1-3 now, SHOW SLAVE STATUS on S2 shows Read_Master_Log_Pos equals to Exec_Master_Log_Pos. 4) Execute STOP SLAVE IO_THREAD on S2. 5) S2 reports in the logs: "Slave I/O thread exiting, ... GTID position 0-1-2". So IO thread didn't realize that it received full transaction for 0-1-3 even though it didn't receive next GTID event. 6) Execute START SLAVE IO_THREAD on S2. 7) At this point SHOW SLAVE STATUS on S2 shows Read_Master_Log_Pos the same as in step 3, but Exec_Master_Log_Pos is now less than in step 3 as if SQL thread didn't catch up with IO thread yet. But despite both threads running and no more transactions executed on master Exec_Master_Log_Pos doesn't change and doesn't become equal to Read_Master_Log_Pos. This apparently happens because IO thread restarts from one transaction behind, adds to relay log Rotate event that master sends with the position of that transaction, but then doesn't add any events for the transaction because it knows they already were added into relay log. I think both problems are bugs. And although after fixing the first it would be really hard (if possible) to reproduce second, I'd think the reporting of SQL thread's position still should be fixed. Thank you, Pavel

2 2

Re: [Maria-developers] [Commits] Rev 3702: Fixed bug mdev-5105. in file:///home/igor/maria/maria-5.3-mdev5105/
by Oleksandr Byelkin 10 Oct '13

10 Oct '13

Hi! 10.10.2013 20:08, Igor Babaev пишет: > At file:///home/igor/maria/maria-5.3-mdev5105/ > > ------------------------------------------------------------ > revno: 3702 > revision-id: igor(a)askmonty.org-20131010170826-dl2xwc048hvnxmau > parent: igor(a)askmonty.org-20131004165107-0hdknrn9111ojupx > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-mdev5105 > timestamp: Thu 2013-10-10 10:08:26 -0700 > message: > Fixed bug mdev-5105. > The bug caused a memory overwrite in the function update_ref_and_keys() > It happened due to a wrong value of SELECT_LEX::cond_count. This value > historically was calculated by the fix_fields method. Now the logic of > calling this method became too complicated and, as a result, this value > is calculated not always correctly. > The patch changes the way how and when the values of SELECT_LEX::cond_count > and of SELECT_LEX::between_count are calculated. The new code does it just at > the beginning of update_ref_and_keys(). > > OK to push.

1 0

[Maria-developers] On predicate pushdown in presense of join buffering.
by Sergei Petrunia 10 Oct '13

10 Oct '13

Hello, Here is the result of discussion on Join Buffering + Static Predicate Pushdown. == Problem setting == Consider a query using join buffering +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 2 | Using where; Using join buffer (flat, BNL join) | | 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | 2 | Using join buffer (flat, BNL join) | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ Consider a predicate "P(t2)". Join buffering works as follows: read records from table t1 and put them into buffer; for each record in table t2 { if (cond(t2)) { for each record in the join buffer { if (cond(t1, t2) { pass record for further processing } } } } cond(2) is stored in "join_tab->cache_select->cond" cond(t1, t2) is stored in join_tab->select_cond". The question is, how should Static Predicate Pushdown work with join buffering? Should the condition be put into cache_select->cond or into select_cond of some later join_tab? Currently, the server will put the predicate P(t2) into both join_tab[t2]->select_cond and join_tab[t2]->cache_select->cond. (And subquery cache will wrap them into different caches) == Solution == First observation: if the expensive subquery predicate depends on several tables (not counting the const tables) then it will never be put into cache_select->cond. join_tab[$TBL]->cache_select->cond can only have conditions that depend only on table $TBL. If a predicate P(T) depends only on table $T, it can be put into 1. join_tab[$T]->cache_select->cond 2. join_tab[$T]->select_cond 3. join_tab[$T+1]->select_cond 4. join_tab[$T+2]->select_cond ... Putting the predicate into both #1 and #2 does not make sense. We have decided that Timour will remove duplicate predicate pushdown from cache_select and select_cond. After that feature, the predicates in cache_select->cond will not have duplicates in cache_select. That way, the options for Static Predicate Pushdown will be: 1. join_tab[$T]->cache_select->cond 3. join_tab[$T+1]->select_cond 4. join_tab[$T+2]->select_cond ... === Needed adjustments in optimizer === Consider the EXPLAIN again: +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 2 | Using where; Using join buffer (flat, BNL join) | | 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | 2 | Using join buffer (flat, BNL join) | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ and a predicate P(t2). Currently, Static Predicate Pushdown will consider putting the predicate at: t2->select_cond: fanout(t1) * fanout(t2) predicate invocations t3->select_cond: fanout(t1) * fanout(t2)* fanout(t3) predicate invocations (here, fanout takes into account selectivity, so it can be that fanout <1) If P(t2) is not attached to t2->select_cond but to t2->cache_select->cond instead, the number of invocations will change to: t2->cache_select->cond: fanout(t1) / number_of_t1_records_in_join_buffer * fanout(t2). The above is obvious for non-BKA case. ==== A note about subquery predicate cache ==== [A theoretical comment] We're not going to do this, but: if P(t2) was attached [ONLY] to t2->select_cond, execution of the query will cause these invocations: P(t2.row1) ---+ P(t2.row1) |-- number_of_t1_records_in_join_buffer times ... | P(t2.row1) ----+ P(t2.row2) ----+ P(t2.row2) |-- number_of_t1_records_in_join_buffer times ... | P(t2.row2) ----+ ... etc. That is, subquery predicate cache will reduce the number of times the predicate is executed. == Applicability for join algorithms == Basic idea: when join buffer is using BKA, join_tab->cache_select==NULL, there is no cache_select. Details: Join algorithms where join_tab->cache_selct->cond is used: - JOIN_CACHE_BNL - JOIN_CACHE_BNLH - JOIN_CACHE_BKAH BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Questions re MDEV-4736 and MDEV-4739 (was Re: Spider's installation sql file)
by Sergey Vojtovich 07 Oct '13

07 Oct '13

Hi Kentoku, I'm adding MariaDB developers to CC. On Thu, Sep 19, 2013 at 01:19:13AM +0900, kentoku wrote: > Hi Sergey, > > > But what kind of errors are possible in your case? Other storage engines > doesn't > seem to suffer from this API violation. > > Spider support bulk updating and deleting for avoiding network roundtrip > between data node. Some times, last bulk updating is executed in rnd_end() > function. So rnd_end() has possibility getting errors from data node. I'm afraid fixing rnd_end() callers in the server may stall for a long time. Is it accaptable for spider to use bulk updates and deletes API instead (see handler.h: start_bulk_update/start_bulk_delete). > By the way, about MDEV-4739. I get the following stack trace. > Program received signal SIGSEGV, Segmentation fault. > 0x00000000005eabf6 in thd_wait_begin (thd=0x29da060, > wait_type=10) > at /ssd1/mariadb-10.0.4/sql/sql_class.cc:4277 > 4277 MYSQL_CALLBACK(thd->scheduler, thd_wait_begin, (thd, wait_type)); > (gdb) print thd > $1 = (THD *) 0x29da060 > (gdb) bt > #0 0x00000000005eabf6 in thd_wait_begin ( > thd=0x29da060, wait_type=10) > at /ssd1/mariadb-10.0.4/sql/sql_class.cc:4277 > #1 0x000000000072a114 in scheduler_wait_sync_begin () > at /ssd1/mariadb-10.0.4/sql/scheduler.cc:59 > #2 0x0000000000d6dc20 in my_sync (fd=23, my_flags=0) > at /ssd1/mariadb-10.0.4/mysys/my_sync.c:76 > #3 0x0000000000d6b54f in my_msync (fd=23, > addr=0x7ffff7ff4000, len=4096, flags=4) > at /ssd1/mariadb-10.0.4/mysys/my_mmap.c:27 > #4 0x00000000008bea03 in TC_LOG_MMAP::open ( > this=0x16e6a00, opt_name=0xe19c87 "tc.log") > at /ssd1/mariadb-10.0.4/sql/log.cc:7735 > #5 0x00000000005751cb in init_server_components () > at /ssd1/mariadb-10.0.4/sql/mysqld.cc:4797 > #6 0x0000000000575a07 in mysqld_main (argc=30, > argv=0x1f204d0) > at /ssd1/mariadb-10.0.4/sql/mysqld.cc:5208 > #7 0x000000000056d884 in main (argc=11, > argv=0x7fffffffe3a8) > at /ssd1/mariadb-10.0.4/sql/main.cc:25 > (gdb) print thd->scheduler > $2 = (scheduler_functions *) 0x8f8f8f8f8f8f8f8f > (gdb) print thd_wait_begin > $3 = {void (THD *, > int)} 0x5eaba4 <thd_wait_begin(THD*, int)> > (gdb) print wait_type > $4 = 10 > > It is looks that "thd->scheduler" is not initialized. What do you think? > Must storage engine set it? Nope, it looks like a bug in thread pool. MDEV-4739 has different trace, how did you get this one? Just executed given test? Regards, Sergey

3 16

[Maria-developers] Comments for parallel replication patch (MDEV-4506)
by Kristian Nielsen 07 Oct '13

07 Oct '13

Hi Monty, I read through your latest patch for the MDEV-4506 task, parallel replication. Just to re-cap, this is a work-in-progress patch, so these are just some comments along the way that I hope will be useful. > -static bool sql_slave_killed(THD* thd, Relay_log_info* rli) > +static bool sql_slave_killed(rpl_group_info *rgi) > -static Log_event* next_event(Relay_log_info* rli) > +static Log_event* next_event(rpl_group_info *rgi) This does not look right to me. Can you explain the reason to use rgi here instead of rli? To me, rgi is about what happens in parallel in each worker thread. But these functions are about what happens in the driver SQL thread, fetching each event from the relay log and checking if the driver SQL thread is done and can exit. It looks from the patch like rgi is always passed the serial_rgi. I think it would be better if it did not use rgi at all, we could pass in any extra information needed directly. Note that serial_rgi should _only_ be used in the case where parallel replication is not activated and actual event execution happens inside the main driver SQL thread. Or did I mis something? > Log_event::enum_skip_reason > -Gtid_log_event::do_shall_skip(Relay_log_info *rli) > +Gtid_log_event::do_shall_skip(rpl_group_info *rgi) > { > + Relay_log_info *rli= rgi->rli; > /* > An event skipped due to @@skip_replication must not be counted towards the > number of events to be skipped due to @@sql_slave_skip_counter. > @@ -6315,10 +6319,13 @@ Gtid_log_event::do_shall_skip(Relay_log_ > if (rli->slave_skip_counter > 0) > { > if (!(flags2 & FL_STANDALONE)) > + { > thd->variables.option_bits|= OPTION_BEGIN; > - return Log_event::continue_group(rli); > + rgi->rli->set_flag(Relay_log_info::IN_TRANSACTION); > + } > + return Log_event::continue_group(rgi); > } > - return Log_event::do_shall_skip(rli); > + return Log_event::do_shall_skip(rgi); > } This looks to me like it cannot work. shall_skip() is called from apply_event_and_update_pos() which runs in each worker thread, in parallel. But rli->slave_skip_counter needs to be checked from the driver SQL thread, in the order that events are read from the relay log. Otherwise, if slave_skip_counter is 1 and we have transactions T1 and T2, T2 might run first and we would end up skipping T2 instead of T1 which is wrong. Besides, it seems better to in as many cases as possible to skip events by not queuing them for a worker thread in the first place. I am wondering if this change came as part of a mechanical change from rli to rgi when is_in_group() was moved from rli to rgi? That is why I had this ToDo comment: > - - Relay_log_info::is_in_group(). This needs to be handled correctly in all > - callers. I think it needs to be split into two, one version in > - Relay_log_info to be used from next_event() in slave.cc, one to be used in > - per-transaction stuff. I think what needs to be done here is that event skip handling needs to be moved, so that it is done in exec_relay_log_event() rather than in apply_event_and_update_pos(). Then it can be done always in the driver SQL thread, both in the parallel and in the non-parallel case. It is possible that we then may need the flag associated with is_in_group() both in rli and in rgi, we will see when we do the change. What do you think? > @@ -71,6 +66,7 @@ rpt_handle_event(rpl_parallel_thread::qu > > /* ToDo: Access to thd, and what about rli, split out a parallel part? */ > mysql_mutex_lock(&rli->data_lock); > + qev->ev->thd= thd; Indeed, this seems to be needed. Wow. I'm surprised that I have been able to get anything to work when this was missing?!? Or is it because in most cases, THD is passed into Log_event:: functions directly, rather than using Log_event::thd ? > @@ -1094,9 +1090,9 @@ bool Relay_log_info::is_until_satisfied( > !replicate_same_server_id) > DBUG_RETURN(FALSE); > log_name= group_master_log_name; > - log_pos= (!ev)? group_master_log_pos : > - ((thd->variables.option_bits & OPTION_BEGIN || !ev->log_pos) ? > - group_master_log_pos : ev->log_pos - ev->data_written); > + log_pos= ((!ev)? group_master_log_pos : > + (get_flag(IN_TRANSACTION) || !ev->log_pos) ? > + group_master_log_pos : ev->log_pos - ev->data_written); Yes, it is clear that we need something like this. You told me on IRC that you still needed to work on this. Eg. I see you started to add a part_of_next_statement() but did not implement this. I also am worried how to make sure that we do not miss any cases here, and agree it needs to be checked. I will just mention that for GTID I already added Log_event::is_part_of_group(), which seems related to your part_of_next_statement()? Maybe they could use the same mechanism. One outstanding issue with is_part_of_group is for row-based replication with multi-table update statements. In such multi-table updates there will be two (or more) Rows_log_event events, and there is a flag to tell which is the last one. They need to be considered part of the same statement. I think this is not yet correctly handled in the GTID code (MDEV-4985). Probably this case also needs to be handled correctly here? > - if ((rgi->thd->variables.option_bits & OPTION_BEGIN) && opt_using_transactions) > + if ((rgi->rli->get_flag(IN_TRANSACTION)) && opt_using_transactions) There is more to be done here than this. Because this runs in each parallel worker thread, in parallel. Wouldn't it be more correct actually to use rgi->thd here, actually? But in any case, we need to think about how to handle old-style replication position updates. Since we already ensure that transactions commit in-order, maybe it is ok to do the updates to position in parallel, as long as we do not decrement the position. So if we have transactions T1 and T2 running in parallel, T1 will commit before T2, but T2 may then race ahead and update replication position before T1. That is ok (both T1 and T2 are committed), but then we need in T1 to check and see that position is already ahead, so we do not set it back to T1 (that could cause duplication of T2 in case of slave restart). And in the GTID case, where the replication position is not used in case of slave restart, then it would be best not to update the master.info and relay-log.info file after every transaction (to save the cost of write + possibly fsync). But it would be best to still update it in-memory so that SHOW SLAVE STATUS shows something meaningful. > + Flags for the state of reading the relay log. Note that these are > + bits "Note that these are bit masks" is less ambiguous. > + This is updated when a temporary table is created or drooped by > + a replication thread. "drooped" -> "dropped". > + /** > + Is the replication inside a group? > + > + Replication is inside a group if either: > + - The IN_TRANSACTION flag is set, meaning we're inside a transaction > + - The IN_STMT flag is set, meaning we're inside a statement > + > + @retval true Replication thread is currently inside a group > + @retval false Replication thread is currently not inside a group > + */ > + bool is_in_group() const { > + return (m_flags & (IN_STMT | IN_TRANSACTION)); > + } Do I understand correctly that this refers to the last event fetched from the relay log, not the last event executed? These are different in the case of parallel replication. I think the comment should be clarified to explicitly say one way or the other. Hm, I see now that this is not your comment (you just moved it), so it is ok, but it would still be good if we could clarify the comment. > -static bool sql_slave_killed(THD* thd, Relay_log_info* rli) > +static bool sql_slave_killed(rpl_group_info *rgi) > { > /* > - The transaction should always be binlogged if OPTION_KEEP_LOG is set > - (it implies that something can not be rolled back). And such case > - should be regarded similarly as modifing a non-transactional table > - because retrying of the transaction will lead to an error or inconsistency > - as well. > - Example: OPTION_KEEP_LOG is set if a temporary table is created or dropped. > + The transaction should always be binlogged if OPTION_KEEP_LOG is > + set (it implies that something can not be rolled back). And such > + case should be regarded similarly as modifing a > + non-transactional table because retrying of the transaction will > + lead to an error or inconsistency as well. > + > + Example: OPTION_KEEP_LOG is set if a temporary table is created > + or dropped. > */ > if ((thd->transaction.all.modified_non_trans_table || > (thd->variables.option_bits & OPTION_KEEP_LOG)) So checking thd->transaction is not meaningful in case of parallel replication. However, this is related to stopping the slave, which as we discussed before is still work-in-progress. I think we can keep this code to handle the non-parallel case, and then add a similar check in the worker threads (maybe in a new function sql_worker_killed()). And maybe then add a check here that all worker threads have stopped. I'm looking into this. So this should be ok for now, just wanted to mention it. - Kristian.

1 0

Re: [Maria-developers] Rev 3847: Client attributes
by Sergei Golubchik 04 Oct '13

04 Oct '13

Hi, Sanja! On Oct 02, sanja(a)askmonty.org wrote: > === modified file 'client/mysqltest.cc' > --- a/client/mysqltest.cc 2013-07-21 14:39:19 +0000 > +++ b/client/mysqltest.cc 2013-10-02 13:54:55 +0000 > @@ -5648,6 +5648,10 @@ void safe_connect(MYSQL* mysql, const ch > verbose_msg("Connecting to server %s:%d (socket %s) as '%s'" > ", connection '%s', attempt %d ...", > host, port, sock, user, name, failed_attempts); > + > + mysql_options(mysql, MYSQL_OPT_CONNECT_ATTR_RESET, 0); > + mysql_options4(mysql, MYSQL_OPT_CONNECT_ATTR_ADD, > + "program_name", "mysqltest"); Did you test connecting without connection attributes? e.g. 5.5 client and 10.0 server? and mysql-5.5 client? And mysql-5.6 client? please do, at least manually > while(!mysql_real_connect(mysql, host,user, pass, db, port, sock, > CLIENT_MULTI_STATEMENTS | CLIENT_REMEMBER_OPTIONS)) > { > > === modified file 'include/mysql.h' > --- a/include/mysql.h 2013-07-18 14:46:57 +0000 > +++ b/include/mysql.h 2013-10-02 13:54:55 +0000 > @@ -557,6 +557,8 @@ int STDCALL mysql_list_proce > int status); > int STDCALL mysql_options(MYSQL *mysql,enum mysql_option option, > const void *arg); > +int STDCALL mysql_options4(MYSQL *mysql,enum mysql_option option, > + const void *arg1, const void *arg2); This is pretty silly, I'd rather have mysql_optionv(MYSQL *mysql, enum mysql_option option, ...) or even (redefining the existing function): mysql_option(MYSQL *mysql, enum mysql_option option, ...) but we're copying MySQL here, so we don't have much of a choice > void STDCALL mysql_free_result(MYSQL_RES *result); > int STDCALL mysql_free_result_start(MYSQL_RES *result); > int STDCALL mysql_free_result_cont(MYSQL_RES *result, int status); > > === modified file 'sql/sql_acl.cc' > --- a/sql/sql_acl.cc 2013-09-21 08:14:42 +0000 > +++ b/sql/sql_acl.cc 2013-10-02 13:54:55 +0000 > @@ -8570,7 +8628,13 @@ static ulong parse_client_handshake_pack > mpvio->acl_user->plugin= old_password_plugin_name; > } > } > - > + > + if (db && Why do you check for db being not NULL? > + (thd->client_capabilities & CLIENT_CONNECT_ATTRS) && > + read_client_connect_attrs(&next_field, ((char *)net->read_pos) + pkt_len, > + mpvio->thd->charset())) > + return packet_error; > + > /* > if the acl_user needs a different plugin to authenticate > (specified in GRANT ... AUTHENTICATED VIA plugin_name ..) > > === modified file 'sql/sys_vars.cc' > --- a/sql/sys_vars.cc 2013-09-21 08:14:42 +0000 > +++ b/sql/sys_vars.cc 2013-10-02 13:54:55 +0000 > @@ -71,6 +71,7 @@ > #define export /* not static */ > > #ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > +#ifndef EMBEDDED_LIBRARY Why is this? > > static Sys_var_mybool Sys_pfs_enabled( > "performance_schema", Regards, Sergei

2 3

[Maria-developers] Adding versions to documentation
by Daniel Bartholomew 03 Oct '13

03 Oct '13

All, Whenever we document new features in the Knowledge Base, one thing we need to do is say when it was (or will be) introduced. For example, if I'm documenting a feature that was introduced in MariaDB 10.0.4, I would add something similar to the following to the top of the page: <<product mariadb from=10.0.4>> This feature was introduced in MariaDB 10.0.4 <</product>> If you're in a hurry and can't remember the <<product>> macro syntax, just use a sentence. We'll wrap it in a product macro later. Also, if you come across an article that doesn't have this information, please add it if you can, or send me an email so that I know it's missing. Thanks! -- Daniel Bartholomew, Sr Tech Writer / Sys Admin MariaDB

1 0

[Maria-developers] [Gsoc]Regexp enhancements License
by Sudheera Palihakkara 03 Oct '13

03 Oct '13

Hi all, This is to inform you that the code that I created for the task "Regexp enhancements" during "Google Summer Of Code 2013" is available to to MariaDB Foundation under terms of the New BSD License. thank you. -- *Sudheera Palihakkara.* Undergraduate Department of *Computer Science and Engineering, *Faculty of Engineering, *University of Moratuwa*, Sri Lanka.

1 0

[Maria-developers] Replicating same server_id problem
by Pavel Ivanov 01 Oct '13

01 Oct '13

Kristian, Currently MariaDB (as well as MySQL of all previous versions) has a very big problem related to replicating same server_id. There is --replicate-same-server-id flag which as I understand (when set to 0) controls two things: 1) It doesn't allow slave to connect to a master with the same server_id. 2) Slave ignores all binlog events in the replication stream that have the same server_id as slave. And this flag cannot be set to 1 when --log-slave-updates is used. And that is a big problem. Consider the following scenario: let's say we have two servers S1 (master) and S2 (slave). Let's say at some moment in time they are completely in sync and you bring down S2 to take cold backup (you can even include binlogs in it). Then you bring it back up, S1 is still master. Now you execute some transactions, then you do a failover, make S2 master and execute some more transactions. Then you bring down S1, restore it from the backup taken earlier and connect to replicate from S2 again. At this point S1 will have to reply some transaction that have the same server_id as S1 has, but S1 doesn't have those transactions yet, so it shouldn't skip them, but it will. What do you think about how this should be fixed? As I understand you explicitly wanted to support replication cycles, so you still want the skipping of transactions with the same server_id to exist. But the situation above is a valid production use case. Maybe in GTID world it can be solved better? E.g. if transaction has the same server_id, but the GTID wasn't applied yet then it shouldn't be skipped? Thank you, Pavel

4 8

[Maria-developers] Galera and aria
by Roberto Spadim 01 Oct '13

01 Oct '13

Hi again guys! =) I have a doubt, i was using a diff -u galera-5.33a mariadb-5.33a | grep -i "myisam" and found some lines... well could we add the aria storage to 'work' like myisam with galera cluster? i don't know if it's the right way (changing "== MYISAM_ENGINE" to something like (== myisam_engine OR == maria_engine)) if yes... i could make this patch and send? should i create a mdev to this task (allow aria engine with galera cluster)? today it's not a trasational engine, but it's better than myisam with the crash safe feature that's all :) thanks guys -- Roberto Spadim

2 4

[Maria-developers] regexp review
by Oleksandr Byelkin 30 Sep '13

30 Sep '13

Hi! Hi! Everything is OK. But there are some small issues: --- mysql-test/include/ctype_utf8mb4.inc 2010-03-05 08:17:19 +0000 +++ mysql-test/include/ctype_utf8mb4.inc 2013-09-26 14:02:17 +0000 @@ -234,15 +234,15 @@ set names utf8mb4; set names utf8mb4; # This should return TRUE -select 'вася' rlike '[[:<:]]вася[[:>:]]'; -select 'вася ' rlike '[[:<:]]вася[[:>:]]'; -select ' вася' rlike '[[:<:]]вася[[:>:]]'; -select ' вася ' rlike '[[:<:]]вася[[:>:]]'; +select 'вася' rlike '\\bвася\\b'; +select 'вася ' rlike '\\bвася\\b'; +select ' вася' rlike '\\bвася\\b'; +select ' вася ' rlike '\\bвася\\b'; Is above unsupported pattern? === modified file 'sql/mysqld.cc' --- sql/mysqld.cc 2013-09-18 11:07:31 +0000 +++ sql/mysqld.cc 2013-09-26 14:02:17 +0000 @@ -1898,7 +1898,7 @@ void clean_up(bool print_message) delete global_rpl_filter; end_ssl(); vio_end(); - my_regex_end(); + //my_regex_end(); #if defined(ENABLED_DEBUG_SYNC) /* End the debug sync facility. See debug_sync.cc. */ debug_sync_end(); @@ -3904,10 +3904,10 @@ static int init_common_variables() return 1; item_init(); #ifndef EMBEDDED_LIBRARY - my_regex_init(&my_charset_latin1, check_enough_stack_size); + //my_regex_init(&my_charset_latin1, check_enough_stack_size); my_string_stack_guard= check_enough_stack_size; #else - my_regex_init(&my_charset_latin1, NULL); + //my_regex_init(&my_charset_latin1, NULL); #endif /* Process a comma-separated character set list and choose Remove it please (I think it was just forgotten).

2 1

[Maria-developers] Empty directory in mariadb's datadir is not handled correctly
by Honza Horak 30 Sep '13

30 Sep '13

Hi guys, quite tricky problem came up after a directory has been created by something else than mysqld in the datadir. Suppose we use datadir=/var/lib/mysql. When we create any file in that directory by hand (or some program does it, which is not just imagination, it really happens), e.g. /var/lib/mysql/.local, MariaDB (Oracle's MySQL will behave the same probably) server thinks this directory is a database, which is wrong. Then we get some weird messages and outputs: # mysqlshow +--------------------+ | Databases | +--------------------+ | information_schema | | #mysql50#.local | | mysql | | performance_schema | | test | +--------------------+ # mysqlcheck -A mysqlcheck: Got error: 1102: Incorrect database name '#mysql50#.local' when selecting the database I believe this is not how the things should work. I'm not sure, but I'd expect there is still some information about existing databases in the information_schema table. So IMHO the server/tool should always fetch such info in addition to pure testing if the database directory exists. Or is such testing simply impossible? Regards, Honza

2 1

Re: [Maria-developers] [All] MariaDB's best (download) month ever
by Kristian Nielsen 30 Sep '13

30 Sep '13

Colin Charles <byte(a)mariadb.com> writes: > It's 17.09 today, and we've hit 50,160 total downloads (this number is > considered company-internal, we don't share this number externally). "company-internal"? What have you been smoking? Colin, you are supposed to be the community guy for crying out lout :-( This is *completely*, utterly unacceptable. MariaDB is a community project. We are about openness. We are about working together, all of us, to create a better world. We are *not* some individuals or companys playground. So no, the community's usage numbers of its own project is not "internal", thank you very much. So please cut out the crap already. - Kristian.

6 11

[Maria-developers] Prep for MariaDB Galera 5.5.33a
by Daniel Bartholomew 28 Sep '13

28 Sep '13

All, I've begun the preparations for the MariaDB Galera 5.5.33a release. Draft release notes and changelog are here: - https://mariadb.com/kb/en/mariadb-galera-5533a-release-notes/ - https://mariadb.com/kb/en/mariadb-galera-5533a-changelog/ Please make necessary changes/additions/etc... to the above pages (or let me know). Thanks. -- Daniel Bartholomew, Sr Tech Writer / Sys Admin MariaDB | t: +1 919-809-6069 | Skype: daniel-bartholomew

1 0

[Maria-developers] MySQL bug affects mariaDB
by Peter Laursen 27 Sep '13

27 Sep '13

I think the 'mysql' client with MariaDB has a similar problem as it has with MySQL - refer? http://bugs.mysql.com/bug.php?id=70443 MariaDB installer installs my.ini to /datadir - what with MariaDB is a subfolder to /basedir. The server will look for it correctly as the registry key defining the service has a --default-file specification. But the command line clients don't read the registry key. @Wlad? -- Peter -- Webyog

3 7

[Maria-developers] EXPLAIN INSERT in MySQL - not useful after all?
by Sergei Petrunia 27 Sep '13

27 Sep '13

4 4

[Maria-developers] [GSoC] MariaDB Contributor Agreement for Kerberos Authentication Plugin
by QIU Shuang 27 Sep '13

27 Sep '13

Hi community, I declare that all the work done so far (and future) on the project Kerberize MariaDB is licenced by the new BSD Lisence. Please contact me if any problems with the license. Thank you all and best regards! Sincerely, Shuang

1 0

[Maria-developers] [GSoC] Request for code review
by QIU Shuang 27 Sep '13

27 Sep '13

Hi Sergei, This is Shuang, a participant for GSoC 2013 Kerberize MariaDB project. I want to ask the community to review the code before requesting for pull and send the MCA. How could I send the codes? In a diff file? or from other approaches? Thank you very much and best regards! Sincerely, Shuang

2 2

[Maria-developers] Review of base64.diff and german2.diff
by Michael Widenius 27 Sep '13

27 Sep '13

Hi! Here is the review for the code that we should put into 10.0 First the base64: > === modified file 'mysql-test/t/func_str.test' > --- mysql-test/t/func_str.test 2013-05-07 11:05:09 +0000 > +++ mysql-test/t/func_str.test 2013-08-28 13:14:24 +0000 > @@ -1555,3 +1555,118 @@ drop table t1,t2; > --echo # End of 5.5 tests > --echo # > > + > +--echo # > +--echo # Start of 5.6 tests > +--echo # Shouldn't this be start of 10.0 tests ? (I know that this code is from MySQL 5.6, but still for us this is 10.0...) > === modified file 'mysys/base64.c' > --- mysys/base64.c 2011-06-30 15:46:53 +0000 > +++ mysys/base64.c 2013-03-09 06:22:59 +0000 > @@ -1,5 +1,4 @@ > -/* Copyright (c) 2003-2008 MySQL AB, 2009 Sun Microsystems, Inc. > - Use is subject to license terms. > +/* Copyright (c) 2003, 2010, Oracle and/or its affiliates. All rights reserved. Removed 'all rights reserved'. You can't have that for GPL code. If there is any new code from us, please add a copyright message for the MariaDB foundation too! > > This program is free software; you can redistribute it and/or modify > it under the terms of the GNU General Public License as published by > @@ -25,6 +24,28 @@ static char base64_table[] = "ABCDEFGHIJ > "abcdefghijklmnopqrstuvwxyz" > "0123456789+/"; > > +/** > + * Maximum length base64_needed_encoded_length() > + * can handle without overflow. > + */ > +int > +base64_encode_max_arg_length() > +{ > +#if (SIZEOF_INT == 8) > + /* > + 6827690988321067803 -> 9223372036854775805 > + 6827690988321067804 -> -9223372036854775807 > + */ Please describe from where the following number comes from (so that anyone can recalculate and check the number if needed): > + return 0x5EC0D4C77B03531BLL > +#else > + /* > + 1589695686 -> 2147483646 > + 1589695687 -> -2147483645 > + */ > + return 0x5EC0D4C6; > +#endif > +} > + > > int > base64_needed_encoded_length(int length_of_data) > @@ -39,6 +60,21 @@ base64_needed_encoded_length(int length_ > } > > > +/** > + * Maximum length base64_needed_decoded_length() > + * can handle without overflow. > + */ > +int > +base64_decode_max_arg_length() > +{ The following also need a comment. > +#if (SIZEOF_INT == 8) > + return 0x2AAAAAAAAAAAAAAALL; > +#else > + return 0x2AAAAAAA; > +#endif > +} > + > + <cut> > +/** > + * Get the next character from a base64 encoded stream. > + * Skip spaces, then scan the next base64 character or a pad character > + * and collect bits into decoder->c. > + * > + * @param decoder Pointer to MY_BASE64_DECODER > + * @return > + * FALSE on success (a valid base64 encoding character found) > + * TRUE on error (unexpected character or unexpected end-of-input found) > + */ Add an empty line here. It would also be good to have a good description of the base64 format in this file so that one can understand what the functions are supposed to do (or at least a link to the specification). Some thing I figured out by asking bar on IRC: - We never generate spaces - When decoding, we allow spaces anywhere. - The end of an 64 base decoded string is always padded with '=' so that the final length is dividable with 4. > +static inline my_bool > +my_base64_decoder_getch(MY_BASE64_DECODER *decoder) Remove inline from the above; It's likely to make the code slower, not faster as this is a big function with lot of if's. > { > - char b[3]; > - size_t i= 0; > - char *dst_base= (char *)dst; > - char const *src= src_base; > - char *d= dst_base; > - size_t j; > + if (my_base64_decoder_skip_spaces(decoder)) > + return TRUE; /* End-of-input */ > > - while (i < len) > + if (!my_base64_add(decoder)) /* Valid base64 character found */ > { > - unsigned c= 0; > - size_t mark= 0; > + if (decoder->mark) > + { > + /* If we have scanned '=' already, then only '=' is valid */ > + DBUG_ASSERT(decoder->state == 3); > + decoder->error= 1; > + decoder->src--; > + return TRUE; /* expected '=', but encoding character found */ > + } > + decoder->state++; > + return FALSE; > + } If you want to have the funtion inline, then move the following to another not inline function; We don't need to generate code for this for every function call: > + > + /* Process error */ > + switch (decoder->state) > + { > + case 0: > + case 1: > + decoder->src--; > + return TRUE; /* base64 character expected */ > + break; > + > + case 2: > + case 3: > + if (decoder->src[-1] == '=') > + { > + decoder->error= 0; /* Not an error - it's a pad character */ > + decoder->mark++; > + } > + else > + { > + decoder->src--; > + return TRUE; /* base64 character or '=' expected */ > + } > + break; > + default: > + DBUG_ASSERT(0); > + return TRUE; /* Wrong state, should not happen */ > + } <cut> Add to the comment for base64_decode that the 'dst' buffer has to be at least 2 character longer than what is needed to decode src_base. > +int > +base64_decode(const char *src_base, size_t len, > + void *dst, const char **end_ptr, int flags) > +{ > + if (my_base64_decoder_getch(&decoder) || > + my_base64_decoder_getch(&decoder) || > + my_base64_decoder_getch(&decoder) || > + my_base64_decoder_getch(&decoder)) > + break; Generating the error handling with a switch for every of the above is wasteful. See previous comment. > + /* Return error if there are more non-space characters */ > + decoder.state= 0; > + if (!my_base64_decoder_skip_spaces(&decoder)) > + decoder.error= 1; > + > if (end_ptr != NULL) Note that base64_needed_decoded_length() used ceil() when it's not needed. <cut> > === modified file 'sql/item_strfunc.cc' > @@ -451,6 +452,101 @@ void Item_func_aes_decrypt::fix_length_a > set_persist_maybe_null(1); > } > > + > +void Item_func_to_base64::fix_length_and_dec() > +{ > + maybe_null= args[0]->maybe_null; > + collation.set(default_charset(), DERIVATION_COERCIBLE, MY_REPERTOIRE_ASCII); Maybe better to cast both arguments to (ulonglong) as the rest of the code is depending on this. Also, why is base64_encode_max_arg_length() int instead of uint or even better size_t? > + if (args[0]->max_length > (uint) base64_encode_max_arg_length()) > + { > + maybe_null= 1; > + fix_char_length_ulonglong((ulonglong) base64_encode_max_arg_length()); > + } > + else > + { > + int length= base64_needed_encoded_length((int) args[0]->max_length); > + DBUG_ASSERT(length > 0); Don't think assert is needed as the function gurantees it already. > + fix_char_length_ulonglong((ulonglong) length - 1); > + } > +} <cut> > +String *Item_func_from_base64::val_str(String *str) > +{ > + String *res= args[0]->val_str_ascii(str); > + bool too_long= false; > + int length; > + const char *end_ptr; > + > + if (!res || > + res->length() > (uint) base64_decode_max_arg_length() || > + (too_long= > + ((uint) (length= base64_needed_decoded_length((int) res->length())) > > + current_thd->variables.max_allowed_packet)) || > + tmp_value.alloc((uint) length) || > + (length= base64_decode(res->ptr(), (int) res->length(), > + (char *) tmp_value.ptr(), &end_ptr, 0)) < 0 || > + end_ptr < res->ptr() + res->length()) > + { Shouldn't we get a readable error or warning if the string contained wrong characters? Now it looks like we will just return null. Something like 'Malformed base64 string. Error at position %d" would be nice. > + null_value= 1; // NULL input, too long input, OOM, or badly formed input > + if (too_long) > + { > + push_warning_printf(current_thd, Sql_condition::WARN_LEVEL_WARN, > + ER_WARN_ALLOWED_PACKET_OVERFLOWED, > + ER(ER_WARN_ALLOWED_PACKET_OVERFLOWED), func_name(), > + current_thd->variables.max_allowed_packet); > + } > + return 0; > + } > + tmp_value.length((uint) length); > + null_value= 0; > + return &tmp_value; > +} <cut> Review of german2.diff === modified file 'include/my_global.h' --- include/my_global.h 2013-07-21 14:39:19 +0000 +++ include/my_global.h 2013-08-29 08:27:09 +0000 @@ -1218,4 +1218,11 @@ static inline double rint(double x) #define HAVE_EXTERNAL_CLIENT #endif /* EMBEDDED_LIBRARY */ + +enum loglevel { + ERROR_LEVEL= 0, + WARNING_LEVEL= 1, + INFORMATION_LEVEL= 2 +}; + #endif /* my_global_h */ Not sure why the above, which is a my_sys construct, should be in my_global.h. Please add at least a comment about this. === modified file 'mysys/charset.c' --- mysys/charset.c 2012-08-14 14:23:34 +0000 +++ mysys/charset.c 2013-08-29 12:38:44 +0000 @@ -66,6 +66,9 @@ static my_bool init_state_maps(struct ch if (!(cs->ident_map= ident_map= (uchar*) my_once_alloc(256, MYF(MY_WME)))) return 1; + state_map= (uchar *) cs->state_map; + ident_map= (uchar *) cs->ident_map; Why is the above needed? They are already set one line above. (In addition cs->state_map and cs->ident_map are already uchar *) --- strings/ctype-simple.c 2013-07-21 14:39:19 +0000 +++ strings/ctype-simple.c 2013-08-29 09:05:07 +0000 @@ -1163,12 +1163,12 @@ static int pcmp(const void * f, const vo return res; } -static my_bool create_fromuni(struct charset_info_st *cs, - void *(*alloc)(size_t)) +static my_bool +create_fromuni(struct charset_info_st *cs, + MY_CHARSET_LOADER *loader) { uni_idx idx[PLANE_NUM]; int i,n; - struct my_uni_idx_st *tab_from_uni; /* Check that Unicode map is loaded. @@ -1209,18 +1209,18 @@ static my_bool create_fromuni(struct cha for (i=0; i < PLANE_NUM; i++) { int ch,numchars; - uchar *tab; /* Skip empty plane */ if (!idx[i].nchars) break; numchars=idx[i].uidx.to-idx[i].uidx.from+1; - if (!(idx[i].uidx.tab= tab= (uchar*) - alloc(numchars * sizeof(*idx[i].uidx.tab)))) + if (!(idx[i].uidx.tab= (uchar *) + (loader->once_alloc) (numchars * + sizeof(*idx[i].uidx.tab)))) return TRUE; - bzero(tab,numchars*sizeof(*tab)); + bzero((char*) idx[i].uidx.tab, numchars * sizeof(*idx[i].uidx.tab)); for (ch=1; ch < PLANE_SIZE; ch++) { @@ -1228,32 +1228,32 @@ static my_bool create_fromuni(struct cha if (wc >= idx[i].uidx.from && wc <= idx[i].uidx.to && wc) { int ofs= wc - idx[i].uidx.from; - tab[ofs]= ch; + ((char *) idx[i].uidx.tab)[ofs]= ch; } } } Why remove tab to be a shortcut for idx[i].uidx.from ? Shouldn't it be faster and shorter to use tab than using idx[i].uidx.from? (At least in this place) -static const uint16 page0FCdata[]= { /* FC00 (3 weights per char) */ +static uint16 page0FCdata[]= { /* FC00 (3 weights per char) */ 0x134F,0x135E,0x0000, 0x134F,0x1364,0x0000, 0x134F,0x13B0,0x0000, 0x134F,0x13C7,0x0000, 0x134F,0x13C8,0x0000, 0x1352,0x135E,0x0000, 0x1352,0x1364,0x0000, 0x1352,0x1365,0x0000, 0x1352,0x13B0,0x0000, @@ -6006,7 +6005,7 @@ static const uint16 page0FCdata[]= { /* 0x1381,0x13C8,0x0000, 0x1382,0x13C7,0x0000, 0x1382,0x13C8,0x0000, 0x1364,0x13C7,0x0000 }; If we are not going to change the above, better to keep these as const! They will then be in the const segment and you will have protection from anyone trying to change these! <cut> +my_uca_add_contraction(MY_CONTRACTIONS *list, my_wc_t *wc, size_t len, + my_bool with_context) { - MY_CONTRACTIONS *list= (MY_CONTRACTIONS*) cs->contractions; MY_CONTRACTION *next= &list->item[list->nitems]; - DBUG_ASSERT(len == 2); /* We currently support only contraction2 */ - next->ch[0]= wc[0]; - next->ch[1]= wc[1]; + size_t i; + /* + Contraction is always at least 2 characters. + Contraction is never longer than MY_UCA_MAX_CONTRACTION, + which is guaranteed by using my_coll_rule_expand() with proper limit. + */ + DBUG_ASSERT(len > 1 && len <= MY_UCA_MAX_CONTRACTION); + for (i= 0; i < len; i++) + { + /* + We don't support contractions with U+0000. + my_coll_rule_expand() guarantees there're no U+0000 in a contraction. + */ + DBUG_ASSERT(wc[i] != 0); + next->ch[i]= wc[i]; + } + if (i < MY_UCA_MAX_CONTRACTION) + next->ch[i]= 0; /* Add end-of-line marker */ Wouldn't it make life easier to always have the marker there? (Ie, always have place for the marker). At least you can remove one if from the code. If there is more than one if to remove, then it's a simple optimzation do to... <cut> +static int +my_coll_parser_too_long_error(MY_COLL_RULE_PARSER *p, const char *name) +{ + my_snprintf(p->errstr, sizeof(p->errstr), "%s is too long", name); + return 0; +} You should limit '%s' to 64 characters so that one gets the name in the output even if it's over sizeof()... %-.64s .... <cut> Regards, Monty

7 40

[Maria-developers] Review request : MDEV-4476 (mytop tracker)
by Jean Weisbuch 26 Sep '13

26 Sep '13

Hi, I have been doing modifications to the MariaDB mytop version (which is shipped on the /scripts directory), especially for bugfix, small display optimizations, use when available the I_S.PROCESSLIST instead of SHOW FULL PROCESSLIST and recently to show 1/10th of seconds on queries running time on servers having the TIME_MS column on the PROCESSLIST. A more detailed list : - Control chars on queries are not displayed anymore to avoid possible command executions/mytop misbehaving : when a query having control chars (ASCII 000 to 037) in it is shown in mytop, it can act like a key was pressed and end up doing strange/unwanted things. - Display cannot overlap the terminal size anymore when fullqueries is activated. - If not all the queries are shown on screen, a "-- Truncated query list --" is shown on the bottom of the query list. - It is now using the informations from INFORMATION_SCHEMA.PROCESSLIST when available instead of the SHOW FULL PROCESSLIST output for a better modularity is a similar maneer as Innotop does. - The query time is now shown with the 1/10th of second precision on servers having the TIME_MS patch (MariaDB and Percona) for queries running for less than 10000 seconds. - Command summary view fixed to retreive/show only the Com_* values and not the Com* (was showing a "Compression" stats row for this boolean variable). - Better have_query_cache test in case other values than ON/OFF/DEMAND would be introduced. Another detail that could be good to fix for MariaDB 10 GA is that the script is shipped as mytop.sh on MariaDB while its a Perl script (it is for example shipped as /usr/bin/mytop on Debian/Ubuntu official packages of mytop). The JIRA task : https://mariadb.atlassian.net/browse/MDEV-4476 The GIT repository where i put the latest revisions (i only have been using it lately but i commented all the changes and bugfixes along the time on the JIRA task and could provide older revisions/diff if required) : https://github.com/jb-boin/mariadb/blob/5.5.30/scripts/mytop.sh Feel free to contact me for more informations. Thanks in advance, Jean Weisbuch

1 0

[Maria-developers] Review of MDEV-4506, parallel replication, part 1
by Michael Widenius 26 Sep '13

26 Sep '13

Hi! Here is the first part of the review. I have 1/2 of one file left (rpl_parallel.cc) to review but I wanted you to have a chance to read what I have done so far. I plan to finish the review tomorrow and start working on the code during the weekend. >>>>> "Kristian" == Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: Kristian> Hi Monty, Kristian> Here is a copy of the status mail I sent earlier. The status is mostly still Kristian> current. Kristian> The code is in this tree: Kristian> lp:~maria-captains/maria/10.0-knielsen Kristian> In that tree, you can extract the parallel replication code as a single patch Kristian> with a command like this: Kristian> bzr diff -rrevid:knielsen@knielsen-hq.org-20130608103621-g91eaielv4n22aha Kristian> In the forwarded mail, I describe getting a 4.5 times speedup on a quad-core Kristian> laptop, so that's quite promising. === modified file 'mysql-test/r/mysqld--help.result' --- mysql-test/r/mysqld--help.result 2013-05-28 11:28:31 +0000 +++ mysql-test/r/mysqld--help.result 2013-08-29 08:18:22 +0000 <cut> @@ -783,6 +794,11 @@ --slave-net-timeout=# Number of seconds to wait for more data from any master/slave connection before aborting the read + --slave-parallel-threads=# + If non-zero, number of threads to spawn to apply in + parallel events on the slave that were group-committed on + the master or were logged with GTID in different + replication domains. What happens if --slave-parallel-threads is 0 ? Does this mean that there will be no parallel threads? Is this the same as 1 ? If same as 1, why not have the option to take values starting from 1 ? IRC: K> yes, if --slave-parallel-threads is 0, then the feature is disabled. The old code is used, and the SQL thread applies the events itself K> if --slave-parallel-threads is > 0, then a pool of threads are spawned. The SQL threads do not apply events themselves, they just put it in queues for the parallel threads to apply M> do we ever need to do this with only one thread? K> (we might want to give an error if user tries to do this, as it does not make much sense) M> So why not have the value from 1 to N M> where 1 is old code and > 1 is using new code ? K> That might make sense === modified file 'mysql-test/suite/perfschema/r/dml_setup_instruments.result' --- mysql-test/suite/perfschema/r/dml_setup_instruments.result 2013-03-26 09:35:34 +0000 +++ mysql-test/suite/perfschema/r/dml_setup_instruments.result 2013-08-29 08:18:22 +0000 @@ -38,14 +38,14 @@ order by name limit 10; NAME ENABLED TIMED wait/synch/cond/sql/COND_flush_thread_cache YES YES wait/synch/cond/sql/COND_manager YES YES +wait/synch/cond/sql/COND_parallel_entry YES YES +wait/synch/cond/sql/COND_prepare_ordered YES YES wait/synch/cond/sql/COND_queue_state YES YES wait/synch/cond/sql/COND_rpl_status YES YES +wait/synch/cond/sql/COND_rpl_thread YES YES +wait/synch/cond/sql/COND_rpl_thread_pool YES YES wait/synch/cond/sql/COND_server_started YES YES wait/synch/cond/sql/COND_thread_cache YES YES -wait/synch/cond/sql/COND_thread_count YES YES -wait/synch/cond/sql/Delayed_insert::cond YES YES -wait/synch/cond/sql/Delayed_insert::cond_client YES YES -wait/synch/cond/sql/Event_scheduler::COND_state YES YES Any idea why the above was deleted? Don't see anything in your patch that could affect that. <cut> === added file 'mysql-test/suite/rpl/t/rpl_parallel.test' --- mysql-test/suite/rpl/t/rpl_parallel.test 1970-01-01 00:00:00 +0000 +++ mysql-test/suite/rpl/t/rpl_parallel.test 2013-08-29 08:18:22 +0000 I didn't find any test results for rpl_parallel.test or rpl_paralell2 in your tree. Did you forget to commit these? I was also missing some comment in these files what they was testing. === modified file 'sql/log.cc' --- sql/log.cc 2013-06-06 15:51:28 +0000 +++ sql/log.cc 2013-08-29 08:18:22 +0000 @@ -6541,44 +6543,201 @@ MYSQL_BIN_LOG::write_transaction_to_binl } Could you please document in detail the arguments and return value for the following function. - I spent 3 hours to try to understand this function in detail and having a full description of the arguments would have helped a bit. bool -MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) +MYSQL_BIN_LOG::queue_for_group_commit(group_commit_entry *entry, + wait_for_commit *wfc) { + group_commit_entry *orig_queue; + wait_for_commit *list, *cur, *last; + /* To facilitate group commit for the binlog, we first queue up ourselves in the group commit queue. Then the first thread to enter the queue waits for the LOCK_log mutex, and commits for everyone in the queue once it gets the lock. Any other threads in the queue just wait for the first one to finish the commit and wake them up. Do you still use LOCK_log for this? It's a bit confusing to mention LOCK_log here when you take the LOCK_prepare_ordered in the function. If LOCK_log is required, shouldn't there be an assert for it in this function? + + To support in-order parallel replication with group commit, after we add + some transaction to the queue, we check if there were other transactions + already prepared to commit but just waiting for the first one to commit. + If so, we add those to the queue as well, transitively for all waiters. */ entry->thd->clear_wakeup_ready(); mysql_mutex_lock(&LOCK_prepare_ordered); <cut> + orig_queue= group_commit_queue; + + /* + Iteratively process everything added to the queue, looking for waiters, + and their waiters, and so on. If a waiter is ready to commit, we + immediately add it to the queue; if not we just wake it up. + + This would be natural to do with recursion, but we want to avoid + potentially unbounded recursion blowing the C stack, so we use the list + approach instead. Add here: cur->next_subsequent_commit is at this point an empty list. 'last' will always point to the last element of the list. We use this list to simulate recursion. I did find it confusing that here you threat next_subsequent_commit as a way to avoid recursion, but you use this list for other things in register_wait_for_prior_commit(). Suggestion: - If you want to reuse memory for the list, please use an union over next_subsequent_commit and have two names for it, so that you don't in the code use the same name for two different list. + */ + list= wfc; + cur= list; + last= list; + for (;;) + { + /* Add the entry to the group commit queue. */ + entry->next= group_commit_queue; + group_commit_queue= entry; + + if (entry->cache_mngr->using_xa) + { + DEBUG_SYNC(entry->thd, "commit_before_prepare_ordered"); + run_prepare_ordered(entry->thd, entry->all); + DEBUG_SYNC(entry->thd, "commit_after_prepare_ordered"); + } + + if (!cur) + break; // Can happen if initial entry has no wait_for_commit + Add a comment what the following if is testing (As there is several lists involved, the code is not that easy to understand) + if (cur->subsequent_commits_list) + { + bool have_lock; + wait_for_commit *waiter; + + mysql_mutex_lock(&cur->LOCK_wait_commit); + have_lock= true; + waiter= cur->subsequent_commits_list; + /* Check again, now safely under lock. */ + if (waiter) + { + /* Grab the list of waiters and process it. */ + cur->subsequent_commits_list= NULL; + do + { + wait_for_commit *next= waiter->next_subsequent_commit; + group_commit_entry *entry2= + (group_commit_entry *)waiter->opaque_pointer; + if (entry2) + { + /* + This is another transaction ready to be written to the binary + log. We can put it into the queue directly, without needing a + separate context switch to the other thread. We just set a flag + so that the other thread will know when it wakes up that it was + already processed. + + So put it at the end of the list to be processed in a subsequent + iteration of the outer loop. + */ + entry2->queued_by_other= true; + last->next_subsequent_commit= waiter; + last= waiter; + /* + As a small optimisation, we do not actually need to set + waiter->next_subsequent_commit to NULL, as we can use the + pointer `last' to check for end-of-list. + */ + } + else + { + /* + Wake up the waiting transaction. + + For this, we need to set the "wakeup running" flag and release + the waitee lock to avoid a deadlock, see comments on + THD::wakeup_subsequent_commits2() for details. + */ + if (have_lock) + { + cur->wakeup_subsequent_commits_running= true; + mysql_mutex_unlock(&cur->LOCK_wait_commit); + have_lock= false; Move have_lock= false to just after the 'if (have_lock)' - Easier to see that you are really reseting the variable. - Faster as there is no need to reload the variable from stack after mysql_mutex_unlock() function. - I try to nowadays in my code to always reset flag variables just after testing them to achive the above. + } + waiter->wakeup(); + } + waiter= next; + } while (waiter); + } You can optimze away one row, one 'if level' and one 'if' away by doing this: Move: 'cur->subsequent_commits_list= NULL;' before the if. Change 'if (waiter)' to 'while (waiter)' Remove 'do' and the corresponding 'while (waiter);' + if (have_lock) + mysql_mutex_unlock(&cur->LOCK_wait_commit); + } + if (cur == last) + break; Add a comment here of what is in next_subsequent_commit list. + cur= cur->next_subsequent_commit; + entry= (group_commit_entry *)cur->opaque_pointer; + DBUG_ASSERT(entry != NULL); } + + /* Now we need to clear the wakeup_subsequent_commits_running flags. */ + if (list) + { + for (;;) + { + if (list->wakeup_subsequent_commits_running) + { + mysql_mutex_lock(&list->LOCK_wait_commit); + list->wakeup_subsequent_commits_running= false; + mysql_mutex_unlock(&list->LOCK_wait_commit); Why do we need the mutex above? The only place where we test this variable, as far as I can find, is in wait_for_commit::register_wait_for_prior_commit() This thread doesn't know if the other thread is just before mysql_mutex_lock(&waitee->LOCK_wait_commit), inside the lock or after the lock. So when we reset the variable should not matter. But somehow this sounds strange. What happens if the above variable is reset just before mysql_mutex_lock(&waitee->LOCK_wait_commit) in wait_for_commit::register_wait_for_prior_commit() ? Isn't the waitee added to the list when it shouldn't? + } + if (list == last) + break; + list= list->next_subsequent_commit; + } + } + + if (opt_binlog_commit_wait_count > 0) + mysql_cond_signal(&COND_prepare_ordered); mysql_mutex_unlock(&LOCK_prepare_ordered); DEBUG_SYNC(entry->thd, "commit_after_release_LOCK_prepare_ordered"); + return orig_queue == NULL; +} + +bool +MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) +{ + wait_for_commit *wfc; + bool is_leader; + + wfc= entry->thd->wait_for_commit_ptr; + entry->queued_by_other= false; + if (wfc && wfc->waiting_for_commit) + { Add comment, something like: /* We have to wait for the threads in wfc->waiting_for_commit */ + mysql_mutex_lock(&wfc->LOCK_wait_commit); + /* Do an extra check here, this time safely under lock. */ + if (wfc->waiting_for_commit) + { + wfc->opaque_pointer= entry; + do + { + mysql_cond_wait(&wfc->COND_wait_commit, &wfc->LOCK_wait_commit); + } while (wfc->waiting_for_commit); + wfc->opaque_pointer= NULL; + } + mysql_mutex_unlock(&wfc->LOCK_wait_commit); + } <cut> @@ -6597,7 +6756,10 @@ MYSQL_BIN_LOG::write_transaction_to_binl if (next) { - next->thd->signal_wakeup_ready(); + if (next->queued_by_other) + next->thd->wait_for_commit_ptr->wakeup(); What does the above code mean ? (add a comment). Why do we wakeup the thread we where waiting for instead of ourselves? + else + next->thd->signal_wakeup_ready(); } else { <cut> @@ -6968,6 +7140,48 @@ MYSQL_BIN_LOG::write_transaction_or_stmt return 0; } + +void +MYSQL_BIN_LOG::wait_for_sufficient_commits() +{ + size_t count; + group_commit_entry *e; + group_commit_entry *last_head; + struct timespec wait_until; + + mysql_mutex_assert_owner(&LOCK_log); + mysql_mutex_assert_owner(&LOCK_prepare_ordered); + + count= 0; + for (e= last_head= group_commit_queue; e; e= e->next) + ++count; + if (count >= opt_binlog_commit_wait_count) + return; Assuming that the queue is bigger may be better: for (e= last_head= group_commit_queue, count=0; e ; e= e->next) if (++count >= opt_binlog_commit_wait_count) return; + mysql_mutex_unlock(&LOCK_log); Here you are going to take LOCK_log and LOCK_prepare_ordered in the wrong order. safe_mutex() should have noticed this. In other code you are first taking LOCK_log and then LOCK_prepare_ordered. If someone else is calling trx_group_commit_leader() while you are in this code you would get a mutex deadlock. + set_timespec_nsec(wait_until, (ulonglong)1000*opt_binlog_commit_wait_usec); + + for (;;) + { + int err; + group_commit_entry *head; + + err= mysql_cond_timedwait(&COND_prepare_ordered, &LOCK_prepare_ordered, + &wait_until); + if (err == ETIMEDOUT) + break; + head= group_commit_queue; + for (e= head; e && e != last_head; e= e->next) + ++count; + if (count >= opt_binlog_commit_wait_count) + break; + last_head= head; + } + + mysql_mutex_lock(&LOCK_log); +} <cut> === modified file 'sql/log_event.cc' --- sql/log_event.cc 2013-06-06 15:51:28 +0000 +++ sql/log_event.cc 2013-08-29 08:18:22 +0000 <cut> @@ -6101,6 +6110,18 @@ Gtid_log_event::Gtid_log_event(const cha domain_id= uint4korr(buf); buf+= 4; flags2= *buf; + if (flags2 & FL_GROUP_COMMIT_ID) + { + if (event_len < (uint)header_size + GTID_HEADER_LEN + 2) + { + seq_no= 0; // So is_valid() returns false + return; + } + ++buf; + commit_id= uint8korr(buf); + } + else + commit_id= 0; } Setting commit_id=0 first instead in the if will generate smaller and faster code (setting a variable is faster than an a jmp). === modified file 'sql/log_event.h' --- sql/log_event.h 2013-06-06 15:51:28 +0000 +++ sql/log_event.h 2013-08-29 08:18:22 +0000 @@ -1317,9 +1317,9 @@ class Log_event @see do_apply_event */ - int apply_event(Relay_log_info const *rli) + int apply_event(struct rpl_group_info *rgi) What is the simple rule to know when to use Relay_log_info and when to use rpl_group_info ? If I understand things correct, relay_log_info should only be used for direct manipulation of data in the relay log. By the way, you should probably change 'struct rpl_group_info' to just rpl_group_info; Now you are mixing both versions in the code. === modified file 'sql/log_event_old.cc' --- sql/log_event_old.cc 2013-02-19 10:45:29 +0000 +++ sql/log_event_old.cc 2013-08-29 08:18:22 +0000 @@ -67,7 +68,7 @@ Old_rows_log_event::do_apply_event(Old_r do_apply_event(). We still check here to prevent future coding errors. */ - DBUG_ASSERT(rli->sql_thd == ev_thd); + DBUG_ASSERT(rgi->thd == ev_thd); Should we change all test for rli->sql_thd to rgi->thd ? You have at least this test in log_event.c and slave.cc. I assume that rli should not have a sql_thd member anymore ? <cut> === added file 'sql/rpl_parallel.cc' --- sql/rpl_parallel.cc 1970-01-01 00:00:00 +0000 +++ sql/rpl_parallel.cc 2013-08-29 08:18:22 +0000 @@ -0,0 +1,699 @@ +#include "my_global.h" +#include "rpl_parallel.h" +#include "slave.h" +#include "rpl_mi.h" + + +/* + Code for optional parallel execution of replicated events on the slave. + + ToDo list: + + - Review every field in Relay_log_info, and all code that accesses it. + Split out the necessary parts into rpl_group_info, to avoid conflicts + between parallel execution of events. (Such as deferred events ...) + + - Error handling. If we fail in one of multiple parallel executions, we + need to make a best effort to complete prior transactions and roll back + following transactions, so slave binlog position will be correct. + And all the retry logic for temporary errors like deadlock. + + - Stopping the slave needs to handle stopping all parallel executions. And + the logic in sql_slave_killed() that waits for current event group to + complete needs to be extended appropriately... + + - Audit the use of Relay_log_info::data_lock. Make sure it is held + correctly in all needed places also when using parallel replication. + + - We need some user-configurable limit on how far ahead the SQL thread will + fetch and queue events for parallel execution (otherwise if slave gets + behind we will fill up memory with pending malloc()'ed events). + + - Fix update of relay-log.info and master.info. In non-GTID replication, + they must be serialised to preserve correctness. In GTID replication, we + should not update them at all except at slave thread stop. + + - All the waits (eg. in struct wait_for_commit and in + rpl_parallel_thread_pool::get_thread()) need to be killable. And on kill, + everything needs to be correctly rolled back and stopped in all threads, + to ensure a consistent slave replication state. I can add kill checking in all functions that waits for a mutex. Will do it with adding enter_cond()/exit_cond() in all functions waiting for mysql_cond_wait(). + + - Handle the case of a partial event group. This occurs when the master + crashes in the middle of writing the event group to the binlog. The + slave rolls back the transaction; parallel execution needs to be able + to deal with this wrt. commit_orderer and such. + + - Relay_log_info::is_in_group(). This needs to be handled correctly in all + callers. I think it needs to be split into two, one version in + Relay_log_info to be used from next_event() in slave.cc, one to be used in + per-transaction stuff. + + - We should fail if we connect to the master with opt_slave_parallel_threads + greater than zero and master does not support GTID. Just to avoid a bunch + of potential problems, we won't be able to do any parallel replication + in this case anyway. If the master doesn't support GTID, I think we should instead of stopping, just issue a warning in the log and continue with normal replication. Otherwise we would have problems when using multi-source replication if a single slave doesn't support GTID. +*/ + +struct rpl_parallel_thread_pool global_rpl_thread_pool; + + +static void +rpt_handle_event(rpl_parallel_thread::queued_event *qev, + struct rpl_parallel_thread *rpt) +{ + int err; + struct rpl_group_info *rgi= qev->rgi; + Relay_log_info *rli= rgi->rli; + THD *thd= rgi->thd; + + thd->rli_slave= rli; + thd->rpl_filter = rli->mi->rpl_filter; + /* ToDo: Get rid of rli->group_info, it is not thread safe. */ + rli->group_info= rgi; + + /* ToDo: Access to thd, and what about rli, split out a parallel part? */ + mysql_mutex_lock(&rli->data_lock); + err= apply_event_and_update_pos(qev->ev, thd, rgi, rpt); We should also get rid of the thd argument and change the function to be a bool so that we can return an error. + /* ToDo: error handling. */ +} + + +pthread_handler_t +handle_rpl_parallel_thread(void *arg) +{ <cut> + mysql_mutex_lock(&rpt->LOCK_rpl_thread); + rpt->thd= thd; + + while (rpt->delay_start) + mysql_cond_wait(&rpt->COND_rpl_thread, &rpt->LOCK_rpl_thread); A much easier way to achive the wait is to have the caller lock mysql_mutex_lock(&rpt->LOCK_rpl_thread) and then just release it when it's time for this thread to continue. - No need for signaling or checking if the thread got the signal - No need to have a cond wait here - No need to have a delay_start flag + rpt->running= true; + + while (!rpt->stop && !thd->killed) + { + rpl_parallel_thread *list; + + old_msg= thd->proc_info; + thd->enter_cond(&rpt->COND_rpl_thread, &rpt->LOCK_rpl_thread, + "Waiting for work from SQL thread"); + while (!rpt->stop && !thd->killed && !(events= rpt->event_queue)) + mysql_cond_wait(&rpt->COND_rpl_thread, &rpt->LOCK_rpl_thread); Add comment: /* Mark that this thread is now executing */ + rpt->free= false; + rpt->event_queue= rpt->last_in_queue= NULL; + thd->exit_cond(old_msg); + Note that events may be undefined here if thd->killed was set above (for the first loop) It would be good to check for thd->killed or rpt->stop here and then abort the loop. Another question: do we really need rpt->stop ? Isn't it enough with using the thd->killed for this ? Otherwise we would need to check for both rpt->top and thd->killed in every loop that may be aborted. + more_events: + while (events) + { + struct rpl_parallel_thread::queued_event *next= events->next; + Log_event_type event_type= events->ev->get_type_code(); + rpl_group_info *rgi= events->rgi; + rpl_parallel_entry *entry= rgi->parallel_entry; + uint64 wait_for_sub_id; + uint64 wait_start_sub_id; + bool end_of_group; + Add a comment why this is handled here and not in rpt_handle_event() + if (event_type == GTID_EVENT) + { + in_event_group= true; Add a comment what the following variable stands for + group_standalone= + (0 != (static_cast<Gtid_log_event *>(events->ev)->flags2 & + Gtid_log_event::FL_STANDALONE)); + Rest of this file will be reviewed tomorrow. <cut> === modified file 'sql/rpl_rli.cc' --- sql/rpl_rli.cc 2013-06-06 15:51:28 +0000 +++ sql/rpl_rli.cc 2013-08-29 08:18:22 +0000 @@ -1194,13 +1194,15 @@ bool Relay_log_info::cached_charset_comp void Relay_log_info::stmt_done(my_off_t event_master_log_pos, - time_t event_creation_time, THD *thd) + time_t event_creation_time, THD *thd, + struct rpl_group_info *rgi) Do we need thd anymore as this is in rgi->thd? If they are not the same, please add a comment about this. It's not otherwise clear when to use rgi->thd and thd as this function uses both. @@ -1265,7 +1267,7 @@ void Relay_log_info::cleanup_context(THD { DBUG_ENTER("Relay_log_info::cleanup_context"); - DBUG_ASSERT(sql_thd == thd); + DBUG_ASSERT(opt_slave_parallel_threads > 0 || sql_thd == thd); Add a comment which THD is passed to the function. <cut> + +int +event_group_new_gtid(rpl_group_info *rgi, Gtid_log_event *gev) +{ + uint64 sub_id= rpl_global_gtid_slave_state.next_subid(gev->domain_id); + if (!sub_id) + { Add a comment that this is an end of memory error. <cut> + +void +delete_or_keep_event_post_apply(Relay_log_info *rli, + Log_event_type typ, Log_event *ev) +{ + /* + ToDo: This needs to work on rpl_group_info, not Relay_log_info, to be + thread-safe for parallel replication. + */ + switch (typ) { + case FORMAT_DESCRIPTION_EVENT: + /* + Format_description_log_event should not be deleted because it + will be used to read info about the relay log's format; + it will be deleted when the SQL thread does not need it, + i.e. when this thread terminates. + */ + break; + case ANNOTATE_ROWS_EVENT: + /* + Annotate_rows event should not be deleted because after it has + been applied, thd->query points to the string inside this event. + The thd->query will be used to generate new Annotate_rows event + during applying the subsequent Rows events. + */ + rli->set_annotate_event((Annotate_rows_log_event*) ev); I assume we should create relay_group_info->set_annotate_event instead of the above. + break; + case DELETE_ROWS_EVENT: + case UPDATE_ROWS_EVENT: + case WRITE_ROWS_EVENT: + /* + After the last Rows event has been applied, the saved Annotate_rows + event (if any) is not needed anymore and can be deleted. + */ + if (((Rows_log_event*)ev)->get_flags(Rows_log_event::STMT_END_F)) + rli->free_annotate_event(); + /* fall through */ + default: + DBUG_PRINT("info", ("Deleting the event after it has been executed")); + if (!rli->is_deferred_event(ev)) + delete ev; I assume that is_deferred_event should also be moved to relay_group_info. + break; + } +} + #endif === modified file 'sql/rpl_rli.h' --- sql/rpl_rli.h 2013-06-06 15:51:28 +0000 +++ sql/rpl_rli.h 2013-08-29 08:18:22 +0000 @@ -22,6 +22,7 @@ #include "log.h" /* LOG_INFO, MYSQL_BIN_LOG */ #include "sql_class.h" /* THD */ #include "log_event.h" +#include "rpl_parallel.h" struct RPL_TABLE_LIST; class Master_info; @@ -52,6 +53,8 @@ class Master_info; *****************************************************************************/ +struct rpl_group_info; + class Relay_log_info : public Slave_reporting_capability { public: @@ -311,13 +314,9 @@ class Relay_log_info : public Slave_repo char slave_patternload_file[FN_REFLEN]; size_t slave_patternload_file_size; - /* - Current GTID being processed. - The sub_id gives the binlog order within one domain_id. A zero sub_id - means that there is no active GTID. - */ - uint64 gtid_sub_id; - rpl_gtid current_gtid; + /* ToDo: We need to remove this, always use the per-transaction one to work with parallel replication. */ + struct rpl_group_info *group_info; + rpl_parallel parallel; Relay_log_info(bool is_slave_recovery); ~Relay_log_info(); @@ -459,7 +458,8 @@ class Relay_log_info : public Slave_repo the <code>Seconds_behind_master</code> field. */ void stmt_done(my_off_t event_log_pos, - time_t event_creation_time, THD *thd); + time_t event_creation_time, THD *thd, + struct rpl_group_info *rgi); /** @@ -594,6 +594,60 @@ class Relay_log_info : public Slave_repo }; +/* + This is data for various state needed to be kept for the processing of + one event group in the SQL thread. + + For single-threaded replication it is linked from the RLI, for parallel + replication it is linked into each event group being executed in parallel. +*/ Could you please rewrite the above to make it more clear. After discussing with you, I think the following could be added: /* In single-threaded replication, there will be one global rpl_group_info and one global RPLI per master connection. They will be linked together. In parallel replication, there will be one rpl_group_info object for each running thd. All rpl_group_info will share the same RLI. */ +struct rpl_group_info +{ + Relay_log_info *rli; + THD *thd; <cut> === modified file 'sql/sql_class.cc' --- sql/sql_class.cc 2013-06-06 15:51:28 +0000 +++ sql/sql_class.cc 2013-08-29 08:18:22 +0000 +void +wait_for_commit::wakeup() +{ + /* + We signal each waiter on their own condition and mutex (rather than using + pthread_cond_broadcast() or something like that). + + Otherwise we would need to somehow ensure that they were done + waking up before we could allow this THD to be destroyed, which would + be annoying and unnecessary. + */ + mysql_mutex_lock(&LOCK_wait_commit); + waiting_for_commit= false; + mysql_cond_signal(&COND_wait_commit); + mysql_mutex_unlock(&LOCK_wait_commit); +} In this particular case it looks safe to move the cond signal out of the mutex. I don't see how anyone could miss the signal. + + +/* + Register that the next commit of this THD should wait to complete until + commit in another THD (the waitee) has completed. + + The wait may occur explicitly, with the waiter sitting in + wait_for_prior_commit() until the waitee calls wakeup_subsequent_commits(). + + Alternatively, the TC (eg. binlog) may do the commits of both waitee and + waiter at once during group commit, resolving both of them in the right + order. + + Only one waitee can be registered for a waiter; it must be removed by + wait_for_prior_commit() or unregister_wait_for_prior_commit() before a new + one is registered. But it is ok for several waiters to register a wait for + the same waitee. It is also permissible for one THD to be both a waiter and + a waitee at the same time. +*/ +void +wait_for_commit::register_wait_for_prior_commit(wait_for_commit *waitee) +{ + waiting_for_commit= true; + DBUG_ASSERT(!this->waitee /* No prior registration allowed */); + this->waitee= waitee; + + mysql_mutex_lock(&waitee->LOCK_wait_commit); + /* + If waitee is in the middle of wakeup, then there is nothing to wait for, + so we need not register. This is necessary to avoid a race in unregister, + see comments on wakeup_subsequent_commits2() for details. + */ + if (waitee->wakeup_subsequent_commits_running) + waiting_for_commit= false; + else + { Please add a comment here what the following code is doing. As this is two different lists involved, it's not trivial code. You also seam to use the list next_subsequent_commit for something different in queue_for_group_commit() code. + this->next_subsequent_commit= waitee->subsequent_commits_list; + waitee->subsequent_commits_list= this; + } + mysql_mutex_unlock(&waitee->LOCK_wait_commit); +} <cut> +/* + Wakeup anyone waiting for us to have committed. + + Note about locking: + + We have a potential race or deadlock between wakeup_subsequent_commits() in + the waitee and unregister_wait_for_prior_commit() in the waiter. + + Both waiter and waitee needs to take their own lock before it is safe to take + a lock on the other party - else the other party might disappear and invalid + memory data could be accessed. But if we take the two locks in different + order, we may end up in a deadlock. + + The waiter needs to lock the waitee to delete itself from the list in + unregister_wait_for_prior_commit(). Thus wakeup_subsequent_commits() can not + hold its own lock while locking waiters, lest we deadlock. lest ? + So we need to prevent unregister_wait_for_prior_commit() running while wakeup + is in progress - otherwise the unregister could complete before the wakeup, + leading to incorrect spurious wakeup or accessing invalid memory. + <cut> + Then also register_wait_for_prior_commit() needs to check if + wakeup_subsequent_commits() is running, and skip the registration if + so. This is needed in case a new waiter manages to register itself and + immediately try to unregister while wakeup_subsequent_commits() is + running. Else the new waiter would also wait rather than unregister, but it + would not be woken up until next wakeup, which could be potentially much + later than necessary. +*/ Add new line here +void +wait_for_commit::wakeup_subsequent_commits2() +{ <cut> === modified file 'sql/sql_class.h' --- sql/sql_class.h 2013-06-06 15:51:28 +0000 +++ sql/sql_class.h 2013-08-29 08:18:22 +0000 @@ -1553,6 +1553,116 @@ class Global_read_lock }; +/* + Class to facilitate the commit of one transactions waiting for the commit of + another transaction to complete first. + + This is used during (parallel) replication, to allow different transactions + to be applied in parallel, but still commit in order. + + The transaction that wants to wait for a prior commit must first register + to wait with register_wait_for_prior_commit(waitee). Such registration + must be done holding the waitee->LOCK_wait_commit, to prevent the other + THD from disappearing during the registration. + + Then during commit, if a THD is registered to wait, it will call + wait_for_prior_commit() as part of ha_commit_trans(). If no wait is + registered, or if the waitee for has already completed commit, then + wait_for_prior_commit() returns immediately. + + And when a THD that may be waited for has completed commit (more precisely + commit_ordered()), then it must call wakeup_subsequent_commits() to wake + up any waiters. Note that this must be done at a point that is guaranteed + to be later than any waiters registering themselves. It is safe to call + wakeup_subsequent_commits() multiple times, as waiters are removed from + registration as part of the wakeup. + + The reason for separate register and wait calls is that this allows to + register the wait early, at a point where the waited-for THD is known to + exist. And then the actual wait can be done much later, where the + waited-for THD may have been long gone. By registering early, the waitee + can signal before disappearing. +*/ Instead of taking locks to ensure that THD will disapper, why not keep all THD objects around until all transactions in a group has been executed? +struct wait_for_commit +{ <cut> + bool waiting_for_commit; + /* + Flag set when wakeup_subsequent_commits_running() is active, see commonts commonts -> comments === modified file 'sql/sys_vars.cc' --- sql/sys_vars.cc 2013-06-07 07:31:11 +0000 +++ sql/sys_vars.cc 2013-08-29 08:18:22 +0000 <cut> +static bool +check_slave_parallel_threads(sys_var *self, THD *thd, set_var *var) +{ + bool running; + + mysql_mutex_lock(&LOCK_active_mi); + running= master_info_index->give_error_if_slave_running(); + mysql_mutex_unlock(&LOCK_active_mi); + if (running) + return true; + + return false; +} + +static bool +fix_slave_parallel_threads(sys_var *self, THD *thd, enum_var_type type) +{ + bool running; + bool err= false; + + mysql_mutex_unlock(&LOCK_global_system_variables); + mysql_mutex_lock(&LOCK_active_mi); + running= master_info_index->give_error_if_slave_running(); + mysql_mutex_unlock(&LOCK_active_mi); You can call check_slave_parallel_threads() above to reuse code. + if (running || rpl_parallel_change_thread_count(&global_rpl_thread_pool, + opt_slave_parallel_threads)) + err= true; + mysql_mutex_lock(&LOCK_global_system_variables); + + return err; +} + + +static Sys_var_ulong Sys_slave_parallel_threads( + "slave_parallel_threads", + "If non-zero, number of threads to spawn to apply in parallel events " + "on the slave that were group-committed on the master or were logged " + "with GTID in different replication domains.", + GLOBAL_VAR(opt_slave_parallel_threads), CMD_LINE(REQUIRED_ARG), + VALID_RANGE(0,16383), DEFAULT(0), BLOCK_SIZE(1), NO_MUTEX_GUARD, + NOT_IN_BINLOG, ON_CHECK(check_slave_parallel_threads), + ON_UPDATE(fix_slave_parallel_threads)); #endif We have talked about having 1 to be the minimum, which would give the old behavior. <cut> Overal things looks very good and it should not be too much work to add the things you have written on the todo part.a Regards Monty

2 10

[Maria-developers] [Spatial] On current implementation approach
by Mateusz Loskot 25 Sep '13

25 Sep '13

Hi, I'm going to ask question about how the current Spatial Extensions are implemented. I have spent some time reading the source code in the current trunk (spatial.h|cc, gcal*.h|cc, related Field and Item definitions, etc.), so I have a rough understanding of the overall structure, how the geometry data types are implemented and exposed to SQL, how the spatial functions are defined and registered. I did not looked into details of implementation of geospatial algorithms, but that's too low level for the question I'm going to ask. Initially, I was going to ask very detailed question, listing all the relevant code definitions and asking separately about each of them, but that's not necessary at this stage, I think. Instead, I'm going to simplify and ask about the bigger picture, more about MariaDB extensions API: 1. Is it possible to implement MariaDB extensions like Spatial (custom type + set of functions) without such a tight coupling with the internal implementation of the type system (without messing Field class with geometry types directly, etc.)? 2. Is it possible to implement Spatial using User-Defined Functions (UDF) defined in shared binary? 3. What is the reason behind using Well-Known-Binary (WKB) stream of bytes to transport geometry values into/from functions? Is it due to limitations of MariaDB type system where String is the only universal carrier for complex data? This concern is related to necessity of encoding/decoding WKB when chaining spatial function calls, and possibilities to avoid it. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net "Participation in this whole process is a form of torture" ~~ Szalony

3 10

[Maria-developers] close mdev-4387
by Roberto Spadim 23 Sep '13

23 Sep '13

hi guys, please close https://mariadb.atlassian.net/browse/MDEV-4387 it's duplicated by alread commited https://mariadb.atlassian.net/browse/MDEV-5025 thanks =) -- Roberto Spadim

1 0

Re: [Maria-developers] [Roles] Final Status
by Vicentiu Ciorbaru 23 Sep '13

23 Sep '13

Hi Sergei! >> Even though I didn't manage to implement everything yet, I want to >> finish this project and get it ready for an upstream push, outside of >> GSoC. > 1. I'd need either from you in email (but then cc: it to > maria-developers) or in your blog (or both) a statement about the > license. A simple one would be to say that this your code is available > to us under BSD license. Another option is our MariaDB Contributor > Agreement: https://mariadb.com/kb/en/mca/ I hereby declare that all my work done so far (and future) by me on the Roles project, is released under the BSD licence. I will also make a statement on my blog in my wrap up post a bit later. If there are any more clarifications needed, let me know! Regards, Vicențiu

2 2