October 2016 - developers - lists.mariadb.org

Re: [Maria-developers] [Commits] aa9bd40: MDEV-10824 - Crash in CREATE OR REPLACE TABLE t1 AS SELECT spfunc()
by Vicențiu Ciorbaru 24 Oct '16

24 Oct '16

Hi Sergey! I think you commited the AAA.test file by mystake. Regards, Vicentiu On Mon, 24 Oct 2016 at 13:47 Sergey Vojtovich <svoj(a)mariadb.org> wrote: > revision-id: aa9bd40f8067e1421ad71d2ada367544a6db78ca > (mariadb-10.0.27-8-gaa9bd40) > parent(s): 4dfb6a3f54cfb26535636197cc5fa70fe5bacc2e > committer: Sergey Vojtovich > timestamp: 2016-10-24 15:26:11 +0400 > message: > > MDEV-10824 - Crash in CREATE OR REPLACE TABLE t1 AS SELECT spfunc() > > Code flow hit incorrect branch while closing table instances before > removal. > This branch expects thread to hold open table instance, whereas CREATE OR > REPLACE doesn't actually hold open table instance. > > Before CREATE OR REPLACE TABLE it was impossible to hit this condition in > LTM_PRELOCKED mode, thus the problem didn't expose itself during DROP TABLE > or DROP DATABASE. > > Fixed by adjusting condition to take into account LTM_PRELOCKED mode, > which can > be set during CREATE OR REPLACE TABLE. > > --- > mysql-test/r/create_or_replace.result | 11 +++++++++++ > mysql-test/t/AAA.test | 24 ++++++++++++++++++++++++ > mysql-test/t/create_or_replace.test | 12 ++++++++++++ > sql/sql_parse.cc | 12 ------------ > sql/sql_table.cc | 3 ++- > 5 files changed, 49 insertions(+), 13 deletions(-) > > diff --git a/mysql-test/r/create_or_replace.result > b/mysql-test/r/create_or_replace.result > index 3a894e9..a43dc2e 100644 > --- a/mysql-test/r/create_or_replace.result > +++ b/mysql-test/r/create_or_replace.result > @@ -442,3 +442,14 @@ KILL QUERY con_id; > ERROR 70100: Query execution was interrupted > drop table t1; > DROP TABLE t2; > +# > +# MDEV-10824 - Crash in CREATE OR REPLACE TABLE t1 AS SELECT spfunc() > +# > +CREATE TABLE t1(a INT); > +CREATE FUNCTION f1() RETURNS VARCHAR(16383) RETURN 'test'; > +CREATE OR REPLACE TABLE t1 AS SELECT f1(); > +LOCK TABLE t1 WRITE; > +CREATE OR REPLACE TABLE t1 AS SELECT f1(); > +UNLOCK TABLES; > +DROP FUNCTION f1; > +DROP TABLE t1; > diff --git a/mysql-test/t/AAA.test b/mysql-test/t/AAA.test > new file mode 100644 > index 0000000..22e22dd > --- /dev/null > +++ b/mysql-test/t/AAA.test > @@ -0,0 +1,24 @@ > +CREATE TABLE t1(a INT); > +DELIMITER $$; > +CREATE FUNCTION f2() RETURNS VARCHAR(16383) RETURN 'test'; > +CREATE FUNCTION f1() RETURNS VARCHAR(16383) > +BEGIN > + INSERT INTO t1 VALUES(1); > + RETURN 'test'; > +END; > +$$ > +CREATE PROCEDURE p1() CREATE OR REPLACE TABLE t1 AS SELECT f2(); > +$$ > +DELIMITER ;$$ > + > +CALL p1; > + > +#CREATE OR REPLACE TABLE t1 AS SELECT f1(); > +LOCK TABLE t1 WRITE; > +#CREATE OR REPLACE TABLE t1 AS SELECT f1(); > +UNLOCK TABLES; > + > +DROP PROCEDURE p1; > +DROP FUNCTION f1; > +DROP FUNCTION f2; > +DROP TABLE t1; > diff --git a/mysql-test/t/create_or_replace.test > b/mysql-test/t/create_or_replace.test > index 7bba2b3..b37417f 100644 > --- a/mysql-test/t/create_or_replace.test > +++ b/mysql-test/t/create_or_replace.test > @@ -386,3 +386,15 @@ drop table t1; > # Cleanup > # > DROP TABLE t2; > + > +--echo # > +--echo # MDEV-10824 - Crash in CREATE OR REPLACE TABLE t1 AS SELECT > spfunc() > +--echo # > +CREATE TABLE t1(a INT); > +CREATE FUNCTION f1() RETURNS VARCHAR(16383) RETURN 'test'; > +CREATE OR REPLACE TABLE t1 AS SELECT f1(); > +LOCK TABLE t1 WRITE; > +CREATE OR REPLACE TABLE t1 AS SELECT f1(); > +UNLOCK TABLES; > +DROP FUNCTION f1; > +DROP TABLE t1; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index cbf723c..70511fc 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2858,12 +2858,6 @@ case SQLCOM_PREPARE: > } > > /* > - For CREATE TABLE we should not open the table even if it exists. > - If the table exists, we should either not create it or replace it > - */ > - lex->query_tables->open_strategy= TABLE_LIST::OPEN_STUB; > - > - /* > If we are a slave, we should add OR REPLACE if we don't have > IF EXISTS. This will help a slave to recover from > CREATE TABLE OR EXISTS failures by dropping the table and > @@ -8225,12 +8219,6 @@ bool create_table_precheck(THD *thd, TABLE_LIST > *tables, > if (check_fk_parent_table_access(thd, &lex->create_info, > &lex->alter_info, create_table->db)) > goto err; > > - /* > - For CREATE TABLE we should not open the table even if it exists. > - If the table exists, we should either not create it or replace it > - */ > - lex->query_tables->open_strategy= TABLE_LIST::OPEN_STUB; > - > error= FALSE; > > err: > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 7cf31ee..050a338 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -2464,7 +2464,8 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST > *tables, bool if_exists, > if (table_type && table_type != view_pseudo_hton) > ha_lock_engine(thd, table_type); > > - if (thd->locked_tables_mode) > + if (thd->locked_tables_mode == LTM_LOCK_TABLES || > + thd->locked_tables_mode == LTM_PRELOCKED_UNDER_LOCK_TABLES) > { > if (wait_while_table_is_used(thd, table->table, > HA_EXTRA_NOT_USED)) > { > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Patch for atomics on MIPS in mroonga
by Kristian Nielsen 22 Oct '16

22 Oct '16

Who should be contacted about issues in the mroonga storage engine? The attached patch is from Debian Bug#838914 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838914 Apparently, libatomic is needed on this platform to support 64-bit atomic operations. The patch looks reasonable and should probably be upstreamed. But I am not sure how the mroonga storage engine is maintained - should this go directly into MariaDB? If there is an upstream maintained mroonga storage engine, probably it should preferably go there first? - Kristian.

2 1

[Maria-developers] 161M for storage/tokudb isn't friendly
by MARK CALLAGHAN 21 Oct '16

21 Oct '16

Filed https://jira.mariadb.org/browse/MDEV-11101 but I have no idea who will fix this. All other engines use less than 20M while tokudb uses 161M including one file in mysql-test that is more than 30M. Lots of time, network and disk space is being wasted by this. -- Mark Callaghan mdcallag(a)gmail.com

1 0

Re: [Maria-developers] [MariaDB/server] MDEV-11064 - Restrict the speed of reading binlog from Master (#246)
by Kristian Nielsen 21 Oct '16

21 Oct '16

vinchen <notifications(a)github.com> writes: > cli_safe_read_reallen() and and my_net_read_packet_reallen() is a good > way to fix the ABI problem. I will fix it like this. > > And the minimum precision is second in slave_sleep(), and it also a > the mutex. I think it is too heavy in most case. (It will wait > millisecond seconds usually) > So I will use my_sleep when wait less than 1 second, otherwise use slave_sleep. > > Is it ok? Yes, it looks fine thanks! And thanks for the explanation, yes now I see that usually the sleep will be quite small, unless there is a huge event. So I agree with using my_sleep() in the common case. I'll merge it like this after checking that everything compiles and tests ok. Thanks! - Kristian.

1 0

Re: [Maria-developers] [MariaDB/server] Restrict the speed of reading binlog from Master (#246)
by Kristian Nielsen 19 Oct '16

19 Oct '16

GCSAdmin <notifications(a)github.com> writes: > In some case, the speed of reading binlog from master is high, especially when doing a new replica. > It would bring the high traffic in master. > So We introduce a new variable "read_binlog_speed_limit" to control the binlog reading rate for IO thread to solve the problem. > It can work when slave_compressed_protocol is on. > But it maybe doesn't work well when the binlog event is very big. > You can view, comment on, or merge this pull request online at: > > https://github.com/MariaDB/server/pull/246 > -- Patch Links -- > > https://github.com/MariaDB/server/pull/246.patch > https://github.com/MariaDB/server/pull/246.diff Overall this looks clean and simple. There is one problem. The patch adds a field real_network_read_len to the NET structure. This will break the client library ABI, because NET is a part of MYSQL. So this would cause client programs to crash if they are linked with a different version of the client library. So this needs to be changed (if I understand correctly). One option might be to introduce new functions like cli_safe_read_reallen() and my_net_read_packet_reallen(), which return in addition the actual amount of bytes read from the server. The old cli_safe_read() and my_net_read_packet() could then become simple wrapper functions around those. And cli_safe_read_reallen() can be used in read_event() in sql/slave.cc. A smaller issue is that in case of a large packet, a large my_sleep() may be invoked, which will cause STOP SLAVE to hang. I think this can be solved simply by calling slave_sleep() instead, it handles terminating the wait early if interrupted by STOP SLAVE. Detailed comments on the patch below. I rebased the series against latest 10.2 to get a clean diff (the pull request includes a couple merges against the main 10.2 tree, these changes are unrelated to the patch). The rebase is in https://github.com/knielsen/server/tree/GCSAdmin-10.2-binlog-speed-limit-2 > diff --git a/include/mysql.h.pp b/include/mysql.h.pp > index 857f5b9..1da038c 100644 > --- a/include/mysql.h.pp > +++ b/include/mysql.h.pp > @@ -35,6 +35,7 @@ typedef struct st_net { > my_bool thread_specific_malloc; > unsigned char compress; > my_bool unused3; > + unsigned long real_network_read_len; As explained above, I believe this would break the ABI (that's the purpose of mysql.h.pp, to catch such problems). > diff --git a/sql/slave.cc b/sql/slave.cc > index 20bf68e..52bb668 100644 > --- a/sql/slave.cc > +++ b/sql/slave.cc > @@ -3307,13 +3308,14 @@ static int request_dump(THD *thd, MYSQL* mysql, Master_info* mi, > try a reconnect. We do not want to print anything to > the error log in this case because this a anormal > event in an idle server. > + network_read_len get the real network read length in VIO, especially using compressed protocol > > RETURN VALUES > 'packet_error' Error > number Length of packet > */ > > -static ulong read_event(MYSQL* mysql, Master_info *mi, bool* suppress_warnings) > +static ulong read_event(MYSQL* mysql, Master_info *mi, bool* suppress_warnings, ulong* network_read_len) Generally, lines longer than 80 characters should be avoided (coding style). > @@ -4473,6 +4479,34 @@ Stopping slave I/O thread due to out-of-memory error from master"); > goto err; > } > > + /* Control the binlog read speed of master when read_binlog_speed_limit is non-zero > + */ > + ulonglong read_binlog_speed_limit_in_bytes = opt_read_binlog_speed_limit * 1024; > + if (read_binlog_speed_limit_in_bytes) > + { > + /* prevent the tokenamount become a large value, > + for example, the IO thread doesn't work for a long time > + */ > + if (tokenamount > read_binlog_speed_limit_in_bytes * 2) > + { > + lastchecktime = my_hrtime().val; > + tokenamount = read_binlog_speed_limit_in_bytes * 2; > + } > + > + do > + { > + ulonglong currenttime = my_hrtime().val; > + tokenamount += (currenttime - lastchecktime) * read_binlog_speed_limit_in_bytes / (1000*1000); > + lastchecktime = currenttime; > + if(tokenamount < network_read_len) > + { > + ulonglong micro_sleeptime = 1000*1000 * (network_read_len - tokenamount) / read_binlog_speed_limit_in_bytes ; > + my_sleep(micro_sleeptime > 1000 ? micro_sleeptime : 1000); // at least sleep 1000 micro second > + } > + }while(tokenamount < network_read_len); > + tokenamount -= network_read_len; > + } > + As explained above, probably better to use slave_sleep() here to allow STOP SLAVE to interrupt a long sleep. Would it make sense to do this wait after calling queue_event()? This way the SQL thread can start applying the event immediately, reducing slave lag. What do you think? Thanks, - Kristian.

1 0

[Maria-developers] Consolidated patch for mdev-8646 introduced a bug MDEV-11081
by Alexander Barkov 18 Oct '16

18 Oct '16

Hello Igor, This patch: commit 2cfc450bf78c2d951729d1a0e8f731c0d987b1d5 Author: Igor Babaev <igor(a)askmonty.org> Date: Tue Feb 9 12:35:59 2016 -0800 seems to have introduced this bug: MDEV-11081 Cursor fetches NULL for aggregate functions Can you please have a look? Thanks!

1 0

[Maria-developers] Article on git rebasing - follow up from Developer Meetup
by Vicențiu Ciorbaru 17 Oct '16

17 Oct '16

Hi everyone! During the Developer Meetup in Amsterdam, we've had quite some productive discussions. One of these was on how to make use of git rebase to clean up our history before pushing final changes. The presentation on it was a bit rushed and perhaps some did not get the full gist out of it. I've decided to write an article on this and go into more details. The content should be generally useful and should give a primer to anyone not familiar with how to use git rebase. You can find the article at: http://vicentiu.ciorbaru.io/keeping-development-history-clean-using-git-reb… Hope it is of use. Regards, Vicențiu

1 0

[Maria-developers] Problem with parallel replication in 10.2
by Michael Widenius 17 Oct '16

17 Oct '16

Hi! I was trying to run a test that fails in the upcoming bb-10.2-jan on the normal 10.2 tree, when I noticed this strange issue: - Test fails with timeout when running with --debug - When looking at the trace file, I notice that we get a duplicate key error for the table gtid_slave_post (MyISAM table). Is this something normal ? To repeat: Store the included test in suite/rpl and run it with: mysql-test-run --debug --record rpl_skr Issue 2: bb-10.2-jan tree is a working tree for a merge of MariaDB 10.2 and MySQL 5.7 When running rpl_skr in 10.2 it takes 2 seconds When running it in the bb-10.2-jan tree it takes either a long time or we get a timeout. This is probably because of the new lock code in lock0lock.cc and lock0wait.cc which doesn't break conflicting transaction but instead waits for a timeout Would appreciate any help with this! Note that rpl_skr is a test that was originally part of rpl_parallel.test, but I have made it separate to be able to more easily test for this issue. Before testing bb-10.2-jan, please apply this patch that fixes one critical issue in this tree related to group-commit: diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index b69468d..0e9caed 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -4976,16 +4953,15 @@ innobase_commit( thd_wakeup_subsequent_commits(thd, 0); /* Now do a write + flush of logs. */ - if (!read_only) { - trx_commit_complete_for_mysql(trx); - } + trx_commit_complete_for_mysql(trx); trx_deregister_from_2pc(trx); } else { /* We just mark the SQL statement ended and do not do a transaction commit */ + DBUG_PRINT("info", ("Just mark SQL statement")); /* If we had reserved the auto-inc lock for some table in this SQL statement we release it now */ Regards, Monty

3 9

Re: [Maria-developers] [MariaDB/server] MDEV-11039 - Add new scheduling algorithm for reducing tail latencies (#245)
by Kristian Nielsen 17 Oct '16

17 Oct '16

Jiamin Huang <notifications(a)github.com> writes: > @sensssz pushed 1 commit. > > 5dc7ad8 Reduce conflict during in-order replication. Cool, that looks nice and simple. Thanks! - Kristian.

1 0

Re: [Maria-developers] MDEV-7145
by Kristian Nielsen 17 Oct '16

17 Oct '16

Phil Sweeney <launchpad(a)sweeneymail.com> writes: > Re: https://jira.mariadb.org/browse/MDEV-7145 > Delayed replication (a feature shipped in MySQL 5.6) has been a MariaDB > feature request since 2014, and has been listed as 'major/red' priority in > Can anyone comment on whether this is still likely to make it into 10.2. I am planning to look into this. - Kristian.

3 4