October 2015 - developers - lists.mariadb.org

[Maria-developers] Moving from Launchpad lists to something else... ?
by Colin Charles 26 Nov '15

26 Nov '15

Hi, I was speaking to Otto (CEO, MariaDB Foundation) and discussing that we should move the lists from Launchpad to something else. We obviously want to preserve the archives, and this isn’t going to happen overnight (something like an aim by the end of the year). Would you be ok with running on Google Groups? Would you prefer being on Mailman? Do you have another self-hosted or (preferably free) hosted solution as your preference? Today, we have several lists (maria-discuss, maria-developers, maria-docs) that are hosted on Launchpad. We also have a few lists (ambassadors, announce, commits, mirroring, maria-buildbot-reports) hosted on Mailman at lists.askmonty.org. Let us know what you think about this Kind Regards, Colin Charles -- Colin Charles, Chief Evangelist, MariaDB Corporation blog: http://bytebot.net/blog/ | t: +1-347-903-3201 | Skype: colincharles | Twitter: @bytebot

6 7

[Maria-developers] On the issue of Seconds_behind_master and Parallel Replication
by Kristian Nielsen 16 Nov '15

16 Nov '15

It was brought to my attention an issue with parallel replication and the Seconds_Behind_Master field of SHOW SLAVE STATUS. I have a possible patch for this, but I wanted to discuss it on the list, as it changes semantics compared to the non-parallel case. Each binlog event contains a timestamp (**) of when the event was created on the master. Whenever the slave SQL thread reads an event from the relay log, it updates the value of Seconds_Behind_Master to the difference between the slave's current time and the event's timestamp. Now in parallel replication, the SQL thread can read a large number of events from the relay log and queue them in-memory for the worker threads. So a small value of Seconds_Behind_Master means only that recent events have been queued - it might still be a long time before the worker threads have had time to actually execute all the queued events. Apparently the problem is (justified) user confusion about this queuing delay not being reflected in Seconds_Behind_Master. The same problem actually exists in the non-parallel case. In case of a large transaction, the Seconds_Behind_Master can be small even though there is still a large amount of execution time remaining for the transaction to complete on the slave. However, in the non-parallel case, at most one transaction can be involved. In the parallel case, the problem is amplified by the potential of thousands of queued transactions awaiting execution. So how to solve it? Attached is a patch that implements one possible solution: the Seconds_Behind_Master is only updated after a transaction commits, with the timestamp of the commit events. This seems more intuitive anyway. But it does introduce a semantic difference between the non-parallel and parallel behaviour for Seconds_Behind_Master. The value will in general be larger on a parallel slave than on a non-parallel slave, for the same actual slave lag. Monty suggested changing the behaviour also for non-parallel mode - letting Seconds_Behind_Master reflect only events actually committed, not just read from the relay log. This would introduce an incompatible behaviour for Seconds_Behind_Master, but could perhaps be done for 10.1, if desired. Doing it in stable 10.0 would be more drastic. So any opinions on this? - Should Seconds_Behind_Master be changed as per above in parallel replication (from 10.0 on)? - If not, any suggestion for another semantics for Seconds_Behind_Master in parallel replication? - If so, should the change to Seconds_Behind_Master also be done in the non-parallel case in 10.1? What about 10.0? - Any comments on the patch? Here is a test case that shows the problem. It simulates a slave catching up with a bunch of INSERT(SLEEP(1)). Without my patch, Seconds_Behind_Master=2 just after starting the parallel slave, while the non-parallel slave shows Seconds_Behind_Master=9. With my patch, the parallel slave shows Seconds_Behind_Master=11. ----------------------------------------------------------------------- --source include/have_innodb.inc --source include/have_binlog_format_statement.inc --let $rpl_topology=1->2 --source include/rpl_init.inc # Test various aspects of parallel replication. --connection server_1 ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; CALL mtr.add_suppression("Unsafe statement written to the binary log using statement format"); CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; --save_master_pos --connection server_2 --sync_with_master --source include/stop_slave.inc SET GLOBAL slave_parallel_threads=1; --connection server_1 --disable_warnings INSERT INTO t1 VALUES (1, SLEEP(1)); INSERT INTO t1 VALUES (2, SLEEP(1)); INSERT INTO t1 VALUES (3, SLEEP(1)); INSERT INTO t1 VALUES (4, SLEEP(1)); INSERT INTO t1 VALUES (5, SLEEP(1)); INSERT INTO t1 VALUES (6, SLEEP(1)); INSERT INTO t1 VALUES (7, SLEEP(1)); INSERT INTO t1 VALUES (8, SLEEP(1)); INSERT INTO t1 VALUES (9, SLEEP(1)); INSERT INTO t1 VALUES (10, SLEEP(1)); --disable_warnings --save_master_pos --connection server_2 --source include/start_slave.inc SELECT SLEEP(1); --let $status_items= Seconds_Behind_Master --source include/show_slave_status.inc --sync_with_master --let $status_items= Seconds_Behind_Master --source include/show_slave_status.inc # Clean up. --connection server_2 --source include/stop_slave.inc SET GLOBAL slave_parallel_threads=DEFAULT; --source include/start_slave.inc --connection server_1 DROP TABLE t1; --source include/rpl_end.inc ----------------------------------------------------------------------- Thanks, - Kristian. ** The full detail is a bit more complex. Some events actually contains the time when the corresponding query _started_ executing, and another value which is the duration of execution. These are added by the slave to get the timestamp for the creation of the event (at the end of query exection). Also, the slave adjusts the timestamp with any time difference between the master and slave clock (eg. time zone for example).

6 10

[Maria-developers] MDEV-8109 unexpected CAST result
by Alexander Barkov 11 Nov '15

11 Nov '15

Hi Serg, Please review my patch for MDEV-8109. Thanks!

2 1

[Maria-developers] Python Storage Engine
by Kim Ebert 02 Nov '15

02 Nov '15

Greetings All, I wanted to share the beginnings of a project. This is a Python Storage Engine. The idea is to plumb in the necessary C to Python calls to allow a storage engine to be written in Python instead of C or C++. This may be valuable to those who want to prototype a database engine in Python, or if the performance is sufficient for the use case, for production. I haven't taken things to far yet, but I think it gives a rough idea of how to move forward with adding the other calls. In my particular case I'm using it to prototype connecting to a search API, like Solr, to provide an SQL like interface. I did have a quick question. I want to license it as a more permissive license, but since I'm using the example engine, does that mean that I need to use the GPL license? Or can I go with the more permissive MIT-new license? https://github.com/perfectsearch/mariadb-plugin-python-engine Thoughts and/or feedback are/is appreciated. Thanks, Kim

3 2

Re: [Maria-developers] MDEV-8692 prefschema test failures on ARM (on Debian build system)
by Sergei Golubchik 28 Oct '15

28 Oct '15

Hi, Alexander! On Oct 23, Alexander Barkov wrote: > Hi, > > - the variant with MY_PERF_CYCLES generally works, but does not fix the > test failure. What do you mean by that? How does it "work" if it does not fix the test failure? > - the variant with MY_CLOCK_REALTIME_FOR_CYCLES works fine > and fixed the problem. > > But it's not really "CYCLES". Perhaps this is not a problem, > because on some other platforms we also fallback cycles to nanoseconds, That commit 49667f0 falls back from cycles to nanoseconds explicitly. So it's ok not to support cycles, as far as I understand, and there's no need to do nanoseconds in place of cycles like your MY_CLOCK_REALTIME_FOR_CYCLES. By the way, why 49667f0 didn't fix the problem? What happens on ARMv7 - no 'cycles' timer at all in perfschema, or there is one and it works incorrectly? > diff --git a/mysql-test/suite/perfschema/t/func_file_io.test b/mysql-test/suite/perfschema/t/func_file_io.test > index 2055fa9..5894cb0 100644 > --- a/mysql-test/suite/perfschema/t/func_file_io.test > +++ b/mysql-test/suite/perfschema/t/func_file_io.test > @@ -43,6 +43,11 @@ TRUNCATE TABLE performance_schema.events_waits_current; > > SELECT * FROM t1 WHERE id = 1; > > +--vertical_results > +SELECT * FROM performance_schema.events_waits_history_long > +WHERE EVENT_NAME = 'wait/io/file/myisam/dfile'; > +--horizontal_results forgotten debug output? > + > SET @before_count = (SELECT SUM(TIMER_WAIT) > FROM performance_schema.events_waits_history_long > WHERE (EVENT_NAME = 'wait/io/file/myisam/dfile') > diff --git a/mysys/my_rdtsc.c b/mysys/my_rdtsc.c > index 028c7f8..a5ed808 100644 > --- a/mysys/my_rdtsc.c > +++ b/mysys/my_rdtsc.c > @@ -533,6 +590,10 @@ void my_timer_init(MY_TIMER_INFO *mti) > mti->cycles.routine= MY_TIMER_ROUTINE_ASM_GCC_SPARC32; > #elif defined(__sgi) && defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_SGI_CYCLE) > mti->cycles.routine= MY_TIMER_ROUTINE_SGI_CYCLE; > +#elif defined(MY_CLOCK_REALTIME_FOR_CYCLES) > + mti->cycles.routine= MY_TIMER_ROUTINE_CLOCK_GETTIME; > +#elif defined(MY_PERF_CYCLES) > + mti->cycles.routine= MY_TIMER_ROUTINE_SGI_CYCLE; I'd say that if we'll have it pushed, you'll need to add a dedicated MY_TIMER_ROUTINE_xxx for this, not reuse MY_TIMER_ROUTINE_SGI_CYCLE. > #elif defined(HAVE_SYS_TIMES_H) && defined(HAVE_GETHRTIME) > mti->cycles.routine= MY_TIMER_ROUTINE_GETHRTIME; > #else Regards, Sergei

1 0

Re: [Maria-developers] [Commits] 72bb358: MDEV-7930 Assertion `table_share->tmp_table != NO_TMP_TABLE || m_lock_type != 2' failed in handler::ha_index_read_map
by Sergey Petrunia 27 Oct '15

27 Oct '15

Hi Sanja, Please find some cosmetic feedback below. Ok to push after it has been addressed. On Mon, Oct 26, 2015 at 05:00:34PM +0100, sanja(a)mariadb.com wrote: > revision-id: 72bb358c2ba98e6e0d4a15b92af514a45751fe8d (mariadb-10.0.21-44-g72bb358) > parent(s): 18f7dfed179204dcfc02a27790e22bb9cc4e2e32 > committer: Oleksandr Byelkin > timestamp: 2015-10-26 16:59:40 +0100 > message: > > MDEV-7930 Assertion `table_share->tmp_table != NO_TMP_TABLE || m_lock_type != 2' failed in handler::ha_index_read_map > > In optimizing aggregate function do not try to touch tables from outer > query. > ... > diff --git a/sql/opt_sum.cc b/sql/opt_sum.cc > index fc3ce09..ad29229 100644 > --- a/sql/opt_sum.cc > +++ b/sql/opt_sum.cc > @@ -341,7 +341,8 @@ int opt_sum_query(THD *thd, > there are no outer joins. > */ > if (!conds && !((Item_sum_count*) item)->get_arg(0)->maybe_null && > - !outer_tables && maybe_exact_count) > + !outer_tables && maybe_exact_count && > + ((item->used_tables() & OUTER_REF_TABLE_BIT) == 0)) > { > if (!is_exact_count) > { > @@ -369,7 +370,8 @@ int opt_sum_query(THD *thd, > indexes to find the key. > */ > Item *expr=item_sum->get_arg(0); > - if (expr->real_item()->type() == Item::FIELD_ITEM) > + if (((expr->used_tables() & OUTER_REF_TABLE_BIT) == 0) && > + expr->real_item()->type() == Item::FIELD_ITEM) > { > uchar key_buff[MAX_KEY_LENGTH]; > TABLE_REF ref; > @@ -394,16 +396,20 @@ int opt_sum_query(THD *thd, > const_result= 0; > break; > } > + /* > + Constant table is already read and the only row is the max > + and the min > + */ I think this comment is not relevant anymore? It's part of the first variant of the fix, which is no longer in the patch. Please remove it. > if (!(error= table->file->ha_index_init((uint) ref.key, 1))) > - error= (is_max ? > + error= (is_max ? > get_index_max_value(table, &ref, range_fl) : > get_index_min_value(table, &ref, item_field, range_fl, > prefix_len)); > > /* Verify that the read tuple indeed matches the search key */ > - if (!error && reckey_in_range(is_max, &ref, item_field->field, > - conds, range_fl, prefix_len)) > - error= HA_ERR_KEY_NOT_FOUND; > + if (!error && reckey_in_range(is_max, &ref, item_field->field, > + conds, range_fl, prefix_len)) > + error= HA_ERR_KEY_NOT_FOUND; These are just endspace changes. Can you remove them? > table->disable_keyread(); > table->file->ha_index_end(); > if (error) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Several CVE's in Oracle MySQL, is MariaDB vulnerable?
by Christian Rebischke 26 Oct '15

26 Oct '15

Hello everyone, Sorry when I am on the wrong mailinglist. I wanted to submit a bugreport or issue for this but I can't signup for your JIRA because of license reasons. I am from the Archlinux-Security Team and want to ask if mariadb in the actual version is vulnerable to the following CVEs: CVE-2015-4913 CVE-2015-4910 CVE-2015-4905 CVE-2015-4904 CVE-2015-4895 CVE-2015-4890 CVE-2015-4879 CVE-2015-4870 CVE-2015-4862 CVE-2015-4864 CVE-2015-4861 CVE-2015-4858 CVE-2015-4836 CVE-2015-4833 CVE-2015-4830 CVE-2015-4826 CVE-2015-4819 CVE-2015-4815 CVE-2015-4807 CVE-2015-4802 CVE-2015-4800 CVE-2015-4792 CVE-2015-4791 CVE-2015-4766 I hope you can help me. Best regards -------------------------------------------------------------- Christian Rebischke Member of Archlinux CVE-Monitoring Team Website : www.nullday.de Twitter : @sh1bumi Jabber : shibumi(a)jabber.ccc.de PGP : 0xDFE2060D Fingerprint: 6DAF 7B80 8F9D F251 3962 0000 D214 61E3 DFE2 060D --------------------------------------------------------------

4 13

[Maria-developers] crosscompiling in buildroot: cannot find -laio and other link error
by Sylvain Raybaud 23 Oct '15

23 Oct '15

Hi I'm trying to get the latest version of mariadb-galera (10.0.21) into buildroot. I'm running into two errors which I have a feeling are related. But I may be wrong. These problems arise during the build of the host variant of the package. I get the two following errors (build does not stop at the first one because they happen in parallel jobs): 1: Linking CXX shared module ha_connect.so /home/sylvain/git/br/output/host/usr/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib/libxml2.so: error adding symbols: File in wrong format --> for host variant ha_connect.so should be linked against /home/sylvain/git/br/output/host/usr/lib/libxml2.so 2: Linking CXX shared module ha_innodb.so /usr/bin/ld: cannot find -laio The thing is, libaio and libxml2 are both present in /home/sylvain/git/br/output/host/usr/lib/. The link command present in storage/connect/CMakeFiles/connect.dir/link.txt (http://pastebin.com/Qn1ZCw2X) contains: /home/sylvain/git/br/output/host/usr/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib/libxml2.so which is incorrect, should be: /home/sylvain/git/br/output/host/usr/lib/libxml2.so The link command present in storage/innobase/CMakeFiles/innobase.dir/link.txt (http://pastebin.com/it6W9WVs) does not contain -L/home/sylvain/git/br/output/host/usr/lib/. I think it should. At least, when I manually run this command adding -L/home/sylvain/git/br/output/host/usr/lib, it works. I'm really not sure how these link.txt are generated. I checked connect/CMakeLists.txt and innobase/CMakeLists.txt and could not find anything obvious problem. Beside, toplevel CMakeCache.txt does contain: CMAKE_EXE_LINKER_FLAGS:STRING=-L/home/sylvain/git/br/output/host/lib -L/home/sylvain/git/br/output/host/usr/lib -Wl,-rpath,/home/sylvain/git/br/output/host/usr/lib which seems correct. I don't know why this setting does not reach storage/innobase and I'm also not sure why ha_connect.so tries to link against target variant of libxml2. It basically comes down to have cmake look for shared libraries in the right place, but I don't seem to be able to determine which lever to pull. Any hint as to how to achieve that is therefore welcome. Cheers, -- Sylvain Raybaud www.green-communications.fr

2 7

[Maria-developers] innodb holepunch compression vs the filesystem
by MARK CALLAGHAN 21 Oct '15

21 Oct '15

InnoDB punch-hole compression is very bad for a filesystem when done on every (or most) page writes. That feature was not designed to be used in that manner and generates far too much in-memory metadata in the filesystem. If nothing else it leads to multi-minute unlink() system calls. Yet both MariaDB and MySQL proudly exclaim this as a feature. Are you going to use the same kludge as MySQL (do unlink() in a background thread) to avoid locking up InnoDB for many minutes during drop table, maybe leading to InnoDB killing mysqld process? Why didn't you catch this problem via testing or reading my blog posts? https://mariadb.com/kb/en/mariadb/compression/ https://blog.mariadb.org/mariadb-10-1-is-stable-ga/ https://bugs.mysql.com/bug.php?id=78277 http://smalldatum.blogspot.com/2015/08/first-day-with-innodb-transparent-pa… http://smalldatum.blogspot.com/2015/09/second-day-with-innodb-transparent-p… http://smalldatum.blogspot.com/2015/09/third-day-with-innodb-transparent-pa… http://smalldatum.blogspot.com/2015/10/wanted-file-system-on-which-innodb.h… -- Mark Callaghan mdcallag(a)gmail.com

2 1

Re: [Maria-developers] Deadlock in Parallel Replication with FLUSH TABLE WITH READ LOCK.
by Kristian Nielsen 20 Oct '15

20 Oct '15

> I enabled // replication on the slave that has MyISAM tables. > The first results are good, but I ended up with what looks like a deadlock. > I opened support # 10289 for "Deadlock in Parallel Replication with > FLUSH TABLE WITH READ LOCK". There is a patch for this: https://lists.launchpad.net/maria-developers/msg08675.html But no reply on the review request so far. - Kristian.

1 0