- developers - lists.mariadb.org

[Maria-developers] New to the list & question about CONNECT
by Martijn Tonies (Upscene Productions) 13 May '14

13 May '14

Hello everyone, My name is Martijn Tonies and I’m a Windows tools developer, we have a tool called “Database Workbench”, a developer tool for several database engines. For it’s next major version, we’re planning to officially support MariaDB in the MySQL module. I’ve investigated several of the features, implemented support for Roles, Virtual Columns etc, but I’m having trouble with the CONNECT engine. Here’s what I did, using MariaDB 10.0.10 1) registered a server with CREATE SERVER: CREATE SERVER mysql55_at_local FOREIGN DATA WRAPPER mysql OPTIONS(HOST 'MT-XP-VM-MYSQL', PORT 3355, DATABASE 'test', USER 'root', PASSWORD ‘secret’); 2) created a table in the MySQL database: CREATE TABLE sfdata ( mark1 Integer(11), mark2 Integer(11), mark3 Integer(11), mark4 Integer(11), `name` VarChar(50) CHARACTER SET latin1 COLLATE latin1_swedish_ci ) ENGINE=InnoDB DEFAULT CHARACTER SET latin1 COLLATE latin1_swedish_ci; 3) created a table in the MariaDB database: CREATE TABLE `sfdata_proxy3` ( `mark1` int(11) DEFAULT NULL, `mark2` int(11) DEFAULT NULL, `mark3` int(11) DEFAULT NULL, `mark4` int(11) DEFAULT NULL, `name` varchar(50) DEFAULT NULL ) ENGINE=CONNECT DEFAULT CHARSET=utf8 CONNECTION='mysql55_at_local/sfdata' I then expected that table “sfdata_proxy3” has the rows as “sfdata” has, but I got nothing returned. Any clue where to look? With regards, Martijn Tonies Upscene Productions http://www.upscene.com Download Database Workbench for Oracle, MS SQL Server, Sybase SQL Anywhere, MySQL, InterBase, NexusDB and Firebird!

1 1

Re: [Maria-developers] Question about GTIDs in MariaBD
by Kristian Nielsen 12 May '14

12 May '14

"AL13N" <alien(a)rmail.be> writes: >> Peter Laursen <peter_laursen(a)webyog.com> writes: >> So yes, there are others that consider Oracle's design of GTID 'a dirty >> hack', >> but MariaDB 10.0 GTID is not affected by that design. > What about mariadb being a slave of a Oracle MySQL GTID? The restrictions are on the master, not on the slaves. So what do you mean? Surely, connecting a MariaDB slave to an Oracle MySQL master will not remove those restrictions. Note that when a MariaDB server connects as a slave to an Oracle MySQL master server, it is not possible to use GTID for that connection (because the GTID implementations are not compatible). Old style non-GTID replication must be used. - Kristian.

2 1

Re: [Maria-developers] Question about GTIDs in MariaBD
by Peter Laursen 12 May '14

12 May '14

With autocommit=1 every statement runs in its own transaction, so obviousluy .. yes! MyISAM is/used to be completely 'transaction-agnosstic'. But GTIDs seem to confict with non-transactional storage engines (as far as I can understand). MySQL/Oracle have "solved" (!) this by not allowing updates to tables using transactional engines and non-transactional storage engines in same transaction. However to me it is nonsense that transactions apply (have any effect at all) for non-transactional storage engines. As a consequence I consider this solution (what must have been enforced in the server layer and not the storeage engine layer) by Oracle 'a dirty hack' (unless somewone can explain me why it it necessary). -- Peter On Sun, May 11, 2014 at 4:42 PM, Roberto Spadim <roberto(a)spadim.com.br>wrote: > With autoclmmit=1 works? > > Em domingo, 11 de maio de 2014, Peter Laursen <peter_laursen(a)webyog.com> > escreveu: > > Thanks for the reply, but I am afraid I don't fully understand. >> >> Here is a simple test case with mySQL 5.6.17: >> >> 1) >> You need all those in configuration [mysqld) section >> >> log-bin >> log-slave-updates >> gtid_mode = ON >> enforce_gtid_consistency = ON >> >> >> 2) >> Then this small script will reproduce: >> >> >> DROP DATABASE IF EXISTS `gtid_test`; >> CREATE DATABASE `gtid_test`; >> USE `gtid_test`; >> >> CREATE TABLE `tab1` ( >> `a` INT DEFAULT NULL >> ) ENGINE=MYISAM ; >> >> CREATE TABLE `tab2` ( >> `b` INT >> ) ENGINE=INNODB; >> >> --start a transaction >> SET autocommit=0; >> >> INSERT INTO `tab1` VALUES (1); >> INSERT INTO `tab2` VALUES (1); >> >> -- now the first UPDATE to the MyISAM table in the transaction triggers >> this error: >> >> UPDATE `tab1` SET `a` = 5 WHERE `a` = 1; >> >> -- and you'll get: >> -- >> -- Error Code: 1785 >> -- When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to >> non-transactional tables can only be done in either autocommitted >> statements or single-statement transactions, and never in the same >> statement as updates to transactional tables. >> >> >> 3) >> ShouldI understand that MariaDB dos not prevent this (does not raise an >> error), but it is still unsafe as binlog - and thus replication - may be >> corrupted? >> >> >> Thanks! >> >> -- Peter >> >> On Sat, May 10, 2014 at 6:09 PM, Pavel Ivanov <pivanof(a)google.com> wrote: >> >>> There is no such limitation in MariaDB as well as no limitation on >>> CREATE TABLE ... SELECT. >>> >>> Although beware that mixing statements changing MyISAM and InnoDB >>> tables in statement binlog mode you may get different results from the >>> statements on slaves. >>> >>> Pavel >>> >>> On Sat, May 10, 2014 at 3:48 AM, Peter Laursen <peter_laursen(a)webyog.com> >>> wrote: >>> > GTID implementation in MySQL 5.6 has the limitation as described here: >>> > >>> https://dev.mysql.com/doc/refman/5.6/en/replication-gtids-restrictions.html >>> > >>> > >>> > "nontransactional storage engines such as MyISAM cannot be made in the >>> same >>> > statement or transaction as updates to tables using transactional >>> storage >>> > engines such as InnoDB." >>> > >>> > If you try you will get the error: >>> > Error Code: 1785 >>> > When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to >>> non-transactional >>> > tables can only be done in either autocommitted statements or >>> > single-statement transactions, and never in the same statement as >>> updates to >>> > transactional tables. >>> > >>> > Is there same or similar/other limitations with GTID in MariaDB 10? >>> > >>> > -- Peter Laursen >>> > -- Webyog >>> > >>> > _______________________________________________ >>> > Mailing list: https://launchpad.net/~maria-developers >>> > Post to : maria-developers(a)lists.launchpad.net >>> > Unsubscribe : https://launchpad.net/~maria-developers >>> > More help : https://help.launchpad.net/ListHelp >>> > >>> >> >> > > -- > Roberto Spadim > SPAEmpresarial > Eng. Automação e Controle > >

3 3

[Maria-developers] Question about GTIDs in MariaBD
by Peter Laursen 11 May '14

11 May '14

GTID implementation in MySQL 5.6 has the limitation as described here: https://dev.mysql.com/doc/refman/5.6/en/replication-gtids-restrictions.html "nontransactional storage engines such as MyISAM<https://dev.mysql.com/doc/refman/5.6/en/myisam-storage-engine.html> cannot be made in the same statement or transaction as updates to tables using transactional storage engines such as InnoDB<https://dev.mysql.com/doc/refman/5.6/en/innodb-storage-engine.html> ." If you try you will get the error: Error Code: 1785 When @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1, updates to non-transactional tables can only be done in either autocommitted statements or single-statement transactions, and never in the same statement as updates to transactional tables. Is there same or similar/other limitations with GTID in MariaDB 10? -- Peter Laursen -- Webyog

2 3

[Maria-developers] MariaDB 10.0.11 release prep
by Daniel Bartholomew 10 May '14

10 May '14

I've started the release prep for MariaDB 10.0.11. Draft release notes and changelog pages are here: - https://mariadb.com/kb/en/mariadb-10011-release-notes/ - https://mariadb.com/kb/en/mariadb-10011-changelog/ Please make necessary additions/modifications/etc... The release is still building at the moment, but when finished I'll upload to the mirrors later today or tomorrow. Current plan, if everything goes well, is to activate and announce the release on Monday, 12 May. Thanks! -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

1 0

[Maria-developers] 10.0 -> 10.1 merge
by Sergey Petrunia 08 May '14

08 May '14

Hello Igor, As discussed, I've done the 10.0 -> 10.1 merge. I didn't get any warnings or new test failures in the 'main' test suite, so I've pushed the merge into 10.1 tree. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] connect.alter fails
by Alexander Barkov 07 May '14

07 May '14

Hello Olivier, connect.alter fails when built without XML support complaining about unsupported table type XML. I guess it should be skipped. connect.alter also fails with libxml2 with these symptoms: --- /home/bar/maria-bzr/maria-10.0-connect/storage/connect/mysql-test/connect/r/alter.result 2014-03-23 11:32:34.519624000 +0400 +++ /home/bar/maria-bzr/maria-10.0-connect/storage/connect/mysql-test/connect/r/alter.reject 2014-03-23 11:48:43.251680097 +0400 @@ -1,3 +1,5 @@ +Warnings: +Warning 1105 No file name. Table will use t1.xml Please fix. A remainder: please check every push in BuildBot: http://buildbot.askmonty.org/buildbot/grid?branch=10.0-connect

3 4

[Maria-developers] Barkov: report for the week starting Apr 29, 2014
by Alexander Barkov 05 May '14

05 May '14

Apr 28 - May 04 =============== Apr 28 Apr 29 Apr 30 May 01 Holiday May 02 Holiday May 03 Weekend May 04 Weekend Vacation days taken: 10/35 - MDEV-5702 Incorrect results are returned with NULLIF() Pushed into 5.5 - MDEV-5459 Illegal mix of collations for datetime Pushed into 5.5 - MDEV-6146 Can't mix (latin1_swedish_ci,NUMERIC) and (utf8_unicode_ci,IMPLICIT An after-fix pushed (the original patch broke "mtr --ps fulltext" - Merging gb18030 from MySQL-5.7 In progress - Discussion the manual changes with Ian

1 0

[Maria-developers] sources won't compile
by AugustQ 03 May '14

03 May '14

Hi, I encountered a problem by compiling the sources of MariaDB 10.04. Some days ago I upgraded my system from Ubuntu 13.10 to 14.04 and now I encounter these problems: gcc used is this: gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) a minor glitch: cmake complains: Syntax Warning in cmake code at /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/cmake/ssl.cmake:210:55 Argument not separated from preceding token by whitespace. This is the reason for this: ELSE() MESSAGE(SEND_ERROR "Wrong option for WITH_SSL. Valid values are : "${WITH_SSL_DOC}) ENDIF() I added a space between "$(WITH and this works now. But in compiling the sources this error was reported: Scanning dependencies of target innobase [ 45%] Building CXX object storage/innobase/CMakeFiles/innobase.dir/row/row0log.cc.o /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/storage/innobase/row/row0log.cc: In function ‘dberr_t row_log_table_apply_ops(que_thr_t*, row_merge_dup_t*)’: /home/august/workspace/MariaDB/mariadb-10.0.4/contrib_AQ_01/storage/innobase/row/row0log.cc:2201:11: error: ‘srv_buf_size’ was not declared in this scope ofs, srv_buf_size); ^ By looking into the code of row0log.cc I found that FALLOC_FL_PUNCH_HOLE seems to be defined now. This is defined in /usr/include/linux/falloc.h. So what happened here during the upgrade of Linux? Regards AugustQ PS: by looking into Ubuntu 12.04 I found that this file also existed here.

2 2

[Maria-developers] Analysing degraded performance at high concurrency in sysbench OLTP
by Kristian Nielsen 02 May '14

02 May '14

At the Barcelona meeting in January, I promised to take a look at the high-concurrency sysbench OLTP benchmarks, and now I finally had the time do do this. There was a lot of work on LOCK_open by Svoj and Serg. If I have understood correctly, the basic problem was that at high concurrency (like, 512 threads), the TPS is only a small fraction of the peak throughput at lower concurrency. Basically, the server "falls over" and starts trashing instead of doing real work, due to some kind of inter-processor communication overhead. I started from Axel' OLTP sysbench runs and scripts, using 10.0 from bzr revno:4151 (revid:svoj@mariadb.org-20140415072957-yeir4jvokyilw5hp). I compiled without performance schema and with PGO, and ran sysbench 0.5 OLTP. (I just realised that my runs are with 32 tables, while I think the benchmarks in January focused on single-table runs. Maybe I need to re-do my analysis with the single-table benchmark, or perhaps it is too artificial to matter much?). In the read-only sysbench, the server mostly does not fall over. I guess this is due to the work by Svoj on eliminating LOCK_open? But in read-write, performance drops dramatically at high concurrency. TPS drops to 2600 at 512 threads compared to a peak of around 13000 (numbers here are approximate only, they vary somewhat between different runs). So I analysed the r/w benchmark with the linux `perf` tool. It turns out two-thirds of the time is spent in a single kernel function _raw_spin_lock(): - 66.26% mysqld [kernel.kallsyms] [k] _raw_spin_lock Digging further using --call-graph, this turns out to be mostly futex waits (and futex wakeups) from inside InnoDB locking primitives. Calls like sync_array_get_and_reserve_cell() and sync_array_wait_event() stand out in particular. So this is related to the non-scalable implementation in InnoDB of locking primitives, which is a known problem. I think Mark Callaghan has written about it a couple of times. Last time I looked at the code, every single mutex wait has to take a global mutex protecting some global arrays and stuff. I even remember seeing code that at mutex release would pthread_signal_broadcast() _every_ waiter, all of them waking up, only to all (except one) go do another wait. This is a kiler for scalability. While investigating, I discovered the variable innodb_sync_array_size, which I did not know about. It seems to split the mutex for some of the synchronisation operations. So I tried to re-run the benchmark with innodb_sync_array_size set to 8 and 64. In both cases, I got significant improvement, TPS increase to 5900, twice the value with innodb_sync_array_size set to the default of 1. So it is clear that the main limitation in this benchmark was the non-scalable InnoDB synchronisation implementation. After tuning innodb_sync_array_size, time spent in _raw_spin_lock() is down to half what it was before (33% of total time): + 33.77% mysqld [kernel.kallsyms] [k] _raw_spin_lock Now investigating call-graphs show that the sync_array operations are much less visible. Instead mutex_create_func(), called from dict_mem_table_create(), is the one that turns up prominently in the profile. I am not familiar with what this part of the InnoDB code is doing, but what I saw from a quick look is that it creates a mutex - and there is another global mutex needed for this, which again limits scalability. It is a bit surprising to see mutex creation being the most significant bottleneck in the benchmark. I would have assumed that most mutexes could be created up-front and re-used? It is possible that this is a warm-up thing, maybe the code is filling up the buffer pool or some table-cache like thing inside InnoDB? Because I see TPS being rather low for the first 150 seconds of the run (around 3000), and then increasing suddenly to around 8000-9000 for the rest. This might be worth investigating further. So in summary, my investigations found that the bottleneck in this benchmark, and the likely cause of the fall-over, is a scalability problem with InnoDB locking primitives. The sync_array part seems to be mitigated to some degree by innodb_sync_array_size, the mutex creation part still needs to be investigated. I wonder if the InnoDB team @ Oracle is doing something for this in 5.7? Does anyone know? I vaguely recall reading something about it, but I am not sure. It would seem a waste to duplicate their efforts. In any case, I hope this was useful. As part of this investigation, I installed a new 3.14 kernel on the lizard2 machine and a new `perf` installation, which seems to work well to do more detailed investigations of these kind of issues. So let me know if there are other benchmarks that I should look into. One thing that could be interesting is to look for false sharing; there are some performance counters that Intel manuals describe can be used for this. As an aside: In my tests, once concurrency becomes high enough that the server falls over, the actual TPS number becomes mostly meaningless. Eg. I saw putting dummy pause loops into the code increasing TPS. If TPS stabilises at N% of peak throughput as concurrency goes to infinity, then we can compare N. But if N goes to zero as concurrency goes to infinite, I think it is meaningless to compare actual TPS numbers - we should instead focus on removing the fall-over behaviour. (Maybe this is already obvious to you, I have not followed the previous benchmark efforts that closely). Hope this helps, - Kristian.

4 4

Re: [Maria-developers] MDEV-5792 - Deadlock between SELECTs from METADATA_LOCK_INFO and another
by Sergei Golubchik 01 May '14

01 May '14

Hi, Sergey! On Apr 23, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4155 > revision-id: svoj(a)mariadb.org-20140423073406-o20n7jvwfjinvxxr > parent: psergey(a)askmonty.org-20140418100754-v5b5sdqbavbcznh7 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0-mdev5792 > timestamp: Wed 2014-04-23 11:34:06 +0400 > message: > MDEV-5792 - Deadlock between SELECTs from METADATA_LOCK_INFO and another > I_S table > > mdl_iterate() helper function (which is used by the plugin to iterate mdl > locks) acquired mutexes in reverse order. > > Fixed by iterating MDL locks in two stages: > 1. Iterate locks hash under the protection of hash mutex, store all > lock pointers in a thread local array and increment reference counter > for the lock. > 2. Iterate local array without protection of hash mutex, handle destroyed > locks. > > It somewhat echoes hack in MDL_map_partition::move_from_hash_to_lock_mutex. Looks ok to me. Regards, Sergei

1 0

[Maria-developers] Next steps in improving single-threaded performance
by Kristian Nielsen 01 May '14

01 May '14

I have been analysing CPU bottlenecks in single-threaded sysbench read-only load. I found that icache misses is the main bottleneck, and that profile-guided compiler optimisation (PGO) with GCC gives a large speedup, 25% or more. (More details in my blog posts: http://kristiannielsen.livejournal.com/17676.html http://kristiannielsen.livejournal.com/18168.html ) Now I would like to ask for some discussions/help in how to get this implemented in practice. It involves changing the build process for our binaries: First compile with gcc --coverage, then run some profile workload, then recompile with -fprofile-use. I implemented a simple program to generate some profile load: https://github.com/knielsen/gen_profile_load It runs a bunch of simple insert/select/update/delete, with different combinations of storage engine, binlog format, and client API. It is designed to run inside the build tree and handle starting and stopping the server being tested, so it is pretty close to a working setup. These commands work to generate a binary that is faster due to PGO: mkdir bld cd bld cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 --coverage" .. make tests/gen_profile_load cmake -DWITHOUT_PERFSCHEMA_STORAGE_ENGINE=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" -DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-Wno-maybe-uninitialized -g -O3 -fprofile-use -fprofile-correction" make So all the pieces really are there, it should be possible to implement it. But we need to find a good way to integrate it into our build system. The best would be to integrate it into our cmake files. The gen_profile_load.c could go into tests/, ideally we would build both a static and dynamically linked version (so we get PGO for both libmysqlclient.a and libmysqlclient.so). Anyone can help me get cmake to do that? And it would be cool if we could get the above procedure to work completely within cmake, so that the user could just do: cmake -DWITH_PGO ... ; make and cmake would itself handle first building with --coverage, then running gen_profile_load.static and gen_profile_load.dynamic, then rebuilding with -fprofile-use. Anyone know if this is possible with cmake, and if so could help implement it? But alternatively, we could integrate a double build, like the commands above, into the buildbot scripts (.deb, .rpm, bintar). Any comments? Here are some more points: - I tested that gen_profile_load gives a good speedup of sysbench read-only (around 30%, so still very significant even though it generates a different and more varied load). - As another test, I removed all SELECT from gen_profile_load, and ran the resulting PGO binary with sysbench read-only. This still gave a fair speedup, despite the PGO load being completely different from the benchmark load. This gives me confidence that the PGO should not cause performance regressions in cases not covered well by gen_profile_load - More tests would be nice, of course. Axel, would you be able to build some binaries following above procedure, and test some different random benchmarks? Anything that is easy to run could be interesting, both to test for improvement, and to check against regressions. - We probably need a recent GCC version to get good results. I used GCC version 4.7.2. Maybe we should install this GCC version in all the VMs we use to build binaries? - Should we do this in 5.5? I think we might want to. The speedup is quite significant, and it seems very safe - no code modifications are involved, only different compiler options. Any thoughts? Volunteeres for helping with the cmake or buildbot parts? - Kristian.

4 14

[Maria-developers] valgrind failure in main.statistics
by Sergey Petrunia 29 Apr '14

29 Apr '14

Hello Igor, Valgrind build in buildbot currently fails in main.statistics test: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… I have tracked it down: the failure appeared after this push by you: revno: 4157 [merge] committer: Igor Babaev <igor(a)askmonty.org> branch nick: maria-10.0 timestamp: Tue 2014-04-22 14:43:13 -0700 message: Merge. ------------------------------------------------------------ revno: 4152.1.1 committer: Igor Babaev <igor(a)askmonty.org> branch nick: maria-10.0-bugs timestamp: Tue 2014-04-22 14:39:57 -0700 message: Fixed the problem of mdev-5947. Back-ported from the mysql 5.6 code line the patch with the following comment: Fix for Bug#11757108 CHANGE IN EXECUTION PLAN FOR COUNT_DISTINCT_GROUP_ON_KEY CAUSES PEFORMANCE REGRESSION I suppose it would take you much less time than anybody else to fix this? The failure is not repeatable on every machine: it doesn't repeat on my machines, but it can be repeated on work.askmonty.org. BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 10.1 planning
by Kristian Nielsen 29 Apr '14

29 Apr '14

Sergei Golubchik <serg(a)mariadb.org> writes: > See below. > What do you think about it? > > On Apr 23, Elena Stepanova wrote: >> >> There is also https://mariadb.atlassian.net/browse/MDEV-4487 >> (Replication from 5.6 with enabled GTID) to be considered. It's >> currently Minor and has 10.0.x as a target version, both of which are >> questionable: recently Colin said on IRC that it should be much more >> than a minor FR, and yet I doubt that somebody will get to it in 10.0. >> Maybe it's more realistic to raise the priority but to re-target it for >> 10.1? I think Monty had some ideas for doing this, IIRC it was discussed at the meeting in Barcelona in January. If what I remember is true, it is actually just that a 5.6 master refuses to replicate if the slave does not say it is configured for MySQL 5.6 GTID. The slave does not need to actually _use_ GTID. So I think Monty's plan was for the MariaDB slave to just lie to the master and say that it's a 5.6 slave with appropriate config. And the MariaDB slave would need to recognise and ignore the new MySQL 5.6 events, such as GTID. - Kristian.

2 2

[Maria-developers] Old autotools cruft in source tree?
by Otto Kekäläinen 28 Apr '14

28 Apr '14

Hello The Debian Lintian QA scanner yields this warning: W outdated-autotools-helper-file (http://lintian.debian.org/tags/outdated-autotools-helper-file.html) extra/jemalloc/config.guess 2012-02-10 extra/jemalloc/config.sub 2012-02-10 storage/tokudb/ft-index/third_party/xz-4.999.9beta/build-aux/config.guess 2009-04-27 storage/tokudb/ft-index/third_party/xz-4.999.9beta/build-aux/config.sub 2009-04-17 Are these still needed in MariaDB or is this simply old cruft from old autotools time that could be removed? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 1

[Maria-developers] "SELECT ... LIMIT 0" is now deterministic
by Alexander Barkov 28 Apr '14

28 Apr '14

Hello Ian, A fix for MDEV-6170 fixed another problem as a side effect: "SELECT ... LIMIT 0" is now deterministic for statement based binary log. Queries with "LIMIT 0" do not return ER_BINLOG_UNSAFE_STATEMENT any more. Can you please mention this in the manual? Some discussion is here: https://mariadb.atlassian.net/browse/MDEV-6170 Related articles in the MySQL manual: https://dev.mysql.com/doc/refman/5.6/en/replication-rbr-safe-unsafe.html https://dev.mysql.com/doc/refman/5.6/en/replication-features-limit.html Thanks.

1 0

[Maria-developers] a Query_log_event charset bug in parallel replication
by nanyi607rao 25 Apr '14

25 Apr '14

hi, If character_set in different Query_log_events changed, worker threads may apply them with wrong character_set. the codes leading this problem is in Query_log_event::do_apply_event, that is: if (charset_inited) { if (rli->cached_charset_compare(charset)) { /* Verify that we support the charsets found in the event. */ if (!(thd->variables.character_set_client= get_charset(uint2korr(charset), MYF(MY_WME))) || !(thd->variables.collation_connection= get_charset(uint2korr(charset+2), MYF(MY_WME))) || !(thd->variables.collation_server= get_charset(uint2korr(charset+4), MYF(MY_WME)))) { There is a charset[6] in rli, which cached last Query_log_event's charset in serial replication. But in parallel replication it would lead mistake, because every worker thread can read and set rli->charset[6], so rli->charset[6] isn't any worker threads' last Query_log_event charset. but rli->charset[6] can affect every worker threads' thd->variables.character_set_* setting. 2014-04-17 nanyi607rao

2 7

[Maria-developers] About the query performance issue reported on IRC (Percona 5.5 vs MariaDB 10.0)
by Elena Stepanova 24 Apr '14

24 Apr '14

Hi all, Over the last couple of days, some of you have seen or even participated in a "thread" on #maria IRC channel regarding the performance problem on query execution that james_woods observed on MariaDB 10.0 instance comparing to a Percona 5.5 instance. This email is for those who are curious about the outcome/current status. Summary ------- The original complaint sounded like this: "After switching to MariaDB 10 I see 5-10 very long running queries - which are not using indices as they do on Percona 5.5." The data in both instances is exactly the same. The HW is essentially different, but it's better on the machine with MariaDB 10. After some back-and-forth, the complaint got split into three main parts: - the query is considerably slower on MariaDB 10.0 instance than on Percona 5.5 instance; - the query plans show very different row counts, although the data is identical; - Percona uses indices when MariaDB doesn't. Sergei Petrunia investigated the alleged issues with query plans and found that there was no problem: - the row counts that Percona 5.5 reported for derived tables were smaller because EXPLAIN in Percona 5.5 had actually executed the query. - the difference in index usage was an effect of table elimination. A table was removed, hence no index use for it was shown in the explain. He also broke down the query and got a simple flat one, without subqueries, which still revealed ~4x difference in execution time. Upon his request, I ran some tests locally to see if I could reproduce such a difference between Percona 5.5 and MariaDB 10.0 and found that I couldn't (of course I don't have the HW that james_woods uses, the but the difference this big was likely to be seen anywhere). Meanwhile, james_woods installed Percona 5.5 on the machine where MariaDB 10.0 was running, and didn't get any performance gain -- that is, the difference was not between Percona 5.5 and MariaDB 10.0, but was caused by something else (HW, environment, ...). As of now, we don't know what it is. (end of summary). Sergei asked me to share the numbers I got while running the local tests. I will also put here the initial queries and explains, for the reference. It's mainly for the record, there is nothing particularly interesting in here; if you stop reading now, you won't lose much. Queries and plans ----------------- The initial offending query: SELECT SUM(visits) FROM ( SELECT IFNULL(ref.probename_unique_visits,'tr_trafficother2_uv') AS probename, SUM(x.value) AS visits, IFNULL(ref.category,'Other') AS category FROM ( SELECT aggregated_data._getSource(source, source2, source3, fb_source, request_ids) AS source, COUNT(DISTINCT u.fbuserid) AS value FROM g9_tracker.app_logins u WHERE u.created_date = '2014-04-22' GROUP BY 1 ) x LEFT JOIN aggregated_data.ref_parameter_mappings ref ON (ref.value_of_ref_parameter = x.source) GROUP BY 1,3 ) y; EXPLAINs (that's where row counts were off; note EXPLAIN execution time): MariaDB 10.0: +------+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 3394453 | | | 2 | DERIVED | <derived3> | ALL | NULL | NULL | NULL | NULL | 3394453 | Using temporary; Using filesort | | 2 | DERIVED | ref | ref | value_of_ref_parameter_UNIQUE | value_of_ref_parameter_UNIQUE | 603 | x.source | 1 | Using where | | 3 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3394453 | Using where; Using filesort | +------+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ 4 rows in set (0,03 sec) Percona 5.5: +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 15 | | | 2 | DERIVED | <derived3> | ALL | NULL | NULL | NULL | NULL | 75 | Using temporary; Using filesort | | 2 | DERIVED | ref | ref | value_of_ref_parameter_UNIQUE | value_of_ref_parameter_UNIQUE | 603 | x.source | 1 | | | 3 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3669609 | Using filesort | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+---------------------------------+ 4 rows in set (1 min 40,20 sec) Example of the indices difference (caused by table elimination): Query: SELECT COUNT(*) FROM ( SELECT aggregated_data._getSource(source, source2, source3, fb_source, request_ids) AS source, COUNT(DISTINCT u.fbuserid) AS value FROM g9_tracker.app_logins u WHERE u.created_date = '2014-04-22' GROUP BY 1 ) x LEFT JOIN aggregated_data.ref_parameter_mappings ref ON (ref.value_of_ref_parameter = x.source); EXPLAINs: MariaDB 10.0: +------+-------------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +------+-------------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 3394453 | 100.00 | | | 2 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3394453 | 100.00 | Using where; Using filesort | +------+-------------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+ 2 rows in set, 1 warning (0,02 sec) Percona 5.5: +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+----------+----------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+----------+----------------+ | 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 75 | 100.00 | | | 1 | PRIMARY | ref | ref | value_of_ref_parameter_UNIQUE | value_of_ref_parameter_UNIQUE | 603 | x.source | 1 | 100.00 | Using index | | 2 | DERIVED | u | ALL | NULL | NULL | NULL | NULL | 3669609 | 100.00 | Using filesort | +----+-------------+------------+------+-------------------------------+-------------------------------+---------+----------+---------+----------+----------------+ 3 rows in set, 1 warning (1 min 43,62 sec) Numbers ------- The query that Sergei suggested to run in local tests: SELECT SUM(varchar_non_indexed_column_with_non_empty_different_values='foo') FROM table_with_3_mio_rows I ran tests on Percona 5.5/5.6, MariaDB 5.5/10.0, MySQL 5.5/5.6. Only the relevant results are here, I'll skip the rest. For every run the caches were cleaned, the server was started, and then the queries were executed, 3 times each. Initially servers were run with the default options (not server default, but packaging default, how they are installed on Wheezy). The difference between 5.5 and 5.6/10.0 was astonishing, but it was explained by Performance Schema. It went like this: 5.5 select sum(c2='foo') from t_innodb; 1 row in set (8.17 sec) 1 row in set (0.99 sec) 1 row in set (0.95 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.04 sec) 1 row in set (0.56 sec) 1 row in set (0.55 sec) 5.6 / 10.0 select sum(c2='foo') from t_innodb; 1 row in set (24.24 sec) 1 row in set (21.65 sec) 1 row in set (23.43 sec) select sum(c2='foo') from t_myisam; 1 row in set (18.83 sec) 1 row in set (17.23 sec) 1 row in set (16.05 sec) Pretty much the same picture for MariaDB, Percona and MySQL. It would explain everything, but it turned out that james_woods already had Performance Schema turned off. (Note: we *really* should consider switching off Performance schema, at least in configs provided upon the new package installation). Then I ran another test, with the PS turned off, and otherwise default options (now really default, without any pre-installed config). There was still a difference between 5.5 and 5.6/10.0, although not that big, and for InnoDB only: Percona 5.5 select sum(c2='foo') from t_innodb; 1 row in set (5.97 sec) 1 row in set (0.79 sec) 1 row in set (0.79 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.68 sec) 1 row in set (0.56 sec) 1 row in set (0.56 sec) Percona 5.6 select sum(c2='foo') from t_innodb; 1 row in set (7.33 sec) 1 row in set (2.32 sec) 1 row in set (1.48 sec) select sum(c2='foo') from t_myisam; 1 row in set (2.90 sec) 1 row in set (0.59 sec) 1 row in set (0.59 sec) MariaDB 5.5 select sum(c2='foo') from t_innodb; 1 row in set (8.17 sec) 1 row in set (0.99 sec) 1 row in set (0.95 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.04 sec) 1 row in set (0.56 sec) 1 row in set (0.55 sec) MariaDB 10.0 select sum(c2='foo') from t_innodb; 1 row in set (7.34 sec) 1 row in set (2.07 sec) 1 row in set (2.07 sec) select sum(c2='foo') from t_myisam; 1 row in set (3.49 sec) 1 row in set (0.58 sec) 1 row in set (0.57 sec) MySQL 5.6 select sum(c2='foo') from t_innodb; 1 row in set (7.88 sec) 1 row in set (1.95 sec) 1 row in set (2.11 sec) select sum(c2='foo') from t_myisam; 1 row in set (4.11 sec) 1 row in set (0.55 sec) 1 row in set (0.57 sec) So, on 5.6/10.0 with default options the query from the "warm" InnoDB table was approximately 2x slower than on 5.5. Then I took non-default options from james_woods and applied them, as much as I could, considering the HW difference, to my modest VM. In fact, only innodb_buffer_pool_size mattered. After doubling it on 5.6/10.0, I got pretty much the same results as on 5.5. Percona 5.5 (default options) select sum(c2='foo') from t1; 1 row in set (5.97 sec) 1 row in set (0.79 sec) 1 row in set (0.79 sec) Percona 5.6 (default options + no PS + increased buffer pool) select sum(c2='foo') from t1; 1 row in set (8.33 sec) 1 row in set (0.83 sec) 1 row in set (0.82 sec) MariaDB 5.5 (default options) select sum(c2='foo') from t1; 1 row in set (8.17 sec) 1 row in set (0.99 sec) 1 row in set (0.95 sec) MariaDB 10.0 (default options + no PS + increased buffer pool) select sum(c2='foo') from t1; 1 row in set (7.85 sec) 1 row in set (0.86 sec) 1 row in set (0.86 sec) There are still some minor differences, but my VM is not suitable for fine-grained performance testing. If somebody wants to investigate it further, it should be done in a proper way; in any case I'm sure it can be tuned and made to perform better on 5.6/10.0, nobody who cares about differences this small actually runs the server with default options. Regards, /E

1 0

[Maria-developers] Difference between 0xHHHH and X'HHHH' hex literals
by Alexander Barkov 24 Apr '14

24 Apr '14

Hello Ian, I tried to explain in https://mariadb.atlassian.net/browse/MDEV-6092 to the complainer why it's not a bug. Can you please document the difference between the two hex notations and then close as "not a bug"? Thanks.

1 0

Re: [Maria-developers] bb01.mariadb.net
by Kristian Nielsen 23 Apr '14

23 Apr '14

Daniel Bartholomew <dbart(a)mariadb.com> writes: > Serg and Knielsen: I've got the box mostly configured I think, but > could be (very) wrong, so I wanted to give you a chance to make sure > everything looks good on it before I go ahead and add it to the It looks like it already mostly works. I noticed an error in a build though: https://buildbot.askmonty.org/buildbot/builders/kvm-tarbake-jaunty-x86/buil… The "kernel" log has this: ----------------------------------------------------------------------- /dev/sda1 contains a file system with errors, check forced. /dev/sda1: Entry 'pack-names' in /home/buildbot/.bzr/repository (3415) has an incorrect filetype (was 1, should be 2). /dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options) ----------------------------------------------------------------------- One likely reason is a failed rsync step. This is the tarbake build; unlike most other virtual machines, this one is actually modified during a build. So it's quite possible that the rsync would go wrong if a tarbake build happened to run in the meantime. So just re-sync that particular VM, and that might solve the problem. Thanks for setting this up! - Kristian.

2 1

[Maria-developers] [GSoC] Accepted student ready to work : )
by Pablo Estrada 23 Apr '14

23 Apr '14

Hello Sergei, Elena and everyone, I'm Pablo, and I was selected as part of Google Summer of Code 2014 to work on the project "*Statistically optimize mysql-test runs by running less tests*". First of all, thank you very much for having selected me for the project. I am very excited to start reading, reviewing data and coding up. I see that Elena will be my mentor. Nice to meet you Elena : ) To begin with, I am focusing on the *Buildbot documentation*, to *(1)understand how the data is organized* inside it, and to *(2)*figure out *how to add the extra functionality* into the BuildBot. I want to get started with the data analysis, so it would be great if you could help me with the following things: 1. Sergei, you said you did a bit of research into this. Do you have your *previous results*, or *any papers* that you have read that you could suggest to me? 2. Can I get *access to the buildbot database* to look at the data? If not, can I get the E/R model, or any scheme of *how the data looks like?* - I understand that maybe the data is not just a single database. I am starting by *reading the buildbot documentation*, to understand how the stuff is organized in it. 3. *Any other information* that you consider useful for me to know? Any advice? And that's all I have for now. I'm looking forward to getting started! Regards Pablo

4 5

[Maria-developers] mdev6027 RLIKE: "." no longer matching new line (default_regex_flags)
by Alexander Barkov 23 Apr '14

23 Apr '14

Hello Serg, Please review a patch implementing a new system variable default_regex_flags, to address the remaining incompatibilities between PCRE and the old regex library. Greetings.

3 5

[Maria-developers] Proposal Acceptance - GSOC
by Puneet Dewan 22 Apr '14

22 Apr '14

Hi Everyone, I, Puneet Dewan have got accepted into Google Summer Of Code 2014. This is the first and the last time I applied for GSOC and getting accepted is *really motivating*. :) My Project is : *Support of UseServerPrepStmts to MariaDB Java Connector.* *Massimo* is my mentor. I will start working on the project after 25th April as my semester exams are going on. I am excited to be part of MariaDB and looking forward to contribute to the community. Regards Puneet.

1 0

[Maria-developers] [GSOC] Proposal accepted!
by sriram patil 22 Apr '14

22 Apr '14

Hi Sergei, Alexander and all, Thank you for accepting my proposal. This was the first time I applied for GSoC, so I was kind of nervous. I was selected to work on the project *"CREATE OR REPLACE, CREATE IF NOT EXISTS and DROP IF EXISTS" *as part of GSoC'14. My end semester exams are going on right now. I will be done with my exams and projects by 29th April. I am totally excited to get started. I have built the source code already. Have learnt a few things about the source code as you must have observed in my proposal. I will continue with the documentation and source code from 1st May. Is there anything else you want me to look into? Thanks, Sriram

1 0

Re: [Maria-developers] [Commits] Rev 4153: MDEV-5963: InnoDB: Assertion failure in file row0sel.cc line 2503 ... in file:///home/psergey/dev2/10.0-cp/
by Jan Lindström 18 Apr '14

18 Apr '14

Sergey, > === modified file 'sql/opt_range.cc' > --- a/sql/opt_range.cc 2014-04-01 16:59:51 +0000 > +++ b/sql/opt_range.cc 2014-04-18 10:07:54 +0000 > @@ -2181,7 +2181,7 @@ int QUICK_ROR_INTERSECT_SELECT::init_ror > quick->record= head->record[0]; > } > > - if (need_to_fetch_row && head->file->ha_rnd_init_with_error(1)) > + if (need_to_fetch_row && head->file->ha_rnd_init_with_error(false)) You had so nice and informative reason why this has to be like this, why not add big fat comment here ? > { > DBUG_PRINT("error", ("ROR index_merge rnd_init call failed")); > DBUG_RETURN(1); > @@ -2363,8 +2363,13 @@ int QUICK_ROR_UNION_SELECT::reset() > quick->save_last_pos(); > queue_insert(&queue, (uchar*)quick); > } > - > - if ((error= head->file->ha_rnd_init(1))) > + /* Prepare for ha_rnd_pos calls. */ > + if (head->file->inited && (error= head->file->ha_rnd_end())) > + { > + DBUG_PRINT("error", ("ROR index_merge rnd_end call failed")); > + DBUG_RETURN(error); > + } > + if ((error= head->file->ha_rnd_init(false))) > { > DBUG_PRINT("error", ("ROR index_merge rnd_init call failed")); > DBUG_RETURN(error); > > _______________________________________________ > R: -- -- Jan Lindström, Principal Engineer SkySQL - The MariaDB Company MariaDB | MaxScale | skype: jan_p_lindstrom www.skysql.com <http://www.skysql.com/> Twitter <http://twitter.com/skysql> Blog <http://www.skysql.com/blog/> Facebook <http://www.facebook.com/skysql> LinkedIn <http://www.linkedin.com/company/1214250> Google+ <https://plus.google.com/117544963211695643458/posts>

1 0

[Maria-developers] TokuDB disabled in Debian and Ubuntu - help me get it working?
by Otto Kekäläinen 18 Apr '14

18 Apr '14

Hello Rick, Last year I spent a lot of time packaging MariaDB 5.5 for Debian and finally this year it has landed in Ubuntu 14.04 and Debian testing. Unfortunately the Debian/Ubuntu version does not include TokuDB and I need your help to get it there. In 5.5.35 (I think) the TokuDB plugn was added to MariaDB but I had issues getting it build 100% correctly and I eventually dropped it (added build parameter -DWITHOUT_TOKUDB=true), as getting MariaDB in Debian at all was a bigger priority than getting it there with every possible plugin. The root cause seems to be that when Debian and Ubuntu packages are built in chroot environments (the build systems of Debian and Ubuntu use pbuilder/sbuilder systems, see https://en.wikipedia.org/wiki/Debian_build_toolchain#Isolated_build_environ…) the code that builds the plugin does not seem to correctly detect the CPU features. It seems to read the values from the build machine and not the inputted target values (in a cross-compile situation). There are two related issues that needs a solution: 1) Currenlty the code that checks what the architecture is (32-bit/64-bit) is the first lines of https://bazaar.launchpad.net/~maria-captains/maria/10.0/view/head:/storage/…. This works well for real and virtual machhines, but it does not seem to work in the pbuilder/sbuilder chroots, as CMAKE_SYSTEM_PROCESSOR always shows the chroot host CPU, not the cross-compile target CPU. Could you please investigate pbuilder/sbuilder and search for some solution that works for reliable target CPU checking? 2) When building TokuDB in Ubuntu (amd64) sbuilder environments something in crashes in the 'toku_os_get_processor_frequency' function. For this too, could you investigate the sbuilde chroot environment and figure out what goes on and how to fix it? Issue 2 has a bug report with the (a bit messy) debugging history documented: https://mariadb.atlassian.net/browse/MDEV-5618 Both of these issues requires learning a bit about sbuilder CPU things, so I assume it is most efficient if the same persons looks into both of these. Thanks! -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

3 5

[Maria-developers] Ubuntu 14.04 and MariaDB
by Otto Kekäläinen 17 Apr '14

17 Apr '14

Ubuntu 14.04 was released today. >From https://wiki.ubuntu.com/TrustyTahr/ReleaseNotes#Ubuntu_Server "MySQL Ubuntu 14.04 LTS continues to provide official support for MySQL 5.5. Three other community supported alternatives of MySQL are also included: * MariaDB 5.5 - a drop in replacement for MySQL 5.5. * Percona XtraDB Cluster 5.5 - a Galera based Active/Active MySQL solution. * MySQL 5.6 - the next release of Oracle's MySQL. Note that upgrading to MySQL 5.6 is an automatic one way process; it is possible to downgrade manually - see the upstream documentation on details of how to perform this process." I have been working on 5.5.37 today, I hope to soon get it uploaded into Debian and then via security update process into Ubuntu too. -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

1 0

Re: [Maria-developers] [GSoC] Student interested in "Optimize mysql-test runs" or adding SQL features
by Sergei Golubchik 17 Apr '14

17 Apr '14

Hi, Pablo! First: please always cc: maria-developers(a)lists.launchpad.net, don't send emails to me only. Now: On Mar 11, Pablo Estrada wrote: > Hello! > I am getting ready to submit my proposal. Is there any feedback for my > application document? Anything that would be good to add, remove, rephrase, > etc? Go on and submit it. There will be a comment form under your proposal, if I will want you to change something, I will leave my comments there. Please mention what tools/languages you think you'll be using. > Also, if I could see the data available (the history of test results), I > could write more in detail : ) Hm. Okay. Attached is the preprocessed table that I used two years ago. It lists test failures in buildbot in a given period of time. Columns are (as far as I remember): * time of the test run start * code branch that was tested * revision id that was tested - it's fake, our buildbot tables don't store revision id. I've constructed it from the set of changed files, revision number, etc. hopefully it's as unique as a real revision id * build number * platform where the tests were run * test type - basically, the set of tests to run and what protocol to use. one test may be run with different protocols * test name * test variant - or combination. certain tests may be run in many combinations Using tis file I tried to rearrange test execution order for every specific build number (using failure data from previous builds in the same table) to detect test failures as early as possible. I managed to have 90% of all test failures happening within first 10% of the tests. The goal of this project is to have something we can put back into buildbot, so that it could actually run only 10% of the tests, still having 90% probability of discovering a failure (or 20%, or 95%, whaveter). See also http://buildbot.askmonty.org/buildbot/ you can click on a link for any branch, for example, http://buildbot.askmonty.org/buildbot/grid?branch=10.0 then on any particular build, for example, http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297 and on a stdio log for a particular test run: http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297/ste… on these pages you'll see what a platform (builder) is, build number, test types, combinations, etc. Regards, Sergei

2 2

[Maria-developers] Problems with Launchpad?
by Andrew McDonnell 17 Apr '14

17 Apr '14

Hi, my mariab bzr branch on launchpad doesnt update the summary page properly, and hasn't done so for a while now, which means recent work I committed is not visible unless you drill right down into the branch change history (My related question on Launchpad Answers - https://answers.launchpad.net/launchpad/+question/246850 - hasn't been touched either) Is anyone else experiencing this? cheers, Andrew -- http://blog.oldcomputerjunk.net https://launchpad.net/~andymc73 https://github.com/andymc73 Twitter: @pastcompute GPG: http://www.andrewmcdonnell.net/gpg.html

2 1

[Maria-developers] MariaDB 5.5.37 release prep
by Daniel Bartholomew 17 Apr '14

17 Apr '14

All, I've begun release prep for MariaDB 5.5.37. Draft changelog and release notes are at: - https://mariadb.com/kb/en/mariadb-5537-release-notes/ - https://mariadb.com/kb/en/mariadb-5537-changelog/ Please let me know ASAP of any notable things that should be mentioned in the release notes. Thanks! -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com

3 3

[Maria-developers] MariaDB scalability: mutexes
by Sergey Vojtovich 16 Apr '14

16 Apr '14

Hi Axel, I benchmarked various mutexes available to MariaDB. Results are here: http://svoj-db.blogspot.ru/2014/02/mariadb-mutexes-scalability.html Among other things I noticed that normal mutexes scale better than adaptive. And it looks like it gives over 5 000 TPS in my benchmarks: Original -------- 64 threads, time spent: 60s, queries executed: 11500723, qps: 191678, 1 thread qps: 2994, tps: 13691 64 threads, time spent: 60s, queries executed: 10983665, qps: 183061, 1 thread qps: 2860, tps: 13075 64 threads, time spent: 60s, queries executed: 11080325, qps: 184672, 1 thread qps: 2885, tps: 13190 slow LOCK_open -------------- 64 threads, time spent: 60s, queries executed: 12948365, qps: 215806, 1 thread qps: 3371, tps: 15414 64 threads, time spent: 60s, queries executed: 13191584, qps: 219859, 1 thread qps: 3435, tps: 15704 64 threads, time spent: 60s, queries executed: 13050647, qps: 217510, 1 thread qps: 3398, tps: 15536 slow LOCK_open + THR_LOCK::mutex -------------------------------- 64 threads, time spent: 60s, queries executed: 15863174, qps: 264386, 1 thread qps: 4131, tps: 18884 64 threads, time spent: 60s, queries executed: 15761944, qps: 262699, 1 thread qps: 4104, tps: 18764 64 threads, time spent: 60s, queries executed: 15773239, qps: 262887, 1 thread qps: 4107, tps: 18777 Could you do official benchmarks (patch attached)? It would be nice to also benchmark MySQL-5.6 with thr_lock part of the patch. Thanks, Sergey

3 8

[Maria-developers] Error on 'make' of mariadb source
by Kesha Shah 14 Apr '14

14 Apr '14

Hi all, I did some changes with in the sql_yacc.yy. During building the source , I am getting following error (attached screenshot) when I try to execute the "make" command after doing a cmake of the source. It says permission denied while trying to make jemalloc-configure. Although i am executing the make command with root user. [image: Inline image 1] [image: Inline image 2] Can somebody provide/help me with some pointers on how to proceed to resolve it ? -- Thanks and Regards, Kesha Shah. www.keshashah.com

2 3

[Maria-developers] MariaDB release and support policies
by Otto Kekäläinen 14 Apr '14

14 Apr '14

Hello! Does MariaDB promise a 8 year release cycle for 5.5 as MySQL does? Does all of the points listed at https://lists.ubuntu.com/archives/technical-board/2014-February/001812.html apply for MariaDB as well? Can we promise Ubuntu that all MariaDB minor releases fulfill the quality requirements listed above? If so, I'll then file a micro release update permission request for Ubuntu. AFAIK as I know it is the only way to get security releases to users anyway. PS. Do you people know about semver.org and try to follow the general versioning policy? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 2

[Maria-developers] Third bunch of unmerged items
by Sergey Vojtovich 14 Apr '14

14 Apr '14

Hi Sergei, here comes third bunch of items which I'm unsure about: WL#5706/Bug#58712/Bug#11746378 - Encrypt or remove passwords from slow, query, and binary logs WL#4677 - Unique Server Ids for Replication Topology (UUIDs) WL#6004 - Server extension for allowing user tables to be put into transactional SE Bug #14112682 : MYSQLTEST: PROVIDE ACCESS TO ERROR NAMES THROUGH NUMERIC CODES AND VICE VERSA Bug #11747313 31741: MYSQLTEST - DEPRECATE "SYSTEM" COMMAND WL#2739: Auditing Password Security (plugin + API) WL#6587: disconnect_on_expired_password WL#5825 Using C++ Standard Library with MySQL code WL#2111: #6 Set CLASS_ORIGIN and SUBCLASS_ORIGIN Regards, Sergey

2 1

Re: [Maria-developers] MariaDB 10.x GTID
by Kristian Nielsen 11 Apr '14

11 Apr '14

Jan Lindström <jplindst(a)mariadb.org> writes: > some background, lets assume we have two galera clusters that are > geographically distributed (lets say for example that cluster A is on Europe > and cluster B on USA). Inside a cluster all nodes are set up so that server_id > and domain_id are the same and we will use strict GTID mode (for a example on > cluster A server_id = 1 and domain_id =1 and in cluster B server_id = 2 and > domain_id = 2). In cluster we naturally use Galera replication. Additionally, > we have asynchronous replication (MySQL replication) between these clusters. > 1) We would like to have some special domain_id (i.e. GTID) for transactions > that are executed only on certain cluster e.g. on cluster A. This would mean > that there is some dynamic option to enable "local" transaction execution mode, > where transactions are executed on cluster A but not on cluster B. I know that > this kind of feature does not exists but is it even possible ? Basically, this > would mean for MySQL replication point of view a fact that if GTID is formed > like this special GTID, they are not replicated to slave, but they are written > to binlog. It sounds like what you want is some option on the slave that would cause it to ignore all GTIDs with some specific domain id, similar to how it is currently possible to filter on server id. That seems a reasonable request; such option does not exist currently though. Another option is to use the @@skip_replication variable. Set this for the sessions doing the "local" transactions. Then set replicate_events_marked_for_skip to either FILTER_ON_SLAVE or FILTER_ON_MASTER, as you like. This is specifically designed to allow to log certain transactions to the binlog but not replicate them, so it sounds like it is what you asked for. > 2) There will be some challenge in maintaining the sequence number for GTID > correctly: for global GTID purposes, we can just count commits, and skip every > local transaction commit, but, we must also initialize the sequence for every > node joining the cluster. In MySQL-5.6 this happens by copying binlog files > (only header part needed) to joining node. I wonder if same strategy works for > MGC-10, is MariaDB GTID is stored in binlog header ? Yes, the GTID is stored in the binlog header. It is also stored in a file master-bin.state. This file is read at server start to initialise the GTID sequence numbers. However, if the file is not found for any reason, the information is extracted from the last binlog file (the master-bin.state is just a cache). However, you need the whole binlog file (if master-bin.state is missing), not just the header. I am sure it must be the same for MySQL-5.6 - you need to scan to the end of the binlog file, as there may be extra GTIDs logged in the file that were not there when the header was written. Of course, just after server start, the most recent binlog file is empty, so just the header works. It sounds like you would want to just copy over the master-bin.state file. However, this whole approach sounds wrong to me. The current GTID state is a cluster-wide property, not a local property, I do not understand why you want to maintain it locally. As I understand it, Galera decides on commit order. The certification process decides for parallel transactions T1 and T2 which one will be committed before the other. The GTID sequence has to match the commit order. So the GTIDs need to be generated by Galera, at the point in the code where the commit order is decided. This Galera code then also needs to do the binlog writing, so that binlog order matches GTID order and commit order. With Galera handling GTID number allocations, I do not understand why you would need to worry about binlog headers. But maybe I have misunderstood, I do not know Galera in details. Maybe you could explain how Galera ensures matching commit order, binlog order, and GTID order? Or alternatively, explain how things work without such ensurance? > 3) Independent table (in data dictionary) or file for replication filters. I > can change the replication filters dynamically, but if the server crash and i > don?t report to my.cnf it?s lost. Therefore, there is a need to store > replication filters persistently and these should be loaded automatically when > server is restarted. Currently we do not have this, replication filters are configuration options, and those are stored in the config file, not in dynamic tables or files. The server_id filters are set by CHANGE MASTER though, IIRC, so they are stored in master.info / relay_log.info. > 4) Dynamic binlog. Currently options are read-only. This we do not have. There has been talk of allowing it under global read lock; this might work, but it would require some work to carefully check all of the code for possible issues. There are a lot of places in the code where expensive synchronisation/locking is avoided based on the knowledge that the binlog can not be enabled or disabled on-the-fly. Hope this helps, - Kristian.

1 0

[Maria-developers] debugging MDEV-5932, wrong result set for tokudb
by Rich Prohaska 10 Apr '14

10 Apr '14

Hello, TokuDB generates an empty result set for a select with sub-select, but generated a non-empty result set for the same query when creating a new table. The result set should be the same in both cases. What is going on here? InnoDB computes an empty result set for both cases. I traced the problem to the tokudb::index_read function. The tokudb::index_read does not handle the null byte key prefix. Unfortunately, I do not understand how the null byte (byte 0) in the key handed to the ::index_read function is supposed to work. Most of the time, the value of this byte is 0. Sometimes it is set to 1. How should the storage engine work WRT this? BTW, this problem exists with TokuDB in MySQL 5.5.36, MySQL 5.6.16. I have not been able to consistently see this on MariaDB 5.5.36 for some reason. In any case, I am interested in how the handler's index_read function is supposed to process the null byte. Thanks

2 1

[Maria-developers] Buildbot machines
by Rasmus Johansson 09 Apr '14

09 Apr '14

Hi, I discussed both with Dbart and Serg last week about the situation with buildbot. Kristian and Elena also had similar discussion on IRC today. To sum it up, there seems to be two aspects to it; 1) more machines, i.e. slaves could/should be added immediately and 2) some development work could/should be done on buildbot to support some other ways of building, like automatically spinning up and down platform specific virtual machines in a public cloud somewhere whenever they are needed. I looked into alternatives for the 1) immediate need of more machines and thought. I found a pretty interesting alternative from OVH.COM, who offers dedicated servers to a very reasonable price. Here is a good example of what they can offer: https://www.ovh.com/us/dedicated-servers/infra/2014-EG-32.xml . I would be prepared to give it a try. Do you think the specs are good enough to help with the load buildbot currently has? BR, Rasmus -- Rasmus Johansson, VP Engineering MariaDB | t: +358 50 499 9589 | Skype: ratzpo

3 5

Re: [Maria-developers] [Commits] Rev 3991: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 09 Apr '14

09 Apr '14

Hi, Alexey! On Apr 02, Alexey Botchkov wrote: > > Still, another question, why did you have to rename net_store_data() to > > net_store_data_cs() ? > > That's personal. > I really don't like functions with exactly same names - makes code harder to > understand and debug. > Do you belive it's better not to rename these here? That's up to you. Sometimes I create functions with the same name, sometimes I don't. For example, tdc_acquire_share() - there are few functions with the same name, because they do exactly the same. Actually, one of tdc_acquire_share() is a real function, others are inline convenience wrappers - they eventually call the "real" one. But tdc_acquire_share_shortlived() - while also only a convenience wrapper - has a different name to emphasize that it has different semantics. But anyway, that's your patch. If you'd like to rename - go ahead. Regards, Sergei

1 0

[Maria-developers] INT thread variables in mysql and mariadb (all versions as far as i know)
by Rich Prohaska 09 Apr '14

09 Apr '14

suppose that I have a THDVAR_INT variable with min = -1 and max = 1, and i set it to -1. how come select @@var returns 4294967295 instead of -1?

2 1

[Maria-developers] Error in "make install"
by sriram patil 09 Apr '14

09 Apr '14

Hello all, I have had built and installed the mariadb source code before, no hassles whatsoever. I was trying to install it again but getting the following error when trying to "make install" ....... -- Installing: /usr/local/mysql/mysql-test/././var/tmp -- Installing: /usr/local/mysql/mysql-test/././var/tmp/mysqld.2 -- Installing: /usr/local/mysql/mysql-test/././var/tmp/mysqld.1.sock CMake Error at mysql-test/cmake_install.cmake:36 (FILE): file INSTALL cannot copy file "/home/sriram/Downloads/mariadb/10.0/mysql-test/./var/tmp/mysqld.1.sock" to "/usr/local/mysql/mysql-test/././var/tmp/mysqld.1.sock". Call Stack (most recent call first): cmake_install.cmake:119 (INCLUDE) make: *** [install] Error 1 I have changed the owner and group of the folder "/usr/local/mysql", so root access is not an issue. I am using Ubuntu 12.04. Did anyone get such error while installing and was able to solve it? Thanks, Sriram

2 3

[Maria-developers] Vision for next 10.x minor releases
by Honza Horak 08 Apr '14

08 Apr '14

Hi guys, first, congratulations for the 10.0 release, well done! Even though it can look insane, I'm already wondering about next minor releases ;) Especially if you have already some idea about 10.1 and later. Specifically, I'm wondering if these releases will still be somehow tight to MySQL's release cycle (say 10.1 will be somehow following 5.7) or if you plan totally independent release schedule like slower/faster the release cadence? In other words, since MySQL changed only minor version and MariaDB bumped the major (we all know the reason), I'm wondering if something changed in thinking about major/minor versioning -- like if the difference between 10.0 and 10.1 will still be considered the same like between say 5.3 and 5.5. I don't expect you can give us any dates, but some expectations, if 10.1 can be expected in months or rather years? If that was already discussed somewhere, please, just point me to the correct place. Thanks, Honza

2 2

Re: [Maria-developers] Rev 4077: MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client
by Sergei Golubchik 07 Apr '14

07 Apr '14

Hi, Holyfoot! On Feb 22, holyfoot(a)askmonty.org wrote: > revno: 4077 > revision-id: holyfoot(a)askmonty.org-20140222173543-xkzlp5rpga41evur > parent: svoj(a)mariadb.org-20140218065405-vw3bfhvhrfjsc6hh > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-4856 > timestamp: Sat 2014-02-22 21:35:43 +0400 > message: > MDEV-4856 SQL_ERROR_LOG shows 1146 errors which didnt appear in mysql client. > The fill_schema_table() function used to call get_table_share() > for a table name in WHERE then clear the error list. That way > plugins receive the superfluous error notification if it happens > in it. Also the problem was that error handler didn't prevent > the suppressed error message from logging anyway as the logging > happens in THD::raise_condition before the handler call. > Fixed by adding the Warnings_only_error_handler before the call. > raise_condition() also fixed. The Internal_error_handler class > was exteded with virtual functions to handle storing the error > message. This looks too complex to me. Why would you extend the base Internal_error_handler class - no other error handler did that or needed that, fwiw. I'd simply push an error handler, store the first message there and access it in get_schema_tables_record() via THD::get_internal_handler(). Regards, Sergei

2 4

[Maria-developers] md5sum mariadb-10.0.10.tar.gz
by Jakub QB Dorňák 07 Apr '14

07 Apr '14

Hi guys, I am getting mariadb-10.0.10.tar.gz with md5 checksum of d41d8cd98f00b204e9800998ecf8427e, while it should be 14ce22b8197d4eae88d237776d47220f. Thank You for any help, QB

2 2

[Maria-developers] Getting started
by sriram patil 04 Apr '14

04 Apr '14

Hi Sergei, I have got myself familiarized with parsing and execution source code. I wanted to start working on the source code a little bit. But I could not find any open bugs on MariaDB launchpad. It would really help if you could give me a few open bugs or enhancements to start with. Thanks, Sriram

2 1

[Maria-developers] [GSOC 2014] CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS
by Kesha Shah 03 Apr '14

03 Apr '14

Dear Sergei Golubchik, Hope this mail finds you well. I am Kesha Shah, a third year undergraduate from DA-IICT, Gandhinagar, India. I was a Google Summer of Code student previous year as well as Google Code-In mentor with BRL-CAD open source solid modeling software. The activities related to my contributions at BRL-CAD at logged by me at - http://brlcad.org/wiki/User:KeshaSShah/GSoC13/Reports. My other projects can be found at - https://github.com/keshashah?tab=repositories. I have good programming hands on C and C++ and would like to take up and contribute at the Mariadb task of 'CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS' [MDEV-5359] as part of GSOC 2014. I have studied database systems and compiler construction during my academics. Your project would help to apply my knowledge by working at deeper level of coding database internals, thereby contributing to open source software. I had checked out the mariadb master branch from bazaar and started looking into the source code. I found the task more related to update of the existing mariadb SQL parser. The parser seems to be implemented as ascendant parser using bison parser generator (GNU version of YACC). From the current findings I could see the YACC grammar file sql_yacc.yy under maria/sql/ where I can see the rules defined for CREATE PROCEDURE_SYM, CREATE FUNCTION_SYM, CREATE TRIGGER_SYM etc. Alternatively I can see the *_SYM lexical tokens defined under the /maria/sql/lex.h file. Similarly for DROP TABLESPACE_SYM, DROP TRIGGER_SYM etc could be found. So when sql text input stream is encountered from SQL interface, it is read and categorized into lexical tokens by the lexer /maria/sql/sql_lex.cc in the lexical analysis step. I tried to figure out that we will have to define additional token 'IF' in the lex.h and 'NOT' and 'EXISTS' are already defined. I am also looking at the existing user defined actions that builds the AST nodes (sp_head being the root node for most types) when relevant grammar rule is satisfied in case of DDL statements. After semantic analysis (create of stored procs/functions), the next step would be to amend/define actions for the newly introduced grammar of handling check of existence of db artefact at the time of DDL execution. Also at this step the compiler will have to query the relevant storage engine on DB level or repository level to check if the DB object exists. This would require me to spend more time on understanding design pattern followed for the implementation of classes and behaviours of each of the underlying db artefacts. It would be interesting for me to know if the steps proposed by me above are in the right direction or not ? Also I am aware that I am late in discussing my proposal with you. But I can assure that I will be able to deliver the requirements in timely manner under your guidance. I have formulated initial draft of my proposal and would like to submit it at the earliest. Looking forward to you reply. Have a nice day! Thanks and Regards, Kesha Shah

2 2

[Maria-developers] new histogram code in 10.0.10
by Axel Schwenke 02 Apr '14

02 Apr '14

Hi Sergei, attached the results from a DBT3/SF30 run vs. 10.0.9 (as released) and 10.0-bzr rev 4128. Both runs use EITS + histograms. I see different EXPLAIN plans for queries 2, 3, 5, 7, 8, 18 and 21. Execution times only for few queries: 10.0.9 rev4128 Q2 44,44,46 44,45,45 Q3 298,298,299 236,243,243 Q5 264,302,303 293,308,308 Q7 199,200,201 204,205,206 Q8 375,375,377 264,274,276 Q18 217,218,221 178,180,182 Q21 251,259,261 258,261,262 Details in attached zip file. XL

1 0

[Maria-developers] PIC build options and MariaDB
by Otto Kekäläinen 02 Apr '14

02 Apr '14

Hello, I noticed that the MySQL packages in Debian have PIC stuff in their alternative build arguments, see http://anonscm.debian.org/gitweb/?p=pkg-mysql/mysql-5.6.git;a=blob;f=debian… http://anonscm.debian.org/gitweb/?p=pkg-mysql/mysql-5.5.git;a=blob;f=debian… Anybody here who understands deeply the build arguments who could say if having these PIC additions is desirable? Also the MySQL rules file has a section like this: # See http://blogs.innodb.com/wp/2010/04/innodb-performance-aio-linux/ 67 # Also #659565 68 ifneq (,$(findstring linux-gnu,$(DEB_HOST_GNU_TYPE))) 69 USE_LINUX_NATIVE_AIO:=-DLINUX_NATIVE_AIO=ON 70 endif Should MariaDB use this too, is it valid for XtraDB? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 1

[Maria-developers] Cassandra plugin build error: _S_empty_rep_storage' can not be used when making a shared object; recompile with -fPIC
by Otto Kekäläinen 02 Apr '14

02 Apr '14

Hello! I am testing out building MariaDB 10.0.10 on a Ubuntu 12.04 system with libthrift-dev installed (latest 0.9.0 from repo 'deb http://ppa.launchpad.net/wnoronha/thrift/ubuntu precise main'). Building fails with this error: make[4]: Siirrytään hakemistoon "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" Scanning dependencies of target cassandra make[4]: Poistutaan hakemistosta "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" make[4]: Siirrytään hakemistoon "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/ha_cassandra.cc.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/cassandra_se.cc.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/Cassandra.cpp.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_types.cpp.o [ 75%] Building CXX object storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_constants.cpp.o Linking CXX shared module ha_cassandra.so /usr/bin/ld.bfd.real: /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a(TApplicationException.o): relocation R_X86_64_32S against `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_empty_rep_storage' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a: could not read symbols: Bad value collect2: ld:n paluuarvo oli 1 make[4]: *** [storage/cassandra/ha_cassandra.so] Virhe 1 Ideas? -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

3 3

Re: [Maria-developers] Cassandra plugin build error: _S_empty_rep_storage' can not be used when making a shared object; recompile with -fPIC
by Otto Kekäläinen 01 Apr '14

01 Apr '14

Hello! Thanks for a quick reply! So I should add -fPIC to cmake arguments? Why isn't it in MariaDB sources by default, what are the downsides? GCC manual said "This option makes a difference on the m68k, PowerPC and SPARC." -> I am running amd64 For background, in my Debian packaging the build command looks like this: sh -c 'PATH=${MYSQL_BUILD_PATH:-"/bin:/usr/bin"} \ CC=${MYSQL_BUILD_CC:-gcc} \ CFLAGS=${MYSQL_BUILD_CFLAGS:-"-O2 -DBIG_JOINS=1 -fno-strict-aliasing "} \ CXX=${MYSQL_BUILD_CXX:-g++} \ CXXFLAGS=${MYSQL_BUILD_CXXFLAGS:-"-O3 -DBIG_JOINS=1 -felide-constructors -fno-exceptions -fno-rtti -fno-strict-aliasing "} \ cmake -DCMAKE_INSTALL_PREFIX=/usr \ \ -DCOMPILATION_COMMENT="(Ubuntu)" \ -DMYSQL_SERVER_SUFFIX="-1" \ -DSYSTEM_TYPE="debian-linux-gnu" \ -DWITHOUT_TOKUDB=true \ -DBUILD_CONFIG=mysql_release \ -DINSTALL_LIBDIR=lib/x86_64-linux-gnu \ -DINSTALL_PLUGINDIR=lib/mysql/plugin \ -DINSTALL_SUPPORTFILESDIR=share/mysql \ -DDEB=1 ..' In upstream MariaDB the build command is: sh -c 'PATH=$${MYSQL_BUILD_PATH:-"/bin:/usr/bin:/usr/local/bin"} \ CC=$${MYSQL_BUILD_CC:-gcc} \ CFLAGS=$${MYSQL_BUILD_CFLAGS:-"-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized $$(case `lsb_release -sc` in (lenny) echo -DWORKAROUND_GCC_4_3_2_BUG ;; esac)"} \ CXX=$${MYSQL_BUILD_CXX:-g++} \ CXXFLAGS=$${MYSQL_BUILD_CXXFLAGS:-"-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized"} \ cmake .. \ -DMYSQL_SERVER_SUFFIX="-$(DEBVERSION)" \ -DBUILD_CONFIG=mysql_release \ -DCOMPILATION_COMMENT="mariadb.org binary distribution" \ -DSYSTEM_TYPE="debian-linux-gnu" \ -DDEB=debian' 2014-04-01 16:21 GMT+03:00 Peter W Bowey <support(a)pbcomp.com.au>: > You require -fPIC to compile. See the info you reported = "> object; > recompile with -fPIC" > > Info is here (via GCC manual): > > -fPIC > If supported for the target machine, emit position-independent code, > suitable for dynamic linking and avoiding any limit on the size of the > global offset table. This option makes a difference on the m68k, PowerPC and > SPARC. > Position-independent code requires special support, and therefore works only > on certain machines. > > When this flag is set, the macros __pic__ and __PIC__ are defined to 2. > > In brief, the term position independent code (PIC) refers to the generated > machine code which is memory address agnostic, i.e. does not make any > assumptions about where it was loaded into RAM. Only position independent > code is supposed to be included into shared objects (SO) as they should have > an ability to dynamically change their location in RAM. > > > ----- Original Message ----- From: "Otto Kekäläinen" <otto(a)seravo.fi> > To: <maria-developers(a)lists.launchpad.net> > Cc: "Roger Meier" <roger(a)bufferoverflow.ch>; "Jake Farrell" > <jfarrell(a)apache.org> > Sent: Tuesday, April 01, 2014 11:16 PM > Subject: [Maria-developers] Cassandra plugin build error: > _S_empty_rep_storage' can not be used when making a shared object; recompile > with -fPIC > > >> Hello! >> >> I am testing out building MariaDB 10.0.10 on a Ubuntu 12.04 system >> with libthrift-dev installed (latest 0.9.0 from repo 'deb >> http://ppa.launchpad.net/wnoronha/thrift/ubuntu precise main'). >> >> Building fails with this error: >> >> make[4]: Siirrytään hakemistoon >> "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" >> Scanning dependencies of target cassandra >> make[4]: Poistutaan hakemistosta >> "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" >> make[4]: Siirrytään hakemistoon >> "/home/otto/MariaDB/pkg-mariadb-10.0/mariadb-10.0/builddir" >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/ha_cassandra.cc.o >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/cassandra_se.cc.o >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/Cassandra.cpp.o >> [ 75%] Building CXX object >> storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_types.cpp.o >> [ 75%] Building CXX object >> >> storage/cassandra/CMakeFiles/cassandra.dir/gen-cpp/cassandra_constants.cpp.o >> Linking CXX shared module ha_cassandra.so >> /usr/bin/ld.bfd.real: >> >> /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a(TApplicationException.o): >> relocation R_X86_64_32S against `std::basic_string<char, >> std::char_traits<char>, std::allocator<char> >>> >>> ::_Rep::_S_empty_rep_storage' can not be used when making a shared >> >> object; recompile with -fPIC >> /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libthrift.a: could >> not read symbols: Bad value >> collect2: ld:n paluuarvo oli 1 >> make[4]: *** [storage/cassandra/ha_cassandra.so] Virhe 1 >> >> >> Ideas? >> >> >> -- >> Check out our blog at http://seravo.fi/blog >> and follow @ottokekalainen >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers(a)lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp >> > > > --- > This email is free from viruses and malware because avast! Antivirus > protection is active. > http://www.avast.com > -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

1 0

[Maria-developers] DBT-3 benchmarks: MariaDB 10 vs MySQL 5.6
by Colin Charles 31 Mar '14

31 Mar '14

Hi XL, Nice work on posting new benchmarks in replication. Are you planning on running DBT-3 benchmarks with 10.0.10 and latest 5.6? Maybe even 5.7 DMR? Thanks -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

1 0

[Maria-developers] maria-developers@lists.launchpad.net
by Sergey Vojtovich 31 Mar '14

31 Mar '14

Hi! This is second list of questions I came up while merging test cases: - --ignore-builtin-innodb is deprecated in 5.6, do we want to deprecate it too? - LOW_PRIORITY WRITE is deprecate in 5.6, do we want to deprecate it too? The following items seem to be not merged to MariaDB. Do we want to merge them? WL#5331 Support Unicode for Windows command line client Bug#12688860 : SECURITY RECOMMENDATION: PASSWORDS ON CLI WL#5855 (EXPLAIN FORMAT=JSON) partition test WL#5980 - Allow remote tablespaces (need to check) WL5602 Providing password hashing with appropriate strength Bug#12316645: Wrong cost calculation with optimizer_join_cache_level settings WL#5602 - SHA256 password authentication Bugs#12635232: VALGRIND WARNINGS: IS_IPV6, IS_IPV4, INET6_ATON, INET6_NTOA + MULTIBYTE CHARSET Bug#14040277 UNINITIALIZED VALUE REFERENCED IN STR_TO_IPV6 And a lot of other IPV6 related stuff. Regards, Sergey

2 2

[Maria-developers] Intermediate status for test cases merge
by Sergey Vojtovich 31 Mar '14

31 Mar '14

Hi! So far I processed 431 test cases from the main suite. Most of them are up to date, a few merged without problems, a few tests fail (bugs reported), a few need extra analysis. There are a few items which I'd like to discuss: - INSERT DELAYED is deprecated in 5.6, do we want to deprecate it too? - SHOW PROFILE is deprecated in 5.6, do we want to deprecate it too? - YEAR(2) is deprecated in 5.6, do we want to deprecate it too? Monty suggests that we shouldn't deprecate it. I find it Ok too. Reasons for YEAR(2) deprecation are not obvious, relevant worklog is private. Relevant revision comment says: "YEAR(2) is a subject to deprecation since it has ill design." - EXPLAIN gives NULL in Extra column in 5.6 (empty string in 10.0), do we want behave similarly? - Do we want to merge fix for BUG#60269 - mysql should reject attempts to create system tables in incorrect engine http://bugs.mysql.com/bug.php?id=60269 - NO_ENGINE_SUBSTITUTION is default mode in 5.6 now, do we want it be default too? - 5.6 has many updates to default values, do we like to merge them too? Thanks, Sergey

2 6

Re: [Maria-developers] [Commits] Rev 3995: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 31 Mar '14

31 Mar '14

Hi, Holyfoot! On Dec 09, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3995 > revision-id: holyfoot(a)askmonty.org-20131209020845-9o3l30ao90vzf600 > parent: holyfoot(a)askmonty.org-20131208143921-3veatgg9wngyzb47 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Mon 2013-12-09 06:08:45 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > If a prepared statement calls an stored procedure, > the thd->server_status out of the SP goes up > to the PS and then to the client. So that the > client gets the SERVER_STATUS_CURSOR_EXISTS status > if the SP uses a cursor. Which makes the embedded > server fail. Why does embedded fail while a normal client-server protocol is ok with that? > Fixed by saving/restoring the upper-level server_status > in sp_head::execute(). Regards, Sergei

2 3

[Maria-developers] Engine independent statistics - naming issues?
by Sergey Petrunia 31 Mar '14

31 Mar '14

Hello, I was looking at whether column names used by EIS feature are consistent with themselves and with ther parts of the server. == Database / table / column names == There are a few naming conventions arready in use: 1. Old school: mysql.columns_priv, tables_priv Db Table_name Column_name 2. SQL STANDARD ALL CAPS HELLO FROM 1970S: information_schema.* (tables, columns, indexes, index_statistics, etc) TABLE_CATALOG (always 'def') TABLE_SCHEMA TABLE_NAME COLUMN_NAME INDEX_NAME 3. InnoDB has its own convention: mysql.innodb_table_stats, innodb_index_stats database_name table_name index_name 4. EITS feature will add: mysql.column_stats, index_stats, table_stats db_name table_name column_name index_name == Other == There are variables @@histogram_size, @@histogram_type. mysql.column_stats has columns named hist_size, hist_type. What's your opinion, should we make some renames to make things more consistent? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] [Commits] Rev 3991: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 31 Mar '14

31 Mar '14

Hi, Alexey! On Mar 26, Alexey Botchkov wrote: > Hi, Sergei. > > The Protocol::net_store_data is virtual in the embedded-server case > because we need it implemented differently for Protocol_binary. Ok, got it (finally), sorry. I'm almost always use ctags when looking for definitions, but in this case ctags didn't handle this correctly: #ifndef EMBEDDED_LIBRARY bool Protocol::net_store_data(const uchar *from, size_t length) #else bool Protocol_binary::net_store_data(const uchar *from, size_t length) #endif { ... so I never noticed that Protocol_binary actually has its own implementation of ::net_store_data() in the embedded case. Still, another question, why did you have to rename net_store_data() to net_store_data_cs() ? I've simply made it virtual and everything compiled just fine - it has a different signature so the compiler can distinguish them all right. Regards, Sergei

1 0

[Maria-developers] MariaDB 10.0.10 release prep
by Daniel Bartholomew 28 Mar '14

28 Mar '14

All, To get a jump on things, I've created the initial changelog page and a placeholder release notes page for the upcoming MariaDB 10.0.10 release. - https://mariadb.com/kb/en/mariadb-10010-changelog/ - https://mariadb.com/kb/en/mariadb-10010-release-notes/ Please let me know of things that need to be in the release notes for this release as soon as you can (or just add them yourself). The changelog page will also likely need to be updated as final pushes are made. As of right now it is up to revision 4135. FYI: I will be unresponsive on IRC or email until around mid-afternoon on Saturday due to some family events, but I will be around in the afternoon/evening on Saturday and Sunday to finish the various necessary tasks in preparation for the release. Thanks! -- Daniel Bartholomew MariaDB

1 0

[Maria-developers] A joint patch for MDEV-5969 and MDEV-5971
by Alexander Barkov 28 Mar '14

28 Mar '14

Hello, please review a joint patch fixing a crashing bug MDEV-5969, together with a non-crashing bug MDEV-5971. Thanks.

1 0

Re: [Maria-developers] parallel replication performance
by Kristian Nielsen 27 Mar '14

27 Mar '14

Axel Schwenke <axel(a)askmonty.org> writes: > The poor performance of MySQL with 1 database is expected. However MySQL is > much better with 32 databases. Even better than MariaDB :( Ok, so I looked closer at the numbers (of which there were quite a few, thanks for doing this work :-). I have some idea of what could be going on. The short story is that for this kind of configuration, it will be necessary to configure @@slave_domain_parallel_threads to be at most the value of @@slave_parallel_threads divided by the number of domain ids used. Eg. with 32 databases/domains, --slave-parallel-threads=64 --slave-domain-parallel-threads=2. Otherwise the domains will starve each other for threads, leading to poor parallelism. In fact, with such high domain parallelism @ 32 databases, I think just using --slave-domain-parallel-threads=1 could give much better performance. The longer story: MariaDB parallel replication uses a single pool of worker threads, of size @@slave_parallel_threads. GTIDs are read serially from the relay log in the SQL driver thread, and scheduled to worker threads. Each replication domain is scheduled independently from all other domains. Each GTID is allocated to a new worker thread if possible. If no worker thread is available (because all workers already have at least one GTID queued for them), then the SQL driver thread waits for one to become free; this stalls _all_ domains from getting more work queued. When your benchmark starts, the SQL driver thread will quickly read 64 events from the relay logs and queue them each to a separate worker thread. Then no more free worker threads are available, and the SQL thread will have to wait. With 64 worker threads and 32 domains, it is quite likely that some domains will not have any work to do. At the same time, the group commit size will most likely be much less than 64, so many workers will have to wait for the previous GTID in their domain without being able to do any work, especially at the "fast" configuration where commit (which can always run in parallel) does not take much time. The result will be a lot of threads waiting on each other and waking each other up, and little real work done. By setting @@slave_domain_parallel_threads, it is possible to limit how many worker threads a single domain is allowed to grab, thereby ensuring that worker threads will be available for all domains to be active doing work. Those limited number of threads will be used round-robin to queue work for; this also reduces thread sleep and wakeup, as with luck a worker thread that completes one commit will already have more work queued up for it at that point that it can process without first returning to the thread pool and going to sleep. In particular, with @@slave_domain_parallel_threads=1, we will just have 32 threads, one for each domain, each processing events independently in parallel at full speed, which is probably the best way to handle a workload as evenly distributed across domains as this one is. It would be very useful to see new benchmarks with those results. And if time permits, @@slave_domain_parallel_threads=2 could also be interesting; maybe there could be some extra performance due to more group commit in the "peer" configuration, or maybe the cost of repeatedly alternating between two worker threads within the domain will cause reduced performance. We can actually see some evidence of this theory in the numbers. Where number of worker threads is much larger than number of domains (so starvation is less of a problem), MariaDB does much better, even better than MySQL 5.6 in some cases. But performance drops as the number of domains increase, possibly because starvation kicks in. Still, even if this theory holds, we do see 5-10% slower performance of MariaDB at eg. slave@1 compared to MySQL. This suggests some higher overhead somewhere, which I need to look into at some point, though it will be a while yet before I have time for that, I am afraid. And in any case, this benchmark is highly useful as a test - as I said before, serious load testing of the domain_id parallelism has received very little testing, so it is quite possible that there is simply some bugs that cause things to work poorly. The extra testing with @@slave_domain_parallel_threads=1 (and possibly 2) will be very useful to help understand this better. > I think 32 databases is a bit too much, not only that MySQL then shines so > bright, it's also unrealistic in the real world. What do you think? I can > rerun the benchmark with 4 or 8 databases (will take 4-5 hours). Quite > certainly MySQL will have worse numbers then :) Well, I think 32 databases is not that unrealistic - shared hosting for example could have many more. The completely even spacing of the load among those databases however is probably unrealistic. But I think we should investigate and solve the issues for MariaDB @ 32 databases. More databases, and thus more parallelism, should be better for the parallel replication, if it is worse then that is a bug that should be fixed. Even if it is just the need to configure @@slave_domain_parallel_threads, maybe we could document that better. > I also notice that for MariaDB 10.0 the tuning of the commit-wait-* > variables is critical. Settings that give trx per commit group tend to slow > down the master a lot. The default settings give only few transactions per Right, this is something to be aware of. Note that for a typical web client application or similar, --commit-wait-* probably will not be much of an issue. An individual client will see slightly higher latency, which should not be much of an issue if it is in the order of a few milliseconds. Throughput is not likely to be affected, just the number of concurrent connections will increase (at least as long as system does not get choked). On the other hand, a load with fixed parallelism, like batch processing or your sysbench runs, can be hurt significantly on throughput by --commit-wait-*, as your tests show. Also, I have heard from several users that the ability to throttle a busy master in order to prevent it from running too far ahead of the slaves, is actually often seen as a benefit rather than a performance problem. Semi-sync replication also is sometimes used to achieve similar throttling. So yes, it is a tuning parameter that is unfortunately rather sensitive depending on the actual load and applications. > Please also check the "master" page of the comparison sheet. It shows the > cost of turning on the binlog. Both MySQL and MariaDB show significant > performance loss at high concurrency when the binlog is turned off. This is > completely counter-intuitive. Kristian, can this be related to MDEV-5802? It might be related. Another possibility is that this is related to InnoDB's known weaknesses at high concurrency. When the binlog is enabled, group commit will handle the commit of a lot of transactions serially, in a tight loop inside a single thread. When the binlog is disabled, all of those transactions will be committing each in their separate thread, all of those threads heavily contending each other for the hot mutexes inside InnoDB. It would need a lot of research to be sure that this is the cause, but it's a plausible guess, at least. Thanks for your efforts so far! Overall, I think the results look pretty good, actually. We see a good speedup from MariaDB parallel replication in many of the configurations. And your comparison shows that using group commit, a large percentage of the speedup can be obtained automatically from group commit, without any need to change applications and partition the data into separate databases. - Kristian.

2 2

Re: [Maria-developers] [Commits] Rev 3991: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 27 Mar '14

27 Mar '14

Hi, Holyfoot! On Nov 30, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3991 > revision-id: holyfoot(a)askmonty.org-20131130132432-etlmes8qhhqhr1iu > parent: elenst(a)wheezy-64.home-20131128160251-vn857wx5qlon6j1p > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Sat 2013-11-30 17:24:32 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > The function Protocol::net_store_data(a, b, CHARSET_A, CHARSET_B) should > be adapted to be working in the embedded server as it's done > with the Protocol::net_store_data(a, b). > That new function renamed as net_store_data_cs, so we can make it > virtual. Why does Protocol::net_store_data() have to be virtual? (if it doesn't, Protocol::net_store_data_cs doesn't have to be virtual either and doesn't have to be renamed) Regards, Sergei

2 1

[Maria-developers] Adding a storage engine - compilation
by Amith Nagarajappa 27 Mar '14

27 Mar '14

Hi, When i am creating a folder in storage/<customEngine>. I have placed the makefile from example folder and in the customEngine folder and have made required changes. However, makefile is not generated in the storage/<customEngine> folder. What change I have to make in configure or configure.in file? Right now, my custom engine code is similar to example code. I would like to build it first and then make the changes accordingly. But makefile is not generated. Appreciate your help. Thanks and Regards, Amith

3 2

Re: [Maria-developers] [Commits] Rev 3994: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 26 Mar '14

26 Mar '14

Hi, Alexey! On Mar 26, Alexey Botchkov wrote: > Hi, Sergei. > > > What do you mean by that? > > It can be seen in the sys_vars.completion_type_func test. oops. obscure feature. Okay, thanks. ok to push that. > > revno: 3994 > > message: > > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > > Thread can be disconnected internally for example after COMMIT statements. > > So we should check this for the statement execution. Regards, Sergei

1 0

[Maria-developers] Core MariaDB 10. Task suggestion.
by Сергей Дудоладов 26 Mar '14

26 Mar '14

2 1

Re: [Maria-developers] parallel replication performance
by Kristian Nielsen 25 Mar '14

25 Mar '14

Axel Schwenke <axel(a)askmonty.org> writes: > The poor performance of MySQL with 1 database is expected. However MySQL is > much better with 32 databases. Even better than MariaDB :( I haven't had time to go through all the numbers and details yet, but just a quick check: In this test, are you sure that you are running MariaDB in GTID mode? CHANGE MASTER TO master_use_gtid=slave_pos If not, you won't get any parallelism out of using 32 databases with different domain_ids. So two interesting, but different comparisons to make here: 1. MySQL multi-threaded slave compared to MariaDB GTID with different domain_id per database. If MySQL is significantly better here it could indicate some problem in MariaDB parallel replication that should be investigated; this is possible as this part has not been tested a lot. 2. MySQL multi-threaded slave with load split into multiple databases, compared to MariaDB without configuring different domains. Or just MariaDB compared without and with different domain ids. This is an "unfair" benchmark; by explicitly declaring that different databases/domains can run in parallel, more opportunity for parallellism is available. But it could still be interesting to see how much of that can be obtained from group commit parallelism, without changing applications. Both of these are interesting, but it was not clear to me which of them you were describing? - Kristian.

2 3

[Maria-developers] Patch for MDEV 5873
by Rohit Kalhans 25 Mar '14

25 Mar '14

Hello Kristian, All, As mentioned earlier i have a patch for MDEV 5873 and i am attaching here in the bundle of the patch. I have run tests on this patch and all of them have passed. Kindly review the patch. Also i am not sure which is the suitable mailing list for sending patches, so it will be great if you can point me to the same. Thanks and regards Rohit. -- Cheerio! *Rohit*

2 1

[Maria-developers] Two 10.0-related things.
by Alexey Botchkov 25 Mar '14

25 Mar '14

Hi, Sergei. I'd like to share my recently obtained knowledge :) 1. Mac OS X and sockets. As you know there on Labrador machine that mysql-test-run.pl failed if not run with --tmpdir=/tmp. The problem apparently is that the /Users directory has the access restrictions that /tmp doesn't. I still don't know what exactly was that, but it helps if we revoke all the ACLs with 'chmod -R -N /Users/mariadb'. 2. The 10.0 fails building on my machine, with linking errors in the extra/yassl/, about missing 'operator new[]' and similar. I fixed it locally with that patch: === modified file 'extra/yassl/CMakeLists.txt' --- extra/yassl/CMakeLists.txt 2014-02-26 14:28:07 +0000 +++ extra/yassl/CMakeLists.txt 2014-03-25 07:45:46 +0000 @@ -33,9 +33,10 @@ SET(YASSL_SOURCES src/buffer.cpp src/ce ADD_CONVENIENCE_LIBRARY(yassl ${YASSL_SOURCES}) RESTRICT_SYMBOL_EXPORTS(yassl) +TARGET_LINK_LIBRARIES(stdc++) + INSTALL_DEBUG_SYMBOLS(yassl) Best regards. HF

2 1

[Maria-developers] valgrind failures in ConnectSE
by Alexander Barkov 23 Mar '14

23 Mar '14

Hi Olivier, Valgrind complains about uninitialized memory read: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… It seems the problem was introduced by this change: http://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/5… Can you please fix this asap? We're going to make a release early next week. It would be nice to merge the latest ConnectSE. Thanks.

2 1

[Maria-developers] A fix for MDEV-5689 ExtractValue(xml, 'substring(/x, /y)') crashes
by Alexander Barkov 23 Mar '14

23 Mar '14

Hi Sergei, please review a fix for MDEV-5689. It also fixes MDEV-5709 ExtractValue() with XPath variable references returns wrong result. Description: 1. The main problem was that that nodeset_func->fix_fields() was called in Item_func_xml_extractvalue::val_str() and Item_func_xml_update::val_str(), which led in some cases to execution of the XPath engine *before* having a parsed XML value. Moved to Item_xml_str_func::fix_fields(). 2. Cleanup: added a new method Item_xml_str_func::fix_fields() and moved most of the code from Item_xml_str_func::fix_length_and_dec() to Item_xml_str_func::fix_fields(), to follow the usual Item layout. 3. Cleanup: a parsed XML value is useless without the raw XML value it was built from. Previously the parsed and the raw values where stored in separate String instances. It was hard to follow how they are synchronized. Added a helper class XML which contains both parsed and raw values. Makes things easier to read and modify. 4. MDEV-5709: const_item() could incorrectly return a "true" result when XPath expression contains users/SP variable references. Now nodeset_func->const_item() is also taken into account to catch such cases. 5. Minor code enhancements.

2 5

[Maria-developers] Transactions behind a failed transaction could commit in parallel replication
by nanyi607rao 21 Mar '14

21 Mar '14

hi kristian, In parallel replication if a transaction executed failed, replication should stop immediately(unconsider retry), and transactions behind the failed one should skip or rollback. you did try to do like this, there are codes like: if (unlikely(entry->stop_on_error_sub_id <= rgi->wait_commit_sub_id)) skip_event_group= true; this codes can tell latter transactions to skip but can't tell them rollback. because if a transaction started commiting before a former transaction failed (such as Lock timeout for unknown reason), the commiting transaction will not be affectd by stop_on_error_sub_id. Then the failed transaction should wakeup latter commiting transactions and tell them to rollback, unfortunately it won't. codes like if (!rgi->is_error && !skip_event_group) err= rpt_handle_event(events, rpt); else err= thd->wait_for_prior_commit(); ... ... finish_event_group(thd, err, event_gtid_sub_id, entry, rgi); if the failed transaction didn't fail at end event, err's value would come from wait_for_prior_commit, the err would be 0 if its former transaction has successed, then the failed transaction would tell latter transactions ok to commit in finish_event_group. 2014-03-19 nanyi607rao

2 2

Re: [Maria-developers] MDEV-5914
by Kristian Nielsen 21 Mar '14

21 Mar '14

Jan Lindström <jplindst(a)mariadb.org> writes: > Here is necessary changes. I have run whole mysql-test-run and I did not see > any failures, thus is is mostly safe. I say mostly, > because we really do not have much test cases that would use multimaster setup > and try to get conflicting transactions > generated from local clients against slave thread. Thanks a lot! I will try to come up with some test cases for this, with replication and external transactions that conflict and show the problem. In parallel, I tried running the parallel replication slaves with READ COMMITTED; that also seems to work and can get Elena able to continue testing. But this change is potentially more correct; I will let you know when I have some test results. - Kristian. > ================================= > > jan@jan-GE70-0NC-0ND ~/mysql/10.0-mdev5914 $ bzr diff > === modified file 'include/mysql/plugin.h' > --- include/mysql/plugin.h 2014-02-26 14:28:07 +0000 > +++ include/mysql/plugin.h 2014-03-21 09:05:58 +0000 > @@ -617,6 +617,7 @@ > int thd_in_lock_tables(const MYSQL_THD thd); > int thd_tablespace_op(const MYSQL_THD thd); > long long thd_test_options(const MYSQL_THD thd, long long test_options); > +unsigned long long thd_group_commit_id(const MYSQL_THD thd); > int thd_sql_command(const MYSQL_THD thd); > void **thd_ha_data(const MYSQL_THD thd, const struct handlerton *hton); > void thd_storage_lock_wait(MYSQL_THD thd, long long value); > > === modified file 'include/mysql/plugin_audit.h.pp' > --- include/mysql/plugin_audit.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_audit.h.pp 2014-03-21 09:06:17 +0000 > @@ -298,6 +298,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'include/mysql/plugin_auth.h.pp' > --- include/mysql/plugin_auth.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_auth.h.pp 2014-03-21 09:06:35 +0000 > @@ -298,6 +298,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'include/mysql/plugin_ftparser.h.pp' > --- include/mysql/plugin_ftparser.h.pp 2013-12-22 16:11:20 +0000 > +++ include/mysql/plugin_ftparser.h.pp 2014-03-21 09:06:59 +0000 > @@ -251,6 +251,7 @@ > int thd_in_lock_tables(const void* thd); > int thd_tablespace_op(const void* thd); > long long thd_test_options(const void* thd, long long test_options); > +unsigned long long thd_group_commit_id(const void* thd); > int thd_sql_command(const void* thd); > void **thd_ha_data(const void* thd, const struct handlerton *hton); > void thd_storage_lock_wait(void* thd, long long value); > > === modified file 'sql/sql_class.cc' > --- sql/sql_class.cc 2014-03-20 14:36:45 +0000 > +++ sql/sql_class.cc 2014-03-21 09:07:12 +0000 > @@ -675,6 +675,12 @@ > } > > extern "C" > +unsigned long long thd_group_commit_id(const THD *thd) > +{ > + return thd->rpl_group_commit_id; > +} > + > +extern "C" > int thd_sql_command(const THD *thd) > { > return (int) thd->lex->sql_command; > > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- storage/innobase/handler/ha_innodb.cc 2014-02-26 18:36:33 +0000 > +++ storage/innobase/handler/ha_innodb.cc 2014-03-20 15:42:27 +0000 > @@ -2048,6 +2048,8 @@ > trx->check_unique_secondary = !thd_test_options( > thd, OPTION_RELAXED_UNIQUE_CHECKS); > > + trx->group_commit_id = thd_group_commit_id(thd); > + > DBUG_VOID_RETURN; > } > > > === modified file 'storage/innobase/include/trx0trx.h' > --- storage/innobase/include/trx0trx.h 2014-02-26 18:36:33 +0000 > +++ storage/innobase/include/trx0trx.h 2014-03-20 15:28:19 +0000 > @@ -840,6 +840,7 @@ > or the state last time became > TRX_STATE_ACTIVE */ > trx_id_t id; /*!< transaction id */ > + ib_uint64_t group_commit_id;/*!< Group commit id or 0 */ > XID xid; /*!< X/Open XA transaction > identification to identify a > transaction branch */ > > === modified file 'storage/innobase/lock/lock0lock.cc' > --- storage/innobase/lock/lock0lock.cc 2014-02-26 18:22:48 +0000 > +++ storage/innobase/lock/lock0lock.cc 2014-03-21 09:32:54 +0000 > @@ -1020,6 +1020,23 @@ > return(FALSE); > } > > + if (trx->group_commit_id != 0 && > + lock2->trx->group_commit_id != 0 && > + trx->group_commit_id == lock2->trx->group_commit_id && > + (type_mode & LOCK_GAP || lock_rec_get_gap(lock2))) { > + /* No lock request needs to wait GAP locks if > + transactions belong to the same group commit group. > + This is because we have already verified these > + transactions on master that they do not conflict > + on record locks but as they could be executed on > + different order on slave we should not wait for > + gap locks here. Note that parallel replication > + makes sure that transaction commits are executed > + in the same order in master and slave. */ > + > + return (FALSE); > + } > + > return(TRUE); > } > > > === modified file 'storage/xtradb/handler/ha_innodb.cc' > --- storage/xtradb/handler/ha_innodb.cc 2014-02-26 18:21:23 +0000 > +++ storage/xtradb/handler/ha_innodb.cc 2014-03-20 15:42:41 +0000 > @@ -2308,6 +2308,8 @@ > > trx->fake_changes = THDVAR(thd, fake_changes); > > + trx->group_commit_id = thd_group_commit_id(thd); > + > #ifdef EXTENDED_SLOWLOG > if (thd_log_slow_verbosity(thd) & (1ULL << SLOG_V_INNODB)) { > trx->take_stats = TRUE; > > === modified file 'storage/xtradb/include/trx0trx.h' > --- storage/xtradb/include/trx0trx.h 2014-02-26 18:21:23 +0000 > +++ storage/xtradb/include/trx0trx.h 2014-03-20 15:28:20 +0000 > @@ -861,6 +861,7 @@ > or the state last time became > TRX_STATE_ACTIVE */ > trx_id_t id; /*!< transaction id */ > + ib_uint64_t group_commit_id;/*!< Group commit id or 0 */ > XID xid; /*!< X/Open XA transaction > identification to identify a > transaction branch */ > > === modified file 'storage/xtradb/lock/lock0lock.cc' > --- storage/xtradb/lock/lock0lock.cc 2013-12-22 16:06:50 +0000 > +++ storage/xtradb/lock/lock0lock.cc 2014-03-21 09:32:46 +0000 > @@ -1021,6 +1021,23 @@ > return(FALSE); > } > > + if (trx->group_commit_id != 0 && > + lock2->trx->group_commit_id != 0 && > + trx->group_commit_id == lock2->trx->group_commit_id && > + (type_mode & LOCK_GAP || lock_rec_get_gap(lock2))) { > + /* No lock request needs to wait GAP locks if > + transactions belong to the same group commit group. > + This is because we have already verified these > + transactions on master that they do not conflict > + on record locks but as they could be executed on > + different order on slave we should not wait for > + gap locks here. Note that parallel replication > + makes sure that transaction commits are executed > + in the same order in master and slave. */ > + > + return (FALSE); > + } > + > return(TRUE); > }

1 0

[Maria-developers] [GSoC 2014] CREATE OR REPLACE, CREATE IF NOT EXISTS and DROP IF EXISTS
by sriram patil 21 Mar '14

21 Mar '14

Hi Sergei, I just went through WL#3129 (thanks to James Briggs for the link). It gave me a detailed insight on what exactly is expected. I have missed 2 object types in my proposal which are, USER and SERVER. I can edit my proposal to incorporate the changes in the deliverables timeline if you want me to. Thanks, Sriram

1 0

Re: [Maria-developers] [GSOC 2014] "CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS" Project
by Sergei Golubchik 21 Mar '14

21 Mar '14

Hi, Sriram! First: please don't forget to cc: the list maria-developers(a)lists.launchpad.net Thank you. On Mar 18, sriram patil wrote: > Hi Sergei, > > I was going through the source code and was able to figure out the > flow of query execution and major functions through which a query goes > before execution. > > The query execution majorly dwells in the "sql" directory of the > source code. So, I think I will have to make changes in the following > files throughout the project, > > - sql_parse.cc (do_command, dispatch_command, mysql_execute_command) Yes. And also sql_yacc.yy - to extend the parser. And the implementation of these commands, for example, sql_acl.cc for CREATE/DROP USER, sql_trigger.cc for CREATE/DROP TRIGGER, etc. > - sql_prepare.cc (Prepared_statement::execute_loop, > Prepared_statement::execute) No, hopefully, this will work automatically. > What I am not sure about is will I have to create a handler for some of > these commands? You are not creating new commands > And will lower level storage engine code changes required? I don't think so. > > > I am interested to work on "CREATE OR REPLACE, CREATE IF NOT EXISTS, and > > > DROP IF EXISTS" project as part of GSOC 2014. I want to apply my subject > > > knowledge at a bigger level which will also give me an idea about the > > > practical issues while implementing the features. > > > Regards, Sergei

3 9

[Maria-developers] [GSoC] MDEV-371- index on BLOB
by smit hinsu 21 Mar '14

21 Mar '14

Hi Sergei and Roberto, My name is Smit Hinsu and I am final year undergrad student. I have good knowledge about database internals and good experience with C++. I find feature of creating index on BLOB quickly using MD-5 interesting. I have understood basic requirements for this feature. I have following questions in my mind currently. I think, HASH index is not supported in MySQL's version of MyISAM storage engine. Has MariaDB removed this restriction? We should support index for BLOB for BTREE index also in which we store only hash value of the value in BTREE. Is this possible? Can you suggest me some startup task related to index or storage engine which can make me familiar with the related parts? I have started to look at code. From Smit Hinsu

2 2

[Maria-developers] Engine independent index statistics - infrastructure questions
by Sergey Petrunia 20 Mar '14

20 Mar '14

Hello Igor and others, I've been playing around with engine independent statistics, and got a few questions. == mysqldump == mysqldump dumps table' data, but besides that it can dump other related objects, e.g. it will dump triggers that are attached to the table. Should it dump table statistics values, also? This question is of interest for DBAs, but also for optimizer developers. If a user has a complaint about the optimizer, we frequently ask them to upload a dataset. How do we request table/column statistics? We can't request output of SELECT * ..., because histogram data is binary and will be corrupted. We should provide users a convinent way to pass us relevant part of the EITS tables. == dropping table statistics == How does one drop table statistics? are we satisfied with needing to run three statements DELETE FROM mysql.column_stats WHERE schema_name=$db AND table_name=$table_name; DELETE FROM mysql.index_stats WHERE table_name=$db AND table_name=$table_name; DELETE FROM mysql.column_stats WHERE table_name=$db AND table_name=$table_name; or we want a simpler SQL, like DROP STATISTICS FOR t1; == Permissions on statistic tables == What should be the permissions on statistics tables? I guess we should follow pattern from other system tables - a user is typically able to read the table but not write to it (so that he can't mess with other users' SPs/triggers/etc). However, if the user is not allowed to modify the table directly, we should provide enough ways to modify the table indirectly. For example, we should provide a way to drop statistics for one's tables (which brings us back to the previous item). We might also want to allow to make other modifications to statistics of tables that one has control of? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] [GSoC 2014]
by Ivan Nikulin 20 Mar '14

20 Mar '14

Hello, I would like to take part into one of your projects on base of GSoC 2014. I am an experienced C/C++ programmer with wide knowledge of algorithms and data structures and huge interest into this sphere. I have some basic knowledge of SQL language, but unfortunatelly I don't have any database development experience. Nevertheless, gaining the latter is my essential purpose. Can I count on a chance of taking part in one of your projects? Sorry for writing to you that late. P.S. I tried writing to you a bit earlier, but the mail wouldn't appear on mailing list. -- Best regards, Ivan Nikulin

2 1

Re: [Maria-developers] [Commits] Rev 4053: MDEV-5864 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.free_tables
by Sergei Golubchik 19 Mar '14

19 Mar '14

Hi, Sergey! On Mar 15, Sergey Vojtovich wrote: > At lp:maria/10.0 > > ------------------------------------------------------------ > revno: 4053 > revision-id: svoj(a)mariadb.org-20140315185301-ce4r6br023co80dm > parent: sanja(a)montyprogram.com-20140314073116-nklb0jkjd4t1w40n > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Sat 2014-03-15 22:53:01 +0400 > message: > MDEV-5864 - Reduce usage of LOCK_open: TABLE_SHARE::tdc.free_tables > > Let TABLE_SHARE::tdc.free_tables, TABLE_SHARE::tdc.all_tables, > TABLE_SHARE::tdc.flushed and corresponding invariants be protected by > per-share TABLE_SHARE::tdc.LOCK_table_share instead of global LOCK_open. This looks pretty good to me! A couple of comments, see below. > === modified file 'sql/sql_plist.h' > --- a/sql/sql_plist.h 2013-11-20 11:05:39 +0000 > +++ b/sql/sql_plist.h 2014-03-15 18:53:01 +0000 > @@ -142,6 +142,14 @@ class I_P_List : public C, public I > > return result; > } > + inline T *back() > + { > + T *result= m_first; > + if (result) > + while (*B::next_ptr(result)) > + result= *B::next_ptr(result); > + return result; > + } I think this is wrong. The correct way would be to use I_P_List<> with the I_P_List_fast_push_back policy, then you could use have get_last() method to get the last element in the list. May be as inline T *back() { return *I::get_last(); } > void swap(I_P_List<T, B, C> &rhs) > { > swap_variables(T *, m_first, rhs.m_first); > > === modified file 'sql/table_cache.cc' > --- a/sql/table_cache.cc 2014-03-06 12:19:12 +0000 > +++ b/sql/table_cache.cc 2014-03-15 18:53:01 +0000 > @@ -70,13 +70,6 @@ static int32 tc_count; /**< Number of TA > > > /** > - Protects TABLE_SHARE::tdc.free_tables, TABLE_SHARE::tdc.all_tables. > -*/ > - > -mysql_mutex_t LOCK_open; So, you've now removed LOCK_open completely? Cool! > - > - > -/** > Protects unused shares list. > > TABLE_SHARE::tdc.prev > @@ -360,12 +378,15 @@ bool tc_release_table(TABLE *table) > table->in_use= 0; > /* Add table to the list of unused TABLE objects for this share. */ > table->s->tdc.free_tables.push_front(table); > - mysql_mutex_unlock(&LOCK_open); > + mysql_mutex_unlock(&table->s->tdc.LOCK_table_share); > return false; > > purge: > + /* Wait for MDL deadlock detector */ > + while (table->s->tdc.all_tables_refs) > + mysql_cond_wait(&table->s->tdc.COND_release, &table->s->tdc.LOCK_table_share); you're using it many times, I'd put it in a small static inline function, like static wait_for_mdl_deadlock_detector(TABLE *table) { mysql_mutex_assert_owner(&table->s->tdc.LOCK_table_share); while (table->s->tdc.all_tables_refs) mysql_cond_wait(&table->s->tdc.COND_release, &table->s->tdc.LOCK_table_share); } > tc_remove_table(table); > - mysql_mutex_unlock(&LOCK_open); > + mysql_mutex_unlock(&table->s->tdc.LOCK_table_share); > table->in_use= 0; > intern_close_table(table); > return true; Regards, Sergei

2 2

[Maria-developers] [GSOC 2014] GIS Enhancement
by bang honam 19 Mar '14

19 Mar '14

Hi everybody. I am Honam Bang, a researcher of Korean Institute of Science Technology and I will be a master student of UCSD, USA from this Sep. I have been working with GIS research for about one year. Therefore, I decided to work on the 'GIS Enhancement topic' because I have a lot of interest for dealing with geospatial data. My topic is 'implementing shp to sql command tool for MariaDB'. #MDEV-5813. Most shptosql tools are enable to convert 3D coordinates as I know. The shape file white paper( http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf) specifies a lot of 3D coordinates like POINT Z, MULTIPOINTZ which does not exist in MariaDB's Geometry type. Does my topic include implementing those type converting as well? Thank you Honam Bang

2 1

[Maria-developers] first results from replication benchmarks
by Axel Schwenke 18 Mar '14

18 Mar '14

Hi everybody, here are some first results from my benchmarks of replication / group commit performance. The benchmark is sysbench OLTP rw, data set fitting in memory. I tested a total of 27 configurations, 9 each for Maria-5.5.36, Maria-10.0.9 and MySQL-5.6.16. Variations were for binlog on/off, SBR vs. RBR and sync_binlog=0/1. For MariaDB I also tested XtraDB vs. InnoDB, for MySQL it was GTID on vs. GTID off. Not surprisingly turning on the binlog comes with a performance penalty. Also sync_binlog=1 adds another penalty. Quite consistently RBR was a little bit slower than SBR for all combinations (I also observed the binlogs to be bigger for RBR by a factor of nearly 2). The only real surprise was the cost of GTID in MySQL as can be seen in the attached plot. MariaDB-10.0 has GTID enabled by default (you cannot even turn it off) Right now the same benchmark is running again, this time using a single disk for binlogs (was a fast RAID before). Next step is applying binlogs to a slave with different settings for number of slave threads etc. XL

2 2

[Maria-developers] Gsoc proposal submitted for user defined event
by Savita Agrawal 18 Mar '14

18 Mar '14

Hi, I have submitted my proposal for user defined event support project. I am still working on the proposal. Please give feedback. Regards, Savita

1 0

[Maria-developers] Group commit id in mysqlbinlog output
by Kristian Nielsen 18 Mar '14

18 Mar '14

There was a question on the output of mysqlbinlog related to group commit. In 10.0, if two transactions group commit together on the master, their GTID event contains a "commit id" - a 64-bit number. This is used by parallel replication; if two transactions have the same commit id, they can be executed in parallel. In mysqlbinlog output, this looks like this: #140314 13:42:56 server id 1 end_log_pos 772 GTID 0-1-12 cid=180 ... #140314 13:42:56 server id 1 end_log_pos 1027 GTID 0-1-13 cid=180 But if a transaction commits alone on the master (no other transactions participate in the group commit), the mysqlbinlog outpus has no commit id: #140314 13:42:56 server id 1 end_log_pos 437 GTID 0-1-10 The question was why there is no cid=X in the second case, as it makes scripting/grepping the output harder. The reason is that there is no commit id in the event in the binlog in this case (to reduce the size of the binlog). So there is no valid number to put in there. One option could be to make the output something like this: #140314 13:42:56 server id 1 end_log_pos 437 GTID 0-1-10 cid=<none> This would allow to grep for "cid=" and catch everything. I do not have a strong opinion one way or the other, if there are people who find this useful, then let me know and I can change it. - Kristian.

4 6

Re: [Maria-developers] rbr triggers (all)
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, Oleksandr! Ok to push. See a couple of comments below. On Mar 18, Oleksandr Byelkin wrote: > === added file 'mysql-test/suite/rpl/t/rpl_row_triggers.test' > --- mysql-test/suite/rpl/t/rpl_row_triggers.test 1970-01-01 00:00:00 +0000 > +++ mysql-test/suite/rpl/t/rpl_row_triggers.test 2014-03-11 08:49:31 +0000 > @@ -0,0 +1,280 @@ > +-- source include/have_binlog_format_row.inc > +-- source include/have_rbr_triggers.inc > +-- source include/master-slave.inc > +-- source include/have_innodb.inc Note that master-slave.inc should always be the last in the series of includes. Everything that can fail should be before it, otherwise you can abort and skip the test after configuring replication, but without a corresponding replication cleanup. It will affect following tests. > === added file 'mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test' > --- mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test 1970-01-01 00:00:00 +0000 > +++ mysql-test/suite/rpl/t/rpl_row_triggers_sbr.test 2014-03-11 08:49:48 +0000 > @@ -0,0 +1,43 @@ > +--source include/master-slave.inc > +--source include/have_binlog_format_statement.inc > +--source include/have_rbr_triggers.inc Same here. > === modified file 'sql/log_event.cc' > --- sql/log_event.cc 2014-03-05 22:20:10 +0000 > +++ sql/log_event.cc 2014-03-11 18:39:51 +0000 > @@ -11121,6 +11233,13 @@ Rows_log_event::write_row(rpl_group_info > TABLE *table= m_table; // pointer to event's table > int error; > int UNINIT_VAR(keynum); > + > +#ifdef RBR_TRIGGERS > + bool invoke_triggers= > + slave_run_triggers_for_rbr && !master_had_triggers && table->triggers; > +#else > +#define invoke_triggers 0 > +#endif //RBR_TRIGGERS I think you don't need this ifdef, just declare invoke_triggers as const: const bool invoke_triggers= slave_run_triggers_for_rbr && !master_had_triggers && table->triggers; > auto_afree_ptr<char> key(NULL); > > prepare_record(table, m_width, Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3994: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, Holyfoot! On Dec 08, holyfoot(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3994 > revision-id: holyfoot(a)askmonty.org-20131208143921-3veatgg9wngyzb47 > parent: holyfoot(a)askmonty.org-20131202104753-srqqm7hv1rtcurx4 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Sun 2013-12-08 18:39:21 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > Thread can be disconnected internally for example after COMMIT statements. What do you mean by that? > So we should check this for the statement execution. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3993: MDEV-5138 Numerous test failures in "mtr --ps --embedded"
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, holyfoot! On Dec 02, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3993 > revision-id: holyfoot(a)askmonty.org-20131202104753-srqqm7hv1rtcurx4 > parent: holyfoot(a)askmonty.org-20131130142537-7j272euoaoy0svg1 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Mon 2013-12-02 14:47:53 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > mysqltest in the 'embedded-server' mode runs queries in a separate thread, > but it didn't do so for the statements - they were run in the main thread. > That leads to inconsistencies. > When a test sets SESSION 'dbug' variable like > SET SESSION debug_dbug="+d,warn_during_ha_commit_trans"; > it is run as a plain query in that separate thread, so the main thread remains unaffected. > After that the statement run in the main thread doesn't produce expected 'dbug' errors, > so the test fail. > To fix that I made statement to be run in that special thread along with the plain queries. > That makes the environment consistent. Ok to push. But sayng "statements" in the comment is a big misleading, it'd be cleared if you'd say "prepared statements". Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3992: MDEV-5138 Numerous test failures in "mtr --ps --embedded". in file:///home/hf/wmar/mdev-5138/
by Sergei Golubchik 18 Mar '14

18 Mar '14

Hi, holyfoot! On Nov 30, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-5138/ > > ------------------------------------------------------------ > revno: 3992 > revision-id: holyfoot(a)askmonty.org-20131130142537-7j272euoaoy0svg1 > parent: holyfoot(a)askmonty.org-20131130132432-etlmes8qhhqhr1iu > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-5138 > timestamp: Sat 2013-11-30 18:25:37 +0400 > message: > MDEV-5138 Numerous test failures in "mtr --ps --embedded". > As Davi added code like > sav_protocol= thd->protocol > thd->protocol= &thd->protocol_binary > ... > thd->protocol= sav_protocol > the fucntions like emb_store_querycache_result() cannot determine > the used protocol testing thd->protocol == &thd->protocol_binary. > Fixed by additional check thd->command == COM_STMT_EXECUTE. What about PREPARE? Regards, Sergei

1 0

Re: [Maria-developers] MaxScale as a binlog server
by Kristian Nielsen 18 Mar '14

18 Mar '14

Mark Riddoch <mark.riddoch(a)skysql.com> writes: > we have been approached by a user with a suggestion for a project that we > might do with them. They have a replication environment with a large fan out > (100 slaves per master) and are looking at doing something to cut one the > load on the master. the thought they had was to use MaxScale to effectively > cache the bin log files, so MaxScale would act as a single Slave to the > Master and would itself have a number of Slaves that read the bin log from > MaxScale. My initial response was why not just use an instance of MariaDB as > the intermediate node and connect slaves to one of those instances, in a > tree structure. They had a few reasons why they did not want to do this; > Looking in more detail I think it is feasible, we essentially write a router > module for MaxSale that acts as a slave, aching the bin log locally on the > MaxScale node. We then have another router, or maybe even the same one, that > relays that binlog to the real slaves. We are looking at doing something for > MariaDB 10, since we would want to utilise the GTID to get the best failover > semantics, I am assuming there must be some differences between the > replication stream that I can find documented for MySQL and what MariaDB 10 > does? Is this stuff anywhere on the Knowledge Base? I tried looking but Right, so I agree with what Serg said, that maybe going back to the original problem and seeing if there are better solutions possible would be better. However, I will try to answer the technical aspect here. I suppose caching and routing binlogs outside of the server should be possible, but it might be more complex than you would think at first. I am not aware of any comprehensive documentation for this. However, it should be relatively easy to see from the code. Basically, it is the function mysql_binlog_send() in sql/sql_repl.cc that handles sending binlog data to a slave. There are good comments in there describing many of the trickier points. You need to be aware that there is more logic in sending binlog data to a slave than just streaming raw binlog files. Maybe you can implement a subset and document any limitations. The old-style replication (not using GTID) is the simplest, but it still does a bit extra, like sending some extra events (FORMAT_DESCRIPTION_EVENT and fake ROTATE_EVENT). There is also the handling of old slaves. A slave sends a value in @mariadb_slave_capability, and the master will rewrite or remove any events that the slave does not understand, as appropriate. And there is the @@skip_replication flag; when the slave sets this, the master removes events that were logged with @@skip_replication set (to reduce bandwidth needs). The slave also does several SQL queries to obtain additional information. For example, the slave does a SELECT binlog_gtid_pos(file, offset) to obtain the GTID position associated with a given old-format replication position, so that it knows how to switch to GTID mode later. Maybe those queries could be handled by relaying them back to a real master server. The GTID mode is significantly more complex, due to a GTID position having multiple streams, one per replication domain. The slave sends @slave_connect_state, @slave_gtid_strict_mode, @slave_gtid_ignore_duplicates, and optionally @slave_until_gtid values. The router would need to parse these values and act accordingly. The binlog sender needs to keep track of each domain as the binlog files are scanned. Events in a domain must be skipped until the point specified in @slave_connect_state is reached. Then events are sent until the point in @slave_until_gtid is reached, at which point further events must again be skipped. The master also sends extra GTID_LIST events, which contain the state of the binlog at the point of the event, which is needed by the slave to correctly handle START SLAVE UNTIL master_gtid_pos=XXX and MASTER_GTID_WAIT(). A lot of effort was put into error handling, to support all reasonable, and even many unreasonable uses of replication, while still giving an error in cases where things are obviously wrong (to prevent silently doing the wrong thing or hanging). For example, if the slave tries to connect at a GTID that the master does not have, this is normally an error. This is important, because that GTID might be some transaction that was executed manually on the slave by mistake; if it only exists on the slave, then the master would endlessly skip events looking for the GTID that never shows up, and replication would silently hang. However, there are special cases where the slave _is_ allowed to connect at a GTID that does not exist on the master. For example, one might have a slave with --log-slave-updates=0. Then if this slave is promoted as a new master, the binlogs will not have the current position, but it is still allowed to connect to it at the point where it became the new master (but not to a point earlier than that). Another special case is if the requested GTID (but no GTIDs following that) was purged from the binary logs, as can easily happen if a replication domain is unused for long time. A _full_ implementation of a router for sending binlog to slaves would be some effort, but it might be reasonable to implement some subset. I suppose the first step would be for you to go through the code in mysql_binlog_send() and the functions it calls, and decide on which parts to implement. I will be happy to answer any questions that might pop up on the way, of course. > 2. They wanted to maintain the same group commit groups as the original > master, so as to benefit from the parallel replication. Note that this may no longer apply to MariaDB 10.0.9. The issue is that an intermediate slave that is itself a master may have less group commit than the original master, which reduces the possibility for a third-level slave to efficiently do parallel replication. But in 10.0.9, it should be possible to configure the intermediate slave with --binlog-commit-wait-* to get better group commit. > 3. They think that can get better failover semantics because they will have > n (maybe 5 or 10) MaxScales and they will all have the same bin log, so > failover between them will be easier. I am curious how it will be assured that they all have the same binlog in case of crash? But maybe you just mean that all will be a subset of the largest one. Hope this helps, - Kristian.

2 1

[Maria-developers] Java Connector Discussion
by Puneet Dewan 18 Mar '14

18 Mar '14

Hi Georg, As we missed our scheduled meeting yesterday, can you suggest a suitable time to discuss? Regards Puneet.

1 0

Re: [Maria-developers] [Maria-discuss] MariaDB vs. MySQL error codes (1900+)
by Kristian Nielsen 17 Mar '14

17 Mar '14

Felipe Gasper <felipe(a)felipegasper.com> writes: > Hi everyone, > > I notice that MariaDB is using the 1900+ range for its own > error codes; however, MySQL 5.7, at least, has a few errors in that > range as well. > > Are error codes just going to be an area of increasing > conflict between the two, or is there a plan to migrate MariaDB codes > to a different numbering scheme? Thanks for bringing this to attention. This is a pretty serious issue, and I've forwarded it to Tomas Ulin, who should be in a position to get it fixed. Tomas, please note this issue. MariaDB has used error codes from 1900 from the very start to avoid these kinds of conflicts. They can not be changed, as the codes are already in GA releases and applications may be relying on the numbers. You will need to ensure that MySQL 5.7 avoids conflicts with these before 5.7 goes GA, or there will be a conflict that will hurt all users and will be very hard to fix later. Felipe, you should probably also file a bug on bugs.mysql.com to try to get this fixed before MySQL 5.7 GA. Thanks, - Kristian.

2 1

[Maria-developers] redundant compilation of targets in mariadb 10.0
by Rohit Kalhans 17 Mar '14

17 Mar '14

hello all, I am trying to fix a few bugs but ever after small modifications in specific files, make command build all targets irrespective of whether or not they are dependent on the modified source. I am no expert in build commands but this seem off since it keeps taking exceptionally long time to build. Is there a trick/workaround that can prevent all targets from being built. Or am I missing something. -- Cheerio! *Rohit* *http://rohitkalhans.com <http://rohitkalhans.com>*

3 6

[Maria-developers] For Google Summer of Code 2014, Interested in the task of "statistically optimize mysql-test runs by running less tests"
by 胡仲义 17 Mar '14

17 Mar '14

Dear Sergei Golubchik, I am a post graduate student of Institute of Software, Chinese Academy of Sciences and my name is Zhongyi Hu. I major in computer science and my research field is data stream mining. Because I have got enough papers and works for graduation, I want to do something interesting, meaningful and valuable in the rest time as student. I have participated in two projects about database, one is main memory database and the other is database cluster. I got some experience of database system design and implementaion from them. Although I am just a beginner of this area, I really like it and expect to make it as my career. I often use Mysql in research and work, but MariaDB is not very familiar to me. I am tremendously optimistic about it's future because all of you. Well, let's come to the point. I am interested in the task of "statistically optimize mysql-test runs by running less tests". I chose this task because I have written a few tools for automatic test. I know the performance is very important if there are a large amount of data or cases to test. I read the MDEV-5776 and I think the major job is as follow. When the code is changed, the mysql-test is used to do the requisite tests. We need to integrate the information of the changes and the scenarios to predict the probability of failure for each test and get the relationships of the tests. Then decide what to test and what test cases should be used. The purpose is to optimize the efficiency of testing. All of these should be done by algorithm and program. In addition, I think that the job is in some ways like mining in data stream, such as many data need to be statistical analyzed and the hidden patterns changing over time. At last, I have two basic questions. 1) What exactly are the builder and the combination? I thought they refer to compiler and runtime environment. 2) What does the "individual tests within a big test file" mean? Maybe I am completely wrong, but I still look forward to your reply. I hope to have the opportunity to learn from you in work and discussion. Best regards Zhongyi Hu 20140314

3 5

[Maria-developers] [GSOC 2014] "CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS" Project
by sriram patil 16 Mar '14

16 Mar '14

Hi Sergei, I am Sriram Patil. I am a Masters' student at International Institute of Information Technology, Hyderabad, India. I have been working on a small DB Engine as part of my course work in C++. I have also worked on a SQLite DB abstraction library FMDB in Objective-C. Also, I am currently working on lock-free concurrent B-trees (Java), and going to evaluate it with an in-memory database to check the performance. I have also worked on libdaap and Rhythembox source code during my undergraduate project. You can check my GitHub<https://github.com/srirampatil?tab=repositories>and BitBucket <https://bitbucket.org/srirampatil> accounts which have some of my projects. I am interested to work on "CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS" project as part of GSOC 2014. I want to apply my subject knowledge at a bigger level which will also give me an idea about the practical issues while implementing the features. I have gone through the project description and understood that a generic interface needs to be implemented for "CREATE OR REPLACE" and "CREATE IF NOT EXISTS" for functions, procedures, tables, triggers, etc. But, I understand I am a pretty late to discuss these things with you. Is it okay if I submit my proposal document directly? Also, do you recommend me to go through some specific things with respect to this project before I start working on my proposal draft? Thanks, Sriram Patil IIIT Hyderabad, India

2 1

[Maria-developers] MDEV-5681- audit log not rotating
by Tan Tran 14 Mar '14

14 Mar '14

Hi maria-developers, This is my patch for MDEV-5681: " audit log will not rotate when the file size exceeds global variable setting": http://bazaar.launchpad.net/~tktran/maria/maria-fix-bug5681/revision/4102 To test, 1. set server_audit_events = 'connect' and keep default rotate size of 1000000. 2. connect with a bad password about 20 times: for i in {1..20}; do mysql -ufail -pfail; done 3. see that server_audit.log is X bytes 4. set server_audit_rotate_size = 1000 5. connect with bad password for another 20 times 6. see original server_audit.log is still X bytes (meaning log rotated immediately) and subsequent server_audit.log.* logs are ~1000 bytes or less Please review. Thanks, Tan Tran Prospective GSoC student

2 1

Re: [Maria-developers] [Commits] Rev 4066: MDEV-5089 - possible deadlocks between rwlocks and mutexes
by Sergei Golubchik 13 Mar '14

13 Mar '14

Hi, Sergey! On Feb 13, Sergey Vojtovich wrote: > revno: 4066 > revision-id: svoj(a)mariadb.org-20140213074049-70zbqa98y95sfpox > parent: sanja(a)askmonty.org-20140212150705-e1tfabmhjggqws7q > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5 > timestamp: Thu 2014-02-13 11:40:49 +0400 > message: > MDEV-5089 - possible deadlocks between rwlocks and mutexes > > Pre-MDL versions had direct relationship between LOCK_open and > LOCK_global_system_variables, e.g.: > intern_sys_var_ptr // locks LOCK_global_system_variable > mysql_sys_var_char > create_options_are_valid > ha_innobase::create > handler::ha_create > ha_create_table > rea_create_table > mysql_create_table_no_lock // locks LOCK_open > mysql_create_table > > With MDL this relationship was removed, but mutex order was still > recorded. In fact there is indirect relationship between LOCK_open > and LOCK_global_system_variables via rwlocks in reverse order. > > Removed LOCK_open and LOCK_global_system_variables order recording, > instead assert that LOCK_open is never held in intern_sys_var_ptr(). > > This solves only one of many problems detected with MDEV-5089. Looks good, thanks! ok to push! Regards, Sergei

1 0

[Maria-developers] TokuDB Performance Regression (MDEV-5777)
by Tim Callaghan 13 Mar '14

13 Mar '14

I'm very concerned about the regression I reported in https://mariadb.atlassian.net/browse/MDEV-5777, as users of the MariaDB.org supplied builds of both 5.5 and 10.0 are getting substantially less throughput than the MariaDB 5.5 build supplied by us at tokutek.com. I see that the fix version is now 5.5.37 and 10.0.10, is there a plan for how that will get done? I'll assist if possible, I just don't know how I can. -Tim Callaghan (Tokutek)

2 1

[Maria-developers] Fwd: Database/kernel community topic at Collaboration Summit 2014
by Hartmut Holzgraefe 13 Mar '14

13 Mar '14

Forwarding this as the original message doesn't seem to have made it to this list ... -------- Original Message -------- Subject: Database/kernel community topic at Collaboration Summit 2014 Date: Mon, 10 Mar 2014 10:15:37 +0000 From: Mel Gorman <mgorman(a)suse.de> To: pgsql-hackers(a)postgresql.org, mysql(a)lists.mysql.com, maria-developers(a)lists.launchpad.net, linux-kernel(a)vger.kernel.org Hi, Arrangements have been made to hold a meeting between database and kernel developers at Collaboration Summit 2014 http://sched.co/1hEBRuq on March 27th 2014. This was organised after discussions on pain points encountered by the PostgreSQL community. Originally the plan had been to just have a topic for LSF/MM there was much more interest in the topic than anticipated so the Collaboration Summit meeting will be much more open. If there are developers attending Collaboration Summit that work in the database or kernel communities, it would be great if you could come along. Previous discussions were on the PostgreSQL list and that should be expanded in case we accidentally build postgres-only features. The intent is to identify the problems encountered by databases and where relevant, test cases that can be used to demonstrate them if they exist. While the kernel community may be aware of some of the problems, they are not always widely known or understood. There is a belief that some interfaces are fine when in reality applications cannot use them properly. The ideal outcome of the meeting would be concrete proposals on kernel features that could be developed over the course of time to address any identified problem. For reference, this is a summary of the discussion that took place when the topic was proposed for LSF/MM. Thanks. ---8<--- On testing of modern kernels ---------------------------- Josh Berkus claims that most people are using Postgres with 2.6.19 and consequently there may be poor awareness of recent kernel developments. This is a disturbingly large window of opportunity for problems to have been introduced. Minimally, Postgres has concerns about IO-related stalls which may or may not exist in current kernels. There were indications that large writes starve reads. There have been variants of this style of bug in the past but it's unclear what the exact shape of this problem is and if IO-less dirty throttling affected it. It is possible that Postgres was burned in the past by data being written back from reclaim context in low memory situations. That would have looked like massive stalls with drops in IO throughput but it was fixed in relatively recent kernels. Any data on historical tests would be helpful. Alternatively, a pgbench-based reproduction test could potentially be used by people in the kernel community that track performance over time and have access to a suitable testing rig. It was mentioned that Postgres has an tool called pg_test_fsync which was mentioned in the context of testing different wal_sync_methods. Potentially it could also be used for evaluating some kernel patches. Gregory Smith highlighted the existence of a benchmark wrapper for pgbench called pgbench-tools: https://github.com/gregs1104/pgbench-tools . It can track statistics of interest to Postgres as well as report in interesting metrics such as transaction latency. He had a lot of information on testing requirements and some very interesting tuning information and it's worth reading the whole mail http://www.postgresql.org/message-id/52D99161.60305@gmail.com Postgres bug reports and LKML ----------------------------- It is claimed that LKML does not welcome bug reports but it's less clear what the basis of this claim is. Is it because the reports are ignored? A possible explanation is that they are simply getting lost in the LKML noise and there would be better luck if the bug report was cc'd to a specific subsystem list. A second possibility is the bug report is against an old kernel and unless it is reproduced on a recent kernel the bug report will be ignored. Finally it is possible that there is not enough data available to debug the problem. The worst explanation is that to date the problem has not been fixable but the details of this have been lost and are now unknown. Is is possible that some of these bug reports can be refreshed so at least there is a chance they get addressed? Apparently there were changes to the reclaim algorithms that crippled performance without any sysctls. The problem may be compounded by the introduction of adaptive replacement cache in the shape of the thrash detection patches currently being reviewed. Postgres investigated the use of ARC in the past and ultimately abandoned it. Details are in the archives (http://www.Postgres.org/search/?m=1&q=arc&l=1&d=-1&s=r) I have not read then, just noting they exist for future reference. Sysctls to control VM behaviour are not popular as such tuning parameters are often used as an excuse to not properly fix the problem. Would it be possible to describe a test case that shows 2.6.19 performing well and a modern kernel failing? That would give the VM people a concrete basis to work from to either fix the problem or identify exactly what sysctls are required to make this work. I am confident that any bug related to VM reclaim in this area has been lost. At least, I recall no instances of it being discussed on linux-mm and it has not featured on LSF/MM during the last years. IO Scheduling ------------- Kevin Grittner has stated that it is known that the DEADLINE and NOOP schedulers perform better than any alternatives for most database loads. It would be desirable to quantify this for some test case and see can the default scheduler cope in some way. The deadline scheduler makes sense to a large extent though. Postgres is sensitive to large latencies due to IO write spikes. It is at least plausible that deadline would give more deterministic behaviour for parallel reads in the presence of large writes assuming there were not ordering problems between the reads/writes and the underlying filesystem. For reference, these IO spikes can be massive. If the shared buffer is completely dirtied in a short space of time then it could be 20-25% of RAM being dirtied and writeback required in typical configurations. There have been cases where it was worked around by limiting the size of the shared buffer to a small enough size so that it can be written back quickly. There are other tuning options available such as altering when dirty background writing starts within the kernel but that will not help if the dirtying happens in a very short space of time. Dave Chinner described the considerations as follows There's no absolute rule here, but the threshold for background writeback needs to consider the amount of dirty data being generated, the rate at which it can be retired and the checkpoint period the application is configured with. i.e. it needs to be slow enough to not cause serious read IO perturbations, but still fast enough that it avoids peaks at synchronisation points. And most importantly, it needs to be fast enought that it can complete writeback of all the dirty data in a checkpoint before the next checkpoint is triggered. In general, I find that threshold to be somewhere around 2-5s worth of data writeback - enough to keep a good amount of write combining and the IO pipeline full as work is done, but no more. e.g. if your workload results in writeback rates of 500MB/s, then I'd be setting the dirty limit somewhere around 1-2GB as an initial guess. It's basically a simple trade off buffering space for writeback latency. Some applications perform well with increased buffering space (e.g. 10-20s of writeback) while others perform better with extremely low writeback latency (e.g. 0.5-1s). Some of this may have been addressed in recent changes with IO-less dirty throttling. When considering stalls related to excessive IO it will be important to check if the kernel was later than 3.2 and what the underlying filesystem was. Again, it really should be possible to demonstrate this with a test case, one driven by pgbench maybe? Workload would generate a bunch of test data, dirty a large percentage of it and try to sync. Metrics would be measuring average read-only query latency when reading in parallel to the write, average latencies from the underlying storage, IO queue lengths etc and comparing default IO scheduler with deadline or noop. NUMA Optimisations ------------------ The primary one that showed up was zone_reclaim_mode. Enabling that parameter is a disaster for many workloads and apparently Postgres is one. It might be time to revisit leaving that thing disabled by default and explicitly requiring that NUMA-aware workloads that are correctly partitioned enable it. Otherwise NUMA considerations are not that much of a concern right now. Bruce Momjian highlighted this block entry that covered zone_reclaim http://frosty-postgres.blogspot.com/2012/08/postgresql-numa-and-zone-reclai… Direct IO, buffered IO, double buffering and wishlists ------------------------------------------------------ The general position of Postgres is that the kernel knows more about storage geometries and IO scheduling that an application can or should know. It would be preferred to have interfaces that allow Postgres to give hints to the kernel about how and when data should be written back. The alternative is exposing details of the underlying storage to userspace so Postgres can implement a full IO scheduler using direct IO. It has been asserted on the kernel side that the optimal IO size and alignment is the most important detail should be all the details that are required in the majority of cases. While some database vendors have this option, the Postgres community do not have the resources to implement something of this magnitude. They also have tried direct IO in the past in the areas where it should have mattered and had mixed results. I can understand Postgres preference for using the kernel to handle these details for them. They are a cross-platform application and the kernel should not be washing its hands of the problem and hiding behind direct IO as a solution. Ted Ts'o summarises the issues as The high order bit is what's the right thing to do when database programmers come to kernel engineers saying, we want to do <FOO> and the performance sucks. Do we say, "Use O_DIRECT, dummy", not withstanding Linus's past comments on the issue? Or do we have some general design principles that we tell database engineers that they should do for better performance, and then all developers for all of the file systems can then try to optimize for a set of new API's, or recommended ways of using the existing API's? In an effort to avoid depending on direct IO there were some proposals and/or wishlist items. These are listed in order of likliehood to be implemented and usefulness to Postgres. 1. Hint to asynchronously queue writeback now in preparation for a fsync in the near future. Postgres dirties a large amount of data and asks the kernel to push it to disk over the next few minutes. Postgres still is required to fsync later but the fsync time should be minimised. vm.dirty_writeback_centisecs is unreliable for this. One possibility would be an fadvise call that queues the data for writeback by a flusher thread now and returns immediately 2. Hint that a page is a prime candidate for reclaim but only if there is reclaim pressure. This avoids a problem where fadvise(DONTNEED) discards a page only to have a read/write or WILLNEED hint immediately read it back in again. The requirements are similar to the volatile range hinting but they do not use mmap() currently and would need a file-descriptor based interface. Robert Hass had some concerns with the general concept and described them thusly This is an interesting idea but it stinks of impracticality. Essentially when the last buffer pin on a page is dropped we'd have to mark it as discardable, and then the next person wanting to pin it would have to check whether it's still there. But the system call overhead of calling vrange() every time the last pin on a page was dropped would probably hose us. Well, I guess it could be done lazily: make periodic sweeps through shared_buffers, looking for pages that haven't been touched in a while, and vrange() them. That's quite a bit of new mechanism, but in theory it could work out to a win. vrange() would have to scale well to millions of separate ranges, though. Will it? And a lot depends on whether the kernel makes the right decision about whether to chunk data from our vrange() vs. any other page it could have reclaimed. 3. Hint that a page should be dropped immediately when IO completes. There is already something like this buried in the kernel internals and sometimes called "immediate reclaim" which comes into play when pages are bgin invalidated. It should just be a case of investigating if that is visible to userspace, if not why not and do it in a semi-sensible fashion. 4. 8kB atomic write with OS support to avoid writing full page images in the WAL. This is a feature that is likely to be delivered anyway and one that Postgres is interested in. 5. Only writeback some pages if explicitly synced or dirty limits are violated. Jeff Janes states that he has problems with large temporary files that generate IO spikes when the data starts hitting the platter even though the data does not need to be preserved. Jim Nasby agreed and commented that he "also frequently see this, and it has an even larger impact if pgsql_tmp is on the same filesystem as WAL. Which *theoretically* shouldn't matter with a BBU controller, except that when the kernel suddenly +decides your *temporary* data needs to hit the media you're screwed." One proposal that may address this is Allow a process with an open fd to hint that pages managed by this inode will have dirty-sticky pages. Pages will be ignored by dirty background writing unless there is an fsync call or dirty page limits are hit. The hint is cleared when no process has the file open. 6. Only writeback pages if explicitly synced. Postgres has strict write ordering requirements. In the words of Tom Lane -- "As things currently stand, we dirty the page in our internal buffers, and we don't write it to the kernel until we've written and fsync'd the WAL data that needs to get to disk first". mmap() would avoid double buffering but it has no control about the write ordering which is a show-stopper. As Andres Freund described; Postgres' durability works by guaranteeing that our journal entries (called WAL := Write Ahead Log) are written & synced to disk before the corresponding entries of tables and indexes reach the disk. That also allows to group together many random-writes into a few contiguous writes fdatasync()ed at once. Only during a checkpointing phase the big bulk of the data is then (slowly, in the background) synced to disk. I don't see how that's doable with holding all pages in mmap()ed buffers. There are also concerns there would be an absurd number of mappings. The problem with this sort of dirty pinning interface is that it can deadlock the kernel if all dirty pages in the system cannot be written back by the kernel. James Bottomley stated No, I'm sorry, that's never going to be possible. No user space application has all the facts. If we give you an interface to force unconditional holding of dirty pages in core you'll livelock the system eventually because you made a wrong decision to hold too many dirty pages. However, it was very clearly stated that the writing ordering is critical. If the kernel breaks the requirement then the database can get trashed in the event of a power failure. This led to a discussion on write barriers which the kernel uses internally but there are scaling concerns both with the number of constraints that would exist and the requirement that Postgres use mapped buffers. There were few solid conclusions on this. It would need major reworking on all sides and it would handing control of system safety to userspace which is going to cause layering violations. This whole idea may be a bust but it is still worth recording. Greg Stark outlined the motivation best as follows; Ted T'so was concerned this would all be a massive layering violation and I have to admit that's a huge risk. It would take some clever API engineering to come with a clean set of primitives to express the kind of ordering guarantees we need without being too tied to Postgres's specific implementation. The reason I think it's more interesting though is that Postgres's journalling and checkpointing architecture is pretty bog-standard CS stuff and there are hundreds or thousands of pieces of software out there that do pretty much the same work and trying to do it efficiently with fsync or O_DIRECT is like working with both hands tied to your feet. 7. Allow userspace process to insert data into the kernel page cache without marking the page dirty. This would allow the application to request that the OS use the application copy of data as page cache if it does not have a copy already. The difficulty here is that the application has no way of knowing if something else has altered the underlying file in the meantime via something like direct IO. Granted, such activity has probably corrupted the database already but initial reactions are that this is not a safe interface and there are coherency concerns. Dave Chinner asked "why, exactly, do you even need the kernel page cache here?" when Postgres already knows how and when data should be written back to disk. The answer boiled down to "To let kernel do the job that it is good at, namely managing the write-back of dirty buffers to disk and to manage (possible) read-ahead pages". Postgres has some ordering requirements but it does not want to be responsible for all cache replacement and IO scheduling. Hannu Krosing summarised it best as Again, as said above the linux file system is doing fine. What we want is a few ways to interact with it to let it do even better when working with Postgres by telling it some stuff it otherwise would have to second guess and by sometimes giving it back some cache pages which were copied away for potential modifying but ended up clean in the end. And let the linux kernel decide if and how long to keep these pages in its cache using its superior knowledge of disk subsystem and about what else is going on in the system in general. 8. Allow copy-on-write of page-cache pages to anonymous. This would limit the double ram usage to some extent. It's not as simple as having a MAP_PRIVATE mapping of a file-backed page because presumably they want this data in a shared buffer shared between Postgres processes. The implementation details of something like this are hairy because it's mmap()-like but not mmap() as it does not have the same writeback semantics due to the write ordering requirements Postgres has for database integrity. Completely nuts and this was not mentioned on the list, but arguably you could try implementing something like this as a character device that allows MAP_SHARED with ioctls with ioctls controlling that file and offset backs pages within the mapping. A new mapping would be forced resident and read-only. A write would COW the page. It's a crazy way of doing something like this but avoids a lot of overhead. Even considering the stupid solution might make the general solution a bit more obvious. For reference, Tom Lane comprehensively described the problems with mmap at http://www.Postgres.org/message-id/17515.1389715715@sss.pgh.pa.us There were some variants of how something like this could be achieved but no finalised proposal at the time of writing. 9. Hint that a page in an anonymous buffer is a copy of a page cache page and invalidate the page cache page on COW. This limits the amount of double buffering. It's in as a low priority item as it's unclear if it's really necessary and also I suspect the implementation would be very heavy because of the amount of information we'd have to track in the kernel. It is important to note in general that Postgres has a problem with some files being written back too aggressively and other files not written back aggressively enough. Temp files for purposes such as sorting should have writeback deferred as long as possible. Data file writes that must complete before portions of the WAL can be discarded should begin writeback early so the final fsync does not stall for too long. As Dave Chinner says IOWs, there are two very different IO and caching requirements in play here and tuning the kernel for one actively degrades the performance of the other. Robert Hass categorised the IO patterns as follows - WAL files are written (and sometimes read) sequentially and fsync'd very frequently and it's always good to write the data out to disk as soon as possible - Temp files are written and read sequentially and never fsync'd. They should only be written to disk when memory pressure demands it (but are a good candidate when that situation comes up) - Data files are read and written randomly. They are fsync'd at checkpoint time; between checkpoints, it's best not to write them sooner than necessary, but when the checkpoint arrives, they all need to get out to the disk without bringing the system to a standstill No matter it was pointed out that fsync should never be able to screw the system. Robert Hass again summaried it as follows IMHO, the problem is simpler than that: no single process should be allowed to completely screw over every other process on the system. When the checkpointer process starts calling fsync(), the system begins writing out the data that needs to be fsync()'d so aggressively that service times for I/O requests from other process go through the roof. It's difficult for me to imagine that any application on any I/O scheduler is ever happy with that behavior. We shouldn't need to sprinkle of fsync() calls with special magic juju sauce that says "hey, when you do this, could you try to avoid causing the rest of the system to COMPLETELY GRIND TO A HALT?". That should be the *default* behavior, if not the *only* behavior. It is important to keep this in mind although sometimes the ordering requirements of the filesystem may make it impossible to achieve. At LSF/MM last year there was a discussion on whether userspace should hint that files are "hot" or "cold" so the underlying layers could decide to relocate some data to faster storage. I tuned out a bit during the discussion and did not track what happened with it since but I guess that any developments of that sort would be of interest to the Postgres community. Some of these wish lists still need polish but could potentially be discussed further at LSF/MM with a wider audience as well as on the lists. Then in a of unicorns and ponies it's a case of picking some of these hinting wishlists, seeing what it takes to implement it in kernel and testing it with a suitably patched version of postgres running a test case driven by something (pgbench presumably). -- Mel Gorman SUSE Labs -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql

1 0

[Maria-developers] GSOC Proposal-Java Connector For MariaDB
by Puneet Dewan 12 Mar '14

12 Mar '14

Hi All, Please find attached my initial proposal for Java Connector for Maria DB. Awaiting for your comments.:) Regards Puneet.

1 0

[Maria-developers] InnoDB selected projects (corrected links)
by Jan Lindström 12 Mar '14

12 Mar '14

Hi all, After careful weighting and selection process, I have selected following two projects as a starting point to improve InnoDB (1) InnoDB file space defragmentation Description: External tool to physically delete delete marked rows from InnoDB file space and freeing unused file space. https://mariadb.atlassian.net/browse/MDEV-5834 (2) InnoDB fast bulk load Description: External tool to load CVS file directly to InnoDB file space (innodb_file_per_table = 1). https://mariadb.atlassian.net/browse/MDEV-5835 Now, I hope these projects mostly represent the most frequently requested features and are most usable to current customers and/or potential new customers. Next, I would hope some indication which one of the selected project you would like to see first. Votes can be added above links. R: -- -- Jan Lindström, Principal Engineer SkySQL - The MariaDB Company MariaDB | MaxScale | skype: jan_p_lindstrom www.skysql.com <http://www.skysql.com/> Twitter <http://twitter.com/skysql> Blog <http://www.skysql.com/blog/> Facebook <http://www.facebook.com/skysql> LinkedIn <http://www.linkedin.com/company/1214250> Google+ <https://plus.google.com/117544963211695643458/posts>

1 0

Re: [Maria-developers] [Commits] Rev 3768: MDEV-5717: Server crash with insert statement containing DEFAULT into view
by Sergei Golubchik 12 Mar '14

12 Mar '14

Hi, Sanja! On Mar 12, sanja(a)montyprogram.com wrote: > revno: 3768 > revision-id: sanja(a)montyprogram.com-20140312103416-3v6tan06crsqry28 > parent: sanja(a)montyprogram.com-20140307115707-poaet8iwv9d55azm > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.3-MDEV-5717 > timestamp: Wed 2014-03-12 12:34:16 +0200 > message: > MDEV-5717: Server crash with insert statement containing DEFAULT into view > > Item_default_value::arg can be NULL so walk() should take it into consideration. > === modified file 'sql/item.h' > --- a/sql/item.h 2014-02-19 13:45:33 +0000 > +++ b/sql/item.h 2014-03-12 10:34:16 +0000 > @@ -3551,7 +3551,7 @@ public: > > bool walk(Item_processor processor, bool walk_subquery, uchar *args) > { > - return arg->walk(processor, walk_subquery, args) || > + return (arg && arg->walk(processor, walk_subquery, args)) || > (this->*processor)(args); > } Ok to push. But *please* install the diff_p bzr plugin as described here: https://mariadb.com/kb/en/how-to-get-more-out-of-bzr-when-working-on-mariad… Regards, Sergei

1 0

Re: [Maria-developers] Maria DB project aspirant for GSoC 2014 : CREATE OR REPLACE, CREATE IF NOT EXISTS, and DROP IF EXISTS
by Sergei 11 Mar '14

11 Mar '14

Hi, Siddharth! First: please send questions like this to maria-developers(a)lists.launchpad.net it's a public mailing list dedicated to MySQL and MariaDB internals, source code, and related things. I am subscribed, so I'll see you mail there, and you may be sure I will, because it won't be accidentally catched by my spam filter, or sorted out in some obscure folder. Furthermore other subscribers will see your question and could reply if I will be not available (e.g. I could be travelling). Thank you. On Mar 11, Siddharth Dalmia wrote: > Hi Sergei, > > I am interested in working on the project. I am new to the open source > world so, can you tell how i should go forward? Do I need to work on > something and show? No, this is not required, but if you do it'll be a bonus point for you - you'll get familiar with MariaDB source code and would be able to write a better proposal. Everything you have to do and how you should move forward is written at the GSoC site. See http://www.google-melange.com/gsoc/homepage/google/gsoc2014 Regards, Sergei

1 0

[Maria-developers] GSOC 14: Discussion on "GIS enhancements to MariaDB"
by Abhishek Kumar 11 Mar '14

11 Mar '14

Hi, I am Abhishek and am interested in knowing about the gsoc project "GIS enhancements to MariaDB". I participate often in the programming contents hosted at these sites: Codeforces: abhi170893 (Division 1) TopCoder: abhi170893 ( Divsion 1) Codechef: abhi_iiit_17 ( Ranked 6 in India in DEC13 . 3 times in top 20 in country.) Spoj: abhi170893 (over 100 problems) The ideas page mentions C as the required skill and I am good at coding in C. Also on the ideas page not much is written about the project. Where can i get more details about it..? what things should i explore first before trying for this project..? the complexity level..? the competition involved..?Do i need to fix bugs first..? If yes, their links..? Thanks Abhishek Kumar CSE, IIIT Hyderabad (India)

2 1

Re: [Maria-developers] Rev 4033: MDEV-5766 - my_atomic_load does memory writes
by Sergei Golubchik 10 Mar '14

10 Mar '14

Hi, Sergey! On Mar 07, Sergey Vojtovich wrote: > revno: 4033 > revision-id: svoj(a)mariadb.org-20140307074306-i3l4fllu2rj5pouf > parent: svoj(a)mariadb.org-20140306121912-i1p8iilpsgc8m4bn > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 10.0 > timestamp: Fri 2014-03-07 11:43:06 +0400 > message: > MDEV-5766 - my_atomic_load does memory writes > > my_atomic_load() is implemented as __sync_fetch_and_or(var, 0) which > writes or-ed value back to var. Memory writes as such have worse > performance and scalability than reads. > > gcc 4.7 and up offers better facility for atomic loads/stores. Use it > whenever it is available. Thanks. ok to push Regards, Sergei

1 0

[Maria-developers] Some questions on GSOC project User Defined Event
by Savita Agrawal 10 Mar '14

10 Mar '14

Hi, what is the difference between a trigger and user defined event? In postgres there is a clear distintion between trigger and event trigger. Triggers are activated when DML statements are executed whereas event triggers are activated when DDL statements are executed. But in soliddb (from which our post in JIRA page is inspired), it seems like user defined events can be activated for DML statements. What is it that can't be done with triggers and can be done with user defined event? Why is the restriction that events can only be posted and received by stored procedures only? Cheers, Savita

3 6