June 2014 - developers - lists.mariadb.org

[Maria-developers] MariaDB Debian packaging: pending merge to upstream
by Otto Kekäläinen 03 Oct '14

03 Oct '14

Hello! Now when the rehearsal is done using simple spelling error fix patches, I now posted the first big merge request about my 5.5/debian upon the upstream 5.5 branch. Issue at https://mariadb.atlassian.net/browse/MDEV-6284 Merge request at https://code.launchpad.net/~otto/maria/maria-fix-debpkg-5.5/+merge/221576 I also have these old merge request still open: https://code.launchpad.net/~otto/maria/maria-fix-debian-copyright-trunk/+me… https://code.launchpad.net/~otto/maria/maria-fix-debian-copyright-5.5/+merg… The sooner somebody has can merge these the less likely it is that there will be merge conflicts.. Thanks! -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 6

Re: [Maria-developers] [Maria-discuss] MariaDB encryption
by Jonas Oreland 23 Sep '14

23 Sep '14

Hi again, > What is the type of license of your code? I asked internally about license, and it seems like we releasing dual gpl2/apache licensed code. > I would like to know, which interfaces from maria-DB you are using. I don't 100% understand the question. We didn't write any actual encryption code, but used the one provided in openssl. Other than that, we didn't really "use interfaces", but rather added/modified functionality/interfaces here and there. Can you be more specific ? /Jonas On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck <elmar(a)eperiesi-beck.de > wrote: > Hi! > We (eperi) would be glad to do a joined work with Google. > Our solution works with MS-SQL, Oracle and other DBs and we are currently > porting it to MariaDB - and - as Monty said - its never to late to put some > sources together and make the best for the open source community. > > What is the type of license of your code? > > Jonas, I am looking forward to connect to you directly. > > Regards > Elmar > > Hi! > > > Hi Jonas, > > (same Jonas we know from NDBCLUSTER? :-) Good to see you again) > > > > On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso(a)google.com> wrote: > > > >> Hi there, > >> I read this blog post > >> > http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.… > >> and wanted to inform you that we at Google has developed > on-disk/block-level encryption for Innodb, aria (as used by temporary > tables), binlogs and temp-files. > >> The code is not yet published, but we expect it to be within a few > weeks or so. > >> We (of course?) think that it would be better if you instead of > developing new code > >> spent the time testing/reviewing ours. > > We are out course happy to do this! > > >> I'm happy to answer questions on the topic, > >> and will let you know once we've published it. > > The main question I have about the Innodb encryption is if it based on the > compression code we did for fusion-io? > The idea we had on our side was that by using the new compression hooks we > could add encryption with very little changes to the Innodb code. > Looking forward to when you are ready to publish the code so we can > discuss your changes in detail. > > > This is great news! > > > > From what I gather, from Monty's blog post (and a 1:1 we had some time > back), this is something done by a partner/external company that has a > mostly OSS solution, that we should integrate into 10.1 > > Yes, that's correct. It I would have known that Google was working on > encryption I would have included them in my discussions with eperi. > Fortunately it's not yet too late to do this. > I am sure eperi would like to work on the Google code as a base! > > > That said, Google's release of something that works for InnoDB, Aria, > binlogs, temp files (and presumably not too hard to add for MyISAM) is > something we should definitely review and target for 10.1 > > Yes! > > Regards, > Monty > > >

3 21

[Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)
by Otto Kekäläinen 01 Sep '14

01 Sep '14

Hello! 2014-02-04 20:41 GMT+02:00 Daniel Bartholomew <dbart(a)mariadb.com>: > On Tue, Feb 4, 2014 at 5:39 AM, Kristian Nielsen > <knielsen(a)knielsen-hq.org> wrote: >>> And while you are at it, please also consider setting up an instance >>> 'kvm-deb-sid-pbuilder-amd64' that provides a Debian unstable >> >> Daniel, can you prepare new sets of VMs for Ubuntu trusty and Debian sid? >> Then I can try to setup Buildbot to do these builds. > > Ok, will do. I should have them up and running before the end of the week. Any updates on this?

3 22

[Maria-developers] [GSoC] Introduction Mail
by Anshu Avinash 04 Aug '14

04 Aug '14

Hi all, I have been accepted for the project "Self-Tuning Optimizer". I will be doing this project under the mentorship of Sergei Golubchik. There is already a task created which will track the progress for this project: https://mariadb.atlassian.net/browse/MDEV-350 . I'll start with getting familiar with the code base, reading some literature on cost estimation in optimizer and interacting with community. About me: I am currently a 4th year student pursuing B.Tech - M.Tech dual degree in Computer Science and Engineering from Indian Institute of Technology (IIT) – Kanpur, India. This would be my first GSoC project. I had done an internship at Facebook last year. I'll also be maintaining a blog at http://igniting.in . I'm really excited to be a part of mariadb and I hope to make a significant contribution to it. Regards Anshu Avinash

5 48

[Maria-developers] Review request MDEV-6274 Collation usage statistics (for feedback plugin)
by Alexander Barkov 30 Jul '14

30 Jul '14

Hello Sergei, Please review a patch for MDEV-6274. Thanks.

2 3

[Maria-developers] MDEV-4511 Assertion `scale <= precision' fails on GROUP BY TIMEDIFF with incorrect types
by Alexander Barkov 25 Jul '14

25 Jul '14

Hello Sergei, Please review a new version of the patch for MDEV-4511, it also fixes more crashing problems found, as well as non-crashing problems reported in MDEV-6302 I removed find_num_type() and fix_num_length_and_dec() which were hard to track, and for some Items these methods were erroneously called twice. fix_num_length_and_dec() now only exists as a non-virtual method in Item_sum and Item_udf_func. The patch is for 5.3, but it is still valid for 10.0, as we still have temporal functions with NOT_FIXED_DEC in 10.0. Thanks.

2 3

[Maria-developers] [GSoC] Optimize mysql-test-runs - Results of new strategy
by Pablo Estrada 17 Jul '14

17 Jul '14

Hello everyone, I took the last couple of days working on a new strategy to calculate the relevance of a test. The results are not sufficient by themselves, but I believe they point to an interesting direction. This strategy uses that rate of co-occurrence of events to estimate the relevance of a test, and the events that it uses are the following: - File editions since last run - Test failure in last run The strategy has also two stages: 1. Training stage 2. Executing stage In the training stage, it goes through the available data, and does the following: - If test A failed: - It counts and stores all the files that were edited since the last test_run (the last test_run depends on BRANCH, PLATFORM, and other factors) - If test A failed also in the previous test run, it also counts that In the executing stage, the training algorithm is still applied, but the decision of whether a test runs is based on its relevance, the relevance is calculated as the sum of the following: - The percentage of times a test has failed in two subsequent test_runs, multiplied by whether the test failed in the previous run (if the test didn't fail in the previous run, this quantity is 0) - For each file that was edited since the last test_run, the percentage of times that the test has failed after this file was edited (The explanation is a bit clumsy, I can clear it up if you wish so) The results have not been too bad, nor too good. With a running set of 200 tests, a training phase of 3000 test runs, and an executing stage of 2000 test runs, I have achieved recall of 0.50. It's not too great, nor too bad. Nonetheless, while running tests, I found something interesting: - I removed the first factor of the relevance. I decided to not care about whether a test failed in the previous test run. I was only using the file-change factor. Naturally, the recall decreased, from 0.50 to 0.39 (the decrease was not too big)... and the distribution of failed tests in the priority queue had a good skew towards the front of the queue (so it seems that the files help somewhat, to indicate the likelihood of a failure). I attached this chart. An interesting problem that I encountered was that about 50% of the test_runs don't have any file changes nor test failures, and so the relevance of all tests is zero. Here is where the original strategy (a weighted average of failures) could be useful, so that even if we don't have any information to guess which tests to run, we just go ahead and run the ones that have failed the most, recently. I will work on mixing up both strategies a bit in the next few days, and see what comes of that. By the way, I pushed the code to github. The code is completely different, so may be better to wait until I have new results soon. Regards! Pablo

2 11

[Maria-developers] Java Connector Coding till 5th June
by Puneet Dewan 17 Jul '14

17 Jul '14

Hi Everyone, The work done till 5th June: 1.I started studying the connector immediately after my semester exams got over(5th May). 2. Understood the relationship between different classes. 3.Read the MySql documentation for text protocol and map the commands from documentation onto code.This thing gave a clear idea about how things are moving from client to server. 4.Setup the code on Ubuntu using eclipse and tested against database. Setup the Team branch in launchpad. 5.Debug the test classes to see how different classes are interlinked and understood that MySQLProtocol.java is the heart of our JDBC driver. 6.Then I started reading Client Server Protocol documentation i.e ServerSide prepared statement and try to see how and where I have to code it. 7.Then I saw that there is a class MySqlServerSidePreparedStatement.java, Initially I coded the prepareStatement() ,close(),and execute() in that class seperately. But after (Code review) discussion with Massimo, we donot need to use that class. Instead use MySqlPreparedStatement.java to do the same work.So made changes accordingly. 8.I had pushed some of the code which I have written to launchpad and discussed with Massimo last week. 9.Last week I was struck with Wireshark on windows and ubuntu both. I was unable to see the packets going to database and seeing the response. 10.On Monday the problem was resolved as I was using the wrong interface.I should have used loopback.io(127.0.0.1) instead on ubuntu(thanks to Georg). Unfortunately the thing does not work on windows.Tried a lot many times. 11.Now I am working on the execute method of client server protocol as I am getting exception while using that method.Wireshark is playing a great role in how the packets are flowing. >From now I will do weekly blogging and keep everyone updated. Thanks & Regards Puneet.

2 9

[Maria-developers] Review request for MDEV-6020, MDEV-5914, MDEV-5941, and MDEV-5262
by Kristian Nielsen 04 Jul '14

04 Jul '14

Hi Serg, We discussed on IRC a review of the $subj bugs related to deadlocks in parallel replication and handling of automatic retry of transactions in case of deadlock. The patches are in my tree: lp:~maria-captains/maria/10.0-knielsen The complete diff can be obtained eg. with: bzr diff -rrevid:psergey@askmonty.org-20140429143317-tkmxrzu6y36hpi7t..revid:knielsen@knielsen-hq.org-20140610081315-o5nx3u32uvqaai7q Let me give some background for these bugs: When two transactions T1 and T2 group-commit on the master, we assume that they have no lock conflicts (otherwise one would have had to wait for the other to commit). So we run them in parallel on the slave. But before we let T2 commit, we wait until T1 has committed first, so that we get the same binlog order on the slave as we had on the master. But if T1 and T2 somehow _do_ get a lock conflict on the slave, and T1 ends up waiting for some lock held by T2, we get a deadlock. This causes the replication to hang (the wait of T2 for T1 to commit is not part of InnoDB deadlock detection), which is quite bad. Elena's testing of the parallel replication in fact uncovered a number of cases where such deadlock issues occur: 1. MDEV-5914. It turns out that InnoDB gap locks can work in a non-symmetric way. If T1 is an INSERT, and T2 is a DELETE, then T1 can run first and not block T2. But if T2 runs first, then it may set a gap lock that blocks T1. Thus, depending on exact execution order, we can get T1 and T2 to group commit on the master, but T2 block T1 on the slave and cause a deadlock. 2. MDEV-5941. If eg. an UPDATE uses different execution plans on the master and slave, it might use index scan on the master, but table scan on the slave. Thus it may take more locks on the slave than on the master, causing it to conflict with another transaction that is run in parallel and cause a deadlock. 3. MDEV-6020. Here, there is a multi-value insert into multiple partitions of a table with auto-increment key. It appears that in this case InnoDB takes an auto-increment table lock on each partition. Since these locks are taken in the order of the inserted values, they can be taken in different order between different INSERT statements, and can cause a deadlock. Since these locks are released during the statement execution (before the commit), such deadlock can occur on the slave even though it did not occur on the master. (Note that this deadlock _is_ detected by InnoDB, and is not related to the commit order on the parallel slave). I know a way to prevent (1) on the slave (and it is implemented in the patches). I do not know if there is a way to prevent (3), it would require understanding better how innodb autoincrement and partitioning works. I do not see any way to prevent (2) from occuring on the slave, in the general case. It also seems to me that there are likely to be more corner cases lurking, and even if I somehow managed to fix the three above problems, there would be some other cases where an extra lock conflict can occur on the slave and cause the slave to hang. So I decided that a general solution is needed to such deadlocks. So the currently proposed patch does two things: A. It adds code to InnoDB, so that whenever T1 starts to wait inside InnoDB for a lock held by T2, this wait is reported to the upper server layer. The replication code will then check if T1 and T2 are part of parallel replication such that T1 must commit before T2. If so, we have a deadlock, and the code will terminate T2 with a deadlock error, allowing T1 to proceed. B. It implements code to handle deadlocks during parallel replication. When a transaction gets a deadlock error (normal, or provoked by the new deadlock detection code from (B)), it is rolled back, and re-executed. The result should be that even if there are other non-discovered cases where locks can conflict in different ways on the master from on the parallel slave, such conflicts will not cause replication to fail. Instead they will be detected, and handled automatically by resolving the deadlock. I think this will be important to ensure the reliability of parallel replication for users. I went through quite a few different ideas and attempts before ending with the present solution. So I wanted to give this background about how the patch relates to the basic problem that needs to be solved. I found a _lot_ of tricky issues during testing, and so I am somewhat unhappy about the complexity of this, even though the final patch may look simple enough on the surface. Maybe you can have some ideas for what could be done simpler, or at least question anything that looks incorrect or unnecessary. Thanks, - Kristian.

2 3

[Maria-developers] Debian packaging - option to enable feedback plugin
by Colin Charles 04 Jul '14

04 Jul '14

Hi! This is probably a great idea: https://answers.launchpad.net/maria/+question/248639 Let's ask dpkg to optionally enable feedback plugin? This could be good for us Just to be clear: I'm referring to this for the packages at mariadb.org I don't know if Debian upstream would like that -- Otto, any thoughts? cheers, -colin -- Colin Charles, Chief Evangelist, SkySQL - The MariaDB Company blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

5 5