November 2010 - developers - lists.mariadb.org

[Maria-developers] MWL#116: Efficient group commit, ready for code review
by Kristian Nielsen 04 Apr '11

04 Apr '11

Hi Serg, Please find attached the patch for code review of MWL#116: Efficient group commit. I went through all your comments in the architecture review and updated the worklog description and the code accordingly. I hope I got everything, otherwise I hope you have the patience to point out any omissions once again! The patch is pushed here if you want to see the complete code: lp:~maria-captains/maria/mariadb-5.1-mwl116 Once the code passes code review, I think the plan is to include this feature in MariaDB 5.3. Please let me know if you need anything else from me for the review. - Kristian.

2 3

[Maria-developers] request for feedback on live schema changes patch
by Zardosht Kasheff 18 Mar '11

18 Mar '11

Hello all, In another thread, there was a discussion about what it would take to allow the storage engine to support live schema changes in mysql and mariadb. I said that I was working porting this existing feature from MySQL Cluster to MySQL 5.1.46. I would like to show the work I have done so far and solicit feedback. Attached is the diff based off of 5.1.46. Before looking at the patch, please be aware of the following things: - I am not an expert in this code. As a result, I had to use an approach of that minimizes the risk of introducing regressions. \ - My main interest is to get something safe that works and can be integrated in 5.1. As a result, kept the current mysql_alter_table functionality in place (as old_mysql_alter_table), and added MySQL Cluster's as new_mysql_alter_table. For an engine to get new_mysql_alter_table to be used, the engine must expose an alter table flag HA_GENERAL_ONLINE. - Another reason why I kept old_mysql_alter_table was that I did not want to deal with engines that implement handler::add_index and make sure that they work properly under the new API. Also, I did not want to port alter table changes related to partitioned tables. - I have run the current 5,1 mysql test suite (that is in the mysql-test directory of source tarballs), and verified that using old_mysql_alter_table and new_mysql_alter_table works fine with them. That being said, I would love to hear feedback on this patch, and would also like to understand what (if anything) can be done to get this into MariaDB 5.1. I realize that may not be possible, but if it is, I would like to try. Thanks -Zardosht

2 5

[Maria-developers] Implementing new "group commit" API in PBXT?
by Kristian Nielsen 13 Dec '10

13 Dec '10

Hi Paul! I want to ask your opinion about implementing in PBXT an extension to the storage engine API that I am working on. There are lots of details in http://askmonty.org/worklog/Server-BackLog/?tid=116 (and even more details in other places), but I thought you would appreciate the short version :-) The idea is to get a well-defined ordering of commits in the server in an efficient way (eg. not break group commit like InnoDB does currently, Bug#13669). For this, two new (optional) storage engine methods are introduced: void (*prepare_ordered)(handlerton *hton, THD *thd, bool all); void (*commit_ordered)(handlerton *hton, THD *thd, bool all); The prepare_ordered() method is called after prepare(), as soon as commit order is decided. The commit_ordered() method is called before commit(), just after the transaction coordinator has made the final decision that the transaction will be durably committed (and not rolled back). The calls into commit_ordered() among different transactions will happen in the order that these transactions are committed, consistently across all engines and the binary log. Same for prepare_ordered(). The idea is that the storage engine should do the minimal amount of work in commit_ordered() necessary to make the commit visible to other threads. And to make sure commits appear to be done in the order of calls to these methods. Do you think either (or both) of these methods could be implemented in PBXT with reasonable effort (and if so, how)? ---- In InnoDB, this was trivial to do, as the InnoDB commit() method already had a "fast" part (which fixed the transaction log order (= "commit order") and made the transaction visible) and a "slow" part (which did the fsync() to make the transaction durable, and handled group commit). (It is necessary that commit_ordered() is fast, as it runs under a global lock. Ideally, it will just allocate an LSN in the transaction log to fix commit order, and perhaps whatever else already needs to happen serialised during engine commit). I hope my explanation was sufficiently clear for you to make a qualified answer. Maybe you can point me to where in the PBXT code a commit becomes visible and the commit order is fixed? In case you were wondering, here are some of the motivations for this feature: 1. For taking a hot backup, it is useful to have consistent commit order between binlog and storage engines. Without it, it can happen that the backed up state of the server has transaction A (but not B) committed in the storage engine, and transaction B (but not A) written to the binlog. Using such backup to provision a new master or slave would leave replication in an inconsistent state. 2. This feature implements working group commit for the binlog while still preserving consistent order as per (1). 3. This will allow to implement START TRANSACTION WITH CONSISTENT SNAPSHOT for multi-engine transactions that is truly consistent (currently is is possible for a transaction to be visible in one engine but not another in such "consistent" snapshot). 4. Galera relies on a consistent commit order, and I believe this feature will allow it to get this in a more engine-independent way. 5. We are planning to use consistent commit order to allow MySQL to recover after a crash transactions that were synced to disk in the binlog but not in the engine. This will allow to reduce the number of fsyncs() during prepare() / commit() from 3 to 1; it only needs to be done in the binlog (with group commit); the engine does not need to fsync(), as any lost transactions will be recovered from the binlog after crash. 6. The prepare_ordered() method is inspired by the Facebook patch to release InnoDB row-level read locks early (before syncing the binlog to disk) to improve performance in the presence of hot spots (probably does not apply to PBXT). - Kristian.

2 9

[Maria-developers] Volunteer my machine for buildbot and database with test results
by Haidong Ji 10 Dec '10

10 Dec '10

To whom it may concern, Per information shown on URL below, with relevant paragraph pasted, I'd love to volunteer to run a build slave and other tests. Let me know how to participate, such as: Will an old laptop work? If so, what do you need? What do I need to do on my side? Does it mean that I need to leave my machine on all the time? What if a reboot happens while it is being used? I am sure more questions as we proceed. Thanks, Haidong "Alex" Ji http://kb.askmonty.org/v/about-buildbot Volunteering to run a build slave Many of our build hosts are run by community members, and we are always looking for additional volunteers to help us cover additional platforms or build options in BuildBot. If you are able to provide a spare machine for this purpose, your help is greatly appreciated! This is a good way to get involved without having to spend a lot of time on it. Get started by writing an email to maria-developers(a)lists.launchpad.net with an offer to run a BuildBot slave. Database with test results Buildbot saves the results of test runs in a database, to be used for enhanced reporting on web pages without need to change the Buildbot code, and for data mining when investigating test failures. The database schema is documented under Buildbot Database Schema. The schema is likely to evolve as we gradually add more kinds of information. For now, the data is not externally available. But the plan is to set up a slave database to replicate the data, and provide access (eg. remote database accounts) to members of the community with interesting ideas about how to present or mine this data, or who are just curious to play with it. If anyone has an interest in this, or wants to volunteer a slave host for this purpose, please send a mail to maria-developers(a)lists.launchpad.net. The more people show interest in this, the faster it is likely to happen!

2 5

[Maria-developers] WL#167 New (by Sergei): sparse bitmaps
by worklog-noreply＠askmonty.org 30 Nov '10

30 Nov '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: sparse bitmaps CREATION DATE..: Tue, 30 Nov 2010, 11:40 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 167 (http://askmonty.org/worklog/?tid=167) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 20 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: DESCRIPTION: implement sparse bitmaps. in mysys, as a generic data structure. preferably with the API compatible to MY_BITMAP. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] New BuildBot tests - compiling the MySQL drivers for PHP and Perl
by Philip Stoev 29 Nov '10

29 Nov '10

Hello, I have installed a new builder "compile-connectors", which will compile the DBD::mysql and the PHP (5.3) mysql connector (the oldest of them all). The purpose is to make sure that MariaDB does not break compatibility with MySQL when it comes to things such as headers, file locations, compilability, linking and loadability of the compiled module. I will try to get some more things tested, such as the newer PHP driver, the C++ connector and drivers containing the embedded server. If you can think of any other connectors or important applications that link or load libmysql, let me know and I will try to add them too. I do not think the new test will catch bugs such as LP#674812, which showed up when mixing and matching OS packages, downloaded packages and self-compiled stuff, but we can at least catch the regression if our headers get modified enough for the third-party drivers to become uncompilable. My next step will be to automate the testing of the Windows installer and the installation of MariaDB as a service. Any thoughts you may have in that area are also appreciated. Thank you. Philip Stoev

2 1

[Maria-developers] Strange pbxt issue in 5.1 (missing rows)
by Michael Widenius 29 Nov '10

29 Nov '10

Hi! I just merged MySQL 5.1.53 into MariaDB and as part of that, did run the pbxt test suite. I encountered this test failure: mysql-test-run --suite=pbxt URRENT_TEST: pbxt.insert --- /home/my/maria-5.1-release/mysql-test/suite/pbxt/r/insert.result 2010-11-25 12:02:26.083226000 +0200 +++ /home/my/maria-5.1-release/mysql-test/suite/pbxt/r/insert.reject 2010-11-25 19:33:48.281738504 +0200 @@ -298,7 +298,7 @@ insert into t2 select t1.* from t1, t2 t, t3 where t1.id1 = t.id2 and t.id2 = t3.id3; select count(*) from t2; count(*) -25500 +25495 drop table t1,t2,t3; create table t1 (a int, b int); insert into t1 (a,b) values (a,b); This only happened once for this test. (I did get similar failures with one other test for that run, but not for new runs). I could not repeat it by running the test or suite again. I also tested to run the test with valgrind, but not problems found. I noticed this also happens in buildbot from time to time: See http://buildbot.askmonty.org/buildbot/builders/gentoo-x86-dbg/builds/396/st…, search after pbxt.insert Paul, have you ever encountered something like this? Any clues where to start looking? (The tree I am working with is 5.1-release) Regards, Monty

2 1

Re: [Maria-developers] MySQL: THDVAR in full-text parser
by Sergei Golubchik 27 Nov '10

27 Nov '10

Hi, 0x00F6! First: please send questions like this to maria-developers(a)lists.launchpad.net it's a public mailing list dedicated to MySQL and MariaDB internals, source code, and related things. I am subscribed, so I'll see you mail there, and you may be sure I will, because it won't be accidentally catched by my spam filter, or sorted out in some obscure folder. Furthermore other subscribers will see your question and could reply if I will be not available (e.g. I could be travelling). Thank you. On Nov 27, 0x00F6 - wrote: > Hello Sergei, > > Would you please help me in one small problem? > I'm writing MySQL full-text parser plugin and I need to configure it > by several session-level variables. So I need to execute THDVAR macro > in my parse function. So how to obtain "thd" variable for this macro? The server uses 'current_thd' in such a case. It should probably be in the plugin.h, I'd say. But at the moment it is not. You can copy the declaration from mysql_priv.h, something like this: extern pthread_key_t THR_THD; THD *current_thd() { return (THD*)pthread_getspecific(THR_THD); } if you want it to work on windows, see mysql_priv.h for the complete definition of current_thd. Regards, Sergei

1 0

Re: [Maria-developers] [Maria-discuss] Problem with MariaDb as a Windows service
by Peter Laursen 25 Nov '10

25 Nov '10

Same problem with MariaDB 5.1.51. --Peter On Sun, Nov 21, 2010 at 02:43, Vladislav Vaintroub < vladislav.vaintroub(a)oracle.com> wrote: > Hi, > It is likely this bug http://bugs.mysql.com/bug.php?id=56821 , was for > some > time in MySQL, already fixed. > > -----Original Message----- > From: maria-developers-bounces+wlad=sun.com(a)lists.launchpad.net > [mailto:maria-developers-bounces+wlad <maria-developers-bounces%2Bwlad>= > sun.com(a)lists.launchpad.net] On Behalf > Of Philip Stoev > Sent: Samstag, 20. November 2010 20:04 > To: Peter Laursen > Cc: maria-discuss(a)lists.launchpad.net; > maria-developers(a)lists.launchpad.net > Subject: Re: [Maria-developers] [Maria-discuss] Problem with MariaDb as a > Windows service > > Hi, > > It seems that the service requires the --console option to operate > properly. > > Here is how I could make it work: > > 1. In the Services Control panel, open up the properties for the service 2. > In the "Start Parameters" box, type "--console" without the quotation marks > 3. In the same dialog box, hit Start > > Philip Stoev > > ----- Original Message ----- > From: "Peter Laursen" <peter_laursen(a)webyog.com> > To: <maria-discuss(a)lists.launchpad.net>; > <maria-developers(a)lists.launchpad.net> > Sent: Saturday, November 20, 2010 3:36 PM > Subject: [Maria-discuss] Problem with MariaDb as a Windows service > > > > Some person claimed here around one week ago that MariaDB would not > > run as service after "mysqld --install". > > I replied that it worked for me - but it does not with the latest > > stable version 5.2.3 (it did with the 5.2.3 RC) (so my apology to this > > person!) > > > > Windows console output: > > > > C:\maria52\bin>sc delete maria52 > > [SC] DeleteService LYKKEDES > > > > C:\maria52\bin>mysqld --install maria52 Service successfully > > installed. > > > > C:\maria52\bin>net start maria52 > > Tjenesten maria52 starter... > > Tjenesten maria52 kunne ikke startes. > > > > Der opstod en systemfejl. > > > > Systemfejlen 1067 opstod. > > > > (From Danish --> service could not start --> the system error 1067 > > occurred) > > > > I have tried with both old datadir and the empty one shipped with > > mariaDB and also my old my.ini as well as a fresh one. > > Also I tried to add --defaults-dir to the registry (just in case I had > > overlooked a my.ini copy in C:\ or C:\Windows) and adding doublequotes > > key manually like "C:\maria52\bin\mysqld" > > --defaults-file="C:\maria52\my.ini" maria52 > > > > I double checked there are no conflicts wit all settings. > > > > The command > > "C:\maria52\bin\mysqld" --defaults-file="C:\maria52\my.ini" > > .. starts the server (also when --defaults-file is omitted) .. but not > > as a service of course. > > > > > > Was it tested with the 5.2.3 GA release that it will run as a Windows > > service? > > The RC worked as a charm - but the GA will not. Any idea? > > (I am using Win7, 64 bit, Home Premium) > > > > > > Peter > > (Webyog) > > > > > > ---------------------------------------------------------------------------- > ---- > > > > _______________________________________________ > > Mailing list: https://launchpad.net/~maria-discuss > > Post to : maria-discuss(a)lists.launchpad.net > > Unsubscribe : https://launchpad.net/~maria-discuss > > More help : https://help.launchpad.net/ListHelp > > > > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-developers > Post to : maria-developers(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-developers > More help : https://help.launchpad.net/ListHelp > >

3 2

Re: [Maria-developers] [Maria-discuss] Problem with MariaDb as a Windows service
by Alex Budovski 25 Nov '10

25 Nov '10

> Right, this was subsequently fixed in MySQL in > http://lists.mysql.com/commits/118824 > Yes, I noticed that too late.

1 0