- developers - lists.mariadb.org

Re: [Maria-developers] [Maria-discuss] Files under /etc/my.cnf.d/ read by default
by Honza Horak 05 Feb '13

05 Feb '13

This should have been posted better to maria-developers, probably, so re-sending.. On 02/05/2013 01:55 PM, Honza Horak wrote: > Hi guys, > > while we were packaging MariaDB into Fedora (finally) we used the > upstream's cnf files structure, which means /etc/my.cnf, that includes > all files from directory /etc/my.cnf.d/* using statement !includedir. > > That works fine when user installs (not updates) the MariaDB packages > without having them installed before. The problem is if user has changed > /etc/my.cnf before updating -- then /etc/my.cnf doesn't get updated > (!includedir won't be added) and we end with files under /etc/my.cnf.d/* > but nothing what includes them. That is quite confusing for users that > may change something in /etc/my.cnf.d/client.cnf but nothing happens. > > Since files under /etc/my.cnf.d/* are empty by default it shouldn't do > any harm if all files corresponding with /etc/my.cnf.d/*.cnf template > get loaded even without !includedir in /etc/my.cnf. Actually I believe > that is what users expect to happen when they see files under > /etc/my.cnf.d.. > > So, that's basically what I'm proposing -- read files under > /etc/my.cnf.d/* by default, without a need to use !includedir. What do > you think? > > Regards, > Honza > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp

1 0

[Maria-developers] InnoDB Compression
by Gordan Bobic 04 Feb '13

04 Feb '13

Hi, I've just tried testing InnoDB page compression, and I'm sure I must be doing something wrong because the disk space usage in my database directory is the same before and after the compression. To compress the tables I am using: ALTER TABLE $table ENGINE=InnoDB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=$size; with $size I tried being 8KB and 4KB, and in each case "du -h" on the DB directory is the same as before the compression. innodb_file_per_table is set. SHOW CREATE TABLE afterwards shows the ROW_FORMAT and KEY_BLOCK_SIZE to be as per the ALTER TABLE statement. 1) Am I doing it wrong? 2) Is the InnoDB compression feature available in MariaDB? 3) Is the compression on by default transparently? 4) Is there any other explanation for lack of effect? As a cross-check, I tried taking my biggest .ibd file and compressing it with "lzop -1" and that compressed it from 80MB down to 22MB, which doesn't sound right if the data in it was already compressed in any way. Has anybody got any possible explanations? TIA Gordan

2 2

Re: [Maria-developers] [Commits] Rev 3657: MDEV-4127 : Export additional symbols when building RPM
by Sergei Golubchik 04 Feb '13

04 Feb '13

Hi, Vladislav! May be you'd rather rename symbols with #define, just as redhat does? On Feb 03, Vladislav Vaintroub wrote: > At file:///H:/bzr/5.5/ > > ------------------------------------------------------------ > revno: 3657 > revision-id: wlad(a)montyprogram.com-20130203164516-ex2ftmyca5b507g3 > parent: elenst(a)ubuntu11.home-20130202225357-za9bglcf2ngv36xj > committer: Vladislav Vaintroub <wlad(a)montyprogram.com> > branch nick: 5.5 > timestamp: Sun 2013-02-03 17:45:16 +0100 > message: > MDEV-4127 : Export additional symbols when building RPM, > to enable both recompiling mysqli or odbc from sources in addition > to drop-in replacement functionality. > > The case in question is compiling mysqli from sources, that needs > client_errors via ER() macro. > > Previously, we exported it as mysql_client_errors (compatibly to > Fedora's style symbol renaming, see MDEV-3842). > However, if MariaDB header files are used when compiling mysqli, > client_errors needs to be exported with its original name. Regards, Sergei

2 2

[Maria-developers] Coverity scans of MariaDB source
by Christian Convey 04 Feb '13

04 Feb '13

Coverity has a deal called Coverity Scan, where they offer free static analyses of open-source programs (http://scan.coverity.com/) I've used Coverity Prevent at work, and I was pretty happy with the analyses it performed. Is it an intentional choice that MariaDB isn't making use of Coverity Scan? I'm interested in starting to contribute to MariaDB, and I'm wondering if it would be useful to the project for me to pursue this as a first task. - Christian

2 2

[Maria-developers] CMake vs. autotools on Linux?
by Christian Convey 04 Feb '13

04 Feb '13

Hi guys, I noticed that MariaDB is built (indirectly) with CMake on Windows, and with autotools on Linux. Why are both systems used? I would have assumed that because CMake is pretty good at generating Makefiles on Linux and OS X, there would be little reason to continue using autotools. Thanks, Christian

2 3

Re: [Maria-developers] Summary of the MDEV-3882 (.deb upgrade from mysql to mariadb fails due to apt-get considering it a downgrade)
by Kristian Nielsen 02 Feb '13

02 Feb '13

[Moving the discussion to maria-developers@, please use the public mailing lists for discussions like this] Alexey Botchkov <holyfoot(a)askmonty.org> writes: > No matter what dependencies or new packages i created inside MariaDB, > Debian's first idea was to try to install the 'later' version of > libmysqlclient18. At the moment > that was the Oracle's one. Yes, we could change our version's number to be > 'later' than Oracle's, but that would turn same problems on MySQL, and Not really. It's not "Oracle's", it is the native Debian/Ubuntu package. So if users want MariaDB, they can add our repository. If they want the official Debian/Ubuntu package, they can just not add our repository. So it makes sense that when users explicitly add the MariaDB repository, they get the MariaDB version of all the packages automatically. > would help us only temporarely - to the moment Oracle releases even > later' version. We can use epoch to ensure that our version will always be bigger than the official Debian/Ubuntu version of MySQL: http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version Something like 1:5.5.29-mariadb1~wheezy. This will always be prefered by apt-get over any official Debian package. > I didn't find anything that could affect apt-get and guys on #Debian > didn't recommend anything. I believe it works to specify desired version explicitly, as mentioned in the bug report as workaround: sudo apt-get install mariadb-server libmysqlclient18=5.5.29-mariadb1~wheezy > So for now i'd do about this: > - change nothing in the existing mariadb-produced debian packages. > - recommend to use 'aptitude' installing the mariadb-server on Debian. > So how do you like my proposal? Any other ideas? I think it is better if we use the epoch to make MariaDB upgrades work automatically with both apt-get and aptitude, like they used to. Did you check how Percona handles the similar problem? - Kristian.

2 1

Re: [Maria-developers] Fwd: [Commits] Rev 3622: Analysis: in file:///home/tsk/mprog/src/5.3-md641/
by Igor Babaev 31 Jan '13

31 Jan '13

On 01/31/2013 04:41 AM, Timour Katchaounov wrote: > Igor, > > Let me know if I can push the patch below. It fixes the bug > according to our last discussion. If the number of group_list > elements was known, I could add a test already in the beginning > of the function, however such a counter doesn't exist, and I > didn't want to complicate the code just for this special case. > That's why the test is done where it is now. > > Timour > > > ------------------------------------------------------------ > revno: 3622 > revision-id: timour(a)askmonty.org-20130131120040-oan9zei8ojyco178 > parent: sergii(a)pisem.net-20130128081223-mp9rsd3t9soz8lly > fixes bug(s): https://mariadb.atlassian.net/browse/MDEV-641 > committer: timour(a)askmonty.org > branch nick: 5.3-md641 > timestamp: Thu 2013-01-31 14:00:40 +0200 > message: > Analysis: > Range analysis discoveres that the query can be executed via loose > index scan for GROUP BY. > Later, GROUP BY analysis fails to confirm that the GROUP operation can > be computed via an > index because there is no logic to handle duplicate field references > in the GROUP clause. > As a result the optimizer the optimizer produces an inconsistent plan. > It constructs a > temporary table, but on the other hand the group fields are not set to > point there. > > Solution: > Make loose scan analysis work in sync with order by analysis. In the > case of duplicate > columns loose scan will not be applicable. This limitation will be > lifted in 10.0 by > removing duplicate columns. > >

1 0

Re: [Maria-developers] what pros/cons of storing binary log in an InnoDB table?
by Kristian Nielsen 31 Jan '13

31 Jan '13

Zardosht Kasheff <zardosht(a)gmail.com> writes: > Instead of having the binary log be stored as a log file, a table with > a transactional engine (like InnoDB) is used. I have NOT done any Yes, this is a tempting idea, is it not? Drizzle even implemented this. But there are some subtle problems, and I think there are better solutions. One problem you will find if you try implementing this is - how are you going to keep the concept of binlog order? A relational table is unordered. You can use an increasing uint64 as the primary key, but then how are you going to ensure that the order in the table is the same as the commit order in InnoDB (InnoDB redo log)? If you take a lock from allocating the sequence number to doing the commit, then you have introduced a serialisation bottleneck and killed group commit. If you accept different order (as Drizzle does), then when you take an XtraBackup, it may not be consistent with the binlog. You will be writing your data *6* times: To the real table, when flushing its buffer pool pages. To the double-write buffer while flushing, and to the redo log before flushing. Then to the binlog table, and before that to the redo log and the doublewrite buffer. I could never reconcile myself with this. So it is tempting, but re-implementing binlog is a huge change (also in terms of end-user visibility), and my personal opinion is that for such huge a change we need to go to the best solution, not second-best. > investigation into whether this is possible, but I'm wondering what > people abstractly think of this idea. > My major motivation is to find a way to have users not be forced to > fsync on every transaction commit on slaves. With crash safety of My preference is MWL#164: http://askmonty.org/worklog/Server-RawIdeaBin/?tid=164 This reduces to one fsync() per group commit. But as you point out, this is still horrendously expensive for a single-threaded slave :-( Abstractly, the proper solution is to store the binlog and the InnoDB redo log (and all other transactional-type logs) in the *same* log. The upper layer would provide a general transactional log facility, and binlog, InnoDB, partitioning, etc. would use this instead of their own logs. But it is hard to see how we could arrive there from current code ... For MariaDB global transaction ID, I support crash-safe without binlog enabled, so there turning off fsync() will be possible. But this does not help if a slave also needs to act as a master. Parallel replication can help a lot, by allowing group commit, amortising the cost of the fsync(). But it would still be nice to solve the orginal problem. As you say, for a slave it is not a problem to loose transactions in a crash as they can be just replicated again. The problem is to be consistent between what is in the InnoDB tables and what is in the binlog. InnoDB already records in a crash-safe way the binlog position of the last commit. So if we can ensure that InnoDB is never ahead of the binlog, we can just cut the binlog after a crash if it has more events than what InnoDB recovered. We could do this by doing binlog writes as async I/O O_DIRECT writes, and sending a signal somehow to InnoDB so it would buffer its own redo log until binlog is safe on disk. Then no fsync() would be needed. But this is so far just a loose idea, it needs more thought to be sure it can work... Anyway, this is my current long-term plan: if we cannot have a proper single database-wide transaction log, then instead coordinate writes between the multiple logs that exist. Delay writes to logs so that they are never ahead of the master log. And at crash, either recover missing pieces of logs from the master log (MWL#164), or cut logs to be consistent. Then we can be consistent without any fsync() delays. > - As I understand it, for two phase commit, storage engines are > currently expected to fsync once after a transaction is prepared and > once after a transaction is committed. If the binary log is an InnoDB > table, then two-phase commit is unnecessary and the number of fsyncs > on the system is cut in half. In MariaDB 10 and MySQL 5.6, the fsync() at commit is no longer needed (InnoDB does not have it). [Note that IIRC, MySQL 5.6 breaks the storage engine API, you will need to implement the skip-fsync-at-commit-stage, or you will loose group commit.] > - With crash safe slaves on MySQL 5.6, one theoretically does not > need to enable fsyncs on transaction commit on slaves. If some data > gets lost after a crash, then the data may be replayed from the > master. However, to get GTIDs working, one must enable the binary log > on slaves, which requires two-phase commit, which require fsyncs. By Yes, requiring binlog on slaves is a serious mistake :-( Ok, as you can perhaps tell, this is something I have thought a lot about over the past years, so I have a lot of ideas but it is a bit difficult to put it all in a single mail reply without it becomming mostly a jumble of loose ideas. You are welcome to ask for more details on any specific points of interest. - Kristian.

5 9

Re: [Maria-developers] reducing fsyncs during handlerton->prepare and handlerton->commit in 5.6
by Kristian Nielsen 31 Jan '13

31 Jan '13

Zardosht Kasheff <zardosht(a)gmail.com> writes: > - in MySQL 5.6 and MariaDB 10.0, what are the fsyncing requirements > for storage engines during prepare and commit? > - are there new APIs that the storage engine must comply with in > order to get these benefits? In MariaDB 10.0, there is a new commit_checkpoint_request handlerton method. If the storage engine implements that, it need not do the fsync() at commit. It must also implement the commit_ordered method from 5.3. Check comments in sql/handler.h for these two methods, and let me know if there are any questions. The intention is that storage engines need not do much extra work for this functionality, eg. the patch for InnoDB support was a couple hundred lines at most IIRC. > - in MySQL 5.6, I see new handlerton methods commit_low and > prepare_low. What do these APIs do? What is their contract? I think MySQL developers should answer for this. But it is related to this new check in InnoDB: thd_requested_durability(trx->mysql_thd) == HA_IGNORE_DURABILITY And the existing flush_logs handlerton method is also used. My guess is that commit_low is just internal server stuff that the storage engine does not need to consider. The MySQL 5.6 mechanism is mostly the same as the MariaDB 10.0 one in what it actually does, but they did not bother to try to be compatible with the storage engine API in MariaDB 5.3. So looks like it is #ifdef time for you if you want to support both. - Kristian.

1 0

[Maria-developers] how to build mariadb RPMs
by Rich Prohaska 30 Jan '13

30 Jan '13

How do you build the mariadb RPMs? Are you using rpmbuild or cpack? Is there a top level script that uses rpmbuild or cpack that I should use? Thanks Rich Prohaska

3 11