- developers - lists.mariadb.org

Re: [Maria-developers] mariadb-server not installable on Debian Testing AMD64
by Kristian Nielsen 06 Feb '13

06 Feb '13

Thomas Hackert <thackert(a)nexgo.de> writes: > I am not sure, how (and if) I can upload to f.a.o., so I will attach it > to this mail. Sorry for the inconvenience ... :( No inconvenience at all, mail is perfect. And indeed, the strace was illuminating. I think the issue is right here: [pid 32673] bind(14, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRNOTAVAIL (Cannot assign requested address) The server fails to start, because it cannot setup a socket to listen on the loopback interface on 127.0.0.1 The manpage for bind(2) says: EADDRNOTAVAIL A nonexistent interface was requested or the requested address was not local. I get the same error on my machine if I try to bind a socket to a non-existing interface. You can try this netcat command to do the same bind, it will probably fail in the same way: nc -l -s 127.0.0.1 -p 3306 So the problem most likely is that your system has a non-standard networking setup, with the normal loopback address 127.0.0.1 not configured. The default configuration file has this line: bind-address = 127.0.0.1 This will fail if the address 127.0.0.1 is not available on the system (normally it is). So I think you can edit /etc/mysql/my.cnf and change this line to use a valid IP address, or replace it with skip-networking to not use any IP address. Then try starting the server and see if it works. If it does, `apt-get install -f` might fix the installation. Can you confirm that your system has no 127.0.0.1 address? (It is somewhat annoying that the install fails in this case, but it does seem a rather rare setup). The second question is why you did not get any error log output with messages about this problem. I suppose this might be because you had no syslog installed. Elena gave you suggestions for how to resolve/investigate this further. But let me know if you have more problems with this. Thanks for your persistence in helping me get to the bottom of this! - Kristian.

2 6

Re: [Maria-developers] Fwd: [Commits] Rev 3622: Fix for MDEV-4140 in file:///home/tsk/mprog/src/5.3-md4140/
by Igor Babaev 06 Feb '13

06 Feb '13

Timour, It's ok to push this patch as a fix. Yet remember that this fix leads us to a performance regression 5.2 >> 5.3 when the range expression contains a constant [single-row] subquery. This is due to the changes when and how we build range trees in 5.3. When we have a range expression with constant expensive subquery we just bail out and do not include this range into the range condition at all. The worst thing about this is that we bail out even for dynamic ranges belonging to te range trees that are build at the execution time when subquery evaluation is allowed. Please open a task to fix all these regressions (it's quite possible). Regards, Igor. On 02/06/2013 03:27 AM, Timour Katchaounov wrote: > Igor, > > please review the following simple patch. > > Timour > > ------------------------------------------------------------ > revno: 3622 > revision-id: timour(a)askmonty.org-20130206112530-nwo4xto0cwv65ece > parent: sergii(a)pisem.net-20130128081223-mp9rsd3t9soz8lly > fixes bug(s): https://mariadb.atlassian.net/browse/MDEV-4140 > committer: timour(a)askmonty.org > branch nick: 5.3-md4140 > timestamp: Wed 2013-02-06 13:25:30 +0200 > message: > Fix for MDEV-4140 > > Analysis: > Range analysis detects that the subquery is expensive and doesn't > build a range access method. Later, the applicability test for loose > scan doesn't take that into account, and builds a loose scan method > without a range scan on the min/max column. As a result loose scan > fetches the first key in each group, rather than the first key that > satisfies the condition on the min/max column. > > Solution: > Since there is no SEL_ARG tree to be used for the min/max column, > it is not possible to use loose scan if the min/max column is compared > with an expensive scalar subquery. Make the test for loose scan > applicability to be in sync with the range analysis code by testing if > the min/max argument is compared with an expensive predicate. > >

1 0

[Maria-developers] MariaDB 10.0.1-alpha release prep
by Daniel Bartholomew 06 Feb '13

06 Feb '13

I've begun prepping MariaDB 10.0.1-alpha for release. Preliminary changelog and release notes are here: - https://kb.askmonty.org/en/mariadb-1001-release-notes/ - https://kb.askmonty.org/en/mariadb-1001-changelog/ I saw in the changelog a couple of features on https://kb.askmonty.org/en/what-is-mariadb-100/ that were added in this release (MDEV-4011 & Cassandra Storage Engine), if there are other new features, or features backported from MySQL 5.6, or anything else I should be sure to mention in the release notes, please let me know ASAP. Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

3 4

[Maria-developers] mariadb-server not installable on Debian Testing AMD64
by Thomas Hackert 05 Feb '13

05 Feb '13

SEND-PR: -*- send-pr -*- SEND-PR: Lines starting with `SEND-PR' will be removed automatically, as SEND-PR: will all comments (text enclosed in `<' and `>'). SEND-PR: From: thackert(a)nexgo.de To: maria-developers(a)lists.launchpad.net Subject: mariadb-server not installable on Debian Testing AMD64 >Description: Hello @ll, this is my first try to report a bug with mysqlbug. If I missed something and/or have something wrong, feel free to ask, but do not condemn me ... ;) I have tried to install mariadb-server on Debian Testing AMD64. I got the error message <quote> <snip> [ ok ] Stopping MariaDB database server: mysqld. 130130 17:28:50 [Note] Plugin 'InnoDB' is disabled. 130130 17:28:50 [Note] Plugin 'FEEDBACK' is disabled. [....] Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . FAIL . . . . . . . failed! invoke-rc.d: initscript mysql, action "start" failed. dpkg: Fehler beim Bearbeiten von mariadb-server-10.0 (--configure): Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück dpkg: Abhängigkeitsprobleme verhindern Konfiguration von mariadb-server: mariadb-server hängt ab von mariadb-server-10.0; aber: Paket mariadb-server-10.0 ist noch nicht konfiguriert. dpkg: Fehler beim Bearbeiten von mariadb-server (--configure): Abhängigkeitsprobleme - verbleibt unkonfiguriert Trigger für menu werden verarbeitet ... Fehler traten auf beim Bearbeiten von: mariadb-server-10.0 mariadb-server </quote> ... :( I get the same message, if I try to install mysql-server from Debian's repository ... :( >How-To-Repeat: 1. Add " # MariaDB 10.0 repository list - created 2013-01-30 16:20 UTC # http://mariadb.org/mariadb/repositories/deb # http://mirror.netcologne.de/mariadb//repo/10.0/debian wheezy main deb-src http://mirror.netcologne.de/mariadb//repo/10.0/debian wheezy main" to your /etc/apt/sources.list. 2. Run "apt-get update" 3. Run "apt-get install mariadb-server" The error message appears nearly at the end of the installation ... :( >Fix: I do not know, sorry ... :( But maybe it is possible to check first, if mysql- or mariadb-server was installed before, and if it was, stop it. If it was not installed before, than it should not try to stop the server ... ;) >Submitter-Id: <submitter ID> >Originator: Thomas Hackert >Organization: <none> >MySQL support: [none | licence | email support | extended email support >] >Synopsis: mariadb-server not installable on Debian Testing AMD64 >Severity: serious >Priority: medium >Category: mysql >Class: <[ sw-bug | doc-bug | change-request | support ] (one >line)> >Release: mysql-10.0.0-mariadb1~wheezy (mariadb.org binary >distribution) >C compiler: gcc-4.7.real (Debian 4.7.1-2) 4.7.1 >C++ compiler: g++-4.7.real (Debian 4.7.1-2) 4.7.1 >Environment: <machine, os, target, libraries (multiple lines)> System: Linux neo 3.2.0-4-amd64 #1 SMP Debian 3.2.35-2 x86_64 GNU/Linux Some paths: /usr/bin/perl /usr/bin/make /usr/bin/gcc /usr/bin/cc GCC: Using built-in specs. COLLECT_GCC=/usr/bin/gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.7/lto-wrapper Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Debian 4.7.2-5' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --with-arch-32=i586 --with-tune=generic --enanable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --with-arch-32=i586 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.7.2 (Debian 4.7.2-5) Compilation info (call): CC='/usr/bin/gcc' CFLAGS='-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' CXX='/usr/bin/g++' CXXFLAGS='-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized -fno-exceptions -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' LDFLAGS='' ASFLAGS='' Compilation info (used): CC='/usr/bin/gcc' CFLAGS='-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' CXX='/usr/bin/g++' CXXFLAGS='-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized -fno-exceptions -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' LDFLAGS='' ASFLAGS='' Perl: This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-gnu-thread-multi -- The way to make a small fortune in the commodities market is to start with a large fortune.

3 7

Re: [Maria-developers] [Maria-discuss] Files under /etc/my.cnf.d/ read by default
by Honza Horak 05 Feb '13

05 Feb '13

This should have been posted better to maria-developers, probably, so re-sending.. On 02/05/2013 01:55 PM, Honza Horak wrote: > Hi guys, > > while we were packaging MariaDB into Fedora (finally) we used the > upstream's cnf files structure, which means /etc/my.cnf, that includes > all files from directory /etc/my.cnf.d/* using statement !includedir. > > That works fine when user installs (not updates) the MariaDB packages > without having them installed before. The problem is if user has changed > /etc/my.cnf before updating -- then /etc/my.cnf doesn't get updated > (!includedir won't be added) and we end with files under /etc/my.cnf.d/* > but nothing what includes them. That is quite confusing for users that > may change something in /etc/my.cnf.d/client.cnf but nothing happens. > > Since files under /etc/my.cnf.d/* are empty by default it shouldn't do > any harm if all files corresponding with /etc/my.cnf.d/*.cnf template > get loaded even without !includedir in /etc/my.cnf. Actually I believe > that is what users expect to happen when they see files under > /etc/my.cnf.d.. > > So, that's basically what I'm proposing -- read files under > /etc/my.cnf.d/* by default, without a need to use !includedir. What do > you think? > > Regards, > Honza > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp

1 0

[Maria-developers] InnoDB Compression
by Gordan Bobic 04 Feb '13

04 Feb '13

Hi, I've just tried testing InnoDB page compression, and I'm sure I must be doing something wrong because the disk space usage in my database directory is the same before and after the compression. To compress the tables I am using: ALTER TABLE $table ENGINE=InnoDB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=$size; with $size I tried being 8KB and 4KB, and in each case "du -h" on the DB directory is the same as before the compression. innodb_file_per_table is set. SHOW CREATE TABLE afterwards shows the ROW_FORMAT and KEY_BLOCK_SIZE to be as per the ALTER TABLE statement. 1) Am I doing it wrong? 2) Is the InnoDB compression feature available in MariaDB? 3) Is the compression on by default transparently? 4) Is there any other explanation for lack of effect? As a cross-check, I tried taking my biggest .ibd file and compressing it with "lzop -1" and that compressed it from 80MB down to 22MB, which doesn't sound right if the data in it was already compressed in any way. Has anybody got any possible explanations? TIA Gordan

2 2

Re: [Maria-developers] [Commits] Rev 3657: MDEV-4127 : Export additional symbols when building RPM
by Sergei Golubchik 04 Feb '13

04 Feb '13

Hi, Vladislav! May be you'd rather rename symbols with #define, just as redhat does? On Feb 03, Vladislav Vaintroub wrote: > At file:///H:/bzr/5.5/ > > ------------------------------------------------------------ > revno: 3657 > revision-id: wlad(a)montyprogram.com-20130203164516-ex2ftmyca5b507g3 > parent: elenst(a)ubuntu11.home-20130202225357-za9bglcf2ngv36xj > committer: Vladislav Vaintroub <wlad(a)montyprogram.com> > branch nick: 5.5 > timestamp: Sun 2013-02-03 17:45:16 +0100 > message: > MDEV-4127 : Export additional symbols when building RPM, > to enable both recompiling mysqli or odbc from sources in addition > to drop-in replacement functionality. > > The case in question is compiling mysqli from sources, that needs > client_errors via ER() macro. > > Previously, we exported it as mysql_client_errors (compatibly to > Fedora's style symbol renaming, see MDEV-3842). > However, if MariaDB header files are used when compiling mysqli, > client_errors needs to be exported with its original name. Regards, Sergei

2 2

[Maria-developers] Coverity scans of MariaDB source
by Christian Convey 04 Feb '13

04 Feb '13

Coverity has a deal called Coverity Scan, where they offer free static analyses of open-source programs (http://scan.coverity.com/) I've used Coverity Prevent at work, and I was pretty happy with the analyses it performed. Is it an intentional choice that MariaDB isn't making use of Coverity Scan? I'm interested in starting to contribute to MariaDB, and I'm wondering if it would be useful to the project for me to pursue this as a first task. - Christian

2 2

[Maria-developers] CMake vs. autotools on Linux?
by Christian Convey 04 Feb '13

04 Feb '13

Hi guys, I noticed that MariaDB is built (indirectly) with CMake on Windows, and with autotools on Linux. Why are both systems used? I would have assumed that because CMake is pretty good at generating Makefiles on Linux and OS X, there would be little reason to continue using autotools. Thanks, Christian

2 3

Re: [Maria-developers] Summary of the MDEV-3882 (.deb upgrade from mysql to mariadb fails due to apt-get considering it a downgrade)
by Kristian Nielsen 02 Feb '13

02 Feb '13

[Moving the discussion to maria-developers@, please use the public mailing lists for discussions like this] Alexey Botchkov <holyfoot(a)askmonty.org> writes: > No matter what dependencies or new packages i created inside MariaDB, > Debian's first idea was to try to install the 'later' version of > libmysqlclient18. At the moment > that was the Oracle's one. Yes, we could change our version's number to be > 'later' than Oracle's, but that would turn same problems on MySQL, and Not really. It's not "Oracle's", it is the native Debian/Ubuntu package. So if users want MariaDB, they can add our repository. If they want the official Debian/Ubuntu package, they can just not add our repository. So it makes sense that when users explicitly add the MariaDB repository, they get the MariaDB version of all the packages automatically. > would help us only temporarely - to the moment Oracle releases even > later' version. We can use epoch to ensure that our version will always be bigger than the official Debian/Ubuntu version of MySQL: http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version Something like 1:5.5.29-mariadb1~wheezy. This will always be prefered by apt-get over any official Debian package. > I didn't find anything that could affect apt-get and guys on #Debian > didn't recommend anything. I believe it works to specify desired version explicitly, as mentioned in the bug report as workaround: sudo apt-get install mariadb-server libmysqlclient18=5.5.29-mariadb1~wheezy > So for now i'd do about this: > - change nothing in the existing mariadb-produced debian packages. > - recommend to use 'aptitude' installing the mariadb-server on Debian. > So how do you like my proposal? Any other ideas? I think it is better if we use the epoch to make MariaDB upgrades work automatically with both apt-get and aptitude, like they used to. Did you check how Percona handles the similar problem? - Kristian.

2 1

Re: [Maria-developers] Fwd: [Commits] Rev 3622: Analysis: in file:///home/tsk/mprog/src/5.3-md641/
by Igor Babaev 31 Jan '13

31 Jan '13

On 01/31/2013 04:41 AM, Timour Katchaounov wrote: > Igor, > > Let me know if I can push the patch below. It fixes the bug > according to our last discussion. If the number of group_list > elements was known, I could add a test already in the beginning > of the function, however such a counter doesn't exist, and I > didn't want to complicate the code just for this special case. > That's why the test is done where it is now. > > Timour > > > ------------------------------------------------------------ > revno: 3622 > revision-id: timour(a)askmonty.org-20130131120040-oan9zei8ojyco178 > parent: sergii(a)pisem.net-20130128081223-mp9rsd3t9soz8lly > fixes bug(s): https://mariadb.atlassian.net/browse/MDEV-641 > committer: timour(a)askmonty.org > branch nick: 5.3-md641 > timestamp: Thu 2013-01-31 14:00:40 +0200 > message: > Analysis: > Range analysis discoveres that the query can be executed via loose > index scan for GROUP BY. > Later, GROUP BY analysis fails to confirm that the GROUP operation can > be computed via an > index because there is no logic to handle duplicate field references > in the GROUP clause. > As a result the optimizer the optimizer produces an inconsistent plan. > It constructs a > temporary table, but on the other hand the group fields are not set to > point there. > > Solution: > Make loose scan analysis work in sync with order by analysis. In the > case of duplicate > columns loose scan will not be applicable. This limitation will be > lifted in 10.0 by > removing duplicate columns. > >

1 0

Re: [Maria-developers] what pros/cons of storing binary log in an InnoDB table?
by Kristian Nielsen 31 Jan '13

31 Jan '13

Zardosht Kasheff <zardosht(a)gmail.com> writes: > Instead of having the binary log be stored as a log file, a table with > a transactional engine (like InnoDB) is used. I have NOT done any Yes, this is a tempting idea, is it not? Drizzle even implemented this. But there are some subtle problems, and I think there are better solutions. One problem you will find if you try implementing this is - how are you going to keep the concept of binlog order? A relational table is unordered. You can use an increasing uint64 as the primary key, but then how are you going to ensure that the order in the table is the same as the commit order in InnoDB (InnoDB redo log)? If you take a lock from allocating the sequence number to doing the commit, then you have introduced a serialisation bottleneck and killed group commit. If you accept different order (as Drizzle does), then when you take an XtraBackup, it may not be consistent with the binlog. You will be writing your data *6* times: To the real table, when flushing its buffer pool pages. To the double-write buffer while flushing, and to the redo log before flushing. Then to the binlog table, and before that to the redo log and the doublewrite buffer. I could never reconcile myself with this. So it is tempting, but re-implementing binlog is a huge change (also in terms of end-user visibility), and my personal opinion is that for such huge a change we need to go to the best solution, not second-best. > investigation into whether this is possible, but I'm wondering what > people abstractly think of this idea. > My major motivation is to find a way to have users not be forced to > fsync on every transaction commit on slaves. With crash safety of My preference is MWL#164: http://askmonty.org/worklog/Server-RawIdeaBin/?tid=164 This reduces to one fsync() per group commit. But as you point out, this is still horrendously expensive for a single-threaded slave :-( Abstractly, the proper solution is to store the binlog and the InnoDB redo log (and all other transactional-type logs) in the *same* log. The upper layer would provide a general transactional log facility, and binlog, InnoDB, partitioning, etc. would use this instead of their own logs. But it is hard to see how we could arrive there from current code ... For MariaDB global transaction ID, I support crash-safe without binlog enabled, so there turning off fsync() will be possible. But this does not help if a slave also needs to act as a master. Parallel replication can help a lot, by allowing group commit, amortising the cost of the fsync(). But it would still be nice to solve the orginal problem. As you say, for a slave it is not a problem to loose transactions in a crash as they can be just replicated again. The problem is to be consistent between what is in the InnoDB tables and what is in the binlog. InnoDB already records in a crash-safe way the binlog position of the last commit. So if we can ensure that InnoDB is never ahead of the binlog, we can just cut the binlog after a crash if it has more events than what InnoDB recovered. We could do this by doing binlog writes as async I/O O_DIRECT writes, and sending a signal somehow to InnoDB so it would buffer its own redo log until binlog is safe on disk. Then no fsync() would be needed. But this is so far just a loose idea, it needs more thought to be sure it can work... Anyway, this is my current long-term plan: if we cannot have a proper single database-wide transaction log, then instead coordinate writes between the multiple logs that exist. Delay writes to logs so that they are never ahead of the master log. And at crash, either recover missing pieces of logs from the master log (MWL#164), or cut logs to be consistent. Then we can be consistent without any fsync() delays. > - As I understand it, for two phase commit, storage engines are > currently expected to fsync once after a transaction is prepared and > once after a transaction is committed. If the binary log is an InnoDB > table, then two-phase commit is unnecessary and the number of fsyncs > on the system is cut in half. In MariaDB 10 and MySQL 5.6, the fsync() at commit is no longer needed (InnoDB does not have it). [Note that IIRC, MySQL 5.6 breaks the storage engine API, you will need to implement the skip-fsync-at-commit-stage, or you will loose group commit.] > - With crash safe slaves on MySQL 5.6, one theoretically does not > need to enable fsyncs on transaction commit on slaves. If some data > gets lost after a crash, then the data may be replayed from the > master. However, to get GTIDs working, one must enable the binary log > on slaves, which requires two-phase commit, which require fsyncs. By Yes, requiring binlog on slaves is a serious mistake :-( Ok, as you can perhaps tell, this is something I have thought a lot about over the past years, so I have a lot of ideas but it is a bit difficult to put it all in a single mail reply without it becomming mostly a jumble of loose ideas. You are welcome to ask for more details on any specific points of interest. - Kristian.

5 9

Re: [Maria-developers] reducing fsyncs during handlerton->prepare and handlerton->commit in 5.6
by Kristian Nielsen 31 Jan '13

31 Jan '13

Zardosht Kasheff <zardosht(a)gmail.com> writes: > - in MySQL 5.6 and MariaDB 10.0, what are the fsyncing requirements > for storage engines during prepare and commit? > - are there new APIs that the storage engine must comply with in > order to get these benefits? In MariaDB 10.0, there is a new commit_checkpoint_request handlerton method. If the storage engine implements that, it need not do the fsync() at commit. It must also implement the commit_ordered method from 5.3. Check comments in sql/handler.h for these two methods, and let me know if there are any questions. The intention is that storage engines need not do much extra work for this functionality, eg. the patch for InnoDB support was a couple hundred lines at most IIRC. > - in MySQL 5.6, I see new handlerton methods commit_low and > prepare_low. What do these APIs do? What is their contract? I think MySQL developers should answer for this. But it is related to this new check in InnoDB: thd_requested_durability(trx->mysql_thd) == HA_IGNORE_DURABILITY And the existing flush_logs handlerton method is also used. My guess is that commit_low is just internal server stuff that the storage engine does not need to consider. The MySQL 5.6 mechanism is mostly the same as the MariaDB 10.0 one in what it actually does, but they did not bother to try to be compatible with the storage engine API in MariaDB 5.3. So looks like it is #ifdef time for you if you want to support both. - Kristian.

1 0

[Maria-developers] how to build mariadb RPMs
by Rich Prohaska 30 Jan '13

30 Jan '13

How do you build the mariadb RPMs? Are you using rpmbuild or cpack? Is there a top level script that uses rpmbuild or cpack that I should use? Thanks Rich Prohaska

3 11

[Maria-developers] MariaDB 5.1.67, 5.2.14, 5.3.12, 5.5.29 Release Prep
by Daniel Bartholomew 30 Jan '13

30 Jan '13

All, I've prepped changelog and release notes pages for MariaDB 5.1.67, 5.2.14, and 5.3.12. I'll shortly be doing the same for MariaDB 5.5.29. https://kb.askmonty.org/en/mariadb-5167-release-notes/ https://kb.askmonty.org/en/mariadb-5167-changelog/ https://kb.askmonty.org/en/mariadb-5214-release-notes/ https://kb.askmonty.org/en/mariadb-5214-changelog/ https://kb.askmonty.org/en/mariadb-5312-release-notes/ https://kb.askmonty.org/en/mariadb-5312-changelog/ If there is anything I should be sure to mention in the release notes, please let me know ASAP. I plan on announcing the releases once 5.5.29 has been uploaded to the mirrors. It's still building at the moment so the earliest the release could happen is tomorrow and it could be pushed back to Thursday if there's a delay of some sort. Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

Re: [Maria-developers] [Commits] Rev 3653: MDEV-3882 .deb upgrade from mysql to mariadb fails due to apt-get considering it a downgrade. in file:///home/hf/wmar/mdev-3882/
by Kristian Nielsen 29 Jan '13

29 Jan '13

<holyfoot(a)askmonty.org> writes: > message: > MDEV-3882 .deb upgrade from mysql to mariadb fails due to apt-get considering it a downgrade. > The libmysqlclient18 package and all the dependencies on it removed. > The libmariadbclient18 set to provide the libmysqlclient's functions and > conflicts with all the libmysqlclients. > > === modified file 'debian/dist/Debian/control' > --- a/debian/dist/Debian/control 2012-08-23 13:32:03 +0000 > +++ b/debian/dist/Debian/control 2013-01-29 09:56:54 +0000 > @@ -13,7 +13,9 @@ Vcs-Bzr: bzr://lp:maria > Package: libmariadbclient18 > Section: libs > Architecture: any > -Depends: mariadb-common, libmysqlclient18 (= ${source:Version}), ${shlibs:Depends}, ${misc:Depends} > +Depends: mariadb-common, ${shlibs:Depends}, ${misc:Depends} > +Provides: libmysqlclient18 > +Conflicts: libmysqlclient10, libmysqlclient12, libmysqlclient14, libmysqlclient15, libmysqlclient16, libmysqlclient18 Are you sure this works? The original reason for introducing a libmysqlclient18 package in the first place (with all the problems in introduces) is that there are standard packages with a versioned dependency on libmysqlclient18. And versioned dependencies can not (used not to?) be satisfied by a Provides: Eg. the following gives 143 packages with versioned dependencies on libmysqlclient18: apt-cache rdepends libmysqlclient18|(read x; read x; while read x; do apt-cache show $x|grep ^Depends: | perl -nle 'print "'"$x:"' $1" if /libmysqlclient18[ \t]*$([^)]+)$/' ; done) You need to test a number of these that they are still installable when MariaDB is installed, also on old .deb distros (ie. Ubuntu 8.04). On the other hand, if this works then it is a nice solution for sure, and could also solve other issues. A related issue is that we might get the distro mysql-common package or the MariaDB mysql-common package somewhat at random, depending on which has the newer version number on any given day. I believe we made it so we work with the distro mysql-common package as well as the MariaDB one, it it will not necessarily cause any problems. It does seem a bit of a mess though, maybe we should just drop the mysql-common package from our repos? - Kristian.

1 0

Re: [Maria-developers] Fwd: [Commits] Rev 3616: Fix for bug in file:///home/tsk/mprog/src/5.3-md641/
by Igor Babaev 28 Jan '13

28 Jan '13

Timour, It's not clear for me: 1. why group by queries with GROUP BY col1, col2, where col1,col2 are defined over the same table, are considered as possibly good for for loose scan optimization. 2. how it comes that unique index is considered as a candidate for a loose scan (see your test case). (The second issue, of course, are unrelated to bug mdev-641). Besides, didn't you notice that you removes duplicates in group by under any circumstances. How it will affect the following query: select a, sum(c) from t1 group by a, a with rollup; ? In general, removing duplicates from GROUP BY and ORDER BY lists is a good optimization, but why it should be done in connection with this bug? Regards, Igor. On 01/24/2013 06:38 AM, Timour Katchaounov wrote: > Igor, > > Could you please review the following patch (we discussed it at the team > call): > > ------------------------------------------------------------ > revno: 3616 > revision-id: timour(a)askmonty.org-20130124143526-bpaiyushble1ieek > parent: timour(a)askmonty.org-20130117140805-4kyoq7azx4v2irhq > fixes bug: https://mariadb.atlassian.net/browse/MDEV-614 > committer: timour(a)askmonty.org > branch nick: 5.3-md641 > timestamp: Thu 2013-01-24 16:35:26 +0200 > message: > Fix for bug > MDEV-641 LP:1002108 - Wrong result (or crash) from a query with > duplicated field in the group list and a limit clause > Bug#11761078: 53534: INCORRECT 'SELECT SQL_BIG_RESULT...' > WITH GROUP BY ON DUPLICATED FIELDS > > Analysis > Range analysis discoveres that the query can be executed via loose > index scan for GROUP BY. > Later, GROUP BY analysis fails to confirm that the GROUP operation can > be computed via an > index because there is no logic to handle duplicate field references > in the GROUP clause. > As a result the optimizer the optimizer produces an inconsistent plan. > It constructs a > temporary table, but on the other hand the group fields are not set to > point there. > > Solution: > Make order by analysis be in sync with loose scan analysis for group > by. Instead of > changing the logic of how indexes are analyzed for order by, the patch > takes the same > approach as MySQL - normalize the query by removing duplicate column > references. The > difference from MySQL is that we remove duplicates earlier, by reusing > remove_const() > which already removed unneeded ORDER elements. > >

1 0

[Maria-developers] MDEV-3875 Wrong result on a DISTINCT query ... and GROUP BY
by Sergei Golubchik 27 Jan '13

27 Jan '13

Hi, Igor, See my patch below. The only questionable part - in remove_dup_with_compare(). Before it was memcpy'ing only part of the record, skipping const fields. Now it memcpy's the complete record. I don't think it matters much performance-wise. An alternative would be to do two memcpy's - for the NULL bitmap and for the fields (because const fields are located between the NULL bitmap that we need, and the other non-const fields). Another alternative would be to have const fields at the end of the record, but it'd be a much larger and more risky change. ------------------------------------------------------------ revno: 3620 fixes bug: https://mariadb.atlassian.net/browse/MDEV-3875 committer: Sergei Golubchik <sergii(a)pisem.net> branch nick: 5.3 timestamp: Sat 2013-01-26 22:33:18 +0100 message: MDEV-3875 Wrong result (missing row) on a DISTINCT query with the same subquery in the SELECT list and GROUP BY fix remove_dup_with_hash_index() and remove_dup_with_compare() to take NULLs into account modified: mysql-test/r/distinct.result mysql-test/t/distinct.test sql/field.cc sql/field.h sql/filesort.cc sql/sql_select.cc diff: === modified file 'mysql-test/r/distinct.result' --- mysql-test/r/distinct.result 2011-12-15 22:26:59 +0000 +++ mysql-test/r/distinct.result 2013-01-26 21:33:18 +0000 @@ -847,3 +847,35 @@ time(f1) 00:00:00.000200 00:00:00.000300 drop table t1; +create table t1(i int, g int); +insert into t1 values (null, 1), (0, 2); +select distinct i from t1 group by g; +i +NULL +0 +drop table t1; +create table t1(i int, g blob); +insert into t1 values (null, 1), (0, 2); +select distinct i from t1 group by g; +i +NULL +0 +drop table t1; +create table t1 (a int) engine=myisam; +insert into t1 values (0),(7); +create table t2 (b int) engine=myisam; +insert into t2 values (7),(0),(3); +create algorithm=temptable view v as +select distinct (select max(a) from t1 where alias.b = a) as field1 from t2 as alias group by field1; +select * from v; +field1 +NULL +0 +7 +select distinct (select max(a) from t1 where alias.b = a) as field1 from t2 as alias group by field1; +field1 +NULL +0 +7 +drop view v; +drop table t1, t2; === modified file 'mysql-test/t/distinct.test' --- mysql-test/t/distinct.test 2011-12-15 22:26:59 +0000 +++ mysql-test/t/distinct.test 2013-01-26 21:33:18 +0000 @@ -658,3 +658,27 @@ select time(f1) from t1 ; select distinct time(f1) from t1 ; drop table t1; +# +# MDEV-3875 Wrong result (missing row) on a DISTINCT query with the same subquery in the SELECT list and GROUP BY +# MySQL Bug#66896 Distinct not distinguishing 0 from NULL when GROUP BY is used +# +create table t1(i int, g int); # remove_dup_with_hash_index +insert into t1 values (null, 1), (0, 2); +select distinct i from t1 group by g; +drop table t1; + +create table t1(i int, g blob); # remove_dup_with_compare +insert into t1 values (null, 1), (0, 2); +select distinct i from t1 group by g; +drop table t1; + +create table t1 (a int) engine=myisam; +insert into t1 values (0),(7); +create table t2 (b int) engine=myisam; +insert into t2 values (7),(0),(3); +create algorithm=temptable view v as +select distinct (select max(a) from t1 where alias.b = a) as field1 from t2 as alias group by field1; +select * from v; +select distinct (select max(a) from t1 where alias.b = a) as field1 from t2 as alias group by field1; +drop view v; +drop table t1, t2; === modified file 'sql/field.cc' --- sql/field.cc 2012-08-29 15:55:59 +0000 +++ sql/field.cc 2013-01-26 21:33:18 +0000 @@ -1062,6 +1062,21 @@ bool Field::type_can_have_key_part(enum } +void Field::make_sort_key(uchar *buff,uint length) +{ + if (maybe_null()) + { + if (is_null()) + { + bzero(buff, length + 1); + return; + } + *buff++= 1; + } + sort_string(buff, length); +} + + /** Numeric fields base class constructor. */ === modified file 'sql/field.h' --- sql/field.h 2012-06-23 22:00:05 +0000 +++ sql/field.h 2013-01-26 21:33:18 +0000 @@ -386,6 +386,7 @@ class Field return bytes; } + void make_sort_key(uchar *buff, uint length); virtual void make_field(Send_field *); virtual void sort_string(uchar *buff,uint length)=0; virtual bool optimize_range(uint idx, uint part); === modified file 'sql/filesort.cc' --- sql/filesort.cc 2012-11-09 08:11:20 +0000 +++ sql/filesort.cc 2013-01-26 21:33:18 +0000 @@ -783,21 +783,9 @@ static void make_sortkey(register SORTPA bool maybe_null=0; if ((field=sort_field->field)) { // Field - if (field->maybe_null()) - { - if (field->is_null()) - { - if (sort_field->reverse) - bfill(to,sort_field->length+1,(char) 255); - else - bzero((char*) to,sort_field->length+1); - to+= sort_field->length+1; - continue; - } - else - *to++=1; - } - field->sort_string(to, sort_field->length); + field->make_sort_key(to, sort_field->length); + if ((maybe_null = field->maybe_null())) + to++; } else { // Item @@ -949,8 +937,11 @@ static void make_sortkey(register SORTPA } if (sort_field->reverse) { /* Revers key */ - if (maybe_null) - to[-1]= ~to[-1]; + if (maybe_null && (to[-1]= !to[-1])) + { + to+= sort_field->length; // don't waste the time reversing all 0's + continue; + } length=sort_field->length; while (length--) { === modified file 'sql/sql_select.cc' --- sql/sql_select.cc 2013-01-16 13:11:13 +0000 +++ sql/sql_select.cc 2013-01-26 21:33:18 +0000 @@ -204,7 +204,7 @@ static int create_sort_index(THD *thd, J static int remove_duplicates(JOIN *join,TABLE *entry,List<Item> &fields, Item *having); static int remove_dup_with_compare(THD *thd, TABLE *entry, Field **field, - ulong offset,Item *having); + Item *having); static int remove_dup_with_hash_index(THD *thd,TABLE *table, uint field_count, Field **first_field, ulong key_length,Item *having); @@ -18891,19 +18891,24 @@ static bool fix_having(JOIN *join, Item #endif -/***************************************************************************** - Remove duplicates from tmp table - This should be recoded to add a unique index to the table and remove - duplicates - Table is a locked single thread table - fields is the number of fields to check (from the end) -*****************************************************************************/ +/** + Compare fields from table->record[0] and table->record[1], + possibly skipping few first fields. + @param table + @param ptr field to start the comparison from, + somewhere in the table->field[] array + + @retval 1 different + @retval 0 identical +*/ static bool compare_record(TABLE *table, Field **ptr) { for (; *ptr ; ptr++) { - if ((*ptr)->cmp_offset(table->s->rec_buff_length)) + Field *f= *ptr; + if (f->is_null() != f->is_null(table->s->rec_buff_length) || + (!f->is_null() && f->cmp_offset(table->s->rec_buff_length))) return 1; } return 0; @@ -18931,15 +18936,15 @@ static void free_blobs(Field **ptr) static int -remove_duplicates(JOIN *join, TABLE *entry,List<Item> &fields, Item *having) +remove_duplicates(JOIN *join, TABLE *table, List<Item> &fields, Item *having) { int error; - ulong reclength,offset; + ulong keylength= 0; uint field_count; THD *thd= join->thd; DBUG_ENTER("remove_duplicates"); - entry->reginfo.lock_type=TL_WRITE; + table->reginfo.lock_type=TL_WRITE; /* Calculate how many saved fields there is in list */ field_count=0; @@ -18956,24 +18961,21 @@ remove_duplicates(JOIN *join, TABLE *ent join->unit->select_limit_cnt= 1; // Only send first row DBUG_RETURN(0); } - Field **first_field=entry->field+entry->s->fields - field_count; - offset= (field_count ? - entry->field[entry->s->fields - field_count]-> - offset(entry->record[0]) : 0); - reclength=entry->s->reclength-offset; - - free_io_cache(entry); // Safety - entry->file->info(HA_STATUS_VARIABLE); - if (entry->s->db_type() == heap_hton || - (!entry->s->blob_fields && - ((ALIGN_SIZE(reclength) + HASH_OVERHEAD) * entry->file->stats.records < + + Field **first_field=table->field+table->s->fields - field_count; + for (Field **ptr=first_field; *ptr; ptr++) + keylength+= (*ptr)->sort_length() + (*ptr)->maybe_null(); + + free_io_cache(table); // Safety + table->file->info(HA_STATUS_VARIABLE); + if (table->s->db_type() == heap_hton || + (!table->s->blob_fields && + ((ALIGN_SIZE(keylength) + HASH_OVERHEAD) * table->file->stats.records < thd->variables.sortbuff_size))) - error=remove_dup_with_hash_index(join->thd, entry, - field_count, first_field, - reclength, having); + error=remove_dup_with_hash_index(join->thd, table, field_count, first_field, + keylength, having); else - error=remove_dup_with_compare(join->thd, entry, first_field, offset, - having); + error=remove_dup_with_compare(join->thd, table, first_field, having); free_blobs(first_field); DBUG_RETURN(error); @@ -18981,18 +18983,13 @@ remove_duplicates(JOIN *join, TABLE *ent static int remove_dup_with_compare(THD *thd, TABLE *table, Field **first_field, - ulong offset, Item *having) + Item *having) { handler *file=table->file; - char *org_record,*new_record; - uchar *record; + uchar *record=table->record[0]; int error; - ulong reclength= table->s->reclength-offset; DBUG_ENTER("remove_dup_with_compare"); - org_record=(char*) (record=table->record[0])+offset; - new_record=(char*) table->record[1]+offset; - if (file->ha_rnd_init_with_error(1)) DBUG_RETURN(1); @@ -19029,7 +19026,7 @@ static int remove_dup_with_compare(THD * error=0; goto err; } - memcpy(new_record,org_record,reclength); + store_record(table,record[1]); /* Read through rest of file and mark duplicated rows deleted */ bool found=0; @@ -19088,8 +19085,9 @@ static int remove_dup_with_hash_index(TH int error; handler *file= table->file; ulong extra_length= ALIGN_SIZE(key_length)-key_length; - uint *field_lengths,*field_length; + uint *field_lengths, *field_length; HASH hash; + Field **ptr; DBUG_ENTER("remove_dup_with_hash_index"); if (!my_multi_malloc(MYF(MY_WME), @@ -19101,21 +19099,8 @@ static int remove_dup_with_hash_index(TH NullS)) DBUG_RETURN(1); - { - Field **ptr; - ulong total_length= 0; - for (ptr= first_field, field_length=field_lengths ; *ptr ; ptr++) - { - uint length= (*ptr)->sort_length(); - (*field_length++)= length; - total_length+= length; - } - DBUG_PRINT("info",("field_count: %u key_length: %lu total_length: %lu", - field_count, key_length, total_length)); - DBUG_ASSERT(total_length <= key_length); - key_length= total_length; - extra_length= ALIGN_SIZE(key_length)-key_length; - } + for (ptr= first_field, field_length=field_lengths ; *ptr ; ptr++) + (*field_length++)= (*ptr)->sort_length(); if (hash_init(&hash, &my_charset_bin, (uint) file->stats.records, 0, key_length, (hash_get_key) 0, 0, 0)) @@ -19155,13 +19140,13 @@ static int remove_dup_with_hash_index(TH /* copy fields to key buffer */ org_key_pos= key_pos; field_length=field_lengths; - for (Field **ptr= first_field ; *ptr ; ptr++) + for (ptr= first_field ; *ptr ; ptr++) { - (*ptr)->sort_string(key_pos,*field_length); - key_pos+= *field_length++; + (*ptr)->make_sort_key(key_pos, *field_length); + key_pos+= (*ptr)->maybe_null() + *field_length++; } /* Check if it exists before */ - if (hash_search(&hash, org_key_pos, key_length)) + if (my_hash_search(&hash, org_key_pos, key_length)) { /* Duplicated found ; Remove the row */ if ((error= file->ha_delete_row(record))) Regards, Sergei

1 0

[Maria-developers] MariaDB RPMs
by Rich Prohaska 24 Jan '13

24 Jan '13

Suppose that we have a RHEL5 machine running and we want to install MariaDB on it. Unfortunately, it conflicts with files installed by the mysql package. # rpm -i MariaDB-5.5.28a-centos5-x86_64-common.rpm warning: MariaDB-5.5.28a-centos5-x86_64-common.rpm: Header V3 DSA signature: NOKEY, key ID 1bb943db file /etc/my.cnf from install of MariaDB-common-5.5.28a-1.x86_64 conflicts with file from package mysql-5.0.77-4.el5_6.6.x86_64 file /usr/share/mysql/charsets/Index.xml from install of MariaDB-common-5.5.28a-1.x86_64 conflicts with file from package mysql-5.0.77-4.el5_6.6.x86_64 file /usr/share/mysql/charsets/cp1250.xml from install of MariaDB-common-5.5.28a-1.x86_64 conflicts with file from package mysql-5.0.77-4.el5_6.6.x86_64 file /usr/share/mysql/charsets/cp1251.xml from install of MariaDB-common-5.5.28a-1.x86_64 conflicts with file from package mysql-5.0.77-4.el5_6.6.x86_64 Unfortunately, a lot of other packages require the mysql package. What is a good approach to solving this problem? Thanks

2 1

[Maria-developers] A new Ubuntu UPDATE event problem smashed MariaDB dead
by Peter W Bowey 23 Jan '13

23 Jan '13

Subject: A new Ubuntu UPDATE event problem smashed MariaDB 5.5.28a dead Hi MariaDB Team, Ubuntu released a update 23 hours ago that 'smashed out' omega8.cc's careful work to support MariaDB 5.5.28-mariadb-a1~precise. Ubuntu's new version is '5.5.29-0ubuntu012.10.1' dated - 2013-01-22 15:07:08 UTC. This does not work yet with MariaDB & Ubuntu 12.04 See Link here: https://launchpad.net/ubuntu/quantal/+source/mysql-5.5/5.5.29-0ubuntu0.12..… Now, I have been 'battling for hours' to work around this Ubuntu pain, while we wait on the good MariaDB team to re-syncs it's own updates with the Ubuntu 12.04 updates. Here is the 'crude *forced* style' I worked with to MariaDB back to life - after the Ubuntu *forced* update and dependency killed the database [MariaDB]: -------------------------------------------------------------------------------- server:~/aegir-boa-installer# aptitude install mariadb-server-5.5 mariadb-common The following NEW packages will be installed: libdbd-mysql-perl{a} libmariadbclient18{ab} libmysqlclient18{a} mariadb-client-5.5{a} mariadb-client-core-5.5{a} mariadb-server-5.5 mariadb-server-core-5.5{a} 0 packages upgraded, 7 newly installed, 0 to remove and 0 not upgraded. Need to get 31.8 MB of archives. After unpacking 107 MB will be used. The following packages have unmet dependencies: mysql-common : Breaks: mysql-client-5.1 which is a virtual package. Breaks: mysql-client-core-5.1 which is a virtual package. Breaks: mysql-server-core-5.1 which is a virtual package. libmariadbclient18 : Depends: libmysqlclient18 (= 5.5.28-mariadb-a1~precise) but 5.5.29-0ubuntu0.12.04.1 is to be installed. The following actions will resolve these dependencies: Keep the following packages at their current version: 1) libmariadbclient18 [Not Installed] 2) mariadb-client-5.5 [Not Installed] 3) mariadb-client-core-5.5 [Not Installed] 4) mariadb-server-5.5 [Not Installed] 5) mariadb-server-core-5.5 [Not Installed] Accept this solution? [Y/n/q/?] n The following actions will resolve these dependencies: Install the following packages: 1) libmysqlclient18 [5.5.28-mariadb-a1~precise (<NULL>)] Downgrade the following packages: 2) mysql-common [5.5.29-0ubuntu0.12.04.1 (now, precise-security, precise-updates) -> 5.5.28-mariadb-a1~precise (<NULL>)] Accept this solution? [Y/n/q/?] The following packages will be DOWNGRADED: mysql-common The following NEW packages will be installed: libdbd-mysql-perl{a} libmariadbclient18{a} libmysqlclient18{a} mariadb-client-5.5{a} mariadb-client-core-5.5{a} mariadb-server-5.5 mariadb-server-core-5.5{a} 0 packages upgraded, 7 newly installed, 1 downgraded, 0 to remove and 0 not upgraded. Need to get 30.9 MB of archives. After unpacking 104 MB will be used. Do you want to continue? [Y/n/?] y Get: 1 http://archive.ubuntu.com/ubuntu/ precise/main libdbd-mysql-perl i386 4.020-1build2 [104 kB] Get: 2 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main mysql-common all 5.5.28-mariadb-a1~precise [8,780 B] Get: 3 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main libmariadbclient18 i386 5.5.28-mariadb-a1~precise [817 kB] Get: 4 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main libmysqlclient18 i386 5.5.28-mariadb-a1~precise [2,920 B] Get: 5 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main mariadb-client-core-5.5 i386 5.5.28-mariadb-a1~precise [1,797 kB] Get: 6 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main mariadb-client-5.5 i386 5.5.28-mariadb-a1~precise [5,097 kB] Get: 7 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main mariadb-server-core-5.5 i386 5.5.28-mariadb-a1~precise [5,113 kB] Get: 8 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main mariadb-server-5.5 i386 5.5.28-mariadb-a1~precise [18.0 MB] Fetched 30.9 MB in 30s (1,020 kB/s) Preconfiguring packages ... dpkg: warning: downgrading mysql-common from 5.5.29-0ubuntu0.12.04.1 to 5.528-mariadb-a1~precise. (Reading database ... 61332 files and directories currently installed.) Preparing to replace mysql-common 5.5.29-0ubuntu0.12.04.1 (using ..../mysql-common_5.5.28-mariadb-a1~precise_all.deb) ... Unpacking replacement mysql-common ... Selecting previously unselected package libmariadbclient18. Unpacking libmariadbclient18 (from ..../libmariadbclient18_5.5.28-mariadb-a1~precise_i386.deb) ... Selecting previously unselected package libmysqlclient18. Unpacking libmysqlclient18 (from ..../libmysqlclient18_5.5.28-mariadb-a1~precise_i386.deb) ... Selecting previously unselected package libdbd-mysql-perl. Unpacking libdbd-mysql-perl (from ..../libdbd-mysql-perl_4.020-1build2_i386deb) ... Selecting previously unselected package mariadb-client-core-5.5. Unpacking mariadb-client-core-5.5 (from ..../mariadb-client-core-5.5_5.5.28-mariadb-a1~precise_i386.deb) ... Selecting previously unselected package mariadb-client-5.5. Unpacking mariadb-client-5.5 (from ..../mariadb-client-5.5_5.5.28-mariadb-a1~precise_i386.deb) ... Selecting previously unselected package mariadb-server-core-5.5. Unpacking mariadb-server-core-5.5 (from ..../mariadb-server-core-5.5_5.5.28-mariadb-a1~precise_i386.deb) ... Selecting previously unselected package mariadb-server-5.5. Unpacking mariadb-server-5.5 (from ..../mariadb-server-5.5_5.5.28-mariadb-a1~precise_i386.deb) ... * Stopping MariaDB database server mysqld [ OK ] Processing triggers for man-db ... Processing triggers for ureadahead ... Setting up mysql-common (5.5.28-mariadb-a1~precise) ... Configuration file `/etc/mysql/my.cnf' ==> Modified (by you or by a script) since installation. ==> Package distributor has shipped an updated version. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version. *** my.cnf (Y/I/N/O/D/Z) [default=N] ? N Setting up libmysqlclient18 (5.5.28-mariadb-a1~precise) ... Setting up libdbd-mysql-perl (4.020-1build2) ... Setting up libmariadbclient18 (5.5.28-mariadb-a1~precise) ... Setting up mariadb-client-core-5.5 (5.5.28-mariadb-a1~precise) ... Setting up mariadb-client-5.5 (5.5.28-mariadb-a1~precise) ... Setting up mariadb-server-core-5.5 (5.5.28-mariadb-a1~precise) ... Setting up mariadb-server-5.5 (5.5.28-mariadb-a1~precise) ... * Stopping MariaDB database server mysqld [ OK ] * Starting MariaDB database server mysqld [ OK ] * Checking for corrupt, not cleanly closed and upgrade needing tables. Processing triggers for libc-bin ... ldconfig deferred processing now taking place -------------------------------------------------------------------------------- Ubuntu will drag in the new update dependants - > mysql-common [5.5.29-0ubuntu0.12.04.1 (now, precise-security, precise-updates) instead of 5.5.28-mariadb-a1~precise (<NULL>)] Peter Peter Bowey Web Design & Developer Peter Bowey Computer Solutions 69 Sutherland Ave, Victor Harbor, South Australia, Australia Tel.: 08 8552 8630 Fax: 08 8552 9185 Email: support(a)pbcomp.com.au Web: http://www.pbcomp.com.au/ Making Windows, Macintosh & Linux work for you! http://www.pbcomp.com.au/ This email and any attachment is intended only for the exclusive and confidential use of the addressee(s). If you are not the intended recipient, any use, interference with, disclosure or copying of this material is unauthorised and prohibited. If you have received this message in error, please notify the sender by return email immediately and delete the message from your computer without making any copies and publishing the contents therein. Consider the environment before printing this e-mail.

2 1

Re: [Maria-developers] Rev 3177: Fixed bug mdev-4063 (bug #56927)
by Sergei Golubchik 21 Jan '13

21 Jan '13

Hi, Igor! On Jan 21, Igor Babaev wrote: > revno: 3177 > revision-id: igor(a)askmonty.org-20130121084657-hvcval7hykojxpmg > parent: sergii(a)pisem.net-20130109225151-z9e7gh8z5nl38dxl > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.1-bug56927 > timestamp: Mon 2013-01-21 00:46:57 -0800 > message: > Fixed bug mdev-4063 (bug #56927). > This bug could result in returning 0 for the expressions of the form > <aggregate_function>(distinct field) when the system variable > max_heap_table_size was set to a small enough number. > It happened because the method Unique::walk() did not support > the case when more than one pass was needed to merge the trees > of distinct values saved in an external file. Looks ok. Just one thought. You use merge_many_buff() if the memory is not enough for the number_of_buffpeks*size. But merge_many_buff() only reduces the number of buffpeks down to 15. What if max_heap_table_size < 15*size ? I'd suggest to either 1) issue an error or 2) merge completely, like in Unique::get(). Actually, 2) should be easy, you just pass the last argument to your new Unique::merge() instead of TRUE, like in your patch, change to merge( ..., max_in_memory_size > MERGEBUFF2*size); Regards, Sergei

1 0

Re: [Maria-developers] Bug in trx_group_commit_leader()
by Kristian Nielsen 21 Jan '13

21 Jan '13

Michael Widenius <monty(a)askmonty.org> writes: > I noticed this compiler error on one of the build machines for 10.0-base: > > /home/buildbot/buildbot/build/mariadb-10.0.0/sql/log.cc:6564:40: > warning: ‘last_in_queue’ may be used uninitialized in this function > [-Wuninitialized] Thanks. I've pushed the fix to 10.0-base. > Looking at the code, this looks like a real potential bug. Yes (though likely dead code, as there's a DBUG_ASSERT before the if () asserting that it is always true). - Kristian.

1 0

[Maria-developers] review of MDEV-4011 per thread memory counting and usage
by Sergei Golubchik 19 Jan '13

19 Jan '13

Hi. Here it is: > === modified file 'client/completion_hash.cc' > --- client/completion_hash.cc 2011-06-30 15:46:53 +0000 > +++ client/completion_hash.cc 2013-01-15 21:00:33 +0000 > @@ -49,7 +49,7 @@ int completion_hash_init(HashTable *ht, > ht->initialized = 0; > return FAILURE; > } > - init_alloc_root(&ht->mem_root, 8192, 0); > + init_alloc_root(&ht->mem_root, 8192, 0, 0); very often you write 0 or MY_THREAD_SPECIFIC instead of MYF(0) and MYF(MY_THREAD_SPECIFIC) > ht->pHashFunction = hashpjw; > ht->nTableSize = nSize; > ht->initialized = 1; > === modified file 'include/my_sys.h' > --- include/my_sys.h 2012-12-23 21:37:11 +0000 > +++ include/my_sys.h 2013-01-15 21:00:33 +0000 > @@ -86,6 +86,8 @@ typedef struct my_aio_result { > #define MY_SYNC 4096 /* my_copy(): sync dst file */ > #define MY_SYNC_DIR 32768 /* my_create/delete/rename: sync directory */ > #define MY_SYNC_FILESIZE 65536 /* my_sync(): safe sync when file is extended */ > +#define MY_THREAD_SPECIFIC 0x10000 /* my_malloc(): thread specific */ > +#define MY_THREAD_MOVE 0x20000 /* realloc(); Memory can move */ may be, better to remove this, as it's unused? it's easier to add it later (just a couple of lines of code), than to keep the dead code around. > #define MY_CHECK_ERROR 1 /* Params to my_end; Check open-close */ > #define MY_GIVE_INFO 2 /* Give time info about process*/ > @@ -148,6 +150,42 @@ typedef struct my_aio_result { > /* Extra length needed for filename if one calls my_create_backup_name */ > #define MY_BACKUP_NAME_EXTRA_LENGTH 17 > > +/* If we have our own safemalloc (for debugging) */ > +#if defined(SAFEMALLOC) > +#define MALLOC_SIZE_AND_FLAG(p,b) sf_malloc_usable_size(p,b) > +#define MALLOC_PREFIX_SIZE 0 > +#define MALLOC_STORE_SIZE(a,b,c,d) > +#define MALLOC_FIX_POINTER_FOR_FREE(a) a > +#define SAFEMALLOC_REPORT_MEMORY(X) sf_report_leaked_memory(X) > +void sf_report_leaked_memory(my_thread_id id); > +extern my_thread_id (*sf_malloc_dbug_id)(void); > +#else > +/* > + * We use double as prefix size as this guarantees the correct > + * alignment on all platforms and will optimize things for > + * memcpy(), memcmp() etc. > + */ > +#define MALLOC_PREFIX_SIZE (sizeof(double)) > +#define MALLOC_SIZE(p) (*(size_t*) ((char*)(p) - MALLOC_PREFIX_SIZE)) > +#define MALLOC_STORE_SIZE(p, type_of_p, size, flag) \ > +{\ > + *(size_t*) p= (size) | (flag); \ > + (p)= (type_of_p) (((char*) (p)) + MALLOC_PREFIX_SIZE); \ > +} > +static inline size_t malloc_size_and_flag(void *p, myf *flags) > +{ > + size_t size= MALLOC_SIZE(p); > + *flags= (size & 1); > + return size & ~ (ulonglong) 1; > +} > +#define MALLOC_SIZE_AND_FLAG(p,b) malloc_size_and_flag(p, b); > +#define MALLOC_FIX_POINTER_FOR_FREE(p) (((char*) (p)) - MALLOC_PREFIX_SIZE) > +#define SAFEMALLOC_REPORT_MEMORY(X) do {} while(0) > +#endif /* SAFEMALLOC */ > + > +typedef void (*MALLOC_SIZE_CB) (long long size, myf my_flags); > +extern void set_malloc_size_cb(MALLOC_SIZE_CB func); I don't think you need many of these defines here. my_sys.h is the interface header. MALLOC_STORE_SIZE and MALLOC_FIX_POINTER_FOR_FREE (for example, I didn't check the others) are only used in my_malloc.c - they should not be used elsewhere, and thus, I think, they should be defined in my_malloc.c And, frankly speaking, I wouldn't bother doing two different ways of storing the size and the flags. my_malloc can always allocate one sizeof(double) more and store its data there. Who cares that in the debug builds safemalloc duplicates that... > /* defines when allocating data */ > extern void *my_malloc(size_t Size,myf MyFlags); > extern void *my_multi_malloc(myf MyFlags, ...); > @@ -323,6 +361,7 @@ typedef struct st_dynamic_array > uint elements,max_element; > uint alloc_increment; > uint size_of_element; > + myf malloc_flags; hm. you've preserved the MEM_ROOT structure for compatibiity reasons, but you've changed DYNAMIC_ARRAY. Why? I'd expect you either to change everything or to change nothing. Besides, what's the point of preserving MEM_ROOT, if you change prototypes of functions that work with it? > } DYNAMIC_ARRAY; > > typedef struct st_my_tmpdir > @@ -768,16 +807,13 @@ extern my_bool real_open_cached_file(IO_ > extern void close_cached_file(IO_CACHE *cache); > File create_temp_file(char *to, const char *dir, const char *pfx, > int mode, myf MyFlags); > -#define my_init_dynamic_array(A,B,C,D) init_dynamic_array2(A,B,NULL,C,D) > -#define my_init_dynamic_array_ci(A,B,C,D) init_dynamic_array2(A,B,NULL,C,D) > -#define my_init_dynamic_array2(A,B,C,D,E) init_dynamic_array2(A,B,C,D,E) > -#define my_init_dynamic_array2_ci(A,B,C,D,E) init_dynamic_array2(A,B,C,D,E) > +#define my_init_dynamic_array(A,B,C,D,E) init_dynamic_array2(A,B,NULL,C,D,E) > +#define my_init_dynamic_array_ci(A,B,C,D,E) init_dynamic_array2(A,B,NULL,C,D,E) > +#define my_init_dynamic_array2(A,B,C,D,E,F) init_dynamic_array2(A,B,C,D,E,F) > +#define my_init_dynamic_array2_ci(A,B,C,D,E,F) init_dynamic_array2(A,B,C,D,E,F) as you're changing function prototypes anyway, you can remove the _ci functions and all other "preserve the ABI" garbage, > extern my_bool init_dynamic_array2(DYNAMIC_ARRAY *array, uint element_size, > void *init_buffer, uint init_alloc, > - uint alloc_increment); > -/* init_dynamic_array() function is deprecated */ > -extern my_bool init_dynamic_array(DYNAMIC_ARRAY *array, uint element_size, > - uint init_alloc, uint alloc_increment); > + uint alloc_increment, myf my_flags); > extern my_bool insert_dynamic(DYNAMIC_ARRAY *array, const uchar * element); > extern uchar *alloc_dynamic(DYNAMIC_ARRAY *array); > extern uchar *pop_dynamic(DYNAMIC_ARRAY*); > @@ -829,7 +865,8 @@ extern void my_free_lock(void *ptr); > #define ALLOC_ROOT_MIN_BLOCK_SIZE (MALLOC_OVERHEAD + sizeof(USED_MEM) + 8) > #define clear_alloc_root(A) do { (A)->free= (A)->used= (A)->pre_alloc= 0; (A)->min_malloc=0;} while(0) > extern void init_alloc_root(MEM_ROOT *mem_root, size_t block_size, > - size_t pre_alloc_size); > + size_t pre_alloc_size, myf my_flags); > +extern void alloc_root_set_min_malloc(MEM_ROOT *mem_root, size_t min_malloc); if you'd store the flag in the lowest bit of the block_size or if you wouldn't try to keep the old MEM_ROOT structure, this function wouldn't be needed > extern void *alloc_root(MEM_ROOT *mem_root, size_t Size); > extern void *multi_alloc_root(MEM_ROOT *mem_root, ...); > extern void free_root(MEM_ROOT *root, myf MyFLAGS); > === modified file 'include/my_tree.h' > --- include/my_tree.h 2011-11-03 18:17:05 +0000 > +++ include/my_tree.h 2013-01-15 21:00:33 +0000 > @@ -68,13 +68,15 @@ typedef struct st_tree { > MEM_ROOT mem_root; > my_bool with_delete; > tree_element_free free; > + myf malloc_flags; > uint flag; > } TREE; > > /* Functions on whole tree */ > void init_tree(TREE *tree, size_t default_alloc_size, size_t memory_limit, > int size, qsort_cmp2 compare, my_bool with_delete, > - tree_element_free free_element, void *custom_arg); > + tree_element_free free_element, void *custom_arg, > + myf my_flags_for_malloc); I'd just call it my_flags. it doesn't matter what tree is doing with them. and there may be other flags too. E.g. you could've removed 'with_delete' and added MY_TREE_WITH_DELETE > void delete_tree(TREE*); > void reset_tree(TREE*); > > === modified file 'include/mysql.h' > --- include/mysql.h 2012-11-20 14:24:39 +0000 > +++ include/mysql.h 2013-01-15 21:00:33 +0000 > @@ -167,7 +167,7 @@ enum mysql_option > MYSQL_OPT_GUESS_CONNECTION, MYSQL_SET_CLIENT_IP, MYSQL_SECURE_AUTH, > MYSQL_REPORT_DATA_TRUNCATION, MYSQL_OPT_RECONNECT, > MYSQL_OPT_SSL_VERIFY_SERVER_CERT, MYSQL_PLUGIN_DIR, MYSQL_DEFAULT_AUTH, > - MYSQL_PROGRESS_CALLBACK, > + MYSQL_PROGRESS_CALLBACK, MYSQL_THREAD_SPECIFIC_MALLOC, I don't think you need this on the client side. (the only use case is federated, and I don't think you need to extend the client API specifically for it, federated can set whatever it needs directliy). > /* MariaDB options */ > MYSQL_OPT_NONBLOCK=6000 > }; > === modified file 'mysys/array.c' > --- mysys/array.c 2012-01-13 14:50:02 +0000 > +++ mysys/array.c 2013-01-15 21:00:33 +0000 > @@ -62,17 +64,19 @@ my_bool init_dynamic_array2(DYNAMIC_ARRA > should not throw an error > */ > if (init_alloc && > - !(array->buffer= (uchar*) my_malloc(element_size*init_alloc, MYF(0)))) > + !(array->buffer= (uchar*) my_malloc(element_size*init_alloc, > + MYF(my_flags)))) > array->max_element=0; > DBUG_RETURN(FALSE); > } > > my_bool init_dynamic_array(DYNAMIC_ARRAY *array, uint element_size, > - uint init_alloc, uint alloc_increment) > + uint init_alloc, uint alloc_increment, > + myf my_flags) > { > /* placeholder to preserve ABI */ > return my_init_dynamic_array_ci(array, element_size, init_alloc, You can remove this, as you're changing the ABI anyway > - alloc_increment); > + alloc_increment, my_flags); > } > /* > Insert element at the end of array. Allocate memory if needed. > === modified file 'mysys/my_alloc.c' > --- mysys/my_alloc.c 2012-01-13 14:50:02 +0000 > +++ mysys/my_alloc.c 2013-01-15 21:00:33 +0000 > @@ -41,16 +48,20 @@ > Altough error can happen during execution of this function if > pre_alloc_size is non-0 it won't be reported. Instead it will be > reported as error in first alloc_root() on this memory root. > + > + We don't want to change the api for MEMROOT. > + Becasue of this, we store in min_malloc of my_thread_specific is set. > */ > > void init_alloc_root(MEM_ROOT *mem_root, size_t block_size, > - size_t pre_alloc_size __attribute__((unused))) > + size_t pre_alloc_size __attribute__((unused)), > + myf my_flags) > { > DBUG_ENTER("init_alloc_root"); > DBUG_PRINT("enter",("root: 0x%lx", (long) mem_root)); > > mem_root->free= mem_root->used= mem_root->pre_alloc= 0; > - mem_root->min_malloc= 32; > + mem_root->min_malloc= 32 | (test(my_flags & MY_THREAD_SPECIFIC) << 16); it'd be safer to put it in the lowest bit. min_malloc or - particularly - block size, are always even numbers, the lowest bit is always 0. or to add a new field to the MEM_ROOT, as compatibility is already broken anyway. > mem_root->block_size= block_size - ALLOC_ROOT_MIN_BLOCK_SIZE; > mem_root->error_handler= 0; > mem_root->block_num= 4; /* We shift this with >>2 */ > === modified file 'mysys/my_malloc.c' > --- mysys/my_malloc.c 2012-01-13 14:50:02 +0000 > +++ mysys/my_malloc.c 2013-01-15 21:00:33 +0000 > @@ -18,6 +18,31 @@ > #include "mysys_err.h" > #include <m_string.h> > > +MALLOC_SIZE_CB malloc_size_cb_func= NULL; > + > +/** > + Inform appliction that memory usage has changed application > + > + @param size Size of memory segment allocated or freed > + @param flag 1 if thread specific (allocated by MY_THREAD_SPECIFIC), > + 0 if system specific. > + > + The type os size is long long, to be able to handle negative numbers to > + decrement the memory usage > +*/ > + > +static void update_malloc_size(long long size, myf my_flags) > +{ > + if (malloc_size_cb_func) > + malloc_size_cb_func(size, my_flags); > +} > + > +void set_malloc_size_cb(MALLOC_SIZE_CB func) > +{ > + malloc_size_cb_func= func; > +} > + > + > /** > Allocate a sized block of memory. > > === modified file 'mysys/mysys_priv.h' > --- mysys/mysys_priv.h 2011-12-12 21:58:24 +0000 > +++ mysys/mysys_priv.h 2013-01-15 21:00:33 +0000 > @@ -70,12 +70,13 @@ extern PSI_file_key key_file_charset, ke > #endif /* HAVE_PSI_INTERFACE */ > > #ifdef SAFEMALLOC > -void *sf_malloc(size_t size); > -void *sf_realloc(void *ptr, size_t size); > +void *sf_malloc(size_t size, myf my_flags); > +void *sf_realloc(void *ptr, size_t size, myf my_flags); > void sf_free(void *ptr); > +size_t sf_malloc_usable_size(void *ptr, myf *my_flags); this is used nowhere. dead code again. > #else > -#define sf_malloc(X) malloc(X) > -#define sf_realloc(X,Y) realloc(X,Y) > +#define sf_malloc(X,Y) malloc(X) > +#define sf_realloc(X,Y,Z) realloc(X,Y) > #define sf_free(X) free(X) > #endif > > === modified file 'mysys/safemalloc.c' > --- mysys/safemalloc.c 2012-04-18 01:29:26 +0000 > +++ mysys/safemalloc.c 2013-01-15 21:00:33 +0000 > @@ -79,11 +81,21 @@ static int bad_ptr(const char *where, vo > static void free_memory(void *ptr); > static void sf_terminate(); > > +/* Setup default call to get a thread id for the memory */ > + > +my_thread_id default_sf_malloc_dbug_id(void) > +{ > + return my_thread_dbug_id(); > +} > + > +my_thread_id (*sf_malloc_dbug_id)(void)= default_sf_malloc_dbug_id; Why do you need different functions for getting the thread id? > + > + > /** > allocates memory > */ > > -void *sf_malloc(size_t size) > +void *sf_malloc(size_t size, myf my_flags) > { > struct st_irem *irem; > uchar *data; > === modified file 'sql/event_scheduler.cc' > --- sql/event_scheduler.cc 2012-09-27 23:06:56 +0000 > +++ sql/event_scheduler.cc 2013-01-15 21:00:33 +0000 > @@ -182,12 +180,15 @@ deinit_event_thread(THD *thd) > void > pre_init_event_thread(THD* thd) > { > + THD *orig_thd= current_thd; > DBUG_ENTER("pre_init_event_thread"); > + > + my_pthread_setspecific_ptr(THR_THD, thd); I'd rather see set_current_thd(thd) here: #define set_current_thd(X) my_pthread_setspecific_ptr(THR_THD, (X)) > thd->client_capabilities= 0; > thd->security_ctx->master_access= 0; > thd->security_ctx->db_access= 0; > thd->security_ctx->host_or_ip= (char*)my_localhost; > - my_net_init(&thd->net, NULL); > + my_net_init(&thd->net, NULL, MY_THREAD_SPECIFIC); > thd->security_ctx->set_user((char*)"event_scheduler"); > thd->net.read_timeout= slave_net_timeout; > thd->variables.option_bits|= OPTION_AUTO_IS_NULL; > === modified file 'sql/handler.cc' > --- sql/handler.cc 2012-12-17 00:49:19 +0000 > +++ sql/handler.cc 2013-01-15 21:00:33 +0000 > @@ -5395,6 +5395,7 @@ fl_log_iterator_buffer_init(struct handl > iterator->buffer= buff; > iterator->next= &fl_log_iterator_next; > iterator->destroy= &fl_log_iterator_destroy; > + my_dir_end(dirp); Why did we never see memory leaks because of that? > return HA_ITERATOR_OK; > } > > === modified file 'sql/mysqld.cc' > --- sql/mysqld.cc 2013-01-09 03:34:33 +0000 > +++ sql/mysqld.cc 2013-01-15 21:00:33 +0000 > @@ -2606,9 +2607,10 @@ bool one_thread_per_connection_end(THD * > > /* It's safe to broadcast outside a lock (COND... is not deleted here) */ > DBUG_PRINT("signal", ("Broadcasting COND_thread_count")); > + mysql_cond_broadcast(&COND_thread_count); > + > DBUG_LEAVE; // Must match DBUG_ENTER() > my_thread_end(); > - mysql_cond_broadcast(&COND_thread_count); why? > > pthread_exit(0); > return 0; // Avoid compiler warnings > @@ -7024,6 +7096,7 @@ SHOW_VAR status_vars[]= { > {"Key", (char*) &show_default_keycache, SHOW_FUNC}, > {"Last_query_cost", (char*) offsetof(STATUS_VAR, last_query_cost), SHOW_DOUBLE_STATUS}, > {"Max_used_connections", (char*) &max_used_connections, SHOW_LONG}, > + {"Memory_used", (char*) &total_memory_used, SHOW_LONGLONG}, May be this should rather be: SHOW SESSION STATUS -> thd->memory_used SHOW GLOBAL STATUS -> total_memory_used (which means that total_memory_used/memory_used should be in the status_var struct) > {"Not_flushed_delayed_rows", (char*) &delayed_rows_in_use, SHOW_LONG_NOFLUSH}, > {"Open_files", (char*) &my_file_opened, SHOW_LONG_NOFLUSH}, > {"Open_streams", (char*) &my_stream_opened, SHOW_LONG_NOFLUSH}, > === modified file 'sql/net_serv.cc' > --- sql/net_serv.cc 2012-02-28 17:53:05 +0000 > +++ sql/net_serv.cc 2013-01-15 21:00:33 +0000 > @@ -139,6 +140,7 @@ my_bool my_net_init(NET *net, Vio* vio) > net->net_skip_rest_factor= 0; > net->last_errno=0; > net->unused= 0; > + net->thread_specific_malloc= test(my_flags & MY_THREAD_SPECIFIC); why do you need that in NET ? > > if (vio != 0) /* If real connection */ > { > === modified file 'sql/sql_analyse.h' > --- sql/sql_analyse.h 2011-11-03 18:17:05 +0000 > +++ sql/sql_analyse.h 2013-01-15 21:00:33 +0000 > @@ -121,7 +121,8 @@ class field_str :public field_info > must_be_blob(0), was_zero_fill(0), > was_maybe_zerofill(0), can_be_still_num(1) > { init_tree(&tree, 0, 0, sizeof(String), (qsort_cmp2) sortcmp2, > - 0, (tree_element_free) free_string, NULL); }; > + 0, (tree_element_free) free_string, NULL, > + MY_THREAD_SPECIFIC); }; Now, when I'm thinking of it, I got the idea that it's better to have MY_THREAD_SPECIFIC as a default, and add MY_GLOBAL_ALLOC instead to mark global allocations explicitly. It's more future-proof. When one copies init_dynamic_array() from somewhere and zeros out unknown or unneeded parameters, the memory will be per-thread by default. And if that's wrong, the test suite will find it, you have an assert for this. To mark memory global as an explicit action, one needs to think about it. Currently, the memory is global by default, and if one doesn't think,# copy-pastes, and leaves new allocation global, it will not be noticed, the memory will simply be unaccounted for. > > void add(); > void get_opt_type(String*, ha_rows); > === modified file 'sql/sql_error.cc' > --- sql/sql_error.cc 2012-01-13 14:50:02 +0000 > +++ sql/sql_error.cc 2013-01-15 21:00:33 +0000 > @@ -457,25 +457,37 @@ Diagnostics_area::disable_status() > m_status= DA_DISABLED; > } > > -Warning_info::Warning_info(ulonglong warn_id_arg, bool allow_unlimited_warnings) > +Warning_info::Warning_info(ulonglong warn_id_arg, > + bool allow_unlimited_warnings, bool initialize) I'd rather add a special constructor for Warning_info that THD would use. Because there's only one place where you need initialize=false, and all other "normal" uses, should not suffer (be changed) because of that. > :m_statement_warn_count(0), > m_current_row_for_warning(1), > m_warn_id(warn_id_arg), > m_allow_unlimited_warnings(allow_unlimited_warnings), > m_read_only(FALSE) > { > - /* Initialize sub structures */ > - init_sql_alloc(&m_warn_root, WARN_ALLOC_BLOCK_SIZE, WARN_ALLOC_PREALLOC_SIZE); > m_warn_list.empty(); > bzero((char*) m_warn_count, sizeof(m_warn_count)); > + if (initialize) > + init(); > } > Regards, Sergei

2 2

Re: [Maria-developers] LP:1024058 - mysqld XA crash in replication slave
by Kristian Nielsen 18 Jan '13

18 Jan '13

"Sergei Golubchik (JIRA)" <jira(a)mariadb.atlassian.net> writes: > [ https://mariadb.atlassian.net/browse/MDEV-633?page=com.atlassian.jira.plugi… ] [Taking this to maria-developers@, which is better for discussions like this] > The problem here is very simple to explain. The server can use either > mmap-based transaction coordinator for 2PC or a binary log. 2PC always uses > binary log, if binary logging is enabled. But even if it is enabled > globally, it is usually disabled in the replication slave thread unless > --log-slave-updates is specified. > > One would probably get the same crash without replication, if one disables > binary log manually with SET SQL_LOG_BIN=0; > > Possible fixes: > * auto-enable binary log for 2PC transactions (bad, binlog will contain > unwanted DDL and DML events). > * abort 2PC transactions if binary log is locally disabled for this thread > (worse, too easy to break the replication) > * write Xid events to binlog even if binary log is locally disabled (best?) > > The last approach seems to be preferable. But in the future if we'll start > recovering transactions from the binary log (doing only one sync per 2PC > transaction), we'll have this problem again, because then we'll need the > actual changes to be logged, not just the Xid. > > [~knielsen] - opinion? First, I do not think SQL_LOG_BIN=0 or binlog enabled without --log-slave-updates are common or very useful. And multi-engine transactions are problematic in several ways. But we should not crash, of course. I would find it acceptable to simply call each engine's commit method separately, without 2PC, especially in stable 5.5 and below. Note that we now have @@skip_replication, which can be used to log a transaction to binlog without replicating it, similar to @@sql_log_bin but maintaining 2PC and recovery. If we really want, we could fix it 10.0 so that 2PC works even for binlog disabled. As you say, probably writing the Xid to the binlog without any associated DML events would be a reasonable way to do this. I would suggest logging a separate new event to not confuse things with XID_LOG_EVENT, which implies a commit of a prior transactions (there are facilities in 10.0 to rewrite such event to a dummy event so we will not break old slaves). With respect to a future recovering transactions from the binary log, it is actually in any case very hard to recover a multi-engine transaction in this way. The problem is when one engine is missing the transaction but the other engine has it, and the event is logged with statement-based replication. But we could simply not do this optimisation for multi-engine transactions, which are really not very commonly used (for multi-engine transactions we would keep the fsync() inside prepare()). Then things should work, for the normal case as well as for binlog locally disabled. So in summary, my immediate opinion would be: - In 5.5, commit each engine individually without 2PC when binlog is locally disabled. - In 10.0, optionally binlog just a new XID_RECOVERY_LOG_EVENT and use it in recovery if binlog is locally disabled. Hope this helps, - Kristian.

1 0

Re: [Maria-developers] Structure of Binary Log
by Kristian Nielsen 16 Jan '13

16 Jan '13

Nuno Rebocho <rebocho.nmg(a)portugalmail.pt> writes: > The attractive hypothesis, develop a data recovery tool, never left my > mind. I've been trying to learn together as a "Maria developers" team > can get information about the structure of the log file from MySql to > navigate this and obtain information needed for recovery. I am not aware of any definitive manual for the MySQL/MariaDB binary log, however there are various resources out there. For example, I wrote two articles that give an overview of the events in the log: http://kristiannielsen.livejournal.com/13382.html http://kristiannielsen.livejournal.com/13253.html My articles on group commit on the same blog would give you lots on information on how the binary log is written. Google can probably find other articles by other authors that might be useful. Otherwise, the source code in the definite source of information. The binary log code is in sql/log.cc. The code for creating and parsing events is in sql/log_event.cc. I noticed you are now on the maria-developers@ mailing list, this will be a good place to ask concrete questions about the binary log. Good luck with your project! - Kristian.

1 0

Re: [Maria-developers] 答复: in-order commit
by Kristian Nielsen 16 Jan '13

16 Jan '13

丁奇 <dingqi.lxb(a)taobao.com> writes: > Hi, Kristian > Ok. I have got the information from JIRA. > > I find you control the commit order inside the user thread. > > Will it be easier to let Trans_worker thread hold this logic? Yes, I think you are right. Of course, the user thread is the one that knows the ordering, but the logic for waiting needs to be in the Trans_worker thread. In fact this is a bug in my first patch: Transaction T3 could wait for the THD of worker thread 1 which has both T1 and T2 queued; then it will wake up too early, when T1 commits rather than when T2 does. I will try to implement the new idea today. > After they have done the execution of one transaction, "register the transaction and wait" if there are transactions from other workers should be commited ahead. > After commit in one worker, wake up another worker, the worker who wait for the next "head of commitee" should be woken up. Right, I'll need to look into this a bit deeper. Actually, in my patch the actual wait and wakeup happens inside ha_commit_trans(), and there is a reason for this. Because eventually I want to do it inside tc_log->log_and_order(), which is called from ha_commit_trans(). Here is how a commit happens: InnoDB prepare step fsync() InnoDB redo log (*A) TC_LOG_BINLOG::log_and_order Write transaction to binlog fsync() binlog (*B) InnoDB commit_ordered() (*C) Write commit record to InnoDB redo log InnoDB commit step The steps (*A) and (*B) are slow, typically around 1-10 milliseconds depending on disk system. So we need many threads to commit in parallel and reach points (*A) and (*B) at the same time, so we only need to do the fsync() once for many threads. This is group commit. Thus for in-order parallel replication, we must not do the wait for the previous commit before the (*B) step. Because if we do, then it becomes impossible for two transactions to be at point (*B) at the same time, and group commit is impossible. On the other hand, point (*C) is where the commit order is determined. So if we do the wait after point (*C), then we cannot enforce that T1 commits before T2. So therefore, the wait must happen exactly around point (B) and (C), inside TC_LOG_BINLOG::log_and_order(). That is why I invented all the register_wait_for_prior_commit() and so on: so that log_and_order() has somewhere to look for exactly who is waiting for who. Then if T2 is waiting for T1 to commit, we can do steps (*B) and (*C) for both of them together, achiving both group commit and in-order parallel replication. Anyway, I just wanted to mention this, I know it will be difficult to understand fully from just this description. This is something that I have been planning to have for years, but I still need to show some real code that actually works. If I manage that, hopefully things will be clearer. (If not - then I need to think again ;-) Thanks, - Kristian.

1 2

[Maria-developers] new timestamp fields
by Zardosht Kasheff 15 Jan '13

15 Jan '13

Hello all, In MySQL/MariaDB 5.1, when a timestamp field was used as a key, we used an integer compare to compare the fields. We assumed all integer fields (including date, time, and timestamp) could use an integer comparison to compare the fields, and that they were either 1, 2, 3, 4 or 8 bytes long. We now see in MariaDB 5.3 and beyond that changes have been made to the timestamp field, such that the length of the field may be something other than 1, 2, 3, 4,or 8 bytes. Given a timestamp field of N bytes, how are we meant to compare them? Are some meant to be integer comparisons? Are others supposed to be memory comparisons? Thanks -Zardosht

2 5

[Maria-developers] MDEV-520: consider parallel replication patch from taobao patches
by Kristian Nielsen 14 Jan '13

14 Jan '13

Hi xiaobin, I read through your patch found here: http://mysql.taobao.org/index.php/RLPR_for_MariaDB I am still a bit confused about the status of this patch. It would be useful if you could explain a bit about the status of the patch: - How have you tested the patch. - How are you currently using the patch? Or is it still just development/test? - What are your current plans for the patch? (You mentioned that you have some changes you are implementing at the moment). Anyway, I think I have a fair understanding of the overall idea, which seems very interesting! I will try to summarise here in a bit more detail, as well as give a few general comments on the patch. Please correct me if I misunderstood anything. ---- The basic idea is to build a set (implemented as a hash table) of every unique key in every row-event in each transaction received from the master. If two transactions have no overlapping keys, they can be applied in parallel (foreign_key_checks is set to OFF to make this work). There is a single thread that processes each event received in sequence. It checks the key set of the new transaction against the key sets of all transactions queued for each of N worker threads: - If there is no conflict with any already queued transaction, the new transaction can be queued to any worker thread (the one with fewest queued transaction is chosen). - If there is a conflict with just one thread, then the transaction can be queued for that thread - it will then execute in correct sequence with the conflicting transactions. - If there is a conflict with multiple threads, wait until the conflicting transactions have been processed. No new transactions are queued while waiting. Then there are N (default 16) threads that apply the queued transactions. Queued transactions are processsed in parallel within one thread, but transaction in different queues are processed in parallel with each other. If a transaction is received which is not using row events (statement-based replication or DDL), then we wait until all queued transactions are processed. Only after that is the new transaction queued, so that such events are processed correctly (but not in parallel, of course). Likewise, if the worker threads get too far behind, we wait with queuing any more transactions. ---- I think this is a very interesting approach, and I am happy that someone have actually tried implementing it. The biggest potential issue seems to be the performance of the thread that checks the transactions for conflicts and queues them for the appropriate thread: - Main issue will be huge transactions, if the size of hash tables of key values become too big for available memory. Maybe this could be solved by having some limit above which transactions are not tried in parallel. - There will be a lot of hash lookups of key values, all done in a single thread. This could become a limit I suppose - testing will be needed to understand this fully. But I think there will be many workloads where this will be useful. Overall, it seems to me this technique is very promising. It seems to be better at getting maximum parallelism on the slave than any of the other approaches that have been considered. As to the patch itself, it seems a bit immature in the present form, but I understood you are still working on it. Let us discuss what the plans could be for finishing it and get it ready for inclusion in MariaDB. The main criteria is that it must be reasonably complete - it must not crash or silently corrupt data in any usage. However, reasonable restrictions for what is supported is fine. (For example, we could say that LOAD DATA is not supported and give an error, but not crash or silently ignore such statement). But we can discuss this in more details. ---- Finally, a couple general comments on the code: 1. get_pk_value() does not seem to handle collations, eg. case insensitivity? It needs to understand that eg. "abab" and "ABAB" can conflict. I think this should be solved by copying in values with non-binary collations using the appropriate strxfrm() variant. 2. remote_query() retries the query in case of deadlock - but that seems wrrong, doesn't it need to retry the whole _transaction_, not just the query? 3. What about other events like User_var_event ? Will they be included in a later patch? 4. There is a my_sleep(10000) in the code when a transaction conflicts with two or more workers. It would be better to replace this with a condition variable so we can wake up as soon as the conflict is resolved. Same for some other sleeps like in wait_for_all_dml_done(). 5. Shouldn't there be some locking in dispatch_transaction()? To protect the hash lookups when checking conflicts against worker threads doing pk_hash_minus() ? 6. get_pk_value(). There seems to be a buffer overflow - st_hash_item::key is fixed size, but there seems no check against overrunning it. One solution is to just truncate if data does not fit - this can lead to false positives where potential parallelism is not exploited, but that is still safe and should be ok. Thanks! - Kristian.

5 21

Re: [Maria-developers] MDEV-318 IF (NOT) EXIST clauses for ALTER TABLE
by Sergei Golubchik 13 Jan '13

13 Jan '13

Hi, Holyfoot! On Jul 08, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-318/ > > ------------------------------------------------------------ > revno: 3552 > revision-id: holyfoot(a)askmonty.org-20120708152354-d7iu7yoqo3cohidz > parent: sanja(a)montyprogram.com-20120626184334-ptpg39ptq3t79dg5 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-318 > timestamp: Sun 2012-07-08 20:23:54 +0500 > message: > MDEV-318 IF (NOT) EXIST clauses for ALTER TABLE (MWL #252). > ALTER TABLE syntax modified and related implementations added to allow: > ALTER TABLE ADD/DROP COLUMN > ALTER TABLE ADD/DROP INDEX > ALTER TABLE ADD/DROP FOREIGN KEY > ALTER TABLE ADD/DROP PARTITION > ALTER TABLE CHANGE COLUMN > ALTER TABLE MODIFY COLUMN Please, find my comments below: > === modified file 'mysql-test/r/alter_table.result' > --- mysql-test/r/alter_table.result 2011-11-21 17:13:14 +0000 > +++ mysql-test/r/alter_table.result 2012-07-11 14:24:32 +0000 > @@ -1282,3 +1282,30 @@ > # Clean-up. > drop table t1; > End of 5.1 tests > +CREATE TABLE t1 ( > +id INT(11) NOT NULL, > +x_param INT(11) DEFAULT NULL, > +PRIMARY KEY (id), > +KEY x_param (x_param) > +); > +ALTER TABLE t1 ADD COLUMN IF NOT EXISTS id INT, > +ADD COLUMN IF NOT EXISTS lol INT AFTER id; > +Warnings: > +Note 1060 Duplicate column name 'id' > +ALTER TABLE t1 ADD COLUMN IF NOT EXISTS lol INT AFTER id; > +Warnings: > +Note 1060 Duplicate column name 'lol' > +ALTER TABLE t1 DROP COLUMN IF EXISTS lol; > +ALTER TABLE t1 DROP COLUMN IF EXISTS lol; > +Warnings: > +Note 1091 Can't DROP 'lol'; check that column/key exists > +ALTER TABLE t1 ADD KEY IF NOT EXISTS x_param(x_param); > +Warnings: > +Note 1061 Duplicate key name 'x_param' > +ALTER TABLE t1 ADD KEY IF NOT EXISTS x_param(x_param); typo? you test ADD KEY IF NOT EXISTS twice. > +Warnings: > +Note 1061 Duplicate key name 'x_param' > +ALTER TABLE t1 MODIFY IF EXISTS lol INT; > +Warnings: > +Note 1054 Unknown column 'lol' in 't1' > +DROP TABLE t1; > === modified file 'sql/field.cc' > --- sql/field.cc 2012-03-13 14:38:43 +0000 > +++ sql/field.cc 2012-07-11 14:28:40 +0000 > @@ -9182,6 +9182,7 @@ void Create_field::init_for_tmp_table(en First: please, see the last section in http://kb.askmonty.org/en/how-to-get-more-out-of-bzr-when-working-on-mariad… and do as it suggests I had to apply your patch locally and re-run bzr diff to get the function names in the diff :( > (maybe_null ? FIELDFLAG_MAYBE_NULL : 0) | > (is_unsigned ? 0 : FIELDFLAG_DECIMAL)); > vcol_info= 0; > + create_if_not_exists= FALSE; > stored_in_db= TRUE; > } > > === modified file 'sql/sql_lex.h' > --- sql/sql_lex.h 2011-12-11 16:39:33 +0000 > +++ sql/sql_lex.h 2012-07-11 14:28:40 +0000 > @@ -1816,6 +1816,7 @@ typedef struct st_lex : public Query_tab > uint16 create_view_algorithm; > uint8 create_view_check; > bool drop_if_exists, drop_temporary, local_file, one_shot_set; > + bool check_exists; /* Enabled if the IF [NOT] EXISTS specified for ALTER */ I think, you can remove drop_if_exists and use check_exists instead > bool autocommit; > bool verbose, no_write_to_binlog; > > === modified file 'sql/sql_yacc.yy' > --- sql/sql_yacc.yy 2012-06-10 09:53:06 +0000 > +++ sql/sql_yacc.yy 2012-07-11 14:28:40 +0000 > @@ -4656,8 +4659,16 @@ table_option: > ; > > opt_if_not_exists: > - /* empty */ { $$= 0; } > - | IF not EXISTS { $$=HA_LEX_CREATE_IF_NOT_EXISTS; } > + /* empty */ > + { > + Lex->check_exists= FALSE; Let's initialize check_exists either before the grammar is parsed (like in the add_create_index_prepare and in the create rule) of in the grammar (here, in the opt_if_not_exists). But don't do it twice. > + $$= 0; > + } > + | IF not EXISTS > + { > + Lex->check_exists= TRUE; > + $$=HA_LEX_CREATE_IF_NOT_EXISTS; > + } > ; > > opt_create_table_options: > @@ -5879,6 +5891,18 @@ opt_ident: > | field_ident { $$=$1.str; } > ; > > +opt_if_not_exists_ident: > + opt_if_not_exists opt_ident > + { > + LEX *lex= Lex; > + if (lex->check_exists && lex->sql_command != SQLCOM_ALTER_TABLE) > + { > + my_parse_error(ER(ER_SYNTAX_ERROR)); > + MYSQL_YYABORT; > + } > + $$= $2; > + }; > + why did you create opt_if_not_exists_ident rule, instead of adding opt_if_not_exists before ident in the alter table grammar? > opt_component: > /* empty */ { $$= null_lex_str; } > | '.' ident { $$= $2; } > @@ -6455,6 +6482,11 @@ opt_column: > | COLUMN_SYM {} > ; > > +opt_column_if_exists: > + opt_column if_exists > + {} > + ; > + same as above - why a special rule? > opt_ignore: > /* empty */ { Lex->ignore= 0;} > | IGNORE_SYM { Lex->ignore= 1;} > @@ -10096,8 +10128,16 @@ table_alias_ref: > ; > > if_exists: please, rename this old rule to opt_if_exists > - /* empty */ { $$= 0; } > - | IF EXISTS { $$= 1; } > + /* empty */ > + { > + Lex->check_exists= FALSE; > + $$= 0; > + } > + | IF EXISTS > + { > + Lex->check_exists= TRUE; > + $$= 1; > + } > ; > > opt_temporary: you didn't change CREATE INDEX and DROP INDEX, did you? and ALTER TABLE DROP PRIMARY KEY isn't covered either > === modified file 'sql/sql_table.cc' > --- sql/sql_table.cc 2012-04-05 21:07:18 +0000 > +++ sql/sql_table.cc 2012-07-11 14:28:40 +0000 > @@ -5788,6 +5788,226 @@ is_index_maintenance_unique (TABLE *tabl > > > /* > + Preparation for table creation > + > + SYNOPSIS > + handle_if_exists_option() Why did you do that in a separate function? You have basically duplicated all checks. normal alter table code checks for duplicate columns/key names and for non-existant columns/keys for DROP. this code is still in place after your patch. so, basically, you'll check for duplicates and non-existant names twice. what's the point of that? I'd expect you to take check_exists flag into account in the existing checking code and convert errors into warnings as appropriate. > + thd Thread object. > + table The altered table. > + alter_info List of columns and indexes to create > + > + DESCRIPTION > + Looks for the IF [NOT] EXISTS options, checks the states and remove items > + from the list if existing found. > + > + RETURN VALUES > + NONE > +*/ > + > +static void > +handle_if_exists_options(THD *thd, TABLE *table, Alter_info *alter_info) > +{ > + Field **f_ptr; Regards, Sergei

2 9

[Maria-developers] Order by speed optimization
by Matthew Lagoe 13 Jan '13

13 Jan '13

I was talking with a few people about a query such as select * from table order by X, Y, Z limit G; > > As the "ORDER BY" for me could change to any number of 12 values in any order I am not able to use a multicolumn index as such I was looking at how to optimize in such case where sorting is dynamic. On my dataset the query takes ~7 seconds for a full table scan but only .3 seconds when using the column index and returns ~80,000 records for the query (before the LIMIT) I was wondering if there is any reason that the order by syntax doesn't use the following logic, I was thinking of looking at the code but was told such a simple optimization must have a reason that its not already in the code, so I am asking here before I spend hours being stupid. This logic block will be based on the above query. 1. If index exists for column X and there is a limit on the query then use the index and sort the list, then return the first G values, continue past G until you hit a different value for X then the value that was at G 2. Use this smaller subset of data for then ordering the data based on Y 3. Order the data again based on Z 4. Return the sorted data like normal If there is a reason this optimization isn't in the code please someone let me know, if not I think I will take a look at adding it. -Botanic

2 2

[Maria-developers] Kubuntu 12.04.1 and MariaDB
by Giacomo Picchiarelli 11 Jan '13

11 Jan '13

Hi! I'm having troubles installing mariadb 10.0 in Kubuntu 12.04.1. When I try to install it, dependencies are getting enraged and apt (through synaptic or muon) suggests to remove kde-plasma-desktop. This happens when I try to install mariadb-server-core 10.0. Of course if I remove kde-plasma-desktop I can't use my kde desktop, because it depends on mysql packages. I was trying to install it for educational purpose, and try to build and make some tests. Do you have any suggestions? Thank you -- Giacomo Picchiarelli <gpicchiarelli(a)gmail.com>

3 3

[Maria-developers] External replication proceeding
by md＠rpzdesign.com 10 Jan '13

10 Jan '13

Monty: I received your email and am re-posting to the maria developers list. It is truly amazing that maria developers are HAPPY with replication as it stands right now. binary logs, Single SLAVE reading from individual masters, bla-bla-bla..... All the current replication code is like walking on the edge of a cliff, anything goes wrong and the replication process STOPS with an error. And a HUMAN BEING must wake up from their blissful sleep to get the process started again and figure out what happened. And I just LOVE having to: LOCK TABLES; RESET MASTER SET SLAVE OFF; MYSQLDUMP; COPY FILES TO SLAVE; SET MASTER TO 'sql_log_bin=45678945"; SET SLAVE ON; SECONDS BEHIND MASTER... WRITE A THESIS FOR MY PHD IN ROCKET SCIENCE Oh my GOD!!!!!! What an absolute nightmare of a design, it works, just not well. I have not made an in depth study of galera, but I refuse to make replication another research project, replication should be EASY, auto-healing, self analyzing, N-MASTER and with simple source code. I will open source the files that I use to change the MYSQLD daemon. The best way to debug MariaDB on Linux is to run the program in Netbeans 7.1.2. You can even give it command line parameters to work against a currently installed system with the data/conf files in place. Your trace idea is very nice, but there is nothing like STEPPING THROUGH the code in NetBeans/GDB in a nice IDE instead of cryptic command line GDB. Cheers, md On 1/2/2013 5:21 AM, Michael Widenius wrote: > Hi! > > A better place to send emails like this is to > maria-developers(a)lists.launchpad.net; That way you can get all > MariaDB developers to answer you questions. > >>>>>> "md" == md <md(a)rpzdesign.com> writes: > md> Hello: > md> Does anybody know the name(s) of the source files I could find the code > md> that processes the Table/Procedure Drop/Create/Update, Alter Table, etc > md> statements. > > md> I want to intercept these SQL statement calls in mysqld before > md> they get sent to the specific storage engine like Aria or Xtradb. > > sql_parse.cc proably contains the code you are looking for. > Another place to look is in the the current replication code, > ie the sql_repl.cc > > md> I am working on an asynchronous external replication daemon > md> that handles all of the data and structure synchronization > md> replication issues (N-master-master, Hot Insert Master additions, DDL > md> changes, etc) > > Sounds very intersting. > Is this a open source project ? > > md> This is one of the missing pieces of the puzzle for MariaDB. > md> (oh yeah, mysql does not exist to me anymore, only Aria/Xtradb) > > :) > > md> I need to have the mysqld daemon issue a signal that a structural change > md> has taken place and the external replication daemon will pick it up > md> and handle all the changes processing across ALL the masters (WITHOUT A > md> HUMAN BEING INVOLVED!!!!) > > md> I figured I would ask since most of you are fluent in the source > md> code and just know where everything is, you would know right away. > > That's ok, but best way to get quick responce is to ask on the > maria-developers list or at #maria channel on Freenode. > > md> I am using MariaDB 5.5.28a from source. > > On thing you can do is: > > ./BUILD/compile-pentium-debug-max > mysqld --debug > > And then start 'mysql' and execute a CREATE TABLE command. > > You can find then find a full trace of all executed functions in: > /tmp/mysqld.trace > > That's probably one of the best ways to learn how MariaDB works. > > See: > > https://kb.askmonty.org/en/creating-a-trace-file/ > > Regards, > Monty >

3 2

[Maria-developers] High Priest of Humility
by md＠rpzdesign.com 08 Jan '13

08 Jan '13

Arjen: I would not go so far as to call it "raving" or boasting about anything. It was a private email message to you only. Not broadcast publicly. For a MariaDB captain, that is pretty bold offering a negative opinion to a potential contributor. Negative comments like that are not the best support for any level of enthusiasm for openness, collaboration, and sharing. Hint: Stop giving hints. marco On 1/7/2013 5:05 PM, Arjen Lentz wrote: > Hi Marco > >> Nice to see your response. >> >> I will be providing the code that I use to modify the MariaDB core >> daemon soon. > I presume you'll put it up in a branch on Launchpad? > > >> You will be amazed how little I am changing the core code. > Hint: modesty/humility are a good traits for an OSS developer. Let other people be amazed and impressed when you deliver code - raving about yourself beforehand is less cool... > > >> Where is the open source for Galera? (could not find it readily) > At Codership, but also in the 5.5 Galera repository of MariaDB. > > > Regards, > Arjen. > >

1 0

[Maria-developers] Missing man/mysql_plugin.1 in tar ball
by Honza Horak 07 Jan '13

07 Jan '13

Hi all, I've noticed that man page for mysql_plugin utility (man/mysql_plugin.1) is not present in mariadb-5.5.28a.tar.gz while it is present in mysql-5.5.28.tar.gz. However, I failed to find the reason why in case there is one. Or is that a mistake? Does anybody know? Regards, Honza

2 1

Re: [Maria-developers] API Documentaion for Replication Features
by Lixun Peng 31 Dec '12

31 Dec '12

Hi Monty, On Tue, Dec 25, 2012 at 8:21 PM, Michael Widenius < michael.widenius(a)gmail.com> wrote: > > Hi! > > Sorry that I missed this email before. > <cut> > Lixun> I can try to implement it. Are you doing it? > > Please try to implemement it? > You can look at the keycache code how we have done this before! > > OK, I'm trying to do it. Thanks, Lixun -- Senior MySQL Developer @ Taobao.com Mobile Phone: +86 18658156856 (Hangzhou) Gtalk: penglixun(at)gmail.com Twitter: http://www.twitter.com/plinux Blog: http://www.penglixun.com

2 1

Re: [Maria-developers] Rev 3480: Post review changes (total (really))
by Sergei Golubchik 24 Dec '12

24 Dec '12

Hi, Oleksandr! On Dec 23, Oleksandr Byelkin wrote: > 22.12.2012 21:28, Sergei Golubchik пишет: > >> @@ -100,6 +101,8 @@ struct st_dynamic_column_value > >> > >> typedef struct st_dynamic_column_value DYNAMIC_COLUMN_VALUE; > >> > >> +/* old functions (deprecated) */ > > I usually prefer code to comments. Kind of a comment that a compiler > > can make use of. That means an assert(aaa == 0) instead of /* aaa > > should be 0 here */ and instead of comment above, I'd rather have > > __attribute__((deprecated)) > > __attribute__((deprecated)) is GCC, we also use MS compiler. So ifdef > is better until we get appropriate multi-platform macro. 1. __attribute__ is defined to be empty for other compilers (somewhere in my_global.h, I guess), so it won't break the compilation. 2. I did not mean to use it instead of the #ifdef, I mean to use it instead of a /* old functions (deprecated) */ comment. Because a compiler cannot read the comment. 3. And anyway, it was just a suggestion, you can keep your current code and not change anything. As you prefer. > everything else will be fixed. Thanks! Regards, Sergei

1 0

Re: [Maria-developers] Rev 3480: Post review changes (total (really))
by Sergei Golubchik 22 Dec '12

22 Dec '12

Hi, Sanja! See below. Just a couple of small API-related comments. And one JSON bug. On Dec 21, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-10.0-cassandra/ > > ------------------------------------------------------------ > revno: 3480 > revision-id: sanja(a)askmonty.org-20121221192211-93tc13j3a6c85qea > parent: sanja(a)montyprogram.com-20121119121604-5h5tu0zn11em0sb3 > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-cassandra > timestamp: Fri 2012-12-21 21:22:11 +0200 > message: > Post review changes (total (really)). > > === modified file 'include/ma_dyncol.h' > --- a/include/ma_dyncol.h 2012-09-28 12:27:16 +0000 > +++ b/include/ma_dyncol.h 2012-12-21 19:22:11 +0000 > @@ -38,12 +38,13 @@ > how the offset are stored. > */ > #define MAX_DYNAMIC_COLUMN_LENGTH 0X1FFFFFFFL > +#define MAX_DYNAMIC_COLUMN_LENGTH_NM 0XFFFFFFFFFL Hm. Two questions. 1. Where does the number come from? 2. You've introduced a constant, but you aren't using it anywhere. What's the point? > /* > Limits of implementation > */ > -#define MAX_NAME_LENGTH 255 > #define MAX_TOTAL_NAME_LENGTH 65535 > +#define MAX_NAME_LENGTH (MAX_TOTAL_NAME_LENGTH/4) > > /* NO and OK is the same used just to show semantics */ > #define ER_DYNCOL_NO ER_DYNCOL_OK > @@ -100,6 +101,8 @@ struct st_dynamic_column_value > > typedef struct st_dynamic_column_value DYNAMIC_COLUMN_VALUE; > > +/* old functions (deprecated) */ I usually prefer code to comments. Kind of a comment that a compiler can make use of. That means an assert(aaa == 0) instead of /* aaa should be 0 here */ and instead of comment above, I'd rather have __attribute__((deprecated)) > +#ifdef MADYNCOL_DEPRECATED > enum enum_dyncol_func_result > dynamic_column_create(DYNAMIC_COLUMN *str, > uint column_nr, DYNAMIC_COLUMN_VALUE *value); > @@ -133,73 +121,105 @@ dynamic_column_update_many(DYNAMIC_COLUM ... > enum enum_dyncol_func_result > -dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys); > +dynamic_column_get(DYNAMIC_COLUMN *org, uint column_nr, > + DYNAMIC_COLUMN_VALUE *store_it_here); > +#endif > > -/* List of not NULL columns */ > +/* new functions */ > enum enum_dyncol_func_result > -dynamic_column_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); > +mariadb_dyncol_create_many(DYNAMIC_COLUMN *str, > + uint column_count, > + uint *column_numbers, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool new_string); > enum enum_dyncol_func_result > -dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr); > +mariadb_dyncol_create_many_named(DYNAMIC_COLUMN *str, > + uint column_count, > + LEX_STRING *column_keys, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool new_string); > + > + > enum enum_dyncol_func_result > -dynamic_column_list_fmt(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array, my_bool string_keys); > +mariadb_dyncol_update_many(DYNAMIC_COLUMN *str, > + uint add_column_count, > + uint *column_keys, > + DYNAMIC_COLUMN_VALUE *values); > +enum enum_dyncol_func_result > +mariadb_dyncol_update_many_named(DYNAMIC_COLUMN *str, > + uint add_column_count, > + LEX_STRING *column_keys, > + DYNAMIC_COLUMN_VALUE *values); > + > + > +enum enum_dyncol_func_result > +mariadb_dyncol_exists(DYNAMIC_COLUMN *org, uint column_nr); > +enum enum_dyncol_func_result > +mariadb_dyncol_exists_named(DYNAMIC_COLUMN *str, LEX_STRING *name); > + > +/* List of not NULL columns */ > +enum enum_dyncol_func_result > +mariadb_dyncol_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); Sanja, let's be constistent. If mariadb_dyncol_list_named() returns an array of LEX_STRING's, than mariadb_dyncol_list should return an array of integers, not a dynamic array. Besides you don't use DYNAMIC_ARRAY *anywhere* else in the new (not deprecated) API. > +enum enum_dyncol_func_result > +mariadb_dyncol_list_named(DYNAMIC_COLUMN *str, uint *count, LEX_STRING **names); > > /* > if the column do not exists it is NULL > */ > enum enum_dyncol_func_result > -dynamic_column_get(DYNAMIC_COLUMN *org, uint column_nr, > +mariadb_dyncol_get(DYNAMIC_COLUMN *org, uint column_nr, > DYNAMIC_COLUMN_VALUE *store_it_here); > enum enum_dyncol_func_result > -dynamic_column_get_str(DYNAMIC_COLUMN *str, LEX_STRING *name, > - DYNAMIC_COLUMN_VALUE *store_it_here); > +mariadb_dyncol_get_named(DYNAMIC_COLUMN *str, LEX_STRING *name, > + DYNAMIC_COLUMN_VALUE *store_it_here); > > -my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str); > +my_bool mariadb_dyncol_has_names(DYNAMIC_COLUMN *str); > > enum enum_dyncol_func_result > -dynamic_column_check(DYNAMIC_COLUMN *str); > +mariadb_dyncol_check(DYNAMIC_COLUMN *str); > > enum enum_dyncol_func_result > -dynamic_column_json(DYNAMIC_COLUMN *str, DYNAMIC_STRING *json); > +mariadb_dyncol_json(DYNAMIC_COLUMN *str, DYNAMIC_STRING *json); > > #define dynamic_column_initialize(A) memset((A), 0, sizeof(*(A))) > #define dynamic_column_column_free(V) dynstr_free(V) > > /* conversion of values to 3 base types */ > enum enum_dyncol_func_result > -dynamic_column_val_str(DYNAMIC_STRING *str, DYNAMIC_COLUMN_VALUE *val, > +mariadb_dyncol_val_str(DYNAMIC_STRING *str, DYNAMIC_COLUMN_VALUE *val, > CHARSET_INFO *cs, my_bool quote); > enum enum_dyncol_func_result > -dynamic_column_val_long(longlong *ll, DYNAMIC_COLUMN_VALUE *val); > +mariadb_dyncol_val_long(longlong *ll, DYNAMIC_COLUMN_VALUE *val); > +enum enum_dyncol_func_result > +mariadb_dyncol_val_double(double *dbl, DYNAMIC_COLUMN_VALUE *val); > + > + > enum enum_dyncol_func_result > -dynamic_column_val_double(double *dbl, DYNAMIC_COLUMN_VALUE *val); > +mariadb_dyncol_unpack(DYNAMIC_COLUMN *str, > + uint *count, > + LEX_STRING **names, DYNAMIC_COLUMN_VALUE **vals); > > +int mariadb_dyncol_column_cmp_named(const LEX_STRING *s1, const LEX_STRING *s2); > enum enum_dyncol_func_result > -dynamic_column_vals(DYNAMIC_COLUMN *str, > - DYNAMIC_ARRAY *names, DYNAMIC_ARRAY *vals, > - char **free_names); > +mariadb_dyncol_column_count(DYNAMIC_COLUMN *str, uint *column_count); > > /*************************************************************************** > Internal functions, don't use if you don't know what you are doing... > ***************************************************************************/ > > -#define dynamic_column_reassociate(V,P,L, A) dynstr_reassociate((V),(P),(L),(A)) > +#define mariadb_dyncol_reassociate(V,P,L, A) dynstr_reassociate((V),(P),(L),(A)) I'd remove it. You can use dynstr_reassociate() directly in Item_func_dyncol_create::val_str(), if you need it, but don't encourage others to do it, by pussing this macro here. > -#define dynamic_column_value_init(V) (V)->type= DYN_COL_NULL > +#define dyncol_value_init(V) (V)->type= DYN_COL_NULL 1. you forgot the mariadb_ prefix 2. This macro seems to be unused. Perhaps you can simply delete it? > === modified file 'mysql-test/r/dyncol.result' > --- a/mysql-test/r/dyncol.result 2012-09-28 11:01:17 +0000 > +++ b/mysql-test/r/dyncol.result 2012-12-21 19:22:11 +0000 > @@ -1577,3 +1587,53 @@ COLUMN_CHECK('') > SELECT COLUMN_CHECK(NULL); > COLUMN_CHECK(NULL) > NULL > +# > +# escaping check > +# > +select column_json(column_create("string", "'\"/\\`.,whatever")),hex(column_create("string", "'\"/\\`.,whatever")); > +column_json(column_create("string", "'\"/\\`.,whatever")) hex(column_create("string", "'\"/\\`.,whatever")) > +[{"string":"'\"/\\`.,whatever"}] 040100060000000300737472696E670827222F5C602E2C7768617465766572 Please, add a test with two values. What it will look like: [{"a":1},{"b":b}] ? or { "a" : 1, "b" : 2 } ? I don't see a reason why it should be an array of one-value objects, instead of one object, with many values. But perhaps I'm missing something, why did you prefer an array? > +# > +# embedding test > +# > +select column_json(column_create("val", "val", "emb", column_create("val2", "val2"))); > +column_json(column_create("val", "val", "emb", column_create("val2", "val2"))) > +[{"emb":[{"val2":"val2"}],{"val":"val"}] ouch. invalid json, curly braces aren't properly paired. > +select column_json(column_create(1, "val", 2, column_create(3, "val2"))); > +column_json(column_create(1, "val", 2, column_create(3, "val2"))) > +[{"1":"val"},{"2":[{"3":"val2"}]] same here. > === modified file 'sql/item_func.h' > --- a/sql/item_func.h 2012-09-27 23:06:56 +0000 > +++ b/sql/item_func.h 2012-12-21 19:22:11 +0000 > @@ -58,7 +58,7 @@ public: > NOW_FUNC, TRIG_COND_FUNC, > SUSERVAR_FUNC, GUSERVAR_FUNC, COLLATE_FUNC, > EXTRACT_FUNC, CHAR_TYPECAST_FUNC, FUNC_SP, UDF_FUNC, > - NEG_FUNC, GSYSVAR_FUNC }; > + NEG_FUNC, GSYSVAR_FUNC, DYNCOL }; DYNCOL_FUNC > enum optimize_type { OPTIMIZE_NONE,OPTIMIZE_KEY,OPTIMIZE_OP, OPTIMIZE_NULL, > OPTIMIZE_EQUAL }; > enum Type type() const { return FUNC_ITEM; } Regards, Sergei

1 0

[Maria-developers] MariaDB Galera 5.5.28a Release Prep
by Daniel Bartholomew 21 Dec '12

21 Dec '12

All, looks like my first message on this never reached the list, so sending again. I've prepped the release notes and changelog pages for this release and will be activating it in a few hours (barring any show stoppers). - https://kb.askmonty.org/en/mariadb-galera-5528a-release-notes/ - https://kb.askmonty.org/en/mariadb-galera-5528a-changelog/ I've updated the "What is MariaDB Galera Cluster?" and "Getting Started..." pages. If other updates are needed, let me know. https://kb.askmonty.org/en/what-is-mariadb-galera-cluster/ https://kb.askmonty.org/en/getting-started-with-mariadb-galera-cluster/ Also please let me know if updates are needed on the "About" and "Known Limitations" pages https://kb.askmonty.org/en/about-galera-replication/ https://kb.askmonty.org/en/mariadb-galera-cluster-known-limitations/ Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

Re: [Maria-developers] [Commits] Rev 3484: postreview fixes (part 5)
by Sergei Golubchik 21 Dec '12

21 Dec '12

Hi, Sanja! On Dec 21, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-10.0-cassandra/ > ------------------------------------------------------------ > revno: 3484 > revision-id: sanja(a)askmonty.org-20121221010609-cnr2z3fg99kpxzma > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-cassandra > timestamp: Fri 2012-12-21 03:06:09 +0200 > message: > postreview fixes (part 5) > === modified file 'include/ma_dyncol.h' > --- a/include/ma_dyncol.h 2012-12-20 23:35:38 +0000 > +++ b/include/ma_dyncol.h 2012-12-21 01:06:09 +0000 > @@ -169,7 +169,8 @@ madyncol_exists_named(DYNAMIC_COLUMN *st > enum enum_dyncol_func_result > madyncol_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); > enum enum_dyncol_func_result > -madyncol_list_named(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr); > +madyncol_list_named(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr, > + void **memory_to_free); Same here. You can put this memory in the dynarray and don't burden the caller with these nuisanses. Or you can use a normal array, not a dynarray. > /* > if the column do not exists it is NULL > > === modified file 'mysql-test/r/dyncol.result' > --- a/mysql-test/r/dyncol.result 2012-12-20 23:48:20 +0000 > +++ b/mysql-test/r/dyncol.result 2012-12-21 01:06:09 +0000 > @@ -1587,3 +1587,7 @@ COLUMN_CHECK('') > SELECT COLUMN_CHECK(NULL); > COLUMN_CHECK(NULL) > NULL > +# quating check "quoting" But in fact, it's not quoting, it's escaping. > +select column_json(column_create("string", "'\"/\\`.,whatever")),hex(column_create("string", "'\"/\\`.,whatever")); > +column_json(column_create("string", "'\"/\\`.,whatever")) hex(column_create("string", "'\"/\\`.,whatever")) > +[{"string":"'\"/\\`.,whatever"}] 040100060000000300737472696E670827222F5C602E2C7768617465766572 > Regards, Sergei

1 0

[Maria-developers] Odd Error in Logs
by Gordan Bobic 21 Dec '12

21 Dec '12

Hi, I recently switched from MySQL 5.6.5 to MariaDB 10.0, and I've been seeing errors like the following in the logs since: 121221 13:22:34 [ERROR] Master 'master1': Found index PRIMARY whose column info does not match that of MySQL. 121221 13:22:34 [ERROR] Master 'master1': Build InnoDB index translation table for Table ./data/aggregate#P#p201006 failed I didn't dump/restore the data during the upgrade. What exactly is the cause of this? Is it dangerous? Everything seems to work fine, I just want to make sure that something subtle isn't going to cause silent data corruption. TIA. Gordan

1 0

Re: [Maria-developers] [Commits] Rev 3482: post review changes - interface (part 3)
by Sergei Golubchik 21 Dec '12

21 Dec '12

Hi, Sanja! On Dec 21, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-10.0-cassandra/ > > ------------------------------------------------------------ > revno: 3482 > revision-id: sanja(a)askmonty.org-20121220233538-b171yhb9isogvd5h > parent: sanja(a)askmonty.org-20121220204008-fox73otsp1lviqda > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-cassandra > timestamp: Fri 2012-12-21 01:35:38 +0200 > message: > post review changes - interface (part 3) > === modified file 'include/ma_dyncol.h' > --- a/include/ma_dyncol.h 2012-12-20 20:40:08 +0000 > +++ b/include/ma_dyncol.h 2012-12-20 23:35:38 +0000 > @@ -100,6 +100,8 @@ struct st_dynamic_column_value > > typedef struct st_dynamic_column_value DYNAMIC_COLUMN_VALUE; > > +/* old functions (depricated) */ > +#ifdef MADYNCOL_DEPRICATED "deprEcated" > enum enum_dyncol_func_result > dynamic_column_create(DYNAMIC_COLUMN *str, > uint column_nr, DYNAMIC_COLUMN_VALUE *value); > @@ -133,75 +120,105 @@ dynamic_column_update_many(DYNAMIC_COLUM > uint *column_numbers, > DYNAMIC_COLUMN_VALUE *values); > enum enum_dyncol_func_result > -dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str, > - uint add_column_count, > - void *column_keys, > - DYNAMIC_COLUMN_VALUE *values, > - my_bool string_keys); > - > -enum enum_dyncol_func_result > dynamic_column_delete(DYNAMIC_COLUMN *org, uint column_nr); > > enum enum_dyncol_func_result > dynamic_column_exists(DYNAMIC_COLUMN *org, uint column_nr); > + > enum enum_dyncol_func_result > -dynamic_column_exists_named(DYNAMIC_COLUMN *str, LEX_STRING *name); > +dynamic_column_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); > + > enum enum_dyncol_func_result > -dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys); > +dynamic_column_get(DYNAMIC_COLUMN *org, uint column_nr, > + DYNAMIC_COLUMN_VALUE *store_it_here); > +#endif > > -/* List of not NULL columns */ > +/* new functions */ > enum enum_dyncol_func_result > -dynamic_column_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); > +madyncol_create_many(DYNAMIC_COLUMN *str, "madyncol" looks silly :) let's agree on one namespace prefix and use it everywhere. either it's ma_ like in ma_open ma_dyncol_create_many ma_alloc or it's maria_ like in maria_open maria_dyncol_create_many maria_alloc > + uint column_count, > + uint *column_numbers, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool new_string); Regards, Sergei

2 1

Re: [Maria-developers] review of the MDEV-377 Name support for dynamic columns
by Sergei Golubchik 21 Dec '12

21 Dec '12

Hi, Oleksandr! On Dec 21, Oleksandr Byelkin wrote: > It looks like you reviewed some old sources in 10.0-cassandra this is > fixed as we discussed it. I've reviewed the 5.5-cassandra tree. And yes, I thought that was already fixed, so I was a bit susprised to find it unfixed :) Ok, wrong tree, then. Thanks! > [skip] > > > > - > > -int dynamic_column_update(DYNAMIC_COLUMN *str, uint column_nr, > > - DYNAMIC_COLUMN_VALUE *value) > > +enum enum_dyncol_func_result > > +dynamic_column_vals(DYNAMIC_COLUMN *str, > > + DYNAMIC_ARRAY *names, DYNAMIC_ARRAY *vals, > > + char **free_names) > > { > > - return dynamic_column_update_many(str, 1, &column_nr, value); > > + DYN_HEADER header; > > + char *nm; > > + uint i; > > + enum enum_dyncol_func_result rc; > > + > > + *free_names= 0; > > + bzero(names, sizeof(DYNAMIC_ARRAY)); /* In case of errors */ > > + bzero(vals, sizeof(DYNAMIC_ARRAY)); /* In case of errors */ > > + if (str->length == 0) > > + return ER_DYNCOL_OK; /* no columns */ > > + > > + if ((rc= init_read_hdr(&header, str)) < 0) > > + return rc; > > + > > + if (header.entry_size * header.column_count + FIXED_HEADER_SIZE > > > + str->length) > > + return ER_DYNCOL_FORMAT; > > + > > + if (init_dynamic_array(names, sizeof(LEX_STRING), > > + header.column_count, 0) || > > + init_dynamic_array(vals, sizeof(DYNAMIC_COLUMN_VALUE), > > + header.column_count, 0) || > > + (header.format == DYNCOL_FMT_NUM && > > + !(*free_names= (char *)malloc(DYNCOL_NUM_CHAR * header.column_count)))) > > why do you need a special malloc() for names, you can put them in > > the buffer of 'names' dynarray. it'll be a lot more convenient for > > the caller (no free_names to care about). > dynamic array also should be freed, so I do not see difference, I can > rename it to memory_to_free or memory_should_be_freed. The difference is - you pass a separate pointer to the caller, that a caller should take care of. If you put this memory in dynarray, than the caller doesn't have to bother. He eventually delete_dynamic() and that does all the necessary cleanup. It's easier to use for the caller - and for API it's extremely important. More important than simple implementation. Regards, Sergei

1 0

[Maria-developers] MDEV-3792: review of the handler part of the cassandra engine
by Sergei Golubchik 20 Dec '12

20 Dec '12

Hi, Sergey, The handler is quite ok. I've only had few mostly cosmetical comments. > === modified file 'cmake/build_configurations/mysql_release.cmake' > --- cmake/build_configurations/mysql_release.cmake 2012-06-06 12:15:29 +0000 > +++ cmake/build_configurations/mysql_release.cmake 2012-11-12 15:11:23 +0000 > @@ -46,6 +46,8 @@ SET(FEATURE_SET_large 5) > SET(FEATURE_SET_xlarge 6) > SET(FEATURE_SET_community 7) > > +SET(WITH_CASSANDRA_STORAGE_ENGINE ON) > + This probably should be reverted before pushing into main, right? > IF(FEATURE_SET) > STRING(TOLOWER ${FEATURE_SET} feature_set) > SET(num ${FEATURE_SET_${feature_set}}) > > === modified file 'sql/sql_join_cache.cc' > --- sql/sql_join_cache.cc 2012-03-24 17:21:22 +0000 > +++ sql/sql_join_cache.cc 2012-11-12 15:11:23 +0000 > @@ -4543,7 +4546,7 @@ bool JOIN_CACHE_BKAH::prepare_look_for_m > { > last_matching_rec_ref_ptr= next_matching_rec_ref_ptr= 0; > if (no_association && > - (curr_matching_chain= get_matching_chain_by_join_key())) > + !(curr_matching_chain= get_matching_chain_by_join_key())) //psergey: added '!' Is that something that should be pushed into the main branch or it's a temporary hack for cassandra, that should be reverted? > return 1; > last_matching_rec_ref_ptr= get_next_rec_ref(curr_matching_chain); > return 0; > > === added file 'storage/cassandra/CMakeLists.txt' > --- storage/cassandra/CMakeLists.txt 1970-01-01 00:00:00 +0000 > +++ storage/cassandra/CMakeLists.txt 2012-11-12 15:11:23 +0000 > @@ -0,0 +1,25 @@ > + > +SET(cassandra_sources > + ha_cassandra.cc > + ha_cassandra.h > + cassandra_se.h > + cassandra_se.cc > + gen-cpp/Cassandra.cpp > + gen-cpp/cassandra_types.h > + gen-cpp/cassandra_types.cpp > + gen-cpp/cassandra_constants.h > + gen-cpp/cassandra_constants.cpp > + gen-cpp/Cassandra.h) > + > +#INCLUDE_DIRECTORIES(BEFORE ${Boost_INCLUDE_DIRS}) > + > +#INCLUDE_DIRECTORIES(AFTER /usr/local/include/thrift) > +INCLUDE_DIRECTORIES(AFTER /home/buildbot/build/thrift-inst/include/thrift/) this needs to be fixed before pushing into the main tree I suppose, you'll need to detect here if thrift is available and disable the engine otherwise > + > +# > +STRING(REPLACE "-fno-exceptions" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS}) > +STRING(REPLACE "-fno-implicit-templates" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS}) > +#LINK_DIRECTORIES(/home/psergey/cassandra/thrift/lib) > + > +MYSQL_ADD_PLUGIN(cassandra ${cassandra_sources} STORAGE_ENGINE LINK_LIBRARIES thrift) > +# was: STORAGE_ENGINE MANDATORY > > === added file 'storage/cassandra/cassandra_se.cc' > --- storage/cassandra/cassandra_se.cc 1970-01-01 00:00:00 +0000 > +++ storage/cassandra/cassandra_se.cc 2012-11-12 15:11:23 +0000 > @@ -0,0 +1,809 @@ > + > +// Cassandra includes: > +#include <inttypes.h> > +#include <netinet/in.h> > +#include <sys/time.h> > +#include <stdio.h> > +#include <stdarg.h> > + > +#include "Thrift.h" > +#include "transport/TSocket.h" > +#include "transport/TTransport.h" > +#include "transport/TBufferTransports.h" > +#include "protocol/TProtocol.h" > +#include "protocol/TBinaryProtocol.h" > +#include "gen-cpp/Cassandra.h" > +// cassandra includes end > + > +#include "cassandra_se.h" > + > +struct st_mysql_lex_string > +{ > + char *str; > + size_t length; > +}; > + > +using namespace std; > +using namespace apache::thrift; > +using namespace apache::thrift::transport; > +using namespace apache::thrift::protocol; > +using namespace org::apache::cassandra; > + > + > +void Cassandra_se_interface::print_error(const char *format, ...) > +{ > + va_list ap; > + va_start(ap, format); > + // it's not a problem if output was truncated > + vsnprintf(err_buffer, sizeof(err_buffer), format, ap); my_vsnprintf please > + va_end(ap); > +} > + ... > === added file 'storage/cassandra/ha_cassandra.h' > --- storage/cassandra/ha_cassandra.h 1970-01-01 00:00:00 +0000 > +++ storage/cassandra/ha_cassandra.h 2012-11-12 15:11:23 +0000 > @@ -0,0 +1,328 @@ > +/* > + Copyright (c) 2012, Monty Program Ab > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ > +#ifdef USE_PRAGMA_INTERFACE > +#pragma interface /* gcc class implementation */ > +#endif > + > + > +#include "my_global.h" /* ulonglong */ > +#include "thr_lock.h" /* THR_LOCK, THR_LOCK_DATA */ > +#include "handler.h" /* handler */ > +#include "my_base.h" /* ha_rows */ > + > +#include "cassandra_se.h" > + > +/** @brief > + CASSANDRA_SHARE is a structure that will be shared among all open handlers. > + This example implements the minimum of what you will probably need. > +*/ > +typedef struct st_cassandra_share { > + char *table_name; > + uint table_name_length,use_count; > + mysql_mutex_t mutex; > + THR_LOCK lock; > +} CASSANDRA_SHARE; > + > +class ColumnDataConverter; > + > +struct ha_table_option_struct; > + > + > +struct st_dynamic_column_value; > + > +typedef bool (* CAS2DYN_CONVERTER)(const char *cass_data, > + int cass_data_len, > + struct st_dynamic_column_value *value); > +typedef bool (* DYN2CAS_CONVERTER)(struct st_dynamic_column_value *value, > + char **cass_data, > + int *cass_data_len, > + void *buf, void **freemem); > +struct cassandra_type_def > +{ > + const char *name; > + CAS2DYN_CONVERTER cassandra_to_dynamic; > + DYN2CAS_CONVERTER dynamic_to_cassandra; > +}; > + > +typedef struct cassandra_type_def CASSANDRA_TYPE_DEF; > + > +enum cassandtra_type_enum {CT_BIGINT, CT_INT, CT_COUNTER, CT_FLOAT, CT_DOUBLE, > + CT_BLOB, CT_ASCII, CT_TEXT, CT_TIMESTAMP, CT_UUID, CT_BOOLEAN, CT_VARINT, > + CT_DECIMAL}; > + > +typedef enum cassandtra_type_enum CASSANDRA_TYPE; > + > + > + > +/** @brief > + Class definition for the storage engine > +*/ > +class ha_cassandra: public handler > +{ > + friend class Column_name_enumerator_impl; > + THR_LOCK_DATA lock; ///< MySQL lock > + CASSANDRA_SHARE *share; ///< Shared lock info > + > + Cassandra_se_interface *se; > + > + /* description of static part of the table definition */ > + ColumnDataConverter **field_converters; > + uint n_field_converters; > + > + CASSANDRA_TYPE_DEF *default_type_def; > + /* description of dynamic columns part */ > + CASSANDRA_TYPE_DEF *special_type_field_converters; > + LEX_STRING *special_type_field_names; > + uint n_special_type_fields; > + DYNAMIC_ARRAY dynamic_values, dynamic_names; > + DYNAMIC_STRING dynamic_rec; > + > + ColumnDataConverter *rowkey_converter; > + > + bool setup_field_converters(Field **field, uint n_fields); > + void free_field_converters(); > + > + int read_cassandra_columns(bool unpack_pk); > + int check_table_options(struct ha_table_option_struct* options); > + > + bool doing_insert_batch; > + ha_rows insert_rows_batched; > + > + uint dyncol_field; > + bool dyncol_set; > + > + /* Used to produce 'wrong column %s at row %lu' warnings */ > + ha_rows insert_lineno; > + void print_conversion_error(const char *field_name, > + char *cass_value, int cass_value_len); > + int connect_and_check_options(TABLE *table_arg); > +public: > + ha_cassandra(handlerton *hton, TABLE_SHARE *table_arg); > + ~ha_cassandra() > + { > + free_field_converters(); > + delete se; > + } > + > + /** @brief > + The name that will be used for display purposes. > + */ > + const char *table_type() const { return "CASSANDRA"; } > + > + /** @brief > + The name of the index type that will be used for display. > + Don't implement this method unless you really have indexes. > + */ > + const char *index_type(uint inx) { return "HASH"; } > + > + /** @brief > + The file extensions. > + */ > + const char **bas_ext() const; > + > + /** @brief > + This is a list of flags that indicate what functionality the storage engine > + implements. The current table flags are documented in handler.h > + */ > + ulonglong table_flags() const > + { > + /* > + HA_BINLOG_STMT_CAPABLE > + We are saying that this engine is just statement capable to have > + an engine that can only handle statement-based logging. This is > + used in testing. > + HA_REC_NOT_IN_SEQ > + If we don't set it, filesort crashes, because it assumes rowids are > + 1..8 byte numbers > + */ > + return HA_BINLOG_STMT_CAPABLE | > + HA_REC_NOT_IN_SEQ; HA_NO_TRANSACTIONS is unset. Is this engine transactional? HA_PARTIAL_COLUMN_READ is unset. Do you always work with all columns, ignoring read_set and write_set? HA_TABLE_SCAN_ON_INDEX is unset. Do you store the data MyISAM-style, index and data in separate files? HA_FAST_KEY_READ is unset. is reading keys in order faster in cassandra, than random reads? HA_REQUIRE_PRIMARY_KEY is unset. but a primary key is required. HA_PRIMARY_KEY_IN_READ_INDEX is unset. but as far as I see, primary key columns are always returned (and needed for position()) HA_PRIMARY_KEY_REQUIRED_FOR_POSITION is unset, but a primary key is required for position() HA_NO_AUTO_INCREMENT is unset. is auto-increment supported? > + > + } > + > + /** @brief > + This is a bitmap of flags that indicates how the storage engine > + implements indexes. The current index flags are documented in > + handler.h. If you do not implement indexes, just return zero here. > + > + @details > + part is the key part to check. First key part is 0. > + If all_parts is set, MySQL wants to know the flags for the combined > + index, up to and including 'part'. > + */ > + ulong index_flags(uint inx, uint part, bool all_parts) const > + { > + return 0; > + } > + > + /** @brief > + unireg.cc will call max_supported_record_length(), max_supported_keys(), > + max_supported_key_parts(), uint max_supported_key_length() > + to make sure that the storage engine can handle the data it is about to > + send. Return *real* limits of your storage engine here; MySQL will do > + min(your_limits, MySQL_limits) automatically. > + */ > + uint max_supported_record_length() const { return HA_MAX_REC_LENGTH; } > + > + /* Support only one Primary Key, for now */ > + uint max_supported_keys() const { return 1; } > + uint max_supported_key_parts() const { return 1; } > + > + /** @brief > + unireg.cc will call this to make sure that the storage engine can handle > + the data it is about to send. Return *real* limits of your storage engine > + here; MySQL will do min(your_limits, MySQL_limits) automatically. > + > + @details > + There is no need to implement ..._key_... methods if your engine doesn't > + support indexes. > + */ > + uint max_supported_key_length() const { return 16*1024; /* just to return something*/ } > + > + int index_init(uint idx, bool sorted); > + > + int index_read_map(uchar * buf, const uchar * key, > + key_part_map keypart_map, > + enum ha_rkey_function find_flag); > + > + /** @brief > + Called in test_quick_select to determine if indexes should be used. > + */ > + virtual double scan_time() { return (double) (stats.records+stats.deleted) / 20.0+10; } does that make any sense? (I understand that it's copy-paste) stats.deleted, for example, is never set. > + > + /** @brief > + This method will never be called if you do not implement indexes. > + */ > + virtual double read_time(uint, uint, ha_rows rows) > + { return (double) rows / 20.0+1; } same question > + > + virtual void start_bulk_insert(ha_rows rows); > + virtual int end_bulk_insert(); > + > + virtual int reset(); > + > + > + int multi_range_read_init(RANGE_SEQ_IF *seq, void *seq_init_param, > + uint n_ranges, uint mode, HANDLER_BUFFER *buf); > + int multi_range_read_next(range_id_t *range_info); > + ha_rows multi_range_read_info_const(uint keyno, RANGE_SEQ_IF *seq, > + void *seq_init_param, > + uint n_ranges, uint *bufsz, > + uint *flags, COST_VECT *cost); > + ha_rows multi_range_read_info(uint keyno, uint n_ranges, uint keys, > + uint key_parts, uint *bufsz, > + uint *flags, COST_VECT *cost); > + int multi_range_read_explain_info(uint mrr_mode, char *str, size_t size); > + > +private: > + bool source_exhausted; > + bool mrr_start_read(); > + int check_field_options(Field **fields); > + int read_dyncol(DYNAMIC_ARRAY *vals, DYNAMIC_ARRAY *names, > + String *valcol, char **freenames); > + int write_dynamic_row(DYNAMIC_ARRAY *names, DYNAMIC_ARRAY *vals); > + void static free_dynamic_row(DYNAMIC_ARRAY *vals, DYNAMIC_ARRAY *names, > + char *free_names); > + CASSANDRA_TYPE_DEF * get_cassandra_field_def(char *cass_name, > + int cass_name_length); > +public: > + > + /* > + Everything below are methods that we implement in ha_example.cc. may be it'd be better to remove the references to ha_example.cc (and example in general) before pushing this into the main tree? > + > + Most of these methods are not obligatory, skip them and > + MySQL will treat them as not implemented > + */ > + /** @brief > + We implement this in ha_example.cc; it's a required method. > + */ > + int open(const char *name, int mode, uint test_if_locked); // required > + > + /** @brief > + We implement this in ha_example.cc; it's a required method. > + */ > + int close(void); // required > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int write_row(uchar *buf); > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int update_row(const uchar *old_data, uchar *new_data); > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int delete_row(const uchar *buf); > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int index_next(uchar *buf); > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int index_prev(uchar *buf); > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int index_first(uchar *buf); why to have unimplemented methods here? I'd rather remove them. > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int index_last(uchar *buf); > + > + /** @brief > + Unlike index_init(), rnd_init() can be called two consecutive times > + without rnd_end() in between (it only makes sense if scan=1). In this > + case, the second call should prepare for the new table scan (e.g if > + rnd_init() allocates the cursor, the second call should position the > + cursor to the start of the table; no need to deallocate and allocate > + it again. This is a required method. > + */ > + int rnd_init(bool scan); //required > + int rnd_end(); > + int rnd_next(uchar *buf); ///< required > + int rnd_pos(uchar *buf, uchar *pos); ///< required > + void position(const uchar *record); ///< required > + int info(uint); ///< required > + int extra(enum ha_extra_function operation); > + int external_lock(THD *thd, int lock_type); ///< required > + int delete_all_rows(void); > + ha_rows records_in_range(uint inx, key_range *min_key, > + key_range *max_key); > + int delete_table(const char *from); > + int create(const char *name, TABLE *form, > + HA_CREATE_INFO *create_info); ///< required > + bool check_if_incompatible_data(HA_CREATE_INFO *info, > + uint table_changes); > + > + THR_LOCK_DATA **store_lock(THD *thd, THR_LOCK_DATA **to, > + enum thr_lock_type lock_type); ///< required > +}; > > === added file 'storage/cassandra/ha_cassandra.cc' > --- storage/cassandra/ha_cassandra.cc 1970-01-01 00:00:00 +0000 > +++ storage/cassandra/ha_cassandra.cc 2012-11-12 15:11:23 +0000 > @@ -0,0 +1,2674 @@ > +/* > + Copyright (c) 2012, Monty Program Ab > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ > + > +#ifdef USE_PRAGMA_IMPLEMENTATION > +#pragma implementation // gcc: Class implementation > +#endif > + > +#include <mysql/plugin.h> > +#include "ha_cassandra.h" > +#include "sql_class.h" > + > +#define DYNCOL_USUAL 20 > +#define DYNCOL_DELTA 100 > +#define DYNCOL_USUAL_REC 1024 > +#define DYNCOL_DELTA_REC 1024 > + > +static handler *cassandra_create_handler(handlerton *hton, > + TABLE_SHARE *table, > + MEM_ROOT *mem_root); > + > +extern int dynamic_column_error_message(enum_dyncol_func_result rc); > + > +handlerton *cassandra_hton; > + > + > +/* > + Hash used to track the number of open tables; variable for example share > + methods > +*/ > +static HASH cassandra_open_tables; > + > +/* The mutex used to init the hash; variable for example share methods */ > +mysql_mutex_t cassandra_mutex; > + > + > +/** > + Structure for CREATE TABLE options (table options). > + It needs to be called ha_table_option_struct. > + > + The option values can be specified in the CREATE TABLE at the end: > + CREATE TABLE ( ... ) *here* > +*/ > + > +struct ha_table_option_struct > +{ > + const char *thrift_host; > + int thrift_port; > + const char *keyspace; > + const char *column_family; > +}; > + > + > +ha_create_table_option cassandra_table_option_list[]= > +{ > + /* > + one option that takes an arbitrary string > + */ > + HA_TOPTION_STRING("thrift_host", thrift_host), > + HA_TOPTION_NUMBER("thrift_port", thrift_port, 9160, 1, 65535, 0), > + HA_TOPTION_STRING("keyspace", keyspace), > + HA_TOPTION_STRING("column_family", column_family), > + HA_TOPTION_END > +}; > + > +/** > + Structure for CREATE TABLE options (field options). > +*/ > + > +struct ha_field_option_struct > +{ > + bool dyncol_field; > +}; > + > +ha_create_table_option cassandra_field_option_list[]= > +{ > + /* > + Collect all other columns as dynamic here, > + the valid values are YES/NO, ON/OFF, 1/0. > + The default is 0, that is true, yes, on. really? looks like a typo. 0 is false, 1 is true. > + */ > + HA_FOPTION_BOOL("DYNAMIC_COLUMN_STORAGE", dyncol_field, 0), > + HA_FOPTION_END > +}; > + > +static MYSQL_THDVAR_ULONG(insert_batch_size, PLUGIN_VAR_RQCMDARG, > + "Number of rows in an INSERT batch", > + NULL, NULL, /*default*/ 100, /*min*/ 1, /*max*/ 1024*1024*1024, 0); > + > +static MYSQL_THDVAR_ULONG(multiget_batch_size, PLUGIN_VAR_RQCMDARG, > + "Number of rows in a multiget(MRR) batch", > + NULL, NULL, /*default*/ 100, /*min*/ 1, /*max*/ 1024*1024*1024, 0); > + > +static MYSQL_THDVAR_ULONG(rnd_batch_size, PLUGIN_VAR_RQCMDARG, > + "Number of rows in an rnd_read (full scan) batch", > + NULL, NULL, /*default*/ 10*1000, /*min*/ 1, /*max*/ 1024*1024*1024, 0); > + > +static MYSQL_THDVAR_ULONG(failure_retries, PLUGIN_VAR_RQCMDARG, > + "Number of times to retry Cassandra calls that failed due to timeouts or " > + "network communication problems. The default, 0, means not to retry.", > + NULL, NULL, /*default*/ 0, /*min*/ 0, /*max*/ 1024*1024*1024, 0); > + > +/* These match values in enum_cassandra_consistency_level */ > +const char *cassandra_consistency_level[] = > +{ > + "ONE", > + "QUORUM", > + "LOCAL_QUORUM", > + "EACH_QUORUM", > + "ALL", > + "ANY", > + "TWO", > + "THREE", > + NullS > +}; > + > +TYPELIB cassandra_consistency_level_typelib= { > + array_elements(cassandra_consistency_level) - 1, "", > + cassandra_consistency_level, NULL > +}; > + > + > +static MYSQL_THDVAR_ENUM(write_consistency, PLUGIN_VAR_RQCMDARG, > + "Cassandra consistency level to use for write operations", NULL, NULL, > + ONE, &cassandra_consistency_level_typelib); > + > +static MYSQL_THDVAR_ENUM(read_consistency, PLUGIN_VAR_RQCMDARG, > + "Cassandra consistency level to use for read operations", NULL, NULL, > + ONE, &cassandra_consistency_level_typelib); > + > + > +mysql_mutex_t cassandra_default_host_lock; > +static char* cassandra_default_thrift_host = NULL; > +static char cassandra_default_host_buf[256]=""; > + > +static void > +cassandra_default_thrift_host_update(THD *thd, > + struct st_mysql_sys_var* var, > + void* var_ptr, /*!< out: where the > + formal string goes */ > + const void* save) /*!< in: immediate result > + from check function */ > +{ > + const char *new_host= *((char**)save); > + const size_t max_len= sizeof(cassandra_default_host_buf); > + > + mysql_mutex_lock(&cassandra_default_host_lock); > + > + if (new_host) > + { > + strncpy(cassandra_default_host_buf, new_host, max_len-1); > + cassandra_default_host_buf[max_len-1]= 0; > + cassandra_default_thrift_host= cassandra_default_host_buf; > + } > + else > + { > + cassandra_default_host_buf[0]= 0; > + cassandra_default_thrift_host= NULL; > + } > + > + *((const char**)var_ptr)= cassandra_default_thrift_host; > + > + mysql_mutex_unlock(&cassandra_default_host_lock); > +} > + > + > +static MYSQL_SYSVAR_STR(default_thrift_host, cassandra_default_thrift_host, > + PLUGIN_VAR_RQCMDARG, > + "Default host for Cassandra thrift connections", > + /*check*/NULL, > + cassandra_default_thrift_host_update, > + /*default*/NULL); > + > +static struct st_mysql_sys_var* cassandra_system_variables[]= { > + MYSQL_SYSVAR(insert_batch_size), > + MYSQL_SYSVAR(multiget_batch_size), > + MYSQL_SYSVAR(rnd_batch_size), > + > + MYSQL_SYSVAR(default_thrift_host), > + MYSQL_SYSVAR(write_consistency), > + MYSQL_SYSVAR(read_consistency), > + MYSQL_SYSVAR(failure_retries), > + NULL > +}; > + > + > +static SHOW_VAR cassandra_status_variables[]= { > + {"row_inserts", > + (char*) &cassandra_counters.row_inserts, SHOW_LONG}, > + {"row_insert_batches", > + (char*) &cassandra_counters.row_insert_batches, SHOW_LONG}, > + > + {"multiget_reads", > + (char*) &cassandra_counters.multiget_reads, SHOW_LONG}, > + {"multiget_keys_scanned", > + (char*) &cassandra_counters.multiget_keys_scanned, SHOW_LONG}, > + {"multiget_rows_read", > + (char*) &cassandra_counters.multiget_rows_read, SHOW_LONG}, > + > + {"timeout_exceptions", > + (char*) &cassandra_counters.timeout_exceptions, SHOW_LONG}, > + {"unavailable_exceptions", > + (char*) &cassandra_counters.unavailable_exceptions, SHOW_LONG}, > + {NullS, NullS, SHOW_LONG} > +}; > + > + > +Cassandra_status_vars cassandra_counters; > +Cassandra_status_vars cassandra_counters_copy; > + > + > +/** > + @brief > + Function we use in the creation of our hash to get key. > +*/ > + > +static uchar* cassandra_get_key(CASSANDRA_SHARE *share, size_t *length, > + my_bool not_used __attribute__((unused))) > +{ > + *length=share->table_name_length; > + return (uchar*) share->table_name; > +} > + > +#ifdef HAVE_PSI_INTERFACE > +static PSI_mutex_key ex_key_mutex_example, ex_key_mutex_CASSANDRA_SHARE_mutex; > + > +static PSI_mutex_info all_cassandra_mutexes[]= > +{ > + { &ex_key_mutex_example, "cassandra", PSI_FLAG_GLOBAL}, > + { &ex_key_mutex_CASSANDRA_SHARE_mutex, "CASSANDRA_SHARE::mutex", 0} > +}; > + > +static void init_cassandra_psi_keys() > +{ > + const char* category= "cassandra"; > + int count; > + > + if (PSI_server == NULL) > + return; > + > + count= array_elements(all_cassandra_mutexes); > + PSI_server->register_mutex(category, all_cassandra_mutexes, count); > +} > +#endif > + > +static int cassandra_init_func(void *p) > +{ > + DBUG_ENTER("cassandra_init_func"); > + > +#ifdef HAVE_PSI_INTERFACE > + init_cassandra_psi_keys(); > +#endif > + > + cassandra_hton= (handlerton *)p; > + mysql_mutex_init(ex_key_mutex_example, &cassandra_mutex, MY_MUTEX_INIT_FAST); > + (void) my_hash_init(&cassandra_open_tables,system_charset_info,32,0,0, > + (my_hash_get_key) cassandra_get_key,0,0); > + > + cassandra_hton->state= SHOW_OPTION_YES; > + cassandra_hton->create= cassandra_create_handler; > + /* > + Don't specify HTON_CAN_RECREATE in flags. re-create is used by TRUNCATE > + TABLE to create an *empty* table from scratch. Cassandra table won't be > + emptied if re-created. > + */ HTON_ALTER_NOT_SUPPORTED is not set. ALTER works? HTON_TEMPORARY_NOT_SUPPORTED is not set. CREATE TEMPORARY TABLE works? HTON_NO_PARTITION is not set. Partitioning works? > + cassandra_hton->flags= 0; > + cassandra_hton->table_options= cassandra_table_option_list; > + cassandra_hton->field_options= cassandra_field_option_list; > + > + mysql_mutex_init(0 /* no instrumentation */, why no instrumentation ? > + &cassandra_default_host_lock, MY_MUTEX_INIT_FAST); > + > + DBUG_RETURN(0); > +} > + > + > +static int cassandra_done_func(void *p) > +{ > + int error= 0; > + DBUG_ENTER("cassandra_done_func"); > + if (cassandra_open_tables.records) > + error= 1; > + my_hash_free(&cassandra_open_tables); > + mysql_mutex_destroy(&cassandra_mutex); > + mysql_mutex_destroy(&cassandra_default_host_lock); > + DBUG_RETURN(error); > +} > + > + > +/** > + @brief > + Example of simple lock controls. The "share" it creates is a > + structure we will pass to each cassandra handler. Do you have to have > + one of these? Well, you have pieces that are used for locking, and > + they are needed to function. > +*/ > + > +static CASSANDRA_SHARE *get_share(const char *table_name, TABLE *table) > +{ > + CASSANDRA_SHARE *share; > + uint length; > + char *tmp_name; > + > + mysql_mutex_lock(&cassandra_mutex); > + length=(uint) strlen(table_name); > + > + if (!(share=(CASSANDRA_SHARE*) my_hash_search(&cassandra_open_tables, > + (uchar*) table_name, > + length))) > + { > + if (!(share=(CASSANDRA_SHARE *) > + my_multi_malloc(MYF(MY_WME | MY_ZEROFILL), > + &share, sizeof(*share), > + &tmp_name, length+1, > + NullS))) > + { > + mysql_mutex_unlock(&cassandra_mutex); > + return NULL; > + } > + > + share->use_count=0; > + share->table_name_length=length; > + share->table_name=tmp_name; > + strmov(share->table_name,table_name); > + if (my_hash_insert(&cassandra_open_tables, (uchar*) share)) > + goto error; > + thr_lock_init(&share->lock); > + mysql_mutex_init(ex_key_mutex_CASSANDRA_SHARE_mutex, > + &share->mutex, MY_MUTEX_INIT_FAST); > + } > + share->use_count++; > + mysql_mutex_unlock(&cassandra_mutex); > + > + return share; > + > +error: > + mysql_mutex_destroy(&share->mutex); > + my_free(share); > + > + return NULL; > +} > + > + > +/** > + @brief > + Free lock controls. We call this whenever we close a table. If the table had > + the last reference to the share, then we free memory associated with it. > +*/ > + > +static int free_share(CASSANDRA_SHARE *share) > +{ > + mysql_mutex_lock(&cassandra_mutex); > + if (!--share->use_count) > + { > + my_hash_delete(&cassandra_open_tables, (uchar*) share); > + thr_lock_delete(&share->lock); > + mysql_mutex_destroy(&share->mutex); > + my_free(share); > + } > + mysql_mutex_unlock(&cassandra_mutex); > + > + return 0; > +} > + > + > +static handler* cassandra_create_handler(handlerton *hton, > + TABLE_SHARE *table, > + MEM_ROOT *mem_root) > +{ > + return new (mem_root) ha_cassandra(hton, table); > +} > + > + > +ha_cassandra::ha_cassandra(handlerton *hton, TABLE_SHARE *table_arg) > + :handler(hton, table_arg), > + se(NULL), field_converters(NULL), > + special_type_field_converters(NULL), > + special_type_field_names(NULL), n_special_type_fields(0), > + rowkey_converter(NULL), > + dyncol_field(0), dyncol_set(0) > +{} > + > + > +static const char *ha_cassandra_exts[] = { > + NullS > +}; > + > +const char **ha_cassandra::bas_ext() const > +{ > + return ha_cassandra_exts; > +} > + > + > +int ha_cassandra::connect_and_check_options(TABLE *table_arg) > +{ > + ha_table_option_struct *options= table_arg->s->option_struct; > + int res; > + DBUG_ENTER("ha_cassandra::connect_and_check_options"); > + > + if ((res= check_field_options(table_arg->s->field)) || > + (res= check_table_options(options))) > + DBUG_RETURN(res); > + > + se= create_cassandra_se(); > + se->set_column_family(options->column_family); > + const char *thrift_host= options->thrift_host? options->thrift_host: > + cassandra_default_thrift_host; > + if (se->connect(thrift_host, options->thrift_port, options->keyspace)) > + { > + my_error(ER_CONNECT_TO_FOREIGN_DATA_SOURCE, MYF(0), se->error_str()); > + DBUG_RETURN(HA_ERR_NO_CONNECTION); > + } > + > + if (setup_field_converters(table_arg->field, table_arg->s->fields)) > + { > + DBUG_RETURN(HA_ERR_NO_CONNECTION); > + } > + > + DBUG_RETURN(0); > +} > + > + > +int ha_cassandra::check_field_options(Field **fields) > +{ > + Field **field; > + uint i; > + DBUG_ENTER("ha_cassandra::check_field_options"); > + for (field= fields, i= 0; *field; field++, i++) > + { > + ha_field_option_struct *field_options= (*field)->option_struct; > + if (field_options && field_options->dyncol_field) > + { > + if (dyncol_set || (*field)->type() != MYSQL_TYPE_BLOB) > + { > + my_error(ER_WRONG_FIELD_SPEC, MYF(0), (*field)->field_name); > + DBUG_RETURN(HA_WRONG_CREATE_OPTION); > + } > + dyncol_set= 1; > + dyncol_field= i; > + bzero(&dynamic_values, sizeof(dynamic_values)); > + bzero(&dynamic_names, sizeof(dynamic_names)); > + bzero(&dynamic_rec, sizeof(dynamic_rec)); > + } > + } > + DBUG_RETURN(0); > +} > + > + > +int ha_cassandra::open(const char *name, int mode, uint test_if_locked) > +{ > + DBUG_ENTER("ha_cassandra::open"); > + > + if (!(share = get_share(name, table))) > + DBUG_RETURN(1); > + thr_lock_data_init(&share->lock,&lock,NULL); > + > + DBUG_ASSERT(!se); > + /* > + Don't do the following on open: it prevents SHOW CREATE TABLE when the server > + has gone away. > + */ > + /* > + int res; > + if ((res= connect_and_check_options(table))) > + { > + DBUG_RETURN(res); > + } > + */ > + > + info(HA_STATUS_NO_LOCK | HA_STATUS_VARIABLE | HA_STATUS_CONST); > + insert_lineno= 0; > + > + DBUG_RETURN(0); > +} > + > + > +int ha_cassandra::close(void) > +{ > + DBUG_ENTER("ha_cassandra::close"); > + delete se; > + se= NULL; > + free_field_converters(); > + DBUG_RETURN(free_share(share)); > +} > + > + > +int ha_cassandra::check_table_options(ha_table_option_struct *options) > +{ > + if (!options->thrift_host && (!cassandra_default_thrift_host || > + !cassandra_default_thrift_host[0])) > + { > + my_error(ER_CONNECT_TO_FOREIGN_DATA_SOURCE, MYF(0), > + "thrift_host table option must be specified, or " > + "@@cassandra_default_thrift_host must be set"); > + return HA_WRONG_CREATE_OPTION; > + } > + > + if (!options->keyspace || !options->column_family) > + { > + my_error(ER_CONNECT_TO_FOREIGN_DATA_SOURCE, MYF(0), > + "keyspace and column_family table options must be specified"); > + return HA_WRONG_CREATE_OPTION; > + } > + return 0; > +} > + > + > +/** > + @brief > + create() is called to create a table. The variable name will have the name > + of the table. > + > + @details > + When create() is called you do not need to worry about > + opening the table. Also, the .frm file will have already been > + created so adjusting create_info is not necessary. You can overwrite > + the .frm file at this point if you wish to change the table > + definition, but there are no methods currently provided for doing > + so. > + > + Called from handle.cc by ha_create_table(). > + > + @see > + ha_create_table() in handle.cc > +*/ > + > +int ha_cassandra::create(const char *name, TABLE *table_arg, > + HA_CREATE_INFO *create_info) > +{ > + int res; > + DBUG_ENTER("ha_cassandra::create"); > + > + Field **pfield= table_arg->s->field; > + if (!((*pfield)->flags & NOT_NULL_FLAG)) > + { > + my_error(ER_WRONG_COLUMN_NAME, MYF(0), "First column must be NOT NULL"); > + DBUG_RETURN(HA_WRONG_CREATE_OPTION); > + } this is unnecessary. the second check guarantees that the first column is NOT NULL, that is, this if() is redundant > + > + if (table_arg->s->keys != 1 || table_arg->s->primary_key !=0 || > + table_arg->key_info[0].key_parts != 1 || > + table_arg->key_info[0].key_part[0].fieldnr != 1) > + { > + my_error(ER_WRONG_COLUMN_NAME, MYF(0), > + "Table must have PRIMARY KEY defined over the first column"); > + DBUG_RETURN(HA_WRONG_CREATE_OPTION); > + } ... > +CASSANDRA_TYPE get_cassandra_type(const char *validator) > +{ > + CASSANDRA_TYPE rc; > + switch(validator[32]) > + { > + case 'L': > + rc= CT_BIGINT; > + break; > + case 'I': > + rc= (validator[35] == '3' ? CT_INT : CT_VARINT); > + rc= CT_INT; > + break; > + case 'C': > + rc= CT_COUNTER; > + break; > + case 'F': > + rc= CT_FLOAT; > + break; > + case 'D': > + switch (validator[33]) > + { > + case 'o': > + rc= CT_DOUBLE; > + break; > + case 'a': > + rc= CT_TIMESTAMP; > + break; > + case 'e': > + rc= CT_DECIMAL; > + break; > + default: > + rc= CT_BLOB; > + break; > + } > + break; > + case 'B': > + rc= (validator[33] == 'o' ? CT_BOOLEAN : CT_BLOB); > + break; > + case 'A': > + rc= CT_ASCII; > + break; > + case 'U': > + rc= (validator[33] == 'T' ? CT_TEXT : CT_UUID); > + break; > + default: > + rc= CT_BLOB; > + } > + DBUG_ASSERT(strcmp(cassandra_types[rc].name, validator) == 0); > + return rc; > +} > + > +ColumnDataConverter *map_field_to_validator(Field *field, const char *validator_name) > +{ > + ColumnDataConverter *res= NULL; > + > + switch(field->type()) { > + case MYSQL_TYPE_TINY: > + if (!strcmp(validator_name, validator_boolean)) why do you strcmp here, while you're only check selected characters in get_cassandra_type() ? you could've called get_cassandra_type() for validator_name instead and compare enum values, instead of strings. > + { > + res= new TinyintDataConverter; > + break; > + } > + /* fall through: */ > + case MYSQL_TYPE_SHORT: > + case MYSQL_TYPE_LONGLONG: > + { > + bool is_counter= false; > + if (!strcmp(validator_name, validator_bigint) || > + !strcmp(validator_name, validator_timestamp) || > + (is_counter= !strcmp(validator_name, validator_counter))) > + res= new BigintDataConverter(!is_counter); > + break; > + } > + case MYSQL_TYPE_FLOAT: > + if (!strcmp(validator_name, validator_float)) > + res= new FloatDataConverter; > + break; > + > + case MYSQL_TYPE_DOUBLE: > + if (!strcmp(validator_name, validator_double)) > + res= new DoubleDataConverter; > + break; > + > + case MYSQL_TYPE_TIMESTAMP: > + if (!strcmp(validator_name, validator_timestamp)) > + res= new TimestampDataConverter; > + break; > + > + case MYSQL_TYPE_STRING: // these are space padded CHAR(n) strings. > + if (!strcmp(validator_name, validator_uuid) && > + field->real_type() == MYSQL_TYPE_STRING && > + field->field_length == 36) > + { > + // UUID maps to CHAR(36), its text representation > + res= new UuidDataConverter; > + break; > + } > + /* fall through: */ > + case MYSQL_TYPE_VAR_STRING: > + case MYSQL_TYPE_VARCHAR: > + { > + /* > + Cassandra's "varint" type is a binary-encoded arbitary-length > + big-endian number. > + - It can be mapped to VARBINARY(N), with sufficiently big N. > + - If the value does not fit into N bytes, it is an error. We should not > + truncate it, because that is just as good as returning garbage. > + - varint should not be mapped to BINARY(N), because BINARY(N) values > + are zero-padded, which will work as multiplying the value by > + 2^k for some value of k. > + */ > + if (field->type() == MYSQL_TYPE_VARCHAR && > + field->binary() && > + (!strcmp(validator_name, validator_varint) || > + !strcmp(validator_name, validator_decimal))) > + { > + res= new StringCopyConverter(field->field_length); > + break; > + } > + > + if (!strcmp(validator_name, validator_blob) || > + !strcmp(validator_name, validator_ascii) || > + !strcmp(validator_name, validator_text)) > + { > + res= new StringCopyConverter((size_t)-1); > + } > + break; > + } > + case MYSQL_TYPE_LONG: > + if (!strcmp(validator_name, validator_int)) > + res= new Int32DataConverter; > + break; > + > + default:; > + } > + return res; > +} > + > + > +bool ha_cassandra::setup_field_converters(Field **field_arg, uint n_fields) > +{ > + char *col_name; > + int col_name_len; > + char *col_type; > + int col_type_len; > + size_t ddl_fields= se->get_ddl_size(); > + const char *default_type= se->get_default_validator(); > + uint max_non_default_fields; > + DBUG_ENTER("ha_cassandra::setup_field_converters"); > + DBUG_ASSERT(default_type); > + > + DBUG_ASSERT(!field_converters); > + DBUG_ASSERT(dyncol_set == 0 || dyncol_set == 1); > + > + /* > + We always should take into account that in case of using dynamic columns > + sql description contain one field which does not described in > + Cassandra DDL also key field is described separately. So that > + is why we use "n_fields - dyncol_set - 1" or "ddl_fields + 2". > + */ > + max_non_default_fields= ddl_fields + 2 - n_fields; > + if (ddl_fields < (n_fields - dyncol_set - 1)) > + { > + se->print_error("Some of SQL fields were not mapped to Cassandra's fields"); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + > + /* allocate memory in one chunk */ > + size_t memsize= sizeof(ColumnDataConverter*) * n_fields + > + (sizeof(LEX_STRING) + sizeof(CASSANDRA_TYPE_DEF))* > + (dyncol_set ? max_non_default_fields : 0); > + if (!(field_converters= (ColumnDataConverter**)my_malloc(memsize, MYF(0)))) > + DBUG_RETURN(true); > + bzero(field_converters, memsize); > + n_field_converters= n_fields; > + > + if (dyncol_set) > + { > + special_type_field_converters= > + (CASSANDRA_TYPE_DEF *)(field_converters + n_fields); > + special_type_field_names= > + ((LEX_STRING*)(special_type_field_converters + max_non_default_fields)); > + } > + > + if (dyncol_set) > + { > + if (init_dynamic_array(&dynamic_values, > + sizeof(DYNAMIC_COLUMN_VALUE), > + DYNCOL_USUAL, DYNCOL_DELTA)) > + DBUG_RETURN(true); > + else > + if (init_dynamic_array(&dynamic_names, > + sizeof(LEX_STRING), > + DYNCOL_USUAL, DYNCOL_DELTA)) > + { > + delete_dynamic(&dynamic_values); > + DBUG_RETURN(true); > + } > + else > + if (init_dynamic_string(&dynamic_rec, NULL, > + DYNCOL_USUAL_REC, DYNCOL_DELTA_REC)) > + { > + delete_dynamic(&dynamic_values); > + delete_dynamic(&dynamic_names); > + DBUG_RETURN(true); > + } > + > + /* Dynamic column field has special processing */ > + field_converters[dyncol_field]= NULL; > + > + default_type_def= cassandra_types + get_cassandra_type(default_type); > + } > + > + se->first_ddl_column(); > + uint n_mapped= 0; > + while (!se->next_ddl_column(&col_name, &col_name_len, &col_type, > + &col_type_len)) > + { > + Field **field; > + uint i; > + /* Mapping for the 1st field is already known */ > + for (field= field_arg + 1, i= 1; *field; field++, i++) > + { > + if ((!dyncol_set || dyncol_field != i) && > + !strcmp((*field)->field_name, col_name)) > + { > + n_mapped++; > + ColumnDataConverter **conv= field_converters + (*field)->field_index; > + if (!(*conv= map_field_to_validator(*field, col_type))) > + { > + se->print_error("Failed to map column %s to datatype %s", > + (*field)->field_name, col_type); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + (*conv)->field= *field; > + } > + } > + if (dyncol_set && !(*field)) // is needed and not found > + { > + DBUG_PRINT("info",("Field not found: %s", col_name)); > + if (strcmp(col_type, default_type)) > + { > + DBUG_PRINT("info",("Field '%s' non-default type: '%s'", > + col_name, col_type)); > + special_type_field_names[n_special_type_fields].length= col_name_len; > + special_type_field_names[n_special_type_fields].str= col_name; > + special_type_field_converters[n_special_type_fields]= > + cassandra_types[get_cassandra_type(col_type)]; > + n_special_type_fields++; > + } > + } > + } > + > + if (n_mapped != n_fields - 1 - dyncol_set) > + { > + Field *first_unmapped= NULL; > + /* Find the first field */ > + for (uint i= 1; i < n_fields;i++) > + { > + if (!field_converters[i]) > + { > + first_unmapped= field_arg[i]; > + break; > + } > + } > + DBUG_ASSERT(first_unmapped); > + > + se->print_error("Field `%s` could not be mapped to any field in Cassandra", generally (here and everywhere) use %`s instead of `%s` > + first_unmapped->field_name); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + > + /* > + Setup type conversion for row_key. > + */ > + se->get_rowkey_type(&col_name, &col_type); > + if (col_name && strcmp(col_name, (*field_arg)->field_name)) > + { > + se->print_error("PRIMARY KEY column must match Cassandra's name '%s'", > + col_name); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + if (!col_name && strcmp("rowkey", (*field_arg)->field_name)) > + { > + se->print_error("target column family has no key_alias defined, " > + "PRIMARY KEY column must be named 'rowkey'"); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + > + if (col_type != NULL) > + { > + if (!(rowkey_converter= map_field_to_validator(*field_arg, col_type))) > + { > + se->print_error("Failed to map PRIMARY KEY to datatype %s", col_type); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + rowkey_converter->field= *field_arg; > + } > + else > + { > + se->print_error("Cassandra's rowkey has no defined datatype (todo: support this)"); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + > + DBUG_RETURN(false); > +} > + > + > +void ha_cassandra::free_field_converters() > +{ > + delete rowkey_converter; > + rowkey_converter= NULL; > + > + if (dyncol_set) > + { > + delete_dynamic(&dynamic_values); > + delete_dynamic(&dynamic_names); > + dynstr_free(&dynamic_rec); > + } > + if (field_converters) > + { > + for (uint i=0; i < n_field_converters; i++) > + if (field_converters[i]) > + { > + DBUG_ASSERT(!dyncol_set || i == dyncol_field); > + delete field_converters[i]; > + } > + my_free(field_converters); > + field_converters= NULL; > + } > +} > + > + > +int ha_cassandra::index_init(uint idx, bool sorted) > +{ > + int ires; > + if (!se && (ires= connect_and_check_options(table))) > + return ires; > + return 0; > +} > + > +void store_key_image_to_rec(Field *field, uchar *ptr, uint len); > + > +int ha_cassandra::index_read_map(uchar *buf, const uchar *key, > + key_part_map keypart_map, > + enum ha_rkey_function find_flag) > +{ > + int rc= 0; > + DBUG_ENTER("ha_cassandra::index_read_map"); > + > + if (find_flag != HA_READ_KEY_EXACT) > + DBUG_RETURN(HA_ERR_WRONG_COMMAND); did you verify that the server expects HA_ERR_WRONG_COMMAND from this method and correctly handles it? or, perhaps, find_flag is always HA_READ_KEY_EXACT here, because of your index_flags() ? > + > + uint key_len= calculate_key_len(table, active_index, key, keypart_map); > + store_key_image_to_rec(table->field[0], (uchar*)key, key_len); > + > + char *cass_key; > + int cass_key_len; > + my_bitmap_map *old_map; > + > + old_map= dbug_tmp_use_all_columns(table, table->read_set); > + > + if (rowkey_converter->mariadb_to_cassandra(&cass_key, &cass_key_len)) > + { > + /* We get here when making lookups like uuid_column='not-an-uuid' */ > + dbug_tmp_restore_column_map(table->read_set, old_map); > + DBUG_RETURN(HA_ERR_KEY_NOT_FOUND); > + } > + > + dbug_tmp_restore_column_map(table->read_set, old_map); > + > + bool found; > + if (se->get_slice(cass_key, cass_key_len, &found)) > + { > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); generally it's better not to use my_error() directly, but only return an error code, and return your se->error_str() from ha_cassandra::get_error_message() > + rc= HA_ERR_INTERNAL_ERROR; > + } > + > + /* TODO: what if we're not reading all columns?? */ > + if (!found) > + rc= HA_ERR_KEY_NOT_FOUND; > + else > + rc= read_cassandra_columns(false); > + > + DBUG_RETURN(rc); > +} > + > + > +void ha_cassandra::print_conversion_error(const char *field_name, > + char *cass_value, > + int cass_value_len) > +{ > + char buf[32]; > + char *p= cass_value; > + size_t i= 0; > + for (; (i < (int)sizeof(buf)-1) && (p < cass_value + cass_value_len); p++) why do you cast here? > + { > + buf[i++]= map2number[(*p >> 4) & 0xF]; > + buf[i++]= map2number[*p & 0xF]; > + } > + buf[i]=0; > + > + se->print_error("Unable to convert value for field `%s` from Cassandra's data" > + " format. Source data is %d bytes, 0x%s%s", > + field_name, cass_value_len, buf, > + (i == sizeof(buf) - 1)? "..." : ""); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > +} > + > + > +void free_strings(DYNAMIC_COLUMN_VALUE *vals, uint num) > +{ > + for (uint i= 0; i < num; i++) > + if (vals[i].type == DYN_COL_STRING && > + !vals[i].x.string.nonfreeable) > + my_free(vals[i].x.string.value.str); > +} > + > + > +CASSANDRA_TYPE_DEF * ha_cassandra::get_cassandra_field_def(char *cass_name, > + int cass_name_len) > +{ > + CASSANDRA_TYPE_DEF *type= default_type_def; > + for(uint i= 0; i < n_special_type_fields; i++) > + { > + if (cass_name_len == (int)special_type_field_names[i].length && > + memcmp(cass_name, special_type_field_names[i].str, > + cass_name_len) == 0) > + { > + type= special_type_field_converters + i; > + break; > + } > + } > + return type; > +} > + > +int ha_cassandra::read_cassandra_columns(bool unpack_pk) > +{ > + char *cass_name; > + char *cass_value; > + int cass_value_len, cass_name_len; > + Field **field; > + int res= 0; > + ulong total_name_len= 0; > + > + /* > + cassandra_to_mariadb() calls will use field->store(...) methods, which > + require that the column is in the table->write_set > + */ > + my_bitmap_map *old_map; > + old_map= dbug_tmp_use_all_columns(table, table->write_set); > + > + /* Start with all fields being NULL */ > + for (field= table->field + 1; *field; field++) > + (*field)->set_null(); > + > + while (!se->get_next_read_column(&cass_name, &cass_name_len, > + &cass_value, &cass_value_len)) > + { > + // map to our column. todo: use hash or something.. > + bool found= 0; > + for (field= table->field + 1; *field; field++) > + { > + uint fieldnr= (*field)->field_index; > + if ((!dyncol_set || dyncol_field != fieldnr) && > + !strcmp((*field)->field_name, cass_name)) > + { > + found= 1; > + (*field)->set_notnull(); > + if (field_converters[fieldnr]->cassandra_to_mariadb(cass_value, > + cass_value_len)) > + { > + print_conversion_error((*field)->field_name, cass_value, > + cass_value_len); > + res=1; > + goto err; > + } > + break; > + } > + } > + if (dyncol_set && !found) > + { > + DYNAMIC_COLUMN_VALUE val; > + LEX_STRING nm; > + CASSANDRA_TYPE_DEF *type= get_cassandra_field_def(cass_name, > + cass_name_len); > + nm.str= cass_name; > + nm.length= cass_name_len; > + if (nm.length > MAX_NAME_LENGTH) > + { > + se->print_error("Unable to convert value for field `%s`" > + " from Cassandra's data format. Name" > + " length exceed limit of %u: '%s'", > + table->field[dyncol_field]->field_name, > + (uint)MAX_NAME_LENGTH, cass_name); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + res=1; > + goto err; > + } > + total_name_len+= cass_name_len; > + if (nm.length > MAX_TOTAL_NAME_LENGTH) > + { > + se->print_error("Unable to convert value for field `%s`" > + " from Cassandra's data format. Sum of all names" > + " length exceed limit of %lu", > + table->field[dyncol_field]->field_name, > + cass_name, (uint)MAX_TOTAL_NAME_LENGTH); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + res=1; > + goto err; > + } > + > + if ((res= (*(type->cassandra_to_dynamic))(cass_value, > + cass_value_len, &val)) || > + insert_dynamic(&dynamic_names, (uchar *) &nm) || > + insert_dynamic(&dynamic_values, (uchar *) &val)) > + { > + if (res) > + { > + print_conversion_error(cass_name, cass_value, cass_value_len); > + } > + free_strings((DYNAMIC_COLUMN_VALUE *)dynamic_values.buffer, > + dynamic_values.elements); > + // EOM shouldm be already reported if happened > + res=1; > + goto err; > + } > + } > + } > + > + dynamic_rec.length= 0; > + if (dyncol_set) > + { > + if (dynamic_column_create_many_internal_fmt(&dynamic_rec, > + dynamic_names.elements, > + dynamic_names.buffer, > + (DYNAMIC_COLUMN_VALUE *) > + dynamic_values.buffer, > + FALSE, > + TRUE) < 0) > + dynamic_rec.length= 0; > + > + free_strings((DYNAMIC_COLUMN_VALUE *)dynamic_values.buffer, > + dynamic_values.elements); > + dynamic_values.elements= dynamic_names.elements= 0; > + } > + if (dyncol_set) why a separate if() ? > + { > + if (dynamic_rec.length == 0) > + table->field[dyncol_field]->set_null(); > + else > + { > + Field_blob *blob= (Field_blob *)table->field[dyncol_field]; > + blob->set_notnull(); > + blob->store_length(dynamic_rec.length); > + *((char **)(((char *)blob->ptr) + blob->pack_length_no_ptr()))= > + dynamic_rec.str; > + } > + } > + > + if (unpack_pk) > + { > + /* Unpack rowkey to primary key */ > + field= table->field; > + (*field)->set_notnull(); > + se->get_read_rowkey(&cass_value, &cass_value_len); > + if (rowkey_converter->cassandra_to_mariadb(cass_value, cass_value_len)) > + { > + print_conversion_error((*field)->field_name, cass_value, cass_value_len); > + res=1; > + goto err; > + } > + } > + > +err: > + dbug_tmp_restore_column_map(table->write_set, old_map); > + return res; > +} ... > +int ha_cassandra::rnd_pos(uchar *buf, uchar *pos) > +{ > + int rc; > + DBUG_ENTER("ha_cassandra::rnd_pos"); > + > + int save_active_index= active_index; > + active_index= 0; /* The primary key */ > + rc= index_read_map(buf, pos, key_part_map(1), HA_READ_KEY_EXACT); > + > + active_index= save_active_index; a bit nicer would be to implement index_read_map_idx and call it from here and from index_read_map > + > + DBUG_RETURN(rc); > +} > + ... > +bool ha_cassandra::mrr_start_read() > +{ > + uint key_len; > + > + my_bitmap_map *old_map; > + old_map= dbug_tmp_use_all_columns(table, table->read_set); > + > + se->new_lookup_keys(); > + > + while (!(source_exhausted= mrr_funcs.next(mrr_iter, &mrr_cur_range))) > + { > + char *cass_key; > + int cass_key_len; > + > + DBUG_ASSERT(mrr_cur_range.range_flag & EQ_RANGE); > + > + uchar *key= (uchar*)mrr_cur_range.start_key.key; > + key_len= mrr_cur_range.start_key.length; > + //key_len= calculate_key_len(table, active_index, key, keypart_map); // NEED THIS?? > + store_key_image_to_rec(table->field[0], (uchar*)key, key_len); > + > + rowkey_converter->mariadb_to_cassandra(&cass_key, &cass_key_len); > + > + // Primitive buffer control > + if (se->add_lookup_key(cass_key, cass_key_len) > > + THDVAR(table->in_use, multiget_batch_size)) > + break; I'd suggest to add a status variable to count how many times the buffer was refilled > + } > + > + dbug_tmp_restore_column_map(table->read_set, old_map); > + > + return se->multiget_slice(); > +} ... > +///////////////////////////////////////////////////////////////////////////// > +// Dummy implementations start > +///////////////////////////////////////////////////////////////////////////// > + > + > +int ha_cassandra::index_next(uchar *buf) Why did you do all these dummy methods? > +{ > + int rc; > + DBUG_ENTER("ha_cassandra::index_next"); > + rc= HA_ERR_WRONG_COMMAND; > + DBUG_RETURN(rc); > +} > + > + > +int ha_cassandra::index_prev(uchar *buf) > +{ > + int rc; > + DBUG_ENTER("ha_cassandra::index_prev"); > + rc= HA_ERR_WRONG_COMMAND; > + DBUG_RETURN(rc); > +} > + > + > +int ha_cassandra::index_first(uchar *buf) > +{ > + int rc; > + DBUG_ENTER("ha_cassandra::index_first"); > + rc= HA_ERR_WRONG_COMMAND; > + DBUG_RETURN(rc); > +} > + > +int ha_cassandra::index_last(uchar *buf) > +{ > + int rc; > + DBUG_ENTER("ha_cassandra::index_last"); > + rc= HA_ERR_WRONG_COMMAND; > + DBUG_RETURN(rc); > +} > + > + > +ha_rows ha_cassandra::records_in_range(uint inx, key_range *min_key, > + key_range *max_key) > +{ > + DBUG_ENTER("ha_cassandra::records_in_range"); > + //DBUG_RETURN(10); // low number to force index usage > + DBUG_RETURN(HA_POS_ERROR); > +} > + > + > +class Column_name_enumerator_impl : public Column_name_enumerator > +{ > + ha_cassandra *obj; > + uint idx; > +public: > + Column_name_enumerator_impl(ha_cassandra *obj_arg) : obj(obj_arg), idx(1) {} > + const char* get_next_name() > + { > + if (idx == obj->table->s->fields) > + return NULL; > + else > + return obj->table->field[idx++]->field_name; > + } > +}; > + > + > +int ha_cassandra::update_row(const uchar *old_data, uchar *new_data) > +{ > + DYNAMIC_ARRAY oldvals, oldnames, vals, names; > + String oldvalcol, valcol; > + char *oldfree_names= NULL, *free_names= NULL; > + my_bitmap_map *old_map; > + int res; > + DBUG_ENTER("ha_cassandra::update_row"); > + /* Currently, it is guaranteed that new_data == table->record[0] */ > + DBUG_ASSERT(new_data == table->record[0]); > + /* For now, just rewrite the full record */ > + se->clear_insert_buffer(); > + > + old_map= dbug_tmp_use_all_columns(table, table->read_set); > + > + char *old_key; > + int old_key_len; > + se->get_read_rowkey(&old_key, &old_key_len); > + > + /* Get the key we're going to write */ > + char *new_key; > + int new_key_len; > + if (rowkey_converter->mariadb_to_cassandra(&new_key, &new_key_len)) > + { > + my_error(ER_WARN_DATA_OUT_OF_RANGE, MYF(0), > + rowkey_converter->field->field_name, insert_lineno); > + dbug_tmp_restore_column_map(table->read_set, old_map); > + DBUG_RETURN(HA_ERR_AUTOINC_ERANGE); > + } > + > + /* > + Compare it to the key we've read. For all types that Cassandra supports, > + binary byte-wise comparison can be used > + */ > + bool new_primary_key; > + if (new_key_len != old_key_len || memcmp(old_key, new_key, new_key_len)) > + new_primary_key= true; > + else > + new_primary_key= false; > + > + if (dyncol_set) > + { > + Field *field= table->field[dyncol_field]; > + /* move to get old_data */ > + my_ptrdiff_t diff; > + diff= (my_ptrdiff_t) (old_data - new_data); > + field->move_field_offset(diff); // Points now at old_data > + if ((res= read_dyncol(&oldvals, &oldnames, &oldvalcol, &oldfree_names))) > + DBUG_RETURN(res); > + field->move_field_offset(-diff); // back to new_data > + if ((res= read_dyncol(&vals, &names, &valcol, &free_names))) > + { > + free_dynamic_row(&oldnames, &oldvals, oldfree_names); > + DBUG_RETURN(res); > + } > + } > + > + if (new_primary_key) > + { > + /* > + Primary key value changed. This is essentially a DELETE + INSERT. > + Add a DELETE operation into the batch > + */ > + Column_name_enumerator_impl name_enumerator(this); > + se->add_row_deletion(old_key, old_key_len, &name_enumerator, > + (LEX_STRING *)oldnames.buffer, > + (dyncol_set ? oldnames.elements : 0)); > + oldnames.elements= oldvals.elements= 0; // they will be deleted > + } > + > + se->start_row_insert(new_key, new_key_len); > + > + /* Convert other fields */ > + for (uint i= 1; i < table->s->fields; i++) I'd probably would've simply done delete+insert in all cases :) but as you've went to the troubles of updating column by column, it really make little sense to rewrite columns that didn't change. could you check table->write_set here and memcmp with the old column value before updating? > + { > + char *cass_data; > + int cass_data_len; > + if (dyncol_set && dyncol_field == i) > + { > + DBUG_ASSERT(field_converters[i] == NULL); > + if ((res= write_dynamic_row(&vals, &names))) > + goto err; > + } > + else > + { > + if (field_converters[i]->mariadb_to_cassandra(&cass_data, &cass_data_len)) > + { > + my_error(ER_WARN_DATA_OUT_OF_RANGE, MYF(0), > + field_converters[i]->field->field_name, insert_lineno); > + dbug_tmp_restore_column_map(table->read_set, old_map); > + DBUG_RETURN(HA_ERR_AUTOINC_ERANGE); > + } > + se->add_insert_column(field_converters[i]->field->field_name, 0, > + cass_data, cass_data_len); > + } > + } > + if (dyncol_set) > + { > + /* find removed fields */ > + uint i= 0, j= 0; > + LEX_STRING *onames= (LEX_STRING *)oldnames.buffer; > + LEX_STRING *nnames= (LEX_STRING *)names.buffer; > + /* both array are sorted */ > + for(; i < oldnames.elements; i++) > + { > + int scmp= 0; > + while (j < names.elements && > + (nnames[j].length < onames[i].length || > + (nnames[j].length == onames[i].length && > + (scmp= memcmp(nnames[j].str, onames[i].str, > + onames[i].length)) < 0))) > + j++; > + if (j < names.elements && > + nnames[j].length == onames[i].length && > + scmp == 0) > + j++; > + else > + se->add_insert_delete_column(onames[i].str, onames[i].length); > + } > + } > + > + dbug_tmp_restore_column_map(table->read_set, old_map); > + > + res= se->do_insert(); > + > + if (res) > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + > +err: > + if (dyncol_set) > + { > + free_dynamic_row(&oldnames, &oldvals, oldfree_names); > + free_dynamic_row(&names, &vals, free_names); > + } > + > + DBUG_RETURN(res? HA_ERR_INTERNAL_ERROR: 0); > +} > + > + > +int ha_cassandra::extra(enum ha_extra_function operation) > +{ > + DBUG_ENTER("ha_cassandra::extra"); > + DBUG_RETURN(0); > +} please. why do you like dummy methods that much? :) > + > + > +/* The following function was copied from ha_blackhole::store_lock: */ > +THR_LOCK_DATA **ha_cassandra::store_lock(THD *thd, > + THR_LOCK_DATA **to, > + enum thr_lock_type lock_type) > +{ > + DBUG_ENTER("ha_cassandra::store_lock"); > + if (lock_type != TL_IGNORE && lock.type == TL_UNLOCK) > + { > + /* > + Here is where we get into the guts of a row level lock. > + If TL_UNLOCK is set > + If we are not doing a LOCK TABLE or DISCARD/IMPORT > + TABLESPACE, then allow multiple writers > + */ > + > + if ((lock_type >= TL_WRITE_CONCURRENT_INSERT && > + lock_type <= TL_WRITE) && !thd_in_lock_tables(thd) > + && !thd_tablespace_op(thd)) 1. tablespace op in cassandra? really? too much copy-paste is confusing. (here and in the comments too) 2. did you test LOCK TABLES? > + lock_type = TL_WRITE_ALLOW_WRITE; that makes all changes to cassanrda immediately visible by concurrently running threads. okay, if that's intentional, but it needs to be documented, as it not the SQL semantics. > + > + /* > + In queries of type INSERT INTO t1 SELECT ... FROM t2 ... > + MySQL would use the lock TL_READ_NO_INSERT on t2, and that > + would conflict with TL_WRITE_ALLOW_WRITE, blocking all inserts > + to t2. Convert the lock to a normal read lock to allow > + concurrent inserts to t2. > + */ > + > + if (lock_type == TL_READ_NO_INSERT && !thd_in_lock_tables(thd)) > + lock_type = TL_READ; this also breaks SBR for INSERT ... SELECT. same as above - okay if intentional, but should be thouroughly documented. > + > + lock.type= lock_type; > + } > + *to++= &lock; > + DBUG_RETURN(to); > +} > + > + > +int ha_cassandra::external_lock(THD *thd, int lock_type) > +{ > + DBUG_ENTER("ha_cassandra::external_lock"); > + DBUG_RETURN(0); > +} > + > +int ha_cassandra::delete_table(const char *name) > +{ > + DBUG_ENTER("ha_cassandra::delete_table"); > + /* > + Cassandra table is just a view. Dropping it doesn't affect the underlying > + column family. > + */ > + DBUG_RETURN(0); > +} more dummies... > + > + > +/** > + check_if_incompatible_data() called if ALTER TABLE can't detect otherwise > + if new and old definition are compatible > + > + @details If there are no other explicit signs like changed number of > + fields this function will be called by compare_tables() > + (sql/sql_tables.cc) to decide should we rewrite whole table or only .frm > + file. > + > +*/ > + > +bool ha_cassandra::check_if_incompatible_data(HA_CREATE_INFO *info, > + uint table_changes) > +{ > + DBUG_ENTER("ha_cassandra::check_if_incompatible_data"); > + /* Checked, we intend to have this empty for Cassandra SE. */ > + DBUG_RETURN(COMPATIBLE_DATA_YES); > +} > + > + > +///////////////////////////////////////////////////////////////////////////// > +// Dummy implementations end > +///////////////////////////////////////////////////////////////////////////// ::update_row and ::store_lock are hardly dummies :) > + > +static int show_cassandra_vars(THD *thd, SHOW_VAR *var, char *buff) > +{ > + cassandra_counters_copy= cassandra_counters; > + > + var->type= SHOW_ARRAY; > + var->value= (char *) &cassandra_status_variables; what's the point? If you don't do anything in this function, you can just as easily list all status variables below in the func_status[] array. > + return 0; > +} > + > + > +struct st_mysql_storage_engine cassandra_storage_engine= > +{ MYSQL_HANDLERTON_INTERFACE_VERSION }; > + > +static struct st_mysql_show_var func_status[]= > +{ > + {"Cassandra", (char *)show_cassandra_vars, SHOW_FUNC}, > + {0,0,SHOW_UNDEF} > +}; > + > +maria_declare_plugin(cassandra) > +{ > + MYSQL_STORAGE_ENGINE_PLUGIN, > + &cassandra_storage_engine, > + "CASSANDRA", > + "Monty Program Ab", > + "Cassandra storage engine", > + PLUGIN_LICENSE_GPL, > + cassandra_init_func, /* Plugin Init */ > + cassandra_done_func, /* Plugin Deinit */ > + 0x0001, /* version number (0.1) */ > + func_status, /* status variables */ > + cassandra_system_variables, /* system variables */ > + "0.1", /* string version */ > + MariaDB_PLUGIN_MATURITY_EXPERIMENTAL /* maturity */ > +} > +maria_declare_plugin_end; Regards, Sergei

2 3

Re: [Maria-developers] [Commits] Rev 3480: Post review changes in the interface part 1.
by Sergei Golubchik 19 Dec '12

19 Dec '12

Hi, Sanja! few preliminary comments: On Dec 19, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-10.0-cassandra/ > ------------------------------------------------------------ > revno: 3480 > revision-id: sanja(a)askmonty.org-20121219200839-oyxvfz7im9p7wqpw > parent: sanja(a)montyprogram.com-20121119121604-5h5tu0zn11em0sb3 > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-cassandra > timestamp: Wed 2012-12-19 22:08:39 +0200 > message: > Post review changes in the interface part 1. > === modified file 'mysys/ma_dyncol.c' > --- a/mysys/ma_dyncol.c 2012-09-28 12:27:16 +0000 > +++ b/mysys/ma_dyncol.c 2012-12-19 20:08:39 +0000 > @@ -26,6 +26,39 @@ > SUCH DAMAGE. > */ > > +/* > + Numeric format: > + =============== > + * Fixed header part > + 1 byte flags: > + 0,1 bits - <offset size> - 1 > + 2-7 bits - 0 > + 2 bytes column counter > + * Columns directory sorted by column number, each entry contains of: > + 2 bytes column number > + <offset size> bytes (1-4) combined offset from beginning of > + the data segment + 3 bit type > + * Data of above columns size of data and length depend on type > + > + Columns with names: > + =================== > + * Fixed header part > + 1 byte flags: > + 0,1 bits - <offset size> - 2 > + 2 bit - 1 (mens format with names) s/mens/means/g > + 3,4 bits - 01 (mens <names offset size> - 1, now 2 is only supported size) eh. I would simply assume that if the bit 2 is 1, this also means offset-2. Like, you have "old format" as above, and "new format" with names, support for recursion (4 bits per type, offset-2), etc. you only have 6 bits here, let's use them sparingly and keep things simple. > + 5-7 bits - 0 > + 2 bytes column counter > + * Variable header part > + <names offset size> (2) bytes size of stored names pool > + * Column directory sorted by names, each consists of > + <names offset size> (2) bytes offset of name > + <offset size> bytes (1-4)bytes combined offset from beginning of > + the data segment + 4 bit type > + * Names stored one after another > + * Data of above columns size of data and length depend on type > +*/ > + > #include "mysys_priv.h" > #include <m_string.h> > #include <ma_dyncol.h> > === modified file 'storage/cassandra/ha_cassandra.cc' > --- a/storage/cassandra/ha_cassandra.cc 2012-11-19 12:16:04 +0000 > +++ b/storage/cassandra/ha_cassandra.cc 2012-12-19 20:08:39 +0000 > @@ -893,9 +893,37 @@ public: > /** > Converting dynamic columns types to/from casandra types > */ > + > + > +/** > + Check and initialize (if it is needed) string MEM_ROOT > +*/ > +static void alloc_strings_memroot(MEM_ROOT *mem_root) > +{ > + if (mem_root->block_size == 0) write if (alloc_root_inited(mem_root)) > + { > + /* > + The mem_root used to allocate UUID (of length 36 + \0) so make > + appropriate allocated size > + */ > + init_alloc_root(mem_root, > + (36 + 1 + ALIGN_SIZE(sizeof(USED_MEM))) * 10 + > + ALLOC_ROOT_MIN_BLOCK_SIZE, > + (36 + 1 + ALIGN_SIZE(sizeof(USED_MEM))) * 10 + > + ALLOC_ROOT_MIN_BLOCK_SIZE); > + } > +} > + > +static void free_strings_memroot(MEM_ROOT *mem_root) > +{ > + if (mem_root->block_size != 0) and here > + free_root(mem_root, MYF(0)); > +} > + > @@ -1730,6 +1759,7 @@ int ha_cassandra::read_cassandra_columns > int res= 0; > ulong total_name_len= 0; > > + strings_root.block_size= 0; // indicates uninitialized MEM_ROOT Use clear_alloc_root(&strings_root); > /* > cassandra_to_mariadb() calls will use field->store(...) methods, which > require that the column is in the table->write_set Regards, Sergei

1 0

Re: [Maria-developers] (MDEV-3949) OTP authenticator plugins
by Roberto Spadim 19 Dec '12

19 Dec '12

hi sergei! sorry about wrong place, i'm new in mariadb and i bit lost, since i changed from mysql development and error correction to mysql user and now mariadb user i will add my email to list and start asking there about s/key, shure! if you could send and maybe put in mariadb by default could be very nice! i'm using some windows and some linux server, in this case i can't use only pam.d, and some linux servers don't give me a unix user to use it :/ that's why i'm trying to find a OTP solution to mariadb, and if not try to develop one (when i have time) well if you could send your solution we could develop some more ideas for sure, i'will not use it to make money, just to secure my self 2012/12/19 <serg(a)montyprogram.com> > ou ask a question - do it on the mailing list. Jira is for > bug reports and feature requests. > -- Roberto Spadim Spadim Technology / SPAEmpresarial

2 1

[Maria-developers] Patches in 10.0 missing in 10.0-base, causing buildbot failures
by Kristian Nielsen 19 Dec '12

19 Dec '12

Hi, I was looking at 10.0-base in Buildbot, in particular at the "update2" failures in all the .deb builds. I fixed one thing in Buildbot, but it still fails. I suspect it is because of these two (at least) commits: sergii(a)pisem.net-20121027121326-q87f5lt6ldxr1em6 sergii(a)pisem.net-20121020163522-uw2apo36c6nn6yi8 These were pushed to 10.0, while it seems they should instead have been pushed to 10.0-base (where they are missing). There are probably also other commits with same problem, eg. this one which is a 5.5 merge according to commit message: sergii(a)pisem.net-20121103112851-bvy1mlhirxbkedvf Any suggestions what to do to fix this and get 10.0-base working in Buildbot? - Kristian.

3 5

Re: [Maria-developers] [Commits] Rev 3449: MDEV-532: Fix some race conditions in test cases. in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Kristian Nielsen 19 Dec '12

19 Dec '12

Hi Serg, Just wanted to be sure you saw this. The async binlogging of the checkpoint event after MDEV-532 means a few test cases need to wait for the event to be logged, or they will get races causing failures with result difference in SHOW BINLOG EVENTS or similar. Typically after running FLUSH LOGS. I fixed the ones I found already. But it is possible that a few others will pop up, and it is useful that someone else than me knows how to fix (so I'm picking on you as the reviewer :-). As seen in the patch, it's just a matter of including the file include/wait_for_binlog_checkpoint.inc in the appropriate place. - Kristian. knielsen(a)knielsen-hq.org writes: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 3449 > revision-id: knielsen(a)knielsen-hq.org-20121217114911-m3qv20xmk7htleez > parent: igor(a)askmonty.org-20121217004919-xgon81zqncwimr3m > committer: knielsen(a)knielsen-hq.org > branch nick: mariadb-10.0-base > timestamp: Mon 2012-12-17 12:49:11 +0100 > message: > MDEV-532: Fix some race conditions in test cases. > > With MDEV-532, the binlog_checkpoint event is logged asynchronously > from a binlog background thread. This causes some sporadic failures > in some test cases whose output depends on order of events in > binlog. > > Fix using an include file that waits until the binlog checkpoint > event has been logged before proceeding with the test case. > === modified file 'mysql-test/extra/rpl_tests/rpl_insert_delayed.test' > --- a/mysql-test/extra/rpl_tests/rpl_insert_delayed.test 2012-09-22 14:11:40 +0000 > +++ b/mysql-test/extra/rpl_tests/rpl_insert_delayed.test 2012-12-17 11:49:11 +0000 > @@ -94,8 +94,10 @@ if (`SELECT @@global.binlog_format = 'S > #flush the logs before the test > connection slave; > FLUSH LOGS; > + source include/wait_for_binlog_checkpoint.inc; > connection master; > FLUSH LOGS; > + source include/wait_for_binlog_checkpoint.inc; > } > > CREATE TABLE t1(a int, UNIQUE(a)); > > === modified file 'mysql-test/extra/rpl_tests/rpl_log.test' > --- a/mysql-test/extra/rpl_tests/rpl_log.test 2011-10-19 19:45:18 +0000 > +++ b/mysql-test/extra/rpl_tests/rpl_log.test 2012-12-17 11:49:11 +0000 > @@ -43,6 +43,7 @@ let $binlog_limit= 1,4; > source include/show_binlog_events.inc; > let $binlog_limit=; > flush logs; > +--source include/wait_for_binlog_checkpoint.inc > > # We need an extra update before doing save_master_pos. > # Otherwise, an unlikely scenario may occur: > > === modified file 'mysql-test/extra/rpl_tests/rpl_show_relaylog_events.inc' > --- a/mysql-test/extra/rpl_tests/rpl_show_relaylog_events.inc 2011-01-13 14:31:37 +0000 > +++ b/mysql-test/extra/rpl_tests/rpl_show_relaylog_events.inc 2012-12-17 11:49:11 +0000 > @@ -41,8 +41,10 @@ INSERT INTO t1 VALUES (3); > # > > FLUSH LOGS; > +--source include/wait_for_binlog_checkpoint.inc > -- connection master > FLUSH LOGS; > +--source include/wait_for_binlog_checkpoint.inc > DROP TABLE t1; > > --let $is_relay_log= 0 > > === added file 'mysql-test/include/wait_for_binlog_checkpoint.inc' > --- a/mysql-test/include/wait_for_binlog_checkpoint.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/wait_for_binlog_checkpoint.inc 2012-12-17 11:49:11 +0000 > @@ -0,0 +1,53 @@ > +# include/wait_for_binlog_checkpoint.inc > +# > +# SUMMARY > +# > +# Wait until binlog checkpoint has been logged for current binlog file. > +# This is useful to avoid races with output difference for binlog > +# checkpoints, as these are logged asynchronously from the binlog > +# background thread. > +# > +# USAGE: > +# > +# --source include/wait_for_binlog_checkpoint.inc > + > +let $_wait_count= 300; > + > +let $_found= 0; > + > +while ($_wait_count) > +{ > + dec $_wait_count; > + let $_cur_binlog= query_get_value(SHOW MASTER STATUS, File, 1); > + let $_more= 1; > + let $_row= 1; > + while ($_more) > + { > + let $_event= query_get_value(SHOW BINLOG EVENTS IN "$_cur_binlog", Event_type, $_row); > + if ($_event == "No such row") > + { > + let $_more= 0; > + } > + if ($_event == "Binlog_checkpoint") > + { > + let $_info= query_get_value(SHOW BINLOG EVENTS IN "$_cur_binlog", Info, $_row); > + if (`SELECT INSTR("$_info", "$_cur_binlog") != 0`) > + { > + let $_more= 0; > + let $_wait_count= 0; > + let $_found= 1; > + } > + } > + inc $_row; > + } > + if ($_wait_count) > + { > + real_sleep 0.1; > + } > +} > + > +if (!$_found) > +{ > + eval SHOW BINLOG EVENTS IN "$_cur_binlog"; > + --die ERROR: failed while waiting for binlog checkpoint $_cur_binlog > +} > > === modified file 'mysql-test/suite/innodb/t/binlog_consistent.test' > --- a/mysql-test/suite/innodb/t/binlog_consistent.test 2012-02-07 15:22:36 +0000 > +++ b/mysql-test/suite/innodb/t/binlog_consistent.test 2012-12-17 11:49:11 +0000 > @@ -72,6 +72,7 @@ connection con3; > --echo # Connection con3 > COMMIT; > FLUSH LOGS; > +--source include/wait_for_binlog_checkpoint.inc > > connection default; > --echo # Connection default > > === modified file 'mysql-test/suite/multi_source/multisource.test' > --- a/mysql-test/suite/multi_source/multisource.test 2012-10-04 20:36:17 +0000 > +++ b/mysql-test/suite/multi_source/multisource.test 2012-12-17 11:49:11 +0000 > @@ -169,6 +169,7 @@ select * from db2.t1; > > --connection master1 > flush logs; > +--source include/wait_for_binlog_checkpoint.inc > --save_master_pos > --connection slave > --sync_with_master 0, 'master1' > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] extended keys
by Igor Babaev 18 Dec '12

18 Dec '12

On 12/17/2012 06:43 AM, Rich Prohaska wrote: > What do I need to change in my storage engine for extended keys? > Where should I look? > Thanks. Hi, see the code /* Currently only InnoDB can use extended keys */ share->set_use_ext_keys_flag(legacy_db_type == DB_TYPE_INNODB); in open_binary_frm(), table.cc (mariadb) You have to change it. There is a similar code in mysql, also in open_binary_frm(),table.cc Regards, Igor. > > On Sun, Dec 16, 2012 at 10:58 PM, Igor Babaev <igor(a)askmonty.org> wrote: >> On 12/14/2012 11:33 AM, Rich Prohaska wrote: >>> MySQL 5.6.9 changed how the number of key parts are measured. They >>> replaced one simple counter (key_parts) with four counters. Glancing >>> at the code, i assume that this was done to support extended keys. If >>> so, the mariadb implementation of extended keys is a lot simpler. We >>> have to maintain the tokudb storage engine on both mysql and mariadb. >>> Is the mariadb design going to change to match the 5.6 implementation, >>> or are the designs now separate? >>> Thanks >>> Rich Prohaska >> >> Hi, >> >> MariaDB supports extended keys starting with first version of 5.5. >> See https://kb.askmonty.org/en/extended-keys/. >> >> See also blogs: >> >> http://igors-notes.blogspot.com/2011/12/3-way-join-that-touches-only-indexe… >> >> http://s.petrunia.net/blog/?p=74 >> >> The interesting thing is that Oracle essentially pulled MariaDB >> implementation for this feature into a mysql version that was in RC. >> Yet one hardly can find anything about the feature in the changelog or >> in the manual. >> >> Regards, >> Igor. >> >> >> >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~maria-developers >>> Post to : maria-developers(a)lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~maria-developers >>> More help : https://help.launchpad.net/ListHelp >>

1 0

Re: [Maria-developers] [Commits] Rev 3592: Test commit for diff options in file:///H:/bzr/5.5/
by Sergei Golubchik 17 Dec '12

17 Dec '12

Hi, Vladislav! On Dec 17, Vladislav Vaintroub wrote: > revno: 3592 > revision-id: wlad(a)montyprogram.com-20121217203725-f5e0mguvmjvb97su > committer: Vladislav Vaintroub <wlad(a)montyprogram.com> > branch nick: 5.5 > timestamp: Mon 2012-12-17 21:37:25 +0100 > message: > Test commit for diff options > === modified file 'sql/mysqld.cc' > --- a/sql/mysqld.cc 2012-11-22 09:19:31 +0000 > +++ b/sql/mysqld.cc 2012-12-17 20:37:25 +0000 > @@ -2441,21 +2441,6 @@ void dec_connection_count(THD *thd) The patch looks ok to me. But I'd prefer a better changeset comment :) Regards, Sergei

1 0

[Maria-developers] extended keys
by Rich Prohaska 17 Dec '12

17 Dec '12

MySQL 5.6.9 changed how the number of key parts are measured. They replaced one simple counter (key_parts) with four counters. Glancing at the code, i assume that this was done to support extended keys. If so, the mariadb implementation of extended keys is a lot simpler. We have to maintain the tokudb storage engine on both mysql and mariadb. Is the mariadb design going to change to match the 5.6 implementation, or are the designs now separate? Thanks Rich Prohaska

2 1

Re: [Maria-developers] [Commits] Rev 3435: MDEV-532: Async InnoDB commit checkpoint. in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Kristian Nielsen 14 Dec '12

14 Dec '12

Hi Serg, As we discussed under review of MDEV-232, here is a separate patch that makes InnoDB/XtraDB commit checkpointing be more asynchroneous. See MDEV-532 for further description of this task. I hope you will review this, at your convenience. - Kristian. knielsen(a)knielsen-hq.org writes: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 3435 > revision-id: knielsen(a)knielsen-hq.org-20120914124453-zsap6hjclq3vrb6n > parent: knielsen(a)knielsen-hq.org-20120913123129-kaujy4cw0jc9o08k > committer: knielsen(a)knielsen-hq.org > branch nick: work-10.0-mdev225-181-232 > timestamp: Fri 2012-09-14 14:44:53 +0200 > message: > MDEV-532: Async InnoDB commit checkpoint. > > Make the commit checkpoint inside InnoDB be asynchroneous. > Implement a background thread in binlog to do the writing and flushing of > binlog checkpoint events to disk. > === modified file 'mysql-test/suite/binlog/r/binlog_checkpoint.result' > --- a/mysql-test/suite/binlog/r/binlog_checkpoint.result 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/r/binlog_checkpoint.result 2012-09-14 12:44:53 +0000 > @@ -70,8 +70,14 @@ show binlog events in 'master-bin.000003 > Log_name Pos Event_type Server_id End_log_pos Info > master-bin.000003 # Format_desc # # SERVER_VERSION, BINLOG_VERSION > master-bin.000003 # Binlog_checkpoint # # master-bin.000001 > +SET DEBUG_SYNC= "RESET"; > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > SET DEBUG_SYNC= "now SIGNAL con2_continue"; > con1 is still pending, no new binlog checkpoint should have been logged. > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > +SET DEBUG_SYNC= "RESET"; > show binlog events in 'master-bin.000003' from <binlog_start>; > Log_name Pos Event_type Server_id End_log_pos Info > master-bin.000003 # Format_desc # # SERVER_VERSION, BINLOG_VERSION > > === modified file 'mysql-test/suite/binlog/r/binlog_xa_recover.result' > --- a/mysql-test/suite/binlog/r/binlog_xa_recover.result 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/r/binlog_xa_recover.result 2012-09-14 12:44:53 +0000 > @@ -118,7 +118,11 @@ master-bin.00000<binlog_start> # Table_m > master-bin.00000<binlog_start> # Write_rows # # table_id: # flags: STMT_END_F > master-bin.00000<binlog_start> # Xid # # COMMIT /* XID */ > SET DEBUG_SYNC= "now SIGNAL con10_cont"; > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > SET DEBUG_SYNC= "now SIGNAL con12_cont"; > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > SET DEBUG_SYNC= "now SIGNAL con11_cont"; > Checking that master-bin.000004 is the last binlog checkpoint > show binlog events in 'master-bin.00000<binlog_start>' from <binlog_start>; > > === modified file 'mysql-test/suite/binlog/t/binlog_checkpoint.test' > --- a/mysql-test/suite/binlog/t/binlog_checkpoint.test 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/t/binlog_checkpoint.test 2012-09-14 12:44:53 +0000 > @@ -71,6 +71,12 @@ SET DEBUG_SYNC= "now WAIT_FOR con2_ready > --let $binlog_file= master-bin.000003 > --source include/show_binlog_events.inc > > +# We need to sync the test case with the background processing of the > +# commit checkpoint, otherwise we get nondeterministic results. > +SET DEBUG_SYNC= "RESET"; > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > + > SET DEBUG_SYNC= "now SIGNAL con2_continue"; > > connection con2; > @@ -78,6 +84,12 @@ reap; > > connection default; > --echo con1 is still pending, no new binlog checkpoint should have been logged. > +# Make sure commit checkpoint is processed before we check that no checkpoint > +# event has been binlogged. > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > +SET DEBUG_SYNC= "RESET"; > + > --let $binlog_file= master-bin.000003 > --source include/show_binlog_events.inc > > > === modified file 'mysql-test/suite/binlog/t/binlog_xa_recover.test' > --- a/mysql-test/suite/binlog/t/binlog_xa_recover.test 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/t/binlog_xa_recover.test 2012-09-14 12:44:53 +0000 > @@ -14,8 +14,24 @@ CREATE TABLE t1 (a INT PRIMARY KEY, b ME > # Insert some data to force a couple binlog rotations (3), so we get some > # normal binlog checkpoints before starting the test. > INSERT INTO t1 VALUES (100, REPEAT("x", 4100)); > +# Wait for the master-bin.000002 binlog checkpoint to appear. > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000002" > +--let $field= Info > +--let $condition= = "master-bin.000002" > +--source include/wait_show_condition.inc > INSERT INTO t1 VALUES (101, REPEAT("x", 4100)); > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000003" > +--let $field= Info > +--let $condition= = "master-bin.000003" > +--source include/wait_show_condition.inc > INSERT INTO t1 VALUES (102, REPEAT("x", 4100)); > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000004" > +--let $field= Info > +--let $condition= = "master-bin.000004" > +--source include/wait_show_condition.inc > > # Now start a bunch of transactions that span multiple binlog > # files. Leave then in the state prepared-but-not-committed in the engine > @@ -153,10 +169,19 @@ SET DEBUG_SYNC= "now SIGNAL con10_cont"; > connection con10; > reap; > connection default; > + > +# We need to sync the test case with the background processing of the > +# commit checkpoint, otherwise we get nondeterministic results. > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > + > SET DEBUG_SYNC= "now SIGNAL con12_cont"; > connection con12; > reap; > connection default; > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > + > SET DEBUG_SYNC= "now SIGNAL con11_cont"; > connection con11; > reap; > @@ -210,7 +235,20 @@ RESET MASTER; > # crash recovery fails due to the error insert used for previous test. > INSERT INTO t1 VALUES (21, REPEAT("x", 4100)); > INSERT INTO t1 VALUES (22, REPEAT("x", 4100)); > +# Wait for the master-bin.000003 binlog checkpoint to appear. > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000003" > +--let $field= Info > +--let $condition= = "master-bin.000003" > +--source include/wait_show_condition.inc > INSERT INTO t1 VALUES (23, REPEAT("x", 4100)); > +# Wait for the last (master-bin.000004) binlog checkpoint to appear. > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000004" > +--let $field= Info > +--let $condition= = "master-bin.000004" > +--source include/wait_show_condition.inc > + > --write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > wait-binlog_xa_recover.test > EOF > > === modified file 'mysql-test/suite/perfschema/r/all_instances.result' > --- a/mysql-test/suite/perfschema/r/all_instances.result 2012-06-22 09:46:28 +0000 > +++ b/mysql-test/suite/perfschema/r/all_instances.result 2012-09-14 12:44:53 +0000 > @@ -76,6 +76,7 @@ wait/synch/mutex/sql/Master_info::run_lo > wait/synch/mutex/sql/Master_info::sleep_lock > wait/synch/mutex/sql/MDL_map::mutex > wait/synch/mutex/sql/MDL_wait::LOCK_wait_status > +wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_binlog_thread > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_index > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_xid_list > wait/synch/mutex/sql/MYSQL_RELAY_LOG::LOCK_index > @@ -129,6 +130,8 @@ wait/synch/cond/sql/Master_info::sleep_c > wait/synch/cond/sql/Master_info::start_cond > wait/synch/cond/sql/Master_info::stop_cond > wait/synch/cond/sql/MDL_context::COND_wait_status > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread_end > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_queue_busy > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list > wait/synch/cond/sql/MYSQL_BIN_LOG::update_cond > > === modified file 'mysql-test/suite/perfschema/r/relaylog.result' > --- a/mysql-test/suite/perfschema/r/relaylog.result 2012-06-22 09:46:28 +0000 > +++ b/mysql-test/suite/perfschema/r/relaylog.result 2012-09-14 12:44:53 +0000 > @@ -56,8 +56,11 @@ where event_name like "%MYSQL_BIN_LOG%" > and event_name not like "%MYSQL_BIN_LOG::update_cond" > order by event_name; > EVENT_NAME COUNT_STAR > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread NONE > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread_end NONE > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_queue_busy NONE > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list NONE > +wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_binlog_thread MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_index MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_xid_list MANY > "Expect no slave relay log" > @@ -131,8 +134,11 @@ where event_name like "%MYSQL_BIN_LOG%" > and event_name not like "%MYSQL_BIN_LOG::update_cond" > order by event_name; > EVENT_NAME COUNT_STAR > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread MANY > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread_end NONE > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_queue_busy NONE > -wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list NONE > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list MANY > +wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_binlog_thread MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_index MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_xid_list MANY > "Expect a slave relay log" > > === modified file 'sql/debug_sync.cc' > --- a/sql/debug_sync.cc 2012-03-28 17:26:00 +0000 > +++ b/sql/debug_sync.cc 2012-09-14 12:44:53 +0000 > @@ -984,6 +984,7 @@ static bool debug_sync_eval_action(THD * > DBUG_ENTER("debug_sync_eval_action"); > DBUG_ASSERT(thd); > DBUG_ASSERT(action_str); > + DBUG_PRINT("debug_sync", ("action_str='%s'", action_str)); > > /* > Get debug sync point name. Or a special command. > > === modified file 'sql/log.cc' > --- a/sql/log.cc 2012-09-13 12:31:29 +0000 > +++ b/sql/log.cc 2012-09-14 12:44:53 +0000 > @@ -53,6 +53,7 @@ > #include "rpl_handler.h" > #include "debug_sync.h" > #include "sql_show.h" > +#include "my_pthread.h" > > /* max size of the log message */ > #define MAX_LOG_BUFFER_SIZE 1024 > @@ -106,6 +107,14 @@ static SHOW_VAR binlog_status_vars_detai > {NullS, NullS, SHOW_LONG} > }; > > +/* Variables for the binlog background thread. */ > +static bool binlog_thread_started= false; > +static bool binlog_background_thread_stop= false; > +static MYSQL_BIN_LOG::xid_count_per_binlog * > + binlog_background_thread_queue= NULL; > + > +static bool start_binlog_background_thread(); > + > > /** > purge logs, master and slave sides both, related error code > @@ -2957,12 +2966,27 @@ void MYSQL_BIN_LOG::cleanup() > my_free(b); > } > > + /* Wait for the binlog thread to stop. */ > + if (!is_relay_log && binlog_thread_started) > + { > + mysql_mutex_lock(&LOCK_binlog_thread); > + binlog_background_thread_stop= true; > + mysql_cond_signal(&COND_binlog_thread); > + while (binlog_background_thread_stop) > + mysql_cond_wait(&COND_binlog_thread_end, &LOCK_binlog_thread); > + mysql_mutex_unlock(&LOCK_binlog_thread); > + binlog_thread_started= false; > + } > + > mysql_mutex_destroy(&LOCK_log); > mysql_mutex_destroy(&LOCK_index); > mysql_mutex_destroy(&LOCK_xid_list); > + mysql_mutex_destroy(&LOCK_binlog_thread); > mysql_cond_destroy(&update_cond); > mysql_cond_destroy(&COND_queue_busy); > mysql_cond_destroy(&COND_xid_list); > + mysql_cond_destroy(&COND_binlog_thread); > + mysql_cond_destroy(&COND_binlog_thread_end); > } > DBUG_VOID_RETURN; > } > @@ -2988,6 +3012,11 @@ void MYSQL_BIN_LOG::init_pthread_objects > mysql_cond_init(m_key_update_cond, &update_cond, 0); > mysql_cond_init(m_key_COND_queue_busy, &COND_queue_busy, 0); > mysql_cond_init(key_BINLOG_COND_xid_list, &COND_xid_list, 0); > + > + mysql_mutex_init(key_BINLOG_LOCK_binlog_thread, > + &LOCK_binlog_thread, MY_MUTEX_INIT_FAST); > + mysql_cond_init(key_BINLOG_COND_binlog_thread, &COND_binlog_thread, 0); > + mysql_cond_init(key_BINLOG_COND_binlog_thread_end, &COND_binlog_thread_end, 0); > } > > > @@ -3085,6 +3114,10 @@ bool MYSQL_BIN_LOG::open(const char *log > DBUG_ENTER("MYSQL_BIN_LOG::open"); > DBUG_PRINT("enter",("log_type: %d",(int) log_type_arg)); > > + if (!is_relay_log && !binlog_thread_started && > + start_binlog_background_thread()) > + DBUG_RETURN(1); > + > if (init_and_set_log_file_name(log_name, new_name, log_type_arg, > io_cache_type_arg)) > { > @@ -5540,11 +5573,7 @@ bool general_log_write(THD *thd, enum en > } > > > -/* > - I would like to make this function static, but this causes compiler warnings > - when it is declared as friend function in log.h. > -*/ > -void > +static void > binlog_checkpoint_callback(void *cookie) > { > MYSQL_BIN_LOG::xid_count_per_binlog *entry= > @@ -8116,9 +8145,128 @@ int TC_LOG_BINLOG::unlog(ulong cookie, m > void > TC_LOG_BINLOG::commit_checkpoint_notify(void *cookie) > { > - mark_xid_done(((xid_count_per_binlog *)cookie)->binlog_id, true); > + xid_count_per_binlog *entry= static_cast<xid_count_per_binlog *>(cookie); > + mysql_mutex_lock(&LOCK_binlog_thread); > + entry->next_in_queue= binlog_background_thread_queue; > + binlog_background_thread_queue= entry; > + mysql_cond_signal(&COND_binlog_thread); > + mysql_mutex_unlock(&LOCK_binlog_thread); > } > > +/* > + Binlog service thread. > + > + This thread is used to log binlog checkpoints in the background, rather than > + in the context of random storage engine threads that happen to call > + commit_checkpoint_notify_ha() and may not like the delays while syncing > + binlog to disk or may not be setup with all my_thread_init() and other > + necessary stuff. > + > + In the future, this thread could also be used to do log rotation in the > + background, which could elimiate all stalls around binlog rotations. > +*/ > +pthread_handler_t > +binlog_background_thread(void *arg __attribute__((unused))) > +{ > + bool stop; > + MYSQL_BIN_LOG::xid_count_per_binlog *queue, *next; > + THD *thd; > + > + my_thread_init(); > + thd= new THD; > + thd->system_thread= SYSTEM_THREAD_BINLOG_BACKGROUND; > + my_pthread_setspecific_ptr(THR_THD, thd); > + mysql_mutex_lock(&LOCK_thread_count); > + thd->thread_id= thread_id++; > + mysql_mutex_unlock(&LOCK_thread_count); > + > + for (;;) > + { > + /* > + Wait until there is something in the queue to process, or we are asked > + to shut down. > + */ > + thd_proc_info(thd, "Waiting for background binlog tasks"); > + mysql_mutex_lock(&mysql_bin_log.LOCK_binlog_thread); > + for (;;) > + { > + stop= binlog_background_thread_stop; > + queue= binlog_background_thread_queue; > + if (stop || queue) > + break; > + mysql_cond_wait(&mysql_bin_log.COND_binlog_thread, > + &mysql_bin_log.LOCK_binlog_thread); > + } > + /* Grab the queue, if any. */ > + binlog_background_thread_queue= NULL; > + mysql_mutex_unlock(&mysql_bin_log.LOCK_binlog_thread); > + > + /* Process any incoming commit_checkpoint_notify() calls. */ > + while (queue) > + { > + thd_proc_info(thd, "Processing binlog checkpoint notification"); > + /* Grab next pointer first, as mark_xid_done() may free the element. */ > + next= queue->next_in_queue; > + mysql_bin_log.mark_xid_done(queue->binlog_id, true); > + queue= next; > + > + DBUG_EXECUTE_IF("binlog_background_checkpoint_processed", > + DBUG_ASSERT(!debug_sync_set_action( > + thd, > + STRING_WITH_LEN("now SIGNAL binlog_background_checkpoint_processed"))); > + ); > + } > + > + if (stop) > + break; > + } > + > + thd_proc_info(thd, "Stopping binlog background thread"); > + > + mysql_mutex_lock(&LOCK_thread_count); > + delete thd; > + mysql_mutex_unlock(&LOCK_thread_count); > + > + my_thread_end(); > + > + /* Signal that we are (almost) stopped. */ > + mysql_mutex_lock(&mysql_bin_log.LOCK_binlog_thread); > + binlog_background_thread_stop= false; > + mysql_cond_signal(&mysql_bin_log.COND_binlog_thread_end); > + mysql_mutex_unlock(&mysql_bin_log.LOCK_binlog_thread); > + > + return 0; > +} > + > +#ifdef HAVE_PSI_INTERFACE > +static PSI_thread_key key_thread_binlog; > + > +static PSI_thread_info all_binlog_threads[]= > +{ > + { &key_thread_binlog, "binlog_background", PSI_FLAG_GLOBAL}, > +}; > +#endif /* HAVE_PSI_INTERFACE */ > + > +static bool > +start_binlog_background_thread() > +{ > + pthread_t th; > + > +#ifdef HAVE_PSI_INTERFACE > + if (PSI_server) > + PSI_server->register_thread("sql", all_binlog_threads, > + array_elements(all_binlog_threads)); > +#endif > + > + if (mysql_thread_create(key_thread_binlog, &th, NULL, > + binlog_background_thread, NULL)) > + return 1; > + > + binlog_thread_started= true; > + return 0; > +} > + > + > int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > IO_CACHE *first_log, > Format_description_log_event *fdle) > > === modified file 'sql/log.h' > --- a/sql/log.h 2012-09-13 12:31:29 +0000 > +++ b/sql/log.h 2012-09-14 12:44:53 +0000 > @@ -395,8 +395,6 @@ class MYSQL_QUERY_LOG: public MYSQL_LOG > #define BINLOG_COOKIE_IS_DUMMY(c) \ > ( ((ulong)(c)>>1) == BINLOG_COOKIE_DUMMY_ID ) > > -void binlog_checkpoint_callback(void *cookie); > - > class binlog_cache_mngr; > class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG > { > @@ -451,27 +449,6 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > }; > > /* > - A list of struct xid_count_per_binlog is used to keep track of how many > - XIDs are in prepared, but not committed, state in each binlog. And how > - many commit_checkpoint_request()'s are pending. > - > - When count drops to zero in a binlog after rotation, it means that there > - are no more XIDs in prepared state, so that binlog is no longer needed > - for XA crash recovery, and we can log a new binlog checkpoint event. > - > - The list is protected against simultaneous access from multiple > - threads by LOCK_xid_list. > - */ > - struct xid_count_per_binlog : public ilink { > - char *binlog_name; > - uint binlog_name_len; > - ulong binlog_id; > - /* Total prepared XIDs and pending checkpoint requests in this binlog. */ > - long xid_count; > - xid_count_per_binlog(); /* Give link error if constructor used. */ > - }; > - I_List<xid_count_per_binlog> binlog_xid_count_list; > - /* > When this is set, a RESET MASTER is in progress. > > Then we should not write any binlog checkpoints into the binlog (that > @@ -480,7 +457,6 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > checkpoint arrives - when all have arrived, RESET MASTER will complete. > */ > bool reset_master_pending; > - friend void binlog_checkpoint_callback(void *cookie); > > /* LOCK_log and LOCK_index are inited by init_pthread_objects() */ > mysql_mutex_t LOCK_index; > @@ -553,10 +529,35 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > int write_transaction_or_stmt(group_commit_entry *entry); > bool write_transaction_to_binlog_events(group_commit_entry *entry); > void trx_group_commit_leader(group_commit_entry *leader); > - void mark_xid_done(ulong cookie, bool write_checkpoint); > - void mark_xids_active(ulong cookie, uint xid_count); > > public: > + /* > + A list of struct xid_count_per_binlog is used to keep track of how many > + XIDs are in prepared, but not committed, state in each binlog. And how > + many commit_checkpoint_request()'s are pending. > + > + When count drops to zero in a binlog after rotation, it means that there > + are no more XIDs in prepared state, so that binlog is no longer needed > + for XA crash recovery, and we can log a new binlog checkpoint event. > + > + The list is protected against simultaneous access from multiple > + threads by LOCK_xid_list. > + */ > + struct xid_count_per_binlog : public ilink { > + char *binlog_name; > + uint binlog_name_len; > + ulong binlog_id; > + /* Total prepared XIDs and pending checkpoint requests in this binlog. */ > + long xid_count; > + /* For linking in requests to the binlog background thread. */ > + xid_count_per_binlog *next_in_queue; > + xid_count_per_binlog(); /* Give link error if constructor used. */ > + }; > + I_List<xid_count_per_binlog> binlog_xid_count_list; > + mysql_mutex_t LOCK_binlog_thread; > + mysql_cond_t COND_binlog_thread; > + mysql_cond_t COND_binlog_thread_end; > + > using MYSQL_LOG::generate_name; > using MYSQL_LOG::is_open; > > @@ -712,6 +713,8 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > bool appendv(const char* buf,uint len,...); > bool append(Log_event* ev); > > + void mark_xids_active(ulong cookie, uint xid_count); > + void mark_xid_done(ulong cookie, bool write_checkpoint); > void make_log_name(char* buf, const char* log_ident); > bool is_active(const char* log_file_name); > bool can_purge_log(const char *log_file_name); > > === modified file 'sql/mysqld.cc' > --- a/sql/mysqld.cc 2012-09-13 12:31:29 +0000 > +++ b/sql/mysqld.cc 2012-09-14 12:44:53 +0000 > @@ -724,6 +724,7 @@ PSI_mutex_key key_LOCK_des_key_file; > #endif /* HAVE_OPENSSL */ > > PSI_mutex_key key_BINLOG_LOCK_index, key_BINLOG_LOCK_xid_list, > + key_BINLOG_LOCK_binlog_thread, > key_delayed_insert_mutex, key_hash_filo_lock, key_LOCK_active_mi, > key_LOCK_connection_count, key_LOCK_crypt, key_LOCK_delayed_create, > key_LOCK_delayed_insert, key_LOCK_delayed_status, key_LOCK_error_log, > @@ -766,6 +767,7 @@ static PSI_mutex_info all_server_mutexes > > { &key_BINLOG_LOCK_index, "MYSQL_BIN_LOG::LOCK_index", 0}, > { &key_BINLOG_LOCK_xid_list, "MYSQL_BIN_LOG::LOCK_xid_list", 0}, > + { &key_BINLOG_LOCK_binlog_thread, "MYSQL_BIN_LOG::LOCK_binlog_thread", 0}, > { &key_RELAYLOG_LOCK_index, "MYSQL_RELAY_LOG::LOCK_index", 0}, > { &key_delayed_insert_mutex, "Delayed_insert::mutex", 0}, > { &key_hash_filo_lock, "hash_filo::lock", 0}, > @@ -834,6 +836,7 @@ PSI_cond_key key_PAGE_cond, key_COND_act > #endif /* HAVE_MMAP */ > > PSI_cond_key key_BINLOG_COND_xid_list, key_BINLOG_update_cond, > + key_BINLOG_COND_binlog_thread, key_BINLOG_COND_binlog_thread_end, > key_COND_cache_status_changed, key_COND_manager, > key_COND_rpl_status, key_COND_server_started, > key_delayed_insert_cond, key_delayed_insert_cond_client, > @@ -863,6 +866,8 @@ static PSI_cond_info all_server_conds[]= > #endif /* HAVE_MMAP */ > { &key_BINLOG_COND_xid_list, "MYSQL_BIN_LOG::COND_xid_list", 0}, > { &key_BINLOG_update_cond, "MYSQL_BIN_LOG::update_cond", 0}, > + { &key_BINLOG_COND_binlog_thread, "MYSQL_BIN_LOG::COND_binlog_thread", 0}, > + { &key_BINLOG_COND_binlog_thread_end, "MYSQL_BIN_LOG::COND_binlog_thread_end", 0}, > { &key_BINLOG_COND_queue_busy, "MYSQL_BIN_LOG::COND_queue_busy", 0}, > { &key_RELAYLOG_update_cond, "MYSQL_RELAY_LOG::update_cond", 0}, > { &key_RELAYLOG_COND_queue_busy, "MYSQL_RELAY_LOG::COND_queue_busy", 0}, > > === modified file 'sql/mysqld.h' > --- a/sql/mysqld.h 2012-09-13 12:31:29 +0000 > +++ b/sql/mysqld.h 2012-09-14 12:44:53 +0000 > @@ -226,6 +226,7 @@ extern PSI_mutex_key key_LOCK_des_key_fi > #endif > > extern PSI_mutex_key key_BINLOG_LOCK_index, key_BINLOG_LOCK_xid_list, > + key_BINLOG_LOCK_binlog_thread, > key_delayed_insert_mutex, key_hash_filo_lock, key_LOCK_active_mi, > key_LOCK_connection_count, key_LOCK_crypt, key_LOCK_delayed_create, > key_LOCK_delayed_insert, key_LOCK_delayed_status, key_LOCK_error_log, > @@ -257,6 +258,7 @@ extern PSI_cond_key key_PAGE_cond, key_C > #endif /* HAVE_MMAP */ > > extern PSI_cond_key key_BINLOG_COND_xid_list, key_BINLOG_update_cond, > + key_BINLOG_COND_binlog_thread, key_BINLOG_COND_binlog_thread_end, > key_COND_cache_status_changed, key_COND_manager, > key_COND_rpl_status, key_COND_server_started, > key_delayed_insert_cond, key_delayed_insert_cond_client, > > === modified file 'sql/rpl_rli.cc' > --- a/sql/rpl_rli.cc 2012-09-13 12:31:29 +0000 > +++ b/sql/rpl_rli.cc 2012-09-14 12:44:53 +0000 > @@ -58,6 +58,7 @@ Relay_log_info::Relay_log_info(bool is_s > { > DBUG_ENTER("Relay_log_info::Relay_log_info"); > > + relay_log.is_relay_log= TRUE; > #ifdef HAVE_PSI_INTERFACE > relay_log.set_psi_keys(key_RELAYLOG_LOCK_index, > key_RELAYLOG_update_cond, > @@ -206,8 +207,6 @@ a file name for --relay-log-index option > name_warning_sent= 1; > } > > - rli->relay_log.is_relay_log= TRUE; > - > /* > note, that if open() fails, we'll still have index file open > but a destructor will take care of that > > === modified file 'sql/sql_class.h' > --- a/sql/sql_class.h 2012-09-13 12:31:29 +0000 > +++ b/sql/sql_class.h 2012-09-14 12:44:53 +0000 > @@ -1244,7 +1244,8 @@ enum enum_thread_type > SYSTEM_THREAD_SLAVE_SQL= 4, > SYSTEM_THREAD_NDBCLUSTER_BINLOG= 8, > SYSTEM_THREAD_EVENT_SCHEDULER= 16, > - SYSTEM_THREAD_EVENT_WORKER= 32 > + SYSTEM_THREAD_EVENT_WORKER= 32, > + SYSTEM_THREAD_BINLOG_BACKGROUND= 64 > }; > > inline char const * > > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- a/storage/innobase/handler/ha_innodb.cc 2012-09-13 12:31:29 +0000 > +++ b/storage/innobase/handler/ha_innodb.cc 2012-09-14 12:44:53 +0000 > @@ -106,6 +106,7 @@ static ulong commit_threads = 0; > static mysql_mutex_t commit_threads_m; > static mysql_cond_t commit_cond; > static mysql_mutex_t commit_cond_m; > +static mysql_mutex_t pending_checkpoint_mutex; > static bool innodb_inited = 0; > > #define INSIDE_HA_INNOBASE_CC > @@ -222,11 +223,13 @@ static mysql_pfs_key_t innobase_share_mu > static mysql_pfs_key_t commit_threads_m_key; > static mysql_pfs_key_t commit_cond_mutex_key; > static mysql_pfs_key_t commit_cond_key; > +static mysql_pfs_key_t pending_checkpoint_mutex_key; > > static PSI_mutex_info all_pthread_mutexes[] = { > {&commit_threads_m_key, "commit_threads_m", 0}, > {&commit_cond_mutex_key, "commit_cond_mutex", 0}, > - {&innobase_share_mutex_key, "innobase_share_mutex", 0} > + {&innobase_share_mutex_key, "innobase_share_mutex", 0}, > + {&pending_checkpoint_mutex_key, "pending_checkpoint_mutex", 0} > }; > > static PSI_cond_info all_innodb_conds[] = { > @@ -2592,6 +2595,9 @@ innobase_init( > mysql_mutex_init(commit_cond_mutex_key, > &commit_cond_m, MY_MUTEX_INIT_FAST); > mysql_cond_init(commit_cond_key, &commit_cond, NULL); > + mysql_mutex_init(pending_checkpoint_mutex_key, > + &pending_checkpoint_mutex, > + MY_MUTEX_INIT_FAST); > innodb_inited= 1; > #ifdef MYSQL_DYNAMIC_PLUGIN > if (innobase_hton != p) { > @@ -2639,6 +2645,7 @@ innobase_end( > mysql_mutex_destroy(&commit_threads_m); > mysql_mutex_destroy(&commit_cond_m); > mysql_cond_destroy(&commit_cond); > + mysql_mutex_destroy(&pending_checkpoint_mutex); > } > > DBUG_RETURN(err); > @@ -3008,6 +3015,16 @@ innobase_rollback_trx( > DBUG_RETURN(convert_error_code_to_mysql(error, 0, NULL)); > } > > + > +struct pending_checkpoint { > + struct pending_checkpoint *next; > + handlerton *hton; > + void *cookie; > + ib_uint64_t lsn; > +}; > +static struct pending_checkpoint *pending_checkpoint_list; > +static struct pending_checkpoint *pending_checkpoint_list_end; > + > /*****************************************************************//** > Handle a commit checkpoint request from server layer. > We simply flush the redo log immediately and do the notify call.*/ > @@ -3017,8 +3034,113 @@ innobase_checkpoint_request( > handlerton *hton, > void *cookie) > { > - log_buffer_flush_to_disk(); > - commit_checkpoint_notify_ha(hton, cookie); > + ib_uint64_t lsn; > + ib_uint64_t flush_lsn; > + struct pending_checkpoint * entry; > + > + /* Do the allocation outside of lock to reduce contention. The normal > + case is that not everything is flushed, so we will need to enqueue. */ > + entry = static_cast<struct pending_checkpoint *> > + (my_malloc(sizeof(*entry), MYF(MY_WME))); > + if (!entry) { > + sql_print_error("Failed to allocate %u bytes." > + " Commit checkpoint will be skipped.", > + static_cast<unsigned>(sizeof(*entry))); > + return; > + } > + > + entry->next = NULL; > + entry->hton = hton; > + entry->cookie = cookie; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + lsn = log_get_lsn(); > + flush_lsn = log_get_flush_lsn(); > + if (lsn > flush_lsn) { > + /* Put the request in queue. > + When the log gets flushed past the lsn, we will remove the > + entry from the queue and notify the upper layer. */ > + entry->lsn = lsn; > + if (pending_checkpoint_list_end) { > + pending_checkpoint_list_end->next = entry; > + } else { > + pending_checkpoint_list = entry; > + } > + pending_checkpoint_list_end = entry; > + entry = NULL; > + } > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (entry) { > + /* We are already flushed. Notify the checkpoint immediately. */ > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + my_free(entry); > + } > +} > + > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +extern "C" UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn) /*!< in: LSN flushed to disk */ > +{ > + struct pending_checkpoint * pending; > + struct pending_checkpoint * entry; > + struct pending_checkpoint * last_ready; > + > + /* It is safe to do a quick check for NULL first without lock. > + Even if we should race, we will at most skip one checkpoint and > + take the next one, which is harmless. */ > + if (!pending_checkpoint_list) > + return; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + pending = pending_checkpoint_list; > + if (!pending) > + { > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + return; > + } > + > + last_ready = NULL; > + for (entry = pending; entry != NULL; entry = entry -> next) > + { > + if (entry->lsn > flush_lsn) > + break; > + last_ready = entry; > + } > + > + if (last_ready) > + { > + /* We found some pending checkpoints that are now flushed to > + disk. So remove them from the list. */ > + pending_checkpoint_list = entry; > + if (!entry) > + pending_checkpoint_list_end = NULL; > + } > + > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (!last_ready) > + return; > + > + /* Now that we have released the lock, notify upper layer about all > + commit checkpoints that have now completed. */ > + for (;;) { > + entry = pending; > + pending = pending->next; > + > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + > + my_free(entry); > + if (entry == last_ready) > + break; > + } > } > > /*****************************************************************//** > > === modified file 'storage/innobase/include/ha_prototypes.h' > --- a/storage/innobase/include/ha_prototypes.h 2011-04-26 17:55:52 +0000 > +++ b/storage/innobase/include/ha_prototypes.h 2012-09-14 12:44:53 +0000 > @@ -136,6 +136,17 @@ innobase_mysql_print_thd( > uint max_query_len); /*!< in: max query length to print, or 0 to > use the default max length */ > > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn); /*!< in: LSN flushed to disk */ > + > /**************************************************************//** > Converts a MySQL type to an InnoDB type. Note that this function returns > the 'mtype' of InnoDB. InnoDB differentiates between MySQL's old <= 4.1 > > === modified file 'storage/innobase/include/log0log.h' > --- a/storage/innobase/include/log0log.h 2012-06-07 13:44:26 +0000 > +++ b/storage/innobase/include/log0log.h 2012-09-14 12:44:53 +0000 > @@ -151,6 +151,13 @@ UNIV_INLINE > ib_uint64_t > log_get_lsn(void); > /*=============*/ > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void); > +/*=============*/ > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/innobase/include/log0log.ic' > --- a/storage/innobase/include/log0log.ic 2011-04-05 07:18:43 +0000 > +++ b/storage/innobase/include/log0log.ic 2012-09-14 12:44:53 +0000 > @@ -411,6 +411,25 @@ log_get_lsn(void) > return(lsn); > } > > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void) > +/*=============*/ > +{ > + ib_uint64_t lsn; > + > + mutex_enter(&(log_sys->mutex)); > + > + lsn = log_sys->flushed_to_disk_lsn; > + > + mutex_exit(&(log_sys->mutex)); > + > + return(lsn); > +} > + > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/innobase/log/log0log.c' > --- a/storage/innobase/log/log0log.c 2012-03-21 03:48:12 +0000 > +++ b/storage/innobase/log/log0log.c 2012-09-14 12:44:53 +0000 > @@ -1353,6 +1353,8 @@ log_write_up_to( > ulint loop_count = 0; > #endif /* UNIV_DEBUG */ > ulint unlock; > + ib_uint64_t write_lsn; > + ib_uint64_t flush_lsn; > > if (recv_no_ibuf_operations) { > /* Recovery is running and no operations on the log files are > @@ -1530,8 +1532,13 @@ log_write_up_to( > > log_flush_do_unlocks(unlock); > > + write_lsn = log_sys->write_lsn; > + flush_lsn = log_sys->flushed_to_disk_lsn; > + > mutex_exit(&(log_sys->mutex)); > > + innobase_mysql_log_notify(write_lsn, flush_lsn); > + > return; > > do_waits: > > === modified file 'storage/xtradb/handler/ha_innodb.cc' > --- a/storage/xtradb/handler/ha_innodb.cc 2012-09-13 12:31:29 +0000 > +++ b/storage/xtradb/handler/ha_innodb.cc 2012-09-14 12:44:53 +0000 > @@ -120,6 +120,7 @@ static ulong commit_threads = 0; > static mysql_mutex_t commit_threads_m; > static mysql_cond_t commit_cond; > static mysql_mutex_t commit_cond_m; > +static mysql_mutex_t pending_checkpoint_mutex; > static bool innodb_inited = 0; > > > @@ -253,11 +254,13 @@ static mysql_pfs_key_t innobase_share_mu > static mysql_pfs_key_t commit_threads_m_key; > static mysql_pfs_key_t commit_cond_mutex_key; > static mysql_pfs_key_t commit_cond_key; > +static mysql_pfs_key_t pending_checkpoint_mutex_key; > > static PSI_mutex_info all_pthread_mutexes[] = { > {&commit_threads_m_key, "commit_threads_m", 0}, > {&commit_cond_mutex_key, "commit_cond_mutex", 0}, > - {&innobase_share_mutex_key, "innobase_share_mutex", 0} > + {&innobase_share_mutex_key, "innobase_share_mutex", 0}, > + {&pending_checkpoint_mutex_key, "pending_checkpoint_mutex", 0} > }; > > static PSI_cond_info all_innodb_conds[] = { > @@ -3060,6 +3063,9 @@ innobase_init( > mysql_mutex_init(commit_cond_mutex_key, > &commit_cond_m, MY_MUTEX_INIT_FAST); > mysql_cond_init(commit_cond_key, &commit_cond, NULL); > + mysql_mutex_init(pending_checkpoint_mutex_key, > + &pending_checkpoint_mutex, > + MY_MUTEX_INIT_FAST); > innodb_inited= 1; > #ifdef MYSQL_DYNAMIC_PLUGIN > if (innobase_hton != p) { > @@ -3107,6 +3113,7 @@ innobase_end( > mysql_mutex_destroy(&commit_threads_m); > mysql_mutex_destroy(&commit_cond_m); > mysql_cond_destroy(&commit_cond); > + mysql_mutex_destroy(&pending_checkpoint_mutex); > } > > DBUG_RETURN(err); > @@ -3500,6 +3507,16 @@ innobase_rollback_trx( > DBUG_RETURN(convert_error_code_to_mysql(error, 0, NULL)); > } > > + > +struct pending_checkpoint { > + struct pending_checkpoint *next; > + handlerton *hton; > + void *cookie; > + ib_uint64_t lsn; > +}; > +static struct pending_checkpoint *pending_checkpoint_list; > +static struct pending_checkpoint *pending_checkpoint_list_end; > + > /*****************************************************************//** > Handle a commit checkpoint request from server layer. > We simply flush the redo log immediately and do the notify call.*/ > @@ -3509,8 +3526,113 @@ innobase_checkpoint_request( > handlerton *hton, > void *cookie) > { > - log_buffer_flush_to_disk(); > - commit_checkpoint_notify_ha(hton, cookie); > + ib_uint64_t lsn; > + ib_uint64_t flush_lsn; > + struct pending_checkpoint * entry; > + > + /* Do the allocation outside of lock to reduce contention. The normal > + case is that not everything is flushed, so we will need to enqueue. */ > + entry = static_cast<struct pending_checkpoint *> > + (my_malloc(sizeof(*entry), MYF(MY_WME))); > + if (!entry) { > + sql_print_error("Failed to allocate %u bytes." > + " Commit checkpoint will be skipped.", > + static_cast<unsigned>(sizeof(*entry))); > + return; > + } > + > + entry->next = NULL; > + entry->hton = hton; > + entry->cookie = cookie; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + lsn = log_get_lsn(); > + flush_lsn = log_get_flush_lsn(); > + if (lsn > flush_lsn) { > + /* Put the request in queue. > + When the log gets flushed past the lsn, we will remove the > + entry from the queue and notify the upper layer. */ > + entry->lsn = lsn; > + if (pending_checkpoint_list_end) { > + pending_checkpoint_list_end->next = entry; > + } else { > + pending_checkpoint_list = entry; > + } > + pending_checkpoint_list_end = entry; > + entry = NULL; > + } > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (entry) { > + /* We are already flushed. Notify the checkpoint immediately. */ > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + my_free(entry); > + } > +} > + > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +extern "C" UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn) /*!< in: LSN flushed to disk */ > +{ > + struct pending_checkpoint * pending; > + struct pending_checkpoint * entry; > + struct pending_checkpoint * last_ready; > + > + /* It is safe to do a quick check for NULL first without lock. > + Even if we should race, we will at most skip one checkpoint and > + take the next one, which is harmless. */ > + if (!pending_checkpoint_list) > + return; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + pending = pending_checkpoint_list; > + if (!pending) > + { > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + return; > + } > + > + last_ready = NULL; > + for (entry = pending; entry != NULL; entry = entry -> next) > + { > + if (entry->lsn > flush_lsn) > + break; > + last_ready = entry; > + } > + > + if (last_ready) > + { > + /* We found some pending checkpoints that are now flushed to > + disk. So remove them from the list. */ > + pending_checkpoint_list = entry; > + if (!entry) > + pending_checkpoint_list_end = NULL; > + } > + > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (!last_ready) > + return; > + > + /* Now that we have released the lock, notify upper layer about all > + commit checkpoints that have now completed. */ > + for (;;) { > + entry = pending; > + pending = pending->next; > + > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + > + my_free(entry); > + if (entry == last_ready) > + break; > + } > } > > /*****************************************************************//** > > === modified file 'storage/xtradb/include/ha_prototypes.h' > --- a/storage/xtradb/include/ha_prototypes.h 2012-02-21 19:51:56 +0000 > +++ b/storage/xtradb/include/ha_prototypes.h 2012-09-14 12:44:53 +0000 > @@ -136,6 +136,17 @@ innobase_mysql_print_thd( > uint max_query_len); /*!< in: max query length to print, or 0 to > use the default max length */ > > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn); /*!< in: LSN flushed to disk */ > + > /**************************************************************//** > Converts a MySQL type to an InnoDB type. Note that this function returns > the 'mtype' of InnoDB. InnoDB differentiates between MySQL's old <= 4.1 > > === modified file 'storage/xtradb/include/log0log.h' > --- a/storage/xtradb/include/log0log.h 2012-08-27 16:13:17 +0000 > +++ b/storage/xtradb/include/log0log.h 2012-09-14 12:44:53 +0000 > @@ -151,6 +151,13 @@ UNIV_INLINE > ib_uint64_t > log_get_lsn(void); > /*=============*/ > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void); > +/*=============*/ > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/xtradb/include/log0log.ic' > --- a/storage/xtradb/include/log0log.ic 2011-07-14 19:22:41 +0000 > +++ b/storage/xtradb/include/log0log.ic 2012-09-14 12:44:53 +0000 > @@ -411,6 +411,25 @@ log_get_lsn(void) > return(lsn); > } > > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void) > +/*=============*/ > +{ > + ib_uint64_t lsn; > + > + mutex_enter(&(log_sys->mutex)); > + > + lsn = log_sys->flushed_to_disk_lsn; > + > + mutex_exit(&(log_sys->mutex)); > + > + return(lsn); > +} > + > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/xtradb/log/log0log.c' > --- a/storage/xtradb/log/log0log.c 2012-08-27 16:13:17 +0000 > +++ b/storage/xtradb/log/log0log.c 2012-09-14 12:44:53 +0000 > @@ -1390,6 +1390,8 @@ log_write_up_to( > ulint loop_count = 0; > #endif /* UNIV_DEBUG */ > ulint unlock; > + ib_uint64_t write_lsn; > + ib_uint64_t flush_lsn; > > if (recv_no_ibuf_operations) { > /* Recovery is running and no operations on the log files are > @@ -1568,8 +1570,13 @@ log_write_up_to( > > log_flush_do_unlocks(unlock); > > + write_lsn = log_sys->write_lsn; > + flush_lsn = log_sys->flushed_to_disk_lsn; > + > mutex_exit(&(log_sys->mutex)); > > + innobase_mysql_log_notify(write_lsn, flush_lsn); > + > return; > > do_waits: > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 2

[Maria-developers] review of the MDEV-377 Name support for dynamic columns
by Sergei Golubchik 14 Dec '12

14 Dec '12

Hi, Sanja, See my review comments below: > === modified file 'include/ma_dyncol.h' > --- include/ma_dyncol.h 2011-09-22 09:04:00 +0000 > +++ include/ma_dyncol.h 2012-11-12 15:11:23 +0000 > @@ -39,6 +39,12 @@ > */ > #define MAX_DYNAMIC_COLUMN_LENGTH 0X1FFFFFFFL > > +/* > + Limits of implementation > +*/ > +#define MAX_NAME_LENGTH 255 If we store only offsets, this limit can go away > +#define MAX_TOTAL_NAME_LENGTH 65535 I'd suggest to store the offset length in the header, just as you do for data offsets. You might hard-code the fixed offset of 2, but if it's in the header, you can later change it, if needed. > + > /* NO and OK is the same used just to show semantics */ > #define ER_DYNCOL_NO ER_DYNCOL_OK > > @@ -81,6 +88,7 @@ struct st_dynamic_column_value > struct { > LEX_STRING value; > CHARSET_INFO *charset; > + my_bool nonfreeable; Yuck. That doesn't look correct. Dynamic column code does not allocate or frees values. The concept of "nonfreeable" values simply has no place in the dynamic column universe. I realize that ha_cassandra needs to know whether to free certain strings or not. But it's the internal problem of ha_cassandra, it has nothing to do with dynaic columns, besides the fact that these certain strings are sometines stored in the struct st_dynamic_column_value. > } string; > struct { > decimal_digit_t buffer[DECIMAL_BUFF_LENGTH]; > @@ -103,6 +111,20 @@ dynamic_column_create_many(DYNAMIC_COLUM > DYNAMIC_COLUMN_VALUE *values); > > enum enum_dyncol_func_result > +dynamic_column_create_many_fmt(DYNAMIC_COLUMN *str, > + uint column_count, > + uchar *column_keys, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool names); I think this should rather be dynamic_column_create_many_named(DYNAMIC_COLUMN *str, uint column_count, LEX_STRING *column_keys, DYNAMIC_COLUMN_VALUE *values); > +enum enum_dyncol_func_result > +dynamic_column_create_many_internal_fmt(DYNAMIC_COLUMN *str, > + uint column_count, > + void *column_keys, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool new_str, > + my_bool string_keys); *internal* should not be in include/, it's not part of the API > + > +enum enum_dyncol_func_result > dynamic_column_update(DYNAMIC_COLUMN *org, uint column_nr, > DYNAMIC_COLUMN_VALUE *value); > enum enum_dyncol_func_result > @@ -110,16 +132,30 @@ dynamic_column_update_many(DYNAMIC_COLUM > uint add_column_count, > uint *column_numbers, > DYNAMIC_COLUMN_VALUE *values); > +enum enum_dyncol_func_result > +dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str, > + uint add_column_count, > + void *column_keys, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool string_keys); same as above, make it dynamic_column_update_many_named > > enum enum_dyncol_func_result > dynamic_column_delete(DYNAMIC_COLUMN *org, uint column_nr); > > enum enum_dyncol_func_result > dynamic_column_exists(DYNAMIC_COLUMN *org, uint column_nr); > +enum enum_dyncol_func_result > +dynamic_column_exists_str(DYNAMIC_COLUMN *str, LEX_STRING *name); > +enum enum_dyncol_func_result > +dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys); remove the second one, and rename the first s/_str/_named/ > > /* List of not NULL columns */ > enum enum_dyncol_func_result > dynamic_column_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); > +enum enum_dyncol_func_result > +dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr); > +enum enum_dyncol_func_result > +dynamic_column_list_fmt(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array, my_bool string_keys); Same. > > /* > if the column do not exists it is NULL > @@ -127,10 +163,36 @@ dynamic_column_list(DYNAMIC_COLUMN *org, > enum enum_dyncol_func_result > dynamic_column_get(DYNAMIC_COLUMN *org, uint column_nr, > DYNAMIC_COLUMN_VALUE *store_it_here); > +enum enum_dyncol_func_result > +dynamic_column_get_str(DYNAMIC_COLUMN *str, LEX_STRING *name, > + DYNAMIC_COLUMN_VALUE *store_it_here); s/_str/_named > + > +my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str); > + > +enum enum_dyncol_func_result > +dynamic_column_check(DYNAMIC_COLUMN *str); > + > +enum enum_dyncol_func_result > +dynamic_column_json(DYNAMIC_COLUMN *str, DYNAMIC_STRING *json); > > #define dynamic_column_initialize(A) memset((A), 0, sizeof(*(A))) > #define dynamic_column_column_free(V) dynstr_free(V) > > +/* conversion of values to 3 base types */ > +enum enum_dyncol_func_result > +dynamic_column_val_str(DYNAMIC_STRING *str, DYNAMIC_COLUMN_VALUE *val, > + CHARSET_INFO *cs, my_bool quote); instead of my_bool quote, better pass a char quote_char. when it's 0 - no quoting. Otherwise it can be " or ` To make it less error prone, you can create an enum NOT_QUOTED=0, QUOTED='`', ANSI_QUOTED='"' > +enum enum_dyncol_func_result > +dynamic_column_val_long(longlong *ll, DYNAMIC_COLUMN_VALUE *val); > +enum enum_dyncol_func_result > +dynamic_column_val_double(double *dbl, DYNAMIC_COLUMN_VALUE *val); > + > + > +enum enum_dyncol_func_result > +dynamic_column_vals(DYNAMIC_COLUMN *str, > + DYNAMIC_ARRAY *names, DYNAMIC_ARRAY *vals, > + char **free_names); Eh. may be dynamic_column_unpack() ? > + > /*************************************************************************** > Internal functions, don't use if you don't know what you are doing... > ***************************************************************************/ > > === modified file 'mysql-test/t/dyncol.test' > --- mysql-test/t/dyncol.test 2012-02-29 20:55:04 +0000 > +++ mysql-test/t/dyncol.test 2012-11-12 15:11:23 +0000 > @@ -550,3 +550,125 @@ select hex(COLUMN_CREATE(0, COLUMN_GET(@ > > select hex(COLUMN_CREATE(0, COLUMN_GET(COLUMN_CREATE(0, 0.0 as decimal), 0 as decimal))); > select hex(COLUMN_CREATE(0, 0.0 as decimal)); > + > +--echo # > +--echo # test of symbolic names > +--echo # > +--echo # creation test (names) > +set names utf8; > +select hex(column_create("адын", 1212)); > +select hex(column_create("1212", 1212)); > +select hex(column_create(1212, 2, "www", 3)); > +select hex(column_create("1212", 2, "www", 3)); > +select hex(column_create("1212", 2, 3, 3)); > +select hex(column_create("1212", 2, "адын", 1, 3, 3)); to test quoting, add a column with a ` (backtick) in the name and few other strange names - with a single quote, double quote, backslash, space, dot, comma, whatever > +set names default; ... > === modified file 'sql/sql_yacc.yy' > --- sql/sql_yacc.yy 2012-06-13 23:28:47 +0000 > +++ sql/sql_yacc.yy 2012-11-12 15:11:23 +0000 > @@ -8799,6 +8801,13 @@ function_call_nonkeyword: > MYSQL_YYABORT; > } > | > + COLUMN_CHECK_SYM '(' expr ')' > + { > + $$= new (YYTHD->mem_root) Item_func_dyncol_check($3); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + | This should be removed completely. See, for example, how EXP function is created (hint: in item_create.cc) and do the same > COLUMN_EXISTS_SYM '(' expr ',' expr ')' > { > $$= new (YYTHD->mem_root) Item_func_dyncol_exists($3, $5); > @@ -8820,6 +8829,13 @@ function_call_nonkeyword: > MYSQL_YYABORT; > } > | > + COLUMN_JSON_SYM '(' expr ')' > + { > + $$= create_func_dyncol_json(YYTHD, $3); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + | same. please, remove the dedicated rule for this and most of other COLUMN_xxx functions. > COLUMN_GET_SYM '(' expr ',' expr AS cast_type ')' > { > LEX *lex= Lex; > > === modified file 'mysys/ma_dyncol.c' > --- mysys/ma_dyncol.c 2011-11-22 17:04:38 +0000 > +++ mysys/ma_dyncol.c 2012-11-12 15:11:23 +0000 > @@ -37,19 +41,42 @@ > */ > /* mask to get above bits */ > #define DYNCOL_FLG_OFFSET 3 > +#define DYNCOL_FLG_NAMES 4 > /* All known flags mask */ > -#define DYNCOL_FLG_KNOWN 3 > +#define DYNCOL_FLG_KNOWN 7 > + > +/* formats */ > +#define DYNCOL_FMT_NUM 0 > +#define DYNCOL_FMT_STR 1 why not an enum? > > /* dynamic column size reserve */ > #define DYNCOL_SYZERESERVE 80 > > +#define DYNCOL_OFFSET_ERROR 0xffffffff > + > /* length of fixed string header 1 byte - flags, 2 bytes - columns counter */ > #define FIXED_HEADER_SIZE 3 > +/* > + length of fixed string header with names > + 1 byte - flags, 2 bytes - columns counter, 2 bytes - name pool size > +*/ > +#define FIXED_HEADER_SIZE_NM 5 > > #define COLUMN_NUMBER_SIZE 2 > +/* 1 byte name length + 2 bytes offset from the name pool */ > +#define COLUMN_NAMEPTR_SIZE 3 > > #define MAX_OFFSET_LENGTH 5 > > +#define DYNCOL_NUM_CHAR 6 > + > +my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str) > +{ > + if (str->length < 1) > + return FALSE; > + return test(str->str[0] & DYNCOL_FLG_NAMES); > +} > + > static enum enum_dyncol_func_result > dynamic_column_time_store(DYNAMIC_COLUMN *str, > MYSQL_TIME *value); > @@ -62,6 +89,311 @@ dynamic_column_time_read_internal(DYNAMI > static enum enum_dyncol_func_result > dynamic_column_date_read_internal(DYNAMIC_COLUMN_VALUE *store_it_here, > uchar *data, size_t length); > +static enum enum_dyncol_func_result > +dynamic_column_get_internal(DYNAMIC_COLUMN *str, > + DYNAMIC_COLUMN_VALUE *store_it_here, > + uint num_key, LEX_STRING *str_key); > +static enum enum_dyncol_func_result > +dynamic_column_exists_internal(DYNAMIC_COLUMN *str, uint num_key, > + LEX_STRING *str_key); > +enum enum_dyncol_func_result > +dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str, > + uint add_column_count, > + void *column_keys, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool string_keys); > +static int plan_sort_num(const void *a, const void *b); > +static int plan_sort_str(const void *a, const void *b); > + > +/* > + Structure to hold information about dynamic columns record and > + iterate through it. > +*/ > + > +struct st_dyn_header > +{ > + uchar *header, *nmpool, *dtpool, *data_end; > + size_t offset_size; > + size_t entry_size; > + size_t header_size; > + size_t nmpool_size; > + size_t data_size; > + /* DYNCOL_FMT_NUM - numeric columns, DYNCOL_FMT_STR - column names */ > + uint format; > + uint column_count; > + > + uchar *entry, *data, *name; > + size_t offset; > + uint length; > + enum enum_dynamic_column_type type; > +}; > + > +typedef struct st_dyn_header DYN_HEADER; > + > +static inline my_bool read_fixed_header(DYN_HEADER *hdr, > + DYNAMIC_COLUMN *str); > +static void set_fixed_header(DYNAMIC_COLUMN *str, > + uint offset_size, > + uint column_count); > +static my_bool type_and_offset_store(uchar *place, size_t offset_size, > + DYNAMIC_COLUMN_TYPE type, > + size_t offset); > + > +/* > + Calculate entry size (E) and header size (H) by offset size (O) and column > + count (C) and fixed part of entry size (F). > +*/ > + > +#define calc_param(E,H,F,O,C) do { \ > + (*(E))= (O) + F; \ > + (*(H))= (*(E)) * (C); \ > +}while(0); > + > + > +/** > + Name pool size functions, for numeric format it is 0 > +*/ > + > +size_t name_size_num(void *keys __attribute__((unused)), > + uint i __attribute__((unused))) > +{ > + return 0; > +} > + > + > +/** > + Name pool size functions. > +*/ > +size_t name_size_str(void *keys, uint i) > +{ > + return ((LEX_STRING *) keys)[i].length; > +} > + > + > +/** > + Comparator function for references on column numbers for qsort > + (numeric format) > +*/ > + > +static int column_sort_num(const void *a, const void *b) > +{ > + return **((uint **)a) - **((uint **)b); > +} > + > + > +/** > + Comparator function for references on column numbers for qsort > + (names format) > +*/ > + > +static int column_sort_str(const void *a, const void *b) > +{ > + LEX_STRING *s1= *((LEX_STRING **)a); > + LEX_STRING *s2= *((LEX_STRING **)b); > + int rc= s1->length - s2->length; > + if (rc == 0) > + rc= memcmp((void *)s1->str, (void *)s2->str, (size_t) s1->length); > + return rc; > +} > + > + > +/** > + Check limit function (numeric format) > +*/ > + > +static my_bool check_limit_num(const void *val) > +{ > + return **((uint **)val) > UINT_MAX16; > +} > + > + > +/** > + Check limit function (names format) > +*/ > + > +static my_bool check_limit_str(const void *val) > +{ > + return (*((LEX_STRING **)val))->length > MAX_NAME_LENGTH; > +} > + > + > +/** > + Write numeric format static header part. > +*/ > + > +void set_fixed_header_num(DYNAMIC_COLUMN *str, DYN_HEADER *hdr) > +{ > + set_fixed_header(str, hdr->offset_size, hdr->column_count); > + hdr->header= (uchar *)str->str + FIXED_HEADER_SIZE; > + hdr->nmpool= hdr->dtpool= hdr->header + hdr->header_size; > +} > + > + > +/** > + Write names format static header part. > +*/ > + > +void set_fixed_header_str(DYNAMIC_COLUMN *str, DYN_HEADER *hdr) > +{ > + set_fixed_header(str, hdr->offset_size, hdr->column_count); There's no need to write the column count there, you save two bytes per record. a column count = (offset of the first name - size of the fized header) / (size of the fixed per column header) > + str->str[0]|= DYNCOL_FLG_NAMES; > + int2store(str->str + 3, hdr->nmpool_size); > + hdr->header= (uchar *)str->str + FIXED_HEADER_SIZE_NM; > + hdr->nmpool= hdr->header + hdr->header_size; > + hdr->dtpool= hdr->nmpool + hdr->nmpool_size; > +} > + > + > +/** > + Write numeric format header entry > + 2 bytes - column number > + 1-4 bytes - data offset combined with type > + > + @param hdr descriptor of dynamic column record > + @param column_key pointer to uint (column number) > + @param value value which will be written (only type used) > + @param offset offset of the data > +*/ > + > +my_bool put_header_entry_num(DYN_HEADER *hdr, this is and many other functions should be static > + void *column_key, > + DYNAMIC_COLUMN_VALUE *value, > + size_t offset) > +{ > + uint *column_number= (uint *)column_key; > + int2store(hdr->entry, *column_number); > + DBUG_ASSERT(hdr->nmpool_size == 0); > + if (type_and_offset_store(hdr->entry, hdr->offset_size, > + value->type, > + offset)) > + return TRUE; > + hdr->entry= hdr->entry + hdr->entry_size; > + return FALSE; > +} > + > + > +/** > + Write names format header entry > + 1 byte - name length > + 2 bytes - name offset in the name pool as discussed on the irc, please store either lengths or offsets, but not both > + 1-4 bytes - data offset combined with type > + > + @param hdr descriptor of dynamic column record > + @param column_key pointer to LEX_STRING (column name) > + @param value value which will be written (only type used) > + @param offset offset of the data > +*/ > + > +my_bool put_header_entry_str(DYN_HEADER *hdr, > + void *column_key, > + DYNAMIC_COLUMN_VALUE *value, > + size_t offset) > +{ > + LEX_STRING *column_name= (LEX_STRING *)column_key; > + DBUG_ASSERT(column_name->length <= MAX_NAME_LENGTH); > + hdr->entry[0]= column_name->length; > + DBUG_ASSERT(hdr->name - hdr->nmpool < (long) 0x10000L); > + int2store(hdr->entry + 1, hdr->name - hdr->nmpool); > + memcpy(hdr->name, column_name->str, column_name->length); > + DBUG_ASSERT(hdr->nmpool_size != 0 || column_name->length == 0); > + if (type_and_offset_store(hdr->entry + 1, hdr->offset_size, > + value->type, > + offset)) > + return TRUE; > + hdr->entry+= hdr->entry_size; > + hdr->name+= column_name->length; > + return FALSE; > +} > + > + > +/** > + Format descriptor, contain constants and function references for > + format processing > +*/ > + > +struct st_service_funcs > +{ > + /* size of fixed header */ > + uint fixed_hdr; > + /* size of fixed part of header entry */ > + uint fixed_hdr_entry; > + > + /*size of array element which stores keys */ > + uint key_size_in_array; > + > + size_t (*name_size) > + (void *, uint); > + int (*column_sort) > + (const void *a, const void *b); > + my_bool (*check_limit) > + (const void *val); > + void (*set_fixed_hdr) > + (DYNAMIC_COLUMN *str, DYN_HEADER *hdr); > + my_bool (*put_header_entry)(DYN_HEADER *hdr, > + void *column_key, > + DYNAMIC_COLUMN_VALUE *value, > + size_t offset); > + int (*plan_sort)(const void *a, const void *b); > +}; > + > + > +/** > + Actual our 2 format descriptors > +*/ > + > +static struct st_service_funcs fmt_data[2]= > +{ > + { > + FIXED_HEADER_SIZE, > + COLUMN_NUMBER_SIZE, > + sizeof(uint), > + &name_size_num, > + &column_sort_num, > + &check_limit_num, > + &set_fixed_header_num, > + &put_header_entry_num, > + &plan_sort_num > + }, > + { > + FIXED_HEADER_SIZE_NM, > + COLUMN_NAMEPTR_SIZE, > + sizeof(LEX_STRING), > + &name_size_str, > + &column_sort_str, > + &check_limit_str, > + &set_fixed_header_str, > + &put_header_entry_str, > + &plan_sort_str > + } > +}; > + > + > +/** > + Read dynamic column record header and fill the descriptor > + > + @param hdr dynamic columns record descriptor to fill > + @param str dynamic columns record > + > + @return ER_DYNCOL_* return code > +*/ > + > +enum enum_dyncol_func_result > +init_read_hdr(DYN_HEADER *hdr, DYNAMIC_COLUMN *str) > +{ > + if (read_fixed_header(hdr, str)) > + return ER_DYNCOL_FORMAT; > + hdr->header= (uchar*)str->str + fmt_data[hdr->format].fixed_hdr; > + calc_param(&hdr->entry_size, &hdr->header_size, > + fmt_data[hdr->format].fixed_hdr_entry, hdr->offset_size, > + hdr->column_count); > + hdr->nmpool= hdr->header + hdr->header_size; > + hdr->dtpool= hdr->nmpool + hdr->nmpool_size; > + hdr->data_size= str->length - fmt_data[hdr->format].fixed_hdr - > + hdr->header_size - hdr->nmpool_size; > + hdr->data_end= (uchar*)str->str + str->length; > + return ER_DYNCOL_OK; > +} > + > > /** > Initialize dynamic column string with (make it empty but correct format) > @@ -1640,83 +2132,763 @@ dynamic_column_list(DYNAMIC_COLUMN *str, > > > /** > + List not-null columns in the packed string (any format) > + > + @param str The packed string > + @param array_of_lexstr Where to put reference on created array > + > + @return ER_DYNCOL_* return code > +*/ > + > +enum enum_dyncol_func_result > +dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr) > +{ > + DYN_HEADER header; > + uchar *read; > + struct st_service_funcs *fmt; > + uint i; > + enum enum_dyncol_func_result rc; > + > + bzero(array_of_lexstr, sizeof(*array_of_lexstr)); /* In case of errors */ > + if (str->length == 0) > + return ER_DYNCOL_OK; /* no columns */ > + > + if ((rc= init_read_hdr(&header, str)) < 0) > + return rc; > + > + fmt= fmt_data + header.format; > + > + if (header.entry_size * header.column_count + fmt->fixed_hdr > > + str->length) > + return ER_DYNCOL_FORMAT; > + > + if (init_dynamic_array(array_of_lexstr, sizeof(LEX_STRING), > + header.column_count, 0)) > + return ER_DYNCOL_RESOURCE; > + > + for (i= 0, read= header.header; > + i < header.column_count; > + i++, read+= header.entry_size) > + { > + LEX_STRING tmp; > + if (header.format == DYNCOL_FMT_NUM) > + { > + uint nm= uint2korr(read); > + tmp.str= my_malloc(DYNCOL_NUM_CHAR, MYF(0)); ouch. do you really need to do header.column_count small mallocs here? > + if (!tmp.str) > + return ER_DYNCOL_RESOURCE; > + tmp.length= snprintf(tmp.str, DYNCOL_NUM_CHAR, "%u", nm); longlong2str or longlong10_to_str > + } > + else > + { > + tmp.length= read[0]; > + tmp.str= my_malloc(tmp.length + 1, MYF(0)); ouch. do you really need to do header.column_count small mallocs here? > + if(!tmp.str) > + return ER_DYNCOL_RESOURCE; > + memcpy(tmp.str, (const void *)header.nmpool + uint2korr(read + 1), > + tmp.length); > + tmp.str[tmp.length]= '\0'; // just for safety > + } > + /* Insert can't never fail as it's pre-allocated above */ > + (void) insert_dynamic(array_of_lexstr, (uchar *)&tmp); > + } > + return ER_DYNCOL_OK; > +} > + > +/** > Find the place of the column in the header or place where it should be put > > - @param num Number of the column > - @param header Pointer to the header > - @param entry_size Size of a header entry > - @param column_count Number of columns in the packed string > - @param entry Return pointer to the entry or next entry > + @param hdr descriptor of dynamic column record > + @param key Name or number of column to fetch > + (depends on string_key) > + @param string_key True if we gave pointer to LEX_STRING. > > @retval TRUE found > @retval FALSE pointer set to the next row > */ > > static my_bool > -find_place(uint num, uchar *header, size_t entry_size, > - uint column_count, uchar **entry) > +find_place(DYN_HEADER *hdr, void *key, my_bool string_keys) > { > uint mid, start, end, val; > int flag; > + LEX_STRING str; > + char buff[DYNCOL_NUM_CHAR]; > + my_bool need_conversion= ((string_keys ? DYNCOL_FMT_STR : DYNCOL_FMT_NUM) != > + hdr->format); > LINT_INIT(flag); /* 100 % safe */ UNINIT_VAR is preferable, when possible (but it doesn't work with structures) > + /* new format can't be numeric if the old one is names */ > + DBUG_ASSERT(string_keys || > + hdr->format == DYNCOL_FMT_NUM); > > start= 0; > - end= column_count -1; > + end= hdr->column_count -1; > mid= 1; > while (start != end) > { > - uint val; > - mid= (start + end) / 2; > - val= uint2korr(header + mid * entry_size); > - if ((flag= CMP_NUM(num, val)) <= 0) > - end= mid; > - else > - start= mid + 1; > + uint val; > + mid= (start + end) / 2; > + hdr->entry= hdr->header + mid * hdr->entry_size; > + if (!string_keys) > + { > + val= uint2korr(hdr->entry); > + flag= CMP_NUM(*((uint *)key), val); > + } > + else > + { > + if (need_conversion) > + { > + str.str= backwritenum(buff + sizeof(buff), uint2korr(hdr->entry)); > + str.length= (buff + sizeof(buff)) - str.str; > + } > + else > + { > + DBUG_ASSERT(hdr->format == DYNCOL_FMT_STR); > + str.length= hdr->entry[0]; > + str.str= (char *)hdr->nmpool + uint2korr(hdr->entry + 1); > + } > + flag= ((LEX_STRING *) key)->length - str.length; > + if (flag == 0) > + flag= memcmp(((LEX_STRING *) key)->str, str.str, str.length); > + } > + if (flag <= 0) > + end= mid; > + else > + start= mid + 1; > } > + hdr->entry= hdr->header + start * hdr->entry_size; > if (start != mid) > { > - val= uint2korr(header + start * entry_size); > - flag= CMP_NUM(num, val); > + if (!string_keys) > + { > + val= uint2korr(hdr->entry); > + flag= CMP_NUM(*((uint *)key), val); > + } > + else > + { > + if (need_conversion) > + { > + str.str= backwritenum(buff + sizeof(buff), uint2korr(hdr->entry)); > + str.length= (buff + sizeof(buff)) - str.str; > + } > + else > + { > + DBUG_ASSERT(hdr->format == DYNCOL_FMT_STR); > + str.length= hdr->entry[0]; > + str.str= (char*) hdr->nmpool + uint2korr(hdr->entry + 1); > + } > + flag= ((LEX_STRING *) key)->length - str.length; > + if (flag == 0) > + flag= memcmp(((LEX_STRING *) key)->str, str.str, str.length); > + } > } > - *entry= header + start * entry_size; > if (flag > 0) > - *entry+= entry_size; /* Point at next bigger key */ > + hdr->entry+= hdr->entry_size; /* Point at next bigger key */ > return flag == 0; > } > > > /* > - Description of plan of adding/removing/updating a packed string > + It is internal structure which describes plan of chenging the record s/chenging/changing/ what do you mean "a plan" ? > + of dynamic columns > */ > > typedef enum {PLAN_REPLACE, PLAN_ADD, PLAN_DELETE, PLAN_NOP} PLAN_ACT; > > struct st_plan { > DYNAMIC_COLUMN_VALUE *val; > - uint *num; > + void *key; > uchar *place; > size_t length; > - int hdelta, ddelta; > + int hdelta, ddelta, ndelta; > + uint mv_offset, mv_length, mv_end; > PLAN_ACT act; > }; > typedef struct st_plan PLAN; > > > -static int plan_sort(const void *a, const void *b) > +/** > + Sort function for plan by column number > +*/ > + > +static int plan_sort_num(const void *a, const void *b) > { > - return ((PLAN *)a)->num[0] - ((PLAN *)b)->num[0]; > + return *((uint *)((PLAN *)a)->key) - *((uint *)((PLAN *)b)->key); > +} > + > + > +/** > + Sort function for plan by column name > +*/ > + > +static int plan_sort_str(const void *a, const void *b) > +{ > + int res= (((LEX_STRING *)((PLAN *)a)->key)->length - > + ((LEX_STRING *)((PLAN *)b)->key)->length); > + if (res == 0) > + res= memcmp(((LEX_STRING *)((PLAN *)a)->key)->str, > + ((LEX_STRING *)((PLAN *)b)->key)->str, > + ((LEX_STRING *)((PLAN *)a)->key)->length); > + return res; > +} > + > +#define DELTA_CHECK(S, D, C) \ > + if ((S) == 0) \ > + (S)= (D); \ > + else if (((S) > 0 && (D) < 0) || \ > + ((S) < 0 && (D) > 0)) \ > + { \ > + (C)= TRUE; \ > + } > + > +/** > + Update dynamic column by copying in a new record (string). > + > + @param str Dynamic column record to change > + @param plan Plan of changing the record > + @param add_column_count number of records in the plan array. > + @param hdr descriptor of old dynamic column record > + @param new_hdr descriptor of new dynamic column record > + @param convert need conversion from numeric to names format > + > + @return ER_DYNCOL_* return code > +*/ > + > +enum enum_dyncol_func_result > +dynamic_column_update_copy(DYNAMIC_COLUMN *str, PLAN *plan, > + uint add_column_count, > + DYN_HEADER *hdr, DYN_HEADER *new_hdr, > + my_bool convert) > +{ > + DYNAMIC_COLUMN tmp; > + struct st_service_funcs *fmt= fmt_data + hdr->format, > + *new_fmt= fmt_data + new_hdr->format; > + uint i, j, k; > + size_t all_headers_size; > + > + if (dynamic_column_init_str(&tmp, > + (new_fmt->fixed_hdr + new_hdr->header_size + > + new_hdr->nmpool_size + > + new_hdr->data_size + DYNCOL_SYZERESERVE))) > + { > + return ER_DYNCOL_RESOURCE; > + } > + bzero(tmp.str, new_fmt->fixed_hdr); > + (*new_fmt->set_fixed_hdr)(&tmp, new_hdr); > + /* Adjust tmp to contain whole the future header */ > + tmp.length= new_fmt->fixed_hdr + new_hdr->header_size + new_hdr->nmpool_size; > + > + > + /* > + Copy data to the new string > + i= index in array of changes > + j= index in packed string header index > + */ > + new_hdr->entry= new_hdr->header; > + new_hdr->name= new_hdr->nmpool; > + all_headers_size= new_fmt->fixed_hdr + > + new_hdr->header_size + new_hdr->nmpool_size; > + for (i= 0, j= 0; i < add_column_count || j < hdr->column_count; i++) > + { > + size_t first_offset; > + uint start= j, end; > + LINT_INIT(first_offset); > + > + /* > + Search in i and j for the next column to add from i and where to > + add. > + */ > + > + while (i < add_column_count && plan[i].act == PLAN_NOP) > + i++; /* skip NOP */ > + > + if (i == add_column_count) > + j= end= hdr->column_count; > + else > + { > + /* > + old data portion. We don't need to check that j < column_count > + as plan[i].place is guaranteed to have a pointer inside the > + data. > + */ > + while (hdr->header + j * hdr->entry_size < plan[i].place) > + j++; > + end= j; > + if ((plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > + j++; /* data at 'j' will be removed */ > + } > + > + /* > + Adjust all headers since last loop. > + We have to do this as the offset for data has moved > + */ > + for (k= start; k < end; k++) > + { > + uchar *read= hdr->header + k * hdr->entry_size; > + void *key; > + LEX_STRING name; > + size_t offs; > + uint nm; > + DYNAMIC_COLUMN_TYPE tp; > + char buff[DYNCOL_NUM_CHAR]; > + > + if (hdr->format == DYNCOL_FMT_NUM) > + { > + if (convert) > + { > + name.str= backwritenum(buff + sizeof(buff), uint2korr(read)); > + name.length= (buff + sizeof(buff)) - name.str; > + key= &name; > + } > + else > + { > + nm= uint2korr(read); /* Column nummber */ > + key= &nm; > + } > + } > + else > + { > + name.length= read[0]; > + name.str= (char *) hdr->nmpool + uint2korr(read + 1); > + key= &name; > + } > + if (type_and_offset_read(&tp, &offs, > + read + fmt->fixed_hdr_entry, hdr->offset_size)) > + goto err; > + if (k == start) > + first_offset= offs; > + else if (offs < first_offset) > + goto err; > + > + offs+= plan[i].ddelta; > + { > + DYNAMIC_COLUMN_VALUE val; > + val.type= tp; // only the type used in the header > + if ((*new_fmt->put_header_entry)(new_hdr, key, &val, offs)) > + goto err; > + } > + } > + > + /* copy first the data that was not replaced in original packed data */ > + if (start < end) > + { > + size_t data_size; > + /* Add old data last in 'tmp' */ > + hdr->entry= hdr->header + start * hdr->entry_size; > + data_size= > + hdr_interval_length(hdr, hdr->header + end * hdr->entry_size); > + if (data_size == DYNCOL_OFFSET_ERROR || > + (long) data_size < 0 || > + data_size > hdr->data_size - first_offset) > + goto err; > + > + memcpy(tmp.str + tmp.length, (char *)hdr->dtpool + first_offset, > + data_size); > + tmp.length+= data_size; > + } > + > + /* new data adding */ > + if (i < add_column_count) > + { > + if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > + { > + if ((*new_fmt->put_header_entry)(new_hdr, plan[i].key, > + plan[i].val, > + tmp.length - all_headers_size)) > + goto err; > + data_store(&tmp, plan[i].val); /* Append new data */ > + } > + } > + } > + dynamic_column_column_free(str); > + *str= tmp; > + return ER_DYNCOL_OK; > +err: > + dynamic_column_column_free(&tmp); > + return ER_DYNCOL_FORMAT; > +} > + > +enum enum_dyncol_func_result > +dynamic_column_update_move_left(DYNAMIC_COLUMN *str, PLAN *plan, > + size_t offset_size, > + size_t entry_size, > + size_t header_size, > + size_t new_offset_size, > + size_t new_entry_size, > + size_t new_header_size, > + uint column_count, > + uint new_column_count, > + uint add_column_count, > + uchar *header_end, > + size_t max_offset) > +{ > + uchar *write; > + uchar *header_base= (uchar *)str->str + FIXED_HEADER_SIZE; > + uint i, j, k; > + size_t curr_offset; > + > + write= (uchar *)str->str + FIXED_HEADER_SIZE; > + set_fixed_header(str, new_offset_size, new_column_count); > + > + /* > + Move headers first. > + i= index in array of changes > + j= index in packed string header index > + */ > + for (curr_offset= 0, i= 0, j= 0; > + i < add_column_count || j < column_count; > + i++) > + { > + size_t first_offset; > + uint start= j, end; > + LINT_INIT(first_offset); > + > + /* > + Search in i and j for the next column to add from i and where to > + add. > + */ > + > + while (i < add_column_count && plan[i].act == PLAN_NOP) > + i++; /* skip NOP */ > + > + if (i == add_column_count) > + j= end= column_count; > + else > + { > + /* > + old data portion. We don't need to check that j < column_count > + as plan[i].place is guaranteed to have a pointer inside the > + data. > + */ > + while (header_base + j * entry_size < plan[i].place) > + j++; > + end= j; > + if ((plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > + j++; /* data at 'j' will be removed */ > + } > + plan[i].mv_end= end; > + > + { > + DYNAMIC_COLUMN_TYPE tp; > + if (type_and_offset_read(&tp, &first_offset, > + header_base + start * entry_size + > + COLUMN_NUMBER_SIZE, offset_size)) > + return ER_DYNCOL_FORMAT; > + } > + /* find data to be moved */ > + if (start < end) > + { > + size_t data_size= > + get_length_interval(header_base + start * entry_size, > + header_base + end * entry_size, > + header_end, offset_size, max_offset); > + if (data_size == DYNCOL_OFFSET_ERROR || > + (long) data_size < 0 || > + data_size > max_offset - first_offset) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + DBUG_ASSERT(curr_offset == first_offset + plan[i].ddelta); > + plan[i].mv_offset= first_offset; > + plan[i].mv_length= data_size; > + curr_offset+= data_size; > + } > + else > + { > + plan[i].mv_length= 0; > + plan[i].mv_offset= curr_offset; > + } > + > + if (plan[i].ddelta == 0 && offset_size == new_offset_size && > + plan[i].act != PLAN_DELETE) > + write+= entry_size * (end - start); > + else > + { > + /* > + Adjust all headers since last loop. > + We have to do this as the offset for data has moved > + */ > + for (k= start; k < end; k++) > + { > + uchar *read= header_base + k * entry_size; > + size_t offs; > + uint nm; > + DYNAMIC_COLUMN_TYPE tp; > + > + nm= uint2korr(read); /* Column nummber */ > + if (type_and_offset_read(&tp, &offs, read + COLUMN_NUMBER_SIZE, > + offset_size)) > + return ER_DYNCOL_FORMAT; > + > + if (k > start && offs < first_offset) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + > + offs+= plan[i].ddelta; > + int2store(write, nm); > + /* write rest of data at write + COLUMN_NUMBER_SIZE */ > + type_and_offset_store(write, new_offset_size, tp, offs); > + write+= new_entry_size; > + } > + } > + > + /* new data adding */ > + if (i < add_column_count) > + { > + if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > + { > + int2store(write, *((uint *)plan[i].key)); > + type_and_offset_store(write, new_offset_size, > + plan[i].val[0].type, > + curr_offset); > + write+= new_entry_size; > + curr_offset+= plan[i].length; > + } > + } > + } > + > + /* > + Move data. > + i= index in array of changes > + j= index in packed string header index > + */ > + str->length= (FIXED_HEADER_SIZE + new_header_size); > + for (i= 0, j= 0; > + i < add_column_count || j < column_count; > + i++) > + { > + uint start= j, end; > + > + /* > + Search in i and j for the next column to add from i and where to > + add. > + */ > + > + while (i < add_column_count && plan[i].act == PLAN_NOP) > + i++; /* skip NOP */ > + > + j= end= plan[i].mv_end; > + if (i != add_column_count && > + (plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > + j++; > + > + /* copy first the data that was not replaced in original packed data */ > + if (start < end && plan[i].mv_length) > + { > + memmove((header_base + new_header_size + > + plan[i].mv_offset + plan[i].ddelta), > + header_base + header_size + plan[i].mv_offset, > + plan[i].mv_length); > + } > + str->length+= plan[i].mv_length; > + > + /* new data adding */ > + if (i < add_column_count) > + { > + if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > + { > + data_store(str, plan[i].val); /* Append new data */ > + } > + } > + } > + return ER_DYNCOL_OK; > +} > + > +enum enum_dyncol_func_result > +dynamic_column_update_move_right(DYNAMIC_COLUMN *str, PLAN *plan, > + size_t offset_size, > + size_t entry_size, > + size_t header_size, > + size_t new_offset_size, > + size_t new_entry_size, > + size_t new_header_size, > + uint column_count, > + uint new_column_count, > + uint add_column_count, > + uchar *header_end, > + size_t max_offset) > +{ > + uchar *write; > + uchar *header_base= (uchar *)str->str + FIXED_HEADER_SIZE; > + uint i, j, k; > + size_t curr_offset; > + > + write= (uchar *)str->str + FIXED_HEADER_SIZE; > + set_fixed_header(str, new_offset_size, new_column_count); > + > + /* > + Move data first. > + i= index in array of changes > + j= index in packed string header index > + */ > + for (curr_offset= 0, i= 0, j= 0; > + i < add_column_count || j < column_count; > + i++) > + { > + size_t first_offset; > + uint start= j, end; > + LINT_INIT(first_offset); > + > + /* > + Search in i and j for the next column to add from i and where to > + add. > + */ > + > + while (i < add_column_count && plan[i].act == PLAN_NOP) > + i++; /* skip NOP */ > + > + if (i == add_column_count) > + j= end= column_count; > + else > + { > + /* > + old data portion. We don't need to check that j < column_count > + as plan[i].place is guaranteed to have a pointer inside the > + data. > + */ > + while (header_base + j * entry_size < plan[i].place) > + j++; > + end= j; > + if ((plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > + j++; /* data at 'j' will be removed */ > + } > + plan[i].mv_end= end; > + > + { > + DYNAMIC_COLUMN_TYPE tp; > + type_and_offset_read(&tp, &first_offset, > + header_base + start * entry_size + COLUMN_NUMBER_SIZE, offset_size); > + } > + /* find data to be moved */ > + if (start < end) > + { > + size_t data_size= > + get_length_interval(header_base + start * entry_size, > + header_base + end * entry_size, > + header_end, offset_size, max_offset); > + if (data_size == DYNCOL_OFFSET_ERROR || > + (long) data_size < 0 || > + data_size > max_offset - first_offset) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + DBUG_ASSERT(curr_offset == first_offset + plan[i].ddelta); > + plan[i].mv_offset= first_offset; > + plan[i].mv_length= data_size; > + curr_offset+= data_size; > + } > + else > + { > + plan[i].mv_length= 0; > + plan[i].mv_offset= curr_offset; > + } > + > + if (plan[i].ddelta == 0 && offset_size == new_offset_size && > + plan[i].act != PLAN_DELETE) > + write+= entry_size * (end - start); > + else > + { > + /* > + Adjust all headers since last loop. > + We have to do this as the offset for data has moved > + */ > + for (k= start; k < end; k++) > + { > + uchar *read= header_base + k * entry_size; > + size_t offs; > + uint nm; > + DYNAMIC_COLUMN_TYPE tp; > + > + nm= uint2korr(read); /* Column nummber */ > + type_and_offset_read(&tp, &offs, read + COLUMN_NUMBER_SIZE, offset_size); > + if (k > start && offs < first_offset) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + > + offs+= plan[i].ddelta; > + int2store(write, nm); > + /* write rest of data at write + COLUMN_NUMBER_SIZE */ > + if (type_and_offset_store(write, new_offset_size, tp, offs)) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + write+= new_entry_size; > + } > + } > + > + /* new data adding */ > + if (i < add_column_count) > + { > + if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > + { > + int2store(write, *((uint *)plan[i].key)); > + if (type_and_offset_store(write, new_offset_size, > + plan[i].val[0].type, > + curr_offset)) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + write+= new_entry_size; > + curr_offset+= plan[i].length; > + } > + } > + } > + > + /* > + Move headers. > + i= index in array of changes > + j= index in packed string header index > + */ > + str->length= (FIXED_HEADER_SIZE + new_header_size); > + for (i= 0, j= 0; > + i < add_column_count || j < column_count; > + i++) > + { > + uint start= j, end; > + > + /* > + Search in i and j for the next column to add from i and where to > + add. > + */ > + > + while (i < add_column_count && plan[i].act == PLAN_NOP) > + i++; /* skip NOP */ > + > + j= end= plan[i].mv_end; > + if (i != add_column_count && > + (plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > + j++; > + > + /* copy first the data that was not replaced in original packed data */ > + if (start < end && plan[i].mv_length) > + { > + memmove((header_base + new_header_size + > + plan[i].mv_offset + plan[i].ddelta), > + header_base + header_size + plan[i].mv_offset, > + plan[i].mv_length); > + } > + str->length+= plan[i].mv_length; > + > + /* new data adding */ > + if (i < add_column_count) > + { > + if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > + { > + data_store(str, plan[i].val); /* Append new data */ > + } > + } > + } > + return ER_DYNCOL_OK; > } > > -#define DELTA_CHECK(S, D, C) \ > - if ((S) == 0) \ > - (S)= (D); \ > - else if (((S) > 0 && (D) < 0) || \ > - ((S) < 0 && (D) > 0)) \ > - { \ > - (C)= TRUE; \ > - break; \ > - } \ > - > > /** > Update the packed string with the given columns > @@ -1826,26 +3044,36 @@ dynamic_column_update_many(DYNAMIC_COLUM > > /* Set common variables for all plans */ > plan[i].ddelta= data_delta; > + plan[i].ndelta= name_delta; > /* get header delta in entries */ > plan[i].hdelta= header_delta; > plan[i].length= 0; /* Length if NULL */ > > - if (find_place(plan[i].num[0], > - (uchar *)str->str + FIXED_HEADER_SIZE, > - entry_size, column_count, &entry)) > + if (find_place(&header, plan[i].key, string_keys)) > { > - size_t entry_data_size; > + size_t entry_data_size, entry_name_size= 0; > > /* Data existed; We have to replace or delete it */ > > - entry_data_size= get_length(entry, header_end, > - offset_size, max_offset); > - if ((long) entry_data_size < 0) > + entry_data_size= hdr_interval_length(&header, header.entry + > + header.entry_size); > + if (entry_data_size == DYNCOL_OFFSET_ERROR || > + (long) entry_data_size < 0) > { > rc= ER_DYNCOL_FORMAT; > goto end; > } > > + //get_length(header.entry, header.dtpool, header.offset_size, > + //header.data_size); did you forget to delete this? > + if (new_header.format == DYNCOL_FMT_STR) > + { > + if (header.format == DYNCOL_FMT_STR) > + entry_name_size= header.entry[0]; > + else > + entry_name_size= numlen(uint2korr(header.entry)); > + } > + > if (plan[i].val->type == DYN_COL_NULL) > { > /* Inserting a NULL means delete the old data */ > @@ -1891,219 +3124,759 @@ dynamic_column_update_many(DYNAMIC_COLUM > goto end; > } > data_delta+= plan[i].length; > + if (new_header.format == DYNCOL_FMT_STR) > + name_delta+= ((LEX_STRING *)plan[i].key)->length; > } > } > - plan[i].place= entry; > + plan[i].place= header.entry; > } > plan[add_column_count].hdelta= header_delta; > plan[add_column_count].ddelta= data_delta; > - new_column_count= column_count + header_delta; > + plan[add_column_count].act= PLAN_NOP; > + plan[add_column_count].place= header.dtpool; > + > + new_header.column_count= header.column_count + header_delta; > > /* > Check if it is only "increasing" or only "decreasing" plan for (header > and data separately). > */ > - data_size= str->length - header_size - FIXED_HEADER_SIZE; > - new_data_size= data_size + data_delta; > - if ((new_offset_size= dynamic_column_offset_bytes(new_data_size)) >= > + new_header.data_size= header.data_size + data_delta; > + new_header.nmpool_size= new_header.nmpool_size + name_delta; > + DBUG_ASSERT(new_header.format != DYNCOL_FMT_NUM || > + new_header.nmpool_size == 0); > + if ((new_header.offset_size= > + dynamic_column_offset_bytes(new_header.data_size)) >= > MAX_OFFSET_LENGTH) > { > rc= ER_DYNCOL_LIMIT; > goto end; > } > > -#ifdef NOT_IMPLEMENTED > - /* if (new_offset_size != offset_size) then we have to rewrite header */ > - header_delta_sign= new_offset_size - offset_size; > + copy= ((header.format != new_header.format) || > + (new_header.format == DYNCOL_FMT_STR)); > + /* if (new_header.offset_size!=offset_size) then we have to rewrite header */ > + header_delta_sign= > + ((int)new_header.offset_size + new_fmt->fixed_hdr_entry) - > + ((int)header.offset_size + fmt->fixed_hdr_entry); > data_delta_sign= 0; > - for (i= 0; i < add_column_count; i++) > + // plan[add_column_count] contains last deltas. > + for (i= 0; i <= add_column_count && !copy; i++) > { > /* This is the check for increasing/decreasing */ > DELTA_CHECK(header_delta_sign, plan[i].hdelta, copy); > DELTA_CHECK(data_delta_sign, plan[i].ddelta, copy); > } > -#endif > - calc_param(&new_entry_size, &new_header_size, > - new_offset_size, new_column_count); > + calc_param(&new_header.entry_size, &new_header.header_size, > + new_fmt->fixed_hdr_entry, > + new_header.offset_size, new_header.column_count); > > /* > - The following code always make a copy. In future we can do a more > - optimized version when data is only increasing / decreasing. > + Need copy because: > + 1. Header/data parts moved in different directions. > + 2. There is no enough allocated space in the string. > + 3. Header and data moved in different directions. > */ > + if (copy || /*1*/ forgot to delete /*1*/ ? > + str->max_length < str->length + header_delta + data_delta || /*2*/ > + ((header_delta_sign < 0 && data_delta_sign > 0) || > + (header_delta_sign > 0 && data_delta_sign < 0))) /*3*/ > + rc= dynamic_column_update_copy(str, plan, add_column_count, > + &header, &new_header, > + convert); > + else > + if (header_delta_sign < 0) > + rc= dynamic_column_update_move_left(str, plan, header.offset_size, > + header.entry_size, > + header.header_size, > + new_header.offset_size, > + new_header.entry_size, > + new_header.header_size, > + header.column_count, > + new_header.column_count, > + add_column_count, header.dtpool, > + header.data_size); > + else > + /* > + rc= dynamic_column_update_move_right(str, plan, offset_size, > + entry_size, header_size, > + new_header.offset_size, > + new_header.entry_size, > + new_heder.header_size, column_count, > + new_header.column_count, > + add_column_count, header_end, > + header.data_size); > + */ > + rc= dynamic_column_update_copy(str, plan, add_column_count, > + &header, &new_header, > + convert); > +end: > + my_free(alloc_plan); > + return rc; > + > +create_new_string: > + /* There is no columns from before, so let's just add the new ones */ > + rc= ER_DYNCOL_OK; > + my_free(alloc_plan); > + if (not_null != 0) > + rc= dynamic_column_create_many_internal_fmt(str, add_column_count, > + (uint*)column_keys, values, > + str->str == NULL, > + string_keys); > + goto end; > +} > + > + > +/** > + Update the packed string with the given column > + > + @param str String where to write the data > + @param column_number Array of columns number > + @param values Array of columns values > + > + @return ER_DYNCOL_* return code > +*/ > + > + > +int dynamic_column_update(DYNAMIC_COLUMN *str, uint column_nr, > + DYNAMIC_COLUMN_VALUE *value) > +{ > + return dynamic_column_update_many(str, 1, &column_nr, value); > +} > + > + > +enum enum_dyncol_func_result > +dynamic_column_check(DYNAMIC_COLUMN *str) > +{ > + struct st_service_funcs *fmt; > + enum enum_dyncol_func_result rc= ER_DYNCOL_FORMAT; > + DYN_HEADER header; > + uint i; > + size_t data_offset= 0, name_offset= 0; > + size_t prev_data_offset= 0, prev_name_offset= 0; > + LEX_STRING name= {0,0}, prev_name= {0,0}; > + uint num= 0, prev_num= 0; > + void *key, *prev_key; > + enum enum_dynamic_column_type type= DYN_COL_NULL, prev_type= DYN_COL_NULL; > + > + DBUG_ENTER("dynamic_column_check"); > > - /*if (copy) */ > + if (str->length == 0) > { > - DYNAMIC_COLUMN tmp; > - uchar *header_base= (uchar *)str->str + FIXED_HEADER_SIZE, > - *write; > - if (dynamic_column_init_str(&tmp, > - (FIXED_HEADER_SIZE + new_header_size + > - new_data_size + DYNCOL_SYZERESERVE))) > - { > - rc= ER_DYNCOL_RESOURCE; > - goto end; > - } > - write= (uchar *)tmp.str + FIXED_HEADER_SIZE; > - /* Adjust tmp to contain whole the future header */ > - tmp.length= FIXED_HEADER_SIZE + new_header_size; > - set_fixed_header(&tmp, new_offset_size, new_column_count); > - data_delta= 0; > + DBUG_PRINT("info", ("empty string is OK")); > + DBUG_RETURN(ER_DYNCOL_OK); > + } > > - /* > - Copy data to the new string > - i= index in array of changes > - j= index in packed string header index > - */ > + bzero(&header, sizeof(header)); > > - for (i= 0, j= 0; i < add_column_count || j < column_count; i++) > - { > - size_t first_offset; > - uint start= j, end; > - LINT_INIT(first_offset); > + /* Check that header is OK */ > + if (read_fixed_header(&header, str)) > + { > + DBUG_PRINT("info", ("Reading fixed string header failed")); > + goto end; > + } > + fmt= fmt_data + header.format; > + calc_param(&header.entry_size, &header.header_size, > + fmt->fixed_hdr_entry, header.offset_size, > + header.column_count); > + /* headers are out of string length (no space for data and part of headers) */ > + if (fmt->fixed_hdr + header.header_size + header.nmpool_size > str->length) > + { > + DBUG_PRINT("info", ("Fixed header: %u Header size: %u " > + "Name pool size: %u but Strig length: %u", > + (uint)fmt->fixed_hdr, > + (uint)header.header_size, > + (uint)header.nmpool_size, > + (uint)str->length)); > + goto end; > + } > + header.header= (uchar*)str->str + fmt->fixed_hdr; > + header.nmpool= header.header + header.header_size; > + header.dtpool= header.nmpool + header.nmpool_size; > + header.data_size= str->length - fmt->fixed_hdr - > + header.header_size - header.nmpool_size; > > - /* > - Search in i and j for the next column to add from i and where to > - add. > - */ > + /* read and check headers */ > + if (header.format == DYNCOL_FMT_NUM) > + { > + key= &num; > + prev_key= &prev_num; > + } > + else > + { > + key= &name; > + prev_key= &prev_name; > + } > + for (i= 0, header.entry= header.header; > + i < header.column_count; > + i++, header.entry+= header.entry_size) > + { > > - while (i < add_column_count && plan[i].act == PLAN_NOP) > - i++; /* skip NOP */ > - if (i == add_column_count) > - j= end= column_count; > - else > + if (header.format == DYNCOL_FMT_NUM) > + { > + num= uint2korr(header.entry); > + } > + else > + { > + DBUG_ASSERT(header.format == DYNCOL_FMT_STR); > + name.length= header.entry[0]; > + name_offset= uint2korr(header.entry + 1); > + name.str= (char *)header.nmpool + name_offset; > + } > + if (type_and_offset_read(&type, &data_offset, > + header.entry + fmt->fixed_hdr_entry, > + header.offset_size)) > + goto end; > + > + DBUG_ASSERT(type != DYN_COL_NULL); > + if (data_offset > header.data_size) > + { > + DBUG_PRINT("info", ("Field order: %u Data offset: %u" > + " > Data pool size: %u", > + (uint)i, > + (uint)name_offset, > + (uint)header.nmpool_size)); > + goto end; > + } > + if (name_offset > header.nmpool_size) > + { > + DBUG_PRINT("info", ("Field order: %u Name offset: %u" > + " > Name pool size: %u", > + (uint)i, > + (uint)name_offset, > + (uint)header.nmpool_size)); > + goto end; > + } > + if (prev_type != DYN_COL_NULL) > + { > + /* It is not first entry */ > + if (prev_data_offset >= data_offset) > { > - /* > - old data portion. We don't need to check that j < column_count > - as plan[i].place is guaranteed to have a pointer inside the > - data. > - */ > - while (header_base + j * entry_size < plan[i].place) > - j++; > - end= j; > - if ((plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > - j++; /* data at 'j' will be removed */ > + DBUG_PRINT("info", ("Field order: %u Previous data offset: %u" > + " >= Current data offset: %u", > + (uint)i, > + (uint)prev_data_offset, > + (uint)data_offset)); > + goto end; > } > - > - if (plan[i].ddelta == 0 && offset_size == new_offset_size) > + if (prev_name_offset > name_offset) > { > - uchar *read= header_base + start * entry_size; > - DYNAMIC_COLUMN_TYPE tp; > - /* > - It's safe to copy the header unchanged. This is usually the > - case for the first header block before any changed data. > - */ > - if (start < end) /* Avoid memcpy with 0 */ > - { > - size_t length= entry_size * (end - start); > - memcpy(write, read, length); > - write+= length; > - } > - /* Read first_offset */ > - type_and_offset_read(&tp, &first_offset, read, offset_size); > + DBUG_PRINT("info", ("Field order: %u Previous name offset: %u" > + " > Current name offset: %u", > + (uint)i, > + (uint)prev_data_offset, > + (uint)data_offset)); > + goto end; > } > - else > + if ((*fmt->column_sort)(&prev_key, &key) >= 0) > { > - /* > - Adjust all headers since last loop. > - We have to do this as the offset for data has moved > - */ > - for (k= start; k < end; k++) > - { > - uchar *read= header_base + k * entry_size; > - size_t offs; > - uint nm; > - DYNAMIC_COLUMN_TYPE tp; > + DBUG_PRINT("info", ("Field order: %u Previous key >= Current key", > + (uint)i)); > + goto end; > + } > + } > + prev_num= num; > + prev_name= name; > + prev_data_offset= data_offset; > + prev_name_offset= name_offset; > + prev_type= type; > + } > + > + /* check data, which we can */ > + for (i= 0, header.entry= header.header; > + i < header.column_count; > + i++, header.entry+= header.entry_size) > + { > + DYNAMIC_COLUMN_VALUE store; > + // already checked by previouse pass > + type_and_offset_read(&header.type, &header.offset, > + header.entry + fmt->fixed_hdr_entry, > + header.offset_size); > + header.length= > + hdr_interval_length(&header, header.entry + header.entry_size); > + header.data= header.dtpool + header.offset; > + switch ((header.type)) { > + case DYN_COL_INT: > + rc= dynamic_column_sint_read(&store, header.data, header.length); > + break; > + case DYN_COL_UINT: > + rc= dynamic_column_uint_read(&store, header.data, header.length); > + break; > + case DYN_COL_DOUBLE: > + rc= dynamic_column_double_read(&store, header.data, header.length); > + break; > + case DYN_COL_STRING: > + rc= dynamic_column_string_read(&store, header.data, header.length); > + break; > + case DYN_COL_DECIMAL: > + rc= dynamic_column_decimal_read(&store, header.data, header.length); > + break; > + case DYN_COL_DATETIME: > + rc= dynamic_column_date_time_read(&store, header.data, > + header.length); > + break; > + case DYN_COL_DATE: > + rc= dynamic_column_date_read(&store, header.data, header.length); > + break; > + case DYN_COL_TIME: > + rc= dynamic_column_time_read(&store, header.data, header.length); > + break; > + case DYN_COL_NULL: > + default: > + rc= ER_DYNCOL_FORMAT; > + goto end; > + } > + if (rc != ER_DYNCOL_OK) > + { > + DBUG_ASSERT(rc < 0); > + DBUG_PRINT("info", ("Field order: %u Can't read data: %i", > + (uint)i, (int) rc)); > + goto end; > + } > + } > > - nm= uint2korr(read); /* Column nummber */ > - type_and_offset_read(&tp, &offs, read, offset_size); > - if (k == start) > - first_offset= offs; > - else if (offs < first_offset) > + rc= ER_DYNCOL_OK; > +end: > + DBUG_RETURN(rc); > +} > + > + > +enum enum_dyncol_func_result > +dynamic_column_val_str(DYNAMIC_STRING *str, DYNAMIC_COLUMN_VALUE *val, > + CHARSET_INFO *cs, my_bool quote) > +{ > + char buff[40]; > + int len; > + switch (val->type) { > + case DYN_COL_INT: > + len= snprintf(buff, sizeof(buff), "%lld", val->x.long_value); > + if (dynstr_append_mem(str, buff, len)) > + return ER_DYNCOL_RESOURCE; > + break; > + case DYN_COL_UINT: > + len= snprintf(buff, sizeof(buff), "%llu", val->x.ulong_value); > + if (dynstr_append_mem(str, buff, len)) > + return ER_DYNCOL_RESOURCE; > + break; > + case DYN_COL_DOUBLE: > + len= snprintf(buff, sizeof(buff), "%lg", val->x.double_value); > + if (dynstr_realloc(str, len + (quote ? 2 : 0))) > + return ER_DYNCOL_RESOURCE; > + if (quote) > + str->str[str->length++]= '"'; > + dynstr_append_mem(str, buff, len); > + if (quote) > + str->str[str->length++]= '"'; > + break; > + case DYN_COL_STRING: > + { > + char *alloc= NULL; > + char *from= val->x.string.value.str; > + uint bufflen; > + my_bool conv= !my_charset_same(val->x.string.charset, cs); > + my_bool rc; > + len= val->x.string.value.length; > + bufflen= (len * (conv ? cs->mbmaxlen : 1)); > + if (dynstr_realloc(str, bufflen)) > + return ER_DYNCOL_RESOURCE; > + > + // guaranty UTF-8 string for value > + if (!my_charset_same(val->x.string.charset, cs)) > + { > + uint dummy_errors; > + if (!quote) > { > - dynamic_column_column_free(&tmp); > - rc= ER_DYNCOL_FORMAT; > - goto end; > + /* convert to the destination */ > + str->length+= copy_and_convert_extended(str->str, bufflen, > + cs, > + from, len, > + val->x.string.charset, > + &dummy_errors); > + return ER_DYNCOL_OK; > } > - > - offs+= plan[i].ddelta; > - int2store(write, nm); > - /* write rest of data at write + COLUMN_NUMBER_SIZE */ > - type_and_offset_store(write, new_offset_size, tp, offs); > - write+= new_entry_size; > + if ((alloc= (char *)my_malloc(bufflen, MYF(0)))) > + { > + len= > + copy_and_convert_extended(alloc, bufflen, cs, > + from, len, val->x.string.charset, > + &dummy_errors); > + from= alloc; > + } > + else > + return ER_DYNCOL_RESOURCE; > } > + if (quote) > + rc= dynstr_append_quoted(str, from, len); > + else > + rc= dynstr_append_mem(str, from, len); > + if (alloc) > + my_free(alloc); > + if (rc) > + return ER_DYNCOL_RESOURCE; > + break; > } > + case DYN_COL_DECIMAL: > + len= sizeof(buff); > + decimal2string(&val->x.decimal.value, buff, &len, > + 0, val->x.decimal.value.frac, > + '0'); > + if (dynstr_append_mem(str, buff, len)) > + return ER_DYNCOL_RESOURCE; > + break; > + case DYN_COL_DATETIME: > + case DYN_COL_DATE: > + case DYN_COL_TIME: > + len= my_TIME_to_str(&val->x.time_value, buff, AUTO_SEC_PART_DIGITS); > + if (dynstr_realloc(str, len + (quote ? 2 : 0))) > + return ER_DYNCOL_RESOURCE; > + if (quote) > + str->str[str->length++]= '"'; > + dynstr_append_mem(str, buff, len); > + if (quote) > + str->str[str->length++]= '"'; > + break; > + case DYN_COL_NULL: > + if (dynstr_append_mem(str, "null", 4)) > + return ER_DYNCOL_RESOURCE; > + break; > + default: > + return(ER_DYNCOL_FORMAT); > + } > + return(ER_DYNCOL_OK); > +} > > - /* copy first the data that was not replaced in original packed data */ > - if (start < end) > + > +enum enum_dyncol_func_result > +dynamic_column_val_long(longlong *ll, DYNAMIC_COLUMN_VALUE *val) > +{ > + enum enum_dyncol_func_result rc= ER_DYNCOL_OK; > + *ll= 0; > + switch (val->type) { > + case DYN_COL_INT: > + *ll= val->x.long_value; > + break; > + case DYN_COL_UINT: > + *ll= (longlong)val->x.ulong_value; > + if (val->x.ulong_value > ULONGLONG_MAX) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_DOUBLE: > + *ll= (longlong)val->x.double_value; > + if (((double) *ll) != val->x.double_value) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_STRING: > { > - /* Add old data last in 'tmp' */ > - size_t data_size= > - get_length_interval(header_base + start * entry_size, > - header_base + end * entry_size, > - header_end, offset_size, max_offset); > - if ((long) data_size < 0 || > - data_size > max_offset - first_offset) > + longlong i= 0, sign= 1; > + char *src= val->x.string.value.str; > + uint len= val->x.string.value.length; > + > + while (len && my_isspace(&my_charset_latin1, *src)) src++,len--; > + > + if (len) > { > - dynamic_column_column_free(&tmp); > - rc= ER_DYNCOL_FORMAT; > - goto end; > + if (*src == '-') > + { > + sign= -1; > + src++; > + } else if (*src == '-') > + src++; > + while(len && my_isdigit(&my_charset_latin1, *src)) > + { > + i= i * 10 + (*src - '0'); > + src++; > + } > } > - > - memcpy(tmp.str + tmp.length, (char *)header_end + first_offset, > - data_size); > - tmp.length+= data_size; > + else > + rc= ER_DYNCOL_TRUNCATED; > + if (len) > + rc= ER_DYNCOL_TRUNCATED; > + *ll= i * sign; > + break; > } > + case DYN_COL_DECIMAL: > + if (decimal2longlong(&val->x.decimal.value, ll) != E_DEC_OK) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_DATETIME: > + *ll= (val->x.time_value.year * 10000000000L + > + val->x.time_value.month * 100000000L + > + val->x.time_value.day * 1000000 + > + val->x.time_value.hour * 10000 + > + val->x.time_value.minute * 100 + > + val->x.time_value.second) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_DATE: > + *ll= (val->x.time_value.year * 10000 + > + val->x.time_value.month * 100 + > + val->x.time_value.day) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_TIME: > + *ll= (val->x.time_value.hour * 10000 + > + val->x.time_value.minute * 100 + > + val->x.time_value.second) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_NULL: > + rc= ER_DYNCOL_TRUNCATED; > + break; > + default: > + return(ER_DYNCOL_FORMAT); > + } > + return(rc); > +} > + > > - /* new data adding */ > - if (i < add_column_count) > +enum enum_dyncol_func_result > +dynamic_column_val_double(double *dbl, DYNAMIC_COLUMN_VALUE *val) > +{ > + enum enum_dyncol_func_result rc= ER_DYNCOL_OK; > + *dbl= 0; > + switch (val->type) { > + case DYN_COL_INT: > + *dbl= (double)val->x.long_value; > + if (((longlong) *dbl) != val->x.long_value) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_UINT: > + *dbl= (double)val->x.ulong_value; > + if (((ulonglong) *dbl) != val->x.ulong_value) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_DOUBLE: > + *dbl= val->x.double_value; > + break; > + case DYN_COL_STRING: > { > - if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > - { > - int2store(write, plan[i].num[0]); > - type_and_offset_store(write, new_offset_size, > - plan[i].val[0].type, > - tmp.length - > - (FIXED_HEADER_SIZE + new_header_size)); > - write+= new_entry_size; > - data_store(&tmp, plan[i].val); /* Append new data */ > - } > - data_delta= plan[i].ddelta; > + char *str, *end; > + if ((str= malloc(val->x.string.value.length + 1))) > + return ER_DYNCOL_RESOURCE; > + memcpy(str, val->x.string.value.str, val->x.string.value.length); > + str[val->x.string.value.length]= '\0'; > + *dbl= strtod(str, &end); > + if (*end != '\0') > + rc= ER_DYNCOL_TRUNCATED; > } > - } > - dynamic_column_column_free(str); > - *str= tmp; > + case DYN_COL_DECIMAL: > + if (decimal2double(&val->x.decimal.value, dbl) != E_DEC_OK) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_DATETIME: > + *dbl= (double)(val->x.time_value.year * 10000000000L + > + val->x.time_value.month * 100000000L + > + val->x.time_value.day * 1000000 + > + val->x.time_value.hour * 10000 + > + val->x.time_value.minute * 100 + > + val->x.time_value.second) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_DATE: > + *dbl= (double)(val->x.time_value.year * 10000 + > + val->x.time_value.month * 100 + > + val->x.time_value.day) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_TIME: > + *dbl= (double)(val->x.time_value.hour * 10000 + > + val->x.time_value.minute * 100 + > + val->x.time_value.second) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_NULL: > + rc= ER_DYNCOL_TRUNCATED; > + break; > + default: > + return(ER_DYNCOL_FORMAT); > } > + return(rc); > +} > > - rc= ER_DYNCOL_OK; > > -end: > - my_free(plan); > - return rc; > +/** > + Convert to JSON > > -create_new_string: > - /* There is no columns from before, so let's just add the new ones */ > - rc= ER_DYNCOL_OK; > - if (not_null != 0) > - rc= dynamic_column_create_many_internal(str, add_column_count, > - column_numbers, values, > - str->str == NULL); > - goto end; > + @param str The packed string > + @param json Where to put json result > + > + @return ER_DYNCOL_* return code > +*/ > + > +enum enum_dyncol_func_result > +dynamic_column_json(DYNAMIC_COLUMN *str, DYNAMIC_STRING *json) > +{ > + DYN_HEADER header; > + uint i; > + enum enum_dyncol_func_result rc; > + > + bzero(json, sizeof(DYNAMIC_STRING)); /* In case of errors */ > + if (str->length == 0) > + return ER_DYNCOL_OK; /* no columns */ > + > + if ((rc= init_read_hdr(&header, str)) < 0) > + return rc; > + > + if (header.entry_size * header.column_count + FIXED_HEADER_SIZE > > + str->length) > + return ER_DYNCOL_FORMAT; > + > + if (init_dynamic_string(json, NULL, str->length * 2, 100)) > + return ER_DYNCOL_RESOURCE; > + > + if (dynstr_append_mem(json, "[", 1)) > + return ER_DYNCOL_RESOURCE; > + rc= ER_DYNCOL_RESOURCE; > + for (i= 0, header.entry= header.header; > + i < header.column_count; > + i++, header.entry+= header.entry_size) > + { > + DYNAMIC_COLUMN_VALUE val; > + if (i != 0 && dynstr_append_mem(json, ",", 1)) > + goto err; > + header.length= > + hdr_interval_length(&header, header.entry + header.entry_size); > + header.data= header.dtpool + header.offset; > + /* > + Check that the found data is withing the ranges. This can happen if > + we get data with wrong offsets. > + */ > + if (header.length == DYNCOL_OFFSET_ERROR || > + header.length > INT_MAX || header.offset > header.data_size) > + { > + rc= ER_DYNCOL_FORMAT; > + goto err; > + } > + if ((rc= dynamic_column_get_value(&header, &val)) < 0 || > + dynstr_append_mem(json, "{", 1)) > + goto err; > + if (header.format == DYNCOL_FMT_NUM) > + { > + uint nm= uint2korr(header.entry); > + if (dynstr_realloc(json, DYNCOL_NUM_CHAR + 3)) > + goto err; > + json->str[json->length++]= '"'; > + json->length+= (snprintf(json->str + json->length, > + DYNCOL_NUM_CHAR, "%u", nm)); > + } > + else > + { > + uint len= header.entry[0]; > + if (dynstr_realloc(json, len + 3)) > + goto err; > + json->str[json->length++]= '"'; > + memcpy(json->str + json->length, (const void *)header.nmpool + > + uint2korr(header.entry + 1), len); > + json->length+= len; > + } > + json->str[json->length++]= '"'; > + json->str[json->length++]= ':'; > + if ((rc= dynamic_column_val_str(json, &val, > + &my_charset_utf8_general_ci, TRUE)) < 0 || > + dynstr_append_mem(json, "}", 1)) > + goto err; > + } > + if (dynstr_append_mem(json, "]", 1)) > + return ER_DYNCOL_RESOURCE; > + return ER_DYNCOL_OK; > + > +err: > + json->length= 0; > + return rc; > } > > > /** > - Update the packed string with the given column > + Convert to DYNAMIC_COLUMN_VALUE values and names (LEX_STING) dynamic array > > - @param str String where to write the data > - @param column_number Array of columns number > - @param values Array of columns values > + @param str The packed string > + @param names Where to put names > + @param vals Where to put values > + @param free_names pointer to free names buffer if there is it. > > @return ER_DYNCOL_* return code > */ > > - > -int dynamic_column_update(DYNAMIC_COLUMN *str, uint column_nr, > - DYNAMIC_COLUMN_VALUE *value) > +enum enum_dyncol_func_result > +dynamic_column_vals(DYNAMIC_COLUMN *str, > + DYNAMIC_ARRAY *names, DYNAMIC_ARRAY *vals, > + char **free_names) > { > - return dynamic_column_update_many(str, 1, &column_nr, value); > + DYN_HEADER header; > + char *nm; > + uint i; > + enum enum_dyncol_func_result rc; > + > + *free_names= 0; > + bzero(names, sizeof(DYNAMIC_ARRAY)); /* In case of errors */ > + bzero(vals, sizeof(DYNAMIC_ARRAY)); /* In case of errors */ > + if (str->length == 0) > + return ER_DYNCOL_OK; /* no columns */ > + > + if ((rc= init_read_hdr(&header, str)) < 0) > + return rc; > + > + if (header.entry_size * header.column_count + FIXED_HEADER_SIZE > > + str->length) > + return ER_DYNCOL_FORMAT; > + > + if (init_dynamic_array(names, sizeof(LEX_STRING), > + header.column_count, 0) || > + init_dynamic_array(vals, sizeof(DYNAMIC_COLUMN_VALUE), > + header.column_count, 0) || > + (header.format == DYNCOL_FMT_NUM && > + !(*free_names= (char *)malloc(DYNCOL_NUM_CHAR * header.column_count)))) why do you need a special malloc() for names, you can put them in the buffer of 'names' dynarray. it'll be a lot more convenient for the caller (no free_names to care about). > + { > + rc= ER_DYNCOL_RESOURCE; > + goto err; > + } > + nm= *free_names; > + > + for (i= 0, header.entry= header.header; > + i < header.column_count; > + i++, header.entry+= header.entry_size) > + { > + DYNAMIC_COLUMN_VALUE val; > + LEX_STRING name; > + header.length= > + hdr_interval_length(&header, header.entry + header.entry_size); > + header.data= header.dtpool + header.offset; > + /* > + Check that the found data is withing the ranges. This can happen if > + we get data with wrong offsets. > + */ > + if (header.length == DYNCOL_OFFSET_ERROR || > + header.length > INT_MAX || header.offset > header.data_size) > + { > + rc= ER_DYNCOL_FORMAT; > + goto err; > + } > + if ((rc= dynamic_column_get_value(&header, &val)) < 0) > + goto err; > + > + if (header.format == DYNCOL_FMT_NUM) > + { > + uint num= uint2korr(header.entry); > + name.str= nm; > + name.length= snprintf(nm, DYNCOL_NUM_CHAR, "%u", num); > + nm+= name.length + 1; > + } > + else > + { > + name.length= header.entry[0]; > + name.str= (char *)header.nmpool + uint2korr(header.entry + 1); > + } > + /* following is preallocated and so do not fail */ > + (void) insert_dynamic(names, (uchar *)&name); > + (void) insert_dynamic(vals, (uchar *)&val); > + } > + return ER_DYNCOL_OK; > + > +err: > + delete_dynamic(names); > + delete_dynamic(vals); > + if (*free_names) > + my_free(*free_names); > + *free_names= 0; > + return rc; > } > Regards, Sergei

1 0

[Maria-developers] a mysterious (to me) block of code in the middle of ha_recover
by Rich Prohaska 13 Dec '12

13 Dec '12

Hello, The ha_recover function in MySQL 5.5.28 has a block of code in the middle of it labelled with WILL_BE_DELETED_LATER that gets compiled into mysqld. The ha_recover function in Mariadb 5.5.28a does not have this code in it. It appears that the code forces prepared transactions to be rolled back in MySQL even when mysqld is past the recover phase. Does anyone know what this code does? Does anyone know why this code exists in MySQL but not in Mariadb? Thanks

2 1

[Maria-developers] a mysterious (to me) block of code in the middle of ha_recover
by Rich Prohaska 13 Dec '12

13 Dec '12

Hello, The ha_recover function in MySQL 5.5.28 has a block of code in the middle of it labelled with WILL_BE_DELETED_LATER that gets compiled into mysqld. The ha_recover function in Mariadb 5.5.28a does not have this code in it. It appears that the code forces prepared transactions to be rolled back in MySQL. Does anyone know what this code does? Does anyone know why this code exists in MySQL but not in Mariadb? Thanks Rich Prohaska

1 0

[Maria-developers] release day of week for security releases
by Daniel Black 07 Dec '12

07 Dec '12

Hi, Thanks for the latest releases with security fixes. While I appreciate that all of the development of these security fixes was in public (without mentioning it was a security fix - well at least the remote code exec), I'm wondering if security releases could occur on a weekday where sysadmins need not forsake part of their weekend to correct a public vulnerability. Just my thoughts and preferences. I appreciate others may consider things different. It also appears that the fedora 17 mariadb galera updates are only partially pushed. Maybe its just my setup after switching from non-galera repo. $ sudo yum update [sudo] password for dan: Loaded plugins: langpacks, presto, priorities, refresh-packagekit, security 38 packages excluded due to repository priority protections Resolving Dependencies --> Running transaction check ---> Package MariaDB-client.x86_64 0:5.5.25-1 will be updated ---> Package MariaDB-client.x86_64 0:5.5.27-1 will be an update ---> Package MariaDB-common.x86_64 0:5.5.25-1 will be updated ---> Package MariaDB-common.x86_64 0:5.5.27-1 will be an update ---> Package MariaDB-server.x86_64 0:5.5.25-1 will be updated ---> Package MariaDB-server.x86_64 0:5.5.27-1 will be obsoleting ---> Package mysql.x86_64 0:5.5.28-1.fc17 will be obsoleted --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================== Installing: MariaDB-server x86_64 5.5.27-1 mariadb 33 M replacing mysql.x86_64 5.5.28-1.fc17 Updating: MariaDB-client x86_64 5.5.27-1 mariadb 8.6 M MariaDB-common x86_64 5.5.27-1 mariadb 23 k Transaction Summary ============================================================================================================================================================== Install 1 Package Upgrade 2 Packages Total size: 42 M Total download size: 8.6 M Is this ok [y/N]: y Downloading Packages: Setting up and reading Presto delta metadata MariaDB-5.5.27-fedora17-x86_64 FAILED HTTP Error 404 - Not Found : http://yum.mariadb.org/5.5-galera/fedora17-amd64/rpms/MariaDB-5.5.27-fedora… http://yum.mariadb.org/5.5-galera/fedora17-amd64/rpms/MariaDB-5.5.27-fedora…: [Errno 14] HTTP Error 404 - Not Found : http://yum.mariadb.org/5.5-galera/fedora17-amd64/rpms/MariaDB-5.5.27-fedora… /etc/yum.repos.d/mariadb.repo : [mariadb] name = MariaDB baseurl = http://yum.mariadb.org/5.5-galera/fedora17-amd64 gpgcheck=1 enabled=1 -- Daniel Black, Engineer @ Open Query (http://openquery.com) Remote expertise & maintenance for MySQL/MariaDB server environments.

2 2

[Maria-developers] [[Bug] skip-character-set-client-handshake]
by spumer-tm＠yandex.ru 07 Dec '12

07 Dec '12

>Description: <I'm using MariaDB with my gameservers. Server works with db through sourcemod. In my db i'am using utf8 encoding, and all works fine on MariaDB 5.3.8 with params: init-connect = "SET NAMES 'utf8' COLLATE 'utf8_general_ci'" skip-character-set-client-handshake But after upgrade to 5.5.27 i found a 'bug'. When i'm send query from my game server, over time character_set_connection, character_set_client and character_set_results set to latin1. After enable log i saw: 88 Connect login@domain as anonymous on sourcebans 88 Query SET NAMES 'utf8' COLLATE 'utf8_general_ci' 88 Query SET NAMES latin1 88 Query SHOW VARIABLES LIKE '%character_set%' But my request is only show vars. I assumed that sourcemod is guilty. But skip-character-set-client-handshake is set! I disable it, and try again. At this time i send SET collation_connection = utf8_general_ci and SET NAMES utf8 after my query. But over time it set to latin1 again. No errosr in error.log Sorry for my english.> >How-To-Repeat: <I don't know how to repeat it manually (without sourcemod app)> >Fix: <It works fine in previous versions. Just not accept NAMES from client.> >Submitter-Id: <submitter ID> >Originator: spumer >Organization: <russian game servers> >MySQL support: [ email support ] >Synopsis: <skip-character-set-client-handshake not work correctly> >Severity: >Priority: >Category: mysql >Class: >Release: mysql-5.5.28a (MariaDB Server) >C compiler: gcc-4.5.3 (Gentoo 4.5.3-r1 p1.0, pie-0.4.5) 4.5.3 >C++ compiler: g++-4.5.3(Gentoo 4.5.3-r1 p1.0, pie-0.4.5) 4.5.3 >Environment: <VMware ESXi 5, Gentoo, Glibc 2.15> System: Linux localhost 3.6.1-gentoo #1 SMP Mon Oct 8 06:00:23 YEKT 2012 x86_64 Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz GenuineIntel GNU/Linux Some paths: /usr/bin/perl /usr/bin/make /usr/bin/gmake /usr/bin/gcc /usr/bin/cc GCC: Используются внутренние спецификации. COLLECT_GCC=/usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/gcc COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/4.5.3/lto-wrapper Целевая архитектура: x86_64-pc-linux-gnu Параметры конфигурации: /var/tmp/portage/sys-devel/gcc-4.5.3-r1/work/gcc-4.5.3/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.5.3/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.5.3 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.5.3/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.5.3/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.5.3/include/g++-v4 --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --disable-altivec --disable-fixed-point --without-ppl --without-cloog --disable-lto --enable-nls --without-included-gettext --with-system-zlib --disable-werror --enable-secureplt --enable-multilib --enable-libmudflap --disable-libssp --enable-libgomp --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/4.5.3/python --enable-checking=release --disable-libgcj --enable-languages=c,c++,fortran --enable-shared --enable-threads=posix --enable-__cxa_a! texit --enable-clocale=gnu --enable-targets=all --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.5.3-r1 p1.0, pie-0.4.5' Модель многопоточности: posix gcc версия 4.5.3 (Gentoo 4.5.3-r1 p1.0, pie-0.4.5) Compilation info (call): CC='/usr/bin/gcc' CFLAGS='-Wall -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' CXX='/usr/bin/c++' CXXFLAGS='-Wall -Wno-unused-parameter -fno-implicit-templates -fno-exceptions -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' LDFLAGS='' ASFLAGS='' Compilation info (used): CC='/usr/bin/gcc' CFLAGS='-Wall -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' CXX='/usr/bin/c++' CXXFLAGS='-Wall -Wno-unused-parameter -fno-implicit-templates -fno-exceptions -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' LDFLAGS='' ASFLAGS='' LIBC: lrwxrwxrwx 1 root root 12 дек. 2 12:43 /lib/libc.so.6 -> libc-2.15.so -rwxr-xr-x 1 root root 1734576 дек. 2 12:42 /lib/libc-2.15.so -rw-r--r-- 1 root root 4594916 дек. 2 12:42 /usr/lib/libc.a -rwxr-xr-x 1 root root 253 дек. 2 12:42 /usr/lib/libc.so Perl: This is perl 5, version 12, subversion 4 (v5.12.4) built for x86_64-linux-thread-multi

2 1

[Maria-developers] test case for mdev 3885 is named incorrectly
by Rich Prohaska 06 Dec '12

06 Dec '12

revision 3584 named the test case for mdev 3885 "test_mdev3855" (NOTE 3855).

2 1

Re: [Maria-developers] How to get JDBC launchpad branch to trigger build in buildbot?
by Kristian Nielsen 03 Dec '12

03 Dec '12

"Vladislav Vaintroub" <wlad(a)montyprogram.com> writes: > there is a change in the Launchpad repository . I have done some things > that I hoped would help, but it looks like it does not help. > I subscribed Buildbot user to the changes in branch > https://code.launchpad.net/~maria-captains/mariadb-java-client/trunk It seems to be subscribed as "Branch attribute notifications only". Check the wiki page, it has to be "branch revision notifications only" (and it's probably not very useful to send review requests to Buildbot :-) > + "lp:~maria-captains/mariadb-java-client/trunk" : "jdbc", As explained on the wiki page, the branch should be on Launchpad under some name lp:~maria-captains/maria/XXX (for some XXX) for this to work. > +f_win_jdbc = factory.BuildFactory() > +f_win32_debug.addStep(ShellCommand( ^^^^^^^^^^^^^ copy-paste error? "f_win_jdbc" ? > +f_win_rqg.addStep(ShellCommand( ^^^^^^^^^ "f_win_jdbc" ? > + name = "bzr_checkout", > + command=["dojob", "bzr" ,"checkout", "-r", WithProperties("%(revision)s"), WithProperties("lp:~maria-captains/maria/%(branch)s"), WithProperties("c:\\buildbot\\%(buildername)s\\build")], Did you mean "lp:~maria-captains/mariadb-java-client/%(branch)s" here? Or did you mean lp:~maria-captains/maria/jdbc in the branch declaration? > + WithProperties("cd c:\\buildbot\\%(buildername)s\\build && C:\\buildscripts\\build_jdbc") Did you consider putting this script into the source tree instead? That makes it easier when you have to modify the script along with the code in the source three: old branches continue to use the old version of the script with the old source version, new branches can use the new script with the new source changes. (But it's up to you what works best in this particular case). > + "nextBuild": myNextBuild, The myNextBuild() function selects among mariadb branches, that's probably not very useful for a jdbc builder? > see. Maybe you can have a quick look, if you got time? Would be a great > help. Ok, I will go through it all and try to get everything working. It's probably better to add a separate configuration (branch map, schedulers, builders) for the jdbc driver to not get it mixed up with the mariadb source builds. - Kristian.

2 2

[Maria-developers] API Documentaion for Replication Features
by Gordan Bobic 03 Dec '12

03 Dec '12

Hi, I have a need for replicating data from multiple masters (independent data going to different tables). I have looked into the following with no success: Tungsten: Slow, CPU hungry, outrageously bloated, and unacceptably unreliable (replication will stall silently at times, and the only way to resume it is to offline and online the replication service). mysql-replication-listener: Just about got it to work with local binlogs, but haven't managed to get it to connect via TCP to a MySQL server. So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: 1) Slave TCP connections to the master and binlog fetching 2) Binlog parsing 3) Config file parsing In the longer term I'd like to write a patch that adds multiple-master functionality into MySQL itself. For configuring it, I was thinking about the following WRT configuration in my.cnf with multiple section instances, e.g.: [replicator] server-id=12345 master-host=server1 master-port=3306 master-user=user1 master-password=password1 replicate-do-table=db1.table1 [replicator] server-id=12345 master-host=server2 master-port=3306 master-user=user2 master-password=password2 replicate-do-table=db2.table2 etc. Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand. TIA. Gordan

4 9

[Maria-developers] MariaDB 5.5.28a release prep
by Sergei Golubchik 28 Nov '12

28 Nov '12

Preliminary release notes and changelog pages have been created for 5.5.28a https://kb.askmonty.org/en/mariadb-5528a-release-notes/ https://kb.askmonty.org/en/mariadb-5528a-changelog/ Please let me know if there is anything in particular which should be mentioned in the release notes. Thanks, Sergei _______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers(a)lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp

1 0

[Maria-developers] Review of mdev-452
by Michael Widenius 28 Nov '12

28 Nov '12

Hi! Here is the review of https://mariadb.atlassian.net/browse/MDEV-452 (Add full support for auto-initialized/updated timestamp and datetime) === modified file 'mysql-test/r/log_tables.result' --- mysql-test/r/log_tables.result 2011-10-19 19:45:18 +0000 +++ mysql-test/r/log_tables.result 2012-10-17 12:43:56 +0000 @@ -53,7 +53,7 @@ ERROR HY000: You can't use locks with lo show create table mysql.general_log; Table Create Table general_log CREATE TABLE `general_log` ( - `event_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + `event_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), Why does all log tables now have CURRENT_TIMESTAMP(6) instead of CURRENT_TIMESTAMP ? This would cause a problem for someone that updates to MySQL 10.0 and then want to go back... Many of the other result file changes comes from the same change for general_log and slow_log. Is this also the case for MySQL 5.6 or only for MariaDB? Answer from Timour on IRC: Timour> This comes from MySQL 5.6 and we need to stay compatbile. === modified file 'sql/event_db_repository.cc' --- sql/event_db_repository.cc 2012-09-04 16:26:30 +0000 +++ sql/event_db_repository.cc 2012-10-17 12:43:56 +0000 @@ -829,9 +829,6 @@ Event_db_repository::update_event(THD *t (int) table->field[ET_FIELD_ON_COMPLETION]->val_int())) goto end; - /* Don't update create on row update. */ - table->timestamp_field_type= TIMESTAMP_NO_AUTO_SET; - Why don't we anymore need to tell the rest of the code that the automatic fields shouldn't be updated from this function? Timour> As I remember, the main change compared to the old design is Timour> that we no longer update timestamps inside the handler Timour> So the place where timestamp updates are triggered is such Timour> that such updates will no longer be triggered in these cases. Yes, mysql_event_fill_row() doesn't call fill_record_n_invoke_before_triggers(). It instead has own code that probably not calling setting of any default fields. This does however mean that one can never get automatic timestamps with events. === modified file 'sql/field.cc' --- sql/field.cc 2012-09-08 22:22:06 +0000 +++ sql/field.cc 2012-10-17 12:43:56 +0000 @@ -4419,10 +4419,12 @@ Field_timestamp::Field_timestamp(uchar * { /* For 4.0 MYD and 4.0 InnoDB compatibility */ flags|= UNSIGNED_FLAG | BINARY_FLAG; - if (unireg_check != NONE && !share->timestamp_field) + if (unireg_check != NONE) { - /* This timestamp has auto-update */ - share->timestamp_field= this; + /* + This TIMESTAMP column is hereby quietly assumed to have an insert or + update default function. + */ We need to here add a comment what the TIMESTAMP_FLAG now means, as it has lost it's orginal meaning. Add to the above comment: /* We mark the flag with TIMESTAMP_FLAG to indicate to the client that this field will be automaticly updated on insert */ flags|= TIMESTAMP_FLAG; if (unireg_check != TIMESTAMP_DN_FIELD) flags|= ON_UPDATE_NOW_FLAG; Please also fix the function comment that refers to TABLE::timestamp_field <cut> +void Field_timestamp::set_explicit_default(Item *value) +{ + if (value && + ((value->type() == Item::DEFAULT_VALUE_ITEM && + !((Item_default_value*)value)->arg) || + (!maybe_null() && value->is_null()))) + return; + flags|= HAS_EXPLICIT_DEFAULT; +} The above function confused me greatly (mostly because of names) You should not have to check for value != 0. An item should never be 0. I noticed that some code called set_explicit_default() with NULL, but it would be better to for this add a general item function: field->set_explicit_default(NULL) -> field->mark_that_field_has_been_given_a_value() Where: Field::mark_that_field_has_been_given_a_value() { flags|= HAS_EXPLICIT_VALUE; } (Note name change for HAS_EXPLICIT_DEFAULT) This can be inline and we will win some function calls. The set_explicit_default() function also needs a comment as it's very confusing. To clarify the code, lets first add a function: /* Returns 1 if the default returns the default value for a column */ Item_default_value::default_column() { return arg != 0; } And then original function would look like: /** Mark columns that has been given a value with HAS_EXPLICIT_VALUE. For timestamp columns, the only case where a column is not marked with been given a value are: - It's explicitely assigned with DEFAULT - We assign NULL to a timestamp field that is defined as NOT NULL. This is how MySQL has worked since it's start. */ void Field_timestamp::set_explicit_default(Item *value) { if (((value->type() == Item::DEFAULT_VALUE_ITEM && !(Item_default_value*) value)->default_column()) || (!maybe_null() && value->is_null())) return; mark_that_field_has_been_given_a_value(); } <cut> + +int Field_datetime::set_time() +{ + THD *thd= current_thd; current_thd -> table->in_use; + MYSQL_TIME now_time; + thd->variables.time_zone->gmt_sec_to_TIME(&now_time, thd->query_start()); + now_time.second_part= thd->query_start_sec_part(); + set_notnull(); + store_TIME(&now_time); + thd->time_zone_used= 1; + return 0; +} @@ -8857,16 +8866,37 @@ bool Create_field::init(THD *thd, char * { uint sign_len, allowed_type_modifier= 0; ulong max_field_charlength= MAX_FIELD_CHARLENGTH; + const bool on_update_is_function= + (fld_on_update_value != NULL && + fld_on_update_value->type() == Item::FUNC_ITEM); DBUG_ENTER("Create_field::init()"); field= 0; field_name= fld_name; - def= fld_default_value; flags= fld_type_modifier; option_list= create_opt; - unireg_check= (fld_type_modifier & AUTO_INCREMENT_FLAG ? - Field::NEXT_NUMBER : Field::NONE); + + if (fld_default_value != NULL && fld_default_value->type() == Item::FUNC_ITEM) + { + /* There is a function default for insertions. */ + def= NULL; + unireg_check= on_update_is_function ? + Field::TIMESTAMP_DNUN_FIELD : // for insertions and for updates. + Field::TIMESTAMP_DN_FIELD; // only for insertions. Add braces around the on_update_is... to get better indentation. + } + else + { + /* No function default for insertions. Either NULL or a constant. */ + def= fld_default_value; + if (on_update_is_function) + unireg_check= Field::TIMESTAMP_UN_FIELD; // function default for updates + else + unireg_check= (fld_type_modifier & AUTO_INCREMENT_FLAG) != 0 ? + Field::NEXT_NUMBER : // Automatic increment. + Field::NONE; Add () around expression. + } + <cut> + + +/** + Mark the field as having an explicit default value. + + @param value if available, the value that the field is being set to + + @note + Fields that have an explicit default value should not be updated + automatically via the DEFAULT or ON UPDATE functions. The functions + that deal with data change functionality (INSERT/UPDATE/LOAD), + determine if there is an explicit value for each field before performing + the data change, and call this method to mark the field. + + If the 'value' parameter is NULL, then the field is marked unconditionally + as having an explicit value. If 'value' is not NULL, then it can be further + analyzed to check if it really should count as a value. +*/ + +void Field::set_explicit_default(Item *value) +{ /* Return if value is DEFAULT */ + if (value && value->type() == Item::DEFAULT_VALUE_ITEM && + !((Item_default_value*)value)->arg) + return; + flags|= HAS_EXPLICIT_DEFAULT; +} Remove checking if value is null. flags|= HAS_EXPLICIT_DEFAULT -> flags|= HAS_EXPLICIT_VALUE === modified file 'sql/field.h' @@ -1239,12 +1270,26 @@ class Field_timestamp :public Field_str virtual int set_time(); <cut> + virtual void set_explicit_default(Item *value); + virtual int evaluate_insert_default_function() + { + int res= 0; + if (has_insert_default_function()) + res= set_time(); + return res; + } + virtual int evaluate_update_default_function() + { + int res= 0; + if (has_update_default_function()) + res= set_time(); + return res; + } /* Get TIMESTAMP field value as seconds since begging of Unix Epoch */ virtual my_time_t get_timestamp(ulong *sec_part) const; virtual void store_TIME(my_time_t timestamp, ulong sec_part) @@ -1252,7 +1297,6 @@ class Field_timestamp :public Field_str int4store(ptr,timestamp); } bool get_date(MYSQL_TIME *ltime, ulonglong fuzzydate); - timestamp_auto_set_type get_auto_set_type() const; uchar *pack(uchar *to, const uchar *from, uint max_length __attribute__((unused))) { @@ -1503,6 +1547,28 @@ class Field_datetime :public Field_tempo void sql_type(String &str) const; bool zero_pack() const { return 1; } bool get_date(MYSQL_TIME *ltime, ulonglong fuzzydate); + virtual int set_time(); + virtual void set_default() + { + if (has_insert_default_function()) + set_time(); + else + Field::set_default(); + } + virtual int evaluate_insert_default_function() + { + int res= 0; + if (has_insert_default_function()) + res= set_time(); + return res; + } + virtual int evaluate_update_default_function() + { + int res= 0; + if (has_update_default_function()) + res= set_time(); + return res; + } It's a bit of a shame that we need to have the exact same functions twice? One way to solve it would be to add these functions to Field_str, as both Field_timestamp() and Field_datetime() depends on this class. === modified file 'sql/sql_base.cc' <cut> static bool -fill_record(THD * thd, List<Item> &fields, List<Item> &values, +fill_record(THD * thd, TABLE *table_arg, List<Item> &fields, List<Item> &values, bool ignore_errors) { Why this change (was not needed for this functionality). timour> I think I had a different implementation before, then I kept this timrou> change, because it looked simpler. bool -fill_record_n_invoke_before_triggers(THD *thd, List<Item> &fields, +fill_record_n_invoke_before_triggers(THD *thd, TABLE *table, List<Item> &fields, List<Item> &values, bool ignore_errors, - Table_triggers_list *triggers, enum trg_event_type event) { bool result; - result= (fill_record(thd, fields, values, ignore_errors) || + Table_triggers_list *triggers= table->triggers; + result= (fill_record(thd, table, fields, values, ignore_errors) || (triggers && triggers->process_triggers(thd, event, TRG_ACTION_BEFORE, TRUE))); /* @@ -8925,7 +8924,6 @@ fill_record_n_invoke_before_triggers(THD */ if (!result && triggers) { - TABLE *table= 0; List_iterator_fast<Item> f(fields); Item *fld; Item_field *item_field; @@ -8933,9 +8931,7 @@ fill_record_n_invoke_before_triggers(THD { fld= (Item_field*)f++; item_field= fld->filed_for_view_update(); - if (item_field && item_field->field && - (table= item_field->field->table) && - table->vfield) + if (item_field && item_field->field && table && table->vfield) result= update_virtual_fields(thd, table, TRUE); Add: DBUG_ASSERT(table == item_field->field->table) Note that if we can only have one table here, than we could also replace: if (!result && triggers) with: if (!result && triggers && table) As the loop will not do anything if table == 0. <cut> bool -fill_record_n_invoke_before_triggers(THD *thd, Field **ptr, +fill_record_n_invoke_before_triggers(THD *thd, TABLE *table, Field **ptr, List<Item> &values, bool ignore_errors, - Table_triggers_list *triggers, enum trg_event_type event) { bool result; - result= (fill_record(thd, ptr, values, ignore_errors, FALSE) || + Table_triggers_list *triggers= table->triggers; + result= (fill_record(thd, table, ptr, values, ignore_errors, FALSE) || (triggers && triggers->process_triggers(thd, event, TRG_ACTION_BEFORE, TRUE))); /* @@ -9071,7 +9064,6 @@ fill_record_n_invoke_before_triggers(THD */ if (!result && triggers && *ptr) { - TABLE *table= (*ptr)->table; Add DBUG_ASSERT(table == (*ptr)->table); if (table->vfield) result= update_virtual_fields(thd, table, TRUE); } <cut> === modified file 'sql/sql_insert.cc' @@ -1696,13 +1660,32 @@ int write_record(THD *thd, TABLE *table, restore_record(table,record[1]); DBUG_ASSERT(info->update_fields->elements == info->update_values->elements); - if (fill_record_n_invoke_before_triggers(thd, *info->update_fields, + if (fill_record_n_invoke_before_triggers(thd, table, *info->update_fields, *info->update_values, info->ignore, - table->triggers, TRG_EVENT_UPDATE)) goto before_trg_err; + bool different_records= (!records_are_comparable(table) || + compare_record(table)); + /* + Default fields must be updated before checking view updateability. + This branch of INSERT is executed only when a UNIQUE key was violated + with the ON DUPLICATE KEY UPDATE option. In this case the INSERT + operation is transformed to an UPDATE, and the default fields must + be updated as if this is an UPDATE. + */ + if (different_records && table->default_field) + { + bool res; + enum_sql_command cmd= thd->lex->sql_command; + thd->lex->sql_command= SQLCOM_UPDATE; + res= table->update_default_fields(); + thd->lex->sql_command= cmd; + if (res) + goto err; + } Another way to do the above nicer would be to give thd->lex->sql_command as an argument to update_default_fields(). Another option would be to have: update_defaults_fields(table) { return update_defaults_fields(table, table->in_use->lex->cmd); } And use the later in this case. However this is not that critical to fix... <cut> + Field **field,**org_field, *found_next_number_field, **dfield_ptr; You need to do dfield_ptr= 0 to avoid a compiler warning. @@ -2390,6 +2370,12 @@ TABLE *Delayed_insert::get_local_table(T bitmap= (uchar*) (field + share->fields + 1); copy->record[0]= (bitmap + share->column_bitmap_size*3); memcpy((char*) copy->record[0], (char*) table->record[0], share->reclength); + if (share->default_fields) + { + copy->default_field= (Field**) client_thd->alloc((share->default_fields+1)* + sizeof(Field**)); You should check for error condition here. + dfield_ptr= copy->default_field; + } /* Make a copy of all fields. The copied fields need to point into the copied record. This is done @@ -2407,18 +2393,19 @@ TABLE *Delayed_insert::get_local_table(T (*field)->move_field_offset(adjust_ptrs); // Point at copy->record[0] if (*org_field == found_next_number_field) (*field)->table->found_next_number_field= *field; + if (share->default_fields && + ((*org_field)->has_insert_default_function() || + (*org_field)->has_update_default_function())) + { + /* Put the newly copied field into the set of default fields. */ + *dfield_ptr= *field; + (*dfield_ptr)->unireg_check= (*org_field)->unireg_check; It's strange that new_field() resets unireg_check. Don't know why this is done and we should at some point fix that it's not reset and see what happens. + dfield_ptr++; + } } *field=0; <cut> @@ -850,7 +837,7 @@ read_fixed_length(THD *thd, COPY_INFO &i ER(ER_WARN_TOO_FEW_RECORDS), thd->warning_info->current_row_for_warning()); if (!field->maybe_null() && field->type() == FIELD_TYPE_TIMESTAMP) - ((Field_timestamp*) field)->set_time(); + field->set_time(); Shouldn't the above be done now by testing for if the field has an insert_default_function? @@ -994,10 +982,11 @@ read_sep_field(THD *thd, COPY_INFO &info DBUG_RETURN(1); } field->set_null(); + field->set_explicit_default(NULL); if (!field->maybe_null()) { if (field->type() == MYSQL_TYPE_TIMESTAMP) - ((Field_timestamp*) field)->set_time(); + field->set_time(); shouldn't we use an insert_default_function here too ? else if (field != table->next_number_field) field->set_warning(MYSQL_ERROR::WARN_LEVEL_WARN, ER_WARN_NULL_TO_NOTNULL, 1); @@ -1066,7 +1056,7 @@ read_sep_field(THD *thd, COPY_INFO &info DBUG_RETURN(1); } if (!field->maybe_null() && field->type() == FIELD_TYPE_TIMESTAMP) - ((Field_timestamp*) field)->set_time(); + field->set_time(); shouldn't we use an insert_default_function here too ? /* TODO: We probably should not throw warning for each field. But how about intention to always have the same number @@ -1206,7 +1196,7 @@ read_xml_field(THD *thd, COPY_INFO &info if (!field->maybe_null()) { if (field->type() == FIELD_TYPE_TIMESTAMP) - ((Field_timestamp *) field)->set_time(); + field->set_time(); shouldn't we use an insert_default_function here too ? === modified file 'sql/sql_parse.cc' --- sql/sql_parse.cc 2012-10-06 07:30:52 +0000 +++ sql/sql_parse.cc 2012-10-18 12:57:12 +0000 @@ -269,12 +269,14 @@ void init_update_queries(void) CF_CAN_GENERATE_ROW_EVENTS; sql_command_flags[SQLCOM_CREATE_INDEX]= CF_CHANGES_DATA | CF_AUTO_COMMIT_TRANS; sql_command_flags[SQLCOM_ALTER_TABLE]= CF_CHANGES_DATA | CF_WRITE_LOGS_COMMAND | - CF_AUTO_COMMIT_TRANS | CF_REPORT_PROGRESS ; + CF_AUTO_COMMIT_TRANS | CF_REPORT_PROGRESS | + CF_INSERTS_DATA; Should we really have CF_INSERTS_DATA here? We should not update any timestamps for ALTER TABLE? === modified file 'sql/table.cc' @@ -2472,7 +2471,7 @@ int open_table_from_share(THD *thd, TABL } /* - Process virtual columns, if any. + Process virtual and default columns, if any. */ if (!share->vfields) outparam->vfield= NULL; @@ -2484,10 +2483,26 @@ int open_table_from_share(THD *thd, TABL goto err; outparam->vfield= vfield_ptr; + } + + if (!share->default_fields) + outparam->default_field= NULL; You don't need to set default_fields to NULL as outparam is zeroed in beginning of the function. (Same is true for outparam->vfield) <cut> Regards, Monty

1 0

[Maria-developers] EXISTS-to-IN can work anywhere?
by Sergei Petrunia 26 Nov '12

26 Nov '12

Hello, Currently, EXISTS-to-IN rewrite only works on WHERE's top level, or under NOT, where the rewrite doesn't change semantics wrt NULL values. It seems, it is possible to make it work in other contexts, too. We need to convert it into a "NULL-ignoring IN", a special kind of IN predicate. For normal IN: NULL IN (select that returns something) --> NULL NULL IN (select that returns nothing) --> FALSE $value IN (select that doesn't return $value) --> FALSE $value IN (select that doesn't return $value but returns NULL) --> NULL null-ignoring-IN should return FALSE in all of the above cases. In fact, the optimizer is perfectly capable of doing this. What we need is just to disable special handling for NULLs. Let's discuss this tomorrow (Tuesday). BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB 5.1.66 release prep
by Sergei Golubchik 26 Nov '12

26 Nov '12

Preliminary release notes and changelog pages have been created for 5.1.66, 5.2.13, 5.3.11. https://kb.askmonty.org/en/mariadb-5166-release-notes/ https://kb.askmonty.org/en/mariadb-5166-changelog/ https://kb.askmonty.org/en/mariadb-5213-release-notes/ https://kb.askmonty.org/en/mariadb-5213-changelog/ https://kb.askmonty.org/en/mariadb-5311-release-notes/ https://kb.askmonty.org/en/mariadb-5311-changelog/ Please let me know if there is anything in particular which should be mentioned in the release notes. Thanks, Sergei

1 0

[Maria-developers] EXISTS-to-IN is ok to push
by Sergei Petrunia 23 Nov '12

23 Nov '12

Hi Sanja, I've made some fixes to comments in lp:~maria-captains/maria/10.0-exists2in. With that, it is ok to push EXISTS-to-IN into 10.0. Sorry for the delay. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] crash in TC_LOG_MMAP::log_one_transaction in maria-10.0.0
by Rich Prohaska 19 Nov '12

19 Nov '12

Hello, I am having problems with the shinny new mariadb-10.0.0. I have a simple test case that replaces a row in innodb and tokudb and then commits. Eventually, mysqld hits this assert: mysqld: /home/prohaska/maria10-build/mariadb-10.0.0/sql/log.cc:7547: int TC_LOG_MMAP::log_one_transaction(my_xid): Assertion `p->ptr < p->end' failed. The test works great when the binlog is enabled. The test crashes when the binlog is OFF. So, maybe the problem is isolated to the TC_LOG_MMAP logic. I noticed that the TC_LOG_MMAP::unlog function is quite different than the mariadb-5.5 version. What is the best way to get this fixed? Would it help if i made the test case available? Thanks Rich Prohaska Here is the stack trace: Program received signal SIGABRT, Aborted. 0x0000003006a32885 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003006a32885 in raise () from /lib64/libc.so.6 #1 0x0000003006a34065 in abort () from /lib64/libc.so.6 #2 0x0000003006a2b9fe in __assert_fail_base () from /lib64/libc.so.6 #3 0x0000003006a2bac0 in __assert_fail () from /lib64/libc.so.6 #4 0x00000000008862bf in TC_LOG_MMAP::log_one_transaction (this=0x1541f00 <tc_log_mmap>, xid=142018) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/log.cc:7547 #5 0x0000000000885442 in TC_LOG_MMAP::log_and_order (this=0x1541f00 <tc_log_mmap>, thd=0x27f0000, xid=142018, all=true, need_prepare_ordered=false, need_commit_ordered=true) at /home/prohaska/maria10-\ build/mariadb-10.0.0/sql/log.cc:7180 #6 0x00000000007b819f in ha_commit_trans (thd=0x27f0000, all=true) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/handler.cc:1329 #7 0x0000000000710307 in trans_commit (thd=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/transaction.cc:213 #8 0x000000000060a2df in mysql_execute_command (thd=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:4084 #9 0x000000000060f1a7 in mysql_parse (thd=0x27f0000, rawbuf=0x7fff6c0050b8 "commit", length=6, parser_state=0x7fffe4112660) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:6056 #10 0x00000000006029a3 in dispatch_command (command=COM_QUERY, thd=0x27f0000, packet=0x27fa341 "commit", packet_length=6) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:1216 #11 0x0000000000601bdb in do_command (thd=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:945 #12 0x00000000006ff6d1 in do_handle_one_connection (thd_arg=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_connect.cc:1254 #13 0x00000000006ff179 in handle_one_connection (arg=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_connect.cc:1168 #14 0x0000003006e077f1 in start_thread () from /lib64/libpthread.so.0 #15 0x0000003006ae592d in clone () from /lib64/libc.so.6

3 9

Re: [Maria-developers] Fwd: [Commits] Rev 3479: MDEV-3858 Change JOIN_TAB::records_read from ha_rows to double in file:///home/tsk/mprog/src/10.0-md3858/
by Sergei Petrunia 15 Nov '12

15 Nov '12

Hi Timour, Ok to push. On Thu, Nov 15, 2012 at 12:56:59PM +0200, Timour Katchaounov wrote: > Sergey, > > Could you please review this trivial task, shouldn't take more than 10 min. > > Timour > > ------------------------------------------------------------ > revno: 3479 > revision-id: timour(a)askmonty.org-20121115105450-zkiva73utg8pg49j > parent: wlad(a)montyprogram.com-20121111095852-1jsqvi3zcdml80lt > fixes bug: https://mariadb.atlassian.net/browse/MDEV-3858 > committer: timour(a)askmonty.org > branch nick: 10.0-md3858 > timestamp: Thu 2012-11-15 12:54:50 +0200 > message: > MDEV-3858 Change JOIN_TAB::records_read from ha_rows to double > > Currently JOIN_TAB::records_read is of type ha_rows. > This is an integer type, which prevents proper selectivity and rows estimates. > > > === modified file 'mysql-test/r/subselect_sj_mat.result' > --- a/mysql-test/r/subselect_sj_mat.result 2012-07-25 16:53:49 +0000 > +++ b/mysql-test/r/subselect_sj_mat.result 2012-11-15 10:54:50 +0000 > @@ -94,7 +94,7 @@ a1 a2 > explain extended > select * from t1i where a1 in (select b1 from t2i where b1 > '0'); > id select_type table type possible_keys key key_len ref rows filtered Extra > -1 PRIMARY t2i index it2i1,it2i3 it2i1 # NULL 5 40.00 Using where; Using index; LooseScan > +1 PRIMARY t2i index it2i1,it2i3 it2i1 # NULL 5 50.00 Using where; Using index; LooseScan > 1 PRIMARY t1i ref _it1_idx _it1_idx # _ref_ 1 100.00 > Warnings: > Note 1003 select `test`.`t1i`.`a1` AS `a1`,`test`.`t1i`.`a2` AS `a2` from `test`.`t1i` semi join (`test`.`t2i`) where ((`test`.`t1i`.`a1` = `test`.`t2i`.`b1`) and (`test`.`t2i`.`b1` > '0')) > @@ -117,7 +117,7 @@ a1 a2 > explain extended > select * from t1i where (a1, a2) in (select b1, b2 from t2i where b1 > '0'); > id select_type table type possible_keys key key_len ref rows filtered Extra > -1 PRIMARY t2i index it2i1,it2i2,it2i3 it2i3 # NULL 5 40.00 Using where; Using index; LooseScan > +1 PRIMARY t2i index it2i1,it2i2,it2i3 it2i3 # NULL 5 50.00 Using where; Using index; LooseScan > 1 PRIMARY t1i ref _it1_idx _it1_idx # _ref_ 1 100.00 > Warnings: > Note 1003 select `test`.`t1i`.`a1` AS `a1`,`test`.`t1i`.`a2` AS `a2` from `test`.`t1i` semi join (`test`.`t2i`) where ((`test`.`t1i`.`a2` = `test`.`t2i`.`b2`) and (`test`.`t1i`.`a1` = `test`.`t2i`.`b1`) and (`test`.`t2i`.`b1` > '0')) > @@ -319,7 +319,7 @@ where (a1, a2) in (select b1, b2 from t2 > (a1, a2) in (select c1, c2 from t3i > where (c1, c2) in (select b1, b2 from t2i where b2 > '0')); > id select_type table type possible_keys key key_len ref rows filtered Extra > -1 PRIMARY t2i index it2i1,it2i2,it2i3 # # # 5 40.00 # > +1 PRIMARY t2i index it2i1,it2i2,it2i3 # # # 5 50.00 # > 1 PRIMARY t1i ref it1i1,it1i2,it1i3 # # # 1 100.00 # > 1 PRIMARY t3i ref it3i1,it3i2,it3i3 # # # 1 100.00 # > 1 PRIMARY t2i ref it2i1,it2i2,it2i3 # # # 2 100.00 # > @@ -407,7 +407,7 @@ id select_type table type possible_keys > 2 MATERIALIZED t2 ALL NULL # # # 5 100.00 # > 4 MATERIALIZED t3 ALL NULL # # # 4 100.00 # > 3 MATERIALIZED t3 ALL NULL # # # 4 100.00 # > -7 UNION t2i index it2i1,it2i2,it2i3 # # # 5 40.00 # > +7 UNION t2i index it2i1,it2i2,it2i3 # # # 5 50.00 # > 7 UNION t1i ref it1i1,it1i2,it1i3 # # # 1 100.00 # > 7 UNION t3i ref it3i1,it3i2,it3i3 # # # 1 100.00 # > 7 UNION t2i ref it2i1,it2i2,it2i3 # # # 2 100.00 # > > === modified file 'sql/item_subselect.cc' > --- a/sql/item_subselect.cc 2012-11-03 11:28:51 +0000 > +++ b/sql/item_subselect.cc 2012-11-15 10:54:50 +0000 > @@ -4553,7 +4553,7 @@ double get_fanout_with_deps(JOIN *join, > !tab->emb_sj_nest && > tab->records_read != 0) > { > - fanout *= rows2double(tab->records_read); > + fanout *= tab->records_read; > } > } > return fanout; > > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2012-11-04 15:32:32 +0000 > +++ b/sql/sql_select.cc 2012-11-15 10:54:50 +0000 > @@ -6789,7 +6789,7 @@ void JOIN::get_prefix_cost_and_fanout(ui > > double JOIN::get_examined_rows() > { > - ha_rows examined_rows; > + double examined_rows; > double prev_fanout= 1; > JOIN_TAB *tab= first_breadth_first_tab(this, WALK_OPTIMIZATION_TABS); > JOIN_TAB *prev_tab= tab; > @@ -6799,7 +6799,7 @@ double JOIN::get_examined_rows() > while ((tab= next_breadth_first_tab(this, WALK_OPTIMIZATION_TABS, tab))) > { > prev_fanout *= prev_tab->records_read; > - examined_rows+= (ha_rows) (tab->get_examined_rows() * prev_fanout); > + examined_rows+= tab->get_examined_rows() * prev_fanout; > prev_tab= tab; > } > return examined_rows; > @@ -7704,7 +7704,8 @@ get_best_combination(JOIN *join) > sub-order > */ > SJ_MATERIALIZATION_INFO *sjm= cur_pos->table->emb_sj_nest->sj_mat_info; > - j->records= j->records_read= (ha_rows)(sjm->is_sj_scan? sjm->rows : 1); > + j->records_read= sjm->is_sj_scan? sjm->rows : 1; > + j->records= (ha_rows) j->records_read; > JOIN_TAB *jt; > JOIN_TAB_RANGE *jt_range; > if (!(jt= (JOIN_TAB*)join->thd->alloc(sizeof(JOIN_TAB)*sjm->tables)) || > @@ -7766,7 +7767,7 @@ get_best_combination(JOIN *join) > Save records_read in JOIN_TAB so that select_describe()/etc don't have > to access join->best_positions[]. > */ > - j->records_read= (ha_rows)join->best_positions[tablenr].records_read; > + j->records_read= join->best_positions[tablenr].records_read; > join->map2table[j->table->tablenr]= j; > > /* If we've reached the end of sjm nest, switch back to main sequence */ > @@ -10460,7 +10461,7 @@ double JOIN_TAB::scan_time() > > ha_rows JOIN_TAB::get_examined_rows() > { > - ha_rows examined_rows; > + double examined_rows; > > if (select && select->quick && use_quick != 2) > examined_rows= select->quick->records; > @@ -10490,7 +10491,7 @@ ha_rows JOIN_TAB::get_examined_rows() > } > } > else > - examined_rows= (ha_rows) records_read; > + examined_rows= records_read; > > return examined_rows; > } > @@ -21972,7 +21973,7 @@ int JOIN::print_explain(select_result_si > } > else > { > - ha_rows examined_rows= tab->get_examined_rows(); > + double examined_rows= tab->get_examined_rows(); > > item_list.push_back(new Item_int((longlong) (ulonglong) examined_rows, > MY_INT64_NUM_DECIMAL_DIGITS)); > @@ -21982,7 +21983,7 @@ int JOIN::print_explain(select_result_si > { > float f= 0.0; > if (examined_rows) > - f= (float) (100.0 * tab->records_read / examined_rows); > + f= (100.0 * (float)tab->records_read) / examined_rows; > set_if_smaller(f, 100.0); > item_list.push_back(new Item_float(f, 2)); > } > > === modified file 'sql/sql_select.h' > --- a/sql/sql_select.h 2012-11-04 15:32:32 +0000 > +++ b/sql/sql_select.h 2012-11-15 10:54:50 +0000 > @@ -280,8 +280,8 @@ typedef struct st_join_table { > */ > double read_time; > > - /* psergey-todo: make the below have type double, like POSITION::records_read? */ > - ha_rows records_read; > + /* Copy of POSITION::records_read, set by get_best_combination() */ > + double records_read; > > /* Startup cost for execution */ > double startup_cost; > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] The patch for the task mdev-539
by Sergei Golubchik 13 Nov '12

13 Nov '12

Hi, Igor! Sorry for a delay :( Please, find my review attached. On Sep 27, Igor Babaev wrote: > === modified file 'mysql-test/r/alter_table.result' > --- a/mysql-test/r/alter_table.result 2012-05-21 13:30:25 +0000 > +++ b/mysql-test/r/alter_table.result 2012-09-27 21:29:26 +0000 > @@ -1340,3 +1340,203 @@ > execute stmt1; > deallocate prepare stmt1; > drop table t2; > +# > +# mdev-539: fast build of unique/primary indexes for MyISAM > +# I'd suggest to have these test cases in a separate file, say, alter_table_539.test But it's just a siggestion, do as you prefer. > +DROP DATABASE IF EXISTS dbt3_s001; > +CREATE DATABASE dbt3_s001; > +use dbt3_s001; > +drop index `primary` on customer; > +show create table customer; > +Table Create Table > +customer CREATE TABLE `customer` ( > + `c_custkey` int(11) NOT NULL, > + `c_name` varchar(25) DEFAULT NULL, > + `c_address` varchar(40) DEFAULT NULL, > + `c_nationkey` int(11) DEFAULT NULL, > + `c_phone` char(15) DEFAULT NULL, > + `c_acctbal` double DEFAULT NULL, > + `c_mktsegment` char(10) DEFAULT NULL, > + `c_comment` varchar(117) DEFAULT NULL, > + KEY `i_c_nationkey` (`c_nationkey`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +alter table customer add primary key (c_custkey); > +show create table customer; > +Table Create Table > +customer CREATE TABLE `customer` ( > + `c_custkey` int(11) NOT NULL, > + `c_name` varchar(25) DEFAULT NULL, > + `c_address` varchar(40) DEFAULT NULL, > + `c_nationkey` int(11) DEFAULT NULL, > + `c_phone` char(15) DEFAULT NULL, > + `c_acctbal` double DEFAULT NULL, > + `c_mktsegment` char(10) DEFAULT NULL, > + `c_comment` varchar(117) DEFAULT NULL, > + PRIMARY KEY (`c_custkey`), > + KEY `i_c_nationkey` (`c_nationkey`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +drop index `primary` on customer; > +insert into customer values > +(3,'Customer#00000000303','MG9kdTD2WBHm',1,'11-719-748-3364',7498.12,'AUTOMOBILE','special packages wake. slyly reg'); > +alter table customer add primary key (c_custkey); please add a test case for alter ignore table > +ERROR 23000: Duplicate entry '3' for key 'PRIMARY' > +show create table customer; > +Table Create Table > +customer CREATE TABLE `customer` ( ... > === modified file 'sql/ha_partition.cc' > --- a/sql/ha_partition.cc 2012-08-31 21:54:54 +0000 > +++ b/sql/ha_partition.cc 2012-09-27 21:29:26 +0000 > @@ -6862,6 +6862,8 @@ > > flags_to_return= ht->alter_table_flags(flags); > flags_to_return|= m_file[0]->alter_table_flags(flags); > + flags_to_return&= > + ~(HA_FAST_ADD_INDEX | HA_FAST_ADD_UNIQUE_INDEX | HA_FAST_ADD_PK_INDEX); You don't need these flags - see below > > /* > If one partition fails we must be able to revert the change for the other, > > === modified file 'sql/sql_table.cc' > --- a/sql/sql_table.cc 2012-09-02 02:41:38 +0000 > +++ b/sql/sql_table.cc 2012-09-27 21:29:26 +0000 > @@ -6743,6 +6745,11 @@ > my_error(ER_LOCK_WAIT_TIMEOUT, MYF(0)); > goto err_new_table_cleanup; > }); > + > + if (!ignore && > + test(alter_flags & (HA_FAST_ADD_UNIQUE_INDEX | HA_FAST_ADD_PK_INDEX))) > + new_table->file->extra(HA_EXTRA_CREATE_UNIQUE_INDEX_BY_SORT); What's the point? extra() functions were always *optional* in the storage engine API, any storage engine is allowed to ignore any HA_EXTRA_* requests, or all of them. It means, you don't need to introduce HA_FAST_whatever flags, and don't need to check them. Just write if (!ignore) new_table->file->extra(HA_EXTRA_CREATE_UNIQUE_INDEX_BY_SORT); and let the engine ignore it, if it wishes to do so. as a small bonus, you won't need to move alter_flags variable around, it can stay where it was, inside the if() block. > + > error= copy_data_between_tables(thd, table, new_table, > alter_info->create_list, ignore, > order_num, order, &copied, &deleted, > > === modified file 'storage/maria/ha_maria.cc' > --- a/storage/maria/ha_maria.cc 2012-08-14 16:59:28 +0000 > +++ b/storage/maria/ha_maria.cc 2012-09-27 21:29:26 +0000 > @@ -1633,6 +1633,9 @@ please, use the recipe from https://kb.askmonty.org/en/how-to-get-more-out-of-bzr-when-working-on-maria… it's much more difficult to review patches without sufficient context (that is, without function names on every chunk) > param->testflag|= T_REP_BY_SORT; > error= maria_repair_by_sort(param, file, fixed_name, > test(param->testflag & T_QUICK)); > + if (error && share->state.dupp_key != MARIA_MAX_KEY) > + print_keydup_error(share->state.dupp_key, > + ER(ER_DUP_ENTRY_WITH_KEY_NAME)); > } > } > else > > === modified file 'storage/myisam/ha_myisam.cc' > --- a/storage/myisam/ha_myisam.cc 2012-07-18 18:40:15 +0000 > +++ b/storage/myisam/ha_myisam.cc 2012-09-27 21:29:26 +0000 > @@ -1130,6 +1130,9 @@ > thd_proc_info(thd, "Repair by sorting"); > error = mi_repair_by_sort(&param, file, fixed_name, > test(param.testflag & T_QUICK)); > + if (error && share->state.dupp_key != MAX_KEY) > + print_keydup_error(share->state.dupp_key, > + ER(ER_DUP_ENTRY_WITH_KEY_NAME)); I don't understand it. print_keydup_error() takes the value to print from table->record[[0]. But - in the case of mi_repair_by_sort - how does the value end up here? What code puts the conflicting key value into table->record[0] ? > } > } > else > > === modified file 'storage/myisam/mi_check.c' > --- a/storage/myisam/mi_check.c 2012-04-07 13:58:46 +0000 > +++ b/storage/myisam/mi_check.c 2012-09-27 21:29:26 +0000 > @@ -2408,12 +2411,22 @@ > (my_bool) (!(param->testflag & T_VERBOSE)), > param->sort_buffer_length)) > { > + > + if (sort_param.sort_info->dupp) > + { > + share->state.dupp_key= sort_param.key; > + keydup_error= 1; > + } > + if (!(keydup_error && param->suppress_keydup_error_handling)) 1. please use param->testflag for flags, instead of a new special field 2. I don't see why are you doing it here, normally the error is issued below, as "Duplicate key for record at %10s against...." I'd expect to see any new code that deals with duplicate key errors at the same place where the old code is > + { > param->retry_repair= 1; > if (! param->error_printed) > mi_check_print_error(param, "Couldn't fix table with create_index_by_sort(). Error: %d", > my_errno); > + } > goto err; > } > + > /* No need to calculate checksum again. */ > sort_param.calc_checksum= 0; > free_root(&sort_param.wordroot, MYF(0)); > @@ -2533,18 +2546,27 @@ > } > if (got_error) > { > + if (!(keydup_error && param->suppress_keydup_error_handling)) > + { > if (! param->error_printed) > mi_check_print_error(param,"%d when fixing table",my_errno); You could simply set param->error_printed when you get dplicate key error - it'll avoid all further errors, you won't need to wrap them all in if()'s > + } > if (new_file >= 0) > { > (void) mysql_file_close(new_file, MYF(0)); > (void) mysql_file_delete(mi_key_file_datatmp, > param->temp_filename, MYF(MY_WME)); > + if (!(keydup_error && param->suppress_keydup_error_handling)) > + { Why do you need this if() ? > if (info->dfile == new_file) /* Retry with key cache */ > if (unlikely(mi_open_datafile(info, share, name, -1))) > param->retry_repair= 0; /* Safety */ > + } > } > + if (!(keydup_error && param->suppress_keydup_error_handling)) > + { > mi_mark_crashed_on_repair(info); > + } Why do you need to avoid mi_mark_crashed_on_repair() ? > if (killed_ptr(param)) > param->retry_repair= 0; /* No use to retry repair */ > } > @@ -4682,8 +4708,9 @@ > (!rows || rows >= MI_MIN_ROWS_TO_DISABLE_INDEXES)); > for (i=0 ; i < share->base.keys ; i++,key++) > { > - if (!(key->flag & (HA_NOSAME | HA_SPATIAL | HA_AUTO_KEY)) && > - ! mi_too_big_key_for_sort(key,rows) && info->s->base.auto_key != i+1) > + if (!(key->flag & (HA_SPATIAL | HA_AUTO_KEY)) && > + ! mi_too_big_key_for_sort(key,rows) && info->s->base.auto_key != i+1 && > + (!(key->flag & HA_NOSAME) || info->create_unique_index_by_sort)) Now, this directly contradicts the function name and the function comment. Could you please rename the function and update the comment? And instead of checking info->create_unique_index_by_sort I'd rather pass it down as an argument: void mi_disable_indexes(MI_INFO *info, ha_rows rows, my_bool only_non_unique) { > { > mi_clear_key_active(share->state.key_map, i); > info->update|= HA_STATE_CHANGED; > > === modified file 'storage/myisam/mi_extra.c' > --- a/storage/myisam/mi_extra.c 2012-07-31 17:29:07 +0000 > +++ b/storage/myisam/mi_extra.c 2012-09-27 21:29:26 +0000 > @@ -365,6 +365,9 @@ > case HA_EXTRA_CHANGE_KEY_TO_DUP: > mi_extra_keyflag(info, function); > break; > + case HA_EXTRA_CREATE_UNIQUE_INDEX_BY_SORT: > + info->create_unique_index_by_sort= 1; > + break; Two comments here: 1. I don't like the name "create unique index by sort", because ALTER TABLE itself does not create any indexes by sort or anything it simply tells the engine to optimize a bulk insertion. and it's myisam that does repair-by-sort. I mean, a better name would be a high-level name that does not imply myisam-specific details. something like HA_EXTRA_BULK_INSERT_DUPLICATES_UNLIKELY which tells the engine to optimize the following bulk insert for the case when unique key conflicts are unlikely to happen. (in myisam case it means to build unique indexes by sort, right) 2. I've discussed this with Monty and we both didn't like using extra() here, even to preserve the API compatibility. Monty thinks there's no need to bother, and we can change start_bulk_insert() any way we want. But to make it better for third-party engines, I'd suggest something like class handler { ... virtual void start_bulk_insert(ha_rows rows) {} virtual void start_bulk_insert(ha_rows rows, ulonglong flags) { start_bulk_insert(rows); } ... } that is, you introduce the new function, and keep the old one as a fallback. This way it stays compatible, and old engines won't need any changes. But myisam/maria will implement the new start_bulk_insert variant, with two arguments, and will recognize the flag HA_BULK_INSERT_DUPLICATES_UNLIKELY And no extra() is needed. > case HA_EXTRA_MMAP: > #ifdef HAVE_MMAP > mysql_mutex_lock(&share->intern_lock); > Regards, Sergei

1 0

[Maria-developers] MariaDB 5.3.10 release prep
by Daniel Bartholomew 12 Nov '12

12 Nov '12

Preliminary release notes and changelog pages have been created for 5.3.10. https://kb.askmonty.org/en/mariadb-5310-release-notes/ https://kb.askmonty.org/en/mariadb-5310-changelog/ Please let me know if there is anything in particular which should be mentioned in the release notes. Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

[Maria-developers] MariaDB 10.0.0 Alpha Release Prep
by Daniel Bartholomew 12 Nov '12

12 Nov '12

All, The files for this release have been uploaded to the mirrors and we're just waiting until Monday morning, U.S. Eastern time, to announce the release. The preliminary release notes and changelog pages are here: - https://kb.askmonty.org/en/mariadb-1000-release-notes/ - https://kb.askmonty.org/en/mariadb-1000-changelog/ The release notes page is pretty sparse. Its content is from the list of "New already done Features" on the "What is MariaDB 10.0?" page. Please send me additions and corrections ASAP (or make the changes yourself). Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

Re: [Maria-developers] [Commits] Rev 3569: MDEV-3830 - fix build on Intel compiler in file:///H:/bzr/5.5/
by Kristian Nielsen 04 Nov '12

04 Nov '12

Vladislav Vaintroub <wlad(a)montyprogram.com> writes: > === modified file 'mysys/my_context.c' > --- a/mysys/my_context.c 2012-10-31 11:47:25 +0000 > +++ b/mysys/my_context.c 2012-11-02 23:31:50 +0000 > @@ -206,7 +206,7 @@ > ( > "movq %%rsp, (%[save])\n\t" > "movq %[stack], %%rsp\n\t" > -#if __GNUC__ >= 4 && __GNUC_MINOR__ >= 4 > +#if __GNUC__ >= 4 && __GNUC_MINOR__ >= 4 && !defined(__INTEL_COMPILER) There are two places this is done (one for 32-bit and one for 64-bit). You probably need the !defined(__INTEL_COMPILER) on both. - Kristian.

1 0

Re: [Maria-developers] [Commits] Rev 3589: Fixed bug mdev-585 (LP bug #637962) in file:///home/igor/maria/maria-5.3-mdev585/
by Sergei Petrunia 02 Nov '12

02 Nov '12

Hello Igor, The fix seems to be ok to push. Which version do you want to push it into? This is not a crash, should we really push it into 5.3, or to 5.5? On Thu, Nov 01, 2012 at 02:54:34PM -0700, Igor Babaev wrote: > At file:///home/igor/maria/maria-5.3-mdev585/ > > ------------------------------------------------------------ > revno: 3589 > revision-id: igor(a)askmonty.org-20121101215433-ctfm3zfhc6uhuy3i > parent: psergey(a)askmonty.org-20121031053425-kfn0l7yj6z7jtnqw > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-mdev585 > timestamp: Thu 2012-11-01 14:54:33 -0700 > message: > Fixed bug mdev-585 (LP bug #637962) > If, when executing a query with ORDER BY col LIMIT n, the optimizer chose > an index-merge scan to access the table containing col while there existed > an index defined over col then optimizer did not consider the possibility > of using an alternative range scan by this index to avoid filesort. This > could cause a performance degradation if the optimizer flag index_merge was > set up to 'on'. > === modified file 'mysql-test/r/range_vs_index_merge.result' > --- a/mysql-test/r/range_vs_index_merge.result 2012-03-21 00:03:28 +0000 > +++ b/mysql-test/r/range_vs_index_merge.result 2012-11-01 21:54:33 +0000 > @@ -1221,6 +1221,150 @@ > Seattle USA 563374 > Caracas VEN 1975294 > set optimizer_switch=@save_optimizer_switch; > +# > +# Bug mdev-585: range vs index-merge with ORDER BY ... LIMIT n > +# (LP bug #637962) > +# > +DROP INDEX CountryPopulation ON City; > +DROP INDEX CountryName ON City; > +DROP INDEX CityName on City; > +CREATE INDEX Name ON City(Name); > +CREATE INDEX Population ON City(Population); > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City index_merge Country,Name,Population Name,Country 35,3 NULL # Using sort_union(Name,Country); Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000); > +ID Name Country Population > +384 Cabo Frio BRA 119503 > +387 Camaragibe BRA 118968 > +403 Catanduva BRA 107761 > +412 Cachoeirinha BRA 103240 > +508 Watford GBR 113080 > +509 Ipswich GBR 114000 > +510 Slough GBR 112000 > +511 Exeter GBR 111000 > +512 Cheltenham GBR 106000 > +513 Gloucester GBR 107000 > +514 Saint Helens GBR 106293 > +515 Sutton Coldfield GBR 106001 > +516 York GBR 104425 > +517 Oldham GBR 103931 > +518 Basildon GBR 100924 > +519 Worthing GBR 100000 > +635 Mallawi EGY 119283 > +636 Bilbays EGY 113608 > +637 Mit Ghamr EGY 101801 > +638 al-Arish EGY 100447 > +701 Tarragona ESP 113016 > +702 Lleida (Lérida) ESP 112207 > +703 Jaén ESP 109247 > +704 Ourense (Orense) ESP 109120 > +705 Mataró ESP 104095 > +706 Algeciras ESP 103106 > +707 Marbella ESP 101144 > +759 Gonder ETH 112249 > +869 Cabuyao PHL 106630 > +870 Calapan PHL 105910 > +873 Cauayan PHL 103952 > +903 Serekunda GMB 102600 > +909 Sohumi GEO 111700 > +913 Tema GHA 109975 > +914 Sekondi-Takoradi GHA 103653 > +924 Villa Nueva GTM 101295 > +1844 Cape Breton CAN 114733 > +1847 Cambridge CAN 109186 > +2406 Herakleion GRC 116178 > +2407 Kallithea GRC 114233 > +2408 Larisa GRC 113090 > +2908 Cajamarca PER 108009 > +3002 Besançon FRA 117733 > +3003 Caen FRA 113987 > +3004 Orléans FRA 113126 > +3005 Mulhouse FRA 110359 > +3006 Rouen FRA 106592 > +3007 Boulogne-Billancourt FRA 106367 > +3008 Perpignan FRA 105115 > +3009 Nancy FRA 103605 > +3411 Ceyhan TUR 102412 > +3567 Carúpano VEN 119639 > +3568 Catia La Mar VEN 117012 > +3571 Calabozo VEN 107146 > +3786 Cam Ranh VNM 114041 > +3792 Tartu EST 101246 > +4002 Carrollton USA 109576 > +4027 Cape Coral USA 102286 > +4032 Cambridge USA 101355 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 2 > +Handler_read_next 385 > +Handler_read_prev 0 > +Handler_read_rnd 377 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City range Country,Name,Population Population 4 NULL # Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +ID Name Country Population > +519 Worthing GBR 100000 > +638 al-Arish EGY 100447 > +518 Basildon GBR 100924 > +707 Marbella ESP 101144 > +3792 Tartu EST 101246 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 1 > +Handler_read_next 59 > +Handler_read_prev 0 > +Handler_read_rnd 0 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +set optimizer_switch='index_merge=off'; > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City range Country,Name,Population Population 4 NULL # Using index condition; Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +ID Name Country Population > +519 Worthing GBR 100000 > +638 al-Arish EGY 100447 > +518 Basildon GBR 100924 > +707 Marbella ESP 101144 > +3792 Tartu EST 101246 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 1 > +Handler_read_next 59 > +Handler_read_prev 0 > +Handler_read_rnd 0 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +set optimizer_switch=@save_optimizer_switch; > DROP DATABASE world; > use test; > CREATE TABLE t1 ( > > === modified file 'mysql-test/r/range_vs_index_merge_innodb.result' > --- a/mysql-test/r/range_vs_index_merge_innodb.result 2012-03-21 00:03:28 +0000 > +++ b/mysql-test/r/range_vs_index_merge_innodb.result 2012-11-01 21:54:33 +0000 > @@ -1222,6 +1222,150 @@ > Seattle USA 563374 > Caracas VEN 1975294 > set optimizer_switch=@save_optimizer_switch; > +# > +# Bug mdev-585: range vs index-merge with ORDER BY ... LIMIT n > +# (LP bug #637962) > +# > +DROP INDEX CountryPopulation ON City; > +DROP INDEX CountryName ON City; > +DROP INDEX CityName on City; > +CREATE INDEX Name ON City(Name); > +CREATE INDEX Population ON City(Population); > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City index_merge Country,Name,Population Name,Country 35,3 NULL # Using sort_union(Name,Country); Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000); > +ID Name Country Population > +384 Cabo Frio BRA 119503 > +387 Camaragibe BRA 118968 > +403 Catanduva BRA 107761 > +412 Cachoeirinha BRA 103240 > +508 Watford GBR 113080 > +509 Ipswich GBR 114000 > +510 Slough GBR 112000 > +511 Exeter GBR 111000 > +512 Cheltenham GBR 106000 > +513 Gloucester GBR 107000 > +514 Saint Helens GBR 106293 > +515 Sutton Coldfield GBR 106001 > +516 York GBR 104425 > +517 Oldham GBR 103931 > +518 Basildon GBR 100924 > +519 Worthing GBR 100000 > +635 Mallawi EGY 119283 > +636 Bilbays EGY 113608 > +637 Mit Ghamr EGY 101801 > +638 al-Arish EGY 100447 > +701 Tarragona ESP 113016 > +702 Lleida (Lérida) ESP 112207 > +703 Jaén ESP 109247 > +704 Ourense (Orense) ESP 109120 > +705 Mataró ESP 104095 > +706 Algeciras ESP 103106 > +707 Marbella ESP 101144 > +759 Gonder ETH 112249 > +869 Cabuyao PHL 106630 > +870 Calapan PHL 105910 > +873 Cauayan PHL 103952 > +903 Serekunda GMB 102600 > +909 Sohumi GEO 111700 > +913 Tema GHA 109975 > +914 Sekondi-Takoradi GHA 103653 > +924 Villa Nueva GTM 101295 > +1844 Cape Breton CAN 114733 > +1847 Cambridge CAN 109186 > +2406 Herakleion GRC 116178 > +2407 Kallithea GRC 114233 > +2408 Larisa GRC 113090 > +2908 Cajamarca PER 108009 > +3002 Besançon FRA 117733 > +3003 Caen FRA 113987 > +3004 Orléans FRA 113126 > +3005 Mulhouse FRA 110359 > +3006 Rouen FRA 106592 > +3007 Boulogne-Billancourt FRA 106367 > +3008 Perpignan FRA 105115 > +3009 Nancy FRA 103605 > +3411 Ceyhan TUR 102412 > +3567 Carúpano VEN 119639 > +3568 Catia La Mar VEN 117012 > +3571 Calabozo VEN 107146 > +3786 Cam Ranh VNM 114041 > +3792 Tartu EST 101246 > +4002 Carrollton USA 109576 > +4027 Cape Coral USA 102286 > +4032 Cambridge USA 101355 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 2 > +Handler_read_next 385 > +Handler_read_prev 0 > +Handler_read_rnd 377 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City range Country,Name,Population Population 4 NULL # Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +ID Name Country Population > +519 Worthing GBR 100000 > +638 al-Arish EGY 100447 > +518 Basildon GBR 100924 > +707 Marbella ESP 101144 > +3792 Tartu EST 101246 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 1 > +Handler_read_next 59 > +Handler_read_prev 0 > +Handler_read_rnd 0 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +set optimizer_switch='index_merge=off'; > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City range Country,Name,Population Population 4 NULL # Using index condition; Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +ID Name Country Population > +519 Worthing GBR 100000 > +638 al-Arish EGY 100447 > +518 Basildon GBR 100924 > +707 Marbella ESP 101144 > +3792 Tartu EST 101246 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 1 > +Handler_read_next 59 > +Handler_read_prev 0 > +Handler_read_rnd 0 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +set optimizer_switch=@save_optimizer_switch; > DROP DATABASE world; > use test; > CREATE TABLE t1 ( > > === modified file 'mysql-test/t/range_vs_index_merge.test' > --- a/mysql-test/t/range_vs_index_merge.test 2012-03-21 00:03:28 +0000 > +++ b/mysql-test/t/range_vs_index_merge.test 2012-11-01 21:54:33 +0000 > @@ -675,6 +675,64 @@ > $cond; > > set optimizer_switch=@save_optimizer_switch; > + > +--echo # > +--echo # Bug mdev-585: range vs index-merge with ORDER BY ... LIMIT n > +--echo # (LP bug #637962) > +--echo # > + > +DROP INDEX CountryPopulation ON City; > +DROP INDEX CountryName ON City; > +DROP INDEX CityName on City; > + > +CREATE INDEX Name ON City(Name); > +CREATE INDEX Population ON City(Population); > + > + > +--replace_column 9 # > +EXPLAIN > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000); > +FLUSH STATUS; > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000); > +SHOW STATUS LIKE 'Handler_read_%'; > + > + > +--replace_column 9 # > +EXPLAIN > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > + > +FLUSH STATUS; > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +SHOW STATUS LIKE 'Handler_read_%'; > + > + > +set optimizer_switch='index_merge=off'; > + > +--replace_column 9 # > +EXPLAIN > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > + > +FLUSH STATUS; > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +SHOW STATUS LIKE 'Handler_read_%'; > + > +set optimizer_switch=@save_optimizer_switch; > > > DROP DATABASE world; > > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2012-10-10 05:21:22 +0000 > +++ b/sql/sql_select.cc 2012-11-01 21:54:33 +0000 > @@ -18079,15 +18079,18 @@ > */ > > if (quick_type == QUICK_SELECT_I::QS_TYPE_INDEX_MERGE || > - quick_type == QUICK_SELECT_I::QS_TYPE_INDEX_INTERSECT || > + quick_type == QUICK_SELECT_I::QS_TYPE_INDEX_INTERSECT || > quick_type == QUICK_SELECT_I::QS_TYPE_ROR_UNION || > quick_type == QUICK_SELECT_I::QS_TYPE_ROR_INTERSECT) > - goto use_filesort; > - ref_key= select->quick->index; > - ref_key_parts= select->quick->used_key_parts; > + ref_key= MAX_KEY; > + else > + { > + ref_key= select->quick->index; > + ref_key_parts= select->quick->used_key_parts; > + } > } > > - if (ref_key >= 0) > + if (ref_key >= 0 && ref_key != MAX_KEY) > { > /* > We come here when there is a REF key. > @@ -18229,7 +18232,8 @@ > else > keys= usable_keys; > > - if (ref_key >= 0 && table->covering_keys.is_set(ref_key)) > + if (ref_key >= 0 && ref_key != MAX_KEY && > + table->covering_keys.is_set(ref_key)) > ref_key_quick_rows= table->quick_rows[ref_key]; > > read_time= join->best_positions[tablenr].read_time; > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] Rev 3176: MDEV-616 fix (MySQL fix accepted)
by Sergei Golubchik 31 Oct '12

31 Oct '12

Hi, Sanja! On Oct 09, sanja(a)montyprogram.com wrote: > revno: 3176 > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.2-MDEV-616 > timestamp: Tue 2012-10-09 17:36:02 +0300 > message: > MDEV-616 fix (MySQL fix accepted) Looks ok, as far as I understand. Regards, Sergei

1 0

[Maria-developers] Fwd: possible bug in MySQL 5.6.7-rc slave replication crash safety?
by Zardosht Kasheff 24 Oct '12

24 Oct '12

Hello, Does Maria 10.0 implement crash safe replication that is found in MySQL 5.6.7? If so, does this issue exist there? Thanks -Zardosht ---------- Forwarded message ---------- From: Zardosht Kasheff <zardosht(a)gmail.com> Date: Wed, Oct 17, 2012 at 8:59 AM Subject: possible bug in MySQL 5.6.7-rc slave replication crash safety? To: internals(a)lists.mysql.com Hello all, We downloaded MySQL 5.6.7-rc to see slave replication crash safety in action. A simple test did not seem to work, as described in http://bugs.mysql.com/bug.php?id=67246. Does anyone have insight into the problem? Perhaps we screwed something up? Thanks -Zardosht

2 8

Re: [Maria-developers] Please look at and advise about MDEV-3801
by Igor Babaev 22 Oct '12

22 Oct '12

On 10/22/2012 12:53 AM, Timour Katchaounov wrote: > Igor, > > Great, the patch indeed fixes the bug both on 5.3 and 5.5, however the > reduced test case I produced still takes 21 MB, and I was not successful > in reducing it to a size that can be added to the regression test suite. > I already spent too much in reducing the test case. Would you agree that > I push your patch without a test case? I think you still have to add the test case case (in a separate file) that is run with the --big option. Regards, Igor. > > Timour > > > On 20.10.2012 08:56, Igor Babaev wrote: >> On 10/19/2012 12:43 PM, Timour Katchaounov wrote: >>> Hi Igor, >>> >>> Could you please look at this bug >>> https://mariadb.atlassian.net/browse/MDEV-3801, >>> and read my last comment. The bug may be (just a guess ATM) related to >>> index >>> creation on the temporary table created for the derived table. Since >>> this is your >>> code, I'd like you to tell me if you have seen anything similar before, >>> or if >>> you have any hints. >>> >>> If not, I will continue with the bug and will trace the cause for the >>> wrong key information. >>> >>> Thanks, >>> Timour >> >> Timour, >> >> The following patch fixes this problem >> >> Regards, >> Igor. >> >> === modified file 'sql/sql_select.cc' >> --- sql/sql_select.cc 2012-10-10 05:21:22 +0000 >> +++ sql/sql_select.cc 2012-10-20 05:49:13 +0000 >> @@ -8981,7 +8981,7 @@ void JOIN::drop_unused_derived_keys() >> JOIN_TAB *tab; >> for (tab= first_linear_tab(this, WITHOUT_CONST_TABLES); >> tab; >> - tab= next_linear_tab(this, tab, WITHOUT_BUSH_ROOTS)) >> + tab= next_linear_tab(this, tab, WITH_BUSH_ROOTS)) >> { >> >> TABLE *table=tab->table; >> >> MariaDB [test]> set @@tmp_table_size=16384; >> ERROR 2006 (HY000): MySQL server has gone away >> No connection. Trying to reconnect... >> Connection id: 1 >> Current database: test >> >> Query OK, 0 rows affected (5.48 sec) >> >> MariaDB [test]> set @@tmp_table_size=16384; >> Query OK, 0 rows affected (0.00 sec) >> >> MariaDB [test]> EXPLAIN SELECT t1.deal_id FROM table_e AS t1 JOIN >> table_d AS t10 ON (t1.deal_id = t10.deal_id) WHERE t1.deal_id IN >> (SELECT * FROM (SELECT DISTINCT t1.deal_id FROM table_e AS >> t1 JOIN table_d AS t2 ON (t1.deal_id = t2.deal_id) >> JOIN table_c_s AS t3 ON (t2.deal_id = t3.deal_id) AND (t2.feid = >> t3.feid) JOIN table_f AS t4 ON t3.exchange = t4.exchange) >> tU) limit 10; >> +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ >> >> | id | select_type | table | type | possible_keys >> | key | key_len | ref | >> rows | Extra | >> +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ >> >> | 1 | PRIMARY | <subquery2> | ALL | distinct_key >> | NULL | NULL | NULL | >> 8488 | | >> | 1 | PRIMARY | t1 | eq_ref | PRIMARY >> | PRIMARY | 4 | tU.deal_id | >> 1 | Using index | >> | 1 | PRIMARY | t10 | ref | PRIMARY,full_id >> | PRIMARY | 4 | tU.deal_id | >> 1 | Using index | >> | 2 | MATERIALIZED | <derived3> | ALL | NULL >> | NULL | NULL | NULL | >> 8488 | | >> | 3 | DERIVED | t4 | system | PRIMARY >> | NULL | NULL | NULL | >> 1 | Using temporary | >> | 3 | DERIVED | t3 | ref | >> PRIMARY,feid,exchange,deal_feid,feid_date | exchange | 3 | const >> | 8488 | | >> | 3 | DERIVED | t1 | eq_ref | PRIMARY >> | PRIMARY | 4 | test.t3.deal_id | >> 1 | Using index | >> | 3 | DERIVED | t2 | eq_ref | PRIMARY,full_id >> | PRIMARY | 8 | test.t3.deal_id,test.t3.feid | >> 1 | Using index; Distinct | >> +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ >> >> 8 rows in set (4.22 sec) >> >> MariaDB [test]> SELECT t1.deal_id FROM table_e AS t1 JOIN table_d AS t10 >> ON (t1.deal_id = t10.deal_id) WHERE t1.deal_id IN (SELECT * FROM >> (SELECT DISTINCT t1.deal_id FROM table_e AS t1 >> JOIN table_d AS t2 ON (t1.deal_id = t2.deal_id) JOIN >> table_c_s AS t3 ON (t2.deal_id = t3.deal_id) AND (t2.feid = t3.feid) >> JOIN table_f AS t4 ON t3.exchange = t4.exchange) tU) limit >> 10; >> +---------+ >> | deal_id | >> +---------+ >> | 30 | >> | 82 | >> | 83 | >> | 108 | >> | 155 | >> | 156 | >> | 162 | >> | 184 | >> | 186 | >> | 195 | >> +---------+ >> 10 rows in set (10.62 sec) >>

1 1

Re: [Maria-developers] Please look at and advise about MDEV-3801
by Igor Babaev 20 Oct '12

20 Oct '12

On 10/19/2012 12:43 PM, Timour Katchaounov wrote: > Hi Igor, > > Could you please look at this bug > https://mariadb.atlassian.net/browse/MDEV-3801, > and read my last comment. The bug may be (just a guess ATM) related to > index > creation on the temporary table created for the derived table. Since > this is your > code, I'd like you to tell me if you have seen anything similar before, > or if > you have any hints. > > If not, I will continue with the bug and will trace the cause for the > wrong key information. > > Thanks, > Timour Timour, The following patch fixes this problem Regards, Igor. === modified file 'sql/sql_select.cc' --- sql/sql_select.cc 2012-10-10 05:21:22 +0000 +++ sql/sql_select.cc 2012-10-20 05:49:13 +0000 @@ -8981,7 +8981,7 @@ void JOIN::drop_unused_derived_keys() JOIN_TAB *tab; for (tab= first_linear_tab(this, WITHOUT_CONST_TABLES); tab; - tab= next_linear_tab(this, tab, WITHOUT_BUSH_ROOTS)) + tab= next_linear_tab(this, tab, WITH_BUSH_ROOTS)) { TABLE *table=tab->table; MariaDB [test]> set @@tmp_table_size=16384; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... Connection id: 1 Current database: test Query OK, 0 rows affected (5.48 sec) MariaDB [test]> set @@tmp_table_size=16384; Query OK, 0 rows affected (0.00 sec) MariaDB [test]> EXPLAIN SELECT t1.deal_id FROM table_e AS t1 JOIN table_d AS t10 ON (t1.deal_id = t10.deal_id) WHERE t1.deal_id IN (SELECT * FROM (SELECT DISTINCT t1.deal_id FROM table_e AS t1 JOIN table_d AS t2 ON (t1.deal_id = t2.deal_id) JOIN table_c_s AS t3 ON (t2.deal_id = t3.deal_id) AND (t2.feid = t3.feid) JOIN table_f AS t4 ON t3.exchange = t4.exchange) tU) limit 10; +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ | 1 | PRIMARY | <subquery2> | ALL | distinct_key | NULL | NULL | NULL | 8488 | | | 1 | PRIMARY | t1 | eq_ref | PRIMARY | PRIMARY | 4 | tU.deal_id | 1 | Using index | | 1 | PRIMARY | t10 | ref | PRIMARY,full_id | PRIMARY | 4 | tU.deal_id | 1 | Using index | | 2 | MATERIALIZED | <derived3> | ALL | NULL | NULL | NULL | NULL | 8488 | | | 3 | DERIVED | t4 | system | PRIMARY | NULL | NULL | NULL | 1 | Using temporary | | 3 | DERIVED | t3 | ref | PRIMARY,feid,exchange,deal_feid,feid_date | exchange | 3 | const | 8488 | | | 3 | DERIVED | t1 | eq_ref | PRIMARY | PRIMARY | 4 | test.t3.deal_id | 1 | Using index | | 3 | DERIVED | t2 | eq_ref | PRIMARY,full_id | PRIMARY | 8 | test.t3.deal_id,test.t3.feid | 1 | Using index; Distinct | +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ 8 rows in set (4.22 sec) MariaDB [test]> SELECT t1.deal_id FROM table_e AS t1 JOIN table_d AS t10 ON (t1.deal_id = t10.deal_id) WHERE t1.deal_id IN (SELECT * FROM (SELECT DISTINCT t1.deal_id FROM table_e AS t1 JOIN table_d AS t2 ON (t1.deal_id = t2.deal_id) JOIN table_c_s AS t3 ON (t2.deal_id = t3.deal_id) AND (t2.feid = t3.feid) JOIN table_f AS t4 ON t3.exchange = t4.exchange) tU) limit 10; +---------+ | deal_id | +---------+ | 30 | | 82 | | 83 | | 108 | | 155 | | 156 | | 162 | | 184 | | 186 | | 195 | +---------+ 10 rows in set (10.62 sec)

1 0

[Maria-developers] my_apc.cc and new innodb
by Sergei Golubchik 19 Oct '12

19 Oct '12

Hi. There's some issue we need to discuss. I've discovered it when merging your APC code into 10.0-monty, but the problem is simple enough and could've been found earlier too. In the thd_killed() you do: 3809 extern "C" int thd_killed(const MYSQL_THD thd) 3810 { 3811 if (!thd) 3812 thd= current_thd; 3813 3814 Apc_target *apc_target= (Apc_target*)&thd->apc_target; 3815 if (apc_target->have_apc_requests()) 3816 apc_target->process_apc_requests(); 3817 3818 if (!(thd->killed & KILL_HARD_BIT)) 3819 return 0; 3820 return thd->killed; 3821 } But there's no rule that thd_killed() cannot be called with thd different from current_thd. That is, with thd of *another thread*. Of course, when it happens, the caller is responsible for making sure that the owner thread of that thd isn't going to use it concurrently. And that's exactly what happens in the new InnoDB. Now, show explain wasn't designed to work in this situation, it created items (which are created using sql_alloc, that uses mem_root of the current thread, and added to the free item list of the current thd). As a short-term fix, I've added a protection to thd_killed() to not call apc functions if thd != current_thd. But I don't like it, I think a better solution would be to switch thds, like thread pool does, and process apc requests anyway. What do you think? Regards, Sergei

1 0

[Maria-developers] libmysqlclient16 for MariaDB 5.5?
by Jim Nelin 19 Oct '12

19 Oct '12

Hi guys! I just recently joined this team/list, so I'm going to be polite and introduce myself. My name is Jim, work mainly as a server technician and developer and I've been using MariaDB for the past 2 years or so. Just recently i upgraded one of my older MySQL 5.1 nodes to MariaDB 5.5 - which comes with a lot of nice new features (I've previously only used MariaDB 5.1 and 5.2) I mainly use Debian nodes (stable/squeeze) for my and my clients projects. One thing i discovered shortly after the first upgrade was that there was no libmysqlclient16 (or libmariadbclient16) for the 5.5 release of MariaDB - only the new libmysqlclient18. This causes my applications to still use the older MySQL 5.1 release of libmysqlclient16 from the stable debian repo. As a lot of packages in Debian Squeeze (and squeeze-backports) still depends on libmysqlclient16 (php5-mysql, postfix-mysql, dovecot-common to mention a few.) It still works without any problems, with the stable/squeeze 5.1 mysqlclientlib16 (afaik anyway), but seems a bit risky to use an older client-lib against a newer server (I'm not aware of the changes in the client lib between 5.1 and 5.5 tho). The work-around this (to make it "a bit" better) was to manually install the .deb packages from MariaDB 5.3 which DO contain libmysqlclient16 (libmysqlclient16-5.3.9-mariadb117~squeeze). This still isn't "perfect" in my opinion - i would love to have a real lib(mysql|mariadb)client16 for the 5.5 release on Debian Squeeze. This is my current "setup" is the following: http://paste.debian.net/hidden/7c8219e0/ (basicly the client16-lib from 5.3 against a 5.5 server). I understand that it won't be any performance increases or similar due to older applications not using the new features in libmysqlclient18 or mDB 5.5. But it would help me to get rid of the warning below in phpMyAdmin and also make me feel more safe with the setup - a more consistent setup. "Your PHP MySQL library version 5.3.9-MariaDB differs from your MySQL server version 5.5.27. This may cause unpredictable behavior." So, the questions is - why is there isn't any libmysqlclient16 for MariaDB 5.5? And it possible to add it to the repos? If not - why? Can this be solved with a virtual package or wrapper-lib of some kind against libmariadbclient18? Please correct me if I've got something wrong. Thanks in advance, Best Regards -- Jim Nelin | +46 (0)720 454 909 Jine | http://jine.se | jim(a)jine.se

2 1

[Maria-developers] Fwd: XA problems in mysql-5.6.7-rc
by Rich Prohaska 19 Oct '12

19 Oct '12

Hello, I have been debugging a deadlock problem in the 2PC logging code in MySQL 5.6.7-rc. Luckily, Sergei pointed me to a bug fix. I integrated this fix into MySQL 5.6.7, and with a couple of tweaks, the deadlock problem is fixed. I verified the fix with helgrind, since helgrind reported a lock ordering violation in the original 5.6.7 code. My tweaks are: 1. Move the mutex unlock to the right place in the TC_LOG_MMAP::log_xid function. WRT the main maria trunk, move line 7087 of log.c to after the break, since the code expects the page lock to be held after the for loop. 2. helgrind reported a race on tc_log_cur_pages_used, so I used statistic_increment on line 6983. Also need to build with SAFE_STATISTICS so that the statistics counters are atomically modified. ---------- Forwarded message ---------- From: Sergei Golubchik <serg(a)askmonty.org> Date: Thu, Oct 11, 2012 at 4:04 PM Subject: Re: XA problems in mysql-5.6.7-rc To: Rich Prohaska <prohaska(a)tokutek.com> Cc: internals(a)lists.mysql.com Hi, Rich! On Oct 11, Rich Prohaska wrote: > Hello Sergei, > Can you give me a pointer to a source files, or a source control changeset? This is the bug report: https://bugs.launchpad.net/maria/+bug/578117 And this is the patch: http://bazaar.launchpad.net/~maria-captains/maria/5.1/revision/2881 > On Thu, Oct 11, 2012 at 3:54 PM, Sergei Golubchik <serg(a)askmonty.org> wrote: > > On Oct 11, Rich Prohaska wrote: > >> Hello All, > >> > >> TokuDB supports XA and runs in both MariaDB and MySQL. It works in > >> MariaDB without any patches. Unfortunately, there are several XA bugs > >> related to multiple XA storage engines in MySQL 5.6.7. It would be > >> great if these bugs were fixed in MySQL. One way to test these > >> changes is to include another example storage engine that supports XA. > >> This storage engine does not even need to work; it just needs to > >> export the prepare method in the handlerton interface. > >> > >> There is a block of code in handler.cc surrounded by > >> WILL_BE_DELETED_LATER. If I disable the debug assert, what else do I > >> need to worry about? > > > > There is a possible deadlock in the MySQL XA code. I don't know how > > probable it is to get it in practice, we've found it with our mutex > > deadlock detector (MariaDB only). I had to rework XA code in MariaDB > > quite a bit to get rid of the deadlock. > > Regards, Sergei

1 0

[Maria-developers] MariaDB and spatial indexes
by Vangelis Katsikaros 16 Oct '12

16 Oct '12

Hi I completed my MSc thesis, that was performed with the co-supervision of Sergei Golubchik from Monty Program AB. It's titled "Towards the universal spatial data model based indexing and its implementation in MySQL" which translates into doing a literature overview for R-tree based spatial indexes, GiST trees (Generalized Search Trees) and implementing them in MyISAM. The work is in progress! In case you are interested in the algorithm details of these indexes: * Thesis text: http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6394 * Current patches: https://www.dropbox.com/s/bw2cq2525zypi6m/patches.tar.gz Many many thanks to Sergei for all his help and support :) Regards Vangelis

1 0

[Maria-developers] FYI: SHOW EXPLAIN pushed to 10.0-base
by Sergei Petrunia 06 Oct '12

06 Oct '12

Hello, FYI, SHOW EXPLAIN has been merged and pushed to 10.0-base. It was not pushed into 5.5. Just in case, the final 5.5-based tree with SHOW EXPLAIN patch can be found here: https://code.launchpad.net/~maria-captains/maria/5.5-show-explain BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Global transaction ID
by Kristian Nielsen 05 Oct '12

05 Oct '12

Hi Kristian, Monty asked me to connect with you regarding global transaction IDs in MariaDB. I reviewed some time ago the MySQL 5.6 global transaction ID design and implementation. Unfortunately I really did not like what I found, to the point that I decided this was not something we would want to have or support in MariaDB (see https://lists.launchpad.net/maria-developers/msg04837.html for an ealier mail on this). Both you and Giuseppe also had critisisms about it. Instead I came up with a (in my opinion) better design. I have now written this up in (great) detail here: https://mariadb.atlassian.net/browse/MDEV-26 It would be great if you (/your team) could comment on this and how it relates to what you need or any comments/suggestions etc. I know it is a long text, and a complex problem, so also feel free to just ask me for any clarifications or explanations you would like, even without digging through the whole text. The basic idea is to keep the existing concept of a stream of replication events and a slave's current position within this stream, but in a way that is uniform across the whole multi-level replication topology (global transaction ID), and that facilitates simple scripting of automated promotion of a new master. I have thought for a long time over this problem and how to best approach it and make it work well both with existing replication usage and with new features like multi-source and parallel replication. It is hard to get right, but I am fairly pleased with what I have ended up with so far. I am hopeful that this will be a good way forward, but I very much welcome any comments or alternate suggestions. Regards, - (the other) Kristian.

2 2

[Maria-developers] MariaDB Optimization for MySQL Users
by Daniel Bartholomew 04 Oct '12

04 Oct '12

MariaDB contains many new options and optimizations which, for compatibility or other reasons, are not enabled in the default install. Enabling them helps you gain extra performance from the same hardware when upgrading from MySQL to MariaDB. Monty thought this would be a good article for the Knowledgebase and I agree. I've created a stub article for this using a few short items Monty provided (just to get the ball rolling), and I am now soliciting suggestions from any and all of you so I can expand it. https://kb.askmonty.org/en/mariadb-optimization-for-mysql-users/ Please send me your MariaDB-only tips and recommended settings and other things people should enable or tweak after upgrading to MariaDB and I'll add them to the page (or feel free to do it yourself). Thanks! -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

Re: [Maria-developers] multi-source replication, first (untested) version
by Kristian Nielsen 02 Oct '12

02 Oct '12

Michael Widenius <monty(a)askmonty.org> writes: > I have now got the first version of multi source patch ported to > MariadB. As requested, I did a quick read through of the patch. I took the patch from your 10.0-mdev253 tree, this revision: revid:monty@askmonty.org-20120930233044-e0m6u6t9q3jeqx03 Below are my comments. Overall, it was nice to see the relatively modest changes needed to get this working. - Kristian. ----------------------------------------------------------------------- > === modified file 'sql/sys_vars.cc' > --- sql/sys_vars.cc 2012-09-22 12:30:24 +0000 > +++ sql/sys_vars.cc 2012-10-01 09:55:23 +0000 > +static Sys_var_multi_source_ulong > +Sys_slave_skip_counter("sql_slave_skip_counter", > + "Skip the next N events from the master log", > + SESSION_VAR(slave_skip_counter), > + NO_CMD_LINE, > + offsetof(Master_info, rli.slave_skip_counter), This use of offsetof() on a C++ class (non-POD) is incorrect. Even if it works for you with your current version of GCC and current compiler options, it is likely to break at some point on some future compiler. And it will be hell to track down. Look, I understand that you do not like some of the ways that the C++ language is defined, like here that offsetof() is not allowed on non-POD data structures. But we have to live with it. Surely, it is better to have a standard language that compiler writers and compiler users agree on how works - even if that language is less convenient than what we would like. So let us please change this to be correct C++. I can change it for you if you want. > === modified file 'sql/slave.cc' > --- sql/slave.cc 2012-09-13 12:31:29 +0000 > +++ sql/slave.cc 2012-10-01 09:55:23 +0000 > +bool show_all_master_info(THD* thd) > +{ > + uint i, elements; > + Master_info **tmp; > + DBUG_ENTER("show_master_info"); > + > + if (send_show_master_info_header(thd, active_mi, 1)) > + DBUG_RETURN(TRUE); > + > + if (!(elements= master_info_index->master_info_hash.records)) > + goto end; > + > + /* > + Sort lines to get them into a predicted order > + (needed for test cases and to not confuse users) > + */ > + if (!(tmp= (Master_info**) thd->alloc(sizeof(Master_info*) * elements))) > + DBUG_RETURN(TRUE); > + > + for (i= 0; i < elements; i++) > + { > + tmp[i]= (Master_info *) my_hash_element(&master_info_index-> > + master_info_hash, i); > + } Isn't there a race condition here? What happens if the number of elements changes between looking up number of elements and the loop? If there is a lock protecting this, please add a safe_mutex_assert_owner() and/or a comment. The call of show_all_master_info() is like this: mysql_mutex_lock(&LOCK_active_mi); if (lex->verbose) res= show_all_master_info(thd); Does LOCK_active_mi protect all changes to multi-master state? If so, I think it should be renamed - "active_mi" is rather confusing in this case. > === modified file 'sql/sql_class.h' > --- sql/sql_class.h 2012-09-22 14:11:40 +0000 > +++ sql/sql_class.h 2012-10-01 09:55:23 +0000 > + /** > + Place holders to store Multi-source variables in sys_var.cc during > + update and show of variables. > + */ > + ulong slave_skip_counter; > + ulong max_relay_log_size; > + /* Names. These will be allocated in buffers in thd */ > + LEX_STRING default_master_connection; > + > + LEX_STRING connection_name; /* If slave */ > + char default_master_connection_buff[MAX_CONNECTION_NAME+1]; Is there not some other way to do this? THD seems to have become some kind of garbage dump for all kinds of special-purpose data. This is not good, it pollutes the CPU caches for all threads, and generally makes everything a mess :-/ I would like to see us move towards having less things in THD that are not relevant to all threads, not more. Could we add a single void * somewhere in THD for special-purpose threads? And then replication threads could use that for their special-purpose stuff. Or maybe sub-class THD. Then later we could start moving other stuff out of THD that is not relevant to all threads... > === modified file 'sql/rpl_rli.h' > --- sql/rpl_rli.h 2012-08-27 16:13:17 +0000 > +++ sql/rpl_rli.h 2012-10-01 09:55:23 +0000 > + volatile ulong slave_skip_counter; /* Must be ulong */ Such comments are extremely unhelpful. Instead explain why it must be ulong. > === modified file 'sql/item_create.cc' > --- sql/item_create.cc 2012-01-13 14:50:02 +0000 > +++ sql/item_create.cc 2012-10-01 09:55:23 +0000 > @@ -4393,27 +4393,36 @@ Create_func_master_pos_wait::create_nati > + thd->lex->safe_to_cache_query= 0; > case 3: > thd->lex->safe_to_cache_query= 0; > + case 4: > + thd->lex->safe_to_cache_query= 0; Isn't it redundant to set safe_to_cache_query twice here? > === modified file 'sql/rpl_mi.cc' > --- sql/rpl_mi.cc 2012-03-27 23:04:46 +0000 > +++ sql/rpl_mi.cc 2012-10-01 09:55:23 +0000 > + /* Store connection name and lower case connection name */ > + connection_name.length= cmp_connection_name.length= > + connection_name_arg->length; > + if ((connection_name.str= (char*) my_malloc(connection_name_arg->length*2+2, > + MYF(MY_WME)))) > + { > + cmp_connection_name.str= (connection_name.str + > + connection_name_arg->length+1); > + strmake(connection_name.str, connection_name_arg->str, > + connection_name.length); > + memcpy(cmp_connection_name.str, connection_name_arg->str, > + connection_name.length+1); > + my_casedn_str(system_charset_info, cmp_connection_name.str); > + } Is it really correct to ignore out-of-memory error here, and just do nothing? Ah, I think this is handled by checking Master_info::error() in other places? If so, then please add a comment here explaining this. > +/** > + Create a log file with a signed suffix. I do not know what is meant with the term "signed suffix". This is not something that people will understand. Does it mean "Create a log file name by appending a dash "-" followed by a suffix" ? Also rename the file create_signed_file_name() - again, the "signed" is confusing. For example "create_augmented_file_name", or "create_name_with_dash_and_suffix" or something. > + It's valid if it's a valid system name, is less than > + MAX_CONNECTION_NAME. "It is valid if it is a valid system name of length less than MAX_CONNECTION_NAME". > === modified file 'client/mysqltest.cc' > --- client/mysqltest.cc 2012-08-31 21:54:54 +0000 > +++ client/mysqltest.cc 2012-10-01 09:55:23 +0000 > + if (buff) > + my_free(start); Better use my_free(buff) here. > === modified file 'sql/rpl_mi.h' > --- sql/rpl_mi.h 2012-03-27 23:04:46 +0000 > +++ sql/rpl_mi.h 2012-10-01 09:55:23 +0000 > + char master_log_name[FN_REFLEN+6]; /* Place for multi-*/ "place for" -> "room for". > === modified file 'sql/mysqld.cc' > --- sql/mysqld.cc 2012-09-22 14:11:40 +0000 > +++ sql/mysqld.cc 2012-10-01 09:55:23 +0000 > + /* > + We must have LOCK_open before LOCK_global_system_variables because > + LOCK_open is hold while sql_plugin.c::intern_sys_var_ptr() is called. "is hold" -> "is held" > === modified file 'sql/item_create.cc' > --- sql/item_create.cc 2012-01-13 14:50:02 +0000 > +++ sql/item_create.cc 2012-10-01 09:55:23 +0000 > @@ -4393,27 +4393,36 @@ Create_func_master_pos_wait::create_nati > if (item_list != NULL) > arg_count= item_list->elements; > > + if (arg_count < 2 || arg_count >4) Missing space before "4"

2 1

[Maria-developers] Json Explain
by Peter Laursen 30 Sep '12

30 Sep '12

Is Explain output in Json format planned in MariaDB (refer http://glebshchepa.blogspot.in/2012/04/optimizer-new-explain-formatjson.html) ? If so when? and will the implementation details and output format be fully identical to Oracle/MySQL 5.6? (I did not find it mentioned here: http://monty-says.blogspot.dk/2012/09/now-is-good-time-to-be-part-of-future… http://blog.mariadb.org/mariadb-directions/ .. sorry if I overlooked it somewhere) -- Peter -- Webyog

4 8

Re: [Maria-developers] [Commits] Rev 3578: MDEV-495 backport --ignore-db-dir
by Sergei Golubchik 25 Sep '12

25 Sep '12

Hi, Holyfoot! On Sep 25, holyfoot(a)askmonty.org wrote: > revno: 3578 > revision-id: holyfoot(a)askmonty.org-20120925094024-yv0wpbl6k7b6bgg8 > parent: sanja(a)montyprogram.com-20120920094859-lcrxe8lr1mtg8h10 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev495 > timestamp: Tue 2012-09-25 14:40:24 +0500 > message: > MDEV-495 backport --ignore-db-dir. > The feature was backported from MySQL 5.6. > Some code was added to make commands as > SELECT * FROM ignored_db.t1; > CALL ignored_db.proc(); > USE ignored_db; > to take that option into account. Just a few comments: 1. add a test case for CREATE DATABASE, CREATE DATABASE ignored_db should fail. 2. I'd prefer to get an error "db not found" instead of "incorrect db name", is it difficult to do? 3. add a test for the following: lost+found is ignored, but CREATE DATABASE `lost+found` should succeed! (indeed, it creates a directory lost@002bfound, which is not ignored), and later SELECT, CALL, USE, SHOW DATABASES, should show that new db. > === added file 'mysql-test/r/ignore_db_dirs_basic.result' > --- a/mysql-test/r/ignore_db_dirs_basic.result 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/r/ignore_db_dirs_basic.result 2012-09-25 09:40:24 +0000 > @@ -0,0 +1,28 @@ > +select @@ignore_db_dirs; > +@@ignore_db_dirs > +e,lost+found,.mysqlgui,ignored_db > +# Check that SHOW DATABASES ignores all directories from > +# @@ignore_db_dirs and all directories with names starting > +# with '.' > +SHOW DATABASES; > +Database > +information_schema > +#mysql50#.otherdir > +mtr > +mysql > +test > +USE ignored_db; > +ERROR 42000: Incorrect database name 'ignored_db' > +SELECT * FROM ignored_db.t1; > +ERROR 42000: Incorrect database name 'ignored_db' > +CALL ignored_db.p1(); > +ERROR 42000: Incorrect database name 'ignored_db' > +SELECT COUNT(*) FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME='ignored_db'; > +COUNT(*) > +1 > +SET @@global.ignore_db_dirs = 'aha'; > +ERROR HY000: Variable 'ignore_db_dirs' is a read only variable > +SET @@local.ignore_db_dirs = 'aha'; > +ERROR HY000: Variable 'ignore_db_dirs' is a read only variable > +SET @@ignore_db_dirs = 'aha'; > +ERROR HY000: Variable 'ignore_db_dirs' is a read only variable When merging to 5.5 you'll need to move this test to sys_vars suite (otherwise sys_vars.all_vars will fail) > === modified file 'sql/sql_show.cc' > --- a/sql/sql_show.cc 2012-08-24 13:39:34 +0000 > +++ b/sql/sql_show.cc 2012-09-25 09:40:24 +0000 > @@ -379,6 +379,239 @@ bool mysqld_show_privileges(THD *thd) > } > > > +/** Hash of LEX_STRINGs used to search for ignored db directories. */ > +static HASH ignore_db_dirs_hash; > + > +/** > + An array of LEX_STRING pointers to collect the options at > + option parsing time. > +*/ > +static DYNAMIC_ARRAY ignore_db_dirs_array; I don't think that sql_show.cc is where all this code really belongs, but ok, as you like. Regards, Sergei

1 0

[Maria-developers] FYI: More ha_cassandra benchmark data
by Sergei Petrunia 24 Sep '12

24 Sep '12

Hi! For the record, posting some ha_cassandra benchmarks: <contents> Run #1: re-run previous benchmarks Run #2: evaluate max. network performance Run #3: Load two dbt3 databases at once Run #4: load two datasets into different nodes Results Conclusions </contents> === Run #1: re-run previous benchmarks === My results with - m1.large mysqld node (mysqld uses optimized binary) - Cassandra on 2*m1.large nodes - Loading data with the standard LOAD DATA INFILE command. - %CPU mysqld =14 sar -n DEV 10: 03:04:41 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:04:51 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 03:04:51 PM eth0 1575.20 156.60 81.47 4458.13 0.00 0.00 0.00 On cassandra node: - jsvc %CPU 70-170... 02:51:00 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 02:51:05 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:51:05 PM eth0 8325.20 7333.80 3376.25 1696.74 0.00 0.00 0.00 Total load time: time: 19 min. === Run #2: evaluate max. network performance === for comparison, sending /dev/zero on one host to /dev/null on another with nc: 03:39:24 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:39:34 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 03:39:34 PM eth0 16756.70 3688.50 851.91 87477.61 0.00 0.00 0.00 sending in both directions: SQL: 03:45:36 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:45:46 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 03:45:46 PM eth0 63308.20 4473.50 73002.80 81484.77 0.00 0.00 0.00 Cassandra: 03:47:32 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:47:42 PM lo 0.20 0.20 0.01 0.01 0.00 0.00 0.00 03:47:42 PM eth0 60741.50 15241.70 84110.07 70712.32 0.00 0.00 0.00 === Run #3: Load two dbt3 databases at once === - Loading two DBT-3 databases at once. - The databases have their own key spaces, and mysql tables - Still, both loaders connect to the same cassandra host. Cassandra host: 04:27:57 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 04:28:07 PM eth0 9979.10 8984.00 6006.77 2621.08 0.00 0.00 0.00 04:29:07 PM eth0 9181.80 7315.00 6416.25 2686.83 0.00 0.00 0.00 04:29:17 PM eth0 10277.00 9629.60 6047.00 2672.02 0.00 0.00 0.00 04:30:23 PM CPU %user %nice %system %iowait %steal %idle 04:30:33 PM all 34.56 24.80 24.71 0.18 0.05 15.69 04:30:53 PM all 32.08 11.20 24.25 1.31 0.08 31.07 04:31:03 PM all 35.59 22.55 22.28 0.27 0.00 19.31 it has periodical slow-downs like this: 04:32:38 PM all 53.25 1.30 19.05 3.03 0.00 23.38 04:32:39 PM all 65.84 10.40 21.78 0.00 0.00 1.98 04:32:40 PM all 37.14 33.81 18.10 0.95 0.00 10.00 04:32:41 PM all 42.65 25.98 26.96 0.00 0.00 4.41 04:32:42 PM all 33.17 39.42 19.23 0.00 0.00 8.17 04:32:43 PM all 36.89 36.89 19.90 0.00 0.00 6.31 04:32:44 PM all 39.81 35.55 13.74 0.00 0.00 10.90 04:32:45 PM all 34.45 33.97 22.01 0.00 0.00 9.57 04:32:46 PM all 35.58 36.06 21.15 0.00 0.00 7.21 04:32:47 PM all 31.05 35.16 15.07 0.00 0.00 18.72 04:32:48 PM all 41.31 26.29 20.66 0.00 0.00 11.74 04:32:49 PM all 39.02 32.20 25.37 0.00 0.00 3.41 04:32:50 PM all 26.64 34.50 13.97 0.00 0.00 24.89 04:32:51 PM all 56.93 27.72 12.87 0.00 0.00 2.48 04:32:52 PM all 71.14 15.42 11.44 0.00 0.00 1.99 04:32:53 PM all 64.22 8.82 23.04 0.00 0.00 3.92 04:32:54 PM all 76.73 1.98 19.31 0.00 0.00 1.98 04:32:55 PM all 61.24 0.00 30.14 0.96 0.00 7.66 04:32:56 PM all 29.20 0.00 32.40 0.00 0.00 38.40 04:32:57 PM all 30.29 0.00 29.88 0.00 0.00 39.83 04:32:58 PM all 23.19 0.00 20.29 0.00 0.36 56.16 04:32:59 PM all 23.34 0.00 16.38 0.00 0.35 59.93 04:33:00 PM all 20.07 0.00 21.90 0.00 0.00 58.03 04:33:01 PM all 24.20 0.00 18.86 0.00 0.00 56.94 04:33:02 PM all 20.07 0.00 22.18 0.00 0.70 57.04 Is this a compaction? mysqld host: 04:29:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 04:29:01 PM eth0 1416.20 247.90 75.43 6578.11 0.00 0.00 0.00 04:29:11 PM eth0 1175.00 199.60 62.49 5292.67 0.00 0.00 0.00 04:29:21 PM eth0 1074.60 186.90 57.21 4952.19 0.00 0.00 0.00 04:30:30 PM CPU %user %nice %system %iowait %steal %idle 04:30:40 PM all 7.77 0.00 0.56 0.00 0.05 91.62 04:30:50 PM all 8.20 0.00 0.60 0.00 0.30 90.89 04:31:00 PM all 8.01 0.00 0.66 0.00 0.05 91.28 Conclusions: - mysqld is still idle most of the time. - Cassandra is busier. Sometimes, it is 1014/Sep/12 12:50 PM0% busy. - network use betwen mysqld and cassandra is 2x more, which is expected Run time: 24 minutes for both (compare to 19 minutes) === Run #4: load two datasets into different nodes === ==== Network ==== SQL node: 05:44:21 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 05:44:41 PM eth0 1178.10 210.10 62.81 5510.03 0.00 0.00 0.00 05:44:51 PM eth0 1062.10 200.90 57.53 5152.63 0.00 0.00 0.00 05:45:01 PM eth0 965.80 194.00 52.53 5012.62 0.00 0.00 0.00 Cassandra1: 05:44:19 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 05:44:49 PM eth0 10159.00 8659.20 4443.26 1798.75 0.00 0.00 0.00 05:44:59 PM eth0 9018.20 7750.10 3982.21 1578.30 0.00 0.00 0.00 05:45:09 PM eth0 8741.30 7521.70 3921.69 1541.55 0.00 0.00 0.00 Cassandra2: 05:44:56 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 05:45:06 PM eth0 9305.60 7906.00 4121.10 1622.92 0.00 0.00 0.00 05:45:16 PM eth0 9186.50 7828.80 4258.12 1681.78 0.00 0.00 0.00 05:45:26 PM eth0 9675.70 8283.10 4306.89 1698.37 0.00 0.00 0.00 ==== CPU ==== SQL node: 05:46:45 PM CPU %user %nice %system %iowait %steal %idle 05:46:55 PM all 8.39 0.00 0.70 0.00 0.40 90.51 05:47:05 PM all 7.65 0.00 0.74 0.00 0.70 90.91 05:47:15 PM all 8.76 0.00 0.64 0.00 0.89 89.70 cassandra 1: 05:47:01 PM CPU %user %nice %system %iowait %steal %idle 05:47:11 PM all 27.28 28.24 27.55 0.00 0.05 16.89 05:47:21 PM all 31.45 24.44 27.56 1.38 0.05 15.13 05:47:31 PM all 29.46 19.60 23.19 2.35 0.04 25.36 05:47:41 PM all 38.47 0.04 14.21 0.15 0.07 47.06 05:47:51 PM all 31.04 9.92 17.97 0.39 0.08 40.61 05:48:01 PM all 24.73 28.07 21.77 0.26 0.04 25.12 05:48:11 PM all 19.79 11.67 19.23 1.83 0.07 47.41 05:48:21 PM all 20.29 0.00 11.07 0.06 0.13 68.44 05:48:31 PM all 24.08 0.00 11.96 0.13 0.13 63.69 Cassandra 2: 05:46:56 PM CPU %user %nice %system %iowait %steal %idle 05:47:06 PM all 31.97 27.14 26.91 0.09 0.05 13.85 05:47:16 PM all 32.79 20.43 23.39 2.34 0.04 21.01 05:47:26 PM all 26.43 27.07 27.29 0.14 0.09 18.98 05:47:36 PM all 17.72 8.59 12.00 2.97 0.21 58.52 05:47:46 PM all 13.63 0.00 6.77 0.06 0.30 79.23 05:47:56 PM all 27.19 3.92 8.52 0.11 0.25 60.01 05:48:06 PM all 19.94 27.83 18.47 0.13 0.08 33.54 05:48:16 PM all 13.69 16.01 14.24 2.69 0.15 53.22 05:48:26 PM all 18.13 0.00 9.57 0.25 0.38 71.66 === Results === The first client failed with ERROR 1928 (HY000) at line 145: Internal error: 'TimedOutException: Default TException.' real 4m3.463s user 0m0.008s sys 0m0.000s The second completed after: real 19m57.875s (which matches load time for loading one dataset). === Conclusions === - need to fix the Timeout error (filed as MDEV-535) - CPU time spent by mysqld is negligible now (I don't know whether this comes from using faster CPU for SQL node, or using a release build of mariadb, or both) - A confirmation of previous findings: if we use only one connection, the speed at which we can insert the data is very limited (see: with two connections, we have inserted 2x data in 20% more time. This is where we used two connections to connect to the *same* node) - I am not sure whether I should just start using multiple connections, or also automatically route the data (both writes and reads) to the right node? - need to check out Hector client. It seems to be able to connect to multiple nodes of Cassandra cluster. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Updates to mytop (from MariaDB 5.5 distribution)
by Michael Widenius 22 Sep '12

22 Sep '12

Hi! I have done some extensions to mytop as part of porting, fixing and enhancing it for MariaDB. I have now merged my changes with 1.91 and thought it was time to share the changes with you. Sorry for not doing it sooner! I also have added Jean Weisbuch patches for --full-queries and reading of .my.cnf Here is a diff of my changes. Hope you find some of them useful! If you have any questions, feel free to ask! Best regards, Monty Creator of MySQL and MariaDB

1 0

[Maria-developers] Cassandra SE: problems including it into release
by Sergei Petrunia 21 Sep '12

21 Sep '12

Hello, I've figured there may be an additional challenge in getting Cassandra storage engine into a *MariaDB release*. The problem is packaging. <contents> Cassandra SE packaging problems Need Thrift library Need recent gcc How did Sphinx SE manage to avoid a similar mess? Possible way out </contents> == Cassandra SE packaging problems == === Need recent gcc === C++ code generated by Thrift uses boost, and so one needs a recent gcc to compile it with MySQL. We have the same issue with oqgraph. See MDEV-501 for details. I am not aware of crashing problems that ograph+boost combination has. Still, recent-gcc requirement may limit the range of platforms we can build for. For example, I never managed to compile on Ubuntu 9.04. === Need Thrift library === Cassandra SE uses Thrift protocol to talk to server. In order to do this, it needs to link againist libthrift.so. There are no Thrift packages in either Fedora or Ubuntu (actually I haven't seen Thrift packages for any OS). Thrift homepage only offers source tarballs. The source tarball uses automake and compiles into 1. thrift compiler binary (we dont actually need it. Source code generated by thrift needs to be modified anyway.) 2. thrift libraries for a dozen of languages, including libthrift.so, which we do need. == How did Sphinx SE manage to avoid a similar mess? == I've asked myself a question how do we manage to include Sphinx SE without having to mess with Sphinx packages. The answer is: Sphinx SE doesn't need any sphinx libraries, either at compile time, or at runtime. It implements its client protocol fully, except for INSERT commands, for which it uses the available mysql client library to talk to Sphinx server using mysql client-server protocol. == An unlikely but possible way out == Cassandra 1.2 (the next version that is promised to be released "in Q4 2012" will introduce native network protocol. A week ago they've pushed first documentation draft. The native network protocol should allow to do everything that Thrift API allows, and more. There are no C++ clients yet, though. If I am faced with a choice betwen 1. implementing a relatively simple network protocol 2. Making Thrift packages for various platforms (and potentially convincing downstream maintainers to accept them) then I will definitely pick #1 (I have implemented network protocols in the past). There is a big risk, though: native protocol support is very new in Cassandra, they pushed it about a week ago. It was pushed into their "trunk" which is under development. I don't know if the code in their trunk is sufficiently stable for us to develop againist. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

3 3

Re: [Maria-developers] New to MariaDB
by Giacomo Picchiarelli 21 Sep '12

21 Sep '12

Thanks! At beginning Il'll try to do some benchmarks! On 13/09/2012 22:11, Michael Widenius wrote: > Hi! > >>>>>> "Sergei" == Sergei Golubchik <serg(a)askmonty.org> writes: > Sergei> Hi, Giacomo! > Sergei> On Aug 11, Giacomo Picchiarelli wrote: >>> Hi, >>> >>> I'm new to MariaDB and I'm reading the knowledge base. >>> I'm interested in studying Aria Engine (and contribute, if I can :) ), >>> are there tasks for newbie developers? > Sergei> I'm afraid, there are no tasks in the Aria engine right now :( > > Sergei> If you're interested to look at other tasks - here is the list of task, > Sergei> feel free to pick anything from it: > > Sergei> https://mariadb.atlassian.net/secure/IssueNavigator.jspa?requestId=11300 > > Actually, there are some open bugs on Aria: > > https://mariadb.atlassian.net/browse/MDEV-333 > > You can go to: > https://mariadb.atlassian.net/secure/QuickSearch.jspa > And search for "Aria" > > Same goes for lanchpad: > > https://bugs.launchpad.net/maria?field.searchtext=aria&search=Search&field.… > > Note that many of the bugs in launchpad for Aria are already fixed. > However it would be quite valuable if you could help us go trough > them and mark which still are open and try to fix them yourself. > > Another thing that can be done on Aria is to run some benchmarks on it > and try to find ways to improve the performance on it. > > Regards, > Michael Widenius > -- Giacomo Picchiarelli Skype: g.picchiarelli

2 1

Re: [Maria-developers] [Commits] Rev 3577: MDEV-521 fix. in file:///home/bell/maria/bzr/work-maria-5.3-MDEV-521/
by Sergei Petrunia 20 Sep '12

20 Sep '12

Hi Sanja, Ok to push. On Thu, Sep 20, 2012 at 12:49:03PM +0300, sanja(a)montyprogram.com wrote: > At file:///home/bell/maria/bzr/work-maria-5.3-MDEV-521/ > > ------------------------------------------------------------ > revno: 3577 > revision-id: sanja(a)montyprogram.com-20120920094859-lcrxe8lr1mtg8h10 > parent: timour(a)askmonty.org-20120917081346-nqd08d0v27w8zp5c > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.3-MDEV-521 > timestamp: Thu 2012-09-20 12:48:59 +0300 > message: > MDEV-521 fix. > > After pullout item during single row subselect transformation it should be fixed properly. > === modified file 'mysql-test/r/subselect.result' > --- a/mysql-test/r/subselect.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect.result 2012-09-20 09:48:59 +0000 > @@ -6163,5 +6163,20 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > > === modified file 'mysql-test/r/subselect_no_mat.result' > --- a/mysql-test/r/subselect_no_mat.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_mat.result 2012-09-20 09:48:59 +0000 > @@ -6162,6 +6162,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set optimizer_switch=default; > > === modified file 'mysql-test/r/subselect_no_opts.result' > --- a/mysql-test/r/subselect_no_opts.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_opts.result 2012-09-20 09:48:59 +0000 > @@ -6158,6 +6158,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set @optimizer_switch_for_subselect_test=null; > > === modified file 'mysql-test/r/subselect_no_scache.result' > --- a/mysql-test/r/subselect_no_scache.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_scache.result 2012-09-20 09:48:59 +0000 > @@ -6169,6 +6169,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set optimizer_switch=default; > > === modified file 'mysql-test/r/subselect_no_semijoin.result' > --- a/mysql-test/r/subselect_no_semijoin.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_semijoin.result 2012-09-20 09:48:59 +0000 > @@ -6158,6 +6158,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set @optimizer_switch_for_subselect_test=null; > > === modified file 'mysql-test/t/subselect.test' > --- a/mysql-test/t/subselect.test 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/t/subselect.test 2012-09-20 09:48:59 +0000 > @@ -5229,5 +5229,18 @@ set optimizer_switch=@mdev367_optimizer_ > > DROP TABLE t1; > > +--echo # > +--echo # MDEV-521 single value subselect transformation problem > +--echo # > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > + > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > + > +# Cleanup > +DROP TABLE t1; > + > --echo # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > > === modified file 'sql/item_subselect.cc' > --- a/sql/item_subselect.cc 2012-09-17 08:13:46 +0000 > +++ b/sql/item_subselect.cc 2012-09-20 09:48:59 +0000 > @@ -1007,11 +1007,9 @@ Item_singlerow_subselect::select_transfo > } > substitution= select_lex->item_list.head(); > /* > - as far as we moved content to upper level, field which depend of > - 'upper' select is not really dependent => we remove this dependence > + as far as we moved content to upper level we have to fix dependences & Co > */ > - substitution->walk(&Item::remove_dependence_processor, 0, > - (uchar *) select_lex->outer_select()); > + substitution->fix_after_pullout(select_lex->outer_select(), &substitution); > } > DBUG_RETURN(false); > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] Server-id patch
by Kristian Nielsen 18 Sep '12

18 Sep '12

Robert Hodges <robert.hodges(a)continuent.com> writes: > I think we need to take this out to one of the email threads. I hope other > people would be interested in these problems. Agree, I've quoted liberally and Cc:'ed maria-developers@ > On Aug 31, 2012, at 6:14 AM PDT, Kristian Nielsen wrote: >> Robert Hodges <robert.hodges(a)continuent.com> writes: >> >>> 1.) The server-id of which I was speaking is a very simple patch to allow us >>> to do the following client command: 'set session server-id=X' where X is >>> whatever we choose. Currently MySQL replication sets the server ID from an >>> internal structure (THD as I recall), which means that applications like >>> Tungsten cannot get the server-id into the binlog when applying updates on a >>> slave. Server-id is essential to prevent replication loops. Monty and I >>> discussed this feature briefly over dinner in April. I hacked in support >>> using a UDF about a year ago but it would be better to have this become a >>> dynamic session variable. >> Yes, it does not seem too hard to do (though as always some care is needed to >> introduce it in a clean way). >> >> I created an issue for this with some descriptions: >> >> https://mariadb.atlassian.net/browse/MDEV-500 > SET SESSION pseudo_server_id=236; > > Then the server_id 236 would to go into the binlog. This seems like a good > idea and avoids accidentally reseting the server_id, which is a pretty > significant variable. Seems there is hope we can have this in 10.0. >>> 2.) I agree with your criticisms of the MySQL 5.6 replication design. It's >>> surprisingly complex and looks as if it will have a lot of bugs. I felt at >> >> Yes. They need this complexity because their parallel replication (which is >> similar to Tungsten parallel replication I think?) can arbitrarily reorder >> transactions in a slave binlog compared to the order in the master binlog. > We just use the sequential log. The slaves are complex but it does not > pollute the log. In fact, everything up to the point of applying to the > slave is purely sequential. I think their design got this wrong. Well, the MySQL --log-slave-updates work by generating completely new events on the slave as the received events are executed by the SQL thread(s). Do you mean that Tungsten does not do this, but uses the original binlog with original events from the master? How is multi-master (one slave replicating from several masters) handled (if it is handled) ? >>> the time that the better approach on transaction IDs would be to put enough >>> information into logs to enable applications to compute a vector clock on >>> the stream of binlog events coming from multiple servers. This is a >> >> Interesting, I though about a concept of "vector clock" before, but did not >> know the terminology. >> >> Aha, so your point is that even though we will apply changes from multiple >> upstream master servers in parallel on a single slave, with a vector clock, we >> can still order them on a slave according to which transactions on one server >> where visible to which transactions on the other. And stuff like >> that. Interesting! >> >> But note that to be truly "theoretically sound", all queries (ie. SELECT) as >> well as all applications that interact directly or indirectly with multiple >> servers need to be included in the vector clock. So it is not without >> complexity. > Are you thinking about holding locks? For practical purposes it's enough to > do writes only. My point is that you have a global ID that is partially > ordered by originating server. Vector clocks are the traditional way of > handling that. This is the basis for most mechanisms of conflict > resolution--you check to see if the clocks are ordered and if they are not > (because two thing happened at the same time, hence are not comparable), it > signals a conflict that must be reconciled. Well, I was mostly just reacting to the phrase "truly theoretically sound". If readers are not considered in the vector clock, then they are able to observe different states of the database at different nodes in the cluster. But as you say, that is probably acceptable. >> This does not immediately apply to the MySQL design, as they work with a >> single master (applying in parallel transactions in different databases), and >> as they arbitrarily reorder between different databases, without any attempt >> to maintain inter-database consistency. But of course, my point is that this >> design is fundamentally wrong. >> >>> theoretically sound approach to enable both synchronous and asynchronous >>> multi-master replication. (Galera incidentally uses approximately this >>> approach for sync replication as you may be aware.) If you give us server >>> IDs we don't need anything else. Oops, I forgot. Making the binlog truly >>> crash safe in every possible case is also very helpful. >> >> I think it is getting closer to being truly crash safe - can you mention the >> situations where it is not? And btw., are you interested in the recent testing >> we (especially Elena) made for testing such crash safety, and the fixes made >> for it? We found some fairly interesting (and scary) stuff... > Actually your (MP's?) fix for group commit is the most important > improvement. Can we turn sync_binlog back on without killing performance? Yes, well at least performance is substantially improved with sync_binlog=1. With a parallel load you can see improvements of 10x or more, and even single-threaded load is improved in 10.0. >> I have been thinking about this for some time. Basically, the correct approach >> is to make the binlog into a true transaction log using standard textbook >> techniques. Pre-allocate the log, make it page based (this will also improve >> performance). Do ping-pong writes at the end so that we can recover even from >> torn pages. Stuff like that. Would be cool to do, but so far I have not really >> seen much interest for something like this. > It would be a lot of work. If group commit is fixed, then most on-premise > people can solve enough problems with RAID and BBU that they don't need much > more. Right, agree, I thought you had something more subtle than sync_binlog=0 in mind. (BTW, I fixed a couple binlog crash recovery bugs recently in MariaDB, which also affect MySQL). >>> 3.) I read the email you cited. I need some time to study it in order to >>> comment intelligently. The most immediate feedback is that I doubt you want >>> to make assumptions about how slaves will apply transactions. The role of >>> the master is to provide local transactions in serial order. Looking across >>> a set of masters you then have a set of easily identifiable streams of >>> partially ordered transactions. That in turn provides a mathematically >>> tractable model for reasoning about things like update order, dependencies, >>> and potential conflicts. Any metadata you can supply about dependencies, >>> e.g., which schema(s) transactions actually touch is useful but may or may >>> not be relevant on slaves depending on the application. It's up to slaves >>> to make decisions on reordering. >> >> But think about a three-level multi-master topology. Servers A and B are >> masters for server C, which is itself a master for D. The DBA/application >> developer will declare that transactions on A and B are independent, so C can >> apply them in any order, and in parallel. Clearly on D we will want to also >> apply them in parallel, and the only way we can do that is to use the >> different server ids A and B in the stream of events in the binlog of C. >> >> So now consider a simple two-level topology with just master C and slave >> D. Why not still let C put different "server_id"s in different events, as a >> way to tell server D that events are independent? For example, if the DBA runs >> a long-running ALTER TABLE, he can ensure that no transactions touch this >> table while it runs, and just SET SESSION SERVER_ID to something else, and it >> can run in parallel on D without delaying normal transactions. >> >> Or think about MySQL 5.6 style parallel replication, where the DBA declares >> that on server A, transactions against different databases are to be >> considered independent and applied in parallel on the slave. While on server B >> such transactions are not independent. Now we want to setup slave C to >> replicate from both A and B. Clearly, C is the wrong place to configure >> whether transactions in different databases are to be considered independent - >> this is a property of how the transactions are generated, not where they are >> replicated. But if we just let the server_id of transactions on master A be a >> hash of the affected database, then MySQL 5.6 style parallel replication just >> falls out of multi-master for tree, without any of their complexity. > I have not ever seen this topology arise in practice within MySQL, as people > tend to keep things pretty homogeneous. It seems to arise mostly in data > warehouse loading where multiple applications load data into the same store > for reporting. These are more heterogeneous but arallel information from > MySQL is of limited value here--data warehouses in Vertica or Hadoop don't > have constraints and you can just jam data into them in parallel. Ok. >> It is just a matter of not artificially restricting transactions generated on >> a given server to have just one server_id value. Which is in fact exactly what >> you asked me about originally with SET SESSION SERVER_ID :) >> >> On the other hand, the slave is free to try to discover more opportunities for >> independence / parallelism if it can, and use them. But automatically >> discovering opportunities for parallelism on the slave is a very hard problem >> in general, as I am sure you agree ... > Meanwhile, I don't like the idea of using server_id in the way you describe > because it overloads the meaning. Server_id just tells us which server > actually committed the transaction in the first place. The reason we need > the session variable was to be able to signal this properly. I was not > intending to use it more broadly. Ok, I've noted your concern. It would be interesting to learn how you use the knowledge that one mysqld instance can have at most one server id in generated transactions? > More generally, I believe that if you want to allow programmers to declare > that things are independent it would be best to add SQL-level features. For > instance, Facebook has a patch that prevents transactions from spanning > schemas. This is a very useful feature, because it ensures that all updates > in one schema are independent from others. That in turn means you can > parallelize without being afraid of deadlocks (a big problem in this space) > or constraint violations. > > You could generalize this into the notion of a shard, which is a scope for > independent transactions and presumably also referential integrity. This > would be a very useful feature for SQL databases. I don't know if you have > read it but Pat Helland has a great paper on this topic called "Life beyond > Distributed Transactions: an Apostate’s Opinion." It was visionary at the > time (and still is) and has never been properly implemented in relational > DBMS to my knowledge. Well, independent schemas are nice, but not enough. There are many applications that need the slave to apply transactions in parallel even within a single table. - Kristian.

1 0

[Maria-developers] Review/help needed with MDEV-532
by Kristian Nielsen 14 Sep '12

14 Sep '12

Hi Serg! I've pushed the MDEV-181/MDEV-225/MDEV-232 to 10.0-base, thanks for review! As you remember, we discussed extending it with a separate patch for doing the checkpoint inside InnoDB in an asynchroneous manner. I have created MDEV-532 for this. It was not hard to create the patch, and it seems to work. But I could use some help with one last test failure that I am somewhat mystified by: rpl.rpl_binlog_index 'row' w7 [ fail ] Found warnings/errors in server log file! Test ended at 2012-09-13 15:44:01 line 120913 15:43:48 [ERROR] mysqld: Table './test/t1' is marked as crashed and should be repaired 120913 15:43:48 [Warning] Checking table: './test/t1' 120913 15:43:48 [ERROR] test.t1: 1 client is using or hasn't closed the table properly I assume this is related to how I create a background thread for the binlog, something that I do wrong or fail to do, so any suggestions on where to look would be appreciated. I have pushed the code here: lp:~maria-captains/maria/10.0-knielsen - Kristian.

1 1

[Maria-developers] New to MariaDB
by Giacomo Picchiarelli 13 Sep '12

13 Sep '12

Hi, I'm new to MariaDB and I'm reading the knowledge base. I'm interested in studying Aria Engine (and contribute, if I can :) ), are there tasks for newbie developers? Thank you. Regards. -- Giacomo Picchiarelli <gpicchiarelli(a)gmail.com> Skype: g.picchiarelli

2 1

Re: [Maria-developers] IDE debugging Mariadb linux build
by md＠rpzdesign.com 10 Sep '12

10 Sep '12

Timour: I respect your technical position, but I do not get my paychecks from monty. My total budgeted time to deal with "issues" related to Aria is exactly zero, so I am not going to try to ramp up learning hardware GDB breakpoints, etc. My broadcast was to enable the many other non-montyprogram developers a chance to load the project into the IDE/debugger so they can tweak or modify something and then contribute back into the community. They likely won't contribute if the only thing they can use is VI and GDB command line. Fortunately, Monty has provided good tested code over the years and that is why Maria will succeed, I do not have to do everything myself, but when I run into a problem, I have a chance at fixing it. Netbeans makes it easy to use the CmakeLists.txt file to load the project. Basically, open source gives me all the power and not ALL the responsibility, bloody magic really. Keep up the good work, marco On 9/10/2012 1:26 AM, Timour Katchaounov wrote: > Hi Marco, > >> I can report to all the developers for MariaDB on linux >> that Netbeans 7.1.2 is working VERY well on Fedora 14 - 64 bit. > > Netbeans has worked well for MySQL already few years ago, I tried it. > The main problem for me with both Netbeans and Eclipse was that both > have limited GDB support. I couldn't find a way to set hardware > breakpoints, neither to inspect easily arrays. AFAIR it was not possible > to evaluate expressions within the debugger, not sure about conditional > breakpoints. > > All the above features are indispensable when working on server bugs, or > analyzing how things work. If you have found a way to overcome these > limitations, then these IDEs might be useful for server development. > > > Timour > > >> >> Vladislav indicated he used only Windows Visual studio and OSX Xcode. >> >> It has a great Find It Grep support and the debugger is easy to set >> breakpoints and run the code. >> >> Marco >> >> On 9/7/2012 3:55 AM, Vladislav Vaintroub wrote: >>> >>>> -----Original Message----- >>>> From: maria-developers- >>>> bounces+wlad=montyprogram.com(a)lists.launchpad.net [mailto:maria- >>>> developers-bounces+wlad=montyprogram.com(a)lists.launchpad.net] On >>>> Behalf Of md(a)rpzdesign.com >>>> Sent: Freitag, 7. September 2012 03:00 >>>> To: maria-developers(a)lists.launchpad.net >>>> Subject: [Maria-developers] IDE debugging Mariadb linux build >>>> >>>> I am using cmake -G"CodeBlocks - Unix Makefiles" to load the 5.5.25 >>>> project into CodeBlocks for building and editing. >>>> >>>> Does anybody use KDevelop4 or Codeblocks to build mariadb? >>> I used KDevelop some time ago (half a year or so), and Codeblocks >>> long-long >>> ago (still with vanilla MySQL) >>> >>>> If so, how do you set the parameters up so you can run the server in >>>> gdb debug? >>> Debugging and building are slightly different things. CMake does not >>> have >>> builtin support for IDE-debugging, so you need to use a semi-manual >>> method >>> . >>> >>> I found the below method useful >>> 1. Switch to build directory >>> 2. cd mysql-test >>> 3. perl mysql-test-run.pl --start 1st --manual-debug >>> >>> This will create database directory, config file, and write path for >>> mysqld >>> executable together working directory and args to the console. You can >>> copy/paste it into IDE, where debug parameters for executable are >>> stored (I >>> do not know btw where KDevelop4 has parameters for debugging, I just >>> describe the process I used in Visual Studio and Xcode ) >>> >>>> Is there a way to run mysqld without mysql_safe monitor daemon in >>>> front >>>> of it? >>> Yes, for example the one above. >>> >>> >>> >>> >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers(a)lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp >

1 0

Re: [Maria-developers] [knielsen@knielsen-hq.org] [Commits] Rev 3459: MDEV-232: Remove one fsync() from commit phase. in http://bazaar.launchpad.net/~maria-captains/maria/5.5
by Kristian Nielsen 09 Sep '12

09 Sep '12

Sergei Golubchik <serg(a)askmonty.org> writes: > Hi, Kristian! > > On Jun 22, Kristian Nielsen wrote: >> Can you review below patch for mdev-232 (At your convenience, not urgent)? > > Here's the review. Mostly questions, I have to confess :) Thanks a lot for your review! I've tried to answer all your questions below, and for many I have tried to clarify the patch, as noted. Do you want to see the changes for when I implement async commit_checkpoint_request() in InnoDB/XtraDB, before I push it to 10.0? >> === modified file 'sql/slave.cc' >> --- a/sql/slave.cc 2012-06-22 09:40:40 +0000 >> +++ b/sql/slave.cc 2012-06-22 10:20:29 +0000 >> @@ -1751,13 +1751,12 @@ when it try to get the value of TIME_ZON >> /* Announce MariaDB slave capabilities. */ >> DBUG_EXECUTE_IF("simulate_slave_capability_none", goto after_set_capability;); >> { >> - const char *q= >> - DBUG_EVALUATE_IF("simulate_slave_capability_old_53", >> - "SET @mariadb_slave_capability=" >> - STRINGIFY_ARG(MARIA_SLAVE_CAPABILITY_ANNOTATE), >> - "SET @mariadb_slave_capability=" >> - STRINGIFY_ARG(MARIA_SLAVE_CAPABILITY_MINE)); >> - if (mysql_real_query(mysql, q, strlen(q))) >> + int rc= DBUG_EVALUATE_IF("simulate_slave_capability_old_53", >> + mysql_real_query(mysql, STRING_WITH_LEN("SET @mariadb_slave_capability=" >> + STRINGIFY_ARG(MARIA_SLAVE_CAPABILITY_ANNOTATE))), >> + mysql_real_query(mysql, STRING_WITH_LEN("SET @mariadb_slave_capability=" >> + STRINGIFY_ARG(MARIA_SLAVE_CAPABILITY_MINE)))); >> + if (rc) > > Why? Monty asked me to not use strlen() which I had introduced due to a request from you in review of an earlier worklog. And you wrote that you did not care much one way or the other (neither do I), so I reverted the change back to the original without strlen(). (Maybe I should have done it as a separate changeset, but I took this short cut since MDEV 225/181/232 are kind of tied together and will likely be pushed together.) >> === modified file 'storage/xtradb/handler/ha_innodb.cc' >> --- a/storage/xtradb/handler/ha_innodb.cc 2012-06-15 12:54:23 +0000 >> +++ b/storage/xtradb/handler/ha_innodb.cc 2012-06-22 10:20:29 +0000 >> @@ -485,8 +486,11 @@ static MYSQL_THDVAR_ULONG(lock_wait_time >> static MYSQL_THDVAR_ULONG(flush_log_at_trx_commit, PLUGIN_VAR_OPCMDARG, >> "Set to 0 (write and flush once per second)," >> " 1 (write and flush at each commit)" >> - " or 2 (write at commit, flush once per second).", >> - NULL, NULL, 1, 0, 2, 0); >> + " or 2 (write at commit, flush once per second)." >> + " Can also be set to 3 to write and flush at each commit, even when this is" >> + " unnecessary because of binlog crash recovery; should not normally be used" >> + " due to performance degration, but provided for completeness.", > > would you like to improve the description on =1 value? For example, > 1 (write and flush on prepare or on commit) > 3 (write and flush both on prepare and on commit, usually it is redundant) Yes, good idea. Here is my suggestion: "Controls the durability/speed trade-off for commits." " Set to 0 (write and flush redo log to disk only once per second)," " 1 (flush to disk at each commit)," " 2 (write to kernel at commit but flush to disk only once per second)" " or 3 (flush to disk twice per commit, usually redundant)." " 1 and 3 guarantees that after a crash, committed transactions will" " not be lost and will be consistent with the binlog and other transactional" " engines. 2 can get inconsistent and lose transactions if there is a" " power failure or kernel crash but not if mysqld crashes. 0 has no" " guarantees in case of crash. 0 and 2 can be faster than 1 or 3.", >> /*****************************************************************//** >> +Handle a commit checkpoint request from server layer. >> +We simply flush the redo log immediately and do the notify call.*/ >> +static >> +void >> +innobase_checkpoint_request( >> + handlerton *hton, >> + void *cookie) >> +{ >> + log_buffer_flush_to_disk(); >> + ha_commit_checkpoint_notify(hton, cookie); >> +} >> + >> +/*****************************************************************//** > > What about making innobase_checkpoint_request() implementation that flushed > in the background? At least to serve as an example of the correct usage of your > API, if not for performance. And to test your code for cases when the notification > isn't coming directly from the checkpoint_request method. > > it looks like it's very easy to make innodb to invoke ha_commit_checkpoint_notify > from the background flusher thread (the one that syncs once a second). > you only need to store the cookie in the log_sys, and invoke > ha_commit_checkpoint_notify after the flush when this stored cookie isn't NULL. Yes, I would like to do this, I will try. It might even help performance, I have seen several blog posts mentioning problems with stalls at binlog rotation, and this might help. As you say, this should be easy. I need to store a list of cookies and their associated LSN (not just a single cookie), but this is infrequently executed so just a simple linked list with malloc/free of elements will work fine. >> +void >> +ha_commit_checkpoint_notify(handlerton *hton, void *cookie) > > are there other ha_xxx methods that a storage engine calls? > > typically server calls methods, like, ha_commit_trans, and they invoke storage > engine methods. Not the other way around. It is exactly for this reason I used names > like "trans_register_ha" - inverse of ha_register_trans - to emphasize that this > method is invoked by the engine to call into the server, not by the server to > call into the engine. Ah, thanks, good point. I suggest "commit_checkpoint_notify_ha()", what do you think? >> === modified file 'sql/log.h' >> --- a/sql/log.h 2012-06-22 09:46:28 +0000 >> +++ b/sql/log.h 2012-06-22 10:20:29 +0000 >> @@ -359,9 +373,14 @@ class MYSQL_QUERY_LOG: public MYSQL_LOG >> Ids start from BINLOG_COOKIE_START; the value BINLOG_COOKIE_DUMMY is special >> meaning "no binlog" (we cannot use zero as that is reserved for error return >> from log_and_order). >> + The low-order bit is reserved for an error flag. >> */ >> -#define BINLOG_COOKIE_DUMMY 1 >> -#define BINLOG_COOKIE_START 2 >> +#define BINLOG_COOKIE_ERROR_FLAG 1 >> +#define BINLOG_COOKIE_ID_INCREMENT (BINLOG_COOKIE_ERROR_FLAG*2) >> +#define BINLOG_COOKIE_DUMMY (BINLOG_COOKIE_ID_INCREMENT*1) > > Is that right? BINLOG_COOKIE_DUMMY == BINLOG_COOKIE_ID_INCREMENT ? It is "right", but I agree it is incomprehensible. I will improve it. The thing is, I need to store a number of different values in the "cookie" used with log_and_order() and unlog(): - 0, which is the error return value from log_and_order() - A pair (binlog_id, error_flag) - A special dummy binlog_id meaning that nothing was logged to the binlog. I will change it to this, what do you think? : /* We assign each binlog file an internal ID, used to identify them for unlog(). The IDs start from 0 and increment for each new binlog created. In unlog() we need to know the ID of the binlog file that the corresponding transaction was written into. We also need a special value for a corner case where there is no corresponding binlog id (since nothing was logged), as well as an error flag to mark that unlog() must return failure. We use the following macros to pack all of this information into the single ulong available with log_and_order() / unlog(). Note that we cannot use the value 0, as that is reserved as error return value from log_and_order(). */ #define BINLOG_COOKIE_ERROR_RETURN 0 #define BINLOG_COOKIE_DUMMY_ID 1 #define BINLOG_COOKIE_BASE 2 #define BINLOG_COOKIE_DUMMY(error_flag) \ ( (BINLOG_COOKIE_DUMMY_ID<<1) | ((error_flag)&1) ) #define BINLOG_COOKIE_MAKE(id, error_flag) \ ( (((id)+BINLOG_COOKIE_BASE)<<1) | ((error_flag)&1) ) #define BINLOG_COOKIE_GET_ERROR_FLAG(c) ((c) & 1) #define BINLOG_COOKIE_GET_ID(c) ( ((ulong)(c)>>1) - BINLOG_COOKIE_BASE ) #define BINLOG_COOKIE_IS_DUMMY(c) \ ( ((ulong)(c)>>1) == BINLOG_COOKIE_DUMMY_ID ) Then the code in log.cc can use these macros, and not have to depend on strange details on how to pack different values into a single ulong. I think the resulting code is much clearer, now BINLOG_COOKIE_* is only used in log_and_order() and unlog() functions, thanks! >> class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG >> @@ -464,15 +492,6 @@ class MYSQL_BIN_LOG: public TC_LOG, priv >> uint file_id; >> uint open_count; // For replication >> int readers_count; >> - bool need_start_event; >> - /* >> - no_auto_events means we don't want any of these automatic events : >> - Start/Rotate/Stop. That is, in 4.x when we rotate a relay log, we don't >> - want a Rotate_log event to be written to the relay log. When we start a >> - relay log etc. So in 4.x this is 1 for relay logs, 0 for binlogs. >> - In 5.0 it's 0 for relay logs too! >> - */ >> - bool no_auto_events; > > Why? Because it's some old code which isn't used nowadays? Yes. This was dead code. >> === modified file 'sql/log.cc' >> --- a/sql/log.cc 2012-06-22 09:46:28 +0000 >> +++ b/sql/log.cc 2012-06-22 10:20:29 +0000 >> @@ -3597,12 +3623,40 @@ bool MYSQL_BIN_LOG::reset_logs(THD* thd) >> + /* >> We are going to nuke all binary log files. >> - So first wait until all pending binlog checkpoints have completed. >> + Without binlog, we cannot XA recover prepared-but-not-committed >> + transactions in engines. So force a commit checkpoint first. >> + >> + Note that we take and immediately release LOCK_commit_ordered. This has >> + the effect to ensure that any on-going group commit (in >> + trx_group_commit_leader()) has completed before we request the checkpoint, >> + due to the chaining of LOCK_log and LOCK_commit_ordered in that function. >> + (We are holding LOCK_log, so no new group commit can start). >> + >> + Without this, it is possible (though perhaps unlikely) that the RESET >> + MASTER could run in-between the write to the binlog and the >> + commit_ordered() in the engine of some transaction, and then a crash >> + later would leave such transaction not recoverable. >> */ >> + mysql_mutex_lock(&LOCK_commit_ordered); >> + mysql_mutex_unlock(&LOCK_commit_ordered); >> + >> + mark_xids_active(current_binlog_id, 1); >> + do_checkpoint_request(current_binlog_id); > > why do_checkpoint_request ? There may be transactions in the current binlog file that are not yet flushed to disk in the engine. If we proceed in RESET MASTER to delete the binlog file, and then crash, then we could lose such transactions. So we request a checkpoint for the current binlog file (we already did so for any previous binlog files). And then wait for this and any other pending checkpoint to complete. Only then can we safely delete binlog files. >> @@ -3731,6 +3780,31 @@ bool MYSQL_BIN_LOG::reset_logs(THD* thd) >> + /* >> + Remove all entries in the xid_count list except the last. >> + Normally we will just be deleting all the entries that we waited for to >> + drop to zero above. But if we fail during RESET MASTER for some reason >> + then we will not have created any new log file, and we may keep the last >> + of the old entries. > > I'm not sure I understand this comment. > What's the point of keeping the last of the old entries, if old binlog > files are deleted and binary logging is probably stopped (when RESET MASTER fails > it's kinda serious)? The code in sql/log.cc assumes that binlog_xid_count_list is never empty. So even in the error case we must be sure to leave at least one entry in there. So the comment is to explain why we do not simply delete everything in the list at the start, and then insert the new entry on top of that. If we get an error half-way, error handling is just easier by doing the deletion at the end, where we know what we can delete and what not. It is true that RESET MASTER failure is quite serious, and if we cannot open a new binlog after deleting the old ones we are indeed in trouble. And there are very likely legacy bugs lurking in this error case. But with this code, at least no new error handling issues are introduced, and the rest of the code is clean and can rely on having at least one entry in the list. > If some engine confirms that everything below a certain cookie of > a particular binlog_id is flushed. Does it automatically mean that > all previous binlog_id's are flushed too in this engine? Yes. This is the relevant comment in handler.h: The TC will periodically (eg. once per binlog rotation) call commit_checkpoint_request(). When this happens, the engine must arrange for all transaction that have completed commit_ordered() to be durably flushed to disk (this does not include transactions that might be in the middle of executing commit_ordered()). When such flush has completed, the engine must call commit_checkpoint_notify_ha(), passing back the opaque "cookie". commit_ordered() calls complete in the same order as transactions are written in the binlog. And we request a checkpoint for a binlog file only when all transactions in it have completed commit_ordered(). So when an engine has flushed all transactions that completed commit_ordered(), this includes everything in the given binlog, as well as anything in previous binlogs. >> @@ -5493,7 +5656,27 @@ int MYSQL_BIN_LOG::rotate(bool force_rot >> if (!write_incident_already_locked(current_thd)) >> flush_and_sync(0); >> >> - *check_purge= true; >> + /* >> + We failed to rotate - so we have to decrement the xid_count back that >> + we incremented before attempting the rotate. >> + */ > > why you don't use mark_xid_done() here? We cannot call mark_xid_done() here, as we are holding LOCK_log - and mark_xid_done() also takes (sometimes) LOCK_log. >> @@ -5920,22 +6115,29 @@ MYSQL_BIN_LOG::write_binlog_checkpoint_e >> */ >> if (!ev.write(&log_file) && !flush_and_sync(0)) >> { >> - bool check_purge= false; >> signal_update(); >> - rotate(false, &check_purge); >> - if (check_purge) >> - purge(); > > Why did you remove rotate/purge from here? Binlog rotation triggers the writing of checkpoint event in the first place. So it does not seem right that writing checkpoint event can trigger binlog rotation. This could in principle lead to infinite rotation of new binlogs, though one would need a _very_ small max binlog size to trigger that. >> +int TC_LOG_MMAP::unlog(ulong cookie, my_xid xid) >> +{ >> + pending_cookies *full_buffer= NULL; >> + DBUG_ASSERT(*(my_xid *)(data+cookie) == xid); >> + >> + /* >> + Do not delete the entry immediately, as there may be participating storage >> + engines which implement commit_checkpoint_request(), and thus have not yet >> + flushed the commit durably to disk. >> + >> + Instead put it in a queue - and periodically, we will request a checkpoint >> + from all engines and delete a whole batch at once. >> + */ >> + mysql_mutex_lock(&LOCK_pending_checkpoint); >> + if (pending_checkpoint == NULL) >> + { >> + uint32 size= sizeof(*pending_checkpoint); >> + if (!(pending_checkpoint= >> + (pending_cookies *)my_malloc(size, MYF(MY_ZEROFILL)))) >> + { >> + my_error(ER_OUTOFMEMORY, MYF(0), size); >> + mysql_mutex_unlock(&LOCK_pending_checkpoint); >> + return 1; >> + } >> + pending_checkpoint->cookies[pending_checkpoint->count++]= cookie; > > Hm, should this be under if (pending_checkpoint == NULL) ? No :-( That seems completely, utterly broken :-( So this means that I must have not tested this code at all. Which makes me sad. Of course, with PBXT dead, I am not sure how it is even possible to test it... ? Thanks for spotting. >> + /* >> + We do an extra increment and notify here - this ensures that >> + things work also if there are no engines at all that support >> + commit_checkpoint_request. >> + */ >> + ++full_buffer->pending_count; >> + ha_commit_checkpoint_request(full_buffer, mmap_do_checkpoint_callback); >> + commit_checkpoint_notify(full_buffer); > > I'm sorry, I don't understand how you handle the case when some engines don't > support commit_checkpoint_request If an engine supports commit_checkpoint_request, then ha_commit_checkpoint_request() will ask the engine to do a checkpoint; and pending_count will be incremented for each such engine to ensure that we wait for each engine to finish the checkpoint before we delete any entries from the on-disk log. If an engine does not support commit_checkpoint_request, then we will not do anything with that engine before deleting entries. But such engines are required to make their transactions durable already in their commit() method, so deletion is always safe. If _no_ engines support ha_commit_checkpoint_request then we would not get any callback to commit_checkpoint_notify() - so we add one extra such a call ourselves, at the end. Also, the increment of pending_count and decrement in the extra commit_checkpoint_notify() call protects against this race: - We call engine A's commit_checkpoint_request() - Before we have time to call engine B's commit_checkpoint_request(), engine A does commit_checkpoint_notify(), so pending_count drops to zero and we delete entries even though B has not had a chance to do a checkpoint yet - We call engine B's commit_checkpoint_request with a void *cookie pointing to freed memory. Hope this clarifies things. >> @@ -7869,6 +8196,11 @@ int TC_LOG_BINLOG::recover(LOG_INFO *lin >> if (!binlog_checkpoint_found) >> break; >> first_round= false; >> + DBUG_EXECUTE_IF("xa_recover_expect_master_bin_000004", >> + if (0 != strcmp("./master-bin.000004", binlog_checkpoint_name) && >> + 0 != strcmp(".\\master-bin.000004", binlog_checkpoint_name)) > > Are you sure the name is always lowercase even on case-insensitive > filesystems? I'd use strcasecmp to avoid the need to think about it :) > Something like > > if (0 != strcasecmp("master-bin.000004", binlog_checkpoint_name+2)) > > (right, unsafe if strlen(binlog_checkpoint_name) < 2, but it's just a test case) I don't understand - this has nothing to do with file system case sensitivity. The string is generated inside log.cc based on what mysql-test-run.pl passes to mysqld for the --log-bin option, I don't see how this could change case. And I thought it was clearer to explicitly show the two possible cases for the full file name (otherwise the offset 2 in the comparison is completely magic). But as you said, this is just a test case, I'm happy to change it if you insists (I just did not understand the reason to change it). BTW, as I remember this was for something that was quite hard to test in mysql-test-run. Since recovery happens early in the server startup, before anything mysqltest can do. So I could not think of anything better than to inject an error insert with a configuration option. - Kristian.

4 9

[Maria-developers] MariaDB 5.5.27 Release Prep
by Daniel Bartholomew 07 Sep '12

07 Sep '12

All, I've prepped the release for MariaDB 5.5.27. We're now waiting for the mirrors to seed the files before turning on the downloads page and sending out announcements. Missing at the moment are our Windows builds because of an issue with the Windows build server. Hopefully those will be ready soon. Draft Release Notes and Changelog are located here: http://kb.askmonty.org/en/mariadb-5527-release-notes/ http://kb.askmonty.org/en/mariadb-5527-changelog/ If there is anything of particular note which should be mentioned in the release notes, please let me know. Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

[Maria-developers] EXPLAIN ANALYZE
by Sergei Petrunia 03 Sep '12

03 Sep '12

Hello Igor, I've got EXPLAIN ANALYZE to work for the first queries, see details at http://kb.askmonty.org/en/explain-analyze/ I need input re whether the shown information is what you needed for 10.0 works. Also, we need to decide what should be shown in these columns for join buffering/subqueries (let's discuss this on the optimizer call tomorrow). BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MDEV-406: EXPLAIN ANALYZE or a different syntax?
by sergii＠pisem.net 03 Sep '12

03 Sep '12

Hi. MDEV-406. The chosen syntax is "EXPLAIN ANALYZE". Pro: 1. PostgreSQL uses this syntax. Contra: 1. Full name is EXPLAIN ANALYZE SELECT. But statements are at most two words in MySQL: SHOW PROCESSLIST, CREATE TABLE, etc. Even the task description talks about EXPLAIN ANALYZE. Having EXPLAIN SELECT and (in the future) EXPLAIN UPDATE and EXPLAIN DELETE, EXPLAIN ANALYZE looks like it's EXPLAIN ANALYZE TABLE. But it is not. 2. In SQL the stament is a verb. That's the spirit of the standard. SELECT something. INSERT something. CREATE something, SHOW... We've deviated from it with the HANDLER command, unfortunately. Still, EXPLAIN ANALYZE SELECT is very unnatural English. EXPLAIN SELECT is better, because "SELECT" is perceived as a noun, as "SELECT statement". So EXPLAIN SELECT means "explain the SELECT statement". But EXPLAIN ANALYZE SELECT just doesn't map into English at all. Possible variants (keep in mind that instead of SELECT one might later have UPDATE and DELETE): EXPLAIN ANALYZED SELECT ... ANALYZE SELECT .... The last syntax is stragely consistent with EXPLAIN: We used to have both EXPLAIN table_name and EXPLAIN SELECT ... So, extending ANALYZE TABLE table_name to ANALYZE SELECT ... follows this weird historical pattern. (which is, of course, no argument, just a funny coincidence) Regards, Sergei

1 0

[Maria-developers] Please test: Cassandra batched-inserts, Batched Key Access
by Sergei Petrunia 31 Aug '12

31 Aug '12

Hi Elena, I've pushed "initial" support for batched INSERT and BKA into 5.5-cassandra, and updated http://kb.askmonty.org/en/cassandra-storage-engine to cover batching. Could you please test it? Cassandra uses "hashed" mode with BKA (see docs), which haven't been used for a while (previous user was Falcon storage engine), so it is possible that something malfunctions in BKA. Names/meaning of control variables and status counter names are preliminary and may change in the future. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] [Commits] Rev 3565: MDEV-492: fixed incorrect error check. in file:///home/bell/maria/bzr/work-maria-5.3-MDEV-492/
by Sergei Golubchik 29 Aug '12

29 Aug '12

Hi, Oleksandr! On Aug 29, Oleksandr Byelkin wrote: > >> === modified file 'sql/sql_update.cc' > >> --- a/sql/sql_update.cc 2012-04-26 17:21:37 +0000 > >> +++ b/sql/sql_update.cc 2012-08-27 14:29:26 +0000 > >> @@ -872,7 +872,7 @@ int mysql_update(THD *thd, > >> id= thd->arg_of_last_insert_id_function ? > >> thd->first_successful_insert_id_in_prev_stmt : 0; > >> > >> - if (error < 0) > >> + if (error < 0 && (!thd->is_error())) > >> { > >> char buff[MYSQL_ERRMSG_SIZE]; > >> my_snprintf(buff, sizeof(buff), ER(ER_UPDATE_INFO), (ulong) found, (ulong) updated, > > Nope, this is not good. > > The bug happens because the error from remove_eq_conds() is ignored. > > You need to catch and process this error as soon as possible. > > While in your patch you let the update work all the way till the end. > > > When "update works all the way" it catched inside update loops, but was > that if update never make any update it ignores errors. > > So actually it should be my fix or both fixes, because errors > (especially such unexpected as "out of resources" could happened almost > anywhere, so issuing OK to the client without checking error state is a > mistake. I know. Before sending a review I actually tried to get a crash by generating an error in different places of the UPDATE. E.g. in the loop: INSERT t1 VALUES (1,2); UPDATE .... or in fill_record(): UPDATE t1 SET a=COLUMN_CREATE(...); or with double my_error(): ... WHERE COLUMN_CREATE(...) OR COLUMN_CREATE(...); all these cases were correctly taken care of, no crash, no ok packet. So, I think, it's enough to fix only that early failure that you have found. Regards, Sergei

1 0