- developers - lists.mariadb.org

Re: [Maria-developers] MariaDB and jemalloc
by Axel Schwenke 24 Apr '13

24 Apr '13

Hi Sanja, Oleksandr Byelkin wrote: > 24.04.2013 19:00, Axel Schwenke пишет: > [skip] >> >> OLTP ro transactions per second relative to glibc malloc >> ---------------------------------------------------------------------- >> Threads jemalloc-3.3.1 glibc malloc tcmalloc jemalloc-3.1.0 >> 4 +3.2% +0.0% +11.6% +61.9% >> 8 +0.9% +0.0% -2.3% +8.0% >> 16 +9.6% +0.0% +4.7% +12.9% >> 32 +1.8% +0.0% +0.9% +5.8% >> 64 -5.1% +0.0% +2.1% +5.4% >> 128 -4.8% +0.0% +0.2% +4.5% >> 256 -4.2% +0.0% +0.7% +5.0% >> 512 -13.3% +0.0% +1.8% +4.1% > > Strange that results jumps up and down when percona had quite smooth > one. Is it possible that computer was doing something else? or we need > more tests to take average? It's not so "jumpy" when you look at the absolute numbers. The difference to Percona is, that they tested their own server, where MariaDB tries to minimize the number of malloc() calls. I wouldn't give too much on the numbers for 4 threads, since this is the buffer pool warmup phase. Also I've run each test for only 100 seconds to get results fast. Re "jumpy" - see the attached pdf with the scatter plot. Each point represents the tps from a 5 second interval. Numbers are in fact rather smooth, except for 512 threads. The main takeaway is, that I can reproduce the regression in jemalloc 3.3. With TokuDB there is no way around jemalloc anyway. XL

2 1

[Maria-developers] MariaDB and jemalloc
by Axel Schwenke 24 Apr '13

24 Apr '13

Hi, Monty asked me to do a quick check how MariaDB performs with jemalloc. This is in the light of the upcoming integration of TokuDB into the MariaDB tree. TokuDB requires jemalloc. While I did use tcmalloc in the connect speed benchmarks with good results, I did no use jemalloc before. This article from Percona http://www.mysqlperformanceblog.com/2013/03/08/mysql-performance-impact-of-… suggests that jemalloc and tcmalloc perform very comparable, but also that jemalloc shows a regression in 3.2.0 and later. The benchmark I used is sysbench OLTP ro, 16 tables, data fitting into buffer pool. I used the new 64 (virtual, 32 real) core machine and compared 4 different malloc implementations as below: OLTP ro transactions per second relative to glibc malloc ---------------------------------------------------------------------- Threads jemalloc-3.3.1 glibc malloc tcmalloc jemalloc-3.1.0 4 +3.2% +0.0% +11.6% +61.9% 8 +0.9% +0.0% -2.3% +8.0% 16 +9.6% +0.0% +4.7% +12.9% 32 +1.8% +0.0% +0.9% +5.8% 64 -5.1% +0.0% +2.1% +5.4% 128 -4.8% +0.0% +0.2% +4.5% 256 -4.2% +0.0% +0.7% +5.0% 512 -13.3% +0.0% +1.8% +4.1% (more data in attached spread sheet) Observations: - the effect is rather limited - jemalloc 3.1.0 clearly wins - jemalloc 3.3.1 performs bad at higher concurrency Conclusion It's fine to build MariaDB with jemalloc in the future. However we mustn't use jemalloc-3.3.1, but rather jemalloc-3.1.0. XL

1 0

[Maria-developers] Fwd: Fwd: gsoc13
by Samarth Mathur 24 Apr '13

24 Apr '13

Hi! I have a few doubts. Going through the internet, I've come across two major implementations of LDAP protocol The openLDAP, and the libpam_ldap library. Both have a detailed information of client side implementation. However, openLDAP gives the slapd which is a standalone server daemon, whereas from the libpam_ldap library, I am able to gather the existence of an inbuilt ldap authentication system. Am I correct about that? Which do you think will be useful to implement as a plugin for mariadb? Thanks!

2 2

[Maria-developers] log analyzer
by Roberto Spadim 23 Apr '13

23 Apr '13

Hi guys, I was looking for something that give me the 'prepared statement' of a SQL query for example: statment = SELECT * FROM table WHERE field="value" prepared statement = SELECT * FROM table WHERE field=? why do this? well my idea is build a log analyzer and get commong queries since some programs don't use native prepared statement, reading the slow query log, or query log only will give me many different queries what i want is: 1)read queries log 2)get queries 3)get the 'prepared statement' of the query 4)create a table with prepared statement / unique id (maybe md5 of (database name + prepared statement) ) 5)create statistics using the unique id of prepared statement and check if the same 'kind' of queries are running with a mean time, or have some low and high times (for example) or getting information if some index is being used or not, well here i have many ideas but it's not the 'problem' for now the problem is the item (3), i have a query and i need the 'prepared statement' of this query, even the query was not executed with prepared statements this should work for INSERT, DELETE, UPDATE, SELECT, CREATE TABLE and others queries too (for now SELECT is enought, but in future others kinds of query will be nice) anyone know something that could do this job? maybe part of the mariadb query parser could/should be used? i was thinking about a command that could be executed in mariadb, today we have the EXPLAIN, we could have PARSE or other name? simplifying the problem: read the query, parse it and change constant values with "?" willcard thanks! any help is welcome -- Roberto Spadim

1 2

[Maria-developers] MariaDB 10.0.2-alpha release prep
by Daniel Bartholomew 23 Apr '13

23 Apr '13

I've begun prepping MariaDB 10.0.2-alpha for release. Draft changelog and release notes are here: - https://kb.askmonty.org/en/mariadb-1002-release-notes/ - https://kb.askmonty.org/en/mariadb-1002-changelog/ In the draft release notes I've listed MDEV-26 (Global Transaction ID), and the CONNECT handler. Please let me know of other features or fixes I should highlight ASAP. Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

2 1

[Maria-developers] ALTER TABLE xxx ENGINE=aria, progress only works in copying stage
by Roberto Spadim 22 Apr '13

22 Apr '13

hi guy, i was changing myisam to aria on some big tables (15gb) and show processlist stop reset the progress at 16% the operation started with 'copying to tmp table', and i saw 1%, 2% ... 16% now the table is doing "repair by sorting" and progress is 0% the point is... could the processlist show, at least, the last progress percentage? without get back to 0%? or maybe a column of current operation progress and another of total process operation? does 'repair by sorting' have progress algoritm implemented? i'm running mariadb 10.0.1 on linux 64bits thanks guys -- Roberto Spadim

1 0

[Maria-developers] performance monitoring
by MARK CALLAGHAN 21 Apr '13

21 Apr '13

After reading this I wonder if there is an opportunity for SkySQL and Percona to offer "performance tuning for performance schema" classes. I don't think performance monitoring should be this hard. What are the plans for MariaDB moving forward -- more user_stats/table_stats/index_stats and maybe statement_stats or integrate with the PS? http://dev.mysql.com/doc/refman/5.6/en/performance-schema.html http://marcalff.blogspot.com/2013/04/on-configuring-performance-schema.html -- Mark Callaghan mdcallag(a)gmail.com

2 1

Re: [Maria-developers] Fwd: gsoc13
by Sergei Golubchik 21 Apr '13

21 Apr '13

Hi, Samarth! On Apr 20, Samarth Mathur wrote: > "See also the authentication plugin examples in plugin/auth/." > > I couldn't locate this folder in the source code (mariadb 10). > Does it not exist there? No, but there are mariadb-10.0 $ ls -dl plugin/auth* drwx------ 3 serg serg 4096 Apr 20 10:13 plugin/auth_dialog drwx------ 3 serg serg 4096 Apr 20 10:13 plugin/auth_examples drwx------ 5 serg serg 4096 Apr 20 10:13 plugin/auth_pam drwx------ 3 serg serg 4096 Apr 20 10:13 plugin/auth_socket make your choice. Regards, Sergei

1 0

[Maria-developers] Aria engine integrity
by Roberto Spadim 21 Apr '13

21 Apr '13

Hi guys, another doubt about mariadb, now about aria... i'm using aria since myisam is a bit unsecure and have a slow recovery/repair process, aria storage is more secure my doubt is... could i use aria engine over a ext2 file system and have a 'security' near a jornaled file system (xfs for example) + myisam? or maybe xfs+aria? what's better? just one repair, from aria engine, or two repairs, one from file system and other from aria? thanks guys -- Roberto Spadim

3 3

[Maria-developers] User question
by Roberto Spadim 20 Apr '13

20 Apr '13

hi guys, i read about multi master replication but didn't found examples there's a wiki or an example of how to setup 2 masters and 1 slave replication? using different databases, like: create database db1; (on slave) create database db2; (on slave) master1 will replicate on db1 and master2 on db2 Thanks! -- Roberto Spadim

3 8

[Maria-developers] Enquire
by MAYANK PANDITA 19 Apr '13

19 Apr '13

I am pursuing B.E. C.S.E. I have experience in C, C++ and SQL. I am keen to work on Regex Enhancements and Roles project. I need to know more about these projects. How can I contact the mentors. -- Mayank Pandita

2 1

[Maria-developers] Fwd: LDAP authentication plugin: GSOC Application
by Ionel Merca 19 Apr '13

19 Apr '13

---------- Forwarded message ---------- From: Ionel Merca <mercaexilonx(a)gmail.com> Date: 2013/4/18 Subject: Re: [Maria-developers] LDAP authentication plugin: GSOC Application To: Sergei Golubchik <serg(a)askmonty.org> Hi, Thank you for the response! I have downloaded the source code from https://downloads.mariadb.org/ the 10.0 Series. I followed these steps : I installed cmake and run cmake . , after that I run make and make install... everything worked ok without any errors, but I don't know how can I run the application. I hope this is not a very stupid question but make run is not working or something like that... Regards, Merca 2013/4/18 Sergei Golubchik <serg(a)askmonty.org> > Hi, Ionel! > > On Apr 18, Ionel Merca wrote: > > Hi, > > > > My name is Merca Ionel and I am a student at Politehnica University of > > Bucharest studying Computer Science in the third year. I am interested in > > the project with ldap authentication. I would like to know how can I > > download the source code and what relevant bugs can I fix to prove that I > > am capable of finishing the job. > > MariaDB is Open Source project, you can download the source code easily. > > No need to fix any bugs, just write a good proposal. Don't forget to > show that you understand how LDAP works, and cover how your LDAP > authentication plugin will be different from using MariaDB PAM > authentication plugin together with pam_ldap.so. > > Regards, > Sergei >

2 1

[Maria-developers] [Gsoc] Regex enhancements Project
by Sudheera Palihakkara 19 Apr '13

19 Apr '13

Hi, I'm Sudheera Palihakkara, an undergraduate of Department of Computer Science and Engineering of University of Moratuwa. I'm interested in "Regex enhancements Project" and I need more information to get a clear picture about the project. Can I know what is the regex implementation used in mariadb is it the Henry Spencer's implementation or any newer implementation? And I want to know that is there any developer documentation on this classes present? I did a bit of search but unable to find. thank you. -- *Sudheera Palihakkara.* Undergraduate Department of *Computer Science and Engineering, *Faculty of Engineering, *University of Moratuwa*, Sri Lanka.

2 1

[Maria-developers] Fwd: gsoc13
by Samarth Mathur 19 Apr '13

19 Apr '13

Hi, I'm interested in the LDAP authentication plugin. I read the earlier correspondence on this particular topic, so as I understand I need to get well acquainted with the LDAP protocol. Do you have more tips or recommendations as requisites for the same? Best Regards looking forward to a reply Sam

2 1

[Maria-developers] GSoC 2013
by Shivam Agarwal 19 Apr '13

19 Apr '13

Hello please kindly provide me the basic requirements of programming languages needed to be participate in your project at GSoC 2013.

2 1

[Maria-developers] GSoC:Regex Project
by Tamás Kövesdan 19 Apr '13

19 Apr '13

Hello, I'm interested in this project. I have read the other thread about this. I've checked the available regex implementations and I've written a draft application. Is there any potential mentor who would review my application and help to improve it? Thanks in advance. Best whishes, Tamás Kövesdán

3 3

[Maria-developers] Fedora test day for replacing MySQL with MariaDB
by Colin Charles 19 Apr '13

19 Apr '13

Hi all, By now you must know of the movement to replace MySQL with MariaDB in Fedora-land (http://fedoraproject.org/wiki/Features/ReplaceMySQLwithMariaDB) One important thing is to ensure that MariaDB is a drop-in replacement. Fedora is a fast-moving high quality Linux distribution, so their focus aims are to ensure that MariaDB and MySQL are compatible, data can be migrated easily and there are minimal/no upgrade issues (and if so, they need to be documented) This is why on 30 April 2013, there is a Fedora test day dedicated to this very important task: http://fedoraproject.org/wiki/Test_Day:2013-04-30_MariaDB I plan to be on IRC for as long as humanly possible on #fedora-test-day (freenode) and will be watching the process closely. If you would like to participate, this is naturally welcome to help new MariaDB users Thanks! cheers, -colin -- Colin Charles, http://bytebot.net/blog/ | twitter: @bytebot | skype: colincharles MariaDB: Community developed. Feature enhanced. Backward compatible. Download it at: http://www.mariadb.org/ Open MariaDB/MySQL documentation at the Knowledgebase: http://kb.askmonty.org/

1 0

Re: [Maria-developers] GTID and replication of mysql database
by Kristian Nielsen 18 Apr '13

18 Apr '13

Pavel Ivanov <pivanof(a)google.com> writes: > Could you please tell me, in your implementation of GTIDs did you make > something special for the case when mysql database is replicated? I > guess mysql.rpl_slave_state cannot be replicated in any circumstances, > so did you put any safeguards to not replicate it or it's totally up > to MariaDB users? In rpl_slave_state::record_gtid(), there is this to prevent replicating updates of the mysql.rpl_slave_state table: table->no_replicate= 1; Other than that, I do not know of any special handling needed for the mysql database. Did you have something special in mind? Note that the user should not update mysql.rpl_slave_state manually. Except perhaps to ALTER TABLE ... ENGINE=xxx (which can be replicated). To change the GTID position, use SET GLOBAL gtid_pos='0-1-10', which will update mysql.rpl_slave_state as needed. BTW, yesterday I merged GTID into lp:maria/10.0, so now the code is in the main tree and will be included in the next 10.0.2 alpha release. - Kristian.

2 4

[Maria-developers] GSoC 2013 -- the Kerberos authentication plugin project
by QIU Shuang 18 Apr '13

18 Apr '13

Hi community, This is Shuang Qiu, a student from Shanghai, China, major in Software Engineering. My field of interests is distributed storage (also a little knowledge of information retrieval). I've focused on the MariaDB project for a while and want to make this GSoC a first step into this community :). Based on my experience, I think the Kerberos plugin project meets more my taste (also the full text search one, unfortunately, which has been crossed out from the slate). Although not a very tough task at first glance, but I believe there must be some tricky things behind the scene. I've skip the pointer on MariaDB GSoC2013 home page and have some point unclear. How will we configure the Kerberos server? Do we need some name resolve protocol to discover the Kerberos service? Thank you very much and best regards! Sincerely, Shuang

2 2

[Maria-developers] LDAP authentication plugin: GSOC Application
by Ionel Merca 18 Apr '13

18 Apr '13

Hi, My name is Merca Ionel and I am a student at Politehnica University of Bucharest studying Computer Science in the third year. I am interested in the project with ldap authentication. I would like to know how can I download the source code and what relevant bugs can I fix to prove that I am capable of finishing the job. Thanks in advance, Merca

2 1

[Maria-developers] GSoC 2013
by Dammina Sahabandu 17 Apr '13

17 Apr '13

Hi All, I'm Dammina Sahabandu, 3rd year undergraduate of University of Moratuwa(dept. of Computer of Science and Engineering) which has consecutively won GSOC’s most number of participant’s award from 2009 to 2012. I'm willing to contribute for a MariaDB project in this years GSoC. I went through the project ideas page and found the project "Self-Tuning Optimizer" interesting. So can you give me an insight of the project. -regards Dammina

2 1

[Maria-developers] MDEV-260 auditing table accesses
by Sergei Golubchik 17 Apr '13

17 Apr '13

Hi, Monty, Here's a patch for MDEV-260 (auditing table accesses). Please review and see whether it's safe for 5.5. Regards, Sergei

1 0

[Maria-developers] Bug in MariaDB 5.5.30 mysql_secure_installation: find_mysql_client command not found
by Viljo Hakala 16 Apr '13

16 Apr '13

When I rn mysql_secure_installation I get a warning: "379: find_mysql_client: command not found" On line 379 on mysql_secure_installation find:mysql_client command/function is called, but there is no such command and no such definition of command in MariaDB 5.5.30. It doesn't cause a problem -- the script runs successfully, but it is just a small annoying message. Can this be fixed?

2 2

Re: [Maria-developers] incomplete index stats in MariaDB
by Axel Schwenke 16 Apr '13

16 Apr '13

Igor Babaev wrote: > On 04/12/2013 09:00 AM, Axel Schwenke wrote: >> Hi Igor, Timour, >> >> when I wrote the script to translate MariaDB table statistics to MySQL, >> I found one problem. Imagine this simple table: >> >> create table t1 (c1 int, c2 int, c3 int, >> primary key (c1, c2), >> index `secondary`(c3)); >> >> Here the secondary index has 3 parts, it can be used as (c3), (c3, c1) >> and (c3, c1, c2). Obviously the avg_frequency is 1 for (c3, c1, c2), but >> it is not obvious how it is for (c3, c1). Such data I'd expect to see in >> the index_stats table under prefix_arity=2, but it's not there. > > To get this you have to run analyze with > set optimizer_switch='extended_keys=on'; Thanks Igor, this is a piece I was missing. While I see the point to obey the extended_keys optimizer switch in ANALYZE TABLE, I wonder if this is a good idea. The optimizer switch is dynamic, so a user can change it, even for a single query. But when the statistics for the extended key is not available, this makes not much sense. I suggest to always collect extended keys statistics, regardless of the setting of the extended_keys optimizer switch. Rationale: it is easy to simply not use the data when extended_keys=off. The only counter argument would be that this is costly - but I cannot see this. XL

1 0

Re: [Maria-developers] I would like to run a buildbot
by Kristian Nielsen 16 Apr '13

16 Apr '13

Dan Demeter <dan(a)apexis.ro> writes: > I would be happy to compile and install the latest CLANG if you > want. Currently I am compiling with > gcc version 4.4.3 ( the stock one from Ubuntu repos ). I think the currently configured builder (dan_demeter1) is fine for now. For the other one, it sounds like a good idea to install CLANG. > Actually, I managed to install Build Slave on a VM and it connected > and it's running. Yes, it seems to be working fine, great! I am looking into the test failures, and I may need help with tracking down one of them. We have a test that causes the server to crash like this: 130414 13:57:41 [ERROR] mysqld got signal 6 ; stack_bottom = 0x0 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xb2ef89] sql/signal_handler.cc:153(handle_fatal_signal)[0x71cb1b] ??:0(??)[0x7f82edc528f0] ??:0(??)[0x7f82ec98fa83] sql/mysqld.cc:5556(handle_connections_sockets())[0x51ca2e] sql/mysqld.cc:5007(mysqld_main(int, char**))[0x52359f] ??:0(??)[0x7f82ec8d1c4d] ??:0(_start)[0x516c4d] Writing a core file As you see, there is not much information to go by. Here are some suggestions you can try on the dan_demeter1 host to get more information available: - Install gdb (if not there already). mysql-test-run can use gdb to obtain better stack traces from core files. - Enable core files in current directory (if not enabled already). It looks like mysql-test-run does not find any core file, even though the log says that one is written. Maybe there is some default ubuntu security configuration or something that disables core files, or puts them in a different location? Not 100% sure, but something to check... - Maybe you can install debug symbols for libc (apt-get install libc6-dbg) to attempt to get better information in the stack trace (if not there already). - Maybe Serg (Cc:'ed) has an idea about what to install to get better stack traces than the above. Thanks, - Kristian.

3 10

[Maria-developers] GSoC 2013: Regex enhancements
by Gábor Kövesdán 14 Apr '13

14 Apr '13

Hi, I'm a student interested in this GSoC project. I already participated in GSoC several times, one of those involved porting the TRE regex engine to FreeBSD. I also worked on some optimizations for that. Since then I've started the PhD course in modeling and domain-specific languages, which made me more familiar with the theory of parsing and automata. I'm experienced in C and I can get along with C++. I'd like to ask what are the exact requirements of this project? Is it about plugging in an existing regexp engine into MariaDB, like RE2 or TRE? Or writing a new implementation? Could you please give me some pointers where to look at the code to see how regexes are used in MariaDB? Also, what other preliminary research should I do to write a good application? Thanks in advance, Gábor

2 1

[Maria-developers] GSOC2013
by Ajaya Agrawal 11 Apr '13

11 Apr '13

Hi All, I am Ajaya and I am interested in participating in gsoc2013. I am interested in gsoc project LDAP authentication plugin and memcached InnoDB interface. Although I have not worked with mariadb before I have submitted couple of working bug-fixes for drizzle. I would like to know which of these two is more important to mariadb? I would like to implement this https://mariadb.atlassian.net/browse/MDEV-4166?jql=project%20in%20(CONC%2C%… I believe I have sufficient theoretical knowledge about the problem but would appreciate if someone can point me to right direction in the codebase. I have compiled and built mariadb on my ubuntu machine. Thank You, Ajaya

2 1

[Maria-developers] We're accepted into Google Summer of Code 2013
by Colin Charles 10 Apr '13

10 Apr '13

Subject says it all We need people to sign up as mentors (http://www.google-melange.com/gsoc/homepage/google/gsoc2013) against the MariaDB project Students start opening up soon If you know of projects a student can complete in 3 months over a summer, this is the ideal place to list the task: https://kb.askmonty.org/en/google-summer-of-code-2013/ This is the first time MariaDB is going to participate. Some of you have done this in the past against MySQL or Drizzle. Big win for MariaDB this year! cheers, -colin -- Colin Charles, http://bytebot.net/blog/ | twitter: @bytebot | skype: colincharles MariaDB: Community developed. Feature enhanced. Backward compatible. Download it at: http://www.mariadb.org/ Open MariaDB/MySQL documentation at the Knowledgebase: http://kb.askmonty.org/

3 2

Re: [Maria-developers] [Commits] Rev 3197: MDEV-4326 fix.
by Sergei Golubchik 09 Apr '13

09 Apr '13

Hi, Oleksandr! On Apr 09, Oleksandr Byelkin wrote: > 09.04.2013 13:38, Sergei Golubchik пишет: > >> === modified file 'sql/sql_prepare.cc' > >> --- a/sql/sql_prepare.cc 2012-11-06 09:52:55 +0000 > >> +++ b/sql/sql_prepare.cc 2013-04-09 06:58:51 +0000 > >> @@ -787,7 +787,7 @@ static bool insert_params_with_log(Prepa > >> if (param->state == Item_param::NO_VALUE) > >> DBUG_RETURN(1); > >> > >> - if (param->limit_clause_param && param->item_type != Item::INT_ITEM) > >> + if (param->limit_clause_param) > > This looks wrong. Why do you need to convert INT_ITEM to INT_ITEM ? > Because it is not INT Item, it is Item with INT cache (which could be > left from previous run). When we assign value from variable we have > exactly the same code (see item.cc). I know that in your bug it's not an INT_ITEM. May be this exactly is a problem? When the param->item_type is not updated correctly for string parameters? > >> === modified file 'tests/mysql_client_test.c' > >> --- a/tests/mysql_client_test.c 2013-04-06 19:29:12 +0000 > >> +++ b/tests/mysql_client_test.c 2013-04-09 06:58:51 +0000 > >> @@ -16750,7 +16750,11 @@ static void test_bug43560(void) > >> fprintf(stdout, "Skipping test_bug43560: server not DEBUG version\n"); > >> DBUG_VOID_RETURN; > >> } > >> - > >> + if (opt_unix_socket) > >> + { > >> + fprintf(stdout, "Skipping test_bug43560: connected via UNIX socket\n"); > >> + DBUG_VOID_RETURN; > >> + } > > Why is that? There's nothing in the bug depends on how the client is > > connected. > they require TCP connection (see following comment). I am not sure that > it is ok to change connection type in the test, so I make it skipped on > unix socket (otherwise it just fails on connection). Ah, it's not your bug. Okay. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3197: MDEV-4326 fix.
by Sergei Golubchik 09 Apr '13

09 Apr '13

Hi, Sanja! On Apr 09, sanja(a)askmonty.org wrote: > revno: 3197 > revision-id: sanja(a)askmonty.org-20130409065851-bb241d1scc8lxo3p > parent: sergii(a)pisem.net-20130406192912-jtwuhe78zijfqfbx > committer: sanja(a)askmonty.org > branch nick: work-maria-5.1-MDEV-4326 > timestamp: Tue 2013-04-09 09:58:51 +0300 > message: > MDEV-4326 fix. > > Removed "optimization" which caused preoblems on second execution of > PS with string parameter in LIMIT clause. > > === modified file 'sql/sql_prepare.cc' > --- a/sql/sql_prepare.cc 2012-11-06 09:52:55 +0000 > +++ b/sql/sql_prepare.cc 2013-04-09 06:58:51 +0000 > @@ -787,7 +787,7 @@ static bool insert_params_with_log(Prepa > if (param->state == Item_param::NO_VALUE) > DBUG_RETURN(1); > > - if (param->limit_clause_param && param->item_type != Item::INT_ITEM) > + if (param->limit_clause_param) This looks wrong. Why do you need to convert INT_ITEM to INT_ITEM ? > { > param->set_int(param->val_int(), MY_INT64_NUM_DECIMAL_DIGITS); > param->item_type= Item::INT_ITEM; > > === modified file 'tests/mysql_client_test.c' > --- a/tests/mysql_client_test.c 2013-04-06 19:29:12 +0000 > +++ b/tests/mysql_client_test.c 2013-04-09 06:58:51 +0000 > @@ -16750,7 +16750,11 @@ static void test_bug43560(void) > fprintf(stdout, "Skipping test_bug43560: server not DEBUG version\n"); > DBUG_VOID_RETURN; > } > - > + if (opt_unix_socket) > + { > + fprintf(stdout, "Skipping test_bug43560: connected via UNIX socket\n"); > + DBUG_VOID_RETURN; > + } Why is that? There's nothing in the bug depends on how the client is connected. > /* > Set up a separate connection for this test to avoid messing up the > general MYSQL object used in other subtests. Use TCP protocol to avoid > @@ -17541,6 +17545,109 @@ static void test_bug13001491() > myquery(rc); > } > > +static void test_mdev4326() > +{ > + MYSQL_STMT *stmt; > + MYSQL_BIND bind; > + char query[]= "SELECT * FROM mysql.user LIMIT ?"; > + char str_data[]= "1"; > + unsigned long length= 0; > + int int_data= 1; > + int rc, count; > + my_bool is_null= 0; > + my_bool error= 0; > + myheader("test_mdev4326"); > + > + rc= mysql_change_user(mysql, opt_user, opt_password, "mysql"); > + myquery(rc); Sanja, I asked to reduce the test case and to remove whatever is not relevant to the bug. Why do you need to change user here? It's not needed for the bug to show up. > + > + rc= mysql_query(mysql, "SET GLOBAL general_log = 1"); > + myquery(rc); > + > + stmt= mysql_stmt_init(mysql); > + check_stmt(stmt); > + > + /* Numeric parameter test */ > + > + rc= mysql_stmt_prepare(stmt, query, strlen(query)); > + check_execute(stmt, rc); > + check_stmt(stmt); > + verify_param_count(stmt, 1); > + > + memset((char *)&bind, 0, sizeof(bind)); > + bind.buffer_type= MYSQL_TYPE_LONG; > + bind.buffer= (char *)&int_data; > + bind.is_null= &is_null; > + bind.length= &length; > + bind.error= &error; Why do you execute the statement with MYSQL_TYPE_LONG parameters? It's not needed for the bug to show up. > + > + rc= mysql_stmt_bind_param(stmt, &bind); > + check_execute(stmt, rc); > + rc= mysql_stmt_execute(stmt); > + check_execute(stmt, rc); > + count= 0; > + while (!(rc= mysql_stmt_fetch(stmt))) > + count++; > + DIE_UNLESS(count == 1); > + rc= mysql_stmt_execute(stmt); > + check_execute(stmt, rc); > + count= 0; > + while (!(rc= mysql_stmt_fetch(stmt))) > + count++; > + DIE_UNLESS(count == 1); > + int_data= 0; > + rc= mysql_stmt_execute(stmt); > + check_execute(stmt, rc); > + count= 0; > + while (!(rc= mysql_stmt_fetch(stmt))) > + count++; > + DIE_UNLESS(count == 0); > + rc= mysql_stmt_close(stmt); > + check_execute(stmt, rc); > + > + /* String parameter test */ > + > + stmt= mysql_stmt_init(mysql); > + rc= mysql_stmt_prepare(stmt, query, strlen(query)); > + check_execute(stmt, rc); > + check_stmt(stmt); > + verify_param_count(stmt, 1); > + > + memset((char *)&bind, 0, sizeof(bind)); > + bind.buffer_type= MYSQL_TYPE_STRING; > + bind.buffer= (char *)str_data; > + length= bind.buffer_length= sizeof(str_data); > + bind.is_null= &is_null; > + bind.length= &length; > + bind.error= &error; > + > + rc= mysql_stmt_bind_param(stmt, &bind); > + check_execute(stmt, rc); > + rc= mysql_stmt_execute(stmt); > + check_execute(stmt, rc); > + count= 0; > + while (!(rc= mysql_stmt_fetch(stmt))) > + count++; > + DIE_UNLESS(count == 1); > + rc= mysql_stmt_execute(stmt); > + check_execute(stmt, rc); > + count= 0; > + while (!(rc= mysql_stmt_fetch(stmt))) > + count++; > + DIE_UNLESS(count == 1); > + str_data[0]= '0'; > + rc= mysql_stmt_execute(stmt); > + check_execute(stmt, rc); > + count= 0; > + while (!(rc= mysql_stmt_fetch(stmt))) > + count++; > + DIE_UNLESS(count == 0); > + rc= mysql_stmt_close(stmt); > + check_execute(stmt, rc); > + > + rc= mysql_change_user(mysql, opt_user, opt_password, current_db); > + myquery(rc); Another useless mysql_change_user. > +} > > static struct my_tests_st my_tests[]= { > { "disable_query_logs", disable_query_logs }, > @@ -17790,6 +17897,7 @@ static struct my_tests_st my_tests[]= { > { "test_bug58036", test_bug58036 }, > { "test_bug56976", test_bug56976 }, > { "test_bug13001491", test_bug13001491 }, > + { "test_mdev4326", test_mdev4326 }, > { 0, 0 } > }; Regards, Sergei

1 0

[Maria-developers] Google Summer of Code 2013 ideas page
by Colin Charles 09 Apr '13

09 Apr '13

Hi all, Team MariaDB wants to participate in Google Summer of Code 2013. We definitely want to ensure that students have lots to do within those three summer months plus continue being MariaDB developers Please contribute to the ideas page: https://kb.askmonty.org/en/google-summer-of-code-2013/ Thanks -- Colin Charles, http://bytebot.net/blog/ | twitter: @bytebot | skype: colincharles MariaDB: Community developed. Feature enhanced. Backward compatible. Download it at: http://www.mariadb.org/ Open MariaDB/MySQL documentation at the Knowledgebase: http://kb.askmonty.org/

1 0

[Maria-developers] Review of last part of discover code
by Michael Widenius 08 Apr '13

08 Apr '13

Hi! Here is finally the review. >=== modified file 'sql/sql_truncate.cc' >--- sql/sql_truncate.cc 2013-01-25 19:40:42 +0000 >+++ sql/sql_truncate.cc 2013-02-27 11:23:42 +0000 >@@ -350,9 +342,15 @@ bool Truncate_statement::lock_table(THD > MYSQL_OPEN_SKIP_TEMPORARY)) > DBUG_RETURN(TRUE); > >- if (dd_check_storage_engine_flag(thd, table_ref->db, table_ref->table_name, >- HTON_CAN_RECREATE, hton_can_recreate)) >+ handlerton *hton; >+ if (!ha_table_exists(thd, table_ref->db, table_ref->table_name, &hton) || >+ !hton || hton == view_pseudo_hton) >+ { >+ my_error(ER_NO_SUCH_TABLE, MYF(0), table_ref->db, table_ref->table_name); > DBUG_RETURN(TRUE); >+ } >+ >+ *hton_can_recreate= hton->flags & HTON_CAN_RECREATE; > } ha_table_exists really needs a comment about what it can return and what the different values of hton means. >=== modified file 'sql/table.cc' >--- sql/table.cc 2013-01-29 14:10:47 +0000 >@@ -292,8 +286,8 @@ TABLE_CATEGORY get_table_category(const > # Share > */ > >-TABLE_SHARE *alloc_table_share(TABLE_LIST *table_list, char *key, >- uint key_length) >+TABLE_SHARE *alloc_table_share(const char *db, const char *table_name, >+ char *key, uint key_length) Just so you know, the reason that alloc_table_share took a table list as argument was that it would enable us: - We should be able to use LEX_STRING for db and table_name - Would be trivial to add schema if we wanted to later We need to at some point change to use at least LEX_STRING as this will make build_table_filename faster. <cut> >+ if (memcmp(head, STRING_WITH_LEN("TYPE=VIEW\n")) == 0) > { >- if (head[2] == FRM_VER || head[2] == FRM_VER+1 || >- (head[2] >= FRM_VER+3 && head[2] <= FRM_VER+4)) >- { >- /* Open view only */ >- if (db_flags & OPEN_VIEW_ONLY) >- { >- error_given= 1; >- goto err; >- } >- table_type= 1; >- } >- else >- { >- error= 6; // Unkown .frm version >- goto err; >- } In your new code, you accept head[2] == FRM_VER +2, which the above code doesn't do. Don't remember why this is skipped, but if we want to be compatible we should skip this one too... But I think it's safe to ignore this change. <cut> >+ if (my_fstat(file, &stats, MYF(0))) >+ goto err; The above I don't like. We already store the max possible length of the frm in frm_header+10. Why not store the exact length there and use this instead of doing fstat()? Would save us a file operation (at least for MariaDB files). It would also ensure that we don't easily break if we find a partial .frm file on disk. <cut> >+ share->init_from_binary_frm_image(thd, false, buf, stats.st_size); >+ error_given= true; Add a comment why error_given is set. (I know it's old code, but it looks confusing) <cut> >+int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, <cut> >+ while (extra2 < e2end) >+ { >+ uchar type= *extra2++; >+ size_t length= *extra2++; >+ if (!length) >+ { >+ length= uint2korr(extra2); >+ extra2+=2; >+ if (length < 256) >+ goto err; >+ } Add a test: if (extra2 + length > e2end) goto err; Please also check in other places that we don't run outside of the image. >+ switch (type) { >+ case EXTRA2_TABLEDEF_VERSION: >+ if (tabledef_version.str) // see init_from_sql_statement_string() >+ { >+ if (length != tabledef_version.length || >+ memcmp(extra2, tabledef_version.str, length)) >+ goto err; >+ } >+ else >+ { >+ uchar *buf= (uchar*) alloc_root(&mem_root, length); >+ if (!buf) >+ goto err; You could have set tabledef_version.str directly and not use buf. >+ memcpy(buf, extra2, length); >+ tabledef_version.str= buf; >+ tabledef_version.length= length; >+ } >+ break; >+ case EXTRA2_ENGINE_TABLEOPTS: >+ if (options) >+ goto err; >+ /* remember but delay parsing until we have read fields and keys */ >+ options= extra2; >+ options_len= length; >+ break; >+ default: >+ /* abort frm parsing if it's an unknown but important extra2 value */ >+ if (type >= 128) 128 should probably be a define. >+ goto err; >+ } >+ extra2+= length; >+ } >+ if (extra2 > e2end) >+ goto err; If you have my earlier test, you don't need this one. Better to do the test early... >+ } > >- mysql_file_seek(file,pos,MY_SEEK_SET,MYF(0)); >- if (mysql_file_read(file, forminfo,288,MYF(MY_NABP))) >+ if (!(pos= uint4korr(frm_image + 64 + len))) > goto err; >- share->frm_version= head[2]; >+ >+ forminfo= frm_image + pos; Check that forminfo + FRM_FORM_LENGTH is inside of image <cut> > /* Read keyinformation */ >- key_info_length= (uint) uint2korr(head+28); >- mysql_file_seek(file, (ulong) uint2korr(head+6), MY_SEEK_SET, MYF(0)); >- if (read_string(file,(uchar**) &disk_buff,key_info_length)) >- goto err; /* purecov: inspected */ >+ disk_buff= frm_image + uint2korr(frm_image+6); Check that disk_buff + key_info_length is within image. >- if (!(extra_segment_buff= (uchar*) my_malloc(n_length + 1, MYF(MY_WME)))) >- goto err; >- next_chunk= extra_segment_buff; >- if (mysql_file_pread(file, extra_segment_buff, >- n_length, record_offset + share->reclength, >- MYF(MY_NABP))) >- { >- goto err; >- } >+ next_chunk= frm_image + record_offset + share->reclength; >+ buff_end= next_chunk + n_length; Check that buff_end is is within image. <cut> >- if (mysql_file_pread(file, record, (size_t) share->reclength, >- record_offset, MYF(MY_NABP))) >- goto err; /* purecov: inspected */ >+ memcpy(record, frm_image + record_offset, share->reclength); Check that disk_buff + share->reclength is withing image > >- mysql_file_seek(file, pos+288, MY_SEEK_SET, MYF(0)); >-#ifdef HAVE_CRYPTED_FRM >- if (crypted) >- { >- crypted->decode((char*) forminfo+256,288-256); >- if (sint2korr(forminfo+284) != 0) // Should be 0 >- goto err; // Wrong password >- } >-#endif >+ disk_buff= frm_image + pos + 288; Check that disk_buff is withing image <cut> >@@ -1420,7 +1360,6 @@ static int open_binary_frm(THD *thd, TAB > geom_type= (Field::geometry_type) strpos[14]; > charset= &my_charset_bin; > #else >- error= 4; // unsupported field type > goto err; > #endif Shouldn't we get a better error for this case than just BAD_FRM ? >@@ -1578,10 +1523,8 @@ static int open_binary_frm(THD *thd, TAB > (TYPELIB*) 0), > share->fieldnames.type_names[i]); > if (!reg_field) // Not supported field type >- { >- error= 4; >- goto err; /* purecov: inspected */ >- } >+ goto err; >+ Shouldn't we get a better error for this case than just BAD_FRM ? >@@ -1607,20 +1550,8 @@ static int open_binary_frm(THD *thd, TAB > if (reg_field->unireg_check == Field::NEXT_NUMBER) > share->found_next_number_field= field_ptr; > >- if (use_hash) >- { >- if (my_hash_insert(&share->name_hash, >- (uchar*) field_ptr)) >- { >- /* >- Set return code 8 here to indicate that an error has >- occurred but that the error message already has been >- sent (OOM). >- */ >- error= 8; >- goto err; >- } >- } >+ if (use_hash && my_hash_insert(&share->name_hash, (uchar*) field_ptr)) >+ goto err; Is this change because my_hash_insert() will give an error for OEM? <cut> >@@ -1950,11 +1877,7 @@ static int open_binary_frm(THD *thd, TAB > share->default_values, reg_field, > &share->next_number_key_offset, > &share->next_number_keypart)) < 0) >- { >- /* Wrong field definition */ >- error= 4; >- goto err; >- } >+ goto err; // Wrong field definition > else > reg_field->flags |= AUTO_INCREMENT_FLAG; As you are cleaning things up, remove else... <cut> >+ case OPEN_FRM_OK: >+ DBUG_ASSERT(0); // open_table_error() is never called for this one Add a break after the assert <cut> >- case 6: >- strxmov(buff, share->normalized_path.str, reg_ext, NullS); >- my_printf_error(ER_NOT_FORM_FILE, >- "Table '%-.64s' was created with a different version " >- "of MySQL and cannot be read", >- MYF(0), buff); >- break; It would be nice to still keep the above error message. We may need it sooner or later! For example, when we find a required table option type that we don't support! In this case we should also print out the MySQL version that was used to create the table! >- case 8: >+ case OPEN_FRM_NOT_A_TABLE: >+ my_error(ER_WRONG_OBJECT, MYF(0), share->db.str, >+ share->table_name.str, "TABLE"); > break; >- default: /* Better wrong error than none */ >- case 4: >+ case OPEN_FRM_DISCOVER: >+ DBUG_ASSERT(0); // open_table_error() is never called for this one Add a break! (Just in case) >+ case OPEN_FRM_CORRUPTED: > strxmov(buff, share->normalized_path.str, reg_ext, NullS); > my_error(ER_NOT_FORM_FILE, errortype, buff); > break; <cut> >+void prepare_fileinfo(THD *thd, uint reclength, uchar *fileinfo, >+ HA_CREATE_INFO *create_info, uint keys, KEY *key_info) >+{ Change name to prepare_frm_header >- DBUG_ENTER("create_frm"); >- >- if (create_info->options & HA_LEX_CREATE_TMP_TABLE) >- create_flags|= O_EXCL | O_NOFOLLOW; I couldn't find where you ensured in writefrm that the above flags was added for temporary tables. We really want to have these flags for temporary files to ensure that we fail if the file already exists! We also don't want anyone to create a symlink in /tmp and use that to overwrite another file! >@@ -3367,7 +3204,7 @@ rename_file_ext(const char * from,const > char from_b[FN_REFLEN],to_b[FN_REFLEN]; > (void) strxmov(from_b,from,ext,NullS); > (void) strxmov(to_b,to,ext,NullS); >- return (mysql_file_rename(key_file_frm, from_b, to_b, MYF(MY_WME))); >+ return (mysql_file_rename(key_file_frm, from_b, to_b, MYF(0))); Why not give a warning if rename fales? At least, use 'ME_JUST_WARNING' >=== modified file 'sql/table.h' >--- sql/table.h 2013-01-29 14:10:47 +0000 >+++ sql/table.h 2013-03-06 18:27:13 +0000 >+/* Adjust number to next larger disk buffer */ >+static inline ulong next_io_size(ulong pos) >+{ >+ return MY_ALIGN(pos, IO_SIZE); >+} >+ Remove above function. Instead store true max length in frominfo+2 >+++ sql/unireg.cc 2013-03-18 15:21:57 +0000 <cut> >+ frm.length= 64; // fileinfo; Change 64 to FRM_HEADER_LENGTH (and use this in other part of the source) >+ frm.length+= extra2_size + 4; // mariadb extra2 frm segment >+ >+ int2store(fileinfo+4, extra2_size); Add comment: // Size of table version and table options Note that you can't use position 4 for this. Position 4 & 5 is reserved for the number of different forms/screens inside the frm. Using this would make the created table unusable for any old version of MySQL / MariaDB. See get_form_pos() in table.cc in MySQL 5.5 Here is the relevant data in the old frm header Pos Information 4-5 Length of form names ; Always 3 in MySQL 8-9 Number of forms ; Always 1 in MySQL 10-13 Position for next form (if we would create one) 64 Name information (int2korr(head+4) bytes) . 4 byte offset/form from file start to where form starts How to fix this in a portable way (instead of using position 4): Position 10-13 are only set by MySQL but never used. This is ALWAYS aligned on IO_SIZE (Ie, byte 10 is always 0). We can use byte 10 to mark that this is a MariaDB .frm file and then we can use bytes 11-13 for whatever we want. >+ int2store(fileinfo+6, frm.length); Add comment: // Position to key information >+ frm.length+= key_buff_length; >+ frm.length+= reclength; // row with default values >+ frm.length+= create_info->extra_size; >+ >+ filepos= frm.length; >+ frm.length+= 288; // forminfo Replace 288 in the code with FRM_FORM_LENGTH >+ frm.length+= packed_fields_length(create_fields); >+ >+ frm_ptr= (uchar*) my_malloc(frm.length, MYF(MY_WME | MY_ZEROFILL | >+ MY_THREAD_SPECIFIC)); >+ if (!frm_ptr) >+ DBUG_RETURN(frm); >+ >+ /* write the extra2 segment */ >+ pos = frm_ptr + 64; 64 -> FRM_HEADER_LENGTH Please move the line: memcpy(frm_ptr, fileinfo, 64); Before the above line. (Makes it easier to see that header is copied) >+ *pos++ = EXTRA2_TABLEDEF_VERSION; We probably want to add a comment that MySQL stores '/' here and that EXTRA2_TABLEDEF_VERSION can't never be '/' <cut> >+ DBUG_ASSERT(options_len <= 65535); // FIXME if necessary >+ int2store(pos + 1, options_len); Why not just use (so that you can get rid of the comment) int3store(pos + 1, options_len); >+ pos+= 3; >+ } >+ pos= engine_table_options_frm_image(pos, create_info->option_list, >+ create_fields, keys, key_info); >+ } >+ >+ int4store(pos, filepos); // end of the extra2 segment >+ pos+= 4; >+ >+ DBUG_ASSERT(pos == frm_ptr + uint2korr(fileinfo+6)); >+ key_info_length= pack_keys(pos, keys, key_info, data_offset); > > maxlength=(uint) next_io_size((ulong) (uint2korr(forminfo)+1000)); > int2store(forminfo+2,maxlength); You can skip the above 2 rows and mark the bytes as free to use. > int4store(fileinfo+10,(ulong) (filepos+maxlength)); Same with the above 4 bytes <cut> > } > } > if (forminfo[46] == (uchar)255) > { Add comment /* New style MySQL 5.5 table comment */ By the way, this was a *stupid* solution for MySQL 5.5 from MySQL 6.0 What it means is that long comments from MySQL 5.5 are hidden in MySQL below 5.5 A better solution would be to always copy up to TABLE_COMMENT_INLINE_MAXLEN bytes into forminfo+47. (Better to have a cut comment than no comment). Can you please do that change? <cut> >+int rea_create_table(THD *thd, LEX_CUSTRING *frm, >+ const char *path, const char *db, const char *table_name, >+ HA_CREATE_INFO *create_info, handler *file) > { > DBUG_ENTER("rea_create_table"); > >- char frm_name[FN_REFLEN]; >- strxmov(frm_name, path, reg_ext, NullS); >- if (mysql_create_frm(thd, frm_name, db, table_name, create_info, >- create_fields, keys, key_info, file)) >+ if (file) >+ { >+ // TODO don't write frm for temp tables >+ if (create_info->tmp_table() && >+ writefrm(path, db, table_name, 0, frm->str, frm->length)) >+ goto err_handler; Can you add a test that if it's a memory table we don't write the frm? (Better to do it for other tmp tables, as we need to delete them from the engine during restart). As far as I can see, we only need one test here, one for the error condition and also when we drop the temporary table. Why not send the 'frm' forwards instead of frm->str and frm->length. These two arguments are passed around for a lot of functions. Would be easier if all functions would just take handler. <cut> >@@ -718,10 +535,8 @@ static bool pack_header(uchar *forminfo, > my_snprintf(warn_buff, sizeof(warn_buff), ER(ER_TOO_LONG_FIELD_COMMENT), > field->field_name, > static_cast<ulong>(COLUMN_COMMENT_MAXLEN)); >- /* do not push duplicate warnings */ >- if (!check_duplicate_warning(current_thd, warn_buff, strlen(warn_buff))) >- push_warning(current_thd, MYSQL_ERROR::WARN_LEVEL_WARN, >- ER_TOO_LONG_FIELD_COMMENT, warn_buff); >+ push_warning(current_thd, MYSQL_ERROR::WARN_LEVEL_WARN, >+ ER_TOO_LONG_FIELD_COMMENT, warn_buff); You could clean up the above code at the same time. Now we duplicate a lot of code above. The full code should changed as follows: if ((current_thd->variables.sql_mode & (MODE_STRICT_TRANS_TABLES | MODE_STRICT_ALL_TABLES))) { my_error(ER_TOO_LONG_FIELD_COMMENT, MYF(0), field->field_name, static_cast<ulong>(COLUMN_COMMENT_MAXLEN)); DBUG_RETURN(1); } char warn_buff[MYSQL_ERRMSG_SIZE]; my_snprintf(warn_buff, sizeof(warn_buff), ER(ER_TOO_LONG_FIELD_COMMENT), field->field_name, static_cast<ulong>(COLUMN_COMMENT_MAXLEN)); push_warning(current_thd, MYSQL_ERROR::WARN_LEVEL_WARN, ER_TOO_LONG_FIELD_COMMENT, warn_buff); field->comment.length= tmp_len; -> bool strict= (current_thd->variables.sql_mode & (MODE_STRICT_TRANS_TABLES | MODE_STRICT_ALL_TABLES))); my_error(ER_TOO_LONG_FIELD_COMMENT, MYF(strict ? 0 : ME_JUST_WARNING), field->field_name, static_cast<ulong>(COLUMN_COMMENT_MAXLEN)); if (strict) DBUG_RETURN(1); field->comment.length= tmp_len; >@@ -833,7 +647,7 @@ static bool pack_header(uchar *forminfo, > } > /* Hack to avoid bugs with small static rows in MySQL */ > reclength=max(file->min_record_length(table_options),reclength); >- if (info_length+(ulong) create_fields.elements*FCOMP+288+ >+ if ((ulong) create_fields.elements*FCOMP+288+ 288 -> FRM_FORM_LENGTH <cut> >- /* Save fields, fieldnames and intervals */ >+static size_t packed_fields_length(List<Create_field> &create_fields) >+{ >+ Create_field *field; >+ size_t length= 0; >+ DBUG_ENTER("packed_fields_length"); > >-static bool pack_fields(File file, List<Create_field> &create_fields, >+ List_iterator<Create_field> it(create_fields); >+ uint int_count=0; >+ while ((field=it++)) >+ { >+ if (field->interval_id > int_count) >+ { >+ int_count= field->interval_id; >+ length++; >+ for (int i=0; field->interval->type_names[i]; i++) >+ { >+ length+= field->interval->type_lengths[i]; >+ length++; >+ } >+ length++; >+ } >+ if (field->vcol_info) >+ { >+ length+= field->vcol_info->expr_str.length + >+ FRM_VCOL_HEADER_SIZE(field->interval); Should be: length+= field->vcol_info->expr_str.length + FRM_VCOL_HEADER_SIZE(field->interval_id); Why? This is the code where the byte is written in pack_fields(). (The code you copied was also wrong, see below) if (field->interval_id) *buff++= (uchar) field->interval_id; <cut> >+static bool pack_fields(uchar *buff, List<Create_field> &create_fields, <cut> >@@ -950,40 +791,29 @@ static bool pack_fields(File file, List< > the additional data saved for the virtual field > */ > buff[12]= cur_vcol_expr_len= field->vcol_info->expr_str.length + >- FRM_VCOL_HEADER_SIZE(field->interval!=NULL); >- vcol_info_length+= cur_vcol_expr_len + >- FRM_VCOL_HEADER_SIZE(field->interval!=NULL); >+ FRM_VCOL_HEADER_SIZE(field->interval); >+ vcol_info_length+= cur_vcol_expr_len; Change field->interval to field->interval_id >=== modified file 'storage/archive/ha_archive.cc' >--- storage/archive/ha_archive.cc 2013-01-23 15:24:04 +0000 >+++ storage/archive/ha_archive.cc 2013-03-06 20:23:01 +0000 >+int archive_discover(handlerton *hton, THD* thd, TABLE_SHARE *share) > { > DBUG_ENTER("archive_discover"); >- DBUG_PRINT("archive_discover", ("db: %s, name: %s", db, name)); >+ DBUG_PRINT("archive_discover", ("db: %s, name: %s", share->db.str, >+ share->table_name.str)); When you change DBUG_PRINT, please also fix the syntax to be same as the rest of the code. The above should be: DBUG_PRINT("archive_discover", ("db: '%s' name: '%s'", share->db.str, share->table_name.str)); (No comma, double space, ' around string arguments) >@@ -738,56 +748,40 @@ int ha_archive::create(const char *name, <cut> >+ /* >+ Here is where we open up the frm and pass it to archive to store >+ */ >+ if (!table_arg->s->read_frm_image(&frm_ptr, &frm_len)) >+ { >+ azwrite_frm(&create_stream, frm_ptr, frm_len); >+ my_free(const_cast<uchar*>(frm_ptr)); >+ } Generally it would be better to have a function: table_arg->s->free_frm_image(&frm_ptr) Because the server could be compiled with another memory allocator than the storage engine (DBUG / NO_DBUG). >=== modified file 'storage/federatedx/ha_federatedx.cc' >--- storage/federatedx/ha_federatedx.cc 2013-01-23 15:24:04 +0000 >+++ storage/federatedx/ha_federatedx.cc 2013-03-10 10:49:49 +0000 >@@ -3588,6 +3574,71 @@ int ha_federatedx::rollback(handlerton * > DBUG_RETURN(return_val); > } > >+ >+/* >+ Federated supports assisted discovery, like >+ CREATE TABLE t1 CONNECTION="mysql://joe:pass@192.168.1.111/federated/t1"; >+ but not a fully automatic discovery where a table magically appear >+ on any use (like, on SELECT * from t1). >+*/ >+int ha_federatedx::discover_assisted(handlerton *hton, THD* thd, >+ TABLE_SHARE *table_s, HA_CREATE_INFO *info) <cut> >+ query.copy(rdata[1], rlen[1], cs); >+ query.append(STRING_WITH_LEN(" CONNECTION=\""), cs); >+ query.append(table_s->connect_string.str, table_s->connect_string.length, cs); >+ query.append('"'); Shouldn't we need to quote the string properly? (It may have " inside) Other things: - Please run sql_bench/test-create on your tree to verify that things are now faster (should be as we don't have to open a frm when a table is created). - Test it also for temporary memory tables (assuming you can get rid of the .frm for them) Regards, Monty

2 3

[Maria-developers] KB Question: Storage engine plugin that "proxies" another engine?
by Daniel Bartholomew 08 Apr '13

08 Apr '13

The following question has been posted to the AskMonty Knowledgebase: https://kb.askmonty.org/en/storage-engine-plugin-that-proxies-another-engin… - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Storage engine plugin that "proxies" another engine? I would like to write a plugin that adds transparent versioning support for table rows. The idea is one would create a table and specify something like "Engine=Versioned", which would create two regular tables in InnoDB/MyISAM. One table to hold the current data, and one to hold a couple of previous versions. Then, when running UPDATE/DELETE statements, the "versioned" engine would first copy the current data to the "history" table and then forward the statement to the underlying storage engine. Is this possible to do in a nice way using the plugin architecture? Thanks Thomas - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

[Maria-developers] mysql test question
by Rich Prohaska 05 Apr '13

05 Apr '13

Hello, I was wondering if there was a way to run a mysql-test on with a prebuilt data directory. If so, can you point me to some example? Thanks Rich Prohaska

3 4

[Maria-developers] Rev 3712: MDEV-4338 : Support atomic option on directFS/FusionIO
by Sergei Golubchik 05 Apr '13

05 Apr '13

Hi, Vladislav! On Mar 29, Vladislav Vaintroub wrote: > ------------------------------------------------------------ > revno: 3712 > revision-id: wlad(a)montyprogram.com-20130328203348-1qe9u9nlq6ol9b8l > parent: igor(a)askmonty.org-20130329021836-x6swah73a59im9tn > fixes bug: https://mariadb.atlassian.net/browse/MDEV-4338 > committer: Vladislav Vaintroub <wlad(a)montyprogram.com> > branch nick: 5.5 > timestamp: Thu 2013-03-28 21:33:48 +0100 > message: > MDEV-4338 : Support atomic option on directFS/FusionIO. > === modified file 'storage/xtradb/handler/ha_innodb.cc' > --- a/storage/xtradb/handler/ha_innodb.cc 2013-03-08 18:08:45 +0000 > +++ b/storage/xtradb/handler/ha_innodb.cc 2013-03-28 20:33:48 +0000 > @@ -185,6 +185,7 @@ static my_bool innobase_file_format_chec > static my_bool innobase_log_archive = FALSE; > static char* innobase_log_arch_dir = NULL; > #endif /* UNIV_LOG_ARCHIVE */ > +static my_bool innobase_use_atomic_writes = TRUE; > static my_bool innobase_use_doublewrite = TRUE; > static my_bool innobase_use_checksums = TRUE; > static my_bool innobase_fast_checksum = FALSE; > @@ -3057,6 +3058,35 @@ innobase_init( > srv_kill_idle_transaction = 0; > #endif > > + srv_use_atomic_writes = (ibool) innobase_use_atomic_writes; Is that the typical usage pattern in the innodb code? Because it looks a bit silly to me, why couldn't you create a sysvar directly on srv_use_atomic_writes, why an intermediate innobase_use_atomic_writes variable? > + if (innobase_use_atomic_writes) { > + fprintf(stderr, "InnoDB: using atomic writes.\n"); > + > + /* Force doublewrite buffer off, atomic writes replace it. */ > + if (srv_use_doublewrite_buf) { > + fprintf(stderr, "InnoDB: Switching off doublewrite buffer " > + "because of atomic writes.\n"); > + innobase_use_doublewrite = srv_use_doublewrite_buf = FALSE; > + } > + > + /* Force O_DIRECT on Unixes (on Windows writes are always unbuffered)*/ > +#ifndef _WIN32 > + if(!innobase_file_flush_method || > + !strstr(innobase_file_flush_method, "O_DIRECT")) { > + innobase_file_flush_method = > + srv_file_flush_method_str = (char*)"O_DIRECT"; > + fprintf(stderr, "InnoDB: using O_DIRECT due to atomic writes.\n"); > + } > +#endif > +#ifdef HAVE_POSIX_FALLOCATE > + /* Due to a bug in directFS, using atomics needs > + * posix_fallocate to extend the file > + * pwrite() past end of the file won't work > + */ > + srv_use_posix_fallocate = TRUE; > +#endif > + } it'd be nice if you could try os_file_set_atomic_writes() here to see if that works. This function creates and opens quite a few files, you could use one of them to mark it for atomic writes, and if that would fail, you'd disable atomic writes, and wouldn't change innobase_file_flush_method and innobase_use_doublewrite. Btw, why not to use posix_fallocate whenever it's available? Or, at least, with its own --innodb-use-fallocate option? Regards, Sergei

3 6

Re: [Maria-developers] MDEV-3917 multiple use locks (GET_LOCK) in one connection
by Sergei Golubchik 04 Apr '13

04 Apr '13

Hi, Holyfoot! (Kostja - there're a couple of questions at the end that you might want to reply to) On Apr 02, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-3917/ > > ------------------------------------------------------------ > revno: 3510 > revision-id: holyfoot(a)askmonty.org-20130402085219-1sxuzssq7segeig4 > parent: igor(a)askmonty.org-20130331221855-1n7hagklb3soplxq > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-3917 > timestamp: Tue 2013-04-02 13:52:19 +0500 > message: > MDEV-3917 multiple use locks (GET_LOCK) in one connection. > The patch contributed by Konstantin Osipov applied. > Native comments: > Implement multiple user-level locks per connection. > > GET_LOCK() function in MySQL allows a connection to hold at most > one user level lock. Taking a new lock automatically releases the > old lock, if any. > > The limit of one lock per session existed since early versions > of MySQL didn't have a deadlock detector for SQL locks. > MDL patches in MySQL 5.5 added a deadlock detector, > so starting from 5.5 it became possible to take multiple locks > in any order -- a deadlock, should it occur, would be detected > and an error returned to the client which closed the wait chain. > > This is exactly what is done in this patch: ULLs are moved > to use MDL subsystem. > > also: > MDL_key::USER renamed to MDL::USER_LOCK What does that mean? > boundary checks for user lock name added. > === modified file 'sql/item_func.cc' > --- a/sql/item_func.cc 2013-03-27 22:41:02 +0000 > +++ b/sql/item_func.cc 2013-04-02 08:52:19 +0000 > @@ -4010,7 +3896,121 @@ int Interruptible_wait::wait(mysql_cond_ > > > /** > - Get a user level lock. If the thread has an old lock this is first released. > + For locks with EXPLICIT duration, MDL returns a new ticket > + every time a lock is granted. This allows to implement recursive > + locks without extra allocation or additional data structures, such > + as below. However, if there are too many tickets in the same > + MDL_context, MDL_context::find_ticket() is getting too slow, > + since it's using a linear search. > + This is why a separate structure is allocated for a user > + level lock, and before requesting a new lock from MDL, > + GET_LOCK() checks thd->ull_hash if such lock is already granted, > + and if so, simply increments a reference counter. > +*/ > + > +class User_level_lock > +{ > +public: > + MDL_ticket *lock; > + int refs; > +}; > + > + > +/** Extract a hash key from User_level_lock. */ > + > +uchar *ull_get_key(const uchar *ptr, size_t *length, > + my_bool not_used __attribute__((unused))) > +{ > + User_level_lock *ull = (User_level_lock*) ptr; > + MDL_key *key = ull->lock->get_key(); > + *length= key->length(); > + return (uchar*) key->ptr(); > +} > + > + > +/** > + Release all user level locks for this THD. > +*/ > + > +void mysql_ull_cleanup(THD *thd) > +{ > + User_level_lock *ull; > + DBUG_ENTER("mysql_ull_cleanup"); > + > + for (uint i= 0; i < thd->ull_hash.records; i++) > + { > + ull = (User_level_lock*) my_hash_element(&thd->ull_hash, i); > + thd->mdl_context.release_lock(ull->lock); > + free(ull); my_free > + } > + > + my_hash_free(&thd->ull_hash); > + > + DBUG_VOID_RETURN; > +} > + > + > +/** > + Set explicit duration for metadata locks corresponding to > + user level locks to protect them from being released at the end > + of transaction. > +*/ > + > +void mysql_ull_set_explicit_lock_duration(THD *thd) > +{ > + User_level_lock *ull; > + DBUG_ENTER("mysql_ull_set_explicit_lock_duration"); > + > + for (uint i= 0; i < thd->ull_hash.records; i++) > + { > + ull= (User_level_lock*) my_hash_element(&thd->ull_hash, i); > + thd->mdl_context.set_lock_duration(ull->lock, MDL_EXPLICIT); > + } > + DBUG_VOID_RETURN; > +} > + > + > +/** > + When MDL detects a lock wait timeout, it pushes > + an error into the statement diagnostics area. > + For GET_LOCK(), lock wait timeout is not an error, > + but a special return value (0). NULL is returned in > + case of error. > + Capture and suppress lock wait timeout. > +*/ > + > +class Lock_wait_timeout_handler: public Internal_error_handler > +{ > +public: > + Lock_wait_timeout_handler() :m_lock_wait_timeout(false) {} > + > + bool m_lock_wait_timeout; > + > + bool handle_condition(THD * /* thd */, uint sql_errno, > + const char * /* sqlstate */, > + MYSQL_ERROR::enum_warning_level /* level */, > + const char *message, > + MYSQL_ERROR ** /* cond_hdl */); > +}; > + > +bool > +Lock_wait_timeout_handler:: > +handle_condition(THD * /* thd */, uint sql_errno, > + const char * /* sqlstate */, > + MYSQL_ERROR::enum_warning_level /* level */, > + const char *message, > + MYSQL_ERROR ** /* cond_hdl */) > +{ > + if (sql_errno == ER_LOCK_WAIT_TIMEOUT) > + { > + m_lock_wait_timeout= true; > + return true; /* condition handled */ > + } > + return false; > +} > + > +/** > + Get a user level lock. > > @retval > 1 : Got lock > @@ -4039,104 +4038,75 @@ longlong Item_func_get_lock::val_int() > it's not guaranteed to be same as on master. > */ > if (thd->slave_thread) > + { > + null_value= 0; > DBUG_RETURN(1); > + } > > if (!res || !res->length()) > + DBUG_RETURN(0); > + > + if (res->length() > NAME_LEN) > { > + my_error(ER_TOO_LONG_IDENT, MYF(0), res->c_ptr_safe()); > DBUG_RETURN(0); > } > + > DBUG_PRINT("info", ("lock %.*s, thd=%ld", res->length(), res->ptr(), > (long) thd->real_id)); > + /* HASH entries are of type User_level_lock. */ > + if (! my_hash_inited(&thd->ull_hash) && > + my_hash_init(&thd->ull_hash, &my_charset_bin, > + 16 /* small hash */, 0, 0, ull_get_key, NULL, 0)) > { > + DBUG_RETURN(0); > } > > + MDL_request ull_request; > + ull_request.init(MDL_key::USER_LOCK, res->c_ptr_safe(), "", > + MDL_SHARED_NO_WRITE, MDL_EXPLICIT); > + MDL_key *ull_key = &ull_request.key; > > > + if ((ull= (User_level_lock*) > + my_hash_search(&thd->ull_hash, ull_key->ptr(), ull_key->length()))) > + { > + /* Recursive lock */ > + ull->refs++; > + null_value = 0; > + DBUG_RETURN(1); > + } > > + Lock_wait_timeout_handler lock_wait_timeout_handler; > + thd->push_internal_handler(&lock_wait_timeout_handler); > + bool error= thd->mdl_context.acquire_lock(&ull_request, timeout); > + (void) thd->pop_internal_handler(); > + if (error) > { > + if (lock_wait_timeout_handler.m_lock_wait_timeout) > + null_value= 0; > + DBUG_RETURN(0); > } > > + ull= (User_level_lock*) malloc(sizeof(User_level_lock)); my_malloc() with MYF(MY_WME|MY_THREAD_SPECIFIC) > + if (ull == NULL) > { > + thd->mdl_context.release_lock(ull_request.ticket); > + DBUG_RETURN(0); > } > + > + ull->lock= ull_request.ticket; > + ull->refs= 1; > + > + if (my_hash_insert(&thd->ull_hash, (uchar*) ull)) > { > + thd->mdl_context.release_lock(ull->lock); > + free(ull); my_free > + DBUG_RETURN(0); > } > + null_value= 0; > > + DBUG_RETURN(1); > } > > > @@ -4151,43 +4121,104 @@ longlong Item_func_get_lock::val_int() > longlong Item_func_release_lock::val_int() > { > DBUG_ASSERT(fixed == 1); > + String *res= args[0]->val_str(&value); > + THD *thd= current_thd; > DBUG_ENTER("Item_func_release_lock::val_int"); > + null_value= 1; > + > if (!res || !res->length()) > + DBUG_RETURN(0); > + > + if (res->length() > NAME_LEN) > { > + my_error(ER_TOO_LONG_IDENT, MYF(0), res->c_ptr_safe()); > DBUG_RETURN(0); > } > + > DBUG_PRINT("info", ("lock %.*s", res->length(), res->ptr())); > > + MDL_key ull_key; > + ull_key.mdl_key_init(MDL_key::USER_LOCK, res->c_ptr_safe(), ""); > + > + User_level_lock *ull; > + > + if (!(ull= > + (User_level_lock*) my_hash_search(&thd->ull_hash, > + ull_key.ptr(), ull_key.length()))) > { > + null_value= thd->mdl_context.get_lock_owner(&ull_key) == 0; > + DBUG_RETURN(0); > } > + null_value= 0; > + if (--ull->refs == 0) > { > + my_hash_delete(&thd->ull_hash, (uchar*) ull); > + thd->mdl_context.release_lock(ull->lock); > + free(ull); my_free > } > + DBUG_RETURN(1); > +} > + > + > +/** > + Check a user level lock. > + > + Sets null_value=TRUE on error. > + > + @retval > + 1 Available > + @retval > + 0 Already taken, or error > +*/ > + > +longlong Item_func_is_free_lock::val_int() > +{ > + DBUG_ASSERT(fixed == 1); > + String *res= args[0]->val_str(&value); > + THD *thd= current_thd; > + null_value= 1; > + this block ----- from here > + if (!res || !res->length()) > + return 0; > + > + if (res->length() > NAME_LEN) > + { > + my_error(ER_TOO_LONG_IDENT, MYF(0), res->c_ptr_safe()); > + return 0; > + } and up ---- to here is repeated four times. please move it to a separate function. Like if (!ull_name_ok(res)) return 0; > + > + MDL_key ull_key; > + ull_key.mdl_key_init(MDL_key::USER_LOCK, res->c_ptr_safe(), ""); > + > + null_value= 0; > + return thd->mdl_context.get_lock_owner(&ull_key) == 0; > +} > + > + > +longlong Item_func_is_used_lock::val_int() > +{ > + DBUG_ASSERT(fixed == 1); > + String *res= args[0]->val_str(&value); > + THD *thd= current_thd; > + null_value= 1; > + > + if (!res || !res->length()) > + return 0; > + > + if (res->length() > NAME_LEN) > + { > + my_error(ER_TOO_LONG_IDENT, MYF(0), res->c_ptr_safe()); > + return 0; > + } > + > + MDL_key ull_key; > + ull_key.mdl_key_init(MDL_key::USER_LOCK, res->c_ptr_safe(), ""); > + ulong thread_id = thd->mdl_context.get_lock_owner(&ull_key); > + if (thread_id == 0) > + return 0; > + > + null_value= 0; > + return thread_id; > } > > === modified file 'sql/mdl.cc' > --- a/sql/mdl.cc 2013-01-29 14:10:47 +0000 > +++ b/sql/mdl.cc 2013-04-02 08:52:19 +0000 > @@ -2094,8 +2151,6 @@ MDL_context::acquire_lock(MDL_request *m > > find_deadlock(); > > - if (lock->needs_notification(ticket)) > - { okay, so before the patch only locks that needs_notification were using the loop with short waits. Now all locks do. Presumably to check whether thd_is_connected still. Before the patch, how were client disconnects handled? Like, one could have started to wait on the mdl lock, and then disconnect. Was the wait aborted? > struct timespec abs_shortwait; > set_timespec(abs_shortwait, 1); > wait_status= MDL_wait::EMPTY; > @@ -2106,17 +2163,25 @@ MDL_context::acquire_lock(MDL_request *m > > if (wait_status != MDL_wait::EMPTY) > break; > + /* Check if the client is gone while we were waiting. */ > + if (! thd_is_connected(m_thd)) > + { > + /* > + * The client is disconnected. Don't wait forever: > + * assume it's the same as a wait timeout, this > + * ensures all error handling is correct. > + */ > + wait_status= MDL_wait::TIMEOUT; > + break; > + } > > mysql_prlock_wrlock(&lock->m_rwlock); > + if (lock->needs_notification(ticket)) why this if() is under the prlock, and not the other way around? > lock->notify_conflicting_locks(this); > mysql_prlock_unlock(&lock->m_rwlock); > set_timespec(abs_shortwait, 1); > } > if (wait_status == MDL_wait::EMPTY) > - wait_status= m_wait.timed_wait(m_thd, &abs_timeout, TRUE, > - mdl_request->key.get_wait_state_name()); > - } > - else > wait_status= m_wait.timed_wait(m_thd, &abs_timeout, TRUE, > mdl_request->key.get_wait_state_name()); Regards, Sergei

1 0

[Maria-developers] MDEV-4145/MWL#253 review feedback (unfinished)
by Sergei Petrunia 03 Apr '13

03 Apr '13

Hello Igor, Please find some review feedback below. ===== sql_statistics.cc has this line with comment: > If the value the variable 'use_stat_tables' is set to > "preferably" the optimizer uses a particular statistical data only if > it can't be found in the statistical data. the part > uses a particular statistical data > only if it can't be found > in the statistical data. doesn't seem to make any sense. Is this a typo? ===== sql_statistics.cc: collect_statistics_for_table() has this code: > table_field->collected_stats->cleanup(); > } > bitmap_clear_all(table->write_set); Why suddenly no indentation on the line with the bitmap_clear_all() call? This looks as if a merge has gone wrong, please fix. ===== sql_statistics.cc has this : > class Table_stat: public Stat_table > { > > private: > > Field *db_name_field; /* Field for the column table_stats.db_name */ Why do all class definitions start with 'private:'? It is active by default in C++. The code looks as if it was auto-translated from some other language. ===== I see that my progress reporting patch from November is not part of the tree yet. ===== I don't any code that would check if table DDLs of statistical tables are what the server expects them to be. I think the check should be made, just like it is made for mysql.proc and other tables. ==== Then I started to play with the code and have filed MDEV-4350, MDEV-4357, MDEV-4359, MDEV-4360, MDEV-4362, MDEV-4363, MDEV-4364. I find it difficult to proceed when there are so many simple counterexamples. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] read_buffer_size query
by Ian Gilfillan 02 Apr '13

02 Apr '13

Hi I'm working on adding complete descriptions to the knowledgebase for all the server variables right now, and have a query on the read_buffer_size variable. The default in my MariaDB 10.0.1 seems to be 2097152, but it's listed on the MySQL docs with a default of 131072. I can't find a mention in the changelogs of this having changed - can anyone tell me in which version the change in default happened? Also, in the MySQL documentation, this variable is described as applying to MyISAM tables, while in the only short description on the Knowledgebase I can find, at https://kb.askmonty.org/en/mysqld-options-full-list/, it's described more generally, not specifically mentioning MyISAM. Does this variable apply to other storage engines as well now (Aria?), or still only MyISAM? thanks, ian gilfillan

3 4

[Maria-developers] Cassandra disabled due to excessive memory usage causing test failures
by Kristian Nielsen 02 Apr '13

02 Apr '13

Hi, When I added my 10.0-mdev26 tree to buildbot, I got some strange errors on the kvm-fulltest2 builder. I finally tracked it down to be caused by the OOM killer killing random processes. The root issue is Cassandra. At some point it seems cassandra was enabled in the VM used by the fulltest2 builder. However, just starting up Cassandra creates a jsvc process that uses 3.4 _gigabyte_ of memory, 1.3 GB of which is resident. That is too much... I also noticed that cassandra tests are not even run, plugins.cassandra is skipped due to cassandra not running. Possibly this is because the OOM killer has killed the process ;-). In any case, for now I have disabled cassandra on the -fulltest VMs: sudo update-rc.d cassandra disable If someone wants to re-enable it, feel free, but please first make sure that it does not cause unrelated test failures due to excessive memory usage! I would suggest not to run cassandra by default, instead add a separate buildbot step that explicitly starts up cassandra when needed. ---- In general, I am disappointed about the state of our trees. This problem seems to have been there for more than a month without anyone doing anything, and there are tons of other test failures in 10.0 and 10.0-base. We need to tighten up our processes here. Remember, the rules are that anything major should be green in a feature tree before pushing to a main tree. Clearly this is not what is happening at the moment. It's ridiculous to constantly hear people pressing for release of 10.0.2-beta when our trees are not even alpha state ... - Kristian.

1 0

[Maria-developers] I have a friend. Her name is Maria
by Кукушкин Александр 01 Apr '13

01 Apr '13

Hello! excuse me for off topic. I have a friend, her name is Maria and she has a birthday today. I thought maybe someone of you could write a commit message mentioning her. Wish her good performance and persistence or something like that. Then I could just show her a page like that one http://bazaar.launchpad.net/~maria-captains/maria/10.0/revision/3505 What do you think about that? is it possible? (sorry if this message is buplicate, I tried to post it yesterday to your mailing list but it seems like I failed)

1 1

Re: [Maria-developers] Rev 3497: MDEV-318 IF (NOT) EXIST clauses for ALTER TABLE (MWL #252)
by Sergei Golubchik 29 Mar '13

29 Mar '13

Hi, Holyfoot! Please push, with one little change: On Feb 19, holyfoot(a)askmonty.org wrote: > > message: > MDEV-318 IF (NOT) EXIST clauses for ALTER TABLE (MWL #252). > Syntax modified to allow statements: > ALTER TABLE ADD/DROP COLUMN > ALTER TABLE ADD/DROP INDEX > ALTER TABLE ADD/DROP FOREIGN KEY > ALTER TABLE ADD/DROP PARTITION > ALTER TABLE CHANGE COLUMN > ALTER TABLE MODIFY COLUMN > DROP INDEX > to have IF (NOT) EXISTS options. > Appropriate implementations added to mysql_alter_table(). ... > key_def: > - normal_key_type opt_ident key_alg '(' key_list ')' > + normal_key_type opt_if_not_exists_ident key_alg '(' key_list ')' > { Lex->option_list= NULL; } remove your new opt_if_not_exists_ident rule, write it like - normal_key_type opt_ident key_alg '(' key_list ')' + normal_key_type opt_if_not_exists opt_ident key_alg '(' key_list ')' Regards, Sergei

2 2

[Maria-developers] New problem with GTID patch
by Pavel Ivanov 28 Mar '13

28 Mar '13

Kristian, Unfortunately with your recent commit (revision 3546) test rpl.rpl_gtid_startpos doesn't work when GTID patch is applied on top of 10.0.1 tarball. It works okay in 10.0-mdev26 though, so I'm not sure if you'll want to look at the problem now. In particular the problematic addition is if (!is_relay_log && read_state_from_file()) DBUG_RETURN(1); to function MYSQL_BIN_LOG::open(). Without it test works. With it test breaks in the section "Test that master gives error when slave asks for empty gtid pos and binlog files have been purged". The problem is when get_gtid_list_event() is called from gtid_find_binlog_file() it returns event with count == 0. It causes contains_all_slave_gtid() to return true and master doesn't send error to slave as expected... I didn't look in any further details because I'm not familiar with everything that should happen here that much. If you want me to look at something else to better understand the problem let me know. Pavel

2 3

[Maria-developers] (no subject)
by Sergei Petrunia 28 Mar '13

28 Mar '13

Hello Igor, I was reviewing histogram code. Not much feedback so far, but I thought I would let you know: == Code questions == Is middle_point_pos() a good functio nname? It was not apparent to me what it meant. For me, "middle point" is a point in the middle of a range, that is, a point equidistant from both ends. Maybe, "position_in_range" will be better? == Histogram storage questions == * What does hist_size column mean? I suppose, it is number of columns in the histogram? * Do we want to support only one kind of histogram? If not, maybe we should provision space for histogram type? * The histogram column is VARBINARY(255). Why set a hard limit of 255 bytes? Maybe, use a BLOB from the start? == A possible bug == I have a column with lots of NULLs: MariaDB [test]> select nulls_ratio from mysql.column_stats where table_name='one_k' and column_name='tnulls'; +-------------+ | nulls_ratio | +-------------+ | 0.9000 | +-------------+ 1 row in set (0.00 sec) The following query crashes in maria-10.0-mwl253 tree. AFAIU, Timour has fixed the crash in 10.0-mdev83 tree. With that tree, I get: MariaDB [test]> explain extended select * from one_k where tnulls is not null; +------+-------------+-------+------+---------------+------+---------+------+------+----------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+----------+-------------+ | 1 | SIMPLE | one_k | ALL | NULL | NULL | NULL | NULL | 969 | 100.00 | Using where | +------+-------------+-------+------+---------------+------+---------+------+------+----------+-------------+ 1 row in set, 1 warning (0.00 sec) Note that filtered=100%. It should be 10%. On Mon, Mar 25, 2013 at 11:48:30PM -0700, Igor Babaev wrote: > At file:///home/igor/maria/maria-10.0-mwl253/ > > ------------------------------------------------------------ > revno: 3502 > revision-id: igor(a)askmonty.org-20130326064829-3x0g8p1g7nbptusd > parent: igor(a)askmonty.org-20130311144424-cwqe1m70in3bora0 > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-10.0-mwl253 > timestamp: Mon 2013-03-25 23:48:29 -0700 > message: > Added histogams for table columns. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] Bugs in GTID implementation
by Kristian Nielsen 27 Mar '13

27 Mar '13

Pavel Ivanov <pivanof(a)google.com> writes: > way to fix them. Also if you can please look at the attached patch > containing fixes to all tests after GTID addition. There are some > questionable changes in there, so I would appreciate if you could > evaluate its correctness and consider including it into MariaDB tree. Right, so now I finished going through the whole testsuite, making all necessary adjustments to .result files (and a bit to the test cases). This is a _huge_ patch, 70k lines :-( This is due to the new GTID events, every SHOW BINLOG EVENTS and mysqldump output needs updating. I did briefly consider instead trying to filter the output to make it identical to the old one, to avoid the huge .result file changes, but I'm not really sure that is any better in the long run... So, I did this in my 10.0-mdev26 tree, which is based on 10.0-base. I noticed at least two things from your patch that are not part of this: - The --let $keep_gtid_events= 1 statements. I am not sure how this $keep_gtid_events came into the 10.0 tree. It is part of the MySQL 5.6 GTID implementation, which we do not want to merge. I would suggest you instead just remove it from show_events.inc. - There were some new testcases in mysql-test/suite/perfschema in your patch that are not in my tree. They update some digests, I did not investigate what those are about, but I will handle when I merge to 10.0 of course. Thanks for your efforts on preparing the initial patch. It was quite a nice help as a starting point, though I did do some parts of it different due to style or correctness issues. I'm likely to have missed a couple cases of course, but at least this should be a starting point, just let me know if you experience more failures. I do see a non-repatable failure in main.plugin and a DBUG crash which is also in other maria trees, I'll look at those next. The fix is pushed to lp:~maria-captains/maria/10.0-mdev26 as usual. - Kristian.

2 3

[Maria-developers] MariaDB 10 galera?
by Jan Kirchhoff 26 Mar '13

26 Mar '13

Hi, we have been playing around with 5.5.28-galera and went into more serious testing with 5.5.29-galera. We are evaluating to move from a traditional mariadb 5.2 master-slave setup to a 5 or 7 node galera cluster for our production database. I do have a good feeling so far and good experiences in our tests, but as I won't be making any major upgrades any more once we really started testing all of our processes on the galera cluster, I'd like to ask if there is any timeline for a mariadb-10.0-galera release? I couldn't find any information on the mariadb-website regarding further plans for mariadb-galera. Is there any technical reason that no binaries of mariadb-10-galera are available? Thanks Jan

2 1

Re: [Maria-developers] Bugs in GTID implementation
by Kristian Nielsen 26 Mar '13

26 Mar '13

Pavel Ivanov <pivanof(a)google.com> writes: > fixes. Meanwhile I've found one more bug in your code. In > Gtid_log_event::do_apply_event() you assume that mysql_parse() always > succeeds, you don't check thd->is_error() and always return 0. I don't Ah, right, I see. Can you test the appended patch if it fixes the problem? I tried to copy the error handling to be identical to what Query_log_event::do_apply_event("BEGIN") does. - Kristian.

2 3

Re: [Maria-developers] [Commits] Rev 3642: MDEV-4292 fix
by Sergei Golubchik 26 Mar '13

26 Mar '13

Hi, Sanja! On Mar 26, sanja(a)askmonty.org wrote: > revno: 3642 > revision-id: sanja(a)askmonty.org-20130326110746-2y3duai4mpflbq3s > parent: holyfoot(a)askmonty.org-20130322133227-szqis3jzf5izvomq > committer: sanja(a)askmonty.org > branch nick: work-maria-5.3-MDEV-4292 > timestamp: Tue 2013-03-26 13:07:46 +0200 > message: > MDEV-4292 fix. > > Fixed printing column_get finction. Is that a complete new fix? Or it's on top of the old one? Looks like new, but your words "post-review patch" confused me. Anyway, if it's a new patch - ok to push! Regards, Sergei

1 0

[Maria-developers] I would like to run a buildbot
by Dan Demeter 26 Mar '13

26 Mar '13

Hi everyone, I have 2 hardware nodes and I would be really happy to run 2 build bots, on some VMs. :) Kind regards, Dan

2 1

Re: [Maria-developers] Rev 3701: Fixed bug mdev-4311
by Sergei Golubchik 25 Mar '13

25 Mar '13

Hi, Igor! On Mar 22, Igor Babaev wrote: > At file:///home/igor/maria/maria-5.5-bug4311/ > > ------------------------------------------------------------ > revno: 3701 > revision-id: igor(a)askmonty.org-20130322224651-r2tpn9in5i1ifv7a > parent: sergii(a)pisem.net-20130317104125-yyp99euwpir5ueho > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.5-bug4311 > timestamp: Fri 2013-03-22 15:46:51 -0700 > message: > Fixed bug mdev-4311 (bug #68749). > This bug was introduced by the patch for WL#3220. > If the memory allocated for the tree to store unique elements > to be counted is not big enough to include all of them then > an external file is used to store the elements. > The unique elements are guaranteed not to be nulls. So, when > reading them from the file we don't have to care about the null > flags of the read values. However, we should remove the flag > at the very beginning of the process. If we don't do it and > if the last value written into the record buffer for the field > whose distinct values needs to be counted happens to be null, > then all values read from the file are considered to be nulls > and are not counted in. > The fix does not remove a possible null flag for the read values. > Rather it just counts the values in the same way it was done > before WL #3220. > === modified file 'sql/item_sum.cc' > --- a/sql/item_sum.cc 2013-02-28 17:42:49 +0000 > +++ b/sql/item_sum.cc 2013-03-22 22:46:51 +0000 > @@ -1460,6 +1460,12 @@ > > bool Aggregator_distinct::unique_walk_function(void *element) > { > + if (item_sum->sum_func() == Item_sum::COUNT_DISTINCT_FUNC) > + { > + Item_sum_count *sum= (Item_sum_count *)item_sum; > + sum->count++; > + return 0; > + } > memcpy(table->field[0]->ptr, element, tree_key_length); > item_sum->add(); > return 0; I'm not sure I like it. You've added a special check for COUNT_DISTINCT_FUNC just before the call of item_sum->add() which is virtual, and Item_sum_count has its own implementation of it. Logically, if there's a virtual method, the check should be in there, not in the caller. And what about SUM(DISTINCT ...) ? Perhaps it's better to mark the item not null, as you've explained in the changeset comment? Regards, Sergei

3 4

Re: [Maria-developers] Rev 3639: MDEV-4292 fix
by Sergei Golubchik 25 Mar '13

25 Mar '13

Hi, Sanja! On Mar 19, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-5.3-MDEV-4292/ > > ------------------------------------------------------------ > revno: 3639 > revision-id: sanja(a)askmonty.org-20130319105139-bxfigoel9pdbgwsn > parent: sergii(a)pisem.net-20130318074424-frdzldas283cvx1w > committer: sanja(a)askmonty.org > branch nick: work-maria-5.3-MDEV-4292 > timestamp: Tue 2013-03-19 12:51:39 +0200 > message: > MDEV-4292 fix. > > Fixed printing column_get finction. > > === modified file 'sql/item_func.cc' > --- a/sql/item_func.cc 2013-03-18 07:44:24 +0000 > +++ b/sql/item_func.cc 2013-03-19 10:51:39 +0000 > @@ -965,7 +965,11 @@ my_decimal *Item_func_numhybrid::val_dec > > void Item_func_signed::print(String *str, enum_query_type query_type) > { > - str->append(STRING_WITH_LEN("cast(")); > + if (args[0]->type() == FUNC_ITEM && > + ((Item_func *) args[0])->functype() == DYNCOL_GET_FUNC) > + str->append(STRING_WITH_LEN("column_get(")); > + else > + str->append(STRING_WITH_LEN("cast(")); > args[0]->print(str, query_type); > str->append(STRING_WITH_LEN(" as signed)")); So, as I said on irc, I don't quite like it. An Item should be independent from its arguments are should not change its name depending on the arguments. I've considered other approaches, like storing item_cast in the Item_dyncol_get or inheriting Item_dyncol_get from cast. But they all complicate the code notably. So, an easy fix that wouldn't contaminate item_cast, could be void Item_dyncol_get::print(String *str, enum_query_type query_type) { DBUG_ASSERT(strncmp(str->ptr() - 5, "cast(", 5) == 0); // see create_func_dyncol_get str->length(str->length() - 5); // removing "cast(" str->append(STRING_WITH_LEN("column_get(")); args[0]->print(str, query_type); str->append(','); args[1]->print(str, query_type); // let the parent cast item add " as <type>)" } Regards, Sergei

1 0

[Maria-developers] Aria/MyISAM and MySQL 5.6.x Compressed Table Compatibility
by Gordan Bobic 22 Mar '13

22 Mar '13

I'm having an issue with MariaDB 10.0.0. I have a bunch of archived compressed MyISAM tables from MySQL 5.6. I can move these around on MySQL 5.6 machines, but when I put them on MariaDB 10.0.0 it fails to recognize them. Has this been fixed in 10.0.1? If not, will it be fixed in 10.0.2? Gordan

2 1

[Maria-developers] Benchmark of connect performance
by Axel Schwenke 22 Mar '13

22 Mar '13

Hi, in the light of http://yoshinorimatsunobu.blogspot.fi/2012/12/non-persistent-connection-per… I did a similar benchmark. I used sysbench-0.4.12 for this with this command line $SYSBENCH --test=oltp --oltp-table-size=1000 --oltp-dist-type=uniform --num-threads=$thread --oltp-read-only=on --oltp-test-mode=simple --oltp-skip-trx --oltp-reconnect-mode=query --oltp-connect-delay=0 --db-ps-mode=disable --mysql-table-engine=InnoDB --mysql-engine-trx=no --mysql-socket=$SOCKET --max-time=60 --max-requests=0 --mysql-user=sysbench --mysql-password=sysbench --percentile=95 run This generates a workload of connect select .. where pk=const disconnect I varied the following: - number of concurrent threads - malloc library: glibc vs. tcmalloc - privileges given to sysbench@localhost * none (--skip-grant-tables) * global = ALL on *.* * db = ALL on sbtest.* * table = ALL on sbtest.sbtest I used the new 64 core machine for this. Everything was compiled locally with the REL_WITH_DEBINFO options. There is a rather huge dependence on the memory allocator. The results don't match Yoshis otherwise. See yourself. XL

3 6

[Maria-developers] Query Stuck in "closing table" State
by Gordan Bobic 20 Mar '13

20 Mar '13

I have a situation with MariaDB 10.0.0 where an INSERT INTO ... SELECT query that takes 3 seconds to run ends up stuck in the closing tables state for 15+ minutes while burning 100% of CPU. I have seen bug reports dating back 5 years that sound very similar but none of them are listed as resolved. Is there a way to debug/troubleshoot this further? The problem is quite intermittent. My data is in a state of constant flux so finding an exact reproducible test case based on data lone isn't likely to be viable. Is there a reasonable way to capture more detailed debug information than what is available from "show full processlist"? Gordan

3 7

Re: [Maria-developers] how to use Select INTO OUTFILE
by Michael Widenius 19 Mar '13

19 Mar '13

Hi! >>>>> "Irwan" == Irwan <irwan_forum(a)yahoo.co.id> writes: First a quick question, which windows version and which MariaDB version are you using? Irwan> how to use into outfile using mariadb to output in specified folder, Irwan> in mysql i use like this Irwan> SELECT * from SSD010.mastergudang INTO OUTFILE Irwan> 'C:/DOCUME~1/USER/LOCALS~1/Temp/17947F77/mastergudang' Irwan> but ini mariadb Irwan> SELECT * from SSD010.mastergudang INTO OUTFILE Irwan> 'C://DOCUME~1//USER//LOCALS~1//Temp//17947F77//mastergudang' Both MariaDB and MYSQL should use the identical syntax here. I wonder if this could be a difference in the clients? What happens if you try to use the MySQL command on windows? Irwan> the output not correct folder like this Irwan> http://prntscr.com/wu0n4 The above screen shows that the output is a into a file, data/DOCUME~1USERLOCALS~1Temp17947F77mastergudang' MariaDB should be able to use both / and \ on windows. In fact, our test suite uses '/' for creating files, not '//', so '/' should work. Irwan> when i use like this failed, Irwan> i'm sure the folder exist Irwan> SELECT * from SSD010.mastergudang INTO OUTFILE Irwan> 'C:\\DOCUME~1\\USER\\LOCALS~1\\Temp\\17947F77\\mastergudang' Irwan> http://prntscr.com/wtypr Errode 13, means 'Access denied'. So this means probably that the directory exists, but the MariaDB service doesn't have right to create a file in it. Wlad, our windows expert says "I think the service is not allowed to write into users temp directory, that's it. But some more context need to be given..." Question from Wlad: "Is mysqld running as service, and if yes, under which account (typically for us it is NetworkService)" ? "As a self-service, first aid to analyze such errors, user can use https://kb.askmonty.org/en/how-to-use-procmon-to-trace-mysqldexe-filesystem…" Hope the above helps. Regards, Monty

1 0

Re: [Maria-developers] utf16le
by Michael Widenius 19 Mar '13

19 Mar '13

Hi! I cc: mariadb-developers to ensure that others can also comment on the patch. Because the patch was not originally submitted there, I am not cutting out ok code parts as I usually do. >>>>> "Alexander" == Alexander Barkov <bar(a)mariadb.org> writes: Alexander> Hi Monty, Alexander> Please review a merge from MySQL-5.6 for utf16le. Alexander> I also merged a few related bug fixes, Alexander> which affected the test "ctype_utf16le": Alexander> - Unreadable warnings in CAST(<ucs2, utf16, utf32 value> AS SIGNED) Alexander> - Errors about unknown collation and unknown charset in cases Alexander> when the SET queries should actually succeed: Alexander> SET collation_connection=<ucs2, utf16, utf32 value>; Alexander> SET character_set_connection=<ucs2, utf16, utf32 value>; Alexander> - MySQL Bug#59145 valgrind warnings for uninitialized values in Alexander> my_strtoll10_mb2 Alexander> - MySQL Bug#32859 Character sets: no warning with non-fitting chariot wheel >---------------------------------------------------------------------- >=== modified file 'cmake/character_sets.cmake' >--- cmake/character_sets.cmake 2011-12-15 21:08:42 +0000 >+++ cmake/character_sets.cmake 2013-03-15 04:14:25 +0000 >@@ -25,14 +25,14 @@ > SET(CHARSETS ${DEFAULT_CHARSET} latin1 utf8 utf8mb4) > SET(CHARSETS_COMPLEX > big5 cp1250 cp932 eucjpms euckr gb2312 gbk latin1 latin2 >- sjis tis620 ucs2 ujis utf8 utf8mb4 utf16 utf32) >+ sjis tis620 ucs2 ujis utf8 utf8mb4 utf16 utf16le utf32) > > SET(CHARSETS_AVAILABLE > binary armscii8 ascii big5 cp1250 cp1251 cp1256 cp1257 > cp850 cp852 cp866 cp932 dec8 eucjpms euckr gb2312 gbk geostd8 > greek hebrew hp8 keybcs2 koi8r koi8u > latin1 latin2 latin5 latin7 macce macroman >-sjis swe7 tis620 ucs2 ujis utf8 utf8mb4 utf16 utf32) >+sjis swe7 tis620 ucs2 ujis utf8 utf8mb4 utf16 utf16le utf32) > > SET (EXTRA_CHARSETS "all") > >=== modified file 'include/m_ctype.h' >--- include/m_ctype.h 2012-04-07 13:58:46 +0000 >+++ include/m_ctype.h 2013-03-14 12:45:40 +0000 >@@ -362,6 +362,8 @@ > extern struct charset_info_st my_charset_utf16_bin; > extern struct charset_info_st my_charset_utf16_general_ci; > extern struct charset_info_st my_charset_utf16_unicode_ci; >+extern struct charset_info_st my_charset_utf16le_bin; >+extern struct charset_info_st my_charset_utf16le_general_ci; > extern struct charset_info_st my_charset_utf32_bin; > extern struct charset_info_st my_charset_utf32_general_ci; > extern struct charset_info_st my_charset_utf32_unicode_ci; > >=== added file 'mysql-test/include/ctype_heap.inc' >--- mysql-test/include/ctype_heap.inc 1970-01-01 00:00:00 +0000 >+++ mysql-test/include/ctype_heap.inc 2012-12-24 16:09:42 +0000 >@@ -0,0 +1,11 @@ >+SELECT @@collation_connection; >+CREATE TABLE t1 ENGINE=HEAP AS SELECT REPEAT (' ', 10) AS a LIMIT 0; >+ALTER TABLE t1 ADD KEY (a); >+CREATE TABLE t2 (a VARCHAR(10)); >+INSERT INTO t2 VALUES ('0'),('1'),('2'),('3'),('4'),('5'),('6'),('7'); >+INSERT INTO t2 VALUES ('8'),('9'),('A'),('B'),('C'),('D'),('E'),('F'); >+INSERT INTO t1 SELECT CONCAT('a',t21.a,t22.a) FROM t2 t21, t2 t22 ORDER BY 1; >+DROP TABLE t2; >+INSERT INTO t1 VALUES ('a '); >+SELECT a, HEX(a) FROM t1 WHERE a='a'; >+DROP TABLE t1; Monty: What new code / issue does the above test? Please add a comment before the test. >=== modified file 'mysql-test/r/ctype_ucs.result' >--- mysql-test/r/ctype_ucs.result 2013-01-15 18:13:32 +0000 >+++ mysql-test/r/ctype_ucs.result 2013-03-15 04:38:24 +0000 >@@ -1274,7 +1274,7 @@ > CONVERT(QUOTE(CHAR(0xf5 using ucs2)), SIGNED) > 0 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: ''Ãµ'' > End of 5.0 tests > # > # Start of 5.1 tests >@@ -4200,3 +4200,22 @@ > # > # End of 5.5 tests > # >+# >+# Start of 5.6 tests >+# >+# >+# Bug#59145 valgrind warnings for uninitialized values in my_strtoll10_mb2 >+# >+SET NAMES latin1; >+SELECT CONVERT(CHAR(NULL USING ucs2), UNSIGNED); >+CONVERT(CHAR(NULL USING ucs2), UNSIGNED) >+0 >+Warnings: >+Warning 1292 Truncated incorrect INTEGER value: '' >+DO IFNULL(CHAR(NULL USING ucs2), ''); >+DO CAST(CONVERT('' USING ucs2) AS UNSIGNED); >+Warnings: >+Warning 1292 Truncated incorrect INTEGER value: '' >+# >+# End of 5.6 tests >+# > >=== added file 'mysql-test/r/ctype_utf16le.result' >--- mysql-test/r/ctype_utf16le.result 1970-01-01 00:00:00 +0000 >+++ mysql-test/r/ctype_utf16le.result 2013-03-14 15:04:17 +0000 >@@ -0,0 +1,1394 @@ >+SET TIME_ZONE='+03:00'; >+DROP TABLE IF EXISTS t1; >+# >+# Start of 5.6 tests >+# >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+SELECT HEX('a'), HEX('a '); >+HEX('a') HEX('a ') >+6100 61002000 >+select 'a' = 'a', 'a' = 'a ', 'a ' = 'a'; >+'a' = 'a' 'a' = 'a ' 'a ' = 'a' >+1 1 1 >+select 'a\0' = 'a', 'a\0' < 'a', 'a\0' > 'a'; >+'a\0' = 'a' 'a\0' < 'a' 'a\0' > 'a' >+0 1 0 >+select 'a' = 'a\0', 'a' < 'a\0', 'a' > 'a\0'; >+'a' = 'a\0' 'a' < 'a\0' 'a' > 'a\0' >+0 0 1 >+select 'a\0' = 'a ', 'a\0' < 'a ', 'a\0' > 'a '; >+'a\0' = 'a ' 'a\0' < 'a ' 'a\0' > 'a ' >+0 1 0 >+select 'a ' = 'a\0', 'a ' < 'a\0', 'a ' > 'a\0'; >+'a ' = 'a\0' 'a ' < 'a\0' 'a ' > 'a\0' >+0 0 1 >+select 'a a' > 'a', 'a \0' < 'a'; >+'a a' > 'a' 'a \0' < 'a' >+1 1 >+select binary 'a a' > 'a', binary 'a \0' > 'a', binary 'a\0' > 'a'; >+binary 'a a' > 'a' binary 'a \0' > 'a' binary 'a\0' > 'a' >+1 1 1 >+# >+# Check that incomplete utf16le characters in HEX notation >+# are left-padded with zeros >+# >+SELECT HEX(_utf16le 0x44); >+HEX(_utf16le 0x44) >+0044 >+SELECT HEX(_utf16le 0x3344); >+HEX(_utf16le 0x3344) >+3344 >+SELECT HEX(_utf16le 0x113344); >+HEX(_utf16le 0x113344) >+00113344 >+# >+# Check that 0x20 is only trimmed when it is >+# a part of real SPACE character, not just a part >+# of a multibyte sequence. >+# Note, CYRILLIC LETTER ER is used as an example, which >+# is stored as 0x0420 in utf16le, thus contains 0x20 in the >+# low byte. The second character is THREE-PER-M, U+2004, >+# which contains 0x20 in the high byte. >+# >+CREATE TABLE t1 (word VARCHAR(64), word2 CHAR(64)) CHARACTER SET utf16le; >+INSERT INTO t1 VALUES (_koi8r 0xF2, _koi8r 0xF2), (_ucs2 X'2004',_ucs2 X'2004'); >+SELECT HEX(word) FROM t1 ORDER BY word; >+HEX(word) >+2004 >+0420 >+SELECT HEX(word2) FROM t1 ORDER BY word2; >+HEX(word2) >+2004 >+0420 >+DELETE FROM t1; >+# >+# Check that real spaces are correctly trimmed. >+# >+INSERT INTO t1 VALUES (_ucs2 X'042000200020', _ucs2 X'042000200020'); >+INSERT INTO t1 VALUES (_ucs2 X'200400200020', _ucs2 X'200400200020'); >+SELECT HEX(word) FROM t1 ORDER BY word; >+HEX(word) >+200420002000 >+042020002000 >+SELECT HEX(word2) FROM t1 ORDER BY word2; >+HEX(word2) >+2004 >+0420 >+DROP TABLE t1; >+# >+# Check LPAD/RPAD >+# >+CREATE TABLE t1 (a VARCHAR(10), pad INT, b VARCHAR(10)) CHARACTER SET utf16le; >+INSERT INTO t1 VALUES (_ucs2 X'0420', 10, _ucs2 X'0421'); >+INSERT INTO t1 VALUES (_ucs2 X'0420', 10, _ucs2 X'04210422'); >+INSERT INTO t1 VALUES (_ucs2 X'0420', 10, _ucs2 X'042104220423'); >+INSERT INTO t1 VALUES (_ucs2 X'0420042104220423042404250426042704280429042A042B',10,_ucs2 X'042104220423'); >+Warnings: >+Warning 1265 Data truncated for column 'a' at row 1 >+INSERT INTO t1 VALUES (_utf32 X'010000', 10, _ucs2 X'0421'); >+INSERT INTO t1 VALUES (_ucs2 X'0421', 10, _utf32 X'010000'); >+SELECT a, pad, b, LPAD(a, pad, b), HEX(LPAD(a, pad, b)) FROM t1; >+a pad b LPAD(a, pad, b) HEX(LPAD(a, pad, b)) >+Ð 10 Ð¡ Ð¡Ð¡Ð¡Ð¡Ð¡Ð¡Ð¡Ð¡Ð¡Ð 2104210421042104210421042104210421042004 >+Ð 10 Ð¡Ð¢ Ð¡Ð¢Ð¡Ð¢Ð¡Ð¢Ð¡Ð¢Ð¡Ð 2104220421042204210422042104220421042004 >+Ð 10 Ð¡Ð¢Ð£ Ð¡Ð¢Ð£Ð¡Ð¢Ð£Ð¡Ð¢Ð£Ð 2104220423042104220423042104220423042004 >+Ð Ð¡Ð¢Ð£Ð¤Ð¥Ð¦Ð§Ð¨Ð© 10 Ð¡Ð¢Ð£ Ð Ð¡Ð¢Ð£Ð¤Ð¥Ð¦Ð§Ð¨Ð© 2004210422042304240425042604270428042904 >+? 10 Ð¡ Ð¡Ð¡Ð¡Ð¡Ð¡Ð¡Ð¡Ð¡Ð¡? 21042104210421042104210421042104210400D800DC >+Ð¡ 10 ? ?????????Ð¡ 00D800DC00D800DC00D800DC00D800DC00D800DC00D800DC00D800DC00D800DC00D800DC2104 Monty: Is the last line with '?' right Note that for test cases, it may sometimes be better to just output the HEX part. Result files with 'wrong' letters will sooner or later cause a problem when someone tries to edit the file for fixing a test result case as the editor may 'scramble some characters. >+DROP TABLE t1; >+CREATE TABLE t1 SELECT >+LPAD(_utf16le X'2004',10,_utf16le X'2104') l, >+RPAD(_utf16le X'2004',10,_utf16le X'2104') r; >+SHOW CREATE TABLE t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `l` varchar(10) CHARACTER SET utf16le NOT NULL DEFAULT '', >+ `r` varchar(10) CHARACTER SET utf16le NOT NULL DEFAULT '' >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+SELECT HEX(l), HEX(r) FROM t1; >+HEX(l) HEX(r) >+2104210421042104210421042104210421042004 2004210421042104210421042104210421042104 >+DROP TABLE t1; >+CREATE TABLE t1 (f1 CHAR(30)); >+INSERT INTO t1 VALUES ("103000"), ("22720000"), ("3401200"), ("78000"); >+SELECT LPAD(f1, 12, "-o-/") FROM t1; >+LPAD(f1, 12, "-o-/") >+-o-/-o103000 >+-o-/22720000 >+-o-/-3401200 >+-o-/-o-78000 >+DROP TABLE t1; >+# >+# Testing LIKE >+# >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+select @@collation_connection; >+@@collation_connection >+utf16le_general_ci >+create table t1 as select repeat(' ',10) as a union select null; >+alter table t1 add key(a); >+show create table t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `a` varchar(10) CHARACTER SET utf16le DEFAULT NULL, >+ KEY `a` (`a`) >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+insert into t1 values ("a"),("abc"),("abcd"),("hello"),("test"); >+explain select * from t1 where a like 'abc%'; >+id select_type table type possible_keys key key_len ref rows Extra >+1 SIMPLE t1 range a a 43 NULL 1 Using where; Using index >+explain select * from t1 where a like concat('abc','%'); >+id select_type table type possible_keys key key_len ref rows Extra >+1 SIMPLE t1 range a a 43 NULL 1 Using where; Using index >+select * from t1 where a like "abc%"; >+a >+abc >+abcd >+select * from t1 where a like concat("abc","%"); >+a >+abc >+abcd >+select * from t1 where a like "ABC%"; >+a >+abc >+abcd >+select * from t1 where a like "test%"; >+a >+test >+select * from t1 where a like "te_t"; >+a >+test >+select * from t1 where a like "%a%"; >+a >+a >+abc >+abcd >+select * from t1 where a like "%abcd%"; >+a >+abcd >+select * from t1 where a like "%abc\d%"; >+a >+abcd >+drop table t1; >+select 'AA' like 'AA'; >+'AA' like 'AA' >+1 >+select 'AA' like 'A%A'; >+'AA' like 'A%A' >+1 >+select 'AA' like 'A%%A'; >+'AA' like 'A%%A' >+1 >+select 'AA' like 'AA%'; >+'AA' like 'AA%' >+1 >+select 'AA' like '%AA%'; >+'AA' like '%AA%' >+1 >+select 'AA' like '%A'; >+'AA' like '%A' >+1 >+select 'AA' like '%AA'; >+'AA' like '%AA' >+1 >+select 'AA' like 'A%A%'; >+'AA' like 'A%A%' >+1 >+select 'AA' like '_%_%'; >+'AA' like '_%_%' >+1 >+select 'AA' like '%A%A'; >+'AA' like '%A%A' >+1 >+select 'AAA'like 'A%A%A'; >+'AAA'like 'A%A%A' >+1 >+select 'AZ' like 'AZ'; >+'AZ' like 'AZ' >+1 >+select 'AZ' like 'A%Z'; >+'AZ' like 'A%Z' >+1 >+select 'AZ' like 'A%%Z'; >+'AZ' like 'A%%Z' >+1 >+select 'AZ' like 'AZ%'; >+'AZ' like 'AZ%' >+1 >+select 'AZ' like '%AZ%'; >+'AZ' like '%AZ%' >+1 >+select 'AZ' like '%Z'; >+'AZ' like '%Z' >+1 >+select 'AZ' like '%AZ'; >+'AZ' like '%AZ' >+1 >+select 'AZ' like 'A%Z%'; >+'AZ' like 'A%Z%' >+1 >+select 'AZ' like '_%_%'; >+'AZ' like '_%_%' >+1 >+select 'AZ' like '%A%Z'; >+'AZ' like '%A%Z' >+1 >+select 'AZ' like 'A_'; >+'AZ' like 'A_' >+1 >+select 'AZ' like '_Z'; >+'AZ' like '_Z' >+1 >+select 'AMZ'like 'A%M%Z'; >+'AMZ'like 'A%M%Z' >+1 >+CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES ('ÑÑÐ²Ð°'),('Ð¤ÑÐ²Ð°'),('ÑÐ«Ð²Ð°'),('ÑÑÐÐ°'),('ÑÑÐ²Ð'),('Ð¤Ð«ÐÐ'); >+INSERT INTO t1 VALUES ('ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶'),('Ð¤ÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶'),('ÑÐ«Ð²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶'),('ÑÑÐÐ°Ð¿ÑÐ¾Ð»Ð´Ð¶'); >+INSERT INTO t1 VALUES ('ÑÑÐ²ÐÐ¿ÑÐ¾Ð»Ð´Ð¶'),('ÑÑÐ²Ð°ÐÑÐ¾Ð»Ð´Ð¶'),('ÑÑÐ²Ð°Ð¿Ð Ð¾Ð»Ð´Ð¶'),('ÑÑÐ²Ð°Ð¿ÑÐÐ»Ð´Ð¶'); >+INSERT INTO t1 VALUES ('ÑÑÐ²Ð°Ð¿ÑÐ¾ÐÐ´Ð¶'),('ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»ÐÐ¶'),('ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð'),('Ð¤Ð«ÐÐÐÐ ÐÐÐÐ'); >+SELECT * FROM t1 WHERE a LIKE '%ÑÐ«Ð²Ð°%' ORDER BY BINARY a; >+a >+Ð¤Ð«ÐÐ >+Ð¤Ð«ÐÐÐÐ ÐÐÐÐ >+Ð¤ÑÐ²Ð° >+Ð¤ÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÐ«Ð²Ð° >+ÑÐ«Ð²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐÐ° >+ÑÑÐÐ°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð >+ÑÑÐ²ÐÐ¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð° >+ÑÑÐ²Ð°ÐÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð°Ð¿Ð Ð¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð°Ð¿ÑÐÐ»Ð´Ð¶ >+ÑÑÐ²Ð°Ð¿ÑÐ¾ÐÐ´Ð¶ >+ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»ÐÐ¶ >+ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð >+ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+SELECT * FROM t1 WHERE a LIKE '%ÑÐ«Ð²%' ORDER BY BINARY a; >+a >+Ð¤Ð«ÐÐ >+Ð¤Ð«ÐÐÐÐ ÐÐÐÐ >+Ð¤ÑÐ²Ð° >+Ð¤ÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÐ«Ð²Ð° >+ÑÐ«Ð²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐÐ° >+ÑÑÐÐ°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð >+ÑÑÐ²ÐÐ¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð° >+ÑÑÐ²Ð°ÐÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð°Ð¿Ð Ð¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð°Ð¿ÑÐÐ»Ð´Ð¶ >+ÑÑÐ²Ð°Ð¿ÑÐ¾ÐÐ´Ð¶ >+ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»ÐÐ¶ >+ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð >+ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+SELECT * FROM t1 WHERE a LIKE 'ÑÐ«Ð²Ð°%' ORDER BY BINARY a; >+a >+Ð¤Ð«ÐÐ >+Ð¤Ð«ÐÐÐÐ ÐÐÐÐ >+Ð¤ÑÐ²Ð° >+Ð¤ÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÐ«Ð²Ð° >+ÑÐ«Ð²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐÐ° >+ÑÑÐÐ°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð >+ÑÑÐ²ÐÐ¿ÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð° >+ÑÑÐ²Ð°ÐÑÐ¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð°Ð¿Ð Ð¾Ð»Ð´Ð¶ >+ÑÑÐ²Ð°Ð¿ÑÐÐ»Ð´Ð¶ >+ÑÑÐ²Ð°Ð¿ÑÐ¾ÐÐ´Ð¶ >+ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»ÐÐ¶ >+ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð >+ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+SELECT * FROM t1 WHERE a LIKE 'ÑÐ«Ð²Ð°%' COLLATE utf16le_bin ORDER BY BINARY a; >+a >+ÑÐ«Ð²Ð° >+ÑÐ«Ð²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶ >+DROP TABLE t1; >+CREATE TABLE t1 (word VARCHAR(64) NOT NULL, PRIMARY KEY (word)) >+ENGINE=MyISAM CHARACTER SET utf16le; >+INSERT INTO t1 (word) VALUES ("cat"); >+SELECT * FROM t1 WHERE word LIKE "c%"; >+word >+cat >+SELECT * FROM t1 WHERE word LIKE "ca_"; >+word >+cat >+SELECT * FROM t1 WHERE word LIKE "cat"; >+word >+cat >+SELECT * FROM t1 WHERE word LIKE _ucs2 x'00630025'; >+word >+cat >+SELECT * FROM t1 WHERE word LIKE _ucs2 x'00630061005F'; >+word >+cat >+DROP TABLE t1; >+# >+# Check that INSERT() works fine. >+# This invokes charpos() function. >+# >+CREATE TABLE t1 ( >+a VARCHAR(10) CHARACTER SET utf16le, >+b VARCHAR(10) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES ('abc', 'def'); >+SELECT INSERT(a, 10, 2, b) FROM t1; >+INSERT(a, 10, 2, b) >+abc >+SELECT INSERT(a, 1, 2, b) FROM t1; >+INSERT(a, 1, 2, b) >+defc >+DROP TABLE t1; >+# >+# Bug#1264 >+# >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+# >+# Two fields, index >+# >+CREATE TABLE t1 ( >+word VARCHAR(64), >+bar INT(11) DEFAULT 0, >+PRIMARY KEY (word)) >+ENGINE=MyISAM >+CHARSET utf16le >+COLLATE utf16le_general_ci ; >+INSERT INTO t1 (word) VALUES ("aar"); >+INSERT INTO t1 (word) VALUES ("a"); >+INSERT INTO t1 (word) VALUES ("aardvar"); >+INSERT INTO t1 (word) VALUES ("aardvark"); >+INSERT INTO t1 (word) VALUES ("aardvara"); >+INSERT INTO t1 (word) VALUES ("aardvarz"); >+EXPLAIN SELECT * FROM t1 ORDER BY word; >+id select_type table type possible_keys key key_len ref rows Extra >+1 SIMPLE t1 ALL NULL NULL NULL NULL 6 Using filesort >+SELECT * FROM t1 ORDER BY word; >+word bar >+a 0 >+aar 0 >+aardvar 0 >+aardvara 0 >+aardvark 0 >+aardvarz 0 >+EXPLAIN SELECT word FROM t1 ORDER BY word; >+id select_type table type possible_keys key key_len ref rows Extra >+1 SIMPLE t1 index NULL PRIMARY 258 NULL 6 Using index >+SELECT word FROM t1 ORDER by word; >+word >+a >+aar >+aardvar >+aardvara >+aardvark >+aardvarz >+DROP TABLE t1; >+# >+# One field, index >+# >+CREATE TABLE t1 ( >+word VARCHAR(64) , >+PRIMARY KEY (word)) >+ENGINE=MyISAM >+CHARSET utf16le >+COLLATE utf16le_general_ci; >+INSERT INTO t1 (word) VALUES ("aar"); >+INSERT INTO t1 (word) VALUES ("a"); >+INSERT INTO t1 (word) VALUES ("aardvar"); >+INSERT INTO t1 (word) VALUES ("aardvark"); >+INSERT INTO t1 (word) VALUES ("aardvara"); >+INSERT INTO t1 (word) VALUES ("aardvarz"); >+EXPLAIN SELECT * FROM t1 ORDER BY WORD; >+id select_type table type possible_keys key key_len ref rows Extra >+1 SIMPLE t1 index NULL PRIMARY 258 NULL 6 Using index >+SELECT * FROM t1 ORDER BY word; >+word >+a >+aar >+aardvar >+aardvara >+aardvark >+aardvarz >+DROP TABLE t1; >+# >+# Two fields, no index >+# >+CREATE TABLE t1 ( >+word TEXT, >+bar INT(11) AUTO_INCREMENT, >+PRIMARY KEY (bar)) >+ENGINE=MyISAM >+CHARSET utf16le >+COLLATE utf16le_general_ci ; >+INSERT INTO t1 (word) VALUES ("aar"); >+INSERT INTO t1 (word) VALUES ("a" ); >+INSERT INTO t1 (word) VALUES ("aardvar"); >+INSERT INTO t1 (word) VALUES ("aardvark"); >+INSERT INTO t1 (word) VALUES ("aardvara"); >+INSERT INTO t1 (word) VALUES ("aardvarz"); >+EXPLAIN SELECT * FROM t1 ORDER BY word; >+id select_type table type possible_keys key key_len ref rows Extra >+1 SIMPLE t1 ALL NULL NULL NULL NULL 6 Using filesort >+SELECT * FROM t1 ORDER BY word; >+word bar >+a 2 >+aar 1 >+aardvar 3 >+aardvara 5 >+aardvark 4 >+aardvarz 6 >+EXPLAIN SELECT word FROM t1 ORDER BY word; >+id select_type table type possible_keys key key_len ref rows Extra >+1 SIMPLE t1 ALL NULL NULL NULL NULL 6 Using filesort >+SELECT word FROM t1 ORDER BY word; >+word >+a >+aar >+aardvar >+aardvara >+aardvark >+aardvarz >+DROP TABLE t1; >+# >+# END OF Bug 1264 test >+# >+# >+# Check alignment for from-binary-conversion with CAST and CONVERT >+# >+SELECT HEX(CAST(0xAA as char CHARACTER SET utf16le)); >+HEX(CAST(0xAA as char CHARACTER SET utf16le)) >+00AA >+SELECT HEX(CONVERT(0xAA USING utf16le)); >+HEX(CONVERT(0xAA USING utf16le)) >+00AA >+# >+# Check alignment for string types >+# >+CREATE TABLE t1 (a CHAR(10) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x1),(0x11),(0x111),(0x1111),(0x11111); >+SELECT HEX(a) FROM t1; >+HEX(a) >+0001 >+0011 >+0111 >+1111 >+00011111 >+DROP TABLE t1; >+CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x1),(0x11),(0x111),(0x1111),(0x11111); >+SELECT HEX(a) FROM t1; >+HEX(a) >+0001 >+0011 >+0111 >+1111 >+00011111 >+DROP TABLE t1; >+CREATE TABLE t1 (a TEXT CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x1),(0x11),(0x111),(0x1111),(0x11111); >+SELECT HEX(a) FROM t1; >+HEX(a) >+0001 >+0011 >+0111 >+1111 >+00011111 >+DROP TABLE t1; >+CREATE TABLE t1 (a MEDIUMTEXT CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x1),(0x11),(0x111),(0x1111),(0x11111); >+SELECT HEX(a) FROM t1; >+HEX(a) >+0001 >+0011 >+0111 >+1111 >+00011111 >+DROP TABLE t1; >+CREATE TABLE t1 (a LONGTEXT CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x1),(0x11),(0x111),(0x1111),(0x11111); >+SELECT HEX(a) FROM t1; >+HEX(a) >+0001 >+0011 >+0111 >+1111 >+00011111 >+DROP TABLE t1; >+# >+# Bug#5081 : UCS2 fields are filled with '0x2020' >+# after extending field length >+# >+CREATE TABLE t1(a CHAR(1)) DEFAULT CHARSET utf16le; >+INSERT INTO t1 VALUES ('a'),('b'),('c'); >+ALTER TABLE t1 MODIFY a CHAR(5); >+SELECT a, HEX(a) FROM t1; >+a HEX(a) >+a 6100 >+b 6200 >+c 6300 >+DROP TABLE t1; >+# >+# Check prepare statement from an UTF16 string >+# >+SET NAMES latin1; >+SET @ivar= 1234; >+SET @str1 = 'SELECT ?'; >+SET @str2 = CONVERT(@str1 USING utf16le); >+PREPARE stmt1 FROM @str2; >+EXECUTE stmt1 USING @ivar; >+? >+1234 >+# >+# Check that utf16le works with ENUM and SET type >+# >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+CREATE TABLE t1 (a ENUM('x','y','z') CHARACTER SET utf16le); >+SHOW CREATE TABLE t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `a` enum('x','y','z') CHARACTER SET utf16le DEFAULT NULL >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+INSERT INTO t1 VALUES ('x'); >+INSERT INTO t1 VALUES ('y'); >+INSERT INTO t1 VALUES ('z'); >+SELECT a, HEX(a) FROM t1 ORDER BY a; >+a HEX(a) >+x 7800 >+y 7900 >+z 7A00 >+ALTER TABLE t1 CHANGE a a ENUM('x','y','z','d','e','Ã¤','Ã¶','Ã¼') CHARACTER SET utf16le; >+SHOW CREATE TABLE t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `a` enum('x','y','z','d','e','Ã¤','Ã¶','Ã¼') CHARACTER SET utf16le DEFAULT NULL >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+INSERT INTO t1 VALUES ('D'); >+INSERT INTO t1 VALUES ('E '); >+INSERT INTO t1 VALUES ('Ã¤'); >+INSERT INTO t1 VALUES ('Ã¶'); >+INSERT INTO t1 VALUES ('Ã¼'); >+SELECT a, HEX(a) FROM t1 ORDER BY a; >+a HEX(a) >+x 7800 >+y 7900 >+z 7A00 >+d 6400 >+e 6500 >+Ã¤ E400 >+Ã¶ F600 >+Ã¼ FC00 >+DROP TABLE t1; >+CREATE TABLE t1 (a set ('x','y','z','Ã¤','Ã¶','Ã¼') CHARACTER SET utf16le); >+SHOW CREATE TABLE t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `a` set('x','y','z','Ã¤','Ã¶','Ã¼') CHARACTER SET utf16le DEFAULT NULL >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+INSERT INTO t1 VALUES ('x'); >+INSERT INTO t1 VALUES ('y'); >+INSERT INTO t1 VALUES ('z'); >+INSERT INTO t1 VALUES ('x,y'); >+INSERT INTO t1 VALUES ('x,y,z,Ã¤,Ã¶,Ã¼'); >+SELECT a, HEX(a) FROM t1 ORDER BY a; >+a HEX(a) >+x 7800 >+y 7900 >+x,y 78002C007900 >+z 7A00 >+x,y,z,Ã¤,Ã¶,Ã¼ 78002C0079002C007A002C00E4002C00F6002C00FC00 >+DROP TABLE t1; >+# >+# Bug#7302 UCS2 data in ENUM fields get truncated when new column is added >+# >+CREATE TABLE t1(a ENUM('a','b','c')) DEFAULT CHARACTER SET utf16le; >+INSERT INTO t1 VALUES('a'),('b'),('c'); >+ALTER TABLE t1 ADD b CHAR(1); >+SHOW WARNINGS; >+Level Code Message >+SELECT * FROM t1 ORDER BY a; >+a b >+a NULL >+b NULL >+c NULL >+DROP TABLE t1; >+SET NAMES utf8, collation_connection='utf16le_general_ci'; >+create table t1 select repeat('a',4000) a; >+delete from t1; >+insert into t1 values ('a'), ('a '), ('a\t'); >+select collation(a),hex(a) from t1 order by a; >+collation(a) hex(a) >+utf16le_general_ci 61000900 >+utf16le_general_ci 6100 >+utf16le_general_ci 61002000 >+drop table t1; >+select @@collation_connection; >+@@collation_connection >+utf16le_general_ci >+create table t1 ROW_FORMAT=DYNAMIC select repeat('a',50) as c1 ; >+insert into t1 values('abcdef'); >+insert into t1 values('_bcdef'); >+insert into t1 values('a_cdef'); >+insert into t1 values('ab_def'); >+insert into t1 values('abc_ef'); >+insert into t1 values('abcd_f'); >+insert into t1 values('abcde_'); >+select c1 as c1u from t1 where c1 like 'ab\_def'; >+c1u >+ab_def >+select c1 as c2h from t1 where c1 like 'ab#_def' escape '#'; >+c2h >+ab_def >+drop table t1; >+SET NAMES utf8, collation_connection='utf16le_bin'; >+create table t1 select repeat('a',4000) a; >+delete from t1; >+insert into t1 values ('a'), ('a '), ('a\t'); >+select collation(a),hex(a) from t1 order by a; >+collation(a) hex(a) >+utf16le_bin 61000900 >+utf16le_bin 6100 >+utf16le_bin 61002000 >+drop table t1; >+# >+# Bug#55980 Character sets: supplementary character _bin ordering is wrong >+# >+CREATE TABLE t1 AS SELECT REPEAT('a',1) AS a LIMIT 0; >+SHOW CREATE TABLE t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `a` varchar(1) CHARACTER SET utf16le COLLATE utf16le_bin NOT NULL DEFAULT '' >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+INSERT INTO t1 VALUES (_utf8mb4 0xEFBE9D),(_utf8mb4 0xF0908E84); >+INSERT INTO t1 VALUES (_utf8mb4 0xCE85),(_utf8mb4 0xF4808080); >+SELECT HEX(a), HEX(CONVERT(a USING utf8mb4)) FROM t1 ORDER BY a; >+HEX(a) HEX(CONVERT(a USING utf8mb4)) >+8503 CE85 >+9DFF EFBE9D >+00D884DF F0908E84 >+C0DB00DC F4808080 >+ALTER TABLE t1 ADD KEY(a); >+SELECT HEX(a), HEX(CONVERT(a USING utf8mb4)) FROM t1 ORDER BY a; >+HEX(a) HEX(CONVERT(a USING utf8mb4)) >+8503 CE85 >+9DFF EFBE9D >+00D884DF F0908E84 >+C0DB00DC F4808080 >+DROP TABLE IF EXISTS t1; >+select @@collation_connection; >+@@collation_connection >+utf16le_bin >+create table t1 ROW_FORMAT=DYNAMIC select repeat('a',50) as c1 ; >+insert into t1 values('abcdef'); >+insert into t1 values('_bcdef'); >+insert into t1 values('a_cdef'); >+insert into t1 values('ab_def'); >+insert into t1 values('abc_ef'); >+insert into t1 values('abcd_f'); >+insert into t1 values('abcde_'); >+select c1 as c1u from t1 where c1 like 'ab\_def'; >+c1u >+ab_def >+select c1 as c2h from t1 where c1 like 'ab#_def' escape '#'; >+c2h >+ab_def >+drop table t1; >+# >+# Bug#10344 Some string functions fail for UCS2 >+# >+CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf16le, pos INT); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,1); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,2); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,3); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,-1); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,-2); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,-3); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, 1); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, 2); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, 3); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, -1); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, -2); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, -3); >+SELECT HEX(SUBSTR(a, pos)), SUBSTR(a, pos) FROM t1; >+HEX(SUBSTR(a, pos)) SUBSTR(a, pos) >+E400E5006800 Ã¤Ã¥h >+E5006800 Ã¥h >+6800 h >+6800 h >+E5006800 Ã¥h >+E400E5006800 Ã¤Ã¥h >+E400E50000D800DC Ã¤Ã¥? >+E50000D800DC Ã¥? >+00D800DC ? >+00D800DC ? >+E50000D800DC Ã¥? >+E400E50000D800DC Ã¤Ã¥? >+DROP TABLE t1; >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+# >+# Bug#9442 Set parameter make query fail if column CHARACTER SET is UCS2 >+# >+CREATE TABLE t1 (utext VARCHAR(20) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES ("lily"); >+INSERT INTO t1 VALUES ("river"); >+PREPARE stmt FROM 'SELECT utext FROM t1 where utext like ?'; >+SET @param1='%%'; >+EXECUTE stmt USING @param1; >+utext >+lily >+river >+EXECUTE stmt USING @param1; >+utext >+lily >+river >+SELECT utext FROM t1 where utext like '%%'; >+utext >+lily >+river >+DROP TABLE t1; >+DEALLOCATE PREPARE stmt; >+# >+# Bug#22052 Trailing spaces are not removed FROM UNICODE fields in an index >+# >+CREATE TABLE t1 ( >+a CHAR(10) CHARACTER SET utf16le NOT NULL, >+INDEX a (a) >+) engine=myisam; >+INSERT INTO t1 VALUES (REPEAT(_ucs2 0x201f, 10)); >+INSERT INTO t1 VALUES (REPEAT(_ucs2 0x2020, 10)); >+INSERT INTO t1 VALUES (REPEAT(_ucs2 0x2021, 10)); >+# make sure "index read" is used >+explain SELECT HEX(a) FROM t1 ORDER BY a; >+id select_type table type possible_keys key key_len ref rows Extra >+1 SIMPLE t1 index NULL a 40 NULL 3 Using index >+SELECT HEX(a) FROM t1 ORDER BY a; >+HEX(a) >+1F201F201F201F201F201F201F201F201F201F20 >+2020202020202020202020202020202020202020 >+2120212021202120212021202120212021202120 >+ALTER TABLE t1 DROP INDEX a; >+SELECT HEX(a) FROM t1 ORDER BY a; >+HEX(a) >+1F201F201F201F201F201F201F201F201F201F20 >+2020202020202020202020202020202020202020 >+2120212021202120212021202120212021202120 >+DROP TABLE t1; >+# >+# Bug #20108: corrupted default enum value for a ucs2 field >+# >+CREATE TABLE t1 ( >+status ENUM('active','passive') CHARACTER SET utf16le COLLATE utf16le_general_ci >+NOT NULL DEFAULT 'passive' >+); >+SHOW CREATE TABLE t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `status` enum('active','passive') CHARACTER SET utf16le NOT NULL DEFAULT 'passive' >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+ALTER TABLE t1 ADD a int NOT NULL AFTER status; >+SHOW CREATE TABLE t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `status` enum('active','passive') CHARACTER SET utf16le NOT NULL DEFAULT 'passive', >+ `a` int(11) NOT NULL >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+DROP TABLE t1; >+# >+# Conversion FROM an UTF16LE string to a decimal column >+# >+CREATE TABLE t1 (a VARCHAR(64) CHARACTER SET utf16le, b DECIMAL(10,3)); >+INSERT INTO t1 VALUES ("1.1", 0), ("2.1", 0); >+UPDATE t1 set b=a; >+SELECT *, HEX(a) FROM t1; >+a b HEX(a) >+1.1 1.100 31002E003100 >+2.1 2.100 32002E003100 >+DROP TABLE t1; >+# >+# Bug#9442 Set parameter make query fail if column CHARACTER SET is UCS2 >+# >+CREATE TABLE t1 (utext VARCHAR(20) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES ("lily"); >+INSERT INTO t1 VALUES ("river"); >+PREPARE stmt FROM 'SELECT utext FROM t1 where utext like ?'; >+SET @param1='%%'; >+EXECUTE stmt USING @param1; >+utext >+lily >+river >+EXECUTE stmt USING @param1; >+utext >+lily >+river >+SELECT utext FROM t1 where utext like '%%'; >+utext >+lily >+river >+DROP TABLE t1; >+DEALLOCATE PREPARE stmt; >+# >+# Bug#22638 SOUNDEX broken for international characters >+# >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+SELECT SOUNDEX(''),SOUNDEX('he'),SOUNDEX('hello all folks'),SOUNDEX('#3556 in bugdb'); >+SOUNDEX('') SOUNDEX('he') SOUNDEX('hello all folks') SOUNDEX('#3556 in bugdb') >+ H000 H4142 I51231 >+SELECT HEX(SOUNDEX('')),HEX(SOUNDEX('he')),HEX(SOUNDEX('hello all folks')),HEX(SOUNDEX('#3556 in bugdb')); >+HEX(SOUNDEX('')) HEX(SOUNDEX('he')) HEX(SOUNDEX('hello all folks')) HEX(SOUNDEX('#3556 in bugdb')) >+ 4800300030003000 48003400310034003200 490035003100320033003100 >+SELECT 'mood' sounds like 'mud'; >+'mood' sounds like 'mud' >+1 >+# Cyrillic A, BE, VE >+SELECT HEX(SOUNDEX(_utf16le 0x041004110412)); >+HEX(SOUNDEX(_utf16le 0x041004110412)) >+0410300030003000 >+# Make sure that "U+00BF INVERTED QUESTION MARK" is not considered as letter >+SELECT HEX(SOUNDEX(_utf16le 0x00BF00C0)); >+HEX(SOUNDEX(_utf16le 0x00BF00C0)) >+00BF300030003000 >+# >+# Bug#14290: character_maximum_length for text fields >+# >+CREATE TABLE t1(a BLOB, b TEXT CHARSET utf16le); >+SELECT data_type, character_octet_length, character_maximum_length >+FROM information_schema.columns where table_name='t1'; >+data_type character_octet_length character_maximum_length >+blob 65535 65535 >+text 65535 32767 >+DROP TABLE t1; >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+# >+# Testing cs->coll->instr() >+# >+SELECT POSITION('bb' IN 'abba'); >+POSITION('bb' IN 'abba') >+2 >+# >+# Testing cs->coll->hash_sort() >+# >+SET NAMES utf8, collation_connection=utf16le_bin; >+SELECT @@collation_connection; >+@@collation_connection >+utf16le_bin >+CREATE TABLE t1 ENGINE=HEAP AS SELECT REPEAT (' ', 10) AS a LIMIT 0; >+ALTER TABLE t1 ADD KEY (a); >+CREATE TABLE t2 (a VARCHAR(10)); >+INSERT INTO t2 VALUES ('0'),('1'),('2'),('3'),('4'),('5'),('6'),('7'); >+INSERT INTO t2 VALUES ('8'),('9'),('A'),('B'),('C'),('D'),('E'),('F'); >+INSERT INTO t1 SELECT CONCAT('a',t21.a,t22.a) FROM t2 t21, t2 t22 ORDER BY 1; >+DROP TABLE t2; >+INSERT INTO t1 VALUES ('a '); >+SELECT a, HEX(a) FROM t1 WHERE a='a'; >+a HEX(a) >+a 61002000 >+DROP TABLE t1; >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+SELECT @@collation_connection; >+@@collation_connection >+utf16le_general_ci >+CREATE TABLE t1 ENGINE=HEAP AS SELECT REPEAT (' ', 10) AS a LIMIT 0; >+ALTER TABLE t1 ADD KEY (a); >+CREATE TABLE t2 (a VARCHAR(10)); >+INSERT INTO t2 VALUES ('0'),('1'),('2'),('3'),('4'),('5'),('6'),('7'); >+INSERT INTO t2 VALUES ('8'),('9'),('A'),('B'),('C'),('D'),('E'),('F'); >+INSERT INTO t1 SELECT CONCAT('a',t21.a,t22.a) FROM t2 t21, t2 t22 ORDER BY 1; >+DROP TABLE t2; >+INSERT INTO t1 VALUES ('a '); >+SELECT a, HEX(a) FROM t1 WHERE a='a'; >+a HEX(a) >+a 61002000 >+DROP TABLE t1; >+# >+# Testing cs->cset->numchars() >+# >+SELECT CHAR_LENGTH('abcd'), OCTET_LENGTH('abcd'); >+CHAR_LENGTH('abcd') OCTET_LENGTH('abcd') >+4 8 >+SELECT CHAR_LENGTH(_utf16le 0x00D800DC), OCTET_LENGTH(_utf16le 0x00D800DC); >+CHAR_LENGTH(_utf16le 0x00D800DC) OCTET_LENGTH(_utf16le 0x00D800DC) >+1 4 >+SELECT CHAR_LENGTH(_utf16le 0x7DD8FFDF), OCTET_LENGTH(_utf16le 0x7FD8DDDF); >+CHAR_LENGTH(_utf16le 0x7DD8FFDF) OCTET_LENGTH(_utf16le 0x7FD8DDDF) >+1 4 >+# >+# Testing cs->cset->charpos() >+# >+SELECT LEFT('abcd',2); >+LEFT('abcd',2) >+ab >+SELECT HEX(LEFT(_utf16le 0x00D800DC7FD8FFDF, 1)); >+HEX(LEFT(_utf16le 0x00D800DC7FD8FFDF, 1)) >+00D800DC >+SELECT HEX(RIGHT(_utf16le 0x00D800DC7FD8FFDF, 1)); >+HEX(RIGHT(_utf16le 0x00D800DC7FD8FFDF, 1)) >+7FD8FFDF >+# >+# Testing cs->cset->well_formed_length() >+# >+CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf16le); >+# Bad sequences >+INSERT INTO t1 VALUES (_utf16le 0x00D8); >+ERROR HY000: Invalid utf16le character string: '00D8' >+INSERT INTO t1 VALUES (_utf16le 0x00DC); >+ERROR HY000: Invalid utf16le character string: '00DC' >+INSERT INTO t1 VALUES (_utf16le 0x00D800D8); >+ERROR HY000: Invalid utf16le character string: '00D800' >+INSERT INTO t1 VALUES (_utf16le 0x00D800E8); >+ERROR HY000: Invalid utf16le character string: '00D800' >+INSERT INTO t1 VALUES (_utf16le 0x00D80008); >+ERROR HY000: Invalid utf16le character string: '00D800' >+# Good sequences >+INSERT INTO t1 VALUES (_utf16le 0x00D800DC); >+INSERT INTO t1 VALUES (_utf16le 0x00D8FFDC); >+INSERT INTO t1 VALUES (_utf16le 0xFFDB00DC); >+INSERT INTO t1 VALUES (_utf16le 0xFFDBFFDC); >+SELECT HEX(a) FROM t1; >+HEX(a) >+00D800DC >+00D8FFDC >+FFDB00DC >+FFDBFFDC >+DROP TABLE t1; >+# >+# Bug#32393 Character sets: illegal characters in utf16le columns >+# >+# Tests that cs->cset->wc_mb() doesn't accept surrogate parts >+# >+# via ALTER >+# >+CREATE TABLE t1 (s1 VARCHAR(50) CHARACTER SET ucs2); >+INSERT INTO t1 VALUES (0xDF84); >+ALTER TABLE t1 MODIFY column s1 VARCHAR(50) CHARACTER SET utf16le; >+Warnings: >+Warning 1366 Incorrect string value: '\xDF\x84' for column 's1' at row 1 >+SELECT HEX(s1) FROM t1; >+HEX(s1) >+3F00 >+DROP TABLE t1; >+# >+# via UPDATE >+# >+CREATE TABLE t1 (s1 VARCHAR(5) CHARACTER SET ucs2, s2 VARCHAR(5) CHARACTER SET utf16le); >+INSERT INTO t1 (s1) VALUES (0xdf84); >+UPDATE t1 set s2 = s1; >+Warnings: >+Warning 1366 Incorrect string value: '\xDF\x84' for column 's2' at row 1 >+SELECT HEX(s2) FROM t1; >+HEX(s2) >+3F00 >+DROP TABLE t1; >+# >+# Testing cs->cset->lengthsp() >+# >+CREATE TABLE t1 (a CHAR(10)) CHARACTER SET utf16le; >+INSERT INTO t1 VALUES ('a '); >+SELECT HEX(a) FROM t1; >+HEX(a) >+6100 >+DROP TABLE t1; >+# >+# Testing cs->cset->caseup() and cs->cset->casedn() >+# >+SELECT UPPER('abcd'), LOWER('ABCD'); >+UPPER('abcd') LOWER('ABCD') >+ABCD abcd >+# >+# Checking str_to_datetime() >+# >+select @@collation_connection; >+@@collation_connection >+utf16le_general_ci >+CREATE TABLE t1 (a date); >+INSERT INTO t1 VALUES ('2007-09-16'); >+SELECT * FROM t1; >+a >+2007-09-16 >+DROP TABLE t1; >+# >+# Testing cs->cset->ll10tostr >+# >+CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (123456); >+SELECT a, HEX(a) FROM t1; >+a HEX(a) >+123456 310032003300340035003600 >+DROP TABLE t1; >+# >+# Testing cs->cset->fill >+# SOUNDEX fills strings with DIGIT ZERO up to four characters >+# >+SELECT SOUNDEX('a'), HEX(SOUNDEX('a')); >+SOUNDEX('a') HEX(SOUNDEX('a')) >+A000 4100300030003000 >+# >+# Testing cs->cset->strntoul >+# >+CREATE TABLE t1 (a enum ('a','b','c')) CHARACTER SET utf16le; >+INSERT INTO t1 VALUES ('1'); >+SELECT * FROM t1; >+a >+a >+DROP TABLE t1; >+# >+# Testing cs->cset->strntoll and cs->cset->strntoull >+# >+SET NAMES latin1; >+SELECT HEX(CONV(CONVERT('123' USING utf16le), -10, 16)); >+HEX(CONV(CONVERT('123' USING utf16le), -10, 16)) >+3742 >+SELECT HEX(CONV(CONVERT('123' USING utf16le), 10, 16)); >+HEX(CONV(CONVERT('123' USING utf16le), 10, 16)) >+3742 >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+# >+# Testing cs->cset->strntod >+# >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+SELECT 1.1 + '1.2'; >+1.1 + '1.2' >+2.3 >+SELECT 1.1 + '1.2xxx'; >+1.1 + '1.2xxx' >+2.3 >+Warnings: >+Warning 1292 Truncated incorrect DOUBLE value: '1.2xxx' >+# >+# Testing cs->cset->strtoll10 >+# >+SELECT LEFT('aaa','1'); >+LEFT('aaa','1') >+a >+CREATE TABLE t1 AS SELECT REPEAT('abcd', 128) AS a; >+SELECT LEFT(a, '2') FROM t1; >+LEFT(a, '2') >+ab >+SELECT LEFT(a, ' \t \t 2') FROM t1; >+LEFT(a, ' \t \t 2') >+ab >+SELECT LEFT(a, ' \t \t +2') FROM t1; >+LEFT(a, ' \t \t +2') >+ab >+SELECT SUBSTR(a, '-2') FROM t1; >+SUBSTR(a, '-2') >+cd >+SELECT SUBSTR(a, ' \t \t -2') FROM t1; >+SUBSTR(a, ' \t \t -2') >+cd >+SELECT LEFT(a, '00002') FROM t1; >+LEFT(a, '00002') >+ab >+SELECT LEFT(a, ' \t \t 00002') FROM t1; >+LEFT(a, ' \t \t 00002') >+ab >+SELECT LEFT(a, ' \t \t +00002') FROM t1; >+LEFT(a, ' \t \t +00002') >+ab >+SELECT SUBSTR(a, '-00002') FROM t1; >+SUBSTR(a, '-00002') >+cd >+SELECT SUBSTR(a, ' \t \t -00002') FROM t1; >+SUBSTR(a, ' \t \t -00002') >+cd >+DROP TABLE t1; >+CREATE TABLE t1 AS SELECT REPEAT('abcd', 128) AS a LIMIT 0; >+INSERT INTO t1 VALUES ('255'), ('65535'),('16777215'),('4294967295'), >+('1099511627775'),('281474976710655'),('72057594037927935'), >+('1844674407370955161'),('18446744073709551614'), ('18446744073709551615'); >+SELECT a, CAST(a AS SIGNED), CAST(a AS UNSIGNED) FROM t1; >+a CAST(a AS SIGNED) CAST(a AS UNSIGNED) >+255 255 255 >+65535 65535 65535 >+16777215 16777215 16777215 >+4294967295 4294967295 4294967295 >+1099511627775 1099511627775 1099511627775 >+281474976710655 281474976710655 281474976710655 >+72057594037927935 72057594037927935 72057594037927935 >+1844674407370955161 1844674407370955161 1844674407370955161 >+18446744073709551614 -2 18446744073709551614 >+18446744073709551615 -1 18446744073709551615 >+Warnings: >+Note 1105 Cast to signed converted positive out-of-range integer to it's negative complement >+Note 1105 Cast to signed converted positive out-of-range integer to it's negative complement >+UPDATE t1 SET a=CONCAT('-', a); >+SELECT a, CAST(a AS SIGNED) FROM t1; >+a CAST(a AS SIGNED) >+-255 -255 >+-65535 -65535 >+-16777215 -16777215 >+-4294967295 -4294967295 >+-1099511627775 -1099511627775 >+-281474976710655 -281474976710655 >+-72057594037927935 -72057594037927935 >+-1844674407370955161 -1844674407370955161 >+-18446744073709551614 -9223372036854775808 >+-18446744073709551615 -9223372036854775808 >+Warnings: >+Warning 1292 Truncated incorrect INTEGER value: '-18446744073709551614' >+Warning 1292 Truncated incorrect INTEGER value: '-18446744073709551615' >+DROP TABLE t1; >+# >+# Testing cs->cset->strntoull10rnd >+# >+CREATE TABLE t1 (a int); >+INSERT INTO t1 VALUES ('-1234.1e2'); >+INSERT INTO t1 VALUES ('-1234.1e2xxxx'); >+Warnings: >+Warning 1265 Data truncated for column 'a' at row 1 >+INSERT INTO t1 VALUES ('-1234.1e2 '); >+INSERT INTO t1 VALUES ('123'); >+INSERT INTO t1 VALUES ('-124'); >+INSERT INTO t1 VALUES ('+125'); >+INSERT INTO t1 VALUES (' \t \t 123'); >+INSERT INTO t1 VALUES (' \t \t -124'); >+INSERT INTO t1 VALUES (' \t \t +125'); >+INSERT INTO t1 VALUES (' \t \t 000123'); >+INSERT INTO t1 VALUES (' \t \t -000124'); >+INSERT INTO t1 VALUES (' \t \t +000125'); >+SELECT * FROM t1; >+a >+-123410 >+-123410 >+-123410 >+123 >+-124 >+125 >+123 >+-124 >+125 >+123 >+-124 >+125 >+DROP TABLE t1; >+CREATE TABLE t1 (a BIGINT UNSIGNED); >+INSERT INTO t1 VALUES ('255'), ('65535'),('16777215'),('4294967295'), >+('1099511627775'),('281474976710655'),('72057594037927935'), >+('1844674407370955161'),('18446744073709551614'), ('18446744073709551615'); >+SELECT * FROM t1; >+a >+255 >+65535 >+16777215 >+4294967295 >+1099511627775 >+281474976710655 >+72057594037927935 >+1844674407370955161 >+18446744073709551614 >+18446744073709551615 >+DROP TABLE t1; >+# >+# Testing cs->cset->scan >+# >+CREATE TABLE t1 (a int); >+INSERT INTO t1 VALUES ('1 '); >+INSERT INTO t1 VALUES ('1 x'); >+Warnings: >+Warning 1265 Data truncated for column 'a' at row 1 >+SELECT * FROM t1; >+a >+1 >+1 >+DROP TABLE t1; >+# >+# Testing auto-conversion to TEXT >+# >+CREATE TABLE t1 (a VARCHAR(17000) CHARACTER SET utf16le); >+Warnings: >+Note 1246 Converting column 'a' from VARCHAR to TEXT >+SHOW CREATE TABLE t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `a` mediumtext CHARACTER SET utf16le >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+DROP TABLE t1; >+# >+# Testing that maximim possible key length is 1000 bytes >+# >+CREATE TABLE t1 (a VARCHAR(250) CHARACTER SET utf16le PRIMARY KEY); >+SHOW CREATE TABLE t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `a` varchar(250) CHARACTER SET utf16le NOT NULL, >+ PRIMARY KEY (`a`) >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+DROP TABLE t1; >+CREATE TABLE t1 (a VARCHAR(334) CHARACTER SET utf16le PRIMARY KEY); >+ERROR 42000: Specified key was too long; max key length is 1000 bytes >+# >+# Conversion to utf8 >+# >+CREATE TABLE t1 (a CHAR(1) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x00D800DC),(0x00D8FFDC),(0x7FDB00DC),(0x7FDBFFDC); >+INSERT INTO t1 VALUES (0xC000), (0xFF00),(0x00E0), (0xFFFF); >+SELECT HEX(a), HEX(@a:=CONVERT(a USING utf8mb4)), HEX(CONVERT(@a USING utf16le)) FROM t1; >+HEX(a) HEX(@a:=CONVERT(a USING utf8mb4)) HEX(CONVERT(@a USING utf16le)) >+00D800DC F0908080 00D800DC >+00D8FFDC F09083BF 00D8FFDC >+7FDB00DC F3AFB080 7FDB00DC >+7FDBFFDC F3AFB3BF 7FDBFFDC >+C000 C380 C000 >+FF00 C3BF FF00 >+00E0 EE8080 00E0 >+FFFF EFBFBF FFFF >+DROP TABLE t1; >+# >+# Test basic regex functionality >+# >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+drop table if exists t1; >+create table t1 as >+select repeat(' ', 64) as s1, repeat(' ',64) as s2 >+union >+select null, null; >+show create table t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `s1` varchar(64) CHARACTER SET utf16le DEFAULT NULL, >+ `s2` varchar(64) CHARACTER SET utf16le DEFAULT NULL >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+delete from t1; >+insert into t1 values('aaa','aaa'); >+insert into t1 values('aaa|qqq','qqq'); >+insert into t1 values('gheis','^[^a-dXYZ]+$'); >+insert into t1 values('aab','^aa?b'); >+insert into t1 values('Baaan','^Ba*n'); >+insert into t1 values('aaa','qqq|aaa'); >+insert into t1 values('qqq','qqq|aaa'); >+insert into t1 values('bbb','qqq|aaa'); >+insert into t1 values('bbb','qqq'); >+insert into t1 values('aaa','aba'); >+insert into t1 values(null,'abc'); >+insert into t1 values('def',null); >+insert into t1 values(null,null); >+insert into t1 values('ghi','ghi['); >+select HIGH_PRIORITY s1 regexp s2 from t1; >+s1 regexp s2 >+0 >+0 >+0 >+1 >+1 >+1 >+1 >+1 >+1 >+1 >+NULL >+NULL >+NULL >+NULL >+drop table t1; >+# >+# Test how CHARACTER SET works with date/time >+# >+CREATE TABLE t1 AS SELECT repeat('a',20) AS s1 LIMIT 0; >+SET timestamp=1216359724; >+INSERT INTO t1 VALUES (current_date); >+INSERT INTO t1 VALUES (current_time); >+INSERT INTO t1 VALUES (current_timestamp); >+SELECT s1, hex(s1) FROM t1; >+s1 hex(s1) >+2008-07-18 32003000300038002D00300037002D0031003800 >+08:42:04 300038003A00340032003A0030003400 >+2008-07-18 08:42:04 32003000300038002D00300037002D00310038002000300038003A00340032003A0030003400 >+DROP TABLE t1; >+SET timestamp=0; >+# >+# Bug#33073 Character sets: ordering fails with utf32 >+# >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+CREATE TABLE t1 AS SELECT REPEAT('a',2) as s1 LIMIT 0; >+SHOW CREATE TABLE t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `s1` varchar(2) CHARACTER SET utf16le NOT NULL DEFAULT '' >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+INSERT INTO t1 VALUES ('ab'),('AE'),('ab'),('AE'); >+SELECT * FROM t1 ORDER BY s1; >+s1 >+ab >+ab >+AE >+AE >+SET max_sort_length=4; >+SELECT * FROM t1 ORDER BY s1; >+s1 >+ab >+ab >+AE >+AE >+DROP TABLE t1; >+SET max_sort_length=DEFAULT; >+# >+# Bug#52520 Difference in tinytext utf column metadata >+# >+CREATE TABLE t1 ( >+s1 TINYTEXT CHARACTER SET utf16le, >+s2 TEXT CHARACTER SET utf16le, >+s3 MEDIUMTEXT CHARACTER SET utf16le, >+s4 LONGTEXT CHARACTER SET utf16le >+); >+SET NAMES utf8, @@character_set_results=NULL; >+SELECT *, HEX(s1) FROM t1; >+Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr >+def test t1 t1 s1 s1 252 255 0 Y 16 0 56 >+def test t1 t1 s2 s2 252 65535 0 Y 16 0 56 >+def test t1 t1 s3 s3 252 16777215 0 Y 16 0 56 >+def test t1 t1 s4 s4 252 4294967295 0 Y 16 0 56 >+def HEX(s1) 253 6120 0 Y 0 0 33 >+s1 s2 s3 s4 HEX(s1) >+SET NAMES latin1; >+SELECT *, HEX(s1) FROM t1; >+Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr >+def test t1 t1 s1 s1 252 127 0 Y 16 0 8 >+def test t1 t1 s2 s2 252 32767 0 Y 16 0 8 >+def test t1 t1 s3 s3 252 8388607 0 Y 16 0 8 >+def test t1 t1 s4 s4 252 2147483647 0 Y 16 0 8 >+def HEX(s1) 253 2040 0 Y 0 0 8 >+s1 s2 s3 s4 HEX(s1) >+SET NAMES utf8; >+SELECT *, HEX(s1) FROM t1; >+Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr >+def test t1 t1 s1 s1 252 381 0 Y 16 0 33 >+def test t1 t1 s2 s2 252 98301 0 Y 16 0 33 >+def test t1 t1 s3 s3 252 25165821 0 Y 16 0 33 >+def test t1 t1 s4 s4 252 4294967295 0 Y 16 0 33 >+def HEX(s1) 253 6120 0 Y 0 0 33 >+s1 s2 s3 s4 HEX(s1) >+CREATE TABLE t2 AS SELECT CONCAT(s1) FROM t1; >+SHOW CREATE TABLE t2; >+Table Create Table >+t2 CREATE TABLE `t2` ( >+ `CONCAT(s1)` varchar(255) CHARACTER SET utf16le DEFAULT NULL >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+DROP TABLE t1, t2; >+# >+# Problem found by Roy during review >+# MY_CS_BINSORT was not set for utf16le_bin, >+# so filesort did not work well >+# >+SET NAMES utf8, @@collation_connection=utf16le_bin; >+CREATE TABLE t1 AS SELECT REPEAT(' ', 10) as c LIMIT 0; >+ALTER TABLE t1 ADD PRIMARY KEY(c); >+SHOW CREATE TABLE t1; >+Table Create Table >+t1 CREATE TABLE `t1` ( >+ `c` varchar(10) CHARACTER SET utf16le COLLATE utf16le_bin NOT NULL DEFAULT '', >+ PRIMARY KEY (`c`) >+) ENGINE=MyISAM DEFAULT CHARSET=latin1 >+INSERT INTO t1 VALUES ('abc'),('zyx'),('acb'); >+SELECT UPPER(c) FROM t1 ORDER BY 1 DESC; >+UPPER(c) >+ZYX >+ACB >+ABC >+DROP TABLE t1; >+# >+# End of 5.6 tests >+# > >=== modified file 'mysql-test/r/ctype_utf32.result' >--- mysql-test/r/ctype_utf32.result 2012-08-09 16:25:47 +0000 >+++ mysql-test/r/ctype_utf32.result 2013-03-15 04:53:03 +0000 >@@ -1126,6 +1126,20 @@ > 00 > DROP TABLE t1; > # >+# Bug#32859 Character sets: no warning with non-fitting chariot wheel >+# >+CREATE TABLE t1 (utf32 CHAR(5) CHARACTER SET utf32, latin1 CHAR(5) CHARACTER SET latin1); >+INSERT INTO t1 (utf32) VALUES (0xc581); >+UPDATE t1 SET latin1 = utf32; >+Warnings: >+Warning 1366 Incorrect string value: '\x00\x00\xC5\x81' for column 'latin1' at row 1 >+DELETE FROM t1; >+INSERT INTO t1 (utf32) VALUES (0x100cc); >+UPDATE t1 SET latin1 = utf32; >+Warnings: >+Warning 1366 Incorrect string value: '\x00\x01\x00\xCC' for column 'latin1' at row 1 >+DROP TABLE t1; >+# > # Bug#55912 FORMAT with locale set fails for numbers < 1000 > # > SET collation_connection=utf32_general_ci; > >=== modified file 'mysql-test/suite/funcs_1/r/innodb_func_view.result' >--- mysql-test/suite/funcs_1/r/innodb_func_view.result 2012-09-08 10:15:55 +0000 >+++ mysql-test/suite/funcs_1/r/innodb_func_view.result 2013-03-15 05:14:48 +0000 >@@ -2675,7 +2675,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2692,7 +2692,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -2709,10 +2709,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > 18446744073709551615 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2727,10 +2727,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > 18446744073709551615 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -2749,7 +2749,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2766,7 +2766,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -2785,7 +2785,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2802,7 +2802,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -3041,7 +3041,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_varbinary_1000` as signed) AS `CAST(my_varbinary_1000 AS SIGNED INTEGER)`,`t1_values`.`my_varbinary_1000` AS `my_varbinary_1000`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3057,7 +3057,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > DROP VIEW v1; > > >@@ -3073,10 +3073,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > -1 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_binary_30` as signed) AS `CAST(my_binary_30 AS SIGNED INTEGER)`,`t1_values`.`my_binary_30` AS `my_binary_30`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3090,10 +3090,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > -1 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > DROP VIEW v1; > > >@@ -3111,7 +3111,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_varchar_1000` as signed) AS `CAST(my_varchar_1000 AS SIGNED INTEGER)`,`t1_values`.`my_varchar_1000` AS `my_varchar_1000`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3127,7 +3127,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > DROP VIEW v1; > > >@@ -3145,7 +3145,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_char_30` as signed) AS `CAST(my_char_30 AS SIGNED INTEGER)`,`t1_values`.`my_char_30` AS `my_char_30`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3161,7 +3161,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > DROP VIEW v1; > > > >=== modified file 'mysql-test/suite/funcs_1/r/memory_func_view.result' >--- mysql-test/suite/funcs_1/r/memory_func_view.result 2012-09-08 10:15:55 +0000 >+++ mysql-test/suite/funcs_1/r/memory_func_view.result 2013-03-15 05:15:06 +0000 >@@ -2676,7 +2676,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2693,7 +2693,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -2710,10 +2710,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > 18446744073709551615 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2728,10 +2728,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > 18446744073709551615 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -2750,7 +2750,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2767,7 +2767,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -2786,7 +2786,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2803,7 +2803,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -3042,7 +3042,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_varbinary_1000` as signed) AS `CAST(my_varbinary_1000 AS SIGNED INTEGER)`,`t1_values`.`my_varbinary_1000` AS `my_varbinary_1000`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3058,7 +3058,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > DROP VIEW v1; > > >@@ -3074,10 +3074,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > -1 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_binary_30` as signed) AS `CAST(my_binary_30 AS SIGNED INTEGER)`,`t1_values`.`my_binary_30` AS `my_binary_30`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3091,10 +3091,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > -1 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > DROP VIEW v1; > > >@@ -3112,7 +3112,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_varchar_1000` as signed) AS `CAST(my_varchar_1000 AS SIGNED INTEGER)`,`t1_values`.`my_varchar_1000` AS `my_varchar_1000`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3128,7 +3128,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > DROP VIEW v1; > > >@@ -3146,7 +3146,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_char_30` as signed) AS `CAST(my_char_30 AS SIGNED INTEGER)`,`t1_values`.`my_char_30` AS `my_char_30`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3162,7 +3162,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > DROP VIEW v1; > > > >=== modified file 'mysql-test/suite/funcs_1/r/myisam_func_view.result' >--- mysql-test/suite/funcs_1/r/myisam_func_view.result 2012-09-08 10:15:55 +0000 >+++ mysql-test/suite/funcs_1/r/myisam_func_view.result 2013-03-15 05:15:27 +0000 >@@ -2676,7 +2676,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2693,7 +2693,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -2710,10 +2710,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > 18446744073709551615 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2728,10 +2728,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > 18446744073709551615 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -2750,7 +2750,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2767,7 +2767,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -2786,7 +2786,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection >@@ -2803,7 +2803,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > Note 1105 Cast to unsigned converted negative integer to it's positive complement > DROP VIEW v1; > >@@ -3042,7 +3042,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_varbinary_1000` as signed) AS `CAST(my_varbinary_1000 AS SIGNED INTEGER)`,`t1_values`.`my_varbinary_1000` AS `my_varbinary_1000`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3058,7 +3058,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- ' > DROP VIEW v1; > > >@@ -3074,10 +3074,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > -1 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_binary_30` as signed) AS `CAST(my_binary_30 AS SIGNED INTEGER)`,`t1_values`.`my_binary_30` AS `my_binary_30`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3091,10 +3091,10 @@ > 0 ---Ã¤ÃÃ¼Ã@Âµ*$-- 4 > -1 -1 5 > Warnings: >-Warning 1292 Truncated incorrect INTEGER value: '' >+Warning 1292 Truncated incorrect INTEGER value: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >-Warning 1292 Truncated incorrect INTEGER value: '-1' >+Warning 1292 Truncated incorrect INTEGER value: ' ---\xC3\xA4\xC3\x96\xC3\xBC\xC3\x9F@\xC2\xB5*$-- \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' >+Warning 1292 Truncated incorrect INTEGER value: '-1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > DROP VIEW v1; > > >@@ -3112,7 +3112,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_varchar_1000` as signed) AS `CAST(my_varchar_1000 AS SIGNED INTEGER)`,`t1_values`.`my_varchar_1000` AS `my_varchar_1000`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3128,7 +3128,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<---------1000 characters-------------------------------------------------------------------------------------------------------' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$-- ' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$-- ' > DROP VIEW v1; > > >@@ -3146,7 +3146,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select cast(`t1_values`.`my_char_30` as signed) AS `CAST(my_char_30 AS SIGNED INTEGER)`,`t1_values`.`my_char_30` AS `my_char_30`,`t1_values`.`id` AS `id` from `t1_values` latin1 latin1_swedish_ci >@@ -3162,7 +3162,7 @@ > Warnings: > Warning 1292 Truncated incorrect INTEGER value: '' > Warning 1292 Truncated incorrect INTEGER value: '<--------30 characters------->' >-Warning 1292 Truncated incorrect INTEGER value: ' ---äÖüß@µ*$--' >+Warning 1292 Truncated incorrect INTEGER value: ' ---Ã¤ÃÃ¼Ã@Âµ*$--' > DROP VIEW v1; > > > >=== modified file 'mysql-test/suite/sys_vars/r/character_set_client_basic.result' >--- mysql-test/suite/sys_vars/r/character_set_client_basic.result 2011-12-12 22:58:40 +0000 >+++ mysql-test/suite/sys_vars/r/character_set_client_basic.result 2013-03-15 05:18:54 +0000 >@@ -430,7 +430,7 @@ > SET @total_charset = (SELECT count(*) FROM INFORMATION_SCHEMA.CHARACTER_SETS); > SELECT @total_charset; > @total_charset >-39 >+40 > '#--------------------FN_DYNVARS_010_10-------------------------#' > SET @@character_set_client = abc; > ERROR 42000: Unknown character set: 'abc' > >=== modified file 'mysql-test/suite/sys_vars/r/character_set_connection_basic.result' >--- mysql-test/suite/sys_vars/r/character_set_connection_basic.result 2011-12-12 22:58:40 +0000 >+++ mysql-test/suite/sys_vars/r/character_set_connection_basic.result 2013-03-15 05:19:08 +0000 >@@ -424,7 +424,7 @@ > SET @total_charset = (SELECT count(*) FROM INFORMATION_SCHEMA.CHARACTER_SETS); > SELECT @total_charset; > @total_charset >-39 >+40 > '#--------------------FN_DYNVARS_011_10-------------------------#' > SET @@character_set_connection = abc; > ERROR 42000: Unknown character set: 'abc' > >=== modified file 'mysql-test/suite/sys_vars/r/character_set_database_basic.result' >--- mysql-test/suite/sys_vars/r/character_set_database_basic.result 2011-12-12 22:58:40 +0000 >+++ mysql-test/suite/sys_vars/r/character_set_database_basic.result 2013-03-15 05:19:20 +0000 >@@ -424,7 +424,7 @@ > SET @total_charset = (SELECT count(*) FROM INFORMATION_SCHEMA.CHARACTER_SETS); > SELECT @total_charset; > @total_charset >-39 >+40 > '#--------------------FN_DYNVARS_012_10-------------------------#' > SET @@character_set_database = "grek"; > ERROR 42000: Unknown character set: 'grek' > >=== modified file 'mysql-test/suite/sys_vars/r/character_set_filesystem_basic.result' >--- mysql-test/suite/sys_vars/r/character_set_filesystem_basic.result 2011-12-12 22:58:40 +0000 >+++ mysql-test/suite/sys_vars/r/character_set_filesystem_basic.result 2013-03-15 05:19:32 +0000 >@@ -402,7 +402,7 @@ > SET @total_charset = (SELECT count(*) FROM INFORMATION_SCHEMA.CHARACTER_SETS); > SELECT @total_charset; > @total_charset >-39 >+40 > '#--------------------FN_DYNVARS_008_10-------------------------#' > SET @@character_set_filesystem = abc; > ERROR 42000: Unknown character set: 'abc' > >=== modified file 'mysql-test/suite/sys_vars/r/character_set_results_basic.result' >--- mysql-test/suite/sys_vars/r/character_set_results_basic.result 2011-12-12 22:58:40 +0000 >+++ mysql-test/suite/sys_vars/r/character_set_results_basic.result 2013-03-15 05:19:46 +0000 >@@ -424,7 +424,7 @@ > SET @total_charset = (SELECT count(*) FROM INFORMATION_SCHEMA.CHARACTER_SETS); > SELECT @total_charset; > @total_charset >-39 >+40 > 'Bug # 34843: character sets are mapped in such a way that 100 and following'; > 'numbers gives error, and before 100 the 36 mapped wraps arround several times.'; > '#--------------------FN_DYNVARS_013_10-------------------------#' > >=== modified file 'mysql-test/t/ctype_ucs.test' >--- mysql-test/t/ctype_ucs.test 2013-01-15 18:13:32 +0000 >+++ mysql-test/t/ctype_ucs.test 2013-03-15 04:05:28 +0000 >@@ -839,3 +839,20 @@ > --echo # > --echo # End of 5.5 tests > --echo # >+ >+ >+--echo # >+--echo # Start of 5.6 tests >+--echo # >+ >+--echo # >+--echo # Bug#59145 valgrind warnings for uninitialized values in my_strtoll10_mb2 >+--echo # >+SET NAMES latin1; >+SELECT CONVERT(CHAR(NULL USING ucs2), UNSIGNED); >+DO IFNULL(CHAR(NULL USING ucs2), ''); >+DO CAST(CONVERT('' USING ucs2) AS UNSIGNED); >+ >+--echo # >+--echo # End of 5.6 tests >+--echo # > >=== added file 'mysql-test/t/ctype_utf16le.test' >--- mysql-test/t/ctype_utf16le.test 1970-01-01 00:00:00 +0000 >+++ mysql-test/t/ctype_utf16le.test 2013-03-14 15:03:57 +0000 >@@ -0,0 +1,729 @@ >+-- source include/have_ucs2.inc >+-- source include/have_utf16.inc >+-- source include/have_utf32.inc >+-- source include/have_utf8mb4.inc >+ >+ >+SET TIME_ZONE='+03:00'; >+ >+--disable_warnings >+DROP TABLE IF EXISTS t1; >+--enable_warnings >+ >+--echo # >+--echo # Start of 5.6 tests >+--echo # >+ >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+SELECT HEX('a'), HEX('a '); >+-- source include/endspace.inc >+ >+ >+--echo # >+--echo # Check that incomplete utf16le characters in HEX notation >+--echo # are left-padded with zeros >+--echo # >+SELECT HEX(_utf16le 0x44); >+SELECT HEX(_utf16le 0x3344); >+SELECT HEX(_utf16le 0x113344); >+ >+--echo # >+--echo # Check that 0x20 is only trimmed when it is >+--echo # a part of real SPACE character, not just a part >+--echo # of a multibyte sequence. >+--echo # Note, CYRILLIC LETTER ER is used as an example, which >+--echo # is stored as 0x0420 in utf16le, thus contains 0x20 in the >+--echo # low byte. The second character is THREE-PER-M, U+2004, >+--echo # which contains 0x20 in the high byte. >+--echo # >+ >+CREATE TABLE t1 (word VARCHAR(64), word2 CHAR(64)) CHARACTER SET utf16le; >+INSERT INTO t1 VALUES (_koi8r 0xF2, _koi8r 0xF2), (_ucs2 X'2004',_ucs2 X'2004'); >+SELECT HEX(word) FROM t1 ORDER BY word; >+SELECT HEX(word2) FROM t1 ORDER BY word2; >+DELETE FROM t1; >+ >+--echo # >+--echo # Check that real spaces are correctly trimmed. >+--echo # >+INSERT INTO t1 VALUES (_ucs2 X'042000200020', _ucs2 X'042000200020'); >+INSERT INTO t1 VALUES (_ucs2 X'200400200020', _ucs2 X'200400200020'); >+SELECT HEX(word) FROM t1 ORDER BY word; >+SELECT HEX(word2) FROM t1 ORDER BY word2; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Check LPAD/RPAD >+--echo # >+CREATE TABLE t1 (a VARCHAR(10), pad INT, b VARCHAR(10)) CHARACTER SET utf16le; >+INSERT INTO t1 VALUES (_ucs2 X'0420', 10, _ucs2 X'0421'); >+INSERT INTO t1 VALUES (_ucs2 X'0420', 10, _ucs2 X'04210422'); >+INSERT INTO t1 VALUES (_ucs2 X'0420', 10, _ucs2 X'042104220423'); >+INSERT INTO t1 VALUES (_ucs2 X'0420042104220423042404250426042704280429042A042B',10,_ucs2 X'042104220423'); >+INSERT INTO t1 VALUES (_utf32 X'010000', 10, _ucs2 X'0421'); >+INSERT INTO t1 VALUES (_ucs2 X'0421', 10, _utf32 X'010000'); >+SELECT a, pad, b, LPAD(a, pad, b), HEX(LPAD(a, pad, b)) FROM t1; >+DROP TABLE t1; >+ >+ >+CREATE TABLE t1 SELECT >+LPAD(_utf16le X'2004',10,_utf16le X'2104') l, >+RPAD(_utf16le X'2004',10,_utf16le X'2104') r; >+SHOW CREATE TABLE t1; >+SELECT HEX(l), HEX(r) FROM t1; >+DROP TABLE t1; >+ >+CREATE TABLE t1 (f1 CHAR(30)); >+INSERT INTO t1 VALUES ("103000"), ("22720000"), ("3401200"), ("78000"); >+SELECT LPAD(f1, 12, "-o-/") FROM t1; >+DROP TABLE t1; >+ >+--echo # >+--echo # Testing LIKE >+--echo # >+ >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+--source include/ctype_like.inc >+ >+CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES ('ÑÑÐ²Ð°'),('Ð¤ÑÐ²Ð°'),('ÑÐ«Ð²Ð°'),('ÑÑÐÐ°'),('ÑÑÐ²Ð'),('Ð¤Ð«ÐÐ'); >+INSERT INTO t1 VALUES ('ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶'),('Ð¤ÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶'),('ÑÐ«Ð²Ð°Ð¿ÑÐ¾Ð»Ð´Ð¶'),('ÑÑÐÐ°Ð¿ÑÐ¾Ð»Ð´Ð¶'); >+INSERT INTO t1 VALUES ('ÑÑÐ²ÐÐ¿ÑÐ¾Ð»Ð´Ð¶'),('ÑÑÐ²Ð°ÐÑÐ¾Ð»Ð´Ð¶'),('ÑÑÐ²Ð°Ð¿Ð Ð¾Ð»Ð´Ð¶'),('ÑÑÐ²Ð°Ð¿ÑÐÐ»Ð´Ð¶'); >+INSERT INTO t1 VALUES ('ÑÑÐ²Ð°Ð¿ÑÐ¾ÐÐ´Ð¶'),('ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»ÐÐ¶'),('ÑÑÐ²Ð°Ð¿ÑÐ¾Ð»Ð´Ð'),('Ð¤Ð«ÐÐÐÐ ÐÐÐÐ'); >+SELECT * FROM t1 WHERE a LIKE '%ÑÐ«Ð²Ð°%' ORDER BY BINARY a; >+SELECT * FROM t1 WHERE a LIKE '%ÑÐ«Ð²%' ORDER BY BINARY a; >+SELECT * FROM t1 WHERE a LIKE 'ÑÐ«Ð²Ð°%' ORDER BY BINARY a; >+SELECT * FROM t1 WHERE a LIKE 'ÑÐ«Ð²Ð°%' COLLATE utf16le_bin ORDER BY BINARY a; >+DROP TABLE t1; >+ >+CREATE TABLE t1 (word VARCHAR(64) NOT NULL, PRIMARY KEY (word)) >+ENGINE=MyISAM CHARACTER SET utf16le; >+INSERT INTO t1 (word) VALUES ("cat"); >+SELECT * FROM t1 WHERE word LIKE "c%"; >+SELECT * FROM t1 WHERE word LIKE "ca_"; >+SELECT * FROM t1 WHERE word LIKE "cat"; >+SELECT * FROM t1 WHERE word LIKE _ucs2 x'00630025'; # "c%" >+SELECT * FROM t1 WHERE word LIKE _ucs2 x'00630061005F'; # "ca_" >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Check that INSERT() works fine. >+--echo # This invokes charpos() function. >+--echo # >+CREATE TABLE t1 ( >+ a VARCHAR(10) CHARACTER SET utf16le, >+ b VARCHAR(10) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES ('abc', 'def'); >+SELECT INSERT(a, 10, 2, b) FROM t1; >+SELECT INSERT(a, 1, 2, b) FROM t1; >+DROP TABLE t1; >+ >+--echo # >+--echo # Bug#1264 >+--echo # >+# Description: >+# >+# When USING a ucs2 TABLE in MySQL, >+# either with ucs2_general_ci or ucs2_bin collation, >+# words are returned in an incorrect order when USING ORDER BY >+# on an _indexed_ CHAR or VARCHAR column. They are sorted with >+# the longest word *first* instead of last. I.E. The word "aardvark" >+# is in the results before the word "a". >+# >+# If there is no index for the column, the problem does not occur. >+# >+# Interestingly, if there is no second column, the words are returned >+# in the correct order. >+# >+# According to EXPLAIN, it looks like when the output includes columns that >+# are not part of the index sorted on, it does a filesort, which fails. >+# Using a straight index yields correct results. >+ >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+ >+--echo # >+--echo # Two fields, index >+--echo # >+ >+CREATE TABLE t1 ( >+ word VARCHAR(64), >+ bar INT(11) DEFAULT 0, >+ PRIMARY KEY (word)) >+ ENGINE=MyISAM >+ CHARSET utf16le >+ COLLATE utf16le_general_ci ; >+ >+INSERT INTO t1 (word) VALUES ("aar"); >+INSERT INTO t1 (word) VALUES ("a"); >+INSERT INTO t1 (word) VALUES ("aardvar"); >+INSERT INTO t1 (word) VALUES ("aardvark"); >+INSERT INTO t1 (word) VALUES ("aardvara"); >+INSERT INTO t1 (word) VALUES ("aardvarz"); >+EXPLAIN SELECT * FROM t1 ORDER BY word; >+SELECT * FROM t1 ORDER BY word; >+EXPLAIN SELECT word FROM t1 ORDER BY word; >+SELECT word FROM t1 ORDER by word; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # One field, index >+--echo # >+ >+CREATE TABLE t1 ( >+ word VARCHAR(64) , >+ PRIMARY KEY (word)) >+ ENGINE=MyISAM >+ CHARSET utf16le >+ COLLATE utf16le_general_ci; >+ >+INSERT INTO t1 (word) VALUES ("aar"); >+INSERT INTO t1 (word) VALUES ("a"); >+INSERT INTO t1 (word) VALUES ("aardvar"); >+INSERT INTO t1 (word) VALUES ("aardvark"); >+INSERT INTO t1 (word) VALUES ("aardvara"); >+INSERT INTO t1 (word) VALUES ("aardvarz"); >+EXPLAIN SELECT * FROM t1 ORDER BY WORD; >+SELECT * FROM t1 ORDER BY word; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Two fields, no index >+--echo # >+ >+CREATE TABLE t1 ( >+ word TEXT, >+ bar INT(11) AUTO_INCREMENT, >+ PRIMARY KEY (bar)) >+ ENGINE=MyISAM >+ CHARSET utf16le >+ COLLATE utf16le_general_ci ; >+INSERT INTO t1 (word) VALUES ("aar"); >+INSERT INTO t1 (word) VALUES ("a" ); >+INSERT INTO t1 (word) VALUES ("aardvar"); >+INSERT INTO t1 (word) VALUES ("aardvark"); >+INSERT INTO t1 (word) VALUES ("aardvara"); >+INSERT INTO t1 (word) VALUES ("aardvarz"); >+EXPLAIN SELECT * FROM t1 ORDER BY word; >+SELECT * FROM t1 ORDER BY word; >+EXPLAIN SELECT word FROM t1 ORDER BY word; >+SELECT word FROM t1 ORDER BY word; >+DROP TABLE t1; >+ >+--echo # >+--echo # END OF Bug 1264 test >+--echo # >+ >+ >+--echo # >+--echo # Check alignment for from-binary-conversion with CAST and CONVERT >+--echo # >+SELECT HEX(CAST(0xAA as char CHARACTER SET utf16le)); >+SELECT HEX(CONVERT(0xAA USING utf16le)); >+ >+--echo # >+--echo # Check alignment for string types >+--echo # >+CREATE TABLE t1 (a CHAR(10) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x1),(0x11),(0x111),(0x1111),(0x11111); >+SELECT HEX(a) FROM t1; >+DROP TABLE t1; >+ >+CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x1),(0x11),(0x111),(0x1111),(0x11111); >+SELECT HEX(a) FROM t1; >+DROP TABLE t1; >+ >+CREATE TABLE t1 (a TEXT CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x1),(0x11),(0x111),(0x1111),(0x11111); >+SELECT HEX(a) FROM t1; >+DROP TABLE t1; >+ >+CREATE TABLE t1 (a MEDIUMTEXT CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x1),(0x11),(0x111),(0x1111),(0x11111); >+SELECT HEX(a) FROM t1; >+DROP TABLE t1; >+ >+CREATE TABLE t1 (a LONGTEXT CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x1),(0x11),(0x111),(0x1111),(0x11111); >+SELECT HEX(a) FROM t1; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Bug#5081 : UCS2 fields are filled with '0x2020' >+--echo # after extending field length >+--echo # >+ >+CREATE TABLE t1(a CHAR(1)) DEFAULT CHARSET utf16le; >+INSERT INTO t1 VALUES ('a'),('b'),('c'); >+ALTER TABLE t1 MODIFY a CHAR(5); >+SELECT a, HEX(a) FROM t1; >+DROP TABLE t1; >+ >+--echo # >+--echo # Check prepare statement from an UTF16 string >+--echo # >+SET NAMES latin1; >+SET @ivar= 1234; >+SET @str1 = 'SELECT ?'; >+SET @str2 = CONVERT(@str1 USING utf16le); >+PREPARE stmt1 FROM @str2; >+EXECUTE stmt1 USING @ivar; >+ >+--echo # >+--echo # Check that utf16le works with ENUM and SET type >+--echo # >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+CREATE TABLE t1 (a ENUM('x','y','z') CHARACTER SET utf16le); >+SHOW CREATE TABLE t1; >+INSERT INTO t1 VALUES ('x'); >+INSERT INTO t1 VALUES ('y'); >+INSERT INTO t1 VALUES ('z'); >+SELECT a, HEX(a) FROM t1 ORDER BY a; >+ALTER TABLE t1 CHANGE a a ENUM('x','y','z','d','e','Ã¤','Ã¶','Ã¼') CHARACTER SET utf16le; >+SHOW CREATE TABLE t1; >+INSERT INTO t1 VALUES ('D'); >+INSERT INTO t1 VALUES ('E '); >+INSERT INTO t1 VALUES ('Ã¤'); >+INSERT INTO t1 VALUES ('Ã¶'); >+INSERT INTO t1 VALUES ('Ã¼'); >+SELECT a, HEX(a) FROM t1 ORDER BY a; >+DROP TABLE t1; >+ >+CREATE TABLE t1 (a set ('x','y','z','Ã¤','Ã¶','Ã¼') CHARACTER SET utf16le); >+SHOW CREATE TABLE t1; >+INSERT INTO t1 VALUES ('x'); >+INSERT INTO t1 VALUES ('y'); >+INSERT INTO t1 VALUES ('z'); >+INSERT INTO t1 VALUES ('x,y'); >+INSERT INTO t1 VALUES ('x,y,z,Ã¤,Ã¶,Ã¼'); >+SELECT a, HEX(a) FROM t1 ORDER BY a; >+DROP TABLE t1; >+ >+--echo # >+--echo # Bug#7302 UCS2 data in ENUM fields get truncated when new column is added >+--echo # >+CREATE TABLE t1(a ENUM('a','b','c')) DEFAULT CHARACTER SET utf16le; >+INSERT INTO t1 VALUES('a'),('b'),('c'); >+ALTER TABLE t1 ADD b CHAR(1); >+SHOW WARNINGS; >+SELECT * FROM t1 ORDER BY a; >+DROP TABLE t1; >+ >+SET NAMES utf8, collation_connection='utf16le_general_ci'; >+-- source include/ctype_filesort.inc >+-- source include/ctype_like_escape.inc >+SET NAMES utf8, collation_connection='utf16le_bin'; >+-- source include/ctype_filesort.inc >+-- source include/ctype_filesort2.inc >+-- source include/ctype_like_escape.inc >+ >+--echo # >+--echo # Bug#10344 Some string functions fail for UCS2 >+--echo # >+CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf16le, pos INT); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,1); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,2); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,3); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,-1); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,-2); >+INSERT INTO t1 VALUES (_ucs2 0x00e400e50068,-3); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, 1); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, 2); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, 3); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, -1); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, -2); >+INSERT INTO t1 VALUES (_utf32 0x000000e4000000e500010000, -3); >+SELECT HEX(SUBSTR(a, pos)), SUBSTR(a, pos) FROM t1; >+DROP TABLE t1; >+ >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+ >+--echo # >+--echo # Bug#9442 Set parameter make query fail if column CHARACTER SET is UCS2 >+--echo # >+CREATE TABLE t1 (utext VARCHAR(20) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES ("lily"); >+INSERT INTO t1 VALUES ("river"); >+PREPARE stmt FROM 'SELECT utext FROM t1 where utext like ?'; >+SET @param1='%%'; >+EXECUTE stmt USING @param1; >+EXECUTE stmt USING @param1; >+SELECT utext FROM t1 where utext like '%%'; >+DROP TABLE t1; >+DEALLOCATE PREPARE stmt; >+ >+--echo # >+--echo # Bug#22052 Trailing spaces are not removed FROM UNICODE fields in an index >+--echo # >+CREATE TABLE t1 ( >+ a CHAR(10) CHARACTER SET utf16le NOT NULL, >+ INDEX a (a) >+) engine=myisam; >+INSERT INTO t1 VALUES (REPEAT(_ucs2 0x201f, 10)); >+INSERT INTO t1 VALUES (REPEAT(_ucs2 0x2020, 10)); >+INSERT INTO t1 VALUES (REPEAT(_ucs2 0x2021, 10)); >+--echo # make sure "index read" is used >+explain SELECT HEX(a) FROM t1 ORDER BY a; >+SELECT HEX(a) FROM t1 ORDER BY a; >+ALTER TABLE t1 DROP INDEX a; >+SELECT HEX(a) FROM t1 ORDER BY a; >+DROP TABLE t1; >+ >+--echo # >+--echo # Bug #20108: corrupted default enum value for a ucs2 field >+--echo # >+CREATE TABLE t1 ( >+ status ENUM('active','passive') CHARACTER SET utf16le COLLATE utf16le_general_ci >+ NOT NULL DEFAULT 'passive' >+); >+SHOW CREATE TABLE t1; >+ALTER TABLE t1 ADD a int NOT NULL AFTER status; >+SHOW CREATE TABLE t1; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Conversion FROM an UTF16LE string to a decimal column >+--echo # >+CREATE TABLE t1 (a VARCHAR(64) CHARACTER SET utf16le, b DECIMAL(10,3)); >+INSERT INTO t1 VALUES ("1.1", 0), ("2.1", 0); >+UPDATE t1 set b=a; >+SELECT *, HEX(a) FROM t1; >+DROP TABLE t1; >+ >+--echo # >+--echo # Bug#9442 Set parameter make query fail if column CHARACTER SET is UCS2 >+--echo # >+CREATE TABLE t1 (utext VARCHAR(20) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES ("lily"); >+INSERT INTO t1 VALUES ("river"); >+PREPARE stmt FROM 'SELECT utext FROM t1 where utext like ?'; >+SET @param1='%%'; >+EXECUTE stmt USING @param1; >+EXECUTE stmt USING @param1; >+SELECT utext FROM t1 where utext like '%%'; >+DROP TABLE t1; >+DEALLOCATE PREPARE stmt; >+ >+--echo # >+--echo # Bug#22638 SOUNDEX broken for international characters >+--echo # >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+SELECT SOUNDEX(''),SOUNDEX('he'),SOUNDEX('hello all folks'),SOUNDEX('#3556 in bugdb'); >+SELECT HEX(SOUNDEX('')),HEX(SOUNDEX('he')),HEX(SOUNDEX('hello all folks')),HEX(SOUNDEX('#3556 in bugdb')); >+SELECT 'mood' sounds like 'mud'; >+--echo # Cyrillic A, BE, VE >+SELECT HEX(SOUNDEX(_utf16le 0x041004110412)); >+--echo # Make sure that "U+00BF INVERTED QUESTION MARK" is not considered as letter >+SELECT HEX(SOUNDEX(_utf16le 0x00BF00C0)); >+ >+--echo # >+--echo # Bug#14290: character_maximum_length for text fields >+--echo # >+CREATE TABLE t1(a BLOB, b TEXT CHARSET utf16le); >+SELECT data_type, character_octet_length, character_maximum_length >+ FROM information_schema.columns where table_name='t1'; >+DROP TABLE t1; >+ >+ >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+ >+--echo # >+--echo # Testing cs->coll->instr() >+--echo # >+SELECT POSITION('bb' IN 'abba'); >+ >+--echo # >+--echo # Testing cs->coll->hash_sort() >+--echo # >+SET NAMES utf8, collation_connection=utf16le_bin; >+--source include/ctype_heap.inc >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+--source include/ctype_heap.inc >+ >+--echo # >+--echo # Testing cs->cset->numchars() >+--echo # >+SELECT CHAR_LENGTH('abcd'), OCTET_LENGTH('abcd'); >+SELECT CHAR_LENGTH(_utf16le 0x00D800DC), OCTET_LENGTH(_utf16le 0x00D800DC); >+SELECT CHAR_LENGTH(_utf16le 0x7DD8FFDF), OCTET_LENGTH(_utf16le 0x7FD8DDDF); >+ >+--echo # >+--echo # Testing cs->cset->charpos() >+--echo # >+SELECT LEFT('abcd',2); >+SELECT HEX(LEFT(_utf16le 0x00D800DC7FD8FFDF, 1)); >+SELECT HEX(RIGHT(_utf16le 0x00D800DC7FD8FFDF, 1)); >+ >+--echo # >+--echo # Testing cs->cset->well_formed_length() >+--echo # >+CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf16le); >+--echo # Bad sequences >+--error ER_INVALID_CHARACTER_STRING >+INSERT INTO t1 VALUES (_utf16le 0x00D8); >+--error ER_INVALID_CHARACTER_STRING >+INSERT INTO t1 VALUES (_utf16le 0x00DC); >+--error ER_INVALID_CHARACTER_STRING >+INSERT INTO t1 VALUES (_utf16le 0x00D800D8); >+--error ER_INVALID_CHARACTER_STRING >+INSERT INTO t1 VALUES (_utf16le 0x00D800E8); >+--error ER_INVALID_CHARACTER_STRING >+INSERT INTO t1 VALUES (_utf16le 0x00D80008); >+--echo # Good sequences >+INSERT INTO t1 VALUES (_utf16le 0x00D800DC); >+INSERT INTO t1 VALUES (_utf16le 0x00D8FFDC); >+INSERT INTO t1 VALUES (_utf16le 0xFFDB00DC); >+INSERT INTO t1 VALUES (_utf16le 0xFFDBFFDC); >+SELECT HEX(a) FROM t1; >+DROP TABLE t1; >+ >+--echo # >+--echo # Bug#32393 Character sets: illegal characters in utf16le columns >+--echo # >+--echo # Tests that cs->cset->wc_mb() doesn't accept surrogate parts >+--echo # >+--echo # via ALTER >+--echo # >+CREATE TABLE t1 (s1 VARCHAR(50) CHARACTER SET ucs2); >+INSERT INTO t1 VALUES (0xDF84); >+ALTER TABLE t1 MODIFY column s1 VARCHAR(50) CHARACTER SET utf16le; >+SELECT HEX(s1) FROM t1; >+DROP TABLE t1; >+ >+--echo # >+--echo # via UPDATE >+--echo # >+CREATE TABLE t1 (s1 VARCHAR(5) CHARACTER SET ucs2, s2 VARCHAR(5) CHARACTER SET utf16le); >+INSERT INTO t1 (s1) VALUES (0xdf84); >+UPDATE t1 set s2 = s1; >+SELECT HEX(s2) FROM t1; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Testing cs->cset->lengthsp() >+--echo # >+CREATE TABLE t1 (a CHAR(10)) CHARACTER SET utf16le; >+INSERT INTO t1 VALUES ('a '); >+SELECT HEX(a) FROM t1; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Testing cs->cset->caseup() and cs->cset->casedn() >+--echo # >+SELECT UPPER('abcd'), LOWER('ABCD'); >+ >+--echo # >+--echo # Checking str_to_datetime() >+--echo # >+select @@collation_connection; >+CREATE TABLE t1 (a date); >+INSERT INTO t1 VALUES ('2007-09-16'); >+SELECT * FROM t1; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Testing cs->cset->ll10tostr >+--echo # >+CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (123456); >+SELECT a, HEX(a) FROM t1; >+DROP TABLE t1; >+ >+--echo # >+--echo # Testing cs->cset->fill >+--echo # SOUNDEX fills strings with DIGIT ZERO up to four characters >+--echo # >+SELECT SOUNDEX('a'), HEX(SOUNDEX('a')); >+ >+ >+--echo # >+--echo # Testing cs->cset->strntoul >+--echo # >+CREATE TABLE t1 (a enum ('a','b','c')) CHARACTER SET utf16le; >+INSERT INTO t1 VALUES ('1'); >+SELECT * FROM t1; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Testing cs->cset->strntoll and cs->cset->strntoull >+--echo # >+SET NAMES latin1; >+SELECT HEX(CONV(CONVERT('123' USING utf16le), -10, 16)); >+SELECT HEX(CONV(CONVERT('123' USING utf16le), 10, 16)); >+SET NAMES utf8, collation_connection=utf16le_general_ci; Monty: You have here two idenitcal SET NAMES in a row. >+ >+ >+--echo # >+--echo # Testing cs->cset->strntod >+--echo # >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+SELECT 1.1 + '1.2'; >+SELECT 1.1 + '1.2xxx'; >+ >+ >+--echo # >+--echo # Testing cs->cset->strtoll10 >+--echo # >+SELECT LEFT('aaa','1'); >+CREATE TABLE t1 AS SELECT REPEAT('abcd', 128) AS a; >+SELECT LEFT(a, '2') FROM t1; >+SELECT LEFT(a, ' \t \t 2') FROM t1; >+SELECT LEFT(a, ' \t \t +2') FROM t1; >+SELECT SUBSTR(a, '-2') FROM t1; >+SELECT SUBSTR(a, ' \t \t -2') FROM t1; >+SELECT LEFT(a, '00002') FROM t1; >+SELECT LEFT(a, ' \t \t 00002') FROM t1; >+SELECT LEFT(a, ' \t \t +00002') FROM t1; >+SELECT SUBSTR(a, '-00002') FROM t1; >+SELECT SUBSTR(a, ' \t \t -00002') FROM t1; >+DROP TABLE t1; >+CREATE TABLE t1 AS SELECT REPEAT('abcd', 128) AS a LIMIT 0; >+INSERT INTO t1 VALUES ('255'), ('65535'),('16777215'),('4294967295'), >+('1099511627775'),('281474976710655'),('72057594037927935'), >+('1844674407370955161'),('18446744073709551614'), ('18446744073709551615'); >+SELECT a, CAST(a AS SIGNED), CAST(a AS UNSIGNED) FROM t1; >+UPDATE t1 SET a=CONCAT('-', a); >+SELECT a, CAST(a AS SIGNED) FROM t1; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Testing cs->cset->strntoull10rnd >+--echo # >+CREATE TABLE t1 (a int); >+INSERT INTO t1 VALUES ('-1234.1e2'); >+INSERT INTO t1 VALUES ('-1234.1e2xxxx'); >+INSERT INTO t1 VALUES ('-1234.1e2 '); >+INSERT INTO t1 VALUES ('123'); >+INSERT INTO t1 VALUES ('-124'); >+INSERT INTO t1 VALUES ('+125'); >+INSERT INTO t1 VALUES (' \t \t 123'); >+INSERT INTO t1 VALUES (' \t \t -124'); >+INSERT INTO t1 VALUES (' \t \t +125'); >+INSERT INTO t1 VALUES (' \t \t 000123'); >+INSERT INTO t1 VALUES (' \t \t -000124'); >+INSERT INTO t1 VALUES (' \t \t +000125'); >+SELECT * FROM t1; >+DROP TABLE t1; >+CREATE TABLE t1 (a BIGINT UNSIGNED); >+INSERT INTO t1 VALUES ('255'), ('65535'),('16777215'),('4294967295'), >+('1099511627775'),('281474976710655'),('72057594037927935'), >+('1844674407370955161'),('18446744073709551614'), ('18446744073709551615'); >+SELECT * FROM t1; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Testing cs->cset->scan >+--echo # >+CREATE TABLE t1 (a int); >+INSERT INTO t1 VALUES ('1 '); >+INSERT INTO t1 VALUES ('1 x'); >+SELECT * FROM t1; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Testing auto-conversion to TEXT >+--echo # >+CREATE TABLE t1 (a VARCHAR(17000) CHARACTER SET utf16le); >+SHOW CREATE TABLE t1; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Testing that maximim possible key length is 1000 bytes >+--echo # >+CREATE TABLE t1 (a VARCHAR(250) CHARACTER SET utf16le PRIMARY KEY); >+SHOW CREATE TABLE t1; >+DROP TABLE t1; >+--error ER_TOO_LONG_KEY >+CREATE TABLE t1 (a VARCHAR(334) CHARACTER SET utf16le PRIMARY KEY); >+ >+ >+--echo # >+--echo # Conversion to utf8 >+--echo # >+CREATE TABLE t1 (a CHAR(1) CHARACTER SET utf16le); >+INSERT INTO t1 VALUES (0x00D800DC),(0x00D8FFDC),(0x7FDB00DC),(0x7FDBFFDC); >+INSERT INTO t1 VALUES (0xC000), (0xFF00),(0x00E0), (0xFFFF); >+SELECT HEX(a), HEX(@a:=CONVERT(a USING utf8mb4)), HEX(CONVERT(@a USING utf16le)) FROM t1; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # Test basic regex functionality >+--echo # >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+--source include/ctype_regex.inc >+ >+ >+--echo # >+--echo # Test how CHARACTER SET works with date/time >+--echo # >+--source include/ctype_datetime.inc >+ >+ >+--echo # >+--echo # Bug#33073 Character sets: ordering fails with utf32 >+--echo # >+SET NAMES utf8, collation_connection=utf16le_general_ci; >+CREATE TABLE t1 AS SELECT REPEAT('a',2) as s1 LIMIT 0; >+SHOW CREATE TABLE t1; >+INSERT INTO t1 VALUES ('ab'),('AE'),('ab'),('AE'); >+SELECT * FROM t1 ORDER BY s1; >+SET max_sort_length=4; >+SELECT * FROM t1 ORDER BY s1; >+DROP TABLE t1; >+SET max_sort_length=DEFAULT; >+ >+ >+--echo # >+--echo # Bug#52520 Difference in tinytext utf column metadata >+--echo # >+CREATE TABLE t1 ( >+ s1 TINYTEXT CHARACTER SET utf16le, >+ s2 TEXT CHARACTER SET utf16le, >+ s3 MEDIUMTEXT CHARACTER SET utf16le, >+ s4 LONGTEXT CHARACTER SET utf16le >+); >+--enable_metadata >+SET NAMES utf8, @@character_set_results=NULL; >+SELECT *, HEX(s1) FROM t1; >+SET NAMES latin1; >+SELECT *, HEX(s1) FROM t1; >+SET NAMES utf8; >+SELECT *, HEX(s1) FROM t1; >+--disable_metadata >+CREATE TABLE t2 AS SELECT CONCAT(s1) FROM t1; >+SHOW CREATE TABLE t2; >+DROP TABLE t1, t2; >+ >+--echo # >+--echo # Problem found by Roy during review >+--echo # MY_CS_BINSORT was not set for utf16le_bin, >+--echo # so filesort did not work well >+--echo # >+SET NAMES utf8, @@collation_connection=utf16le_bin; >+CREATE TABLE t1 AS SELECT REPEAT(' ', 10) as c LIMIT 0; >+# the problem in fact reproduced even without the primary key: >+ALTER TABLE t1 ADD PRIMARY KEY(c); >+SHOW CREATE TABLE t1; >+INSERT INTO t1 VALUES ('abc'),('zyx'),('acb'); >+SELECT UPPER(c) FROM t1 ORDER BY 1 DESC; >+DROP TABLE t1; >+ >+ >+--echo # >+--echo # End of 5.6 tests >+--echo # > >=== modified file 'mysql-test/t/ctype_utf32.test' >--- mysql-test/t/ctype_utf32.test 2012-08-09 16:25:47 +0000 >+++ mysql-test/t/ctype_utf32.test 2013-03-15 04:52:28 +0000 >@@ -813,6 +813,17 @@ > DROP TABLE t1; > > --echo # >+--echo # Bug#32859 Character sets: no warning with non-fitting chariot wheel >+--echo # >+CREATE TABLE t1 (utf32 CHAR(5) CHARACTER SET utf32, latin1 CHAR(5) CHARACTER SET latin1); >+INSERT INTO t1 (utf32) VALUES (0xc581); >+UPDATE t1 SET latin1 = utf32; >+DELETE FROM t1; >+INSERT INTO t1 (utf32) VALUES (0x100cc); >+UPDATE t1 SET latin1 = utf32; >+DROP TABLE t1; >+ >+--echo # > --echo # Bug#55912 FORMAT with locale set fails for numbers < 1000 > --echo # > SET collation_connection=utf32_general_ci; > >=== modified file 'mysys/charset-def.c' >--- mysys/charset-def.c 2012-07-31 17:29:07 +0000 >+++ mysys/charset-def.c 2013-03-14 12:43:39 +0000 >@@ -304,6 +304,8 @@ > #ifdef HAVE_CHARSET_utf16 > add_compiled_collation(&my_charset_utf16_general_ci); > add_compiled_collation(&my_charset_utf16_bin); >+ add_compiled_collation(&my_charset_utf16le_general_ci); >+ add_compiled_collation(&my_charset_utf16le_bin); > #ifdef HAVE_UCA_COLLATIONS > add_compiled_collation(&my_charset_utf16_unicode_ci); > add_compiled_collation(&my_charset_utf16_icelandic_uca_ci); > >=== modified file 'sql/item_func.cc' >--- sql/item_func.cc 2013-01-29 14:10:47 +0000 >+++ sql/item_func.cc 2013-03-14 14:52:45 +0000 >@@ -1027,13 +1027,11 @@ > value= cs->cset->strtoll10(cs, start, &end, error); > if (*error > 0 || end != start+ length) > { >- char err_buff[128]; >- String err_tmp(err_buff,(uint32) sizeof(err_buff), system_charset_info); >- err_tmp.copy(start, length, system_charset_info); >+ ErrConvString err(res); > push_warning_printf(current_thd, MYSQL_ERROR::WARN_LEVEL_WARN, > ER_TRUNCATED_WRONG_VALUE, > ER(ER_TRUNCATED_WRONG_VALUE), "INTEGER", >- err_tmp.c_ptr()); >+ err.ptr()); > } > return value; > } > >=== modified file 'sql/sys_vars.cc' >--- sql/sys_vars.cc 2013-02-25 23:20:17 +0000 >+++ sql/sys_vars.cc 2013-03-15 04:23:14 +0000 >@@ -385,16 +385,19 @@ > if (var->value->result_type() == STRING_RESULT) > { > String str(buff, sizeof(buff), system_charset_info), *res; >- if (!(res=var->value->val_str(&str))) >+ if (!(res= var->value->val_str(&str))) > var->save_result.ptr= NULL; >- else if (!(var->save_result.ptr= get_charset_by_csname(res->c_ptr(), >- MY_CS_PRIMARY, >- MYF(0))) && >- !(var->save_result.ptr=get_old_charset_by_name(res->c_ptr()))) >+ else > { >- ErrConvString err(res); >- my_error(ER_UNKNOWN_CHARACTER_SET, MYF(0), err.ptr()); >- return true; >+ ErrConvString err(res); /* Get utf8 '\0' terminated string */ >+ if (!(var->save_result.ptr= get_charset_by_csname(err.ptr(), >+ MY_CS_PRIMARY, >+ MYF(0))) && >+ !(var->save_result.ptr= get_old_charset_by_name(err.ptr()))) >+ { >+ my_error(ER_UNKNOWN_CHARACTER_SET, MYF(0), err.ptr()); >+ return true; >+ } > } Monty: Please add a comment why we need to use ErrConvString instead of the original string. (As ErrConvString should normally only be used for errors). I assume this is because of: SET collation_connection=<ucs2, utf16, utf32 value>; SET character_set_connection=<ucs2, utf16, utf32 value>; Wouldn't it be better to have a function String change_character_set(CHARSET_INFO *cs) for cases like this? (I am mostly worried someone will try to copy the above code for things where ErrConvString is not suitable) After all, instead of using ErrConvString() I assume one could simple have: char buff[256]; String utf8(buff, sizeof(buff), system_charset_info); utf8.copy(res); Which I think is probably faster and more clear than using ErrConvString(). > } > else // INT_RESULT >@@ -505,11 +508,14 @@ > String str(buff, sizeof(buff), system_charset_info), *res; > if (!(res= var->value->val_str(&str))) > var->save_result.ptr= NULL; >- else if (!(var->save_result.ptr= get_charset_by_name(res->c_ptr(), MYF(0)))) >+ else > { >- ErrConvString err(res); >- my_error(ER_UNKNOWN_COLLATION, MYF(0), err.ptr()); >- return true; >+ ErrConvString err(res); /* Get utf8 '\0'-terminated string */ >+ if (!(var->save_result.ptr= get_charset_by_name(err.ptr(), MYF(0)))) >+ { >+ my_error(ER_UNKNOWN_COLLATION, MYF(0), err.ptr()); >+ return true; >+ } > } > } Monty: same argument as above. > else // INT_RESULT > >=== modified file 'strings/ctype-latin1.c' >--- strings/ctype-latin1.c 2013-03-12 14:33:19 +0000 >+++ strings/ctype-latin1.c 2013-03-15 04:51:34 +0000 >@@ -385,7 +385,10 @@ > if (str >= end) > return MY_CS_TOOSMALL; > >- pl= uni_to_cs[(wc>>8) & 0xFF]; >+ if (wc > 0xFFFF) >+ return MY_CS_ILUNI; >+ >+ pl= uni_to_cs[wc >> 8]; > str[0]= pl ? pl[wc & 0xFF] : '\0'; > return (!str[0] && wc) ? MY_CS_ILUNI : 1; > } > >=== modified file 'strings/ctype-ucs2.c' >--- strings/ctype-ucs2.c 2013-01-29 14:10:47 +0000 >+++ strings/ctype-ucs2.c 2013-03-15 04:27:10 +0000 >@@ -718,17 +718,30 @@ > ulonglong li; > int negative; > ulong cutoff, cutoff2, cutoff3; >+ my_wc_t wc; >+ int res; > > s= nptr; > /* If fixed length string */ > if (endptr) > { >- /* Make sure string length is even */ >+ /* >+ Make sure string length is even. >+ Odd length indicates a bug in the caller. >+ Assert in debug, round in production. >+ */ >+ DBUG_ASSERT((*endptr - s) % 2 == 0); > end= s + ((*endptr - s) / 2) * 2; >- while (s < end && !s[0] && (s[1] == ' ' || s[1] == '\t')) >- s+= 2; >- if (s == end) >- goto no_conv; >+ >+ for ( ; ; ) /* Skip leading spaces and tabs */ >+ { >+ res= cs->cset->mb_wc(cs, &wc, (const uchar *) s, (const uchar *) end); >+ if (res <= 0) >+ goto no_conv; >+ s+= res; >+ if (wc != ' ' && wc != '\t') >+ break; >+ } > } > else > { >@@ -738,13 +751,14 @@ > > /* Check for a sign. */ > negative= 0; >- if (!s[0] && s[1] == '-') >+ if (wc == '-') > { > *error= -1; /* Mark as negative number */ > negative= 1; >- s+= 2; >- if (s == end) >+ res= cs->cset->mb_wc(cs, &wc, (const uchar *) s, (const uchar *) end); >+ if (res < 0) > goto no_conv; >+ s+= res; Monty: Add a comment: /* wc should contain the first number in the code below */ It would also be nice to have a comment why we can't assume for this code that each character is 2 bytes (like the old code assumed). I assume it's to detect wrong characters ? > cutoff= MAX_NEGATIVE_NUMBER / LFACTOR2; > cutoff2= (MAX_NEGATIVE_NUMBER % LFACTOR2) / 100; > cutoff3= MAX_NEGATIVE_NUMBER % 100; >@@ -752,46 +766,55 @@ > else > { > *error= 0; >- if (!s[0] && s[1] == '+') >+ if (wc == '+') > { >- s+= 2; >- if (s == end) >+ res= cs->cset->mb_wc(cs, &wc, (const uchar *) s, (const uchar *) end); >+ if (res < 0) > goto no_conv; >+ s+= res; Monty: Add same comment as for the earlier comment > } > cutoff= ULONGLONG_MAX / LFACTOR2; > cutoff2= ULONGLONG_MAX % LFACTOR2 / 100; > cutoff3= ULONGLONG_MAX % 100; > } > >+ > /* Handle case where we have a lot of pre-zero */ >- if (!s[0] && s[1] == '0') >+ if (wc == '0') > { > i= 0; >- do >+ for ( ; ; s+= res) > { >- s+= 2; > if (s == end) > goto end_i; /* Return 0 */ >+ res= cs->cset->mb_wc(cs, &wc, (const uchar *) s, (const uchar *) end); >+ if (res < 0) >+ goto no_conv; >+ if (wc != '0') >+ break; > } >- while (!s[0] && s[1] == '0'); >+ while (wc == '0'); Monty: The above line is a bug (even if condition is impossible). Remove it! (The above code is probably wrongly left there when you converted the do-while to a for()). > n_end= s + 2 * INIT_CNT; > } > else > { > /* Read first digit to check that it's a valid number */ >- if (s[0] || (c= (s[1]-'0')) > 9) >+ if ((c= (wc - '0')) > 9) > goto no_conv; > i= c; >- s+= 2; > n_end= s + 2 * (INIT_CNT-1); > } Monty: Above you can set 'i' directly. No reason to use 'c' > /* Handle first 9 digits and store them in i */ > if (n_end > end) > n_end= end; >- for (; s != n_end ; s+= 2) >+ for ( ; ; ) > { >- if (s[0] || (c= (s[1]-'0')) > 9) >+ res= cs->cset->mb_wc(cs, &wc, (const uchar *) s, (const uchar *) n_end); >+ if (res < 0) >+ break; Monty: In general I prefer the code: if ((res= cs->cset->mb_wc(cs, &wc, (const uchar *) s, (const uchar *) n_end) < 0) By the way, wouldn't the code be simpler if you would change the type of 's' and 'end' and 'n_end' to const uchar * ? >+ s+= res; >+ if ((c= (wc - '0')) > 9) > goto end_i; > i= i*10+c; > } >@@ -806,10 +829,13 @@ > n_end= end; > do > { >- if (s[0] || (c= (s[1]-'0')) > 9) >+ res= cs->cset->mb_wc(cs, &wc, (const uchar *) s, (const uchar *) end); >+ if (res < 0) >+ goto no_conv; >+ s+= res; >+ if ((c= (wc - '0')) > 9) > goto end_i_and_j; > j= j*10+c; >- s+= 2; > } while (s != n_end); > if (s == end) > { >@@ -817,20 +843,28 @@ > goto end_i_and_j; > goto end3; > } >- if (s[0] || (c= (s[1]-'0')) > 9) >+ res= cs->cset->mb_wc(cs, &wc, (const uchar *) s, (const uchar *) end); >+ if (res < 0) >+ goto no_conv; >+ s+= res; >+ if ((c= (wc - '0')) > 9) > goto end3; Monty: Here you could store directly in 'k' and remove next line Isn't it wrong to increment 's' before going to to end3? 's' should point to the first not-number character. Now it will point one character past it. Why wasn't this shown in the test cases? Maybe because you didn't test with big numbers? > /* Handle the next 1 or 2 digits and store them in k */ > k=c; >- s+= 2; >- if (s == end || s[0] || (c= (s[1]-'0')) > 9) >+ if (s == end) >+ goto end4; >+ res= cs->cset->mb_wc(cs, &wc, (const uchar *) s, (const uchar *) end); >+ if (res < 0) >+ goto no_conv; >+ s+= res; >+ if ((c= (wc - '0')) > 9) > goto end4; Monty: Isn't it wrong to increment 's' before going to to end4? 's' should point to the first not-number character. Now it will point one character past it. Why wasn't this shown in the test cases? Please check the same issue with all the other calls to end_i, end_i_and_j, end3 and end4. Even some of the earlier code looks not safe. > k= k*10+c; >- s+= 2; > *endptr= (char*) s; > > /* number string should have ended here */ >- if (s != end && !s[0] && (c= (s[1]-'0')) <= 9) >+ if (s != end && (c= (wc - '0')) <= 9) > goto overflow; > > /* Check that we didn't get an overflow with the last digit */ >@@ -882,15 +916,19 @@ > const char *str, const char *end, int sequence_type) > { > const char *str0= str; >- end--; /* for easier loop condition, because of two bytes per character */ >- >+ my_wc_t wc; >+ int res; >+ > switch (sequence_type) > { > case MY_SEQ_SPACES: >- for ( ; str < end; str+= 2) >+ for (res= cs->cset->mb_wc(cs, &wc, >+ (const uchar *) str, (const uchar *) end); >+ res > 0 && wc == ' '; >+ str+= res, >+ res= cs->cset->mb_wc(cs, &wc, >+ (const uchar *) str, (const uchar *) end)) > { >- if (str[0] != '\0' || str[1] != ' ') >- break; > } > return (size_t) (str - str0); > default: >@@ -900,11 +938,34 @@ > > > static void >-my_fill_mb2(CHARSET_INFO *cs __attribute__((unused)), >- char *s, size_t l, int fill) >+my_fill_mb2(CHARSET_INFO *cs, char *s, size_t slen, int fill) > { >- DBUG_ASSERT(fill <= 0xFFFF); >- for ( ; l >= 2; s[0]= (fill >> 8), s[1]= (fill & 0xFF), s+= 2, l-= 2); >+ char buf[10]; >+ int buflen; >+ >+ DBUG_ASSERT((slen % 2) == 0); >+ >+ buflen= cs->cset->wc_mb(cs, (my_wc_t) fill, (uchar*) buf, >+ (uchar*) buf + sizeof(buf)); >+ >+ DBUG_ASSERT(buflen > 0); >+ >+ while (slen >= (size_t) buflen) >+ { >+ /* Enough space for the characer */ >+ memcpy(s, buf, (size_t) buflen); >+ s+= buflen; >+ slen-= buflen; >+ } Monty: I generally prefer loops like: end= s + slen; for ( ; s < end; s+=buflen) memcpy(s, buf, (size_t) buflen); Better to only have to decrement one variable instead of two. >+ >+ /* >+ If there are some more space which is not enough >+ for the whole multibyte character, then add trailing zeros. >+ */ >+ for ( ; slen; slen--) >+ { >+ *s++= 0x00; >+ } Monty: Use instead: if (slen) bzero(s, slen); > } > > >@@ -1018,11 +1079,26 @@ > DB80..DBFF - Private surrogate high (128 pages) > DC00..DFFF - Surrogate low (1024 codes in a page) > */ >+#define MY_UTF16_SURROGATE_HIGH_FIRST 0xD800 >+#define MY_UTF16_SURROGATE_HIGH_LAST 0xDBFF >+#define MY_UTF16_SURROGATE_LOW_FIRST 0xDC00 >+#define MY_UTF16_SURROGATE_LOW_LAST 0xDFFF > > #define MY_UTF16_HIGH_HEAD(x) ((((uchar) (x)) & 0xFC) == 0xD8) > #define MY_UTF16_LOW_HEAD(x) ((((uchar) (x)) & 0xFC) == 0xDC) > #define MY_UTF16_SURROGATE(x) (((x) & 0xF800) == 0xD800) > >+#define MY_UTF16_WC2(a, b) ((a << 8) + b) >+ >+/* >+ a= 110110?? (<< 18) >+ b= ???????? (<< 10) >+ c= 110111?? (<< 8) >+ d= ???????? (<< 0) >+*/ >+#define MY_UTF16_WC4(a, b, c, d) (((a & 3) << 18) + (b << 10) + \ >+ ((c & 3) << 8) + d + 0x10000) >+ > static int > my_utf16_uni(CHARSET_INFO *cs __attribute__((unused)), > my_wc_t *pwc, const uchar *s, const uchar *e) >@@ -1044,23 +1120,14 @@ > if (!MY_UTF16_LOW_HEAD(s[2])) /* Broken surrigate pair */ > return MY_CS_ILSEQ; > >- /* >- s[0]= 110110?? (<< 18) >- s[1]= ???????? (<< 10) >- s[2]= 110111?? (<< 8) >- s[3]= ???????? (<< 0) >- */ >- >- *pwc= ((s[0] & 3) << 18) + (s[1] << 10) + >- ((s[2] & 3) << 8) + s[3] + 0x10000; >- >+ *pwc= MY_UTF16_WC4(s[0], s[1], s[2], s[3]); > return 4; > } > > if (MY_UTF16_LOW_HEAD(*s)) /* Low surrogate part without high part */ > return MY_CS_ILSEQ; >- >- *pwc= (s[0] << 8) + s[1]; >+ >+ *pwc= MY_UTF16_WC2(s[0], s[1]); > return 2; > } > >@@ -1141,10 +1208,10 @@ > DBUG_ASSERT(src == dst && srclen == dstlen); > > while ((src < srcend) && >- (res= my_utf16_uni(cs, &wc, (uchar *)src, (uchar*) srcend)) > 0) >+ (res= cs->cset->mb_wc(cs, &wc, (uchar *) src, (uchar *) srcend)) > 0) > { > my_toupper_utf16(uni_plane, &wc); >- if (res != my_uni_utf16(cs, wc, (uchar*) src, (uchar*) srcend)) >+ if (res != cs->cset->wc_mb(cs, wc, (uchar *) src, (uchar *) srcend)) > break; > src+= res; > } Monty: Check the assembler if it would make sence to store cs->cset->mb_wc in a register. (It probably would as the assembler can't assume that the pointer is unchanged over calls). This applies to all cases where you use 'cs->cset->mb_wc' in a loop. >@@ -1158,13 +1225,11 @@ > { > my_wc_t wc; > int res; >- const uchar *e= s+slen; >+ const uchar *e= s + cs->cset->lengthsp(cs, (const char *) s, slen); > MY_UNICASE_INFO *const *uni_plane= cs->caseinfo; > >- while (e > s + 1 && e[-1] == ' ' && e[-2] == '\0') >- e-= 2; >- >- while ((s < e) && (res= my_utf16_uni(cs, &wc, (uchar *)s, (uchar*)e)) > 0) >+ while ((s < e) && (res= cs->cset->mb_wc(cs, &wc, >+ (uchar *) s, (uchar *) e)) > 0) > { > my_tosort_utf16(uni_plane, &wc); > n1[0]^= (((n1[0] & 63) + n2[0]) * (wc & 0xFF)) + (n1[0] << 8); >@@ -1188,10 +1253,10 @@ > DBUG_ASSERT(src == dst && srclen == dstlen); > > while ((src < srcend) && >- (res= my_utf16_uni(cs, &wc, (uchar*) src, (uchar*) srcend)) > 0) >+ (res= cs->cset->mb_wc(cs, &wc, (uchar *) src, (uchar *) srcend)) > 0) > { > my_tolower_utf16(uni_plane, &wc); >- if (res != my_uni_utf16(cs, wc, (uchar*) src, (uchar*) srcend)) >+ if (res != cs->cset->wc_mb(cs, wc, (uchar *) src, (uchar *) srcend)) > break; > src+= res; > } >@@ -1213,8 +1278,8 @@ > > while (s < se && t < te) > { >- s_res= my_utf16_uni(cs, &s_wc, s, se); >- t_res= my_utf16_uni(cs, &t_wc, t, te); >+ s_res= cs->cset->mb_wc(cs, &s_wc, s, se); >+ t_res= cs->cset->mb_wc(cs, &t_wc, t, te); > > if (s_res <= 0 || t_res <= 0) > { >@@ -1283,8 +1348,8 @@ > > while (s < se && t < te) > { >- int s_res= my_utf16_uni(cs, &s_wc, s, se); >- int t_res= my_utf16_uni(cs, &t_wc, t, te); >+ int s_res= cs->cset->mb_wc(cs, &s_wc, s, se); >+ int t_res= cs->cset->mb_wc(cs, &t_wc, t, te); > > if (s_res <= 0 || t_res <= 0) > { >@@ -1324,7 +1389,7 @@ > > for ( ; s < se; s+= s_res) > { >- if ((s_res= my_utf16_uni(cs, &s_wc, s, se)) < 0) >+ if ((s_res= cs->cset->mb_wc(cs, &s_wc, s, se)) < 0) > { > DBUG_ASSERT(0); > return 0; >@@ -1338,22 +1403,11 @@ > > > static uint >-my_ismbchar_utf16(CHARSET_INFO *cs __attribute__((unused)), >- const char *b __attribute__((unused)), >- const char *e __attribute__((unused))) >+my_ismbchar_utf16(CHARSET_INFO *cs, const char *b, const char *e) > { >- if (b + 2 > e) >- return 0; >- >- if (MY_UTF16_HIGH_HEAD(*b)) >- { >- return (b + 4 <= e) && MY_UTF16_LOW_HEAD(b[2]) ? 4 : 0; >- } >- >- if (MY_UTF16_LOW_HEAD(*b)) >- return 0; >- >- return 2; >+ my_wc_t wc; >+ int res= cs->cset->mb_wc(cs, &wc, (const uchar *) b, (const uchar *) e); >+ return (uint) (res > 0 ? res : 0); > } > > >@@ -1361,6 +1415,7 @@ > my_mbcharlen_utf16(CHARSET_INFO *cs __attribute__((unused)), > uint c __attribute__((unused))) > { >+ DBUG_ASSERT(0); > return MY_UTF16_HIGH_HEAD(c) ? 4 : 2; > } > >@@ -1454,8 +1509,8 @@ > > while ( s < se && t < te ) > { >- s_res= my_utf16_uni(cs,&s_wc, s, se); >- t_res= my_utf16_uni(cs,&t_wc, t, te); >+ s_res= cs->cset->mb_wc(cs, &s_wc, s, se); >+ t_res= cs->cset->mb_wc(cs, &t_wc, t, te); > > if (s_res <= 0 || t_res <= 0) > { >@@ -1493,8 +1548,8 @@ > > while (s < se && t < te) > { >- int s_res= my_utf16_uni(cs, &s_wc, s, se); >- int t_res= my_utf16_uni(cs, &t_wc, t, te); >+ int s_res= cs->cset->mb_wc(cs, &s_wc, s, se); >+ int t_res= cs->cset->mb_wc(cs, &t_wc, t, te); > > if (s_res <= 0 || t_res <= 0) > { >@@ -1531,7 +1586,7 @@ > > for ( ; s < se; s+= s_res) > { >- if ((s_res= my_utf16_uni(cs, &s_wc, s, se)) < 0) >+ if ((s_res= cs->cset->mb_wc(cs, &s_wc, s, se)) < 0) > { > DBUG_ASSERT(0); > return 0; >@@ -1545,17 +1600,11 @@ > > > static void >-my_hash_sort_utf16_bin(CHARSET_INFO *cs __attribute__((unused)), >- const uchar *key, size_t len,ulong *nr1, ulong *nr2) >+my_hash_sort_utf16_bin(CHARSET_INFO *cs, >+ const uchar *pos, size_t len, ulong *nr1, ulong *nr2) > { >- const uchar *pos = key; >- >- key+= len; >- >- while (key > pos + 1 && key[-1] == ' ' && key[-2] == '\0') >- key-= 2; >- >- for (; pos < (uchar*) key ; pos++) >+ const uchar *end= pos + cs->cset->lengthsp(cs, (const char *) pos, len); >+ for ( ; pos < end ; pos++) > { > nr1[0]^= (ulong) ((((uint) nr1[0] & 63) + nr2[0]) * > ((uint)*pos)) + (nr1[0] << 8); >@@ -1664,7 +1713,7 @@ > struct charset_info_st my_charset_utf16_bin= > { > 55,0,0, /* number */ >- MY_CS_COMPILED|MY_CS_BINSORT|MY_CS_UNICODE|MY_CS_NONASCII, >+ MY_CS_COMPILED|MY_CS_BINSORT|MY_CS_STRNXFRM|MY_CS_UNICODE|MY_CS_NONASCII, > "utf16", /* cs name */ > "utf16_bin", /* name */ > "UTF-16 Unicode", /* comment */ >@@ -1693,6 +1742,173 @@ > &my_collation_utf16_bin_handler > }; > >+ >+static int >+my_utf16le_uni(const CHARSET_INFO *cs __attribute__((unused)), >+ my_wc_t *pwc, const uchar *s, const uchar *e) >+{ >+ my_wc_t lo; >+ >+ if (s + 2 > e) >+ return MY_CS_TOOSMALL2; >+ >+ if ((*pwc= uint2korr(s)) < MY_UTF16_SURROGATE_HIGH_FIRST || >+ (*pwc > MY_UTF16_SURROGATE_LOW_LAST)) >+ return 2; /* [0000-D7FF,E000-FFFF] */ >+ >+ if (*pwc >= MY_UTF16_SURROGATE_LOW_FIRST) >+ return MY_CS_ILSEQ; /* [DC00-DFFF] Low surrogate part without high part */ >+ >+ if (s + 4 > e) >+ return MY_CS_TOOSMALL4; >+ >+ s+= 2; >+ >+ if ((lo= uint2korr(s)) < MY_UTF16_SURROGATE_LOW_FIRST || >+ lo > MY_UTF16_SURROGATE_LOW_LAST) >+ return MY_CS_ILSEQ; /* Expected low surrogate part, got something else */ >+ >+ *pwc= 0x10000 + (((*pwc & 0x3FF) << 10) | (lo & 0x3FF)); >+ return 4; >+} >+ >+ >+static int >+my_uni_utf16le(const CHARSET_INFO *cs __attribute__((unused)), >+ my_wc_t wc, uchar *s, uchar *e) >+{ >+ if (wc < MY_UTF16_SURROGATE_HIGH_FIRST || >+ (wc > MY_UTF16_SURROGATE_LOW_LAST && >+ wc <= 0xFFFF)) >+ { >+ if (s + 2 > e) >+ return MY_CS_TOOSMALL2; >+ int2store(s, wc); >+ return 2; /* [0000-D7FF,E000-FFFF] */ >+ } >+ >+ if (wc < 0xFFFF || wc > 0x10FFFF) >+ return MY_CS_ILUNI; /* [D800-DFFF,10FFFF+] */ >+ >+ if (s + 4 > e) >+ return MY_CS_TOOSMALL4; >+ >+ wc-= 0x10000; >+ int2store(s, (0xD800 | ((wc >> 10) & 0x3FF))); s+= 2; >+ int2store(s, (0xDC00 | (wc & 0x3FF))); Monty: Use int2store(s+2... for the second row and remove s+2. Makes the line easier to read... Note that as int2store is a macro, doing it this is maybe faster: uint16 first_part, second_part; uint32 total; first_part= (0xD800 | ((wc >> 10) & 0x3FF))); second_part= (0xDC00 | (wc & 0x3FF))); total= first_part + (second_part << 16) int4store(s, total); >+ return 4; /* [010000-10FFFF] */ >+} >+ >+ >+static size_t >+my_lengthsp_utf16le(const CHARSET_INFO *cs __attribute__((unused)), >+ const char *ptr, size_t length) >+{ >+ const char *end= ptr + length; >+ while (end > ptr + 1 && uint2korr(end - 2) == 0x20) Monty Using ' ' instead of 0x20 is maybe more clear? >+ end-= 2; >+ return (size_t) (end - ptr); >+} >+ >+ >+static MY_CHARSET_HANDLER my_charset_utf16le_handler= >+{ >+ NULL, /* init */ >+ my_ismbchar_utf16, >+ my_mbcharlen_utf16, >+ my_numchars_utf16, >+ my_charpos_utf16, >+ my_well_formed_len_utf16, >+ my_lengthsp_utf16le, >+ my_numcells_mb, >+ my_utf16le_uni, /* mb_wc */ >+ my_uni_utf16le, /* wc_mb */ >+ my_mb_ctype_mb, >+ my_caseup_str_mb2_or_mb4, >+ my_casedn_str_mb2_or_mb4, >+ my_caseup_utf16, >+ my_casedn_utf16, >+ my_snprintf_mb2, >+ my_l10tostr_mb2_or_mb4, >+ my_ll10tostr_mb2_or_mb4, >+ my_fill_mb2, >+ my_strntol_mb2_or_mb4, >+ my_strntoul_mb2_or_mb4, >+ my_strntoll_mb2_or_mb4, >+ my_strntoull_mb2_or_mb4, >+ my_strntod_mb2_or_mb4, >+ my_strtoll10_mb2, >+ my_strntoull10rnd_mb2_or_mb4, >+ my_scan_mb2 >+}; >+ >+ >+struct charset_info_st my_charset_utf16le_general_ci= >+{ >+ 56,0,0, /* number */ >+ MY_CS_COMPILED|MY_CS_PRIMARY|MY_CS_STRNXFRM|MY_CS_UNICODE|MY_CS_NONASCII, >+ "utf16le", /* cs name */ >+ "utf16le_general_ci",/* name */ >+ "UTF-16LE Unicode", /* comment */ >+ NULL, /* tailoring */ >+ NULL, /* ctype */ >+ NULL, /* to_lower */ >+ NULL, /* to_upper */ >+ NULL, /* sort_order */ >+ NULL, /* contractions */ >+ NULL, /* sort_order_big*/ >+ NULL, /* tab_to_uni */ >+ NULL, /* tab_from_uni */ >+ my_unicase_default, /* caseinfo */ >+ NULL, /* state_map */ >+ NULL, /* ident_map */ >+ 1, /* strxfrm_multiply */ >+ 1, /* caseup_multiply */ >+ 1, /* casedn_multiply */ >+ 2, /* mbminlen */ >+ 4, /* mbmaxlen */ >+ 0, /* min_sort_char */ >+ 0xFFFF, /* max_sort_char */ >+ ' ', /* pad char */ >+ 0, /* escape_with_backslash_is_dangerous */ >+ &my_charset_utf16le_handler, >+ &my_collation_utf16_general_ci_handler >+}; >+ >+ >+struct charset_info_st my_charset_utf16le_bin= >+{ >+ 62,0,0, /* number */ >+ MY_CS_COMPILED|MY_CS_BINSORT|MY_CS_STRNXFRM|MY_CS_UNICODE|MY_CS_NONASCII, >+ "utf16le", /* cs name */ >+ "utf16le_bin", /* name */ >+ "UTF-16LE Unicode", /* comment */ >+ NULL, /* tailoring */ >+ NULL, /* ctype */ >+ NULL, /* to_lower */ >+ NULL, /* to_upper */ >+ NULL, /* sort_order */ >+ NULL, /* contractions */ >+ NULL, /* sort_order_big*/ >+ NULL, /* tab_to_uni */ >+ NULL, /* tab_from_uni */ >+ my_unicase_default, /* caseinfo */ >+ NULL, /* state_map */ >+ NULL, /* ident_map */ >+ 1, /* strxfrm_multiply */ >+ 1, /* caseup_multiply */ >+ 1, /* casedn_multiply */ >+ 2, /* mbminlen */ >+ 4, /* mbmaxlen */ >+ 0, /* min_sort_char */ >+ 0xFFFF, /* max_sort_char */ >+ ' ', /* pad char */ >+ 0, /* escape_with_backslash_is_dangerous */ >+ &my_charset_utf16le_handler, >+ &my_collation_utf16_bin_handler >+}; >+ >+ > #endif /* HAVE_CHARSET_utf16 */ > > >@@ -2589,7 +2805,7 @@ > struct charset_info_st my_charset_utf32_bin= > { > 61,0,0, /* number */ >- MY_CS_COMPILED|MY_CS_BINSORT|MY_CS_UNICODE|MY_CS_NONASCII, >+ MY_CS_COMPILED|MY_CS_BINSORT|MY_CS_STRNXFRM|MY_CS_UNICODE|MY_CS_NONASCII, > "utf32", /* cs name */ > "utf32_bin", /* name */ > "UTF-32 Unicode", /* comment */ >@@ -2708,6 +2924,35 @@ > } > > >+static inline void >+my_tolower_ucs2(MY_UNICASE_INFO *const *uni_plane, my_wc_t *wc) >+{ >+ int page= *wc >> 8; >+ DBUG_ASSERT(page < 256); monty: make page uint both here and in other places. >+ if (uni_plane[page]) >+ *wc= uni_plane[page][*wc & 0xFF].tolower; >+} >+ >+ >+static inline void >+my_toupper_ucs2(MY_UNICASE_INFO *const *uni_plane, my_wc_t *wc) >+{ >+ int page= *wc >> 8; >+ DBUG_ASSERT(page < 256); >+ if (uni_plane[page]) >+ *wc= uni_plane[page][*wc & 0xFF].toupper; >+} >+ >+ >+static inline void >+my_tosort_ucs2(MY_UNICASE_INFO *const *uni_plane, my_wc_t *wc) >+{ >+ int page= *wc >> 8; >+ DBUG_ASSERT(page < 256); >+ if (uni_plane[page]) >+ *wc= uni_plane[page][*wc & 0xFF].sort; >+} >+ > static size_t my_caseup_ucs2(CHARSET_INFO *cs, char *src, size_t srclen, > char *dst __attribute__((unused)), > size_t dstlen __attribute__((unused))) >@@ -2721,8 +2966,7 @@ > while ((src < srcend) && > (res= my_ucs2_uni(cs, &wc, (uchar *)src, (uchar*) srcend)) > 0) > { >- int plane= (wc>>8) & 0xFF; >- wc= uni_plane[plane] ? uni_plane[plane][wc & 0xFF].toupper : wc; >+ my_toupper_ucs2(uni_plane, &wc); > if (res != my_uni_ucs2(cs, wc, (uchar*) src, (uchar*) srcend)) > break; > src+= res; >@@ -2744,8 +2988,7 @@ > > while ((s < e) && (res=my_ucs2_uni(cs,&wc, (uchar *)s, (uchar*)e)) >0) > { >- int plane = (wc>>8) & 0xFF; >- wc = uni_plane[plane] ? uni_plane[plane][wc & 0xFF].sort : wc; >+ my_tosort_ucs2(uni_plane, &wc); > n1[0]^= (((n1[0] & 63)+n2[0])*(wc & 0xFF))+ (n1[0] << 8); > n2[0]+=3; > n1[0]^= (((n1[0] & 63)+n2[0])*(wc >> 8))+ (n1[0] << 8); >@@ -2768,8 +3011,7 @@ > while ((src < srcend) && > (res= my_ucs2_uni(cs, &wc, (uchar*) src, (uchar*) srcend)) > 0) > { >- int plane= (wc>>8) & 0xFF; >- wc= uni_plane[plane] ? uni_plane[plane][wc & 0xFF].tolower : wc; >+ my_tolower_ucs2(uni_plane, &wc); > if (res != my_uni_ucs2(cs, wc, (uchar*) src, (uchar*) srcend)) > break; > src+= res; >@@ -2778,6 +3020,14 @@ > } > > >+static void >+my_fill_ucs2(CHARSET_INFO *cs __attribute__((unused)), >+ char *s, size_t l, int fill) monty: Is it time to make 'fill' an unsigned int? >+{ >+ DBUG_ASSERT(fill <= 0xFFFF); >+ for ( ; l >= 2; s[0]= (fill >> 8), s[1]= (fill & 0xFF), s+= 2, l-= 2); >+} Monty: You can make the above faster if you store fill >>8 and fill & 0xff in registers. >+ > static int my_strnncoll_ucs2(CHARSET_INFO *cs, > const uchar *s, size_t slen, > const uchar *t, size_t tlen, >@@ -2791,7 +3041,6 @@ > > while ( s < se && t < te ) > { >- int plane; > s_res=my_ucs2_uni(cs,&s_wc, s, se); > t_res=my_ucs2_uni(cs,&t_wc, t, te); > >@@ -2801,10 +3050,9 @@ > return ((int)s[0]-(int)t[0]); > } > >- plane=(s_wc>>8) & 0xFF; >- s_wc = uni_plane[plane] ? uni_plane[plane][s_wc & 0xFF].sort : s_wc; >- plane=(t_wc>>8) & 0xFF; >- t_wc = uni_plane[plane] ? uni_plane[plane][t_wc & 0xFF].sort : t_wc; >+ my_tosort_ucs2(uni_plane, &s_wc); >+ my_tosort_ucs2(uni_plane, &t_wc); >+ > if ( s_wc != t_wc ) > { > return s_wc > t_wc ? 1 : -1; >@@ -3115,7 +3363,7 @@ > my_snprintf_mb2, > my_l10tostr_mb2_or_mb4, > my_ll10tostr_mb2_or_mb4, >- my_fill_mb2, >+ my_fill_ucs2, > my_strntol_mb2_or_mb4, > my_strntoul_mb2_or_mb4, > my_strntoll_mb2_or_mb4, > Regards, Monty

2 1

Re: [Maria-developers] Aria vs MyISAM key_buffer_size
by Michael Widenius 19 Mar '13

19 Mar '13

Hi! I am cc:ing Maria-developers list, as this is a typical question that can be asked and answered here. >>>>> "Ian" == Ian Gilfillan <ian(a)greenman.co.za> writes: Ian> Hi Monty Ian> I have a question on key-buffer-size that wasn't answered definitively Ian> by others on IRC. Ian> As I understand it, by default, Aria doesn't use the key_buffer_size Ian> setting at all, it makes use of aria-pagecache-buffer-size. Yes, both are basicly doing the same thing, but for different storage engines. The main difference is that for Aria we can also store rows in it if you user ROW_FORMAT=PAGE Ian> However, if Ian> Aria tables are created with ROW_FORMAT=FIXED or ROW_FORMAT=DYNAMIC to Ian> emulate MyISAM, are they then emulating MyISAM's use of key-buffer-size Ian> as well, and not making use of Aria cache settings? It's not really emulation, but more that Aria has, for these cases, identical code for handling row data.. For the above ROW_FORMAT's we store *ONLY* the key pages in the aria page cache. The rows are not cashed (like we do with MyISAM). Regards, Monty

1 0

[Maria-developers] impact of segmented table_open_cache on sysbench results
by Axel Schwenke 16 Mar '13

16 Mar '13

Hi, I have been running a sysbench on recent 5.5, 5.6 and 10.0 releases with tuning a'la Dimitri K. There is one feature that is only in MySQL-5.6.10: table_open_cache_instances = 16 and I wondered how it would affect the sysbench result. It makes a *huge* difference. For the curious I also add the result from a previous run that shows the impact of tcmalloc vs. glibc malloc. Disclaimer: hardware is 64 cores (32 w/o HT) XL

2 3

[Maria-developers] mariadb++
by Sylvain Rochette 14 Mar '13

14 Mar '13

I just want to inform that the last few weeks I worked on a new open source project called mariadb++. It's use mariadb-native-client lib, it an alternative over MySQL++/MySQL connector. It's available here: https://launchpad.net/mariadb++ Regards, Sylvain Rochette

2 1

Re: [Maria-developers] [Commits] Rev 3697: MDEV-4272 fix. in file:///home/bell/maria/bzr/work-maria-5.5-MDEV-4272/
by Sergei Golubchik 14 Mar '13

14 Mar '13

Hi, Sanja! The idea of the fix is correct. See a couple of comments below. On Mar 14, sanja(a)askmonty.org wrote: > message: > MDEV-4272 fix. > > Incorrect NULL value handling in Item_func_conv_charset fixed. > === modified file 'sql/item_strfunc.cc' > --- a/sql/item_strfunc.cc 2013-01-28 12:36:05 +0000 > +++ b/sql/item_strfunc.cc 2013-03-14 13:22:32 +0000 > @@ -2987,7 +2987,7 @@ String *Item_func_conv_charset::val_str( > return null_value ? 0 : &str_value; > String *arg= args[0]->val_str(str); > uint dummy_errors; > - if (!arg) > + if (!arg || args[0]->null_value) it's redundant. One of these conditions is enough, no need to check both. > { > null_value=1; > return 0; > > === modified file 'sql/item_strfunc.h' > --- a/sql/item_strfunc.h 2013-01-15 18:07:46 +0000 > +++ b/sql/item_strfunc.h 2013-03-14 13:22:32 +0000 > @@ -859,25 +859,45 @@ public: > { > if (args[0]->result_type() == STRING_RESULT) > return Item_str_func::val_int(); > - return args[0]->val_int(); > + { > + longlong res= args[0]->val_int(); > + if ((null_value= args[0]->null_value)) > + return 0; > + return res; > + } Why did you add an extra pair of curly brackets? it's also redundant. Regards, Sergei

1 0

[Maria-developers] accelerating CREATE TABLE
by Toshikuni Fukaya 13 Mar '13

13 Mar '13

Hi, I made a patch to accelerate CREATE TABLE on the innodb plugin. To zero table spaces, I used fallocate instead of normal writes and sync. I compared MariaDB 5.5 trunk and the patched one by our own application, which creates > 1000 tables in series, with ext4 filesystem on Ubuntu 12.04. Results are shown below. Original MariaDB 5.5 trunk: 64.9 seconds (5 times avg.) Patched MariaDB 5.5 trunk: 36.0 seconds (5 times avg.) Any comments or suggestions? Thanks, Toshikuni Fukaya

3 8

[Maria-developers] MariaDB 5.5.30 Release Prep
by Daniel Bartholomew 12 Mar '13

12 Mar '13

All, I've prepped changelog and release notes pages for MariaDB 5.5.30. https://kb.askmonty.org/en/mariadb-5530-release-notes/ https://kb.askmonty.org/en/mariadb-5530-changelog/ If there is anything I should be sure to mention in the release notes, please let me know ASAP. Unless something comes up, I plan on activating and announcing the release tomorrow (12 Mar 2013). Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

[Maria-developers] issue with Index Condition Pushdown in MariaDB 5.5.28a
by Zardosht Kasheff 11 Mar '13

11 Mar '13

Hello, I have found an interesting issue with index condition pushdown. I do not know if it is a bug, or if I am misusing the feature. I have found that in one scenario, when performing a join, an index condition is pushed down, but because handler->end_range is not set, handler_index_cond_check never returns ICP_OUT_OF_RANGE, even though we are out of range. Here is the relevant stack trace, I cannot yet send the entire trace as it may include customer information, and I need to double check it is ok to send: #15 0x00002aaac4f33593 in ha_tokudb::index_next_same (this=0x2fb33d8, buf=0x2fb53b8 "\302\177\254\353/\036", key=0x31b36d0 "Y\001", keylen=395) ---Type <return> to continue, or q <return> to quit--- at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/storage/tokudb/ha_tokudb.cc:4768 #16 0x00000000005e782c in handler::ha_index_next_same (this=0x2fb33d8, buf=0x2fb53b8 "\302\177\254\353/\036", key=0x31b36d0 "Y\001", keylen=395) at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/sql/sql_class.h:4246 #17 0x000000000065d103 in join_read_next_same (info=0x31b2f68) at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/sql/sql_select.cc:16929 #18 0x000000000065b54a in sub_select (join=0x319f588, join_tab=0x31b2eb8, end_of_records=false) at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/sql/sql_select.cc:16142 #19 0x000000000065bb5f in evaluate_join_record (join=0x319f588, join_tab=0x31b2b98, error=0) at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/sql/sql_select.cc:16342 #20 0x000000000065b628 in sub_select (join=0x319f588, join_tab=0x31b2b98, end_of_records=false) at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/sql/sql_select.cc:16162 #21 0x000000000065bb5f in evaluate_join_record (join=0x319f588, join_tab=0x31b2878, error=0) at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/sql/sql_select.cc:16342 #22 0x000000000065b498 in sub_select (join=0x319f588, join_tab=0x31b2878, end_of_records=false) at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/sql/sql_select.cc:16123 #23 0x000000000065bb5f in evaluate_join_record (join=0x319f588, join_tab=0x31b2558, error=0) at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/sql/sql_select.cc:16342 #24 0x000000000065b628 in sub_select (join=0x319f588, join_tab=0x31b2558, end_of_records=false) at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/sql/sql_select.cc:16162 #25 0x000000000065ad52 in do_select (join=0x319f588, fields=0x0, table=0x31bf368, procedure=0x0) at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/sql/sql_select.cc:15793 #26 0x0000000000639c5d in JOIN::exec (this=0x319f588) at /home/zardosht/maria/mysql-build/mariadb-5.5.28a/sql/sql_select.cc:2396 In this stack, we are performing an index_next_same, and a condition has been pushed down. When we call handler_index_cond_check. we always get ICP_NO_MATCH and never get ICP_OUT_OF_RANGE, even though we go past the key that we should be retrieving. Is this by design? Is this a bug? Thanks -Zardosht

2 2

[Maria-developers] Spam in KB?
by Sergei Petrunia 07 Mar '13

07 Mar '13

Hi Daniel, I am looking at https://kb.askmonty.org/en/mariadb-5529-release-notes/ and I see at the top in the breadcrumbs: [Home] [ Why Shared Web Hosting Service Is Ideal For Small Businesses] [Development] and when I click on the shared hosting I get to: https://kb.askmonty.org/en/why-shared-web-hosting-service-is-ideal-for-smal… In my understanding this text has nothing to do with MariaDB? Is it link spam? BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] Rev 3633: MDEV-4241 fix
by Sergei Golubchik 06 Mar '13

06 Mar '13

Hi, Sanja! On Mar 06, sanja(a)askmonty.org wrote: > revno: 3633 > revision-id: sanja(a)askmonty.org-20130306191058-d293nik3l66vn4tp > parent: psergey(a)askmonty.org-20130301042335-x16j7uo1dwxxgqs9 > committer: sanja(a)askmonty.org > branch nick: work-maria-5.3-MDEV-4241 > timestamp: Wed 2013-03-06 21:10:58 +0200 > message: > MDEV-4241 fix. > > Field_enum incorrectly inherited decimals() from Field_string. > Field_enum should be always integer in numeric context. Ok to push! Regards, Sergei

1 0

[Maria-developers] WL#253 New (by Igor): Take into account the selectivity of single-table range predicates on non-indexed columns
by worklog-noreply＠askmonty.org 06 Mar '13

06 Mar '13

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Take into account the selectivity of single-table range predicates on non-indexed columns CREATION DATE..: Wed, 06 Mar 2013, 17:25 SUPERVISOR.....: IMPLEMENTOR....: Igor COPIES TO......: Psergey CATEGORY.......: Server-BackLog TASK ID........: 253 (http://askmonty.org/worklog/?tid=253) VERSION........: Server-9.x STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Description of the problem ========================== Currently the optimizer completely ignores any conditions on non-indexed columns when searching for the best execution plan. As a result in many cases it chooses a suboptimal plan. Let's consider the query: SELECT * FROM t1,t2 WHERE t1.a=t2.a and t2.b BETWEEN (1,3) assuming that: -table t1 contains 100 records -table t2 contains 1000 records -there is a primary index on t1(a) -there is a secondary index on t2(a) -there is no index defined on column t2.b -the selectivity of the condition t2.b BETWEEN (1,3) is high (~ 1%) Under these conditions currently the optimizer will choose the plan that accesses t1 using a table scan accesses t2 using index t2(a) checks the condition t2.b BETWEEN (1,3) This plan examines all rows of both tables and it performs 100 index look-ups. The alternative plan that: accesses table t2 in a table scan checks the condition t2.b BETWEEN (1,3) accesses t1 using index t1(a) also would examine all rows of t2, but it performs only ~10 look-ups to access ~10 rows of table t1. The second plan is expected to be more efficient and it is so. The plan to resolve the problem =============================== 1. Build single-table range conditions over different columns of all tables of the join query. 2. Calculate the selectivity of each such condition using persistent statistics on table columns. 3. Take this statistics into account when calculated the cardinality of each partial join that are evaluated by the optimizer. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] deadlock, rotate and prepared xids
by MARK CALLAGHAN 05 Mar '13

05 Mar '13

Kristian, Did you do anything special to avoid deadlock with ::new_file_impl and prepared XIDs? My memory is vague but I thought you handled it. I added code to the FB patch group commit implementation to handle it. Both Yoshi and I hit deadlocks with that code in official MySQL 5.6: http://bugs.mysql.com/bug.php?id=68569 http://bugs.mysql.com/bug.php?id=68251 -- Mark Callaghan mdcallag(a)gmail.com

2 3

[Maria-developers] C API for dynamic columns - in which versions?
by Honza Horak 01 Mar '13

01 Mar '13

Hi guys, I failed to find the info in which versions of MariaDB "C API for dynamic columns" is available as described here: https://kb.askmonty.org/en/dynamic-columns-api/ I think the information should be the best included directly on the page itself. Regards, Honza

2 2

[Maria-developers] How to copy IO_CACHE into a string?
by Lixun Peng 28 Feb '13

28 Feb '13

Hi Guys, I want to copy the contents of the IO_CACHE into a string. But I'm not found a function can do it, but I found "my_b_copy_to_file" can do the similar thing, so I implement a function "my_b_copy_to_string" by imitating "my_b_copy_to_file", to do this thing. Could you please review my code? char *my_b_copy_to_string(IO_CACHE *cache, size_t *bytes_in_cache) { char *buff; char *tmp_buff; size_t now_size; size_t inc_size; /* Reinit the cache to read from the beginning of the cache */ if (reinit_io_cache(cache, READ_CACHE, 0L, FALSE, FALSE)) return NULL; now_size= my_b_bytes_in_cache(cache); inc_size= 0; buff= (char *) my_malloc(now_size + 1, MYF(0)); tmp_buff= buff; do { now_size+= inc_size; if(inc_size > 0) { buff= (char *) my_realloc(buff, now_size + 1, MYF(0)); tmp_buff= buff + (now_size - inc_size); memcpy(tmp_buff, cache->read_pos, inc_size); } else { memcpy(tmp_buff, cache->read_pos, now_size); } cache->read_pos= cache->read_end; } while ((inc_size= my_b_fill(cache))); buff[now_size]= '\0'; reinit_io_cache(cache, WRITE_CACHE, 0, FALSE, TRUE); *bytes_in_cache= now_size; return buff; } Thanks, Lixun -- Senior MySQL Developer @ Taobao.com Mobile Phone: +86 18658156856 (Hangzhou) Gtalk: penglixun(at)gmail.com Twitter: http://www.twitter.com/plinux Blog: http://www.penglixun.com

2 2

Re: [Maria-developers] GTID: clueless user experience, day 1
by Kristian Nielsen 27 Feb '13

27 Feb '13

[Cc: maria-developers@, in case others are interested or have comments] Elena Stepanova <elenst(a)montyprogram.com> writes: >> So if you agree, I will put on my ToDo to remove that omitted domain_id >> defaults to 0. > > It's fine with me. Pushed to lp:~maria-captains/maria/10.0-mdev26/ >> Clearly I need to give a better error message, this is highly >> confusing. Thanks for bringing this to attention! What do you think of: >> >> ERROR XXXX: Duplicate domain_id 0 for GTID 0-2-1 I made it like this: MariaDB [test]> change master to master_gtid_pos="0-1-2,1-1000-1000,2-3-1,0-2-100,4-4-4"; ERROR 1941 (HY000): GTID 0-2-100 and 0-1-2 conflict (duplicate domain id 0) > Sounds good. Only, if I understand correctly, it's nowadays not > domain_id, but gtid_domain_id ? Then it's better be named as such in > the error message (/me tried in vain to set domain_id, first on the > server command line, and then in a session). Well, these are different but related concepts. @@gtid_domain_id is a system variable. A "domain id" is a conceptual thing when taking about global transaction ID. It is like the InnoDB buffer pool. The system variable is "innodb_buffer_pool_size". The concept is "buffer pool size". But I agree the concept should be "domain ID", not domain_id (with an underscore). The reason I spell this out is not to nitpick, but rather that I would welcome suggestions for naming, I am terrible with naming. Do you think I should rename @@gtid_domain_id and @@gtid_seq_no to just @@domain_id and @@seq_no? I thought that was perhaps a bit too generic names, though I did keep plain @@server_id for backwards compatibility. Another option is @@binlog_domain_id and @@binlog_seq_no, this is perhaps better than the random "gtid" prefix? The variables are for controlling what goes into the binlog, after all. > But on the other hand, why should it be several rows? If > binlog_gtid_pos() returns a *single* comma-separated list of GTIDs for > the given position, as it does now, we can just as well add this list > to SHOW MASTER STATUS as a *column*, no? Sorry, it was me who was confused. I meant adding extra columns, not rows. I am still not sure if it is safe to add extra columns. It would seem useful, unless it breaks too many existing applications. >> Start after. GTID is the last event executed. > Hm... I must admit I don't quite understand the reasoning. I'll ask Let me give an example. Suppose server 1 is master. It binlogs an event with GTID 0-1-10. This event is currently the last in the binlog. Server 2 is a slave of server 1. When it catches up, GTID 0-1-10 is the last event applied on the slave. So now, what should be the replication state of slave server 2? If we want to use "start from" we could try GTID 0-1-11? But then suppose we now switch master to a new server 3. Then the event following 0-1-10 is 0-2-11, not 0-1-11. So if server 2 would try to connect as a slave to server 3 and start from 0-1-11, it will not work, this event does not exist anywhere. That is why it has to be "start after", as when we save the state on the slave, we know only the GTID of the last event applied, not the one to apply next. > There might be more in there for me to look into.. Indeed. This is a critical part of the design, I spent a _lot_ of thought to get this working, and I think I got it right, but I will feel a lot better knowing you spent your best efforts to break it :-). But please wait with this for a later stage, I have some planned development around this that is not ready yet. >> Gtid_Pos_Auto=1 means the new method, whether AUTO or explicit 'xxx' gtid, >> So what should we call it? How about just "Using_Gtid" ? > > Yep, Using_Gtid sounds fine. This is fixed in the tree now. > This brings a more conceptual question for me... Does Using_Gtid > indicate how the slave *replicates*, or just how it determines the > starting position while connecting to master? In other words, is there There is a difference. There are two ways a slave can connect. The old way using binlog file and position. And the new way using GTID. I think we need to have both. The Using_Gtid records which of the two is used. I made it this way so that users do not need to set some specific option in my.cnf, and so that with multi-source, one slave connection can use the old style (eg. master not supporting GTID?), while another can use new style. The main difference is finding the starting position. But this also implies difference for how the slave replicates. Because with GTID there can be *multiple* starting positions (one for each domain id). So after connecting, some events can be skipped during replication if one domain_id is ahead of another in the master binlog. > If not (and I couldn't yet figure the difference, since, as you said, > the slave reads the GTID state anyway when it connects with file > name/offset), then I start having doubts that we need the Using_Gtid > flag at all. I mean, if it only indicates what I provided last time in > CHANGE MASTER, how is this information useful? It is used when the slave connects (which happens at every START SLAVE, and every server restart, including mysqld_safe autostart). If Using_Gtid is false, then slave connects at the last file/offset found in relay-log.info. It also obtains the corresponding GTID position from the master, so that it is ready in case the user wants to switch to CHANGE MASTER TO ... MASTER_GTID_POS=AUTO. If Using_Gtid is true, slave connects at the GTID position read from the mysql.rpl_slave_state table, which is crash safe. And multiple start positions are used if there is more than one domain id in the GTID position. So you see, the information is needed, otherwise the slave and the master would not know how to handle slave connect. >> But frankly, I am not sure if the right place is in SHOW ALL SLAVES STATUS or >> in plain SHOW SLAVE STATUS. Can you help me understand what the difference is >> between putting the column in one or the other of these? > > I guess the only real reason was that Monty didn't want to change the > output format of existing SHOW SLAVE STATUS, not to break anything. Yes, this is an important issue. I will discuss it with Monty when I visit him next week. > Ideally, I'd expect the global values be shown both in SHOW SLAVE > STATUS, e.g. at the end of it, and once for all slaves in SHOW ALL > SLAVES STATUS, again at the end. But I don't know if it's doable, and > if it is, we need to think what it might break. Yes. The code actually seems to have been written for something like SHOW FULL SLAVE STATUS, which would include the extra data. I will need to ask Monty to know more. > Then I assumed you decided not to print current GTIDs in SHOW SLAVE > STATUS on purpose, to avoid the mess like MySQL has -- which was also > surprising since we wouldn't have that insanity anyway. > I didn't ask how I could obtain them on slave, as I figured it was > supposed to be done via mysql.rpl_slave_state. But it didn't occur to > me that it was printed in SHOW ALL SLAVES STATUS... Yes. In any case, the user interface is very much not finalised yet. Sorry for the rough experience, but hopefully we can use your experience to come up with a good interface for the final release. >> Maybe the problem is that if I add a column in SHOW SLAVE STATUS, and MySQL >> adds another column in the same position, then scripts will break because the >> column count for one or the other will change? > > eh.. MySQL's show slave status currently has like +13 columns > comparing to ours, I really don't think that scripts relying on such > level of detail stand any chance to survive. My main worry is not if one has more than the other. The worry is if some column is in position 25 (say) in MySQL but 28 (say) in MariaDB. Is there even any easy way to pick out a column by name, rather than numerical position? > Good thing you mentioned it, I actually totally forgot to tell the > tale of my *very* first experience. Well, I started when I mentioned > Then I followed the instruction again, and... it didn't work. It > didn't crash, replication didn't fail, but when I switched master and > slave, the new slave (old master) just wouldn't replicate the last > line, (6,3). > > I couldn't understand it until I compared the flow and configuration > with the MTR test that you have (it does basically the same, and it > works), and only then I figured that I have to have > log-slave-updates=on. It wasn't clear to me why, I mean, I had the > transaction in quesiton in the binlog, and GTID was there too, > so?.. That's when I switched from AUTO to explicit values and started > experimenting with them. Well, in my example, first server 1 is the master, then I switch around so that server 2 is the master. For a slave to also be able to serve as master, it needs --log-slave-updates=1, right? On the other hand, the design specifically allows for --log-slave-updates=0. I did fix one or two possibly related bugs lately, so please try again and if you can reproduce the problem I will look into it. (I was a bit confused how you could have the transaction in question in the binlog on the server that was changed from slave to master, if it did not have --log-slave-updates=1 ...) > > But nothing happens... No errors in the slave, but it isn't moving, either: > This looks like a bug, your commands look correct. I could reproduce, I will > investigate and fix ASAP and let you know. This should be fixed now and pushed to the tree. > > Getting the wrong mutex usage warnings. I saw a discussion on IRC the other day, but don't know the current status -- whether it's supposed to be fixed, or not fixed yet, or is considered innocent... This also should be fixed in the tree now. - Kristian. (who now has to run from the computer and hopes the above makes sense...)

1 0

Re: [Maria-developers] Fwd: some question on bundled libedit in mysql
by Sergei Golubchik 25 Feb '13

25 Feb '13

Hi, AL13N! On Feb 25, AL13N wrote: > > > > In MariaDB we prefer to link with readline. We never link with bundled > > libedit at all. Either with system readline, if it's usable, or with > > system libedit, or with bundled readline. > > iinm, MariaDB is GPLv2 and libreadline (version 6) is GPLv3, so they don't > mix. but bundled libreadline (version 5) is GPLv2 and that one is > possible. That's why I said "with system readline, if it's usable". We only link with system readline if it's V5 or earlier. Regards, Sergei

1 0

[Maria-developers] Fwd: some question on bundled libedit in mysql
by xiaobing jiang 25 Feb '13

25 Feb '13

hi all: No reply from jonathan.perkin, so post to the maillist. can some one explain to me? Thanks! ---------- Forwarded message ---------- From: xiaobing jiang <s7v7nislands(a)gmail.com> Date: Fri, Feb 22, 2013 at 3:52 PM Subject: some question on bundled libedit in mysql To: jonathan.perkin(a)oracle.com hi jonathan: after read the README in source, I have some question about libedit. in recently, we find a bug. when using bundled libedit, we can't input the chinese using GBK. after debug, I find this code may be cause the bug. protected int terminal__putc(EditLine *el, Int c) { char buf[MB_LEN_MAX +1]; ssize_t i; mbstate_t state; memset(&state, 0, sizeof(mbstate_t)); if (c == (Int)MB_FILL_CHAR) return 0; i = ct_encode_char(buf, (size_t)MB_CUR_MAX, c, &state); // this should be: ct_encode_char(buf, (size_t)MB_LEN_MAX, c, &state); if (i <= 0) return (int)i; buf[i] = '\0'; return fputs(buf, el->el_outfile); } after change the code, I can fix the bug. and I find upstream also using MB_LEN_MAX. http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libedit/terminal.c protected int terminal__putc(EditLine *el, Int c) { char buf[MB_LEN_MAX +1]; ssize_t i; if (c == (Int)MB_FILL_CHAR) return 0; i = ct_encode_char(buf, (size_t)MB_LEN_MAX, c); if (i <= 0) return (int)i; buf[i] = '\0'; return fputs(buf, el->el_outfile); } so the merge from upstream may be wrong. And this bug may be the same http://bugs.mysql.com/bug.php?id=23097 and why mysql5.6 remove the bundled readline? I think readline is used more than libedit. and when using system readline, why perfer libedit to readline ? look at cmake/readline.cmake. FIND_SYSTEM_LIBEDIT(edit) -> FIND_SYSTEM_LIBEDIT(readline) thanks! xiaobing jiang

2 1

[Maria-developers] understanding commit and prepare APIs in MariaDB 5.5
by Zardosht Kasheff 21 Feb '13

21 Feb '13

Hello all, I am investigating what our storage engine, TokuDB, needs to do to achieve its best performance in MariaDB 5.5 with respect to preparing and committing transactions, and with respect to binary logging. As I understand, group commit has been added to the binary log in Maria 5.3, and is also in MariaDB 5.5. Additionally, new APIs prepare_ordered and commit_ordered are added. I am trying to understand how these fit together. >From reading http://kristiannielsen.livejournal.com/12408.html and comments in handler.h, here is what I THINK I know: - In MariaDB 5.5, we must fsync our recovery log on handlerton->prepare and handlerton->commit - If we want to ensure that the order we commit transactions is the same order that they appear committed in the binary log, then we should perform our commit in two phases, commit_ordered and commit, where commit_ordered is serialized under a global mutex that ensures the correct order - InnoDB must do this for hot backup to work Here is my big question: given TokuDB is not designed to work with InnoDB's hot backup solution, is there ANY reason why we should care about the commit order of our transactions? Are there any performance reasons related to group commit? >From everything I read, I think the answer is no, but I would appreciate confirmation. I think there are no performance gains to be made by implementing commit_ordered, and given we fsync on commit, there should be no issues with syncing our engine up with the binary log after a crash. Also, I assume the fsync after commit is necessary. Only in 10.0 will some new functionality be available that allows us to not fsync on commit. Is my understanding correct? Thanks -Zardosht

2 7

Re: [Maria-developers] Rev 3663: MDEV-3819 missing constraints for spatial column types
by Sergei Golubchik 20 Feb '13

20 Feb '13

Hi, Holyfoot! On Feb 20, holyfoot(a)askmonty.org wrote: > revno: 3663 > revision-id: holyfoot(a)askmonty.org-20130220195237-99gp9p29xemltqcm > parent: vvaintroub@pcbsd-20130219224652-t84v8wepdqvhxfwb > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-3819 > timestamp: Wed 2013-02-20 23:52:37 +0400 > message: > MDEV-3819 missing constraints for spatial column types. > Checks added to return and error when inappropriate > geometry type is stored in a field. > === modified file 'mysql-test/r/gis.result' > --- a/mysql-test/r/gis.result 2012-09-05 21:14:33 +0000 > +++ b/mysql-test/r/gis.result 2013-02-20 19:52:37 +0000 > @@ -1491,4 +1491,11 @@ SELECT 1 FROM g1 WHERE a >= ANY > (SELECT 1 FROM g1 WHERE a = geomfromtext('') OR a) ; > 1 > DROP TABLE g1; > +# > +# MDEV-3819 missing constraints for spatial column types > +# > +create table t1 (pt point); > +insert into t1 values(Geomfromtext('POLYGON((1 1, 2 2, 2 1, 1 1))')); > +ERROR HY000: Incorrect POLYGON value: 'POINT' I'm sorry, but you got it backwards :) The value above is POLYGON. The type is POINT. The error message should be ERROR HY000: Incorrect POINT value: 'POLYGON' Regards, Sergei

2 1

Re: [Maria-developers] Rev 3663: MDEV-3819 missing constraints for spatial column types
by Sergei Golubchik 20 Feb '13

20 Feb '13

Hi, Holyfoot! On Feb 20, holyfoot(a)askmonty.org wrote: > revno: 3663 > revision-id: holyfoot(a)askmonty.org-20130220172419-j5sad8p7t3191kii > parent: vvaintroub@pcbsd-20130219224652-t84v8wepdqvhxfwb > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-3819 > timestamp: Wed 2013-02-20 21:24:19 +0400 > message: > MDEV-3819 missing constraints for spatial column types. > Checks added to return and error when inappropriate > geometry type is stored in a field. > > === modified file 'sql/field.cc' > --- a/sql/field.cc 2013-01-28 12:36:05 +0000 > +++ b/sql/field.cc 2013-02-20 17:24:19 +0000 > @@ -7580,6 +7580,12 @@ int Field_geom::store(const char *from, > if (wkb_type < (uint32) Geometry::wkb_point || > wkb_type > (uint32) Geometry::wkb_last) > goto err; > + > + if (geom_type != Field::GEOM_GEOMETRY && > + geom_type != Field::GEOM_GEOMETRYCOLLECTION && > + (uint32) geom_type != wkb_type) > + goto err_wrong_type; why goto err_wrong_type? there's no common cleanup or any common code at all at the end, you can put it right here in the if(). > + > Field_blob::store_length(length); > if (table->copy_blobs || length <= MAX_FIELD_WIDTH) > { // Must make a copy > @@ -7595,6 +7601,12 @@ int Field_geom::store(const char *from, > my_message(ER_CANT_CREATE_GEOMETRY_OBJECT, > ER(ER_CANT_CREATE_GEOMETRY_OBJECT), MYF(0)); > return -1; > + > +err_wrong_type: > + bzero(ptr, Field_blob::pack_length()); > + my_message(ER_WRONG_GEOMETRY_TYPE, > + ER(ER_WRONG_GEOMETRY_TYPE), MYF(0)); > + return -1; > } > > #endif /*HAVE_SPATIAL*/ > > === modified file 'sql/share/errmsg-utf8.txt' > --- a/sql/share/errmsg-utf8.txt 2013-02-08 21:24:06 +0000 > +++ b/sql/share/errmsg-utf8.txt 2013-02-20 17:24:19 +0000 > @@ -6588,3 +6588,5 @@ ER_QUERY_EXCEEDED_ROWS_EXAMINED_LIMIT > ER_NO_SUCH_TABLE_IN_ENGINE 42S02 > eng "Table '%-.192s.%-.192s' doesn't exist in engine" > swe "Det finns ingen tabell som heter '%-.192s.%-.192s' i handlern" > +ER_WRONG_GEOMETRY_TYPE > + eng "Unsuitable geometry type for the field" Ouch, is that really needed? Can you use ER_WRONG_VALUE instead? Like Incorrect POINT value: 'POLYGON((1 1, 2 2, 2 1, 1 1))' Regards, Sergei

1 0

[Maria-developers] Determining to which project a patch should be sent.
by Christian Convey 20 Feb '13

20 Feb '13

Hi guys, As I work through the Coverity Scan results of MariaDB's trunk, I find myself having to research what some functions do, and the meanings of their parameters and return values. Rather than just recording what I learn on a piece of paper, I thought it might be nice to write Doxygen comments for some of those functions / structs / etc. That way, perhaps other persons trying to learn the code base could benefit. Here's the problem. Based on Monty's blog posting about MariaDB 10 vs. MySQL 5.6 ( http://blog.mariadb.org/mariadb-10-0-and-mysql-5-6/ ), it's not obvious to me which files / functions / etc. in MariaDB have been forked from MariaDB, vs. which ones will be replaced with the corresponding MySQL code when the next MySQL release comes out. This means that when I write Doxygen comments for a given file / function /struct / etc., I don't know whether to submit that comment patch to the MariaDB project or to the MySQL project. Does anyone know I I can figure that out? Thanks, Christian

3 5

[Maria-developers] binlog event of COMMIT statement
by nanyi607rao 20 Feb '13

20 Feb '13

I am little confused about binlog event of COMMIT statement, In most of time, a COMMIT statement corresponds a XID_event in the binlog, but sometimes is a Query_event. how that happened ? or can I do any operations to make a COMMIT statement correspond a Query_event certainly in the binlog. Thankes, Longhui 2013-02-20 nanyi607rao

2 1

[Maria-developers] my_hash_sort_bin
by MARK CALLAGHAN 20 Feb '13

20 Feb '13

Hello MariaDB, While working with Dmitry Lenev to understand why MySQL 5.6 was slower than 5.1 on one benchmark ( https://www.facebook.com/notes/mysql-at-facebook/my-mysql-is-faster-than-yo…) we realized that the hash function used in my_hash_sort_bin is lousy for this input: test.sbest1, test.sbtest2, ..., test.sbtest10. The problem is made worse when a small number of hash buckets is used because the hash function output doesn't do the right thing for the least significant bits so that all 10 inputs map to the same hash bucket. More details are at http://bugs.mysql.com/bug.php?id=66473 The InnoDB hash function is much better. Details for that and a test program are in the bug report. Does anyone remember why this hash function was chosen? strings/ctype-bin.c doesn't have any comments explaining why this hash function was selected. This is another peeve for me. Critical code like this should be explained if we expect anyone new to begin working on this code. I haven't done a formal test of the hash function, and http://burtleburtle.net/bob/hash/hashfaq.html would be a good reference for that. But I think it is very likely that my assertion that this hash function is lousy holds in general, rather than is limited to this input. And given the large number of machines that run MySQL, MariaDB and Percona Server (and the FB patch) around the world I suspect that we should figure this out. -- Mark Callaghan mdcallag(a)gmail.com

2 1

[Maria-developers] implementing multi range read at the storage engine in MariaDB 5.5
by Zardosht Kasheff 19 Feb '13

19 Feb '13

Hello all, Is there anything to implementing multi range read for storage engines? Or is there a default implementation that is likely good enough. Also, is there anything one can read to understand how the feature works and its benefits? Thanks -Zardosht

3 3

[Maria-developers] Using C++ features in MariaDB
by Christian Convey 19 Feb '13

19 Feb '13

I just read in Pachev's "MySQL Internals" book that C++ STL and exceptions aren't allowed in the MySQL code. Are those restrictions also in effect for MariaDB? If so, can anyone explain the reason for them? Thanks, Christian

2 8

Re: [Maria-developers] [Commits] Rev 3497: MDEV-537 Make multi-column non-top level subqueries to be executed via index (index/unique subquery) in file:///home/tsk/mprog/src/10.0-md537-merge/
by Sergei Petrunia 18 Feb '13

18 Feb '13

Hi Timour, With the below patch, conditions are unnecessarily wraped into two Item_func_trigcond objects. See EXPLAIN EXTENDED ouput in .result files below - one object is wrapping the other. Debugging shows that they are triggered by the same flag get_cond_guard(i). Two wrappings are just a few code lines away, so it should be totally feasible to get rid of double wrapping. On Thu, Feb 07, 2013 at 01:33:35PM +0000, timour(a)askmonty.org wrote: > At file:///home/tsk/mprog/src/10.0-md537-merge/ > > ------------------------------------------------------------ > revno: 3497 > revision-id: timour(a)askmonty.org-20130207133324-qrcjga4z9heaezb9 > parent: sergii(a)pisem.net-20130205094326-4yeuozh5iemlqley > fixes bug(s): https://mariadb.atlassian.net/browse/MDEV-537 > committer: timour(a)askmonty.org > branch nick: 10.0-md537-merge > timestamp: Thu 2013-02-07 15:33:24 +0200 > message: > MDEV-537 Make multi-column non-top level subqueries to be executed via index (index/unique subquery) > instead of single_select_engine > > This task changes the IN-EXISTS rewrite for multi-column subqueries > "(a, b) IN (select b, c ...)" to work in the same way as for > single-column subqueries "a IN (select b ...) with respect to the > injection of NULL-rejecting predicates. > > More specifically, the method > Item_in_subselect::create_row_in_to_exists_cond() > adds Item_is_not_null_test and Item_func_trig_cond only if the left > IN operand can be NULL. Not having these predicates when not necessary, > makes it possible to rewrite the subquery into a "unique_subquery" or > "index_subquery" when there is a suitable index on the only > subquery table. > === modified file 'mysql-test/r/derived_view.result' > --- a/mysql-test/r/derived_view.result 2013-01-15 18:07:46 +0000 > +++ b/mysql-test/r/derived_view.result 2013-02-07 13:33:24 +0000 > @@ -1468,14 +1468,14 @@ WHERE (t2.a ,t1.b) NOT IN (SELECT DISTIN > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 2 > 1 PRIMARY t2 ALL NULL NULL NULL NULL 2 Using where; Using join buffer (flat, BNL join) > -2 DEPENDENT SUBQUERY t eq_ref PRIMARY,c PRIMARY 4 func 1 Using where > +2 DEPENDENT SUBQUERY t unique_subquery PRIMARY,c PRIMARY 4 func 1 Using where > EXPLAIN > SELECT * FROM t1 , t2 > WHERE (t2.a ,t1.b) NOT IN (SELECT DISTINCT c,a FROM (SELECT * FROM t3) t); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 2 > 1 PRIMARY t2 ALL NULL NULL NULL NULL 2 Using where; Using join buffer (flat, BNL join) > -2 DEPENDENT SUBQUERY t3 eq_ref PRIMARY,c PRIMARY 4 func 1 Using where > +2 DEPENDENT SUBQUERY t3 unique_subquery PRIMARY,c PRIMARY 4 func 1 Using where > SELECT * FROM t1 , t2 > WHERE (t2.a ,t1.b) NOT IN (SELECT DISTINCT c,a FROM (SELECT * FROM t3) t); > b a > > === modified file 'mysql-test/r/subselect.result' > --- a/mysql-test/r/subselect.result 2012-12-17 00:49:19 +0000 > +++ b/mysql-test/r/subselect.result 2013-02-07 13:33:24 +0000 > @@ -2966,7 +2966,7 @@ id select_type table type possible_keys > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 > 2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 9 100.00 Using where > Warnings: > -Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<expr_cache><`test`.`t1`.`one`,`test`.`t1`.`two`>(<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`))) and trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`)))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`)))))) AS `test` from `test`.`t1` > +Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<expr_cache><`test`.`t1`.`one`,`test`.`t1`.`two`>(<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`)))) and trigcond(trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`))))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`)))))) AS `test` from `test`.`t1` > explain extended SELECT one,two from t1 where ROW(one,two) IN (SELECT one,two FROM t2 WHERE flag = 'N'); > id select_type table type possible_keys key key_len ref rows filtered Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 > @@ -2979,7 +2979,7 @@ id select_type table type possible_keys > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 > 2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 9 100.00 Using where > Warnings: > -Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<expr_cache><`test`.`t1`.`one`,`test`.`t1`.`two`>(<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`))) and trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`)))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`)))))) AS `test` from `test`.`t1` > +Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<expr_cache><`test`.`t1`.`one`,`test`.`t1`.`two`>(<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`)))) and trigcond(trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`))))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`)))))) AS `test` from `test`.`t1` > DROP TABLE t1,t2; > set optimizer_switch=@tmp11867_optimizer_switch; > CREATE TABLE t1 (a char(5), b char(5)); > > === modified file 'mysql-test/r/subselect3.result' > --- a/mysql-test/r/subselect3.result 2012-10-25 12:50:10 +0000 > +++ b/mysql-test/r/subselect3.result 2013-02-07 13:33:24 +0000 > @@ -252,7 +252,7 @@ id select_type table type possible_keys > 2 DEPENDENT SUBQUERY t1 index_subquery a a 5 func 2 100.00 Using where; Full scan on NULL key > Warnings: > Note 1276 Field or reference 'test.t2.oref' of SELECT #2 was resolved in SELECT #1 > -Note 1003 select `test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,`test`.`t2`.`oref` AS `oref`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on a checking NULL where ((`test`.`t1`.`c` = `test`.`t2`.`oref`) and trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`) or isnull(`test`.`t1`.`a`))) and trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`b`) or isnull(`test`.`t1`.`b`)))) having (trigcond(<is_not_null_test>(`test`.`t1`.`a`)) and trigcond(<is_not_null_test>(`test`.`t1`.`b`))))))) AS `Z` from `test`.`t2` > +Note 1003 select `test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,`test`.`t2`.`oref` AS `oref`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on a checking NULL where ((`test`.`t1`.`c` = `test`.`t2`.`oref`) and trigcond(trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`) or isnull(`test`.`t1`.`a`)))) and trigcond(trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`b`) or isnull(`test`.`t1`.`b`))))) having (trigcond(<is_not_null_test>(`test`.`t1`.`a`)) and trigcond(<is_not_null_test>(`test`.`t1`.`b`))))))) AS `Z` from `test`.`t2` > select a,b, oref, (a,b) in (select a,b from t1 where c=t2.oref) Z from t2; > a b oref Z > NULL 1 100 0 > @@ -269,7 +269,7 @@ id select_type table type possible_keys > 2 DEPENDENT SUBQUERY t4 ALL NULL NULL NULL NULL 100 100.00 Using where; Using join buffer (flat, BNL join) > Warnings: > Note 1276 Field or reference 'test.t2.oref' of SELECT #2 was resolved in SELECT #1 > -Note 1003 select `test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,`test`.`t2`.`oref` AS `oref`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(select `test`.`t1`.`a`,`test`.`t1`.`b` from `test`.`t1` join `test`.`t4` where ((`test`.`t1`.`c` = `test`.`t2`.`oref`) and trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`) or isnull(`test`.`t1`.`a`))) and trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`b`) or isnull(`test`.`t1`.`b`)))) having (trigcond(<is_not_null_test>(`test`.`t1`.`a`)) and trigcond(<is_not_null_test>(`test`.`t1`.`b`)))))) AS `Z` from `test`.`t2` > +Note 1003 select `test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,`test`.`t2`.`oref` AS `oref`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(select `test`.`t1`.`a`,`test`.`t1`.`b` from `test`.`t1` join `test`.`t4` where ((`test`.`t1`.`c` = `test`.`t2`.`oref`) and trigcond(trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`) or isnull(`test`.`t1`.`a`)))) and trigcond(trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`b`) or isnull(`test`.`t1`.`b`))))) having (trigcond(<is_not_null_test>(`test`.`t1`.`a`)) and trigcond(<is_not_null_test>(`test`.`t1`.`b`)))))) AS `Z` from `test`.`t2` > select a,b, oref, > (a,b) in (select a,b from t1,t4 where c=t2.oref) Z > from t2; > @@ -314,7 +314,7 @@ id select_type table type possible_keys > 2 DEPENDENT SUBQUERY t1 index_subquery idx idx 5 func 4 100.00 Using where; Full scan on NULL key > Warnings: > Note 1276 Field or reference 'test.t2.oref' of SELECT #2 was resolved in SELECT #1 > -Note 1003 select `test`.`t2`.`oref` AS `oref`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on idx checking NULL where ((`test`.`t1`.`oref` = `test`.`t2`.`oref`) and trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`ie1`) or isnull(`test`.`t1`.`ie1`))) and trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`ie2`) or isnull(`test`.`t1`.`ie2`)))) having (trigcond(<is_not_null_test>(`test`.`t1`.`ie1`)) and trigcond(<is_not_null_test>(`test`.`t1`.`ie2`))))))) AS `Z` from `test`.`t2` where ((`test`.`t2`.`b` = 10) and (`test`.`t2`.`a` = 10)) > +Note 1003 select `test`.`t2`.`oref` AS `oref`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on idx checking NULL where ((`test`.`t1`.`oref` = `test`.`t2`.`oref`) and trigcond(trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`ie1`) or isnull(`test`.`t1`.`ie1`)))) and trigcond(trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`ie2`) or isnull(`test`.`t1`.`ie2`))))) having (trigcond(<is_not_null_test>(`test`.`t1`.`ie1`)) and trigcond(<is_not_null_test>(`test`.`t1`.`ie2`))))))) AS `Z` from `test`.`t2` where ((`test`.`t2`.`b` = 10) and (`test`.`t2`.`a` = 10)) > drop table t1, t2; > create table t1 (oref char(4), grp int, ie int); > insert into t1 (oref, grp, ie) values > @@ -584,7 +584,7 @@ id select_type table type possible_keys > 2 DEPENDENT SUBQUERY t1 index_subquery idx idx 5 func 4 100.00 Using where; Full scan on NULL key > Warnings: > Note 1276 Field or reference 'test.t2.oref' of SELECT #2 was resolved in SELECT #1 > -Note 1003 select `test`.`t2`.`oref` AS `oref`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on idx checking NULL where ((`test`.`t1`.`oref` = `test`.`t2`.`oref`) and trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`ie1`) or isnull(`test`.`t1`.`ie1`))) and trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`ie2`) or isnull(`test`.`t1`.`ie2`)))) having (trigcond(<is_not_null_test>(`test`.`t1`.`ie1`)) and trigcond(<is_not_null_test>(`test`.`t1`.`ie2`))))))) AS `Z` from `test`.`t2` > +Note 1003 select `test`.`t2`.`oref` AS `oref`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on idx checking NULL where ((`test`.`t1`.`oref` = `test`.`t2`.`oref`) and trigcond(trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`ie1`) or isnull(`test`.`t1`.`ie1`)))) and trigcond(trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`ie2`) or isnull(`test`.`t1`.`ie2`))))) having (trigcond(<is_not_null_test>(`test`.`t1`.`ie1`)) and trigcond(<is_not_null_test>(`test`.`t1`.`ie2`))))))) AS `Z` from `test`.`t2` > drop table t1,t2; > create table t1 (oref char(4), grp int, ie int primary key); > insert into t1 (oref, grp, ie) values > > === modified file 'mysql-test/r/subselect3_jcl6.result' > --- a/mysql-test/r/subselect3_jcl6.result 2012-10-25 12:50:10 +0000 > +++ b/mysql-test/r/subselect3_jcl6.result 2013-02-07 13:33:24 +0000 > @@ -262,7 +262,7 @@ id select_type table type possible_keys > 2 DEPENDENT SUBQUERY t1 index_subquery a a 5 func 2 100.00 Using where; Full scan on NULL key > Warnings: > Note 1276 Field or reference 'test.t2.oref' of SELECT #2 was resolved in SELECT #1 > -Note 1003 select `test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,`test`.`t2`.`oref` AS `oref`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on a checking NULL where ((`test`.`t1`.`c` = `test`.`t2`.`oref`) and trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`) or isnull(`test`.`t1`.`a`))) and trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`b`) or isnull(`test`.`t1`.`b`)))) having (trigcond(<is_not_null_test>(`test`.`t1`.`a`)) and trigcond(<is_not_null_test>(`test`.`t1`.`b`))))))) AS `Z` from `test`.`t2` > +Note 1003 select `test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,`test`.`t2`.`oref` AS `oref`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on a checking NULL where ((`test`.`t1`.`c` = `test`.`t2`.`oref`) and trigcond(trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`) or isnull(`test`.`t1`.`a`)))) and trigcond(trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`b`) or isnull(`test`.`t1`.`b`))))) having (trigcond(<is_not_null_test>(`test`.`t1`.`a`)) and trigcond(<is_not_null_test>(`test`.`t1`.`b`))))))) AS `Z` from `test`.`t2` > select a,b, oref, (a,b) in (select a,b from t1 where c=t2.oref) Z from t2; > a b oref Z > NULL 1 100 0 > @@ -279,7 +279,7 @@ id select_type table type possible_keys > 2 DEPENDENT SUBQUERY t4 ALL NULL NULL NULL NULL 100 100.00 Using where; Using join buffer (flat, BNL join) > Warnings: > Note 1276 Field or reference 'test.t2.oref' of SELECT #2 was resolved in SELECT #1 > -Note 1003 select `test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,`test`.`t2`.`oref` AS `oref`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(select `test`.`t1`.`a`,`test`.`t1`.`b` from `test`.`t1` join `test`.`t4` where ((`test`.`t1`.`c` = `test`.`t2`.`oref`) and trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`) or isnull(`test`.`t1`.`a`))) and trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`b`) or isnull(`test`.`t1`.`b`)))) having (trigcond(<is_not_null_test>(`test`.`t1`.`a`)) and trigcond(<is_not_null_test>(`test`.`t1`.`b`)))))) AS `Z` from `test`.`t2` > +Note 1003 select `test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,`test`.`t2`.`oref` AS `oref`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(select `test`.`t1`.`a`,`test`.`t1`.`b` from `test`.`t1` join `test`.`t4` where ((`test`.`t1`.`c` = `test`.`t2`.`oref`) and trigcond(trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`a`) or isnull(`test`.`t1`.`a`)))) and trigcond(trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`b`) or isnull(`test`.`t1`.`b`))))) having (trigcond(<is_not_null_test>(`test`.`t1`.`a`)) and trigcond(<is_not_null_test>(`test`.`t1`.`b`)))))) AS `Z` from `test`.`t2` > select a,b, oref, > (a,b) in (select a,b from t1,t4 where c=t2.oref) Z > from t2; > @@ -324,7 +324,7 @@ id select_type table type possible_keys > 2 DEPENDENT SUBQUERY t1 index_subquery idx idx 5 func 4 100.00 Using where; Full scan on NULL key > Warnings: > Note 1276 Field or reference 'test.t2.oref' of SELECT #2 was resolved in SELECT #1 > -Note 1003 select `test`.`t2`.`oref` AS `oref`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on idx checking NULL where ((`test`.`t1`.`oref` = `test`.`t2`.`oref`) and trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`ie1`) or isnull(`test`.`t1`.`ie1`))) and trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`ie2`) or isnull(`test`.`t1`.`ie2`)))) having (trigcond(<is_not_null_test>(`test`.`t1`.`ie1`)) and trigcond(<is_not_null_test>(`test`.`t1`.`ie2`))))))) AS `Z` from `test`.`t2` where ((`test`.`t2`.`b` = 10) and (`test`.`t2`.`a` = 10)) > +Note 1003 select `test`.`t2`.`oref` AS `oref`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on idx checking NULL where ((`test`.`t1`.`oref` = `test`.`t2`.`oref`) and trigcond(trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`ie1`) or isnull(`test`.`t1`.`ie1`)))) and trigcond(trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`ie2`) or isnull(`test`.`t1`.`ie2`))))) having (trigcond(<is_not_null_test>(`test`.`t1`.`ie1`)) and trigcond(<is_not_null_test>(`test`.`t1`.`ie2`))))))) AS `Z` from `test`.`t2` where ((`test`.`t2`.`b` = 10) and (`test`.`t2`.`a` = 10)) > drop table t1, t2; > create table t1 (oref char(4), grp int, ie int); > insert into t1 (oref, grp, ie) values > @@ -594,7 +594,7 @@ id select_type table type possible_keys > 2 DEPENDENT SUBQUERY t1 index_subquery idx idx 5 func 4 100.00 Using where; Full scan on NULL key > Warnings: > Note 1276 Field or reference 'test.t2.oref' of SELECT #2 was resolved in SELECT #1 > -Note 1003 select `test`.`t2`.`oref` AS `oref`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on idx checking NULL where ((`test`.`t1`.`oref` = `test`.`t2`.`oref`) and trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`ie1`) or isnull(`test`.`t1`.`ie1`))) and trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`ie2`) or isnull(`test`.`t1`.`ie2`)))) having (trigcond(<is_not_null_test>(`test`.`t1`.`ie1`)) and trigcond(<is_not_null_test>(`test`.`t1`.`ie2`))))))) AS `Z` from `test`.`t2` > +Note 1003 select `test`.`t2`.`oref` AS `oref`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b`,<expr_cache><`test`.`t2`.`a`,`test`.`t2`.`b`,`test`.`t2`.`oref`>(<in_optimizer>((`test`.`t2`.`a`,`test`.`t2`.`b`),<exists>(<index_lookup>(<cache>(`test`.`t2`.`a`) in t1 on idx checking NULL where ((`test`.`t1`.`oref` = `test`.`t2`.`oref`) and trigcond(trigcond(((<cache>(`test`.`t2`.`a`) = `test`.`t1`.`ie1`) or isnull(`test`.`t1`.`ie1`)))) and trigcond(trigcond(((<cache>(`test`.`t2`.`b`) = `test`.`t1`.`ie2`) or isnull(`test`.`t1`.`ie2`))))) having (trigcond(<is_not_null_test>(`test`.`t1`.`ie1`)) and trigcond(<is_not_null_test>(`test`.`t1`.`ie2`))))))) AS `Z` from `test`.`t2` > drop table t1,t2; > create table t1 (oref char(4), grp int, ie int primary key); > insert into t1 (oref, grp, ie) values > > === modified file 'mysql-test/r/subselect4.result' > --- a/mysql-test/r/subselect4.result 2013-01-29 14:10:47 +0000 > +++ b/mysql-test/r/subselect4.result 2013-02-07 13:33:24 +0000 > @@ -303,7 +303,7 @@ EXPLAIN > SELECT * FROM t1 WHERE (NULL, 1) NOT IN (SELECT t2a.i, t2a.pk FROM t2a WHERE t2a.pk = t1.pk); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 4 Using where > -2 DEPENDENT SUBQUERY t2a eq_ref PRIMARY PRIMARY 8 const,test.t1.pk 1 Using where; Using index; Full scan on NULL key > +2 DEPENDENT SUBQUERY t2a unique_subquery PRIMARY PRIMARY 8 const,test.t1.pk 1 Using index; Using where; Full scan on NULL key > SELECT * FROM t1 WHERE (NULL, 1) NOT IN (SELECT t2a.i, t2a.pk FROM t2a WHERE t2a.pk = t1.pk); > pk i > 0 10 > @@ -335,7 +335,7 @@ EXPLAIN > SELECT * FROM t1 WHERE (NULL, 1) NOT IN (SELECT t2c.i, t2c.pk FROM t2c WHERE t2c.pk = t1.pk); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 4 Using where > -2 DEPENDENT SUBQUERY t2c ref it2c it2c 8 const,test.t1.pk 2 Using where; Using index; Full scan on NULL key > +2 DEPENDENT SUBQUERY t2c index_subquery it2c it2c 8 const,test.t1.pk 2 Using index; Using where; Full scan on NULL key > SELECT * FROM t1 WHERE (NULL, 1) NOT IN (SELECT t2c.i, t2c.pk FROM t2c WHERE t2c.pk = t1.pk); > pk i > 0 10 > > === modified file 'mysql-test/r/subselect_mat_cost.result' > --- a/mysql-test/r/subselect_mat_cost.result 2012-12-20 18:58:40 +0000 > +++ b/mysql-test/r/subselect_mat_cost.result 2013-02-07 13:33:24 +0000 > @@ -348,7 +348,7 @@ FROM City LEFT JOIN Country ON (Country > AND Language IN ('English','Spanish'); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY CountryLanguage range Language Language 30 NULL 72 Using index condition; Using where; Rowid-ordered scan > -2 DEPENDENT SUBQUERY City ref CityName CityName 35 func 1 Using index condition; Using where > +2 DEPENDENT SUBQUERY City ref CityName CityName 35 func 1 Using index condition > 2 DEPENDENT SUBQUERY Country eq_ref PRIMARY PRIMARY 3 world.City.Country 1 Using where; Using index > select count(*) > from CountryLanguage > > === modified file 'mysql-test/r/subselect_no_mat.result' > --- a/mysql-test/r/subselect_no_mat.result 2012-12-17 00:49:19 +0000 > +++ b/mysql-test/r/subselect_no_mat.result 2013-02-07 13:33:24 +0000 > @@ -2973,7 +2973,7 @@ id select_type table type possible_keys > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 > 2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 9 100.00 Using where > Warnings: > -Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<expr_cache><`test`.`t1`.`one`,`test`.`t1`.`two`>(<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`))) and trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`)))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`)))))) AS `test` from `test`.`t1` > +Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<expr_cache><`test`.`t1`.`one`,`test`.`t1`.`two`>(<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`)))) and trigcond(trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`))))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`)))))) AS `test` from `test`.`t1` > explain extended SELECT one,two from t1 where ROW(one,two) IN (SELECT one,two FROM t2 WHERE flag = 'N'); > id select_type table type possible_keys key key_len ref rows filtered Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 > @@ -2985,7 +2985,7 @@ id select_type table type possible_keys > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 > 2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 9 100.00 Using where > Warnings: > -Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<expr_cache><`test`.`t1`.`one`,`test`.`t1`.`two`>(<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`))) and trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`)))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`)))))) AS `test` from `test`.`t1` > +Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<expr_cache><`test`.`t1`.`one`,`test`.`t1`.`two`>(<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`)))) and trigcond(trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`))))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`)))))) AS `test` from `test`.`t1` > DROP TABLE t1,t2; > set optimizer_switch=@tmp11867_optimizer_switch; > CREATE TABLE t1 (a char(5), b char(5)); > > === modified file 'mysql-test/r/subselect_no_opts.result' > --- a/mysql-test/r/subselect_no_opts.result 2012-12-17 00:49:19 +0000 > +++ b/mysql-test/r/subselect_no_opts.result 2013-02-07 13:33:24 +0000 > @@ -2969,7 +2969,7 @@ id select_type table type possible_keys > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 > 2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 9 100.00 Using where > Warnings: > -Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`))) and trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`)))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`))))) AS `test` from `test`.`t1` > +Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`)))) and trigcond(trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`))))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`))))) AS `test` from `test`.`t1` > explain extended SELECT one,two from t1 where ROW(one,two) IN (SELECT one,two FROM t2 WHERE flag = 'N'); > id select_type table type possible_keys key key_len ref rows filtered Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 Using where > @@ -2981,7 +2981,7 @@ id select_type table type possible_keys > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 > 2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 9 100.00 Using where > Warnings: > -Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`))) and trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`)))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`))))) AS `test` from `test`.`t1` > +Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`)))) and trigcond(trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`))))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`))))) AS `test` from `test`.`t1` > DROP TABLE t1,t2; > set optimizer_switch=@tmp11867_optimizer_switch; > CREATE TABLE t1 (a char(5), b char(5)); > > === modified file 'mysql-test/r/subselect_no_scache.result' > --- a/mysql-test/r/subselect_no_scache.result 2012-12-17 00:49:19 +0000 > +++ b/mysql-test/r/subselect_no_scache.result 2013-02-07 13:33:24 +0000 > @@ -2972,7 +2972,7 @@ id select_type table type possible_keys > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 > 2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 9 100.00 Using where > Warnings: > -Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`))) and trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`)))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`))))) AS `test` from `test`.`t1` > +Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`)))) and trigcond(trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`))))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`))))) AS `test` from `test`.`t1` > explain extended SELECT one,two from t1 where ROW(one,two) IN (SELECT one,two FROM t2 WHERE flag = 'N'); > id select_type table type possible_keys key key_len ref rows filtered Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 > @@ -2985,7 +2985,7 @@ id select_type table type possible_keys > 1 PRIMARY t1 ALL NULL NULL NULL NULL 8 100.00 > 2 DEPENDENT SUBQUERY t2 ALL NULL NULL NULL NULL 9 100.00 Using where > Warnings: > -Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`))) and trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`)))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`))))) AS `test` from `test`.`t1` > +Note 1003 select `test`.`t1`.`one` AS `one`,`test`.`t1`.`two` AS `two`,<in_optimizer>((`test`.`t1`.`one`,`test`.`t1`.`two`),<exists>(select `test`.`t2`.`one`,`test`.`t2`.`two` from `test`.`t2` where ((`test`.`t2`.`flag` = '0') and trigcond(trigcond(((<cache>(`test`.`t1`.`one`) = `test`.`t2`.`one`) or isnull(`test`.`t2`.`one`)))) and trigcond(trigcond(((<cache>(`test`.`t1`.`two`) = `test`.`t2`.`two`) or isnull(`test`.`t2`.`two`))))) having (trigcond(<is_not_null_test>(`test`.`t2`.`one`)) and trigcond(<is_not_null_test>(`test`.`t2`.`two`))))) AS `test` from `test`.`t1` > DROP TABLE t1,t2; > set optimizer_switch=@tmp11867_optimizer_switch; > CREATE TABLE t1 (a char(5), b char(5)); > > === modified file 'sql/item_subselect.cc' > --- a/sql/item_subselect.cc 2013-01-29 14:10:47 +0000 > +++ b/sql/item_subselect.cc 2013-02-07 13:33:24 +0000 > @@ -2306,7 +2306,7 @@ Item_in_subselect::create_row_in_to_exis > ref_pointer_array+i, > (char *)"<no matter>", > (char *)"<list ref>")); > - if (!abort_on_null) > + if (!abort_on_null && select_lex->ref_pointer_array[i]->maybe_null) > { > Item *having_col_item= > new Item_is_not_null_test(this, > @@ -2325,10 +2325,6 @@ Item_in_subselect::create_row_in_to_exis > (char *)"<no matter>", > (char *)"<list ref>")); > item= new Item_cond_or(item, item_isnull); > - /* > - TODO: why we create the above for cases where the right part > - cant be NULL? > - */ > if (left_expr->element_index(i)->maybe_null) > { > if (!(item= new Item_func_trig_cond(item, get_cond_guard(i)))) > @@ -2339,6 +2335,11 @@ Item_in_subselect::create_row_in_to_exis > } > *having_item= and_items(*having_item, having_col_item); > } > + if (!abort_on_null && left_expr->element_index(i)->maybe_null) > + { > + if (!(item= new Item_func_trig_cond(item, get_cond_guard(i)))) > + DBUG_RETURN(true); > + } > *where_item= and_items(*where_item, item); > } > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Idea for parallel replication with statement-based binlogging
by Kristian Nielsen 17 Feb '13

17 Feb '13

Hi Xiaobin, I hope you are enjoying the Spring Festival! All the best wishes for you and your family for the coming year. I had an idea for your parallel replication work, inspired by my work on in-order commit. I wanted to hear your opinion on it, if you are interested. The idea is to make your patch work with statement-based replication. For row-based InnoDB events, you hash all the unique keys to check for conflicts, and if no conflict you can run transactions in parallel even against the same table. For MyISAM, I get deadlock issues in my in-order-commit patch, if I try to replicate in parallel against the same table. So I hash just the table names, so that two transactions on the same table always conflict. Then I can still replicate in parallel events against different tables. Now suppose that for statement-based events, we obtain the names of all tables used in the queries. We could add a new event to the binlog with the table names, or perhaps parse the statement on the slave, if that is not too expensive. Then we could hash just the names to check for conflicts, just like I did for MyISAM with row-based events. It seems to me that we could then safely replicate statement-based events in parallel on the slave if they are to different tables. This would be great for people that do not want to use row-based replication, it would surely be better than the MySQL 5.6 multi-threaded slave which can only do queries against different databases in parallel. What do you think? - Kristian.

2 1

[Maria-developers] implementing index condition pushdown in MariaDB 5.5
by Zardosht Kasheff 16 Feb '13

16 Feb '13

Hello, Is there any documentation for what a storage engine needs to do to implement index condition pushdown in MariaDB 5.5? I see some related things, such as handler_index_cond_check, HA_DO_INDEX_COND_PUSHDOWN, and idx_cond_push, but I don't understand how these all interact with each other. Thanks -Zardosht

2 6

[Maria-developers] Help with running a transaction inside the server
by Kristian Nielsen 14 Feb '13

14 Feb '13

Hi Serg, Can you give me a tip for doing the following (for MDEV-26, global transaction ID): I need to run a transaction inside the server. This is in the slave I/O thread, when it connects to the master it receives the current replication position as a GTID, and I want to store it in the mysql.rpl_slave_state so that next reconnect of the slave can use the GTID automatically. So I basically need to run the equivalent of something like this: BEGIN; INSERT INTO mysql.rpl_slave_state VALUES (...); DELETE FROM mysql.rpl_slave_state WHERE id=...; COMMIT; I think I already have everything except the "BEGIN" part (if you want, you can check the code in rpl_slave_state::record_gtid() in my branch lp:~maria-captains/maria/10.0-mdev26/ on Launchpad). So how do I do the equivalent of BEGIN inside the server code? Can you think of any existing code I could use as example? - Kristian.

3 7

[Maria-developers] Fwd: debugging how query plans are made
by Zardosht Kasheff 12 Feb '13

12 Feb '13

Is there something similar in MariaDB? ---------- Forwarded message ---------- From: Øystein Grøvlen <oystein.grovlen(a)oracle.com> Date: Tue, Feb 12, 2013 at 9:11 AM Subject: Re: debugging how query plans are made To: Zardosht Kasheff <zardosht(a)gmail.com> Cc: "internals(a)lists.mysql.com" <internals(a)lists.mysql.com> On 02/12/13 02:54 PM, Zardosht Kasheff wrote: > > Hello all, > > In MySQL and/or MariaDB, are there any hooks or methods one can use to > learn why a query plan is chosen over alternatives? We sometimes > notice that a different query plan is chosen for us than InnoDB (which > may be fine, given we are a different storage engine), and we would > like to better understand why. Is there any debug parameter that gets > MySQL to output decisions made in the optimizer? MySQL 5.6 has optimizer tracing which outputs the decisions made by the optimizer. See http://dev.mysql.com/doc/internals/en/optimizer-tracing.html Hope this helps, -- Øystein

2 1

Re: [Maria-developers] Files under /etc/my.cnf.d/ read by default
by Sergei Golubchik 12 Feb '13

12 Feb '13

Hi, Honza! On Feb 05, Honza Horak wrote: > Hi guys, > > while we were packaging MariaDB into Fedora (finally) we used the > upstream's cnf files structure, which means /etc/my.cnf, that includes > all files from directory /etc/my.cnf.d/* using statement !includedir. > > That works fine when user installs (not updates) the MariaDB packages > without having them installed before. The problem is if user has changed > /etc/my.cnf before updating -- then /etc/my.cnf doesn't get updated > (!includedir won't be added) and we end with files under /etc/my.cnf.d/* > but nothing what includes them. That is quite confusing for users that > may change something in /etc/my.cnf.d/client.cnf but nothing happens. Agree. Especially as we've started to use /etc/my.cnf.d/ now - e.g. CassandraSE comes in a separate rpm, that (besides the plugin itself) drops a file into /etc/my.cnf.d/ to enable the engine in the server. I expect that more separately-packed plugins will start doing the same. > Since files under /etc/my.cnf.d/* are empty by default it shouldn't do > any harm if all files corresponding with /etc/my.cnf.d/*.cnf template > get loaded even without !includedir in /etc/my.cnf. Actually I believe > that is what users expect to happen when they see files under > /etc/my.cnf.d.. > > So, that's basically what I'm proposing -- read files under > /etc/my.cnf.d/* by default, without a need to use !includedir. What do > you think? What if we add !includedir to the /etc/my.cnf from the post-inst script? Something like grep -q '!includedir /etc/my.cnf.d' /etc/my.cnf || \ (echo; echo '!includedir /etc/my.cnf.d') >> /etc/my.cnf Would that be enough? Regards, Sergei

2 1

[Maria-developers] No Doxygen?
by Christian Convey 11 Feb '13

11 Feb '13

I was kind of surprised to not find anything in MariaDB's build system for using Doxygen to create HTML docs. Have people decided this isn't a worthwhile thing to do? Or has just no one gotten around to it yet? - Christian

3 4

Re: [Maria-developers] Question
by Michael Widenius 10 Feb '13

10 Feb '13

Hi! First, the MariaDB developers list is the right place to ask questions like this. >>>>> "Aamir" == Aamir Abbas <Abbas> writes: Aamir> Hi Monty, Aamir> I have researched this question about MySQL without luck and if you cannot answer it then nobody can. On some MySQL servers the version number returns a "-log" suffixed to the version number. For example, 5.0.45-log. What does the "-log" mean or signify? This is not a standard version number as far as I can tell. Should it not simply return 5.0.45? I have posted the question to a couple of forums but nobody's able to answer. I have just updated the documentation at https://kb.askmonty.org/en/version/ to answer this questions in detail. Here is a copy of the new documentation: The VERSION() string may have one or more of the following suffixes: Suffix Description -embedded The server is an embedded server (libmysqld). -log General logging, slow logging or binary (replication) logging is enabled. -debug The server is compiled for debugging. -valgrind The server is compiled to be instrumented with valgrind. Regards, Monty

1 0

[Maria-developers] Coverity scan results
by Christian Convey 07 Feb '13

07 Feb '13

Hey guys, For those of you who missed it, I volunteered to try gettin MariaDB scanned as part of the Coverity Scan service. I just got the trunk scanned for the first time, and a lot of potential-problem reports came up. I did a spot check on one of them, and the report looks like a true-positive. Coverity Scan reports: 178 high-impact problems 1020 medium-impact problems 47 low-impact problems. Any suggestions for how to get these looked at by the appropriate developers? It seems crazy for me to enter 1000+ bug reports into the bug tracker. I believe I have the power to create new user accounts in Coverity Scan's MariaDB project, for any MariaDB developer who wants to see the results. Thanks, Christian

3 12

Re: [Maria-developers] Fwd: [Commits] Rev 3622: Fix for bug MDEV-765 (LP:825075) in file:///home/tsk/mprog/src/5.3-md765/
by Sergei Petrunia 07 Feb '13

07 Feb '13

Hi Timour, On Mon, Feb 04, 2013 at 05:39:02PM +0200, Timour Katchaounov wrote: > > Sergey, > > Could you please review the following patch. The fix implements > your suggestion from your previous review: > https://lists.launchpad.net/maria-developers/msg04597.html > Ok to push. > Timour > > ------------------------------------------------------------ > revno: 3622 > revision-id: timour(a)askmonty.org-20130204153548-njv08hcdskv6ttjk > parent: sergii(a)pisem.net-20130128081223-mp9rsd3t9soz8lly > fixes bug: https://mariadb.atlassian.net/browse/MDEV-765 > committer: timour(a)askmonty.org > branch nick: 5.3-md765 > timestamp: Mon 2013-02-04 17:35:48 +0200 > message: > Fix for bug MDEV-765 (LP:825075) > > Analys: > The cause for the wrong result was that the optimizer > incorrectly chose min/max loose scan when it is not > applicable. The applicability test missed the case when > a condition on the MIN/MAX argument was OR-ed with a > condition on some other field. In this case, the MIN/MAX > condition cannot be used for loose scan. > > Solution: > Extend the test check_group_min_max_predicates() to check > that the WHERE clause is of the form: "cond1 AND cond2" > where > cond1 - does not use min_max_column at all. > cond2 - is an AND/OR tree with leaves in form "min_max_column $CMP$ const" > or $CMP$ is one of the functions between, is [not] null > > > === modified file 'mysql-test/r/group_min_max.result' > --- a/mysql-test/r/group_min_max.result 2013-01-28 08:12:23 +0000 > +++ b/mysql-test/r/group_min_max.result 2013-02-04 15:35:48 +0000 > @@ -3186,3 +3186,61 @@ a b > 109 19 > drop table t1; > End of 5.1 tests > +# > +# MDEV-765: LP:825075 - Wrong result with GROUP BY + multipart key + MIN/MAX loose scan > +# > +CREATE TABLE t1 (a varchar(1), b varchar(1), KEY (b,a)); > +INSERT INTO t1 VALUES > +('0',NULL),('9',NULL),('8','c'),('4','d'),('7','d'),(NULL,'f'), > +('7','f'),('8','g'),(NULL,'j'); > +explain > +SELECT max(a) , b FROM t1 WHERE a IS NULL OR b = 'z' GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index b b 8 NULL 9 Using where; Using index > +SELECT max(a) , b FROM t1 WHERE a IS NULL OR b = 'z' GROUP BY b; > +max(a) b > +NULL f > +NULL j > +explain > +SELECT b, min(a) FROM t1 WHERE a = '7' OR b = 'z' GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index b b 8 NULL 9 Using where; Using index > +SELECT b, min(a) FROM t1 WHERE a = '7' OR b = 'z' GROUP BY b; > +b min(a) > +d 7 > +f 7 > +explain > +SELECT b, min(a) FROM t1 WHERE (a = b OR b = 'd' OR b is NULL) GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index b b 8 NULL 9 Using where; Using index > +SELECT b, min(a) FROM t1 WHERE (a = b OR b = 'd' OR b is NULL) GROUP BY b; > +b min(a) > +NULL 0 > +d 4 > +explain > +SELECT b, min(a) FROM t1 WHERE a > ('0' = b) AND b = 'z' GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ref b b 4 const 1 Using where; Using index > +SELECT b, min(a) FROM t1 WHERE a > ('0' = b) AND b = 'z' GROUP BY b; > +b min(a) > +explain > +SELECT b, min(a) FROM t1 WHERE a > '0' AND (b < (a = '7')) GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index NULL b 8 NULL 9 Using where; Using index > +SELECT b, min(a) FROM t1 WHERE a > '0' AND (b < (a = '7')) GROUP BY b; > +b min(a) > +d 7 > +f 7 > +explain > +SELECT b, min(a) FROM t1 WHERE (a > '0' AND (a > '1' OR b = 'd')) GROUP BY b; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index b b 8 NULL 9 Using where; Using index > +SELECT b, min(a) FROM t1 WHERE (a > '0' AND (a > '1' OR b = 'd')) GROUP BY b; > +b min(a) > +NULL 9 > +c 8 > +d 4 > +f 7 > +g 8 > +drop table t1; > +End of 5.3 tests > > === modified file 'mysql-test/t/group_min_max.test' > --- a/mysql-test/t/group_min_max.test 2013-01-28 08:12:23 +0000 > +++ b/mysql-test/t/group_min_max.test 2013-02-04 15:35:48 +0000 > @@ -1203,3 +1203,40 @@ WHERE EXISTS ( SELECT DISTINCT b FROM t1 > drop table t1; > > --echo End of 5.1 tests > + > +--echo # > +--echo # MDEV-765: LP:825075 - Wrong result with GROUP BY + multipart key + MIN/MAX loose scan > +--echo # > + > +CREATE TABLE t1 (a varchar(1), b varchar(1), KEY (b,a)); > +INSERT INTO t1 VALUES > +('0',NULL),('9',NULL),('8','c'),('4','d'),('7','d'),(NULL,'f'), > +('7','f'),('8','g'),(NULL,'j'); > + > +explain > +SELECT max(a) , b FROM t1 WHERE a IS NULL OR b = 'z' GROUP BY b; > +SELECT max(a) , b FROM t1 WHERE a IS NULL OR b = 'z' GROUP BY b; > + > +explain > +SELECT b, min(a) FROM t1 WHERE a = '7' OR b = 'z' GROUP BY b; > +SELECT b, min(a) FROM t1 WHERE a = '7' OR b = 'z' GROUP BY b; > + > +explain > +SELECT b, min(a) FROM t1 WHERE (a = b OR b = 'd' OR b is NULL) GROUP BY b; > +SELECT b, min(a) FROM t1 WHERE (a = b OR b = 'd' OR b is NULL) GROUP BY b; > + > +explain > +SELECT b, min(a) FROM t1 WHERE a > ('0' = b) AND b = 'z' GROUP BY b; > +SELECT b, min(a) FROM t1 WHERE a > ('0' = b) AND b = 'z' GROUP BY b; > + > +explain > +SELECT b, min(a) FROM t1 WHERE a > '0' AND (b < (a = '7')) GROUP BY b; > +SELECT b, min(a) FROM t1 WHERE a > '0' AND (b < (a = '7')) GROUP BY b; > + > +explain > +SELECT b, min(a) FROM t1 WHERE (a > '0' AND (a > '1' OR b = 'd')) GROUP BY b; > +SELECT b, min(a) FROM t1 WHERE (a > '0' AND (a > '1' OR b = 'd')) GROUP BY b; > + > +drop table t1; > + > +--echo End of 5.3 tests > > === modified file 'sql/opt_range.cc' > --- a/sql/opt_range.cc 2013-01-28 08:12:23 +0000 > +++ b/sql/opt_range.cc 2013-02-04 15:35:48 +0000 > @@ -11485,7 +11485,8 @@ static bool get_constant_key_infix(KEY * > KEY_PART_INFO **first_non_infix_part); > static bool > check_group_min_max_predicates(Item *cond, Item_field *min_max_arg_item, > - Field::imagetype image_type); > + Field::imagetype image_type, > + bool *has_min_max_fld, bool *has_other_fld); > > static void > cost_group_min_max(TABLE* table, KEY *index_info, uint used_key_parts, > @@ -12005,10 +12006,12 @@ get_best_group_min_max(PARAM *param, SEL > DBUG_RETURN(NULL); > > /* Check (SA3) for the where clause. */ > + bool has_min_max_fld= false, has_other_fld= false; > if (join->conds && min_max_arg_item && > !check_group_min_max_predicates(join->conds, min_max_arg_item, > (index_info->flags & HA_SPATIAL) ? > - Field::itMBR : Field::itRAW)) > + Field::itMBR : Field::itRAW, > + &has_min_max_fld, &has_other_fld)) > DBUG_RETURN(NULL); > > /* The query passes all tests, so construct a new TRP object. */ > @@ -12043,16 +12046,24 @@ get_best_group_min_max(PARAM *param, SEL > > SYNOPSIS > check_group_min_max_predicates() > - cond tree (or subtree) describing all or part of the WHERE > - clause being analyzed > - min_max_arg_item the field referenced by the MIN/MAX function(s) > - min_max_arg_part the keypart of the MIN/MAX argument if any > + cond [in] the expression tree being analyzed > + min_max_arg [in] the field referenced by the MIN/MAX function(s) > + image_type [in] > + has_min_max_arg [out] true if the subtree being analyzed references min_max_arg > + has_other_arg [out] true if the subtree being analyzed references a column > + other min_max_arg > > DESCRIPTION > The function walks recursively over the cond tree representing a WHERE > clause, and checks condition (SA3) - if a field is referenced by a MIN/MAX > aggregate function, it is referenced only by one of the following > - predicates: {=, !=, <, <=, >, >=, between, is null, is not null}. > + predicates $FUNC$: > + {=, !=, <, <=, >, >=, between, is [not] null, multiple equal}. > + In addition the function checks that the WHERE condition is equivalent to > + "cond1 AND cond2" where : > + cond1 - does not use min_max_column at all. > + cond2 - is an AND/OR tree with leaves in form > + "$FUNC$(min_max_column[, const])". > > RETURN > TRUE if cond passes the test > @@ -12061,7 +12072,8 @@ get_best_group_min_max(PARAM *param, SEL > > static bool > check_group_min_max_predicates(Item *cond, Item_field *min_max_arg_item, > - Field::imagetype image_type) > + Field::imagetype image_type, > + bool *has_min_max_arg, bool *has_other_arg) > { > DBUG_ENTER("check_group_min_max_predicates"); > DBUG_ASSERT(cond && min_max_arg_item); > @@ -12073,12 +12085,24 @@ check_group_min_max_predicates(Item *con > DBUG_PRINT("info", ("Analyzing: %s", ((Item_func*) cond)->func_name())); > List_iterator_fast<Item> li(*((Item_cond*) cond)->argument_list()); > Item *and_or_arg; > + Item_func::Functype func_type= ((Item_cond*) cond)->functype(); > + bool has_min_max= false, has_other= false; > while ((and_or_arg= li++)) > { > - if (!check_group_min_max_predicates(and_or_arg, min_max_arg_item, > - image_type)) > + /* > + The WHERE clause doesn't pass the condition if: > + (1) any subtree doesn't pass the condition or > + (2) the subtree passes the test, but it is an OR and it references both > + the min/max argument and other columns. > + */ > + if (!check_group_min_max_predicates(and_or_arg, min_max_arg_item, //1 > + image_type, > + &has_min_max, &has_other) || > + (func_type == Item_func::COND_OR_FUNC && has_min_max && has_other))//2 > DBUG_RETURN(FALSE); > } > + *has_min_max_arg= has_min_max || *has_min_max_arg; > + *has_other_arg= has_other || *has_other_arg; > DBUG_RETURN(TRUE); > } > > @@ -12112,6 +12136,10 @@ check_group_min_max_predicates(Item *con > if (cond_type == Item::FIELD_ITEM) > { > DBUG_PRINT("info", ("Analyzing: %s", cond->full_name())); > + if (min_max_arg_item->eq((Item_field*)cond, 1)) > + *has_min_max_arg= true; > + else > + *has_other_arg= true; > DBUG_RETURN(TRUE); > } > > @@ -12120,9 +12148,33 @@ check_group_min_max_predicates(Item *con > > /* Test if cond references only group-by or non-group fields. */ > Item_func *pred= (Item_func*) cond; > + Item_func::Functype pred_type= pred->functype(); > + DBUG_PRINT("info", ("Analyzing: %s", pred->func_name())); > + if (pred_type == Item_func::MULT_EQUAL_FUNC) > + { > + /* > + Check that each field in a multiple equality is either a constant or > + it is a reference to the min/max argument, or it doesn't contain the > + min/max argument at all. > + */ > + Item_equal_fields_iterator eq_it(*((Item_equal*)pred)); > + Item *eq_item; > + bool has_min_max= false, has_other= false; > + while ((eq_item= eq_it++)) > + { > + if (min_max_arg_item->eq(eq_item->real_item(), 1)) > + has_min_max= true; > + else > + has_other= true; > + } > + *has_min_max_arg= has_min_max || *has_min_max_arg; > + *has_other_arg= has_other || *has_other_arg; > + DBUG_RETURN(!(has_min_max && has_other)); > + } > + > Item **arguments= pred->arguments(); > Item *cur_arg; > - DBUG_PRINT("info", ("Analyzing: %s", pred->func_name())); > + bool has_min_max= false, has_other= false; > for (uint arg_idx= 0; arg_idx < pred->argument_count (); arg_idx++) > { > cur_arg= arguments[arg_idx]->real_item(); > @@ -12131,11 +12183,11 @@ check_group_min_max_predicates(Item *con > { > if (min_max_arg_item->eq(cur_arg, 1)) > { > - /* > - If pred references the MIN/MAX argument, check whether pred is a range > - condition that compares the MIN/MAX argument with a constant. > - */ > - Item_func::Functype pred_type= pred->functype(); > + has_min_max= true; > + /* > + If pred references the MIN/MAX argument, check whether pred is a range > + condition that compares the MIN/MAX argument with a constant. > + */ > if (pred_type != Item_func::EQUAL_FUNC && > pred_type != Item_func::LT_FUNC && > pred_type != Item_func::LE_FUNC && > @@ -12175,11 +12227,13 @@ check_group_min_max_predicates(Item *con > min_max_arg_item->field->cmp_type() != args[1]->result_type()))) > DBUG_RETURN(FALSE); > } > + else > + has_other= true; > } > else if (cur_arg->type() == Item::FUNC_ITEM) > { > - if (!check_group_min_max_predicates(cur_arg, min_max_arg_item, > - image_type)) > + if (!check_group_min_max_predicates(cur_arg, min_max_arg_item, image_type, > + &has_min_max, &has_other)) > DBUG_RETURN(FALSE); > } > else if (cur_arg->const_item()) > @@ -12192,7 +12246,11 @@ check_group_min_max_predicates(Item *con > } > else > DBUG_RETURN(FALSE); > + if(has_min_max && has_other) > + DBUG_RETURN(FALSE); > } > + *has_min_max_arg= has_min_max || *has_min_max_arg; > + *has_other_arg= has_other || *has_other_arg; > > DBUG_RETURN(TRUE); > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Igor please review: EXPLAIN for DELETE
by Sergei Petrunia 07 Feb '13

07 Feb '13

Hello Igor, Hello Igor, Could you please review the following two patches: http://lists.askmonty.org/pipermail/commits/2012-December/003944.html http://lists.askmonty.org/pipermail/commits/2012-December/003945.html I think they implement the ideas about MDEV-3798 we've discussed earlier. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] mariadb-server not installable on Debian Testing AMD64
by Kristian Nielsen 06 Feb '13

06 Feb '13

Thomas Hackert <thackert(a)nexgo.de> writes: > I am not sure, how (and if) I can upload to f.a.o., so I will attach it > to this mail. Sorry for the inconvenience ... :( No inconvenience at all, mail is perfect. And indeed, the strace was illuminating. I think the issue is right here: [pid 32673] bind(14, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRNOTAVAIL (Cannot assign requested address) The server fails to start, because it cannot setup a socket to listen on the loopback interface on 127.0.0.1 The manpage for bind(2) says: EADDRNOTAVAIL A nonexistent interface was requested or the requested address was not local. I get the same error on my machine if I try to bind a socket to a non-existing interface. You can try this netcat command to do the same bind, it will probably fail in the same way: nc -l -s 127.0.0.1 -p 3306 So the problem most likely is that your system has a non-standard networking setup, with the normal loopback address 127.0.0.1 not configured. The default configuration file has this line: bind-address = 127.0.0.1 This will fail if the address 127.0.0.1 is not available on the system (normally it is). So I think you can edit /etc/mysql/my.cnf and change this line to use a valid IP address, or replace it with skip-networking to not use any IP address. Then try starting the server and see if it works. If it does, `apt-get install -f` might fix the installation. Can you confirm that your system has no 127.0.0.1 address? (It is somewhat annoying that the install fails in this case, but it does seem a rather rare setup). The second question is why you did not get any error log output with messages about this problem. I suppose this might be because you had no syslog installed. Elena gave you suggestions for how to resolve/investigate this further. But let me know if you have more problems with this. Thanks for your persistence in helping me get to the bottom of this! - Kristian.

2 6

Re: [Maria-developers] Fwd: [Commits] Rev 3622: Fix for MDEV-4140 in file:///home/tsk/mprog/src/5.3-md4140/
by Igor Babaev 06 Feb '13

06 Feb '13

Timour, It's ok to push this patch as a fix. Yet remember that this fix leads us to a performance regression 5.2 >> 5.3 when the range expression contains a constant [single-row] subquery. This is due to the changes when and how we build range trees in 5.3. When we have a range expression with constant expensive subquery we just bail out and do not include this range into the range condition at all. The worst thing about this is that we bail out even for dynamic ranges belonging to te range trees that are build at the execution time when subquery evaluation is allowed. Please open a task to fix all these regressions (it's quite possible). Regards, Igor. On 02/06/2013 03:27 AM, Timour Katchaounov wrote: > Igor, > > please review the following simple patch. > > Timour > > ------------------------------------------------------------ > revno: 3622 > revision-id: timour(a)askmonty.org-20130206112530-nwo4xto0cwv65ece > parent: sergii(a)pisem.net-20130128081223-mp9rsd3t9soz8lly > fixes bug(s): https://mariadb.atlassian.net/browse/MDEV-4140 > committer: timour(a)askmonty.org > branch nick: 5.3-md4140 > timestamp: Wed 2013-02-06 13:25:30 +0200 > message: > Fix for MDEV-4140 > > Analysis: > Range analysis detects that the subquery is expensive and doesn't > build a range access method. Later, the applicability test for loose > scan doesn't take that into account, and builds a loose scan method > without a range scan on the min/max column. As a result loose scan > fetches the first key in each group, rather than the first key that > satisfies the condition on the min/max column. > > Solution: > Since there is no SEL_ARG tree to be used for the min/max column, > it is not possible to use loose scan if the min/max column is compared > with an expensive scalar subquery. Make the test for loose scan > applicability to be in sync with the range analysis code by testing if > the min/max argument is compared with an expensive predicate. > >

1 0

[Maria-developers] MariaDB 10.0.1-alpha release prep
by Daniel Bartholomew 06 Feb '13

06 Feb '13

I've begun prepping MariaDB 10.0.1-alpha for release. Preliminary changelog and release notes are here: - https://kb.askmonty.org/en/mariadb-1001-release-notes/ - https://kb.askmonty.org/en/mariadb-1001-changelog/ I saw in the changelog a couple of features on https://kb.askmonty.org/en/what-is-mariadb-100/ that were added in this release (MDEV-4011 & Cassandra Storage Engine), if there are other new features, or features backported from MySQL 5.6, or anything else I should be sure to mention in the release notes, please let me know ASAP. Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

3 4

[Maria-developers] mariadb-server not installable on Debian Testing AMD64
by Thomas Hackert 05 Feb '13

05 Feb '13

SEND-PR: -*- send-pr -*- SEND-PR: Lines starting with `SEND-PR' will be removed automatically, as SEND-PR: will all comments (text enclosed in `<' and `>'). SEND-PR: From: thackert(a)nexgo.de To: maria-developers(a)lists.launchpad.net Subject: mariadb-server not installable on Debian Testing AMD64 >Description: Hello @ll, this is my first try to report a bug with mysqlbug. If I missed something and/or have something wrong, feel free to ask, but do not condemn me ... ;) I have tried to install mariadb-server on Debian Testing AMD64. I got the error message <quote> <snip> [ ok ] Stopping MariaDB database server: mysqld. 130130 17:28:50 [Note] Plugin 'InnoDB' is disabled. 130130 17:28:50 [Note] Plugin 'FEEDBACK' is disabled. [....] Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . FAIL . . . . . . . failed! invoke-rc.d: initscript mysql, action "start" failed. dpkg: Fehler beim Bearbeiten von mariadb-server-10.0 (--configure): Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück dpkg: Abhängigkeitsprobleme verhindern Konfiguration von mariadb-server: mariadb-server hängt ab von mariadb-server-10.0; aber: Paket mariadb-server-10.0 ist noch nicht konfiguriert. dpkg: Fehler beim Bearbeiten von mariadb-server (--configure): Abhängigkeitsprobleme - verbleibt unkonfiguriert Trigger für menu werden verarbeitet ... Fehler traten auf beim Bearbeiten von: mariadb-server-10.0 mariadb-server </quote> ... :( I get the same message, if I try to install mysql-server from Debian's repository ... :( >How-To-Repeat: 1. Add " # MariaDB 10.0 repository list - created 2013-01-30 16:20 UTC # http://mariadb.org/mariadb/repositories/deb # http://mirror.netcologne.de/mariadb//repo/10.0/debian wheezy main deb-src http://mirror.netcologne.de/mariadb//repo/10.0/debian wheezy main" to your /etc/apt/sources.list. 2. Run "apt-get update" 3. Run "apt-get install mariadb-server" The error message appears nearly at the end of the installation ... :( >Fix: I do not know, sorry ... :( But maybe it is possible to check first, if mysql- or mariadb-server was installed before, and if it was, stop it. If it was not installed before, than it should not try to stop the server ... ;) >Submitter-Id: <submitter ID> >Originator: Thomas Hackert >Organization: <none> >MySQL support: [none | licence | email support | extended email support >] >Synopsis: mariadb-server not installable on Debian Testing AMD64 >Severity: serious >Priority: medium >Category: mysql >Class: <[ sw-bug | doc-bug | change-request | support ] (one >line)> >Release: mysql-10.0.0-mariadb1~wheezy (mariadb.org binary >distribution) >C compiler: gcc-4.7.real (Debian 4.7.1-2) 4.7.1 >C++ compiler: g++-4.7.real (Debian 4.7.1-2) 4.7.1 >Environment: <machine, os, target, libraries (multiple lines)> System: Linux neo 3.2.0-4-amd64 #1 SMP Debian 3.2.35-2 x86_64 GNU/Linux Some paths: /usr/bin/perl /usr/bin/make /usr/bin/gcc /usr/bin/cc GCC: Using built-in specs. COLLECT_GCC=/usr/bin/gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.7/lto-wrapper Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Debian 4.7.2-5' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --with-arch-32=i586 --with-tune=generic --enanable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --with-arch-32=i586 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.7.2 (Debian 4.7.2-5) Compilation info (call): CC='/usr/bin/gcc' CFLAGS='-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' CXX='/usr/bin/g++' CXXFLAGS='-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized -fno-exceptions -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' LDFLAGS='' ASFLAGS='' Compilation info (used): CC='/usr/bin/gcc' CFLAGS='-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' CXX='/usr/bin/g++' CXXFLAGS='-O2 -fno-omit-frame-pointer -g -pipe -Wall -Wno-uninitialized -fno-exceptions -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' LDFLAGS='' ASFLAGS='' Perl: This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-gnu-thread-multi -- The way to make a small fortune in the commodities market is to start with a large fortune.

3 7

Re: [Maria-developers] [Maria-discuss] Files under /etc/my.cnf.d/ read by default
by Honza Horak 05 Feb '13

05 Feb '13

This should have been posted better to maria-developers, probably, so re-sending.. On 02/05/2013 01:55 PM, Honza Horak wrote: > Hi guys, > > while we were packaging MariaDB into Fedora (finally) we used the > upstream's cnf files structure, which means /etc/my.cnf, that includes > all files from directory /etc/my.cnf.d/* using statement !includedir. > > That works fine when user installs (not updates) the MariaDB packages > without having them installed before. The problem is if user has changed > /etc/my.cnf before updating -- then /etc/my.cnf doesn't get updated > (!includedir won't be added) and we end with files under /etc/my.cnf.d/* > but nothing what includes them. That is quite confusing for users that > may change something in /etc/my.cnf.d/client.cnf but nothing happens. > > Since files under /etc/my.cnf.d/* are empty by default it shouldn't do > any harm if all files corresponding with /etc/my.cnf.d/*.cnf template > get loaded even without !includedir in /etc/my.cnf. Actually I believe > that is what users expect to happen when they see files under > /etc/my.cnf.d.. > > So, that's basically what I'm proposing -- read files under > /etc/my.cnf.d/* by default, without a need to use !includedir. What do > you think? > > Regards, > Honza > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp

1 0

[Maria-developers] InnoDB Compression
by Gordan Bobic 04 Feb '13

04 Feb '13

Hi, I've just tried testing InnoDB page compression, and I'm sure I must be doing something wrong because the disk space usage in my database directory is the same before and after the compression. To compress the tables I am using: ALTER TABLE $table ENGINE=InnoDB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=$size; with $size I tried being 8KB and 4KB, and in each case "du -h" on the DB directory is the same as before the compression. innodb_file_per_table is set. SHOW CREATE TABLE afterwards shows the ROW_FORMAT and KEY_BLOCK_SIZE to be as per the ALTER TABLE statement. 1) Am I doing it wrong? 2) Is the InnoDB compression feature available in MariaDB? 3) Is the compression on by default transparently? 4) Is there any other explanation for lack of effect? As a cross-check, I tried taking my biggest .ibd file and compressing it with "lzop -1" and that compressed it from 80MB down to 22MB, which doesn't sound right if the data in it was already compressed in any way. Has anybody got any possible explanations? TIA Gordan

2 2

Re: [Maria-developers] [Commits] Rev 3657: MDEV-4127 : Export additional symbols when building RPM
by Sergei Golubchik 04 Feb '13

04 Feb '13

Hi, Vladislav! May be you'd rather rename symbols with #define, just as redhat does? On Feb 03, Vladislav Vaintroub wrote: > At file:///H:/bzr/5.5/ > > ------------------------------------------------------------ > revno: 3657 > revision-id: wlad(a)montyprogram.com-20130203164516-ex2ftmyca5b507g3 > parent: elenst(a)ubuntu11.home-20130202225357-za9bglcf2ngv36xj > committer: Vladislav Vaintroub <wlad(a)montyprogram.com> > branch nick: 5.5 > timestamp: Sun 2013-02-03 17:45:16 +0100 > message: > MDEV-4127 : Export additional symbols when building RPM, > to enable both recompiling mysqli or odbc from sources in addition > to drop-in replacement functionality. > > The case in question is compiling mysqli from sources, that needs > client_errors via ER() macro. > > Previously, we exported it as mysql_client_errors (compatibly to > Fedora's style symbol renaming, see MDEV-3842). > However, if MariaDB header files are used when compiling mysqli, > client_errors needs to be exported with its original name. Regards, Sergei

2 2

[Maria-developers] Coverity scans of MariaDB source
by Christian Convey 04 Feb '13

04 Feb '13

Coverity has a deal called Coverity Scan, where they offer free static analyses of open-source programs (http://scan.coverity.com/) I've used Coverity Prevent at work, and I was pretty happy with the analyses it performed. Is it an intentional choice that MariaDB isn't making use of Coverity Scan? I'm interested in starting to contribute to MariaDB, and I'm wondering if it would be useful to the project for me to pursue this as a first task. - Christian

2 2

[Maria-developers] CMake vs. autotools on Linux?
by Christian Convey 04 Feb '13

04 Feb '13

Hi guys, I noticed that MariaDB is built (indirectly) with CMake on Windows, and with autotools on Linux. Why are both systems used? I would have assumed that because CMake is pretty good at generating Makefiles on Linux and OS X, there would be little reason to continue using autotools. Thanks, Christian

2 3

Re: [Maria-developers] Summary of the MDEV-3882 (.deb upgrade from mysql to mariadb fails due to apt-get considering it a downgrade)
by Kristian Nielsen 02 Feb '13

02 Feb '13

[Moving the discussion to maria-developers@, please use the public mailing lists for discussions like this] Alexey Botchkov <holyfoot(a)askmonty.org> writes: > No matter what dependencies or new packages i created inside MariaDB, > Debian's first idea was to try to install the 'later' version of > libmysqlclient18. At the moment > that was the Oracle's one. Yes, we could change our version's number to be > 'later' than Oracle's, but that would turn same problems on MySQL, and Not really. It's not "Oracle's", it is the native Debian/Ubuntu package. So if users want MariaDB, they can add our repository. If they want the official Debian/Ubuntu package, they can just not add our repository. So it makes sense that when users explicitly add the MariaDB repository, they get the MariaDB version of all the packages automatically. > would help us only temporarely - to the moment Oracle releases even > later' version. We can use epoch to ensure that our version will always be bigger than the official Debian/Ubuntu version of MySQL: http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version Something like 1:5.5.29-mariadb1~wheezy. This will always be prefered by apt-get over any official Debian package. > I didn't find anything that could affect apt-get and guys on #Debian > didn't recommend anything. I believe it works to specify desired version explicitly, as mentioned in the bug report as workaround: sudo apt-get install mariadb-server libmysqlclient18=5.5.29-mariadb1~wheezy > So for now i'd do about this: > - change nothing in the existing mariadb-produced debian packages. > - recommend to use 'aptitude' installing the mariadb-server on Debian. > So how do you like my proposal? Any other ideas? I think it is better if we use the epoch to make MariaDB upgrades work automatically with both apt-get and aptitude, like they used to. Did you check how Percona handles the similar problem? - Kristian.

2 1

Re: [Maria-developers] Fwd: [Commits] Rev 3622: Analysis: in file:///home/tsk/mprog/src/5.3-md641/
by Igor Babaev 31 Jan '13

31 Jan '13

On 01/31/2013 04:41 AM, Timour Katchaounov wrote: > Igor, > > Let me know if I can push the patch below. It fixes the bug > according to our last discussion. If the number of group_list > elements was known, I could add a test already in the beginning > of the function, however such a counter doesn't exist, and I > didn't want to complicate the code just for this special case. > That's why the test is done where it is now. > > Timour > > > ------------------------------------------------------------ > revno: 3622 > revision-id: timour(a)askmonty.org-20130131120040-oan9zei8ojyco178 > parent: sergii(a)pisem.net-20130128081223-mp9rsd3t9soz8lly > fixes bug(s): https://mariadb.atlassian.net/browse/MDEV-641 > committer: timour(a)askmonty.org > branch nick: 5.3-md641 > timestamp: Thu 2013-01-31 14:00:40 +0200 > message: > Analysis: > Range analysis discoveres that the query can be executed via loose > index scan for GROUP BY. > Later, GROUP BY analysis fails to confirm that the GROUP operation can > be computed via an > index because there is no logic to handle duplicate field references > in the GROUP clause. > As a result the optimizer the optimizer produces an inconsistent plan. > It constructs a > temporary table, but on the other hand the group fields are not set to > point there. > > Solution: > Make loose scan analysis work in sync with order by analysis. In the > case of duplicate > columns loose scan will not be applicable. This limitation will be > lifted in 10.0 by > removing duplicate columns. > >

1 0

Re: [Maria-developers] what pros/cons of storing binary log in an InnoDB table?
by Kristian Nielsen 31 Jan '13

31 Jan '13

Zardosht Kasheff <zardosht(a)gmail.com> writes: > Instead of having the binary log be stored as a log file, a table with > a transactional engine (like InnoDB) is used. I have NOT done any Yes, this is a tempting idea, is it not? Drizzle even implemented this. But there are some subtle problems, and I think there are better solutions. One problem you will find if you try implementing this is - how are you going to keep the concept of binlog order? A relational table is unordered. You can use an increasing uint64 as the primary key, but then how are you going to ensure that the order in the table is the same as the commit order in InnoDB (InnoDB redo log)? If you take a lock from allocating the sequence number to doing the commit, then you have introduced a serialisation bottleneck and killed group commit. If you accept different order (as Drizzle does), then when you take an XtraBackup, it may not be consistent with the binlog. You will be writing your data *6* times: To the real table, when flushing its buffer pool pages. To the double-write buffer while flushing, and to the redo log before flushing. Then to the binlog table, and before that to the redo log and the doublewrite buffer. I could never reconcile myself with this. So it is tempting, but re-implementing binlog is a huge change (also in terms of end-user visibility), and my personal opinion is that for such huge a change we need to go to the best solution, not second-best. > investigation into whether this is possible, but I'm wondering what > people abstractly think of this idea. > My major motivation is to find a way to have users not be forced to > fsync on every transaction commit on slaves. With crash safety of My preference is MWL#164: http://askmonty.org/worklog/Server-RawIdeaBin/?tid=164 This reduces to one fsync() per group commit. But as you point out, this is still horrendously expensive for a single-threaded slave :-( Abstractly, the proper solution is to store the binlog and the InnoDB redo log (and all other transactional-type logs) in the *same* log. The upper layer would provide a general transactional log facility, and binlog, InnoDB, partitioning, etc. would use this instead of their own logs. But it is hard to see how we could arrive there from current code ... For MariaDB global transaction ID, I support crash-safe without binlog enabled, so there turning off fsync() will be possible. But this does not help if a slave also needs to act as a master. Parallel replication can help a lot, by allowing group commit, amortising the cost of the fsync(). But it would still be nice to solve the orginal problem. As you say, for a slave it is not a problem to loose transactions in a crash as they can be just replicated again. The problem is to be consistent between what is in the InnoDB tables and what is in the binlog. InnoDB already records in a crash-safe way the binlog position of the last commit. So if we can ensure that InnoDB is never ahead of the binlog, we can just cut the binlog after a crash if it has more events than what InnoDB recovered. We could do this by doing binlog writes as async I/O O_DIRECT writes, and sending a signal somehow to InnoDB so it would buffer its own redo log until binlog is safe on disk. Then no fsync() would be needed. But this is so far just a loose idea, it needs more thought to be sure it can work... Anyway, this is my current long-term plan: if we cannot have a proper single database-wide transaction log, then instead coordinate writes between the multiple logs that exist. Delay writes to logs so that they are never ahead of the master log. And at crash, either recover missing pieces of logs from the master log (MWL#164), or cut logs to be consistent. Then we can be consistent without any fsync() delays. > - As I understand it, for two phase commit, storage engines are > currently expected to fsync once after a transaction is prepared and > once after a transaction is committed. If the binary log is an InnoDB > table, then two-phase commit is unnecessary and the number of fsyncs > on the system is cut in half. In MariaDB 10 and MySQL 5.6, the fsync() at commit is no longer needed (InnoDB does not have it). [Note that IIRC, MySQL 5.6 breaks the storage engine API, you will need to implement the skip-fsync-at-commit-stage, or you will loose group commit.] > - With crash safe slaves on MySQL 5.6, one theoretically does not > need to enable fsyncs on transaction commit on slaves. If some data > gets lost after a crash, then the data may be replayed from the > master. However, to get GTIDs working, one must enable the binary log > on slaves, which requires two-phase commit, which require fsyncs. By Yes, requiring binlog on slaves is a serious mistake :-( Ok, as you can perhaps tell, this is something I have thought a lot about over the past years, so I have a lot of ideas but it is a bit difficult to put it all in a single mail reply without it becomming mostly a jumble of loose ideas. You are welcome to ask for more details on any specific points of interest. - Kristian.

5 9

Re: [Maria-developers] reducing fsyncs during handlerton->prepare and handlerton->commit in 5.6
by Kristian Nielsen 31 Jan '13

31 Jan '13

Zardosht Kasheff <zardosht(a)gmail.com> writes: > - in MySQL 5.6 and MariaDB 10.0, what are the fsyncing requirements > for storage engines during prepare and commit? > - are there new APIs that the storage engine must comply with in > order to get these benefits? In MariaDB 10.0, there is a new commit_checkpoint_request handlerton method. If the storage engine implements that, it need not do the fsync() at commit. It must also implement the commit_ordered method from 5.3. Check comments in sql/handler.h for these two methods, and let me know if there are any questions. The intention is that storage engines need not do much extra work for this functionality, eg. the patch for InnoDB support was a couple hundred lines at most IIRC. > - in MySQL 5.6, I see new handlerton methods commit_low and > prepare_low. What do these APIs do? What is their contract? I think MySQL developers should answer for this. But it is related to this new check in InnoDB: thd_requested_durability(trx->mysql_thd) == HA_IGNORE_DURABILITY And the existing flush_logs handlerton method is also used. My guess is that commit_low is just internal server stuff that the storage engine does not need to consider. The MySQL 5.6 mechanism is mostly the same as the MariaDB 10.0 one in what it actually does, but they did not bother to try to be compatible with the storage engine API in MariaDB 5.3. So looks like it is #ifdef time for you if you want to support both. - Kristian.

1 0

[Maria-developers] how to build mariadb RPMs
by Rich Prohaska 30 Jan '13

30 Jan '13

How do you build the mariadb RPMs? Are you using rpmbuild or cpack? Is there a top level script that uses rpmbuild or cpack that I should use? Thanks Rich Prohaska

3 11

[Maria-developers] MariaDB 5.1.67, 5.2.14, 5.3.12, 5.5.29 Release Prep
by Daniel Bartholomew 29 Jan '13

29 Jan '13

All, I've prepped changelog and release notes pages for MariaDB 5.1.67, 5.2.14, and 5.3.12. I'll shortly be doing the same for MariaDB 5.5.29. https://kb.askmonty.org/en/mariadb-5167-release-notes/ https://kb.askmonty.org/en/mariadb-5167-changelog/ https://kb.askmonty.org/en/mariadb-5214-release-notes/ https://kb.askmonty.org/en/mariadb-5214-changelog/ https://kb.askmonty.org/en/mariadb-5312-release-notes/ https://kb.askmonty.org/en/mariadb-5312-changelog/ If there is anything I should be sure to mention in the release notes, please let me know ASAP. I plan on announcing the releases once 5.5.29 has been uploaded to the mirrors. It's still building at the moment so the earliest the release could happen is tomorrow and it could be pushed back to Thursday if there's a delay of some sort. Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

Re: [Maria-developers] [Commits] Rev 3653: MDEV-3882 .deb upgrade from mysql to mariadb fails due to apt-get considering it a downgrade. in file:///home/hf/wmar/mdev-3882/
by Kristian Nielsen 29 Jan '13

29 Jan '13

<holyfoot(a)askmonty.org> writes: > message: > MDEV-3882 .deb upgrade from mysql to mariadb fails due to apt-get considering it a downgrade. > The libmysqlclient18 package and all the dependencies on it removed. > The libmariadbclient18 set to provide the libmysqlclient's functions and > conflicts with all the libmysqlclients. > > === modified file 'debian/dist/Debian/control' > --- a/debian/dist/Debian/control 2012-08-23 13:32:03 +0000 > +++ b/debian/dist/Debian/control 2013-01-29 09:56:54 +0000 > @@ -13,7 +13,9 @@ Vcs-Bzr: bzr://lp:maria > Package: libmariadbclient18 > Section: libs > Architecture: any > -Depends: mariadb-common, libmysqlclient18 (= ${source:Version}), ${shlibs:Depends}, ${misc:Depends} > +Depends: mariadb-common, ${shlibs:Depends}, ${misc:Depends} > +Provides: libmysqlclient18 > +Conflicts: libmysqlclient10, libmysqlclient12, libmysqlclient14, libmysqlclient15, libmysqlclient16, libmysqlclient18 Are you sure this works? The original reason for introducing a libmysqlclient18 package in the first place (with all the problems in introduces) is that there are standard packages with a versioned dependency on libmysqlclient18. And versioned dependencies can not (used not to?) be satisfied by a Provides: Eg. the following gives 143 packages with versioned dependencies on libmysqlclient18: apt-cache rdepends libmysqlclient18|(read x; read x; while read x; do apt-cache show $x|grep ^Depends: | perl -nle 'print "'"$x:"' $1" if /libmysqlclient18[ \t]*$([^)]+)$/' ; done) You need to test a number of these that they are still installable when MariaDB is installed, also on old .deb distros (ie. Ubuntu 8.04). On the other hand, if this works then it is a nice solution for sure, and could also solve other issues. A related issue is that we might get the distro mysql-common package or the MariaDB mysql-common package somewhat at random, depending on which has the newer version number on any given day. I believe we made it so we work with the distro mysql-common package as well as the MariaDB one, it it will not necessarily cause any problems. It does seem a bit of a mess though, maybe we should just drop the mysql-common package from our repos? - Kristian.

1 0

Re: [Maria-developers] Fwd: [Commits] Rev 3616: Fix for bug in file:///home/tsk/mprog/src/5.3-md641/
by Igor Babaev 28 Jan '13

28 Jan '13

Timour, It's not clear for me: 1. why group by queries with GROUP BY col1, col2, where col1,col2 are defined over the same table, are considered as possibly good for for loose scan optimization. 2. how it comes that unique index is considered as a candidate for a loose scan (see your test case). (The second issue, of course, are unrelated to bug mdev-641). Besides, didn't you notice that you removes duplicates in group by under any circumstances. How it will affect the following query: select a, sum(c) from t1 group by a, a with rollup; ? In general, removing duplicates from GROUP BY and ORDER BY lists is a good optimization, but why it should be done in connection with this bug? Regards, Igor. On 01/24/2013 06:38 AM, Timour Katchaounov wrote: > Igor, > > Could you please review the following patch (we discussed it at the team > call): > > ------------------------------------------------------------ > revno: 3616 > revision-id: timour(a)askmonty.org-20130124143526-bpaiyushble1ieek > parent: timour(a)askmonty.org-20130117140805-4kyoq7azx4v2irhq > fixes bug: https://mariadb.atlassian.net/browse/MDEV-614 > committer: timour(a)askmonty.org > branch nick: 5.3-md641 > timestamp: Thu 2013-01-24 16:35:26 +0200 > message: > Fix for bug > MDEV-641 LP:1002108 - Wrong result (or crash) from a query with > duplicated field in the group list and a limit clause > Bug#11761078: 53534: INCORRECT 'SELECT SQL_BIG_RESULT...' > WITH GROUP BY ON DUPLICATED FIELDS > > Analysis > Range analysis discoveres that the query can be executed via loose > index scan for GROUP BY. > Later, GROUP BY analysis fails to confirm that the GROUP operation can > be computed via an > index because there is no logic to handle duplicate field references > in the GROUP clause. > As a result the optimizer the optimizer produces an inconsistent plan. > It constructs a > temporary table, but on the other hand the group fields are not set to > point there. > > Solution: > Make order by analysis be in sync with loose scan analysis for group > by. Instead of > changing the logic of how indexes are analyzed for order by, the patch > takes the same > approach as MySQL - normalize the query by removing duplicate column > references. The > difference from MySQL is that we remove duplicates earlier, by reusing > remove_const() > which already removed unneeded ORDER elements. > >

1 0

[Maria-developers] MDEV-3875 Wrong result on a DISTINCT query ... and GROUP BY
by Sergei Golubchik 27 Jan '13

27 Jan '13

Hi, Igor, See my patch below. The only questionable part - in remove_dup_with_compare(). Before it was memcpy'ing only part of the record, skipping const fields. Now it memcpy's the complete record. I don't think it matters much performance-wise. An alternative would be to do two memcpy's - for the NULL bitmap and for the fields (because const fields are located between the NULL bitmap that we need, and the other non-const fields). Another alternative would be to have const fields at the end of the record, but it'd be a much larger and more risky change. ------------------------------------------------------------ revno: 3620 fixes bug: https://mariadb.atlassian.net/browse/MDEV-3875 committer: Sergei Golubchik <sergii(a)pisem.net> branch nick: 5.3 timestamp: Sat 2013-01-26 22:33:18 +0100 message: MDEV-3875 Wrong result (missing row) on a DISTINCT query with the same subquery in the SELECT list and GROUP BY fix remove_dup_with_hash_index() and remove_dup_with_compare() to take NULLs into account modified: mysql-test/r/distinct.result mysql-test/t/distinct.test sql/field.cc sql/field.h sql/filesort.cc sql/sql_select.cc diff: === modified file 'mysql-test/r/distinct.result' --- mysql-test/r/distinct.result 2011-12-15 22:26:59 +0000 +++ mysql-test/r/distinct.result 2013-01-26 21:33:18 +0000 @@ -847,3 +847,35 @@ time(f1) 00:00:00.000200 00:00:00.000300 drop table t1; +create table t1(i int, g int); +insert into t1 values (null, 1), (0, 2); +select distinct i from t1 group by g; +i +NULL +0 +drop table t1; +create table t1(i int, g blob); +insert into t1 values (null, 1), (0, 2); +select distinct i from t1 group by g; +i +NULL +0 +drop table t1; +create table t1 (a int) engine=myisam; +insert into t1 values (0),(7); +create table t2 (b int) engine=myisam; +insert into t2 values (7),(0),(3); +create algorithm=temptable view v as +select distinct (select max(a) from t1 where alias.b = a) as field1 from t2 as alias group by field1; +select * from v; +field1 +NULL +0 +7 +select distinct (select max(a) from t1 where alias.b = a) as field1 from t2 as alias group by field1; +field1 +NULL +0 +7 +drop view v; +drop table t1, t2; === modified file 'mysql-test/t/distinct.test' --- mysql-test/t/distinct.test 2011-12-15 22:26:59 +0000 +++ mysql-test/t/distinct.test 2013-01-26 21:33:18 +0000 @@ -658,3 +658,27 @@ select time(f1) from t1 ; select distinct time(f1) from t1 ; drop table t1; +# +# MDEV-3875 Wrong result (missing row) on a DISTINCT query with the same subquery in the SELECT list and GROUP BY +# MySQL Bug#66896 Distinct not distinguishing 0 from NULL when GROUP BY is used +# +create table t1(i int, g int); # remove_dup_with_hash_index +insert into t1 values (null, 1), (0, 2); +select distinct i from t1 group by g; +drop table t1; + +create table t1(i int, g blob); # remove_dup_with_compare +insert into t1 values (null, 1), (0, 2); +select distinct i from t1 group by g; +drop table t1; + +create table t1 (a int) engine=myisam; +insert into t1 values (0),(7); +create table t2 (b int) engine=myisam; +insert into t2 values (7),(0),(3); +create algorithm=temptable view v as +select distinct (select max(a) from t1 where alias.b = a) as field1 from t2 as alias group by field1; +select * from v; +select distinct (select max(a) from t1 where alias.b = a) as field1 from t2 as alias group by field1; +drop view v; +drop table t1, t2; === modified file 'sql/field.cc' --- sql/field.cc 2012-08-29 15:55:59 +0000 +++ sql/field.cc 2013-01-26 21:33:18 +0000 @@ -1062,6 +1062,21 @@ bool Field::type_can_have_key_part(enum } +void Field::make_sort_key(uchar *buff,uint length) +{ + if (maybe_null()) + { + if (is_null()) + { + bzero(buff, length + 1); + return; + } + *buff++= 1; + } + sort_string(buff, length); +} + + /** Numeric fields base class constructor. */ === modified file 'sql/field.h' --- sql/field.h 2012-06-23 22:00:05 +0000 +++ sql/field.h 2013-01-26 21:33:18 +0000 @@ -386,6 +386,7 @@ class Field return bytes; } + void make_sort_key(uchar *buff, uint length); virtual void make_field(Send_field *); virtual void sort_string(uchar *buff,uint length)=0; virtual bool optimize_range(uint idx, uint part); === modified file 'sql/filesort.cc' --- sql/filesort.cc 2012-11-09 08:11:20 +0000 +++ sql/filesort.cc 2013-01-26 21:33:18 +0000 @@ -783,21 +783,9 @@ static void make_sortkey(register SORTPA bool maybe_null=0; if ((field=sort_field->field)) { // Field - if (field->maybe_null()) - { - if (field->is_null()) - { - if (sort_field->reverse) - bfill(to,sort_field->length+1,(char) 255); - else - bzero((char*) to,sort_field->length+1); - to+= sort_field->length+1; - continue; - } - else - *to++=1; - } - field->sort_string(to, sort_field->length); + field->make_sort_key(to, sort_field->length); + if ((maybe_null = field->maybe_null())) + to++; } else { // Item @@ -949,8 +937,11 @@ static void make_sortkey(register SORTPA } if (sort_field->reverse) { /* Revers key */ - if (maybe_null) - to[-1]= ~to[-1]; + if (maybe_null && (to[-1]= !to[-1])) + { + to+= sort_field->length; // don't waste the time reversing all 0's + continue; + } length=sort_field->length; while (length--) { === modified file 'sql/sql_select.cc' --- sql/sql_select.cc 2013-01-16 13:11:13 +0000 +++ sql/sql_select.cc 2013-01-26 21:33:18 +0000 @@ -204,7 +204,7 @@ static int create_sort_index(THD *thd, J static int remove_duplicates(JOIN *join,TABLE *entry,List<Item> &fields, Item *having); static int remove_dup_with_compare(THD *thd, TABLE *entry, Field **field, - ulong offset,Item *having); + Item *having); static int remove_dup_with_hash_index(THD *thd,TABLE *table, uint field_count, Field **first_field, ulong key_length,Item *having); @@ -18891,19 +18891,24 @@ static bool fix_having(JOIN *join, Item #endif -/***************************************************************************** - Remove duplicates from tmp table - This should be recoded to add a unique index to the table and remove - duplicates - Table is a locked single thread table - fields is the number of fields to check (from the end) -*****************************************************************************/ +/** + Compare fields from table->record[0] and table->record[1], + possibly skipping few first fields. + @param table + @param ptr field to start the comparison from, + somewhere in the table->field[] array + + @retval 1 different + @retval 0 identical +*/ static bool compare_record(TABLE *table, Field **ptr) { for (; *ptr ; ptr++) { - if ((*ptr)->cmp_offset(table->s->rec_buff_length)) + Field *f= *ptr; + if (f->is_null() != f->is_null(table->s->rec_buff_length) || + (!f->is_null() && f->cmp_offset(table->s->rec_buff_length))) return 1; } return 0; @@ -18931,15 +18936,15 @@ static void free_blobs(Field **ptr) static int -remove_duplicates(JOIN *join, TABLE *entry,List<Item> &fields, Item *having) +remove_duplicates(JOIN *join, TABLE *table, List<Item> &fields, Item *having) { int error; - ulong reclength,offset; + ulong keylength= 0; uint field_count; THD *thd= join->thd; DBUG_ENTER("remove_duplicates"); - entry->reginfo.lock_type=TL_WRITE; + table->reginfo.lock_type=TL_WRITE; /* Calculate how many saved fields there is in list */ field_count=0; @@ -18956,24 +18961,21 @@ remove_duplicates(JOIN *join, TABLE *ent join->unit->select_limit_cnt= 1; // Only send first row DBUG_RETURN(0); } - Field **first_field=entry->field+entry->s->fields - field_count; - offset= (field_count ? - entry->field[entry->s->fields - field_count]-> - offset(entry->record[0]) : 0); - reclength=entry->s->reclength-offset; - - free_io_cache(entry); // Safety - entry->file->info(HA_STATUS_VARIABLE); - if (entry->s->db_type() == heap_hton || - (!entry->s->blob_fields && - ((ALIGN_SIZE(reclength) + HASH_OVERHEAD) * entry->file->stats.records < + + Field **first_field=table->field+table->s->fields - field_count; + for (Field **ptr=first_field; *ptr; ptr++) + keylength+= (*ptr)->sort_length() + (*ptr)->maybe_null(); + + free_io_cache(table); // Safety + table->file->info(HA_STATUS_VARIABLE); + if (table->s->db_type() == heap_hton || + (!table->s->blob_fields && + ((ALIGN_SIZE(keylength) + HASH_OVERHEAD) * table->file->stats.records < thd->variables.sortbuff_size))) - error=remove_dup_with_hash_index(join->thd, entry, - field_count, first_field, - reclength, having); + error=remove_dup_with_hash_index(join->thd, table, field_count, first_field, + keylength, having); else - error=remove_dup_with_compare(join->thd, entry, first_field, offset, - having); + error=remove_dup_with_compare(join->thd, table, first_field, having); free_blobs(first_field); DBUG_RETURN(error); @@ -18981,18 +18983,13 @@ remove_duplicates(JOIN *join, TABLE *ent static int remove_dup_with_compare(THD *thd, TABLE *table, Field **first_field, - ulong offset, Item *having) + Item *having) { handler *file=table->file; - char *org_record,*new_record; - uchar *record; + uchar *record=table->record[0]; int error; - ulong reclength= table->s->reclength-offset; DBUG_ENTER("remove_dup_with_compare"); - org_record=(char*) (record=table->record[0])+offset; - new_record=(char*) table->record[1]+offset; - if (file->ha_rnd_init_with_error(1)) DBUG_RETURN(1); @@ -19029,7 +19026,7 @@ static int remove_dup_with_compare(THD * error=0; goto err; } - memcpy(new_record,org_record,reclength); + store_record(table,record[1]); /* Read through rest of file and mark duplicated rows deleted */ bool found=0; @@ -19088,8 +19085,9 @@ static int remove_dup_with_hash_index(TH int error; handler *file= table->file; ulong extra_length= ALIGN_SIZE(key_length)-key_length; - uint *field_lengths,*field_length; + uint *field_lengths, *field_length; HASH hash; + Field **ptr; DBUG_ENTER("remove_dup_with_hash_index"); if (!my_multi_malloc(MYF(MY_WME), @@ -19101,21 +19099,8 @@ static int remove_dup_with_hash_index(TH NullS)) DBUG_RETURN(1); - { - Field **ptr; - ulong total_length= 0; - for (ptr= first_field, field_length=field_lengths ; *ptr ; ptr++) - { - uint length= (*ptr)->sort_length(); - (*field_length++)= length; - total_length+= length; - } - DBUG_PRINT("info",("field_count: %u key_length: %lu total_length: %lu", - field_count, key_length, total_length)); - DBUG_ASSERT(total_length <= key_length); - key_length= total_length; - extra_length= ALIGN_SIZE(key_length)-key_length; - } + for (ptr= first_field, field_length=field_lengths ; *ptr ; ptr++) + (*field_length++)= (*ptr)->sort_length(); if (hash_init(&hash, &my_charset_bin, (uint) file->stats.records, 0, key_length, (hash_get_key) 0, 0, 0)) @@ -19155,13 +19140,13 @@ static int remove_dup_with_hash_index(TH /* copy fields to key buffer */ org_key_pos= key_pos; field_length=field_lengths; - for (Field **ptr= first_field ; *ptr ; ptr++) + for (ptr= first_field ; *ptr ; ptr++) { - (*ptr)->sort_string(key_pos,*field_length); - key_pos+= *field_length++; + (*ptr)->make_sort_key(key_pos, *field_length); + key_pos+= (*ptr)->maybe_null() + *field_length++; } /* Check if it exists before */ - if (hash_search(&hash, org_key_pos, key_length)) + if (my_hash_search(&hash, org_key_pos, key_length)) { /* Duplicated found ; Remove the row */ if ((error= file->ha_delete_row(record))) Regards, Sergei

1 0

[Maria-developers] MariaDB RPMs
by Rich Prohaska 24 Jan '13

24 Jan '13

Suppose that we have a RHEL5 machine running and we want to install MariaDB on it. Unfortunately, it conflicts with files installed by the mysql package. # rpm -i MariaDB-5.5.28a-centos5-x86_64-common.rpm warning: MariaDB-5.5.28a-centos5-x86_64-common.rpm: Header V3 DSA signature: NOKEY, key ID 1bb943db file /etc/my.cnf from install of MariaDB-common-5.5.28a-1.x86_64 conflicts with file from package mysql-5.0.77-4.el5_6.6.x86_64 file /usr/share/mysql/charsets/Index.xml from install of MariaDB-common-5.5.28a-1.x86_64 conflicts with file from package mysql-5.0.77-4.el5_6.6.x86_64 file /usr/share/mysql/charsets/cp1250.xml from install of MariaDB-common-5.5.28a-1.x86_64 conflicts with file from package mysql-5.0.77-4.el5_6.6.x86_64 file /usr/share/mysql/charsets/cp1251.xml from install of MariaDB-common-5.5.28a-1.x86_64 conflicts with file from package mysql-5.0.77-4.el5_6.6.x86_64 Unfortunately, a lot of other packages require the mysql package. What is a good approach to solving this problem? Thanks

2 1

[Maria-developers] A new Ubuntu UPDATE event problem smashed MariaDB dead
by Peter W Bowey 23 Jan '13

23 Jan '13

Subject: A new Ubuntu UPDATE event problem smashed MariaDB 5.5.28a dead Hi MariaDB Team, Ubuntu released a update 23 hours ago that 'smashed out' omega8.cc's careful work to support MariaDB 5.5.28-mariadb-a1~precise. Ubuntu's new version is '5.5.29-0ubuntu012.10.1' dated - 2013-01-22 15:07:08 UTC. This does not work yet with MariaDB & Ubuntu 12.04 See Link here: https://launchpad.net/ubuntu/quantal/+source/mysql-5.5/5.5.29-0ubuntu0.12..… Now, I have been 'battling for hours' to work around this Ubuntu pain, while we wait on the good MariaDB team to re-syncs it's own updates with the Ubuntu 12.04 updates. Here is the 'crude *forced* style' I worked with to MariaDB back to life - after the Ubuntu *forced* update and dependency killed the database [MariaDB]: -------------------------------------------------------------------------------- server:~/aegir-boa-installer# aptitude install mariadb-server-5.5 mariadb-common The following NEW packages will be installed: libdbd-mysql-perl{a} libmariadbclient18{ab} libmysqlclient18{a} mariadb-client-5.5{a} mariadb-client-core-5.5{a} mariadb-server-5.5 mariadb-server-core-5.5{a} 0 packages upgraded, 7 newly installed, 0 to remove and 0 not upgraded. Need to get 31.8 MB of archives. After unpacking 107 MB will be used. The following packages have unmet dependencies: mysql-common : Breaks: mysql-client-5.1 which is a virtual package. Breaks: mysql-client-core-5.1 which is a virtual package. Breaks: mysql-server-core-5.1 which is a virtual package. libmariadbclient18 : Depends: libmysqlclient18 (= 5.5.28-mariadb-a1~precise) but 5.5.29-0ubuntu0.12.04.1 is to be installed. The following actions will resolve these dependencies: Keep the following packages at their current version: 1) libmariadbclient18 [Not Installed] 2) mariadb-client-5.5 [Not Installed] 3) mariadb-client-core-5.5 [Not Installed] 4) mariadb-server-5.5 [Not Installed] 5) mariadb-server-core-5.5 [Not Installed] Accept this solution? [Y/n/q/?] n The following actions will resolve these dependencies: Install the following packages: 1) libmysqlclient18 [5.5.28-mariadb-a1~precise (<NULL>)] Downgrade the following packages: 2) mysql-common [5.5.29-0ubuntu0.12.04.1 (now, precise-security, precise-updates) -> 5.5.28-mariadb-a1~precise (<NULL>)] Accept this solution? [Y/n/q/?] The following packages will be DOWNGRADED: mysql-common The following NEW packages will be installed: libdbd-mysql-perl{a} libmariadbclient18{a} libmysqlclient18{a} mariadb-client-5.5{a} mariadb-client-core-5.5{a} mariadb-server-5.5 mariadb-server-core-5.5{a} 0 packages upgraded, 7 newly installed, 1 downgraded, 0 to remove and 0 not upgraded. Need to get 30.9 MB of archives. After unpacking 104 MB will be used. Do you want to continue? [Y/n/?] y Get: 1 http://archive.ubuntu.com/ubuntu/ precise/main libdbd-mysql-perl i386 4.020-1build2 [104 kB] Get: 2 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main mysql-common all 5.5.28-mariadb-a1~precise [8,780 B] Get: 3 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main libmariadbclient18 i386 5.5.28-mariadb-a1~precise [817 kB] Get: 4 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main libmysqlclient18 i386 5.5.28-mariadb-a1~precise [2,920 B] Get: 5 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main mariadb-client-core-5.5 i386 5.5.28-mariadb-a1~precise [1,797 kB] Get: 6 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main mariadb-client-5.5 i386 5.5.28-mariadb-a1~precise [5,097 kB] Get: 7 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main mariadb-server-core-5.5 i386 5.5.28-mariadb-a1~precise [5,113 kB] Get: 8 http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu/ precise/main mariadb-server-5.5 i386 5.5.28-mariadb-a1~precise [18.0 MB] Fetched 30.9 MB in 30s (1,020 kB/s) Preconfiguring packages ... dpkg: warning: downgrading mysql-common from 5.5.29-0ubuntu0.12.04.1 to 5.528-mariadb-a1~precise. (Reading database ... 61332 files and directories currently installed.) Preparing to replace mysql-common 5.5.29-0ubuntu0.12.04.1 (using ..../mysql-common_5.5.28-mariadb-a1~precise_all.deb) ... Unpacking replacement mysql-common ... Selecting previously unselected package libmariadbclient18. Unpacking libmariadbclient18 (from ..../libmariadbclient18_5.5.28-mariadb-a1~precise_i386.deb) ... Selecting previously unselected package libmysqlclient18. Unpacking libmysqlclient18 (from ..../libmysqlclient18_5.5.28-mariadb-a1~precise_i386.deb) ... Selecting previously unselected package libdbd-mysql-perl. Unpacking libdbd-mysql-perl (from ..../libdbd-mysql-perl_4.020-1build2_i386deb) ... Selecting previously unselected package mariadb-client-core-5.5. Unpacking mariadb-client-core-5.5 (from ..../mariadb-client-core-5.5_5.5.28-mariadb-a1~precise_i386.deb) ... Selecting previously unselected package mariadb-client-5.5. Unpacking mariadb-client-5.5 (from ..../mariadb-client-5.5_5.5.28-mariadb-a1~precise_i386.deb) ... Selecting previously unselected package mariadb-server-core-5.5. Unpacking mariadb-server-core-5.5 (from ..../mariadb-server-core-5.5_5.5.28-mariadb-a1~precise_i386.deb) ... Selecting previously unselected package mariadb-server-5.5. Unpacking mariadb-server-5.5 (from ..../mariadb-server-5.5_5.5.28-mariadb-a1~precise_i386.deb) ... * Stopping MariaDB database server mysqld [ OK ] Processing triggers for man-db ... Processing triggers for ureadahead ... Setting up mysql-common (5.5.28-mariadb-a1~precise) ... Configuration file `/etc/mysql/my.cnf' ==> Modified (by you or by a script) since installation. ==> Package distributor has shipped an updated version. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version. *** my.cnf (Y/I/N/O/D/Z) [default=N] ? N Setting up libmysqlclient18 (5.5.28-mariadb-a1~precise) ... Setting up libdbd-mysql-perl (4.020-1build2) ... Setting up libmariadbclient18 (5.5.28-mariadb-a1~precise) ... Setting up mariadb-client-core-5.5 (5.5.28-mariadb-a1~precise) ... Setting up mariadb-client-5.5 (5.5.28-mariadb-a1~precise) ... Setting up mariadb-server-core-5.5 (5.5.28-mariadb-a1~precise) ... Setting up mariadb-server-5.5 (5.5.28-mariadb-a1~precise) ... * Stopping MariaDB database server mysqld [ OK ] * Starting MariaDB database server mysqld [ OK ] * Checking for corrupt, not cleanly closed and upgrade needing tables. Processing triggers for libc-bin ... ldconfig deferred processing now taking place -------------------------------------------------------------------------------- Ubuntu will drag in the new update dependants - > mysql-common [5.5.29-0ubuntu0.12.04.1 (now, precise-security, precise-updates) instead of 5.5.28-mariadb-a1~precise (<NULL>)] Peter Peter Bowey Web Design & Developer Peter Bowey Computer Solutions 69 Sutherland Ave, Victor Harbor, South Australia, Australia Tel.: 08 8552 8630 Fax: 08 8552 9185 Email: support(a)pbcomp.com.au Web: http://www.pbcomp.com.au/ Making Windows, Macintosh & Linux work for you! http://www.pbcomp.com.au/ This email and any attachment is intended only for the exclusive and confidential use of the addressee(s). If you are not the intended recipient, any use, interference with, disclosure or copying of this material is unauthorised and prohibited. If you have received this message in error, please notify the sender by return email immediately and delete the message from your computer without making any copies and publishing the contents therein. Consider the environment before printing this e-mail.

2 1

Re: [Maria-developers] Rev 3177: Fixed bug mdev-4063 (bug #56927)
by Sergei Golubchik 21 Jan '13

21 Jan '13

Hi, Igor! On Jan 21, Igor Babaev wrote: > revno: 3177 > revision-id: igor(a)askmonty.org-20130121084657-hvcval7hykojxpmg > parent: sergii(a)pisem.net-20130109225151-z9e7gh8z5nl38dxl > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.1-bug56927 > timestamp: Mon 2013-01-21 00:46:57 -0800 > message: > Fixed bug mdev-4063 (bug #56927). > This bug could result in returning 0 for the expressions of the form > <aggregate_function>(distinct field) when the system variable > max_heap_table_size was set to a small enough number. > It happened because the method Unique::walk() did not support > the case when more than one pass was needed to merge the trees > of distinct values saved in an external file. Looks ok. Just one thought. You use merge_many_buff() if the memory is not enough for the number_of_buffpeks*size. But merge_many_buff() only reduces the number of buffpeks down to 15. What if max_heap_table_size < 15*size ? I'd suggest to either 1) issue an error or 2) merge completely, like in Unique::get(). Actually, 2) should be easy, you just pass the last argument to your new Unique::merge() instead of TRUE, like in your patch, change to merge( ..., max_in_memory_size > MERGEBUFF2*size); Regards, Sergei

1 0

Re: [Maria-developers] Bug in trx_group_commit_leader()
by Kristian Nielsen 21 Jan '13

21 Jan '13

Michael Widenius <monty(a)askmonty.org> writes: > I noticed this compiler error on one of the build machines for 10.0-base: > > /home/buildbot/buildbot/build/mariadb-10.0.0/sql/log.cc:6564:40: > warning: ‘last_in_queue’ may be used uninitialized in this function > [-Wuninitialized] Thanks. I've pushed the fix to 10.0-base. > Looking at the code, this looks like a real potential bug. Yes (though likely dead code, as there's a DBUG_ASSERT before the if () asserting that it is always true). - Kristian.

1 0

[Maria-developers] review of MDEV-4011 per thread memory counting and usage
by Sergei Golubchik 18 Jan '13

18 Jan '13

Hi. Here it is: > === modified file 'client/completion_hash.cc' > --- client/completion_hash.cc 2011-06-30 15:46:53 +0000 > +++ client/completion_hash.cc 2013-01-15 21:00:33 +0000 > @@ -49,7 +49,7 @@ int completion_hash_init(HashTable *ht, > ht->initialized = 0; > return FAILURE; > } > - init_alloc_root(&ht->mem_root, 8192, 0); > + init_alloc_root(&ht->mem_root, 8192, 0, 0); very often you write 0 or MY_THREAD_SPECIFIC instead of MYF(0) and MYF(MY_THREAD_SPECIFIC) > ht->pHashFunction = hashpjw; > ht->nTableSize = nSize; > ht->initialized = 1; > === modified file 'include/my_sys.h' > --- include/my_sys.h 2012-12-23 21:37:11 +0000 > +++ include/my_sys.h 2013-01-15 21:00:33 +0000 > @@ -86,6 +86,8 @@ typedef struct my_aio_result { > #define MY_SYNC 4096 /* my_copy(): sync dst file */ > #define MY_SYNC_DIR 32768 /* my_create/delete/rename: sync directory */ > #define MY_SYNC_FILESIZE 65536 /* my_sync(): safe sync when file is extended */ > +#define MY_THREAD_SPECIFIC 0x10000 /* my_malloc(): thread specific */ > +#define MY_THREAD_MOVE 0x20000 /* realloc(); Memory can move */ may be, better to remove this, as it's unused? it's easier to add it later (just a couple of lines of code), than to keep the dead code around. > #define MY_CHECK_ERROR 1 /* Params to my_end; Check open-close */ > #define MY_GIVE_INFO 2 /* Give time info about process*/ > @@ -148,6 +150,42 @@ typedef struct my_aio_result { > /* Extra length needed for filename if one calls my_create_backup_name */ > #define MY_BACKUP_NAME_EXTRA_LENGTH 17 > > +/* If we have our own safemalloc (for debugging) */ > +#if defined(SAFEMALLOC) > +#define MALLOC_SIZE_AND_FLAG(p,b) sf_malloc_usable_size(p,b) > +#define MALLOC_PREFIX_SIZE 0 > +#define MALLOC_STORE_SIZE(a,b,c,d) > +#define MALLOC_FIX_POINTER_FOR_FREE(a) a > +#define SAFEMALLOC_REPORT_MEMORY(X) sf_report_leaked_memory(X) > +void sf_report_leaked_memory(my_thread_id id); > +extern my_thread_id (*sf_malloc_dbug_id)(void); > +#else > +/* > + * We use double as prefix size as this guarantees the correct > + * alignment on all platforms and will optimize things for > + * memcpy(), memcmp() etc. > + */ > +#define MALLOC_PREFIX_SIZE (sizeof(double)) > +#define MALLOC_SIZE(p) (*(size_t*) ((char*)(p) - MALLOC_PREFIX_SIZE)) > +#define MALLOC_STORE_SIZE(p, type_of_p, size, flag) \ > +{\ > + *(size_t*) p= (size) | (flag); \ > + (p)= (type_of_p) (((char*) (p)) + MALLOC_PREFIX_SIZE); \ > +} > +static inline size_t malloc_size_and_flag(void *p, myf *flags) > +{ > + size_t size= MALLOC_SIZE(p); > + *flags= (size & 1); > + return size & ~ (ulonglong) 1; > +} > +#define MALLOC_SIZE_AND_FLAG(p,b) malloc_size_and_flag(p, b); > +#define MALLOC_FIX_POINTER_FOR_FREE(p) (((char*) (p)) - MALLOC_PREFIX_SIZE) > +#define SAFEMALLOC_REPORT_MEMORY(X) do {} while(0) > +#endif /* SAFEMALLOC */ > + > +typedef void (*MALLOC_SIZE_CB) (long long size, myf my_flags); > +extern void set_malloc_size_cb(MALLOC_SIZE_CB func); I don't think you need many of these defines here. my_sys.h is the interface header. MALLOC_STORE_SIZE and MALLOC_FIX_POINTER_FOR_FREE (for example, I didn't check the others) are only used in my_malloc.c - they should not be used elsewhere, and thus, I think, they should be defined in my_malloc.c And, frankly speaking, I wouldn't bother doing two different ways of storing the size and the flags. my_malloc can always allocate one sizeof(double) more and store its data there. Who cares that in the debug builds safemalloc duplicates that... > /* defines when allocating data */ > extern void *my_malloc(size_t Size,myf MyFlags); > extern void *my_multi_malloc(myf MyFlags, ...); > @@ -323,6 +361,7 @@ typedef struct st_dynamic_array > uint elements,max_element; > uint alloc_increment; > uint size_of_element; > + myf malloc_flags; hm. you've preserved the MEM_ROOT structure for compatibiity reasons, but you've changed DYNAMIC_ARRAY. Why? I'd expect you either to change everything or to change nothing. Besides, what's the point of preserving MEM_ROOT, if you change prototypes of functions that work with it? > } DYNAMIC_ARRAY; > > typedef struct st_my_tmpdir > @@ -768,16 +807,13 @@ extern my_bool real_open_cached_file(IO_ > extern void close_cached_file(IO_CACHE *cache); > File create_temp_file(char *to, const char *dir, const char *pfx, > int mode, myf MyFlags); > -#define my_init_dynamic_array(A,B,C,D) init_dynamic_array2(A,B,NULL,C,D) > -#define my_init_dynamic_array_ci(A,B,C,D) init_dynamic_array2(A,B,NULL,C,D) > -#define my_init_dynamic_array2(A,B,C,D,E) init_dynamic_array2(A,B,C,D,E) > -#define my_init_dynamic_array2_ci(A,B,C,D,E) init_dynamic_array2(A,B,C,D,E) > +#define my_init_dynamic_array(A,B,C,D,E) init_dynamic_array2(A,B,NULL,C,D,E) > +#define my_init_dynamic_array_ci(A,B,C,D,E) init_dynamic_array2(A,B,NULL,C,D,E) > +#define my_init_dynamic_array2(A,B,C,D,E,F) init_dynamic_array2(A,B,C,D,E,F) > +#define my_init_dynamic_array2_ci(A,B,C,D,E,F) init_dynamic_array2(A,B,C,D,E,F) as you're changing function prototypes anyway, you can remove the _ci functions and all other "preserve the ABI" garbage, > extern my_bool init_dynamic_array2(DYNAMIC_ARRAY *array, uint element_size, > void *init_buffer, uint init_alloc, > - uint alloc_increment); > -/* init_dynamic_array() function is deprecated */ > -extern my_bool init_dynamic_array(DYNAMIC_ARRAY *array, uint element_size, > - uint init_alloc, uint alloc_increment); > + uint alloc_increment, myf my_flags); > extern my_bool insert_dynamic(DYNAMIC_ARRAY *array, const uchar * element); > extern uchar *alloc_dynamic(DYNAMIC_ARRAY *array); > extern uchar *pop_dynamic(DYNAMIC_ARRAY*); > @@ -829,7 +865,8 @@ extern void my_free_lock(void *ptr); > #define ALLOC_ROOT_MIN_BLOCK_SIZE (MALLOC_OVERHEAD + sizeof(USED_MEM) + 8) > #define clear_alloc_root(A) do { (A)->free= (A)->used= (A)->pre_alloc= 0; (A)->min_malloc=0;} while(0) > extern void init_alloc_root(MEM_ROOT *mem_root, size_t block_size, > - size_t pre_alloc_size); > + size_t pre_alloc_size, myf my_flags); > +extern void alloc_root_set_min_malloc(MEM_ROOT *mem_root, size_t min_malloc); if you'd store the flag in the lowest bit of the block_size or if you wouldn't try to keep the old MEM_ROOT structure, this function wouldn't be needed > extern void *alloc_root(MEM_ROOT *mem_root, size_t Size); > extern void *multi_alloc_root(MEM_ROOT *mem_root, ...); > extern void free_root(MEM_ROOT *root, myf MyFLAGS); > === modified file 'include/my_tree.h' > --- include/my_tree.h 2011-11-03 18:17:05 +0000 > +++ include/my_tree.h 2013-01-15 21:00:33 +0000 > @@ -68,13 +68,15 @@ typedef struct st_tree { > MEM_ROOT mem_root; > my_bool with_delete; > tree_element_free free; > + myf malloc_flags; > uint flag; > } TREE; > > /* Functions on whole tree */ > void init_tree(TREE *tree, size_t default_alloc_size, size_t memory_limit, > int size, qsort_cmp2 compare, my_bool with_delete, > - tree_element_free free_element, void *custom_arg); > + tree_element_free free_element, void *custom_arg, > + myf my_flags_for_malloc); I'd just call it my_flags. it doesn't matter what tree is doing with them. and there may be other flags too. E.g. you could've removed 'with_delete' and added MY_TREE_WITH_DELETE > void delete_tree(TREE*); > void reset_tree(TREE*); > > === modified file 'include/mysql.h' > --- include/mysql.h 2012-11-20 14:24:39 +0000 > +++ include/mysql.h 2013-01-15 21:00:33 +0000 > @@ -167,7 +167,7 @@ enum mysql_option > MYSQL_OPT_GUESS_CONNECTION, MYSQL_SET_CLIENT_IP, MYSQL_SECURE_AUTH, > MYSQL_REPORT_DATA_TRUNCATION, MYSQL_OPT_RECONNECT, > MYSQL_OPT_SSL_VERIFY_SERVER_CERT, MYSQL_PLUGIN_DIR, MYSQL_DEFAULT_AUTH, > - MYSQL_PROGRESS_CALLBACK, > + MYSQL_PROGRESS_CALLBACK, MYSQL_THREAD_SPECIFIC_MALLOC, I don't think you need this on the client side. (the only use case is federated, and I don't think you need to extend the client API specifically for it, federated can set whatever it needs directliy). > /* MariaDB options */ > MYSQL_OPT_NONBLOCK=6000 > }; > === modified file 'mysys/array.c' > --- mysys/array.c 2012-01-13 14:50:02 +0000 > +++ mysys/array.c 2013-01-15 21:00:33 +0000 > @@ -62,17 +64,19 @@ my_bool init_dynamic_array2(DYNAMIC_ARRA > should not throw an error > */ > if (init_alloc && > - !(array->buffer= (uchar*) my_malloc(element_size*init_alloc, MYF(0)))) > + !(array->buffer= (uchar*) my_malloc(element_size*init_alloc, > + MYF(my_flags)))) > array->max_element=0; > DBUG_RETURN(FALSE); > } > > my_bool init_dynamic_array(DYNAMIC_ARRAY *array, uint element_size, > - uint init_alloc, uint alloc_increment) > + uint init_alloc, uint alloc_increment, > + myf my_flags) > { > /* placeholder to preserve ABI */ > return my_init_dynamic_array_ci(array, element_size, init_alloc, You can remove this, as you're changing the ABI anyway > - alloc_increment); > + alloc_increment, my_flags); > } > /* > Insert element at the end of array. Allocate memory if needed. > === modified file 'mysys/my_alloc.c' > --- mysys/my_alloc.c 2012-01-13 14:50:02 +0000 > +++ mysys/my_alloc.c 2013-01-15 21:00:33 +0000 > @@ -41,16 +48,20 @@ > Altough error can happen during execution of this function if > pre_alloc_size is non-0 it won't be reported. Instead it will be > reported as error in first alloc_root() on this memory root. > + > + We don't want to change the api for MEMROOT. > + Becasue of this, we store in min_malloc of my_thread_specific is set. > */ > > void init_alloc_root(MEM_ROOT *mem_root, size_t block_size, > - size_t pre_alloc_size __attribute__((unused))) > + size_t pre_alloc_size __attribute__((unused)), > + myf my_flags) > { > DBUG_ENTER("init_alloc_root"); > DBUG_PRINT("enter",("root: 0x%lx", (long) mem_root)); > > mem_root->free= mem_root->used= mem_root->pre_alloc= 0; > - mem_root->min_malloc= 32; > + mem_root->min_malloc= 32 | (test(my_flags & MY_THREAD_SPECIFIC) << 16); it'd be safer to put it in the lowest bit. min_malloc or - particularly - block size, are always even numbers, the lowest bit is always 0. or to add a new field to the MEM_ROOT, as compatibility is already broken anyway. > mem_root->block_size= block_size - ALLOC_ROOT_MIN_BLOCK_SIZE; > mem_root->error_handler= 0; > mem_root->block_num= 4; /* We shift this with >>2 */ > === modified file 'mysys/my_malloc.c' > --- mysys/my_malloc.c 2012-01-13 14:50:02 +0000 > +++ mysys/my_malloc.c 2013-01-15 21:00:33 +0000 > @@ -18,6 +18,31 @@ > #include "mysys_err.h" > #include <m_string.h> > > +MALLOC_SIZE_CB malloc_size_cb_func= NULL; > + > +/** > + Inform appliction that memory usage has changed application > + > + @param size Size of memory segment allocated or freed > + @param flag 1 if thread specific (allocated by MY_THREAD_SPECIFIC), > + 0 if system specific. > + > + The type os size is long long, to be able to handle negative numbers to > + decrement the memory usage > +*/ > + > +static void update_malloc_size(long long size, myf my_flags) > +{ > + if (malloc_size_cb_func) > + malloc_size_cb_func(size, my_flags); > +} > + > +void set_malloc_size_cb(MALLOC_SIZE_CB func) > +{ > + malloc_size_cb_func= func; > +} > + > + > /** > Allocate a sized block of memory. > > === modified file 'mysys/mysys_priv.h' > --- mysys/mysys_priv.h 2011-12-12 21:58:24 +0000 > +++ mysys/mysys_priv.h 2013-01-15 21:00:33 +0000 > @@ -70,12 +70,13 @@ extern PSI_file_key key_file_charset, ke > #endif /* HAVE_PSI_INTERFACE */ > > #ifdef SAFEMALLOC > -void *sf_malloc(size_t size); > -void *sf_realloc(void *ptr, size_t size); > +void *sf_malloc(size_t size, myf my_flags); > +void *sf_realloc(void *ptr, size_t size, myf my_flags); > void sf_free(void *ptr); > +size_t sf_malloc_usable_size(void *ptr, myf *my_flags); this is used nowhere. dead code again. > #else > -#define sf_malloc(X) malloc(X) > -#define sf_realloc(X,Y) realloc(X,Y) > +#define sf_malloc(X,Y) malloc(X) > +#define sf_realloc(X,Y,Z) realloc(X,Y) > #define sf_free(X) free(X) > #endif > > === modified file 'mysys/safemalloc.c' > --- mysys/safemalloc.c 2012-04-18 01:29:26 +0000 > +++ mysys/safemalloc.c 2013-01-15 21:00:33 +0000 > @@ -79,11 +81,21 @@ static int bad_ptr(const char *where, vo > static void free_memory(void *ptr); > static void sf_terminate(); > > +/* Setup default call to get a thread id for the memory */ > + > +my_thread_id default_sf_malloc_dbug_id(void) > +{ > + return my_thread_dbug_id(); > +} > + > +my_thread_id (*sf_malloc_dbug_id)(void)= default_sf_malloc_dbug_id; Why do you need different functions for getting the thread id? > + > + > /** > allocates memory > */ > > -void *sf_malloc(size_t size) > +void *sf_malloc(size_t size, myf my_flags) > { > struct st_irem *irem; > uchar *data; > === modified file 'sql/event_scheduler.cc' > --- sql/event_scheduler.cc 2012-09-27 23:06:56 +0000 > +++ sql/event_scheduler.cc 2013-01-15 21:00:33 +0000 > @@ -182,12 +180,15 @@ deinit_event_thread(THD *thd) > void > pre_init_event_thread(THD* thd) > { > + THD *orig_thd= current_thd; > DBUG_ENTER("pre_init_event_thread"); > + > + my_pthread_setspecific_ptr(THR_THD, thd); I'd rather see set_current_thd(thd) here: #define set_current_thd(X) my_pthread_setspecific_ptr(THR_THD, (X)) > thd->client_capabilities= 0; > thd->security_ctx->master_access= 0; > thd->security_ctx->db_access= 0; > thd->security_ctx->host_or_ip= (char*)my_localhost; > - my_net_init(&thd->net, NULL); > + my_net_init(&thd->net, NULL, MY_THREAD_SPECIFIC); > thd->security_ctx->set_user((char*)"event_scheduler"); > thd->net.read_timeout= slave_net_timeout; > thd->variables.option_bits|= OPTION_AUTO_IS_NULL; > === modified file 'sql/handler.cc' > --- sql/handler.cc 2012-12-17 00:49:19 +0000 > +++ sql/handler.cc 2013-01-15 21:00:33 +0000 > @@ -5395,6 +5395,7 @@ fl_log_iterator_buffer_init(struct handl > iterator->buffer= buff; > iterator->next= &fl_log_iterator_next; > iterator->destroy= &fl_log_iterator_destroy; > + my_dir_end(dirp); Why did we never see memory leaks because of that? > return HA_ITERATOR_OK; > } > > === modified file 'sql/mysqld.cc' > --- sql/mysqld.cc 2013-01-09 03:34:33 +0000 > +++ sql/mysqld.cc 2013-01-15 21:00:33 +0000 > @@ -2606,9 +2607,10 @@ bool one_thread_per_connection_end(THD * > > /* It's safe to broadcast outside a lock (COND... is not deleted here) */ > DBUG_PRINT("signal", ("Broadcasting COND_thread_count")); > + mysql_cond_broadcast(&COND_thread_count); > + > DBUG_LEAVE; // Must match DBUG_ENTER() > my_thread_end(); > - mysql_cond_broadcast(&COND_thread_count); why? > > pthread_exit(0); > return 0; // Avoid compiler warnings > @@ -7024,6 +7096,7 @@ SHOW_VAR status_vars[]= { > {"Key", (char*) &show_default_keycache, SHOW_FUNC}, > {"Last_query_cost", (char*) offsetof(STATUS_VAR, last_query_cost), SHOW_DOUBLE_STATUS}, > {"Max_used_connections", (char*) &max_used_connections, SHOW_LONG}, > + {"Memory_used", (char*) &total_memory_used, SHOW_LONGLONG}, May be this should rather be: SHOW SESSION STATUS -> thd->memory_used SHOW GLOBAL STATUS -> total_memory_used (which means that total_memory_used/memory_used should be in the status_var struct) > {"Not_flushed_delayed_rows", (char*) &delayed_rows_in_use, SHOW_LONG_NOFLUSH}, > {"Open_files", (char*) &my_file_opened, SHOW_LONG_NOFLUSH}, > {"Open_streams", (char*) &my_stream_opened, SHOW_LONG_NOFLUSH}, > === modified file 'sql/net_serv.cc' > --- sql/net_serv.cc 2012-02-28 17:53:05 +0000 > +++ sql/net_serv.cc 2013-01-15 21:00:33 +0000 > @@ -139,6 +140,7 @@ my_bool my_net_init(NET *net, Vio* vio) > net->net_skip_rest_factor= 0; > net->last_errno=0; > net->unused= 0; > + net->thread_specific_malloc= test(my_flags & MY_THREAD_SPECIFIC); why do you need that in NET ? > > if (vio != 0) /* If real connection */ > { > === modified file 'sql/sql_analyse.h' > --- sql/sql_analyse.h 2011-11-03 18:17:05 +0000 > +++ sql/sql_analyse.h 2013-01-15 21:00:33 +0000 > @@ -121,7 +121,8 @@ class field_str :public field_info > must_be_blob(0), was_zero_fill(0), > was_maybe_zerofill(0), can_be_still_num(1) > { init_tree(&tree, 0, 0, sizeof(String), (qsort_cmp2) sortcmp2, > - 0, (tree_element_free) free_string, NULL); }; > + 0, (tree_element_free) free_string, NULL, > + MY_THREAD_SPECIFIC); }; Now, when I'm thinking of it, I got the idea that it's better to have MY_THREAD_SPECIFIC as a default, and add MY_GLOBAL_ALLOC instead to mark global allocations explicitly. It's more future-proof. When one copies init_dynamic_array() from somewhere and zeros out unknown or unneeded parameters, the memory will be per-thread by default. And if that's wrong, the test suite will find it, you have an assert for this. To mark memory global as an explicit action, one needs to think about it. Currently, the memory is global by default, and if one doesn't think,# copy-pastes, and leaves new allocation global, it will not be noticed, the memory will simply be unaccounted for. > > void add(); > void get_opt_type(String*, ha_rows); > === modified file 'sql/sql_error.cc' > --- sql/sql_error.cc 2012-01-13 14:50:02 +0000 > +++ sql/sql_error.cc 2013-01-15 21:00:33 +0000 > @@ -457,25 +457,37 @@ Diagnostics_area::disable_status() > m_status= DA_DISABLED; > } > > -Warning_info::Warning_info(ulonglong warn_id_arg, bool allow_unlimited_warnings) > +Warning_info::Warning_info(ulonglong warn_id_arg, > + bool allow_unlimited_warnings, bool initialize) I'd rather add a special constructor for Warning_info that THD would use. Because there's only one place where you need initialize=false, and all other "normal" uses, should not suffer (be changed) because of that. > :m_statement_warn_count(0), > m_current_row_for_warning(1), > m_warn_id(warn_id_arg), > m_allow_unlimited_warnings(allow_unlimited_warnings), > m_read_only(FALSE) > { > - /* Initialize sub structures */ > - init_sql_alloc(&m_warn_root, WARN_ALLOC_BLOCK_SIZE, WARN_ALLOC_PREALLOC_SIZE); > m_warn_list.empty(); > bzero((char*) m_warn_count, sizeof(m_warn_count)); > + if (initialize) > + init(); > } > Regards, Sergei

2 2

Re: [Maria-developers] LP:1024058 - mysqld XA crash in replication slave
by Kristian Nielsen 18 Jan '13

18 Jan '13

"Sergei Golubchik (JIRA)" <jira(a)mariadb.atlassian.net> writes: > [ https://mariadb.atlassian.net/browse/MDEV-633?page=com.atlassian.jira.plugi… ] [Taking this to maria-developers@, which is better for discussions like this] > The problem here is very simple to explain. The server can use either > mmap-based transaction coordinator for 2PC or a binary log. 2PC always uses > binary log, if binary logging is enabled. But even if it is enabled > globally, it is usually disabled in the replication slave thread unless > --log-slave-updates is specified. > > One would probably get the same crash without replication, if one disables > binary log manually with SET SQL_LOG_BIN=0; > > Possible fixes: > * auto-enable binary log for 2PC transactions (bad, binlog will contain > unwanted DDL and DML events). > * abort 2PC transactions if binary log is locally disabled for this thread > (worse, too easy to break the replication) > * write Xid events to binlog even if binary log is locally disabled (best?) > > The last approach seems to be preferable. But in the future if we'll start > recovering transactions from the binary log (doing only one sync per 2PC > transaction), we'll have this problem again, because then we'll need the > actual changes to be logged, not just the Xid. > > [~knielsen] - opinion? First, I do not think SQL_LOG_BIN=0 or binlog enabled without --log-slave-updates are common or very useful. And multi-engine transactions are problematic in several ways. But we should not crash, of course. I would find it acceptable to simply call each engine's commit method separately, without 2PC, especially in stable 5.5 and below. Note that we now have @@skip_replication, which can be used to log a transaction to binlog without replicating it, similar to @@sql_log_bin but maintaining 2PC and recovery. If we really want, we could fix it 10.0 so that 2PC works even for binlog disabled. As you say, probably writing the Xid to the binlog without any associated DML events would be a reasonable way to do this. I would suggest logging a separate new event to not confuse things with XID_LOG_EVENT, which implies a commit of a prior transactions (there are facilities in 10.0 to rewrite such event to a dummy event so we will not break old slaves). With respect to a future recovering transactions from the binary log, it is actually in any case very hard to recover a multi-engine transaction in this way. The problem is when one engine is missing the transaction but the other engine has it, and the event is logged with statement-based replication. But we could simply not do this optimisation for multi-engine transactions, which are really not very commonly used (for multi-engine transactions we would keep the fsync() inside prepare()). Then things should work, for the normal case as well as for binlog locally disabled. So in summary, my immediate opinion would be: - In 5.5, commit each engine individually without 2PC when binlog is locally disabled. - In 10.0, optionally binlog just a new XID_RECOVERY_LOG_EVENT and use it in recovery if binlog is locally disabled. Hope this helps, - Kristian.

1 0

Re: [Maria-developers] Structure of Binary Log
by Kristian Nielsen 16 Jan '13

16 Jan '13

Nuno Rebocho <rebocho.nmg(a)portugalmail.pt> writes: > The attractive hypothesis, develop a data recovery tool, never left my > mind. I've been trying to learn together as a "Maria developers" team > can get information about the structure of the log file from MySql to > navigate this and obtain information needed for recovery. I am not aware of any definitive manual for the MySQL/MariaDB binary log, however there are various resources out there. For example, I wrote two articles that give an overview of the events in the log: http://kristiannielsen.livejournal.com/13382.html http://kristiannielsen.livejournal.com/13253.html My articles on group commit on the same blog would give you lots on information on how the binary log is written. Google can probably find other articles by other authors that might be useful. Otherwise, the source code in the definite source of information. The binary log code is in sql/log.cc. The code for creating and parsing events is in sql/log_event.cc. I noticed you are now on the maria-developers@ mailing list, this will be a good place to ask concrete questions about the binary log. Good luck with your project! - Kristian.

1 0

Re: [Maria-developers] 答复: in-order commit
by Kristian Nielsen 16 Jan '13

16 Jan '13

丁奇 <dingqi.lxb(a)taobao.com> writes: > Hi, Kristian > Ok. I have got the information from JIRA. > > I find you control the commit order inside the user thread. > > Will it be easier to let Trans_worker thread hold this logic? Yes, I think you are right. Of course, the user thread is the one that knows the ordering, but the logic for waiting needs to be in the Trans_worker thread. In fact this is a bug in my first patch: Transaction T3 could wait for the THD of worker thread 1 which has both T1 and T2 queued; then it will wake up too early, when T1 commits rather than when T2 does. I will try to implement the new idea today. > After they have done the execution of one transaction, "register the transaction and wait" if there are transactions from other workers should be commited ahead. > After commit in one worker, wake up another worker, the worker who wait for the next "head of commitee" should be woken up. Right, I'll need to look into this a bit deeper. Actually, in my patch the actual wait and wakeup happens inside ha_commit_trans(), and there is a reason for this. Because eventually I want to do it inside tc_log->log_and_order(), which is called from ha_commit_trans(). Here is how a commit happens: InnoDB prepare step fsync() InnoDB redo log (*A) TC_LOG_BINLOG::log_and_order Write transaction to binlog fsync() binlog (*B) InnoDB commit_ordered() (*C) Write commit record to InnoDB redo log InnoDB commit step The steps (*A) and (*B) are slow, typically around 1-10 milliseconds depending on disk system. So we need many threads to commit in parallel and reach points (*A) and (*B) at the same time, so we only need to do the fsync() once for many threads. This is group commit. Thus for in-order parallel replication, we must not do the wait for the previous commit before the (*B) step. Because if we do, then it becomes impossible for two transactions to be at point (*B) at the same time, and group commit is impossible. On the other hand, point (*C) is where the commit order is determined. So if we do the wait after point (*C), then we cannot enforce that T1 commits before T2. So therefore, the wait must happen exactly around point (B) and (C), inside TC_LOG_BINLOG::log_and_order(). That is why I invented all the register_wait_for_prior_commit() and so on: so that log_and_order() has somewhere to look for exactly who is waiting for who. Then if T2 is waiting for T1 to commit, we can do steps (*B) and (*C) for both of them together, achiving both group commit and in-order parallel replication. Anyway, I just wanted to mention this, I know it will be difficult to understand fully from just this description. This is something that I have been planning to have for years, but I still need to show some real code that actually works. If I manage that, hopefully things will be clearer. (If not - then I need to think again ;-) Thanks, - Kristian.

1 2

[Maria-developers] new timestamp fields
by Zardosht Kasheff 15 Jan '13

15 Jan '13

Hello all, In MySQL/MariaDB 5.1, when a timestamp field was used as a key, we used an integer compare to compare the fields. We assumed all integer fields (including date, time, and timestamp) could use an integer comparison to compare the fields, and that they were either 1, 2, 3, 4 or 8 bytes long. We now see in MariaDB 5.3 and beyond that changes have been made to the timestamp field, such that the length of the field may be something other than 1, 2, 3, 4,or 8 bytes. Given a timestamp field of N bytes, how are we meant to compare them? Are some meant to be integer comparisons? Are others supposed to be memory comparisons? Thanks -Zardosht

2 5

[Maria-developers] MDEV-520: consider parallel replication patch from taobao patches
by Kristian Nielsen 14 Jan '13

14 Jan '13

Hi xiaobin, I read through your patch found here: http://mysql.taobao.org/index.php/RLPR_for_MariaDB I am still a bit confused about the status of this patch. It would be useful if you could explain a bit about the status of the patch: - How have you tested the patch. - How are you currently using the patch? Or is it still just development/test? - What are your current plans for the patch? (You mentioned that you have some changes you are implementing at the moment). Anyway, I think I have a fair understanding of the overall idea, which seems very interesting! I will try to summarise here in a bit more detail, as well as give a few general comments on the patch. Please correct me if I misunderstood anything. ---- The basic idea is to build a set (implemented as a hash table) of every unique key in every row-event in each transaction received from the master. If two transactions have no overlapping keys, they can be applied in parallel (foreign_key_checks is set to OFF to make this work). There is a single thread that processes each event received in sequence. It checks the key set of the new transaction against the key sets of all transactions queued for each of N worker threads: - If there is no conflict with any already queued transaction, the new transaction can be queued to any worker thread (the one with fewest queued transaction is chosen). - If there is a conflict with just one thread, then the transaction can be queued for that thread - it will then execute in correct sequence with the conflicting transactions. - If there is a conflict with multiple threads, wait until the conflicting transactions have been processed. No new transactions are queued while waiting. Then there are N (default 16) threads that apply the queued transactions. Queued transactions are processsed in parallel within one thread, but transaction in different queues are processed in parallel with each other. If a transaction is received which is not using row events (statement-based replication or DDL), then we wait until all queued transactions are processed. Only after that is the new transaction queued, so that such events are processed correctly (but not in parallel, of course). Likewise, if the worker threads get too far behind, we wait with queuing any more transactions. ---- I think this is a very interesting approach, and I am happy that someone have actually tried implementing it. The biggest potential issue seems to be the performance of the thread that checks the transactions for conflicts and queues them for the appropriate thread: - Main issue will be huge transactions, if the size of hash tables of key values become too big for available memory. Maybe this could be solved by having some limit above which transactions are not tried in parallel. - There will be a lot of hash lookups of key values, all done in a single thread. This could become a limit I suppose - testing will be needed to understand this fully. But I think there will be many workloads where this will be useful. Overall, it seems to me this technique is very promising. It seems to be better at getting maximum parallelism on the slave than any of the other approaches that have been considered. As to the patch itself, it seems a bit immature in the present form, but I understood you are still working on it. Let us discuss what the plans could be for finishing it and get it ready for inclusion in MariaDB. The main criteria is that it must be reasonably complete - it must not crash or silently corrupt data in any usage. However, reasonable restrictions for what is supported is fine. (For example, we could say that LOAD DATA is not supported and give an error, but not crash or silently ignore such statement). But we can discuss this in more details. ---- Finally, a couple general comments on the code: 1. get_pk_value() does not seem to handle collations, eg. case insensitivity? It needs to understand that eg. "abab" and "ABAB" can conflict. I think this should be solved by copying in values with non-binary collations using the appropriate strxfrm() variant. 2. remote_query() retries the query in case of deadlock - but that seems wrrong, doesn't it need to retry the whole _transaction_, not just the query? 3. What about other events like User_var_event ? Will they be included in a later patch? 4. There is a my_sleep(10000) in the code when a transaction conflicts with two or more workers. It would be better to replace this with a condition variable so we can wake up as soon as the conflict is resolved. Same for some other sleeps like in wait_for_all_dml_done(). 5. Shouldn't there be some locking in dispatch_transaction()? To protect the hash lookups when checking conflicts against worker threads doing pk_hash_minus() ? 6. get_pk_value(). There seems to be a buffer overflow - st_hash_item::key is fixed size, but there seems no check against overrunning it. One solution is to just truncate if data does not fit - this can lead to false positives where potential parallelism is not exploited, but that is still safe and should be ok. Thanks! - Kristian.

5 21

Re: [Maria-developers] MDEV-318 IF (NOT) EXIST clauses for ALTER TABLE
by Sergei Golubchik 13 Jan '13

13 Jan '13

Hi, Holyfoot! On Jul 08, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/mdev-318/ > > ------------------------------------------------------------ > revno: 3552 > revision-id: holyfoot(a)askmonty.org-20120708152354-d7iu7yoqo3cohidz > parent: sanja(a)montyprogram.com-20120626184334-ptpg39ptq3t79dg5 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev-318 > timestamp: Sun 2012-07-08 20:23:54 +0500 > message: > MDEV-318 IF (NOT) EXIST clauses for ALTER TABLE (MWL #252). > ALTER TABLE syntax modified and related implementations added to allow: > ALTER TABLE ADD/DROP COLUMN > ALTER TABLE ADD/DROP INDEX > ALTER TABLE ADD/DROP FOREIGN KEY > ALTER TABLE ADD/DROP PARTITION > ALTER TABLE CHANGE COLUMN > ALTER TABLE MODIFY COLUMN Please, find my comments below: > === modified file 'mysql-test/r/alter_table.result' > --- mysql-test/r/alter_table.result 2011-11-21 17:13:14 +0000 > +++ mysql-test/r/alter_table.result 2012-07-11 14:24:32 +0000 > @@ -1282,3 +1282,30 @@ > # Clean-up. > drop table t1; > End of 5.1 tests > +CREATE TABLE t1 ( > +id INT(11) NOT NULL, > +x_param INT(11) DEFAULT NULL, > +PRIMARY KEY (id), > +KEY x_param (x_param) > +); > +ALTER TABLE t1 ADD COLUMN IF NOT EXISTS id INT, > +ADD COLUMN IF NOT EXISTS lol INT AFTER id; > +Warnings: > +Note 1060 Duplicate column name 'id' > +ALTER TABLE t1 ADD COLUMN IF NOT EXISTS lol INT AFTER id; > +Warnings: > +Note 1060 Duplicate column name 'lol' > +ALTER TABLE t1 DROP COLUMN IF EXISTS lol; > +ALTER TABLE t1 DROP COLUMN IF EXISTS lol; > +Warnings: > +Note 1091 Can't DROP 'lol'; check that column/key exists > +ALTER TABLE t1 ADD KEY IF NOT EXISTS x_param(x_param); > +Warnings: > +Note 1061 Duplicate key name 'x_param' > +ALTER TABLE t1 ADD KEY IF NOT EXISTS x_param(x_param); typo? you test ADD KEY IF NOT EXISTS twice. > +Warnings: > +Note 1061 Duplicate key name 'x_param' > +ALTER TABLE t1 MODIFY IF EXISTS lol INT; > +Warnings: > +Note 1054 Unknown column 'lol' in 't1' > +DROP TABLE t1; > === modified file 'sql/field.cc' > --- sql/field.cc 2012-03-13 14:38:43 +0000 > +++ sql/field.cc 2012-07-11 14:28:40 +0000 > @@ -9182,6 +9182,7 @@ void Create_field::init_for_tmp_table(en First: please, see the last section in http://kb.askmonty.org/en/how-to-get-more-out-of-bzr-when-working-on-mariad… and do as it suggests I had to apply your patch locally and re-run bzr diff to get the function names in the diff :( > (maybe_null ? FIELDFLAG_MAYBE_NULL : 0) | > (is_unsigned ? 0 : FIELDFLAG_DECIMAL)); > vcol_info= 0; > + create_if_not_exists= FALSE; > stored_in_db= TRUE; > } > > === modified file 'sql/sql_lex.h' > --- sql/sql_lex.h 2011-12-11 16:39:33 +0000 > +++ sql/sql_lex.h 2012-07-11 14:28:40 +0000 > @@ -1816,6 +1816,7 @@ typedef struct st_lex : public Query_tab > uint16 create_view_algorithm; > uint8 create_view_check; > bool drop_if_exists, drop_temporary, local_file, one_shot_set; > + bool check_exists; /* Enabled if the IF [NOT] EXISTS specified for ALTER */ I think, you can remove drop_if_exists and use check_exists instead > bool autocommit; > bool verbose, no_write_to_binlog; > > === modified file 'sql/sql_yacc.yy' > --- sql/sql_yacc.yy 2012-06-10 09:53:06 +0000 > +++ sql/sql_yacc.yy 2012-07-11 14:28:40 +0000 > @@ -4656,8 +4659,16 @@ table_option: > ; > > opt_if_not_exists: > - /* empty */ { $$= 0; } > - | IF not EXISTS { $$=HA_LEX_CREATE_IF_NOT_EXISTS; } > + /* empty */ > + { > + Lex->check_exists= FALSE; Let's initialize check_exists either before the grammar is parsed (like in the add_create_index_prepare and in the create rule) of in the grammar (here, in the opt_if_not_exists). But don't do it twice. > + $$= 0; > + } > + | IF not EXISTS > + { > + Lex->check_exists= TRUE; > + $$=HA_LEX_CREATE_IF_NOT_EXISTS; > + } > ; > > opt_create_table_options: > @@ -5879,6 +5891,18 @@ opt_ident: > | field_ident { $$=$1.str; } > ; > > +opt_if_not_exists_ident: > + opt_if_not_exists opt_ident > + { > + LEX *lex= Lex; > + if (lex->check_exists && lex->sql_command != SQLCOM_ALTER_TABLE) > + { > + my_parse_error(ER(ER_SYNTAX_ERROR)); > + MYSQL_YYABORT; > + } > + $$= $2; > + }; > + why did you create opt_if_not_exists_ident rule, instead of adding opt_if_not_exists before ident in the alter table grammar? > opt_component: > /* empty */ { $$= null_lex_str; } > | '.' ident { $$= $2; } > @@ -6455,6 +6482,11 @@ opt_column: > | COLUMN_SYM {} > ; > > +opt_column_if_exists: > + opt_column if_exists > + {} > + ; > + same as above - why a special rule? > opt_ignore: > /* empty */ { Lex->ignore= 0;} > | IGNORE_SYM { Lex->ignore= 1;} > @@ -10096,8 +10128,16 @@ table_alias_ref: > ; > > if_exists: please, rename this old rule to opt_if_exists > - /* empty */ { $$= 0; } > - | IF EXISTS { $$= 1; } > + /* empty */ > + { > + Lex->check_exists= FALSE; > + $$= 0; > + } > + | IF EXISTS > + { > + Lex->check_exists= TRUE; > + $$= 1; > + } > ; > > opt_temporary: you didn't change CREATE INDEX and DROP INDEX, did you? and ALTER TABLE DROP PRIMARY KEY isn't covered either > === modified file 'sql/sql_table.cc' > --- sql/sql_table.cc 2012-04-05 21:07:18 +0000 > +++ sql/sql_table.cc 2012-07-11 14:28:40 +0000 > @@ -5788,6 +5788,226 @@ is_index_maintenance_unique (TABLE *tabl > > > /* > + Preparation for table creation > + > + SYNOPSIS > + handle_if_exists_option() Why did you do that in a separate function? You have basically duplicated all checks. normal alter table code checks for duplicate columns/key names and for non-existant columns/keys for DROP. this code is still in place after your patch. so, basically, you'll check for duplicates and non-existant names twice. what's the point of that? I'd expect you to take check_exists flag into account in the existing checking code and convert errors into warnings as appropriate. > + thd Thread object. > + table The altered table. > + alter_info List of columns and indexes to create > + > + DESCRIPTION > + Looks for the IF [NOT] EXISTS options, checks the states and remove items > + from the list if existing found. > + > + RETURN VALUES > + NONE > +*/ > + > +static void > +handle_if_exists_options(THD *thd, TABLE *table, Alter_info *alter_info) > +{ > + Field **f_ptr; Regards, Sergei

2 9

[Maria-developers] Order by speed optimization
by Matthew Lagoe 13 Jan '13

13 Jan '13

I was talking with a few people about a query such as select * from table order by X, Y, Z limit G; > > As the "ORDER BY" for me could change to any number of 12 values in any order I am not able to use a multicolumn index as such I was looking at how to optimize in such case where sorting is dynamic. On my dataset the query takes ~7 seconds for a full table scan but only .3 seconds when using the column index and returns ~80,000 records for the query (before the LIMIT) I was wondering if there is any reason that the order by syntax doesn't use the following logic, I was thinking of looking at the code but was told such a simple optimization must have a reason that its not already in the code, so I am asking here before I spend hours being stupid. This logic block will be based on the above query. 1. If index exists for column X and there is a limit on the query then use the index and sort the list, then return the first G values, continue past G until you hit a different value for X then the value that was at G 2. Use this smaller subset of data for then ordering the data based on Y 3. Order the data again based on Z 4. Return the sorted data like normal If there is a reason this optimization isn't in the code please someone let me know, if not I think I will take a look at adding it. -Botanic

2 2

[Maria-developers] Kubuntu 12.04.1 and MariaDB
by Giacomo Picchiarelli 11 Jan '13

11 Jan '13

Hi! I'm having troubles installing mariadb 10.0 in Kubuntu 12.04.1. When I try to install it, dependencies are getting enraged and apt (through synaptic or muon) suggests to remove kde-plasma-desktop. This happens when I try to install mariadb-server-core 10.0. Of course if I remove kde-plasma-desktop I can't use my kde desktop, because it depends on mysql packages. I was trying to install it for educational purpose, and try to build and make some tests. Do you have any suggestions? Thank you -- Giacomo Picchiarelli <gpicchiarelli(a)gmail.com>

3 3

[Maria-developers] External replication proceeding
by md＠rpzdesign.com 10 Jan '13

10 Jan '13

Monty: I received your email and am re-posting to the maria developers list. It is truly amazing that maria developers are HAPPY with replication as it stands right now. binary logs, Single SLAVE reading from individual masters, bla-bla-bla..... All the current replication code is like walking on the edge of a cliff, anything goes wrong and the replication process STOPS with an error. And a HUMAN BEING must wake up from their blissful sleep to get the process started again and figure out what happened. And I just LOVE having to: LOCK TABLES; RESET MASTER SET SLAVE OFF; MYSQLDUMP; COPY FILES TO SLAVE; SET MASTER TO 'sql_log_bin=45678945"; SET SLAVE ON; SECONDS BEHIND MASTER... WRITE A THESIS FOR MY PHD IN ROCKET SCIENCE Oh my GOD!!!!!! What an absolute nightmare of a design, it works, just not well. I have not made an in depth study of galera, but I refuse to make replication another research project, replication should be EASY, auto-healing, self analyzing, N-MASTER and with simple source code. I will open source the files that I use to change the MYSQLD daemon. The best way to debug MariaDB on Linux is to run the program in Netbeans 7.1.2. You can even give it command line parameters to work against a currently installed system with the data/conf files in place. Your trace idea is very nice, but there is nothing like STEPPING THROUGH the code in NetBeans/GDB in a nice IDE instead of cryptic command line GDB. Cheers, md On 1/2/2013 5:21 AM, Michael Widenius wrote: > Hi! > > A better place to send emails like this is to > maria-developers(a)lists.launchpad.net; That way you can get all > MariaDB developers to answer you questions. > >>>>>> "md" == md <md(a)rpzdesign.com> writes: > md> Hello: > md> Does anybody know the name(s) of the source files I could find the code > md> that processes the Table/Procedure Drop/Create/Update, Alter Table, etc > md> statements. > > md> I want to intercept these SQL statement calls in mysqld before > md> they get sent to the specific storage engine like Aria or Xtradb. > > sql_parse.cc proably contains the code you are looking for. > Another place to look is in the the current replication code, > ie the sql_repl.cc > > md> I am working on an asynchronous external replication daemon > md> that handles all of the data and structure synchronization > md> replication issues (N-master-master, Hot Insert Master additions, DDL > md> changes, etc) > > Sounds very intersting. > Is this a open source project ? > > md> This is one of the missing pieces of the puzzle for MariaDB. > md> (oh yeah, mysql does not exist to me anymore, only Aria/Xtradb) > > :) > > md> I need to have the mysqld daemon issue a signal that a structural change > md> has taken place and the external replication daemon will pick it up > md> and handle all the changes processing across ALL the masters (WITHOUT A > md> HUMAN BEING INVOLVED!!!!) > > md> I figured I would ask since most of you are fluent in the source > md> code and just know where everything is, you would know right away. > > That's ok, but best way to get quick responce is to ask on the > maria-developers list or at #maria channel on Freenode. > > md> I am using MariaDB 5.5.28a from source. > > On thing you can do is: > > ./BUILD/compile-pentium-debug-max > mysqld --debug > > And then start 'mysql' and execute a CREATE TABLE command. > > You can find then find a full trace of all executed functions in: > /tmp/mysqld.trace > > That's probably one of the best ways to learn how MariaDB works. > > See: > > https://kb.askmonty.org/en/creating-a-trace-file/ > > Regards, > Monty >

3 2

[Maria-developers] High Priest of Humility
by md＠rpzdesign.com 08 Jan '13

08 Jan '13

Arjen: I would not go so far as to call it "raving" or boasting about anything. It was a private email message to you only. Not broadcast publicly. For a MariaDB captain, that is pretty bold offering a negative opinion to a potential contributor. Negative comments like that are not the best support for any level of enthusiasm for openness, collaboration, and sharing. Hint: Stop giving hints. marco On 1/7/2013 5:05 PM, Arjen Lentz wrote: > Hi Marco > >> Nice to see your response. >> >> I will be providing the code that I use to modify the MariaDB core >> daemon soon. > I presume you'll put it up in a branch on Launchpad? > > >> You will be amazed how little I am changing the core code. > Hint: modesty/humility are a good traits for an OSS developer. Let other people be amazed and impressed when you deliver code - raving about yourself beforehand is less cool... > > >> Where is the open source for Galera? (could not find it readily) > At Codership, but also in the 5.5 Galera repository of MariaDB. > > > Regards, > Arjen. > >

1 0

[Maria-developers] Missing man/mysql_plugin.1 in tar ball
by Honza Horak 07 Jan '13

07 Jan '13

Hi all, I've noticed that man page for mysql_plugin utility (man/mysql_plugin.1) is not present in mariadb-5.5.28a.tar.gz while it is present in mysql-5.5.28.tar.gz. However, I failed to find the reason why in case there is one. Or is that a mistake? Does anybody know? Regards, Honza

2 1

Re: [Maria-developers] API Documentaion for Replication Features
by Lixun Peng 31 Dec '12

31 Dec '12

Hi Monty, On Tue, Dec 25, 2012 at 8:21 PM, Michael Widenius < michael.widenius(a)gmail.com> wrote: > > Hi! > > Sorry that I missed this email before. > <cut> > Lixun> I can try to implement it. Are you doing it? > > Please try to implemement it? > You can look at the keycache code how we have done this before! > > OK, I'm trying to do it. Thanks, Lixun -- Senior MySQL Developer @ Taobao.com Mobile Phone: +86 18658156856 (Hangzhou) Gtalk: penglixun(at)gmail.com Twitter: http://www.twitter.com/plinux Blog: http://www.penglixun.com

2 1

Re: [Maria-developers] Rev 3480: Post review changes (total (really))
by Sergei Golubchik 24 Dec '12

24 Dec '12

Hi, Oleksandr! On Dec 23, Oleksandr Byelkin wrote: > 22.12.2012 21:28, Sergei Golubchik пишет: > >> @@ -100,6 +101,8 @@ struct st_dynamic_column_value > >> > >> typedef struct st_dynamic_column_value DYNAMIC_COLUMN_VALUE; > >> > >> +/* old functions (deprecated) */ > > I usually prefer code to comments. Kind of a comment that a compiler > > can make use of. That means an assert(aaa == 0) instead of /* aaa > > should be 0 here */ and instead of comment above, I'd rather have > > __attribute__((deprecated)) > > __attribute__((deprecated)) is GCC, we also use MS compiler. So ifdef > is better until we get appropriate multi-platform macro. 1. __attribute__ is defined to be empty for other compilers (somewhere in my_global.h, I guess), so it won't break the compilation. 2. I did not mean to use it instead of the #ifdef, I mean to use it instead of a /* old functions (deprecated) */ comment. Because a compiler cannot read the comment. 3. And anyway, it was just a suggestion, you can keep your current code and not change anything. As you prefer. > everything else will be fixed. Thanks! Regards, Sergei

1 0

Re: [Maria-developers] Rev 3480: Post review changes (total (really))
by Sergei Golubchik 22 Dec '12

22 Dec '12

Hi, Sanja! See below. Just a couple of small API-related comments. And one JSON bug. On Dec 21, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-10.0-cassandra/ > > ------------------------------------------------------------ > revno: 3480 > revision-id: sanja(a)askmonty.org-20121221192211-93tc13j3a6c85qea > parent: sanja(a)montyprogram.com-20121119121604-5h5tu0zn11em0sb3 > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-cassandra > timestamp: Fri 2012-12-21 21:22:11 +0200 > message: > Post review changes (total (really)). > > === modified file 'include/ma_dyncol.h' > --- a/include/ma_dyncol.h 2012-09-28 12:27:16 +0000 > +++ b/include/ma_dyncol.h 2012-12-21 19:22:11 +0000 > @@ -38,12 +38,13 @@ > how the offset are stored. > */ > #define MAX_DYNAMIC_COLUMN_LENGTH 0X1FFFFFFFL > +#define MAX_DYNAMIC_COLUMN_LENGTH_NM 0XFFFFFFFFFL Hm. Two questions. 1. Where does the number come from? 2. You've introduced a constant, but you aren't using it anywhere. What's the point? > /* > Limits of implementation > */ > -#define MAX_NAME_LENGTH 255 > #define MAX_TOTAL_NAME_LENGTH 65535 > +#define MAX_NAME_LENGTH (MAX_TOTAL_NAME_LENGTH/4) > > /* NO and OK is the same used just to show semantics */ > #define ER_DYNCOL_NO ER_DYNCOL_OK > @@ -100,6 +101,8 @@ struct st_dynamic_column_value > > typedef struct st_dynamic_column_value DYNAMIC_COLUMN_VALUE; > > +/* old functions (deprecated) */ I usually prefer code to comments. Kind of a comment that a compiler can make use of. That means an assert(aaa == 0) instead of /* aaa should be 0 here */ and instead of comment above, I'd rather have __attribute__((deprecated)) > +#ifdef MADYNCOL_DEPRECATED > enum enum_dyncol_func_result > dynamic_column_create(DYNAMIC_COLUMN *str, > uint column_nr, DYNAMIC_COLUMN_VALUE *value); > @@ -133,73 +121,105 @@ dynamic_column_update_many(DYNAMIC_COLUM ... > enum enum_dyncol_func_result > -dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys); > +dynamic_column_get(DYNAMIC_COLUMN *org, uint column_nr, > + DYNAMIC_COLUMN_VALUE *store_it_here); > +#endif > > -/* List of not NULL columns */ > +/* new functions */ > enum enum_dyncol_func_result > -dynamic_column_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); > +mariadb_dyncol_create_many(DYNAMIC_COLUMN *str, > + uint column_count, > + uint *column_numbers, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool new_string); > enum enum_dyncol_func_result > -dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr); > +mariadb_dyncol_create_many_named(DYNAMIC_COLUMN *str, > + uint column_count, > + LEX_STRING *column_keys, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool new_string); > + > + > enum enum_dyncol_func_result > -dynamic_column_list_fmt(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array, my_bool string_keys); > +mariadb_dyncol_update_many(DYNAMIC_COLUMN *str, > + uint add_column_count, > + uint *column_keys, > + DYNAMIC_COLUMN_VALUE *values); > +enum enum_dyncol_func_result > +mariadb_dyncol_update_many_named(DYNAMIC_COLUMN *str, > + uint add_column_count, > + LEX_STRING *column_keys, > + DYNAMIC_COLUMN_VALUE *values); > + > + > +enum enum_dyncol_func_result > +mariadb_dyncol_exists(DYNAMIC_COLUMN *org, uint column_nr); > +enum enum_dyncol_func_result > +mariadb_dyncol_exists_named(DYNAMIC_COLUMN *str, LEX_STRING *name); > + > +/* List of not NULL columns */ > +enum enum_dyncol_func_result > +mariadb_dyncol_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); Sanja, let's be constistent. If mariadb_dyncol_list_named() returns an array of LEX_STRING's, than mariadb_dyncol_list should return an array of integers, not a dynamic array. Besides you don't use DYNAMIC_ARRAY *anywhere* else in the new (not deprecated) API. > +enum enum_dyncol_func_result > +mariadb_dyncol_list_named(DYNAMIC_COLUMN *str, uint *count, LEX_STRING **names); > > /* > if the column do not exists it is NULL > */ > enum enum_dyncol_func_result > -dynamic_column_get(DYNAMIC_COLUMN *org, uint column_nr, > +mariadb_dyncol_get(DYNAMIC_COLUMN *org, uint column_nr, > DYNAMIC_COLUMN_VALUE *store_it_here); > enum enum_dyncol_func_result > -dynamic_column_get_str(DYNAMIC_COLUMN *str, LEX_STRING *name, > - DYNAMIC_COLUMN_VALUE *store_it_here); > +mariadb_dyncol_get_named(DYNAMIC_COLUMN *str, LEX_STRING *name, > + DYNAMIC_COLUMN_VALUE *store_it_here); > > -my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str); > +my_bool mariadb_dyncol_has_names(DYNAMIC_COLUMN *str); > > enum enum_dyncol_func_result > -dynamic_column_check(DYNAMIC_COLUMN *str); > +mariadb_dyncol_check(DYNAMIC_COLUMN *str); > > enum enum_dyncol_func_result > -dynamic_column_json(DYNAMIC_COLUMN *str, DYNAMIC_STRING *json); > +mariadb_dyncol_json(DYNAMIC_COLUMN *str, DYNAMIC_STRING *json); > > #define dynamic_column_initialize(A) memset((A), 0, sizeof(*(A))) > #define dynamic_column_column_free(V) dynstr_free(V) > > /* conversion of values to 3 base types */ > enum enum_dyncol_func_result > -dynamic_column_val_str(DYNAMIC_STRING *str, DYNAMIC_COLUMN_VALUE *val, > +mariadb_dyncol_val_str(DYNAMIC_STRING *str, DYNAMIC_COLUMN_VALUE *val, > CHARSET_INFO *cs, my_bool quote); > enum enum_dyncol_func_result > -dynamic_column_val_long(longlong *ll, DYNAMIC_COLUMN_VALUE *val); > +mariadb_dyncol_val_long(longlong *ll, DYNAMIC_COLUMN_VALUE *val); > +enum enum_dyncol_func_result > +mariadb_dyncol_val_double(double *dbl, DYNAMIC_COLUMN_VALUE *val); > + > + > enum enum_dyncol_func_result > -dynamic_column_val_double(double *dbl, DYNAMIC_COLUMN_VALUE *val); > +mariadb_dyncol_unpack(DYNAMIC_COLUMN *str, > + uint *count, > + LEX_STRING **names, DYNAMIC_COLUMN_VALUE **vals); > > +int mariadb_dyncol_column_cmp_named(const LEX_STRING *s1, const LEX_STRING *s2); > enum enum_dyncol_func_result > -dynamic_column_vals(DYNAMIC_COLUMN *str, > - DYNAMIC_ARRAY *names, DYNAMIC_ARRAY *vals, > - char **free_names); > +mariadb_dyncol_column_count(DYNAMIC_COLUMN *str, uint *column_count); > > /*************************************************************************** > Internal functions, don't use if you don't know what you are doing... > ***************************************************************************/ > > -#define dynamic_column_reassociate(V,P,L, A) dynstr_reassociate((V),(P),(L),(A)) > +#define mariadb_dyncol_reassociate(V,P,L, A) dynstr_reassociate((V),(P),(L),(A)) I'd remove it. You can use dynstr_reassociate() directly in Item_func_dyncol_create::val_str(), if you need it, but don't encourage others to do it, by pussing this macro here. > -#define dynamic_column_value_init(V) (V)->type= DYN_COL_NULL > +#define dyncol_value_init(V) (V)->type= DYN_COL_NULL 1. you forgot the mariadb_ prefix 2. This macro seems to be unused. Perhaps you can simply delete it? > === modified file 'mysql-test/r/dyncol.result' > --- a/mysql-test/r/dyncol.result 2012-09-28 11:01:17 +0000 > +++ b/mysql-test/r/dyncol.result 2012-12-21 19:22:11 +0000 > @@ -1577,3 +1587,53 @@ COLUMN_CHECK('') > SELECT COLUMN_CHECK(NULL); > COLUMN_CHECK(NULL) > NULL > +# > +# escaping check > +# > +select column_json(column_create("string", "'\"/\\`.,whatever")),hex(column_create("string", "'\"/\\`.,whatever")); > +column_json(column_create("string", "'\"/\\`.,whatever")) hex(column_create("string", "'\"/\\`.,whatever")) > +[{"string":"'\"/\\`.,whatever"}] 040100060000000300737472696E670827222F5C602E2C7768617465766572 Please, add a test with two values. What it will look like: [{"a":1},{"b":b}] ? or { "a" : 1, "b" : 2 } ? I don't see a reason why it should be an array of one-value objects, instead of one object, with many values. But perhaps I'm missing something, why did you prefer an array? > +# > +# embedding test > +# > +select column_json(column_create("val", "val", "emb", column_create("val2", "val2"))); > +column_json(column_create("val", "val", "emb", column_create("val2", "val2"))) > +[{"emb":[{"val2":"val2"}],{"val":"val"}] ouch. invalid json, curly braces aren't properly paired. > +select column_json(column_create(1, "val", 2, column_create(3, "val2"))); > +column_json(column_create(1, "val", 2, column_create(3, "val2"))) > +[{"1":"val"},{"2":[{"3":"val2"}]] same here. > === modified file 'sql/item_func.h' > --- a/sql/item_func.h 2012-09-27 23:06:56 +0000 > +++ b/sql/item_func.h 2012-12-21 19:22:11 +0000 > @@ -58,7 +58,7 @@ public: > NOW_FUNC, TRIG_COND_FUNC, > SUSERVAR_FUNC, GUSERVAR_FUNC, COLLATE_FUNC, > EXTRACT_FUNC, CHAR_TYPECAST_FUNC, FUNC_SP, UDF_FUNC, > - NEG_FUNC, GSYSVAR_FUNC }; > + NEG_FUNC, GSYSVAR_FUNC, DYNCOL }; DYNCOL_FUNC > enum optimize_type { OPTIMIZE_NONE,OPTIMIZE_KEY,OPTIMIZE_OP, OPTIMIZE_NULL, > OPTIMIZE_EQUAL }; > enum Type type() const { return FUNC_ITEM; } Regards, Sergei

1 0

[Maria-developers] MariaDB Galera 5.5.28a Release Prep
by Daniel Bartholomew 21 Dec '12

21 Dec '12

All, looks like my first message on this never reached the list, so sending again. I've prepped the release notes and changelog pages for this release and will be activating it in a few hours (barring any show stoppers). - https://kb.askmonty.org/en/mariadb-galera-5528a-release-notes/ - https://kb.askmonty.org/en/mariadb-galera-5528a-changelog/ I've updated the "What is MariaDB Galera Cluster?" and "Getting Started..." pages. If other updates are needed, let me know. https://kb.askmonty.org/en/what-is-mariadb-galera-cluster/ https://kb.askmonty.org/en/getting-started-with-mariadb-galera-cluster/ Also please let me know if updates are needed on the "About" and "Known Limitations" pages https://kb.askmonty.org/en/about-galera-replication/ https://kb.askmonty.org/en/mariadb-galera-cluster-known-limitations/ Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

Re: [Maria-developers] [Commits] Rev 3484: postreview fixes (part 5)
by Sergei Golubchik 21 Dec '12

21 Dec '12

Hi, Sanja! On Dec 21, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-10.0-cassandra/ > ------------------------------------------------------------ > revno: 3484 > revision-id: sanja(a)askmonty.org-20121221010609-cnr2z3fg99kpxzma > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-cassandra > timestamp: Fri 2012-12-21 03:06:09 +0200 > message: > postreview fixes (part 5) > === modified file 'include/ma_dyncol.h' > --- a/include/ma_dyncol.h 2012-12-20 23:35:38 +0000 > +++ b/include/ma_dyncol.h 2012-12-21 01:06:09 +0000 > @@ -169,7 +169,8 @@ madyncol_exists_named(DYNAMIC_COLUMN *st > enum enum_dyncol_func_result > madyncol_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); > enum enum_dyncol_func_result > -madyncol_list_named(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr); > +madyncol_list_named(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr, > + void **memory_to_free); Same here. You can put this memory in the dynarray and don't burden the caller with these nuisanses. Or you can use a normal array, not a dynarray. > /* > if the column do not exists it is NULL > > === modified file 'mysql-test/r/dyncol.result' > --- a/mysql-test/r/dyncol.result 2012-12-20 23:48:20 +0000 > +++ b/mysql-test/r/dyncol.result 2012-12-21 01:06:09 +0000 > @@ -1587,3 +1587,7 @@ COLUMN_CHECK('') > SELECT COLUMN_CHECK(NULL); > COLUMN_CHECK(NULL) > NULL > +# quating check "quoting" But in fact, it's not quoting, it's escaping. > +select column_json(column_create("string", "'\"/\\`.,whatever")),hex(column_create("string", "'\"/\\`.,whatever")); > +column_json(column_create("string", "'\"/\\`.,whatever")) hex(column_create("string", "'\"/\\`.,whatever")) > +[{"string":"'\"/\\`.,whatever"}] 040100060000000300737472696E670827222F5C602E2C7768617465766572 > Regards, Sergei

1 0

[Maria-developers] Odd Error in Logs
by Gordan Bobic 21 Dec '12

21 Dec '12

Hi, I recently switched from MySQL 5.6.5 to MariaDB 10.0, and I've been seeing errors like the following in the logs since: 121221 13:22:34 [ERROR] Master 'master1': Found index PRIMARY whose column info does not match that of MySQL. 121221 13:22:34 [ERROR] Master 'master1': Build InnoDB index translation table for Table ./data/aggregate#P#p201006 failed I didn't dump/restore the data during the upgrade. What exactly is the cause of this? Is it dangerous? Everything seems to work fine, I just want to make sure that something subtle isn't going to cause silent data corruption. TIA. Gordan

1 0

Re: [Maria-developers] [Commits] Rev 3482: post review changes - interface (part 3)
by Sergei Golubchik 21 Dec '12

21 Dec '12

Hi, Sanja! On Dec 21, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-10.0-cassandra/ > > ------------------------------------------------------------ > revno: 3482 > revision-id: sanja(a)askmonty.org-20121220233538-b171yhb9isogvd5h > parent: sanja(a)askmonty.org-20121220204008-fox73otsp1lviqda > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-cassandra > timestamp: Fri 2012-12-21 01:35:38 +0200 > message: > post review changes - interface (part 3) > === modified file 'include/ma_dyncol.h' > --- a/include/ma_dyncol.h 2012-12-20 20:40:08 +0000 > +++ b/include/ma_dyncol.h 2012-12-20 23:35:38 +0000 > @@ -100,6 +100,8 @@ struct st_dynamic_column_value > > typedef struct st_dynamic_column_value DYNAMIC_COLUMN_VALUE; > > +/* old functions (depricated) */ > +#ifdef MADYNCOL_DEPRICATED "deprEcated" > enum enum_dyncol_func_result > dynamic_column_create(DYNAMIC_COLUMN *str, > uint column_nr, DYNAMIC_COLUMN_VALUE *value); > @@ -133,75 +120,105 @@ dynamic_column_update_many(DYNAMIC_COLUM > uint *column_numbers, > DYNAMIC_COLUMN_VALUE *values); > enum enum_dyncol_func_result > -dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str, > - uint add_column_count, > - void *column_keys, > - DYNAMIC_COLUMN_VALUE *values, > - my_bool string_keys); > - > -enum enum_dyncol_func_result > dynamic_column_delete(DYNAMIC_COLUMN *org, uint column_nr); > > enum enum_dyncol_func_result > dynamic_column_exists(DYNAMIC_COLUMN *org, uint column_nr); > + > enum enum_dyncol_func_result > -dynamic_column_exists_named(DYNAMIC_COLUMN *str, LEX_STRING *name); > +dynamic_column_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); > + > enum enum_dyncol_func_result > -dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys); > +dynamic_column_get(DYNAMIC_COLUMN *org, uint column_nr, > + DYNAMIC_COLUMN_VALUE *store_it_here); > +#endif > > -/* List of not NULL columns */ > +/* new functions */ > enum enum_dyncol_func_result > -dynamic_column_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); > +madyncol_create_many(DYNAMIC_COLUMN *str, "madyncol" looks silly :) let's agree on one namespace prefix and use it everywhere. either it's ma_ like in ma_open ma_dyncol_create_many ma_alloc or it's maria_ like in maria_open maria_dyncol_create_many maria_alloc > + uint column_count, > + uint *column_numbers, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool new_string); Regards, Sergei

2 1

Re: [Maria-developers] review of the MDEV-377 Name support for dynamic columns
by Sergei Golubchik 21 Dec '12

21 Dec '12

Hi, Oleksandr! On Dec 21, Oleksandr Byelkin wrote: > It looks like you reviewed some old sources in 10.0-cassandra this is > fixed as we discussed it. I've reviewed the 5.5-cassandra tree. And yes, I thought that was already fixed, so I was a bit susprised to find it unfixed :) Ok, wrong tree, then. Thanks! > [skip] > > > > - > > -int dynamic_column_update(DYNAMIC_COLUMN *str, uint column_nr, > > - DYNAMIC_COLUMN_VALUE *value) > > +enum enum_dyncol_func_result > > +dynamic_column_vals(DYNAMIC_COLUMN *str, > > + DYNAMIC_ARRAY *names, DYNAMIC_ARRAY *vals, > > + char **free_names) > > { > > - return dynamic_column_update_many(str, 1, &column_nr, value); > > + DYN_HEADER header; > > + char *nm; > > + uint i; > > + enum enum_dyncol_func_result rc; > > + > > + *free_names= 0; > > + bzero(names, sizeof(DYNAMIC_ARRAY)); /* In case of errors */ > > + bzero(vals, sizeof(DYNAMIC_ARRAY)); /* In case of errors */ > > + if (str->length == 0) > > + return ER_DYNCOL_OK; /* no columns */ > > + > > + if ((rc= init_read_hdr(&header, str)) < 0) > > + return rc; > > + > > + if (header.entry_size * header.column_count + FIXED_HEADER_SIZE > > > + str->length) > > + return ER_DYNCOL_FORMAT; > > + > > + if (init_dynamic_array(names, sizeof(LEX_STRING), > > + header.column_count, 0) || > > + init_dynamic_array(vals, sizeof(DYNAMIC_COLUMN_VALUE), > > + header.column_count, 0) || > > + (header.format == DYNCOL_FMT_NUM && > > + !(*free_names= (char *)malloc(DYNCOL_NUM_CHAR * header.column_count)))) > > why do you need a special malloc() for names, you can put them in > > the buffer of 'names' dynarray. it'll be a lot more convenient for > > the caller (no free_names to care about). > dynamic array also should be freed, so I do not see difference, I can > rename it to memory_to_free or memory_should_be_freed. The difference is - you pass a separate pointer to the caller, that a caller should take care of. If you put this memory in dynarray, than the caller doesn't have to bother. He eventually delete_dynamic() and that does all the necessary cleanup. It's easier to use for the caller - and for API it's extremely important. More important than simple implementation. Regards, Sergei

1 0

[Maria-developers] MDEV-3792: review of the handler part of the cassandra engine
by Sergei Golubchik 20 Dec '12

20 Dec '12

Hi, Sergey, The handler is quite ok. I've only had few mostly cosmetical comments. > === modified file 'cmake/build_configurations/mysql_release.cmake' > --- cmake/build_configurations/mysql_release.cmake 2012-06-06 12:15:29 +0000 > +++ cmake/build_configurations/mysql_release.cmake 2012-11-12 15:11:23 +0000 > @@ -46,6 +46,8 @@ SET(FEATURE_SET_large 5) > SET(FEATURE_SET_xlarge 6) > SET(FEATURE_SET_community 7) > > +SET(WITH_CASSANDRA_STORAGE_ENGINE ON) > + This probably should be reverted before pushing into main, right? > IF(FEATURE_SET) > STRING(TOLOWER ${FEATURE_SET} feature_set) > SET(num ${FEATURE_SET_${feature_set}}) > > === modified file 'sql/sql_join_cache.cc' > --- sql/sql_join_cache.cc 2012-03-24 17:21:22 +0000 > +++ sql/sql_join_cache.cc 2012-11-12 15:11:23 +0000 > @@ -4543,7 +4546,7 @@ bool JOIN_CACHE_BKAH::prepare_look_for_m > { > last_matching_rec_ref_ptr= next_matching_rec_ref_ptr= 0; > if (no_association && > - (curr_matching_chain= get_matching_chain_by_join_key())) > + !(curr_matching_chain= get_matching_chain_by_join_key())) //psergey: added '!' Is that something that should be pushed into the main branch or it's a temporary hack for cassandra, that should be reverted? > return 1; > last_matching_rec_ref_ptr= get_next_rec_ref(curr_matching_chain); > return 0; > > === added file 'storage/cassandra/CMakeLists.txt' > --- storage/cassandra/CMakeLists.txt 1970-01-01 00:00:00 +0000 > +++ storage/cassandra/CMakeLists.txt 2012-11-12 15:11:23 +0000 > @@ -0,0 +1,25 @@ > + > +SET(cassandra_sources > + ha_cassandra.cc > + ha_cassandra.h > + cassandra_se.h > + cassandra_se.cc > + gen-cpp/Cassandra.cpp > + gen-cpp/cassandra_types.h > + gen-cpp/cassandra_types.cpp > + gen-cpp/cassandra_constants.h > + gen-cpp/cassandra_constants.cpp > + gen-cpp/Cassandra.h) > + > +#INCLUDE_DIRECTORIES(BEFORE ${Boost_INCLUDE_DIRS}) > + > +#INCLUDE_DIRECTORIES(AFTER /usr/local/include/thrift) > +INCLUDE_DIRECTORIES(AFTER /home/buildbot/build/thrift-inst/include/thrift/) this needs to be fixed before pushing into the main tree I suppose, you'll need to detect here if thrift is available and disable the engine otherwise > + > +# > +STRING(REPLACE "-fno-exceptions" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS}) > +STRING(REPLACE "-fno-implicit-templates" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS}) > +#LINK_DIRECTORIES(/home/psergey/cassandra/thrift/lib) > + > +MYSQL_ADD_PLUGIN(cassandra ${cassandra_sources} STORAGE_ENGINE LINK_LIBRARIES thrift) > +# was: STORAGE_ENGINE MANDATORY > > === added file 'storage/cassandra/cassandra_se.cc' > --- storage/cassandra/cassandra_se.cc 1970-01-01 00:00:00 +0000 > +++ storage/cassandra/cassandra_se.cc 2012-11-12 15:11:23 +0000 > @@ -0,0 +1,809 @@ > + > +// Cassandra includes: > +#include <inttypes.h> > +#include <netinet/in.h> > +#include <sys/time.h> > +#include <stdio.h> > +#include <stdarg.h> > + > +#include "Thrift.h" > +#include "transport/TSocket.h" > +#include "transport/TTransport.h" > +#include "transport/TBufferTransports.h" > +#include "protocol/TProtocol.h" > +#include "protocol/TBinaryProtocol.h" > +#include "gen-cpp/Cassandra.h" > +// cassandra includes end > + > +#include "cassandra_se.h" > + > +struct st_mysql_lex_string > +{ > + char *str; > + size_t length; > +}; > + > +using namespace std; > +using namespace apache::thrift; > +using namespace apache::thrift::transport; > +using namespace apache::thrift::protocol; > +using namespace org::apache::cassandra; > + > + > +void Cassandra_se_interface::print_error(const char *format, ...) > +{ > + va_list ap; > + va_start(ap, format); > + // it's not a problem if output was truncated > + vsnprintf(err_buffer, sizeof(err_buffer), format, ap); my_vsnprintf please > + va_end(ap); > +} > + ... > === added file 'storage/cassandra/ha_cassandra.h' > --- storage/cassandra/ha_cassandra.h 1970-01-01 00:00:00 +0000 > +++ storage/cassandra/ha_cassandra.h 2012-11-12 15:11:23 +0000 > @@ -0,0 +1,328 @@ > +/* > + Copyright (c) 2012, Monty Program Ab > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ > +#ifdef USE_PRAGMA_INTERFACE > +#pragma interface /* gcc class implementation */ > +#endif > + > + > +#include "my_global.h" /* ulonglong */ > +#include "thr_lock.h" /* THR_LOCK, THR_LOCK_DATA */ > +#include "handler.h" /* handler */ > +#include "my_base.h" /* ha_rows */ > + > +#include "cassandra_se.h" > + > +/** @brief > + CASSANDRA_SHARE is a structure that will be shared among all open handlers. > + This example implements the minimum of what you will probably need. > +*/ > +typedef struct st_cassandra_share { > + char *table_name; > + uint table_name_length,use_count; > + mysql_mutex_t mutex; > + THR_LOCK lock; > +} CASSANDRA_SHARE; > + > +class ColumnDataConverter; > + > +struct ha_table_option_struct; > + > + > +struct st_dynamic_column_value; > + > +typedef bool (* CAS2DYN_CONVERTER)(const char *cass_data, > + int cass_data_len, > + struct st_dynamic_column_value *value); > +typedef bool (* DYN2CAS_CONVERTER)(struct st_dynamic_column_value *value, > + char **cass_data, > + int *cass_data_len, > + void *buf, void **freemem); > +struct cassandra_type_def > +{ > + const char *name; > + CAS2DYN_CONVERTER cassandra_to_dynamic; > + DYN2CAS_CONVERTER dynamic_to_cassandra; > +}; > + > +typedef struct cassandra_type_def CASSANDRA_TYPE_DEF; > + > +enum cassandtra_type_enum {CT_BIGINT, CT_INT, CT_COUNTER, CT_FLOAT, CT_DOUBLE, > + CT_BLOB, CT_ASCII, CT_TEXT, CT_TIMESTAMP, CT_UUID, CT_BOOLEAN, CT_VARINT, > + CT_DECIMAL}; > + > +typedef enum cassandtra_type_enum CASSANDRA_TYPE; > + > + > + > +/** @brief > + Class definition for the storage engine > +*/ > +class ha_cassandra: public handler > +{ > + friend class Column_name_enumerator_impl; > + THR_LOCK_DATA lock; ///< MySQL lock > + CASSANDRA_SHARE *share; ///< Shared lock info > + > + Cassandra_se_interface *se; > + > + /* description of static part of the table definition */ > + ColumnDataConverter **field_converters; > + uint n_field_converters; > + > + CASSANDRA_TYPE_DEF *default_type_def; > + /* description of dynamic columns part */ > + CASSANDRA_TYPE_DEF *special_type_field_converters; > + LEX_STRING *special_type_field_names; > + uint n_special_type_fields; > + DYNAMIC_ARRAY dynamic_values, dynamic_names; > + DYNAMIC_STRING dynamic_rec; > + > + ColumnDataConverter *rowkey_converter; > + > + bool setup_field_converters(Field **field, uint n_fields); > + void free_field_converters(); > + > + int read_cassandra_columns(bool unpack_pk); > + int check_table_options(struct ha_table_option_struct* options); > + > + bool doing_insert_batch; > + ha_rows insert_rows_batched; > + > + uint dyncol_field; > + bool dyncol_set; > + > + /* Used to produce 'wrong column %s at row %lu' warnings */ > + ha_rows insert_lineno; > + void print_conversion_error(const char *field_name, > + char *cass_value, int cass_value_len); > + int connect_and_check_options(TABLE *table_arg); > +public: > + ha_cassandra(handlerton *hton, TABLE_SHARE *table_arg); > + ~ha_cassandra() > + { > + free_field_converters(); > + delete se; > + } > + > + /** @brief > + The name that will be used for display purposes. > + */ > + const char *table_type() const { return "CASSANDRA"; } > + > + /** @brief > + The name of the index type that will be used for display. > + Don't implement this method unless you really have indexes. > + */ > + const char *index_type(uint inx) { return "HASH"; } > + > + /** @brief > + The file extensions. > + */ > + const char **bas_ext() const; > + > + /** @brief > + This is a list of flags that indicate what functionality the storage engine > + implements. The current table flags are documented in handler.h > + */ > + ulonglong table_flags() const > + { > + /* > + HA_BINLOG_STMT_CAPABLE > + We are saying that this engine is just statement capable to have > + an engine that can only handle statement-based logging. This is > + used in testing. > + HA_REC_NOT_IN_SEQ > + If we don't set it, filesort crashes, because it assumes rowids are > + 1..8 byte numbers > + */ > + return HA_BINLOG_STMT_CAPABLE | > + HA_REC_NOT_IN_SEQ; HA_NO_TRANSACTIONS is unset. Is this engine transactional? HA_PARTIAL_COLUMN_READ is unset. Do you always work with all columns, ignoring read_set and write_set? HA_TABLE_SCAN_ON_INDEX is unset. Do you store the data MyISAM-style, index and data in separate files? HA_FAST_KEY_READ is unset. is reading keys in order faster in cassandra, than random reads? HA_REQUIRE_PRIMARY_KEY is unset. but a primary key is required. HA_PRIMARY_KEY_IN_READ_INDEX is unset. but as far as I see, primary key columns are always returned (and needed for position()) HA_PRIMARY_KEY_REQUIRED_FOR_POSITION is unset, but a primary key is required for position() HA_NO_AUTO_INCREMENT is unset. is auto-increment supported? > + > + } > + > + /** @brief > + This is a bitmap of flags that indicates how the storage engine > + implements indexes. The current index flags are documented in > + handler.h. If you do not implement indexes, just return zero here. > + > + @details > + part is the key part to check. First key part is 0. > + If all_parts is set, MySQL wants to know the flags for the combined > + index, up to and including 'part'. > + */ > + ulong index_flags(uint inx, uint part, bool all_parts) const > + { > + return 0; > + } > + > + /** @brief > + unireg.cc will call max_supported_record_length(), max_supported_keys(), > + max_supported_key_parts(), uint max_supported_key_length() > + to make sure that the storage engine can handle the data it is about to > + send. Return *real* limits of your storage engine here; MySQL will do > + min(your_limits, MySQL_limits) automatically. > + */ > + uint max_supported_record_length() const { return HA_MAX_REC_LENGTH; } > + > + /* Support only one Primary Key, for now */ > + uint max_supported_keys() const { return 1; } > + uint max_supported_key_parts() const { return 1; } > + > + /** @brief > + unireg.cc will call this to make sure that the storage engine can handle > + the data it is about to send. Return *real* limits of your storage engine > + here; MySQL will do min(your_limits, MySQL_limits) automatically. > + > + @details > + There is no need to implement ..._key_... methods if your engine doesn't > + support indexes. > + */ > + uint max_supported_key_length() const { return 16*1024; /* just to return something*/ } > + > + int index_init(uint idx, bool sorted); > + > + int index_read_map(uchar * buf, const uchar * key, > + key_part_map keypart_map, > + enum ha_rkey_function find_flag); > + > + /** @brief > + Called in test_quick_select to determine if indexes should be used. > + */ > + virtual double scan_time() { return (double) (stats.records+stats.deleted) / 20.0+10; } does that make any sense? (I understand that it's copy-paste) stats.deleted, for example, is never set. > + > + /** @brief > + This method will never be called if you do not implement indexes. > + */ > + virtual double read_time(uint, uint, ha_rows rows) > + { return (double) rows / 20.0+1; } same question > + > + virtual void start_bulk_insert(ha_rows rows); > + virtual int end_bulk_insert(); > + > + virtual int reset(); > + > + > + int multi_range_read_init(RANGE_SEQ_IF *seq, void *seq_init_param, > + uint n_ranges, uint mode, HANDLER_BUFFER *buf); > + int multi_range_read_next(range_id_t *range_info); > + ha_rows multi_range_read_info_const(uint keyno, RANGE_SEQ_IF *seq, > + void *seq_init_param, > + uint n_ranges, uint *bufsz, > + uint *flags, COST_VECT *cost); > + ha_rows multi_range_read_info(uint keyno, uint n_ranges, uint keys, > + uint key_parts, uint *bufsz, > + uint *flags, COST_VECT *cost); > + int multi_range_read_explain_info(uint mrr_mode, char *str, size_t size); > + > +private: > + bool source_exhausted; > + bool mrr_start_read(); > + int check_field_options(Field **fields); > + int read_dyncol(DYNAMIC_ARRAY *vals, DYNAMIC_ARRAY *names, > + String *valcol, char **freenames); > + int write_dynamic_row(DYNAMIC_ARRAY *names, DYNAMIC_ARRAY *vals); > + void static free_dynamic_row(DYNAMIC_ARRAY *vals, DYNAMIC_ARRAY *names, > + char *free_names); > + CASSANDRA_TYPE_DEF * get_cassandra_field_def(char *cass_name, > + int cass_name_length); > +public: > + > + /* > + Everything below are methods that we implement in ha_example.cc. may be it'd be better to remove the references to ha_example.cc (and example in general) before pushing this into the main tree? > + > + Most of these methods are not obligatory, skip them and > + MySQL will treat them as not implemented > + */ > + /** @brief > + We implement this in ha_example.cc; it's a required method. > + */ > + int open(const char *name, int mode, uint test_if_locked); // required > + > + /** @brief > + We implement this in ha_example.cc; it's a required method. > + */ > + int close(void); // required > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int write_row(uchar *buf); > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int update_row(const uchar *old_data, uchar *new_data); > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int delete_row(const uchar *buf); > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int index_next(uchar *buf); > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int index_prev(uchar *buf); > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int index_first(uchar *buf); why to have unimplemented methods here? I'd rather remove them. > + > + /** @brief > + We implement this in ha_example.cc. It's not an obligatory method; > + skip it and and MySQL will treat it as not implemented. > + */ > + int index_last(uchar *buf); > + > + /** @brief > + Unlike index_init(), rnd_init() can be called two consecutive times > + without rnd_end() in between (it only makes sense if scan=1). In this > + case, the second call should prepare for the new table scan (e.g if > + rnd_init() allocates the cursor, the second call should position the > + cursor to the start of the table; no need to deallocate and allocate > + it again. This is a required method. > + */ > + int rnd_init(bool scan); //required > + int rnd_end(); > + int rnd_next(uchar *buf); ///< required > + int rnd_pos(uchar *buf, uchar *pos); ///< required > + void position(const uchar *record); ///< required > + int info(uint); ///< required > + int extra(enum ha_extra_function operation); > + int external_lock(THD *thd, int lock_type); ///< required > + int delete_all_rows(void); > + ha_rows records_in_range(uint inx, key_range *min_key, > + key_range *max_key); > + int delete_table(const char *from); > + int create(const char *name, TABLE *form, > + HA_CREATE_INFO *create_info); ///< required > + bool check_if_incompatible_data(HA_CREATE_INFO *info, > + uint table_changes); > + > + THR_LOCK_DATA **store_lock(THD *thd, THR_LOCK_DATA **to, > + enum thr_lock_type lock_type); ///< required > +}; > > === added file 'storage/cassandra/ha_cassandra.cc' > --- storage/cassandra/ha_cassandra.cc 1970-01-01 00:00:00 +0000 > +++ storage/cassandra/ha_cassandra.cc 2012-11-12 15:11:23 +0000 > @@ -0,0 +1,2674 @@ > +/* > + Copyright (c) 2012, Monty Program Ab > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ > + > +#ifdef USE_PRAGMA_IMPLEMENTATION > +#pragma implementation // gcc: Class implementation > +#endif > + > +#include <mysql/plugin.h> > +#include "ha_cassandra.h" > +#include "sql_class.h" > + > +#define DYNCOL_USUAL 20 > +#define DYNCOL_DELTA 100 > +#define DYNCOL_USUAL_REC 1024 > +#define DYNCOL_DELTA_REC 1024 > + > +static handler *cassandra_create_handler(handlerton *hton, > + TABLE_SHARE *table, > + MEM_ROOT *mem_root); > + > +extern int dynamic_column_error_message(enum_dyncol_func_result rc); > + > +handlerton *cassandra_hton; > + > + > +/* > + Hash used to track the number of open tables; variable for example share > + methods > +*/ > +static HASH cassandra_open_tables; > + > +/* The mutex used to init the hash; variable for example share methods */ > +mysql_mutex_t cassandra_mutex; > + > + > +/** > + Structure for CREATE TABLE options (table options). > + It needs to be called ha_table_option_struct. > + > + The option values can be specified in the CREATE TABLE at the end: > + CREATE TABLE ( ... ) *here* > +*/ > + > +struct ha_table_option_struct > +{ > + const char *thrift_host; > + int thrift_port; > + const char *keyspace; > + const char *column_family; > +}; > + > + > +ha_create_table_option cassandra_table_option_list[]= > +{ > + /* > + one option that takes an arbitrary string > + */ > + HA_TOPTION_STRING("thrift_host", thrift_host), > + HA_TOPTION_NUMBER("thrift_port", thrift_port, 9160, 1, 65535, 0), > + HA_TOPTION_STRING("keyspace", keyspace), > + HA_TOPTION_STRING("column_family", column_family), > + HA_TOPTION_END > +}; > + > +/** > + Structure for CREATE TABLE options (field options). > +*/ > + > +struct ha_field_option_struct > +{ > + bool dyncol_field; > +}; > + > +ha_create_table_option cassandra_field_option_list[]= > +{ > + /* > + Collect all other columns as dynamic here, > + the valid values are YES/NO, ON/OFF, 1/0. > + The default is 0, that is true, yes, on. really? looks like a typo. 0 is false, 1 is true. > + */ > + HA_FOPTION_BOOL("DYNAMIC_COLUMN_STORAGE", dyncol_field, 0), > + HA_FOPTION_END > +}; > + > +static MYSQL_THDVAR_ULONG(insert_batch_size, PLUGIN_VAR_RQCMDARG, > + "Number of rows in an INSERT batch", > + NULL, NULL, /*default*/ 100, /*min*/ 1, /*max*/ 1024*1024*1024, 0); > + > +static MYSQL_THDVAR_ULONG(multiget_batch_size, PLUGIN_VAR_RQCMDARG, > + "Number of rows in a multiget(MRR) batch", > + NULL, NULL, /*default*/ 100, /*min*/ 1, /*max*/ 1024*1024*1024, 0); > + > +static MYSQL_THDVAR_ULONG(rnd_batch_size, PLUGIN_VAR_RQCMDARG, > + "Number of rows in an rnd_read (full scan) batch", > + NULL, NULL, /*default*/ 10*1000, /*min*/ 1, /*max*/ 1024*1024*1024, 0); > + > +static MYSQL_THDVAR_ULONG(failure_retries, PLUGIN_VAR_RQCMDARG, > + "Number of times to retry Cassandra calls that failed due to timeouts or " > + "network communication problems. The default, 0, means not to retry.", > + NULL, NULL, /*default*/ 0, /*min*/ 0, /*max*/ 1024*1024*1024, 0); > + > +/* These match values in enum_cassandra_consistency_level */ > +const char *cassandra_consistency_level[] = > +{ > + "ONE", > + "QUORUM", > + "LOCAL_QUORUM", > + "EACH_QUORUM", > + "ALL", > + "ANY", > + "TWO", > + "THREE", > + NullS > +}; > + > +TYPELIB cassandra_consistency_level_typelib= { > + array_elements(cassandra_consistency_level) - 1, "", > + cassandra_consistency_level, NULL > +}; > + > + > +static MYSQL_THDVAR_ENUM(write_consistency, PLUGIN_VAR_RQCMDARG, > + "Cassandra consistency level to use for write operations", NULL, NULL, > + ONE, &cassandra_consistency_level_typelib); > + > +static MYSQL_THDVAR_ENUM(read_consistency, PLUGIN_VAR_RQCMDARG, > + "Cassandra consistency level to use for read operations", NULL, NULL, > + ONE, &cassandra_consistency_level_typelib); > + > + > +mysql_mutex_t cassandra_default_host_lock; > +static char* cassandra_default_thrift_host = NULL; > +static char cassandra_default_host_buf[256]=""; > + > +static void > +cassandra_default_thrift_host_update(THD *thd, > + struct st_mysql_sys_var* var, > + void* var_ptr, /*!< out: where the > + formal string goes */ > + const void* save) /*!< in: immediate result > + from check function */ > +{ > + const char *new_host= *((char**)save); > + const size_t max_len= sizeof(cassandra_default_host_buf); > + > + mysql_mutex_lock(&cassandra_default_host_lock); > + > + if (new_host) > + { > + strncpy(cassandra_default_host_buf, new_host, max_len-1); > + cassandra_default_host_buf[max_len-1]= 0; > + cassandra_default_thrift_host= cassandra_default_host_buf; > + } > + else > + { > + cassandra_default_host_buf[0]= 0; > + cassandra_default_thrift_host= NULL; > + } > + > + *((const char**)var_ptr)= cassandra_default_thrift_host; > + > + mysql_mutex_unlock(&cassandra_default_host_lock); > +} > + > + > +static MYSQL_SYSVAR_STR(default_thrift_host, cassandra_default_thrift_host, > + PLUGIN_VAR_RQCMDARG, > + "Default host for Cassandra thrift connections", > + /*check*/NULL, > + cassandra_default_thrift_host_update, > + /*default*/NULL); > + > +static struct st_mysql_sys_var* cassandra_system_variables[]= { > + MYSQL_SYSVAR(insert_batch_size), > + MYSQL_SYSVAR(multiget_batch_size), > + MYSQL_SYSVAR(rnd_batch_size), > + > + MYSQL_SYSVAR(default_thrift_host), > + MYSQL_SYSVAR(write_consistency), > + MYSQL_SYSVAR(read_consistency), > + MYSQL_SYSVAR(failure_retries), > + NULL > +}; > + > + > +static SHOW_VAR cassandra_status_variables[]= { > + {"row_inserts", > + (char*) &cassandra_counters.row_inserts, SHOW_LONG}, > + {"row_insert_batches", > + (char*) &cassandra_counters.row_insert_batches, SHOW_LONG}, > + > + {"multiget_reads", > + (char*) &cassandra_counters.multiget_reads, SHOW_LONG}, > + {"multiget_keys_scanned", > + (char*) &cassandra_counters.multiget_keys_scanned, SHOW_LONG}, > + {"multiget_rows_read", > + (char*) &cassandra_counters.multiget_rows_read, SHOW_LONG}, > + > + {"timeout_exceptions", > + (char*) &cassandra_counters.timeout_exceptions, SHOW_LONG}, > + {"unavailable_exceptions", > + (char*) &cassandra_counters.unavailable_exceptions, SHOW_LONG}, > + {NullS, NullS, SHOW_LONG} > +}; > + > + > +Cassandra_status_vars cassandra_counters; > +Cassandra_status_vars cassandra_counters_copy; > + > + > +/** > + @brief > + Function we use in the creation of our hash to get key. > +*/ > + > +static uchar* cassandra_get_key(CASSANDRA_SHARE *share, size_t *length, > + my_bool not_used __attribute__((unused))) > +{ > + *length=share->table_name_length; > + return (uchar*) share->table_name; > +} > + > +#ifdef HAVE_PSI_INTERFACE > +static PSI_mutex_key ex_key_mutex_example, ex_key_mutex_CASSANDRA_SHARE_mutex; > + > +static PSI_mutex_info all_cassandra_mutexes[]= > +{ > + { &ex_key_mutex_example, "cassandra", PSI_FLAG_GLOBAL}, > + { &ex_key_mutex_CASSANDRA_SHARE_mutex, "CASSANDRA_SHARE::mutex", 0} > +}; > + > +static void init_cassandra_psi_keys() > +{ > + const char* category= "cassandra"; > + int count; > + > + if (PSI_server == NULL) > + return; > + > + count= array_elements(all_cassandra_mutexes); > + PSI_server->register_mutex(category, all_cassandra_mutexes, count); > +} > +#endif > + > +static int cassandra_init_func(void *p) > +{ > + DBUG_ENTER("cassandra_init_func"); > + > +#ifdef HAVE_PSI_INTERFACE > + init_cassandra_psi_keys(); > +#endif > + > + cassandra_hton= (handlerton *)p; > + mysql_mutex_init(ex_key_mutex_example, &cassandra_mutex, MY_MUTEX_INIT_FAST); > + (void) my_hash_init(&cassandra_open_tables,system_charset_info,32,0,0, > + (my_hash_get_key) cassandra_get_key,0,0); > + > + cassandra_hton->state= SHOW_OPTION_YES; > + cassandra_hton->create= cassandra_create_handler; > + /* > + Don't specify HTON_CAN_RECREATE in flags. re-create is used by TRUNCATE > + TABLE to create an *empty* table from scratch. Cassandra table won't be > + emptied if re-created. > + */ HTON_ALTER_NOT_SUPPORTED is not set. ALTER works? HTON_TEMPORARY_NOT_SUPPORTED is not set. CREATE TEMPORARY TABLE works? HTON_NO_PARTITION is not set. Partitioning works? > + cassandra_hton->flags= 0; > + cassandra_hton->table_options= cassandra_table_option_list; > + cassandra_hton->field_options= cassandra_field_option_list; > + > + mysql_mutex_init(0 /* no instrumentation */, why no instrumentation ? > + &cassandra_default_host_lock, MY_MUTEX_INIT_FAST); > + > + DBUG_RETURN(0); > +} > + > + > +static int cassandra_done_func(void *p) > +{ > + int error= 0; > + DBUG_ENTER("cassandra_done_func"); > + if (cassandra_open_tables.records) > + error= 1; > + my_hash_free(&cassandra_open_tables); > + mysql_mutex_destroy(&cassandra_mutex); > + mysql_mutex_destroy(&cassandra_default_host_lock); > + DBUG_RETURN(error); > +} > + > + > +/** > + @brief > + Example of simple lock controls. The "share" it creates is a > + structure we will pass to each cassandra handler. Do you have to have > + one of these? Well, you have pieces that are used for locking, and > + they are needed to function. > +*/ > + > +static CASSANDRA_SHARE *get_share(const char *table_name, TABLE *table) > +{ > + CASSANDRA_SHARE *share; > + uint length; > + char *tmp_name; > + > + mysql_mutex_lock(&cassandra_mutex); > + length=(uint) strlen(table_name); > + > + if (!(share=(CASSANDRA_SHARE*) my_hash_search(&cassandra_open_tables, > + (uchar*) table_name, > + length))) > + { > + if (!(share=(CASSANDRA_SHARE *) > + my_multi_malloc(MYF(MY_WME | MY_ZEROFILL), > + &share, sizeof(*share), > + &tmp_name, length+1, > + NullS))) > + { > + mysql_mutex_unlock(&cassandra_mutex); > + return NULL; > + } > + > + share->use_count=0; > + share->table_name_length=length; > + share->table_name=tmp_name; > + strmov(share->table_name,table_name); > + if (my_hash_insert(&cassandra_open_tables, (uchar*) share)) > + goto error; > + thr_lock_init(&share->lock); > + mysql_mutex_init(ex_key_mutex_CASSANDRA_SHARE_mutex, > + &share->mutex, MY_MUTEX_INIT_FAST); > + } > + share->use_count++; > + mysql_mutex_unlock(&cassandra_mutex); > + > + return share; > + > +error: > + mysql_mutex_destroy(&share->mutex); > + my_free(share); > + > + return NULL; > +} > + > + > +/** > + @brief > + Free lock controls. We call this whenever we close a table. If the table had > + the last reference to the share, then we free memory associated with it. > +*/ > + > +static int free_share(CASSANDRA_SHARE *share) > +{ > + mysql_mutex_lock(&cassandra_mutex); > + if (!--share->use_count) > + { > + my_hash_delete(&cassandra_open_tables, (uchar*) share); > + thr_lock_delete(&share->lock); > + mysql_mutex_destroy(&share->mutex); > + my_free(share); > + } > + mysql_mutex_unlock(&cassandra_mutex); > + > + return 0; > +} > + > + > +static handler* cassandra_create_handler(handlerton *hton, > + TABLE_SHARE *table, > + MEM_ROOT *mem_root) > +{ > + return new (mem_root) ha_cassandra(hton, table); > +} > + > + > +ha_cassandra::ha_cassandra(handlerton *hton, TABLE_SHARE *table_arg) > + :handler(hton, table_arg), > + se(NULL), field_converters(NULL), > + special_type_field_converters(NULL), > + special_type_field_names(NULL), n_special_type_fields(0), > + rowkey_converter(NULL), > + dyncol_field(0), dyncol_set(0) > +{} > + > + > +static const char *ha_cassandra_exts[] = { > + NullS > +}; > + > +const char **ha_cassandra::bas_ext() const > +{ > + return ha_cassandra_exts; > +} > + > + > +int ha_cassandra::connect_and_check_options(TABLE *table_arg) > +{ > + ha_table_option_struct *options= table_arg->s->option_struct; > + int res; > + DBUG_ENTER("ha_cassandra::connect_and_check_options"); > + > + if ((res= check_field_options(table_arg->s->field)) || > + (res= check_table_options(options))) > + DBUG_RETURN(res); > + > + se= create_cassandra_se(); > + se->set_column_family(options->column_family); > + const char *thrift_host= options->thrift_host? options->thrift_host: > + cassandra_default_thrift_host; > + if (se->connect(thrift_host, options->thrift_port, options->keyspace)) > + { > + my_error(ER_CONNECT_TO_FOREIGN_DATA_SOURCE, MYF(0), se->error_str()); > + DBUG_RETURN(HA_ERR_NO_CONNECTION); > + } > + > + if (setup_field_converters(table_arg->field, table_arg->s->fields)) > + { > + DBUG_RETURN(HA_ERR_NO_CONNECTION); > + } > + > + DBUG_RETURN(0); > +} > + > + > +int ha_cassandra::check_field_options(Field **fields) > +{ > + Field **field; > + uint i; > + DBUG_ENTER("ha_cassandra::check_field_options"); > + for (field= fields, i= 0; *field; field++, i++) > + { > + ha_field_option_struct *field_options= (*field)->option_struct; > + if (field_options && field_options->dyncol_field) > + { > + if (dyncol_set || (*field)->type() != MYSQL_TYPE_BLOB) > + { > + my_error(ER_WRONG_FIELD_SPEC, MYF(0), (*field)->field_name); > + DBUG_RETURN(HA_WRONG_CREATE_OPTION); > + } > + dyncol_set= 1; > + dyncol_field= i; > + bzero(&dynamic_values, sizeof(dynamic_values)); > + bzero(&dynamic_names, sizeof(dynamic_names)); > + bzero(&dynamic_rec, sizeof(dynamic_rec)); > + } > + } > + DBUG_RETURN(0); > +} > + > + > +int ha_cassandra::open(const char *name, int mode, uint test_if_locked) > +{ > + DBUG_ENTER("ha_cassandra::open"); > + > + if (!(share = get_share(name, table))) > + DBUG_RETURN(1); > + thr_lock_data_init(&share->lock,&lock,NULL); > + > + DBUG_ASSERT(!se); > + /* > + Don't do the following on open: it prevents SHOW CREATE TABLE when the server > + has gone away. > + */ > + /* > + int res; > + if ((res= connect_and_check_options(table))) > + { > + DBUG_RETURN(res); > + } > + */ > + > + info(HA_STATUS_NO_LOCK | HA_STATUS_VARIABLE | HA_STATUS_CONST); > + insert_lineno= 0; > + > + DBUG_RETURN(0); > +} > + > + > +int ha_cassandra::close(void) > +{ > + DBUG_ENTER("ha_cassandra::close"); > + delete se; > + se= NULL; > + free_field_converters(); > + DBUG_RETURN(free_share(share)); > +} > + > + > +int ha_cassandra::check_table_options(ha_table_option_struct *options) > +{ > + if (!options->thrift_host && (!cassandra_default_thrift_host || > + !cassandra_default_thrift_host[0])) > + { > + my_error(ER_CONNECT_TO_FOREIGN_DATA_SOURCE, MYF(0), > + "thrift_host table option must be specified, or " > + "@@cassandra_default_thrift_host must be set"); > + return HA_WRONG_CREATE_OPTION; > + } > + > + if (!options->keyspace || !options->column_family) > + { > + my_error(ER_CONNECT_TO_FOREIGN_DATA_SOURCE, MYF(0), > + "keyspace and column_family table options must be specified"); > + return HA_WRONG_CREATE_OPTION; > + } > + return 0; > +} > + > + > +/** > + @brief > + create() is called to create a table. The variable name will have the name > + of the table. > + > + @details > + When create() is called you do not need to worry about > + opening the table. Also, the .frm file will have already been > + created so adjusting create_info is not necessary. You can overwrite > + the .frm file at this point if you wish to change the table > + definition, but there are no methods currently provided for doing > + so. > + > + Called from handle.cc by ha_create_table(). > + > + @see > + ha_create_table() in handle.cc > +*/ > + > +int ha_cassandra::create(const char *name, TABLE *table_arg, > + HA_CREATE_INFO *create_info) > +{ > + int res; > + DBUG_ENTER("ha_cassandra::create"); > + > + Field **pfield= table_arg->s->field; > + if (!((*pfield)->flags & NOT_NULL_FLAG)) > + { > + my_error(ER_WRONG_COLUMN_NAME, MYF(0), "First column must be NOT NULL"); > + DBUG_RETURN(HA_WRONG_CREATE_OPTION); > + } this is unnecessary. the second check guarantees that the first column is NOT NULL, that is, this if() is redundant > + > + if (table_arg->s->keys != 1 || table_arg->s->primary_key !=0 || > + table_arg->key_info[0].key_parts != 1 || > + table_arg->key_info[0].key_part[0].fieldnr != 1) > + { > + my_error(ER_WRONG_COLUMN_NAME, MYF(0), > + "Table must have PRIMARY KEY defined over the first column"); > + DBUG_RETURN(HA_WRONG_CREATE_OPTION); > + } ... > +CASSANDRA_TYPE get_cassandra_type(const char *validator) > +{ > + CASSANDRA_TYPE rc; > + switch(validator[32]) > + { > + case 'L': > + rc= CT_BIGINT; > + break; > + case 'I': > + rc= (validator[35] == '3' ? CT_INT : CT_VARINT); > + rc= CT_INT; > + break; > + case 'C': > + rc= CT_COUNTER; > + break; > + case 'F': > + rc= CT_FLOAT; > + break; > + case 'D': > + switch (validator[33]) > + { > + case 'o': > + rc= CT_DOUBLE; > + break; > + case 'a': > + rc= CT_TIMESTAMP; > + break; > + case 'e': > + rc= CT_DECIMAL; > + break; > + default: > + rc= CT_BLOB; > + break; > + } > + break; > + case 'B': > + rc= (validator[33] == 'o' ? CT_BOOLEAN : CT_BLOB); > + break; > + case 'A': > + rc= CT_ASCII; > + break; > + case 'U': > + rc= (validator[33] == 'T' ? CT_TEXT : CT_UUID); > + break; > + default: > + rc= CT_BLOB; > + } > + DBUG_ASSERT(strcmp(cassandra_types[rc].name, validator) == 0); > + return rc; > +} > + > +ColumnDataConverter *map_field_to_validator(Field *field, const char *validator_name) > +{ > + ColumnDataConverter *res= NULL; > + > + switch(field->type()) { > + case MYSQL_TYPE_TINY: > + if (!strcmp(validator_name, validator_boolean)) why do you strcmp here, while you're only check selected characters in get_cassandra_type() ? you could've called get_cassandra_type() for validator_name instead and compare enum values, instead of strings. > + { > + res= new TinyintDataConverter; > + break; > + } > + /* fall through: */ > + case MYSQL_TYPE_SHORT: > + case MYSQL_TYPE_LONGLONG: > + { > + bool is_counter= false; > + if (!strcmp(validator_name, validator_bigint) || > + !strcmp(validator_name, validator_timestamp) || > + (is_counter= !strcmp(validator_name, validator_counter))) > + res= new BigintDataConverter(!is_counter); > + break; > + } > + case MYSQL_TYPE_FLOAT: > + if (!strcmp(validator_name, validator_float)) > + res= new FloatDataConverter; > + break; > + > + case MYSQL_TYPE_DOUBLE: > + if (!strcmp(validator_name, validator_double)) > + res= new DoubleDataConverter; > + break; > + > + case MYSQL_TYPE_TIMESTAMP: > + if (!strcmp(validator_name, validator_timestamp)) > + res= new TimestampDataConverter; > + break; > + > + case MYSQL_TYPE_STRING: // these are space padded CHAR(n) strings. > + if (!strcmp(validator_name, validator_uuid) && > + field->real_type() == MYSQL_TYPE_STRING && > + field->field_length == 36) > + { > + // UUID maps to CHAR(36), its text representation > + res= new UuidDataConverter; > + break; > + } > + /* fall through: */ > + case MYSQL_TYPE_VAR_STRING: > + case MYSQL_TYPE_VARCHAR: > + { > + /* > + Cassandra's "varint" type is a binary-encoded arbitary-length > + big-endian number. > + - It can be mapped to VARBINARY(N), with sufficiently big N. > + - If the value does not fit into N bytes, it is an error. We should not > + truncate it, because that is just as good as returning garbage. > + - varint should not be mapped to BINARY(N), because BINARY(N) values > + are zero-padded, which will work as multiplying the value by > + 2^k for some value of k. > + */ > + if (field->type() == MYSQL_TYPE_VARCHAR && > + field->binary() && > + (!strcmp(validator_name, validator_varint) || > + !strcmp(validator_name, validator_decimal))) > + { > + res= new StringCopyConverter(field->field_length); > + break; > + } > + > + if (!strcmp(validator_name, validator_blob) || > + !strcmp(validator_name, validator_ascii) || > + !strcmp(validator_name, validator_text)) > + { > + res= new StringCopyConverter((size_t)-1); > + } > + break; > + } > + case MYSQL_TYPE_LONG: > + if (!strcmp(validator_name, validator_int)) > + res= new Int32DataConverter; > + break; > + > + default:; > + } > + return res; > +} > + > + > +bool ha_cassandra::setup_field_converters(Field **field_arg, uint n_fields) > +{ > + char *col_name; > + int col_name_len; > + char *col_type; > + int col_type_len; > + size_t ddl_fields= se->get_ddl_size(); > + const char *default_type= se->get_default_validator(); > + uint max_non_default_fields; > + DBUG_ENTER("ha_cassandra::setup_field_converters"); > + DBUG_ASSERT(default_type); > + > + DBUG_ASSERT(!field_converters); > + DBUG_ASSERT(dyncol_set == 0 || dyncol_set == 1); > + > + /* > + We always should take into account that in case of using dynamic columns > + sql description contain one field which does not described in > + Cassandra DDL also key field is described separately. So that > + is why we use "n_fields - dyncol_set - 1" or "ddl_fields + 2". > + */ > + max_non_default_fields= ddl_fields + 2 - n_fields; > + if (ddl_fields < (n_fields - dyncol_set - 1)) > + { > + se->print_error("Some of SQL fields were not mapped to Cassandra's fields"); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + > + /* allocate memory in one chunk */ > + size_t memsize= sizeof(ColumnDataConverter*) * n_fields + > + (sizeof(LEX_STRING) + sizeof(CASSANDRA_TYPE_DEF))* > + (dyncol_set ? max_non_default_fields : 0); > + if (!(field_converters= (ColumnDataConverter**)my_malloc(memsize, MYF(0)))) > + DBUG_RETURN(true); > + bzero(field_converters, memsize); > + n_field_converters= n_fields; > + > + if (dyncol_set) > + { > + special_type_field_converters= > + (CASSANDRA_TYPE_DEF *)(field_converters + n_fields); > + special_type_field_names= > + ((LEX_STRING*)(special_type_field_converters + max_non_default_fields)); > + } > + > + if (dyncol_set) > + { > + if (init_dynamic_array(&dynamic_values, > + sizeof(DYNAMIC_COLUMN_VALUE), > + DYNCOL_USUAL, DYNCOL_DELTA)) > + DBUG_RETURN(true); > + else > + if (init_dynamic_array(&dynamic_names, > + sizeof(LEX_STRING), > + DYNCOL_USUAL, DYNCOL_DELTA)) > + { > + delete_dynamic(&dynamic_values); > + DBUG_RETURN(true); > + } > + else > + if (init_dynamic_string(&dynamic_rec, NULL, > + DYNCOL_USUAL_REC, DYNCOL_DELTA_REC)) > + { > + delete_dynamic(&dynamic_values); > + delete_dynamic(&dynamic_names); > + DBUG_RETURN(true); > + } > + > + /* Dynamic column field has special processing */ > + field_converters[dyncol_field]= NULL; > + > + default_type_def= cassandra_types + get_cassandra_type(default_type); > + } > + > + se->first_ddl_column(); > + uint n_mapped= 0; > + while (!se->next_ddl_column(&col_name, &col_name_len, &col_type, > + &col_type_len)) > + { > + Field **field; > + uint i; > + /* Mapping for the 1st field is already known */ > + for (field= field_arg + 1, i= 1; *field; field++, i++) > + { > + if ((!dyncol_set || dyncol_field != i) && > + !strcmp((*field)->field_name, col_name)) > + { > + n_mapped++; > + ColumnDataConverter **conv= field_converters + (*field)->field_index; > + if (!(*conv= map_field_to_validator(*field, col_type))) > + { > + se->print_error("Failed to map column %s to datatype %s", > + (*field)->field_name, col_type); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + (*conv)->field= *field; > + } > + } > + if (dyncol_set && !(*field)) // is needed and not found > + { > + DBUG_PRINT("info",("Field not found: %s", col_name)); > + if (strcmp(col_type, default_type)) > + { > + DBUG_PRINT("info",("Field '%s' non-default type: '%s'", > + col_name, col_type)); > + special_type_field_names[n_special_type_fields].length= col_name_len; > + special_type_field_names[n_special_type_fields].str= col_name; > + special_type_field_converters[n_special_type_fields]= > + cassandra_types[get_cassandra_type(col_type)]; > + n_special_type_fields++; > + } > + } > + } > + > + if (n_mapped != n_fields - 1 - dyncol_set) > + { > + Field *first_unmapped= NULL; > + /* Find the first field */ > + for (uint i= 1; i < n_fields;i++) > + { > + if (!field_converters[i]) > + { > + first_unmapped= field_arg[i]; > + break; > + } > + } > + DBUG_ASSERT(first_unmapped); > + > + se->print_error("Field `%s` could not be mapped to any field in Cassandra", generally (here and everywhere) use %`s instead of `%s` > + first_unmapped->field_name); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + > + /* > + Setup type conversion for row_key. > + */ > + se->get_rowkey_type(&col_name, &col_type); > + if (col_name && strcmp(col_name, (*field_arg)->field_name)) > + { > + se->print_error("PRIMARY KEY column must match Cassandra's name '%s'", > + col_name); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + if (!col_name && strcmp("rowkey", (*field_arg)->field_name)) > + { > + se->print_error("target column family has no key_alias defined, " > + "PRIMARY KEY column must be named 'rowkey'"); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + > + if (col_type != NULL) > + { > + if (!(rowkey_converter= map_field_to_validator(*field_arg, col_type))) > + { > + se->print_error("Failed to map PRIMARY KEY to datatype %s", col_type); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + rowkey_converter->field= *field_arg; > + } > + else > + { > + se->print_error("Cassandra's rowkey has no defined datatype (todo: support this)"); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + DBUG_RETURN(true); > + } > + > + DBUG_RETURN(false); > +} > + > + > +void ha_cassandra::free_field_converters() > +{ > + delete rowkey_converter; > + rowkey_converter= NULL; > + > + if (dyncol_set) > + { > + delete_dynamic(&dynamic_values); > + delete_dynamic(&dynamic_names); > + dynstr_free(&dynamic_rec); > + } > + if (field_converters) > + { > + for (uint i=0; i < n_field_converters; i++) > + if (field_converters[i]) > + { > + DBUG_ASSERT(!dyncol_set || i == dyncol_field); > + delete field_converters[i]; > + } > + my_free(field_converters); > + field_converters= NULL; > + } > +} > + > + > +int ha_cassandra::index_init(uint idx, bool sorted) > +{ > + int ires; > + if (!se && (ires= connect_and_check_options(table))) > + return ires; > + return 0; > +} > + > +void store_key_image_to_rec(Field *field, uchar *ptr, uint len); > + > +int ha_cassandra::index_read_map(uchar *buf, const uchar *key, > + key_part_map keypart_map, > + enum ha_rkey_function find_flag) > +{ > + int rc= 0; > + DBUG_ENTER("ha_cassandra::index_read_map"); > + > + if (find_flag != HA_READ_KEY_EXACT) > + DBUG_RETURN(HA_ERR_WRONG_COMMAND); did you verify that the server expects HA_ERR_WRONG_COMMAND from this method and correctly handles it? or, perhaps, find_flag is always HA_READ_KEY_EXACT here, because of your index_flags() ? > + > + uint key_len= calculate_key_len(table, active_index, key, keypart_map); > + store_key_image_to_rec(table->field[0], (uchar*)key, key_len); > + > + char *cass_key; > + int cass_key_len; > + my_bitmap_map *old_map; > + > + old_map= dbug_tmp_use_all_columns(table, table->read_set); > + > + if (rowkey_converter->mariadb_to_cassandra(&cass_key, &cass_key_len)) > + { > + /* We get here when making lookups like uuid_column='not-an-uuid' */ > + dbug_tmp_restore_column_map(table->read_set, old_map); > + DBUG_RETURN(HA_ERR_KEY_NOT_FOUND); > + } > + > + dbug_tmp_restore_column_map(table->read_set, old_map); > + > + bool found; > + if (se->get_slice(cass_key, cass_key_len, &found)) > + { > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); generally it's better not to use my_error() directly, but only return an error code, and return your se->error_str() from ha_cassandra::get_error_message() > + rc= HA_ERR_INTERNAL_ERROR; > + } > + > + /* TODO: what if we're not reading all columns?? */ > + if (!found) > + rc= HA_ERR_KEY_NOT_FOUND; > + else > + rc= read_cassandra_columns(false); > + > + DBUG_RETURN(rc); > +} > + > + > +void ha_cassandra::print_conversion_error(const char *field_name, > + char *cass_value, > + int cass_value_len) > +{ > + char buf[32]; > + char *p= cass_value; > + size_t i= 0; > + for (; (i < (int)sizeof(buf)-1) && (p < cass_value + cass_value_len); p++) why do you cast here? > + { > + buf[i++]= map2number[(*p >> 4) & 0xF]; > + buf[i++]= map2number[*p & 0xF]; > + } > + buf[i]=0; > + > + se->print_error("Unable to convert value for field `%s` from Cassandra's data" > + " format. Source data is %d bytes, 0x%s%s", > + field_name, cass_value_len, buf, > + (i == sizeof(buf) - 1)? "..." : ""); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > +} > + > + > +void free_strings(DYNAMIC_COLUMN_VALUE *vals, uint num) > +{ > + for (uint i= 0; i < num; i++) > + if (vals[i].type == DYN_COL_STRING && > + !vals[i].x.string.nonfreeable) > + my_free(vals[i].x.string.value.str); > +} > + > + > +CASSANDRA_TYPE_DEF * ha_cassandra::get_cassandra_field_def(char *cass_name, > + int cass_name_len) > +{ > + CASSANDRA_TYPE_DEF *type= default_type_def; > + for(uint i= 0; i < n_special_type_fields; i++) > + { > + if (cass_name_len == (int)special_type_field_names[i].length && > + memcmp(cass_name, special_type_field_names[i].str, > + cass_name_len) == 0) > + { > + type= special_type_field_converters + i; > + break; > + } > + } > + return type; > +} > + > +int ha_cassandra::read_cassandra_columns(bool unpack_pk) > +{ > + char *cass_name; > + char *cass_value; > + int cass_value_len, cass_name_len; > + Field **field; > + int res= 0; > + ulong total_name_len= 0; > + > + /* > + cassandra_to_mariadb() calls will use field->store(...) methods, which > + require that the column is in the table->write_set > + */ > + my_bitmap_map *old_map; > + old_map= dbug_tmp_use_all_columns(table, table->write_set); > + > + /* Start with all fields being NULL */ > + for (field= table->field + 1; *field; field++) > + (*field)->set_null(); > + > + while (!se->get_next_read_column(&cass_name, &cass_name_len, > + &cass_value, &cass_value_len)) > + { > + // map to our column. todo: use hash or something.. > + bool found= 0; > + for (field= table->field + 1; *field; field++) > + { > + uint fieldnr= (*field)->field_index; > + if ((!dyncol_set || dyncol_field != fieldnr) && > + !strcmp((*field)->field_name, cass_name)) > + { > + found= 1; > + (*field)->set_notnull(); > + if (field_converters[fieldnr]->cassandra_to_mariadb(cass_value, > + cass_value_len)) > + { > + print_conversion_error((*field)->field_name, cass_value, > + cass_value_len); > + res=1; > + goto err; > + } > + break; > + } > + } > + if (dyncol_set && !found) > + { > + DYNAMIC_COLUMN_VALUE val; > + LEX_STRING nm; > + CASSANDRA_TYPE_DEF *type= get_cassandra_field_def(cass_name, > + cass_name_len); > + nm.str= cass_name; > + nm.length= cass_name_len; > + if (nm.length > MAX_NAME_LENGTH) > + { > + se->print_error("Unable to convert value for field `%s`" > + " from Cassandra's data format. Name" > + " length exceed limit of %u: '%s'", > + table->field[dyncol_field]->field_name, > + (uint)MAX_NAME_LENGTH, cass_name); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + res=1; > + goto err; > + } > + total_name_len+= cass_name_len; > + if (nm.length > MAX_TOTAL_NAME_LENGTH) > + { > + se->print_error("Unable to convert value for field `%s`" > + " from Cassandra's data format. Sum of all names" > + " length exceed limit of %lu", > + table->field[dyncol_field]->field_name, > + cass_name, (uint)MAX_TOTAL_NAME_LENGTH); > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + res=1; > + goto err; > + } > + > + if ((res= (*(type->cassandra_to_dynamic))(cass_value, > + cass_value_len, &val)) || > + insert_dynamic(&dynamic_names, (uchar *) &nm) || > + insert_dynamic(&dynamic_values, (uchar *) &val)) > + { > + if (res) > + { > + print_conversion_error(cass_name, cass_value, cass_value_len); > + } > + free_strings((DYNAMIC_COLUMN_VALUE *)dynamic_values.buffer, > + dynamic_values.elements); > + // EOM shouldm be already reported if happened > + res=1; > + goto err; > + } > + } > + } > + > + dynamic_rec.length= 0; > + if (dyncol_set) > + { > + if (dynamic_column_create_many_internal_fmt(&dynamic_rec, > + dynamic_names.elements, > + dynamic_names.buffer, > + (DYNAMIC_COLUMN_VALUE *) > + dynamic_values.buffer, > + FALSE, > + TRUE) < 0) > + dynamic_rec.length= 0; > + > + free_strings((DYNAMIC_COLUMN_VALUE *)dynamic_values.buffer, > + dynamic_values.elements); > + dynamic_values.elements= dynamic_names.elements= 0; > + } > + if (dyncol_set) why a separate if() ? > + { > + if (dynamic_rec.length == 0) > + table->field[dyncol_field]->set_null(); > + else > + { > + Field_blob *blob= (Field_blob *)table->field[dyncol_field]; > + blob->set_notnull(); > + blob->store_length(dynamic_rec.length); > + *((char **)(((char *)blob->ptr) + blob->pack_length_no_ptr()))= > + dynamic_rec.str; > + } > + } > + > + if (unpack_pk) > + { > + /* Unpack rowkey to primary key */ > + field= table->field; > + (*field)->set_notnull(); > + se->get_read_rowkey(&cass_value, &cass_value_len); > + if (rowkey_converter->cassandra_to_mariadb(cass_value, cass_value_len)) > + { > + print_conversion_error((*field)->field_name, cass_value, cass_value_len); > + res=1; > + goto err; > + } > + } > + > +err: > + dbug_tmp_restore_column_map(table->write_set, old_map); > + return res; > +} ... > +int ha_cassandra::rnd_pos(uchar *buf, uchar *pos) > +{ > + int rc; > + DBUG_ENTER("ha_cassandra::rnd_pos"); > + > + int save_active_index= active_index; > + active_index= 0; /* The primary key */ > + rc= index_read_map(buf, pos, key_part_map(1), HA_READ_KEY_EXACT); > + > + active_index= save_active_index; a bit nicer would be to implement index_read_map_idx and call it from here and from index_read_map > + > + DBUG_RETURN(rc); > +} > + ... > +bool ha_cassandra::mrr_start_read() > +{ > + uint key_len; > + > + my_bitmap_map *old_map; > + old_map= dbug_tmp_use_all_columns(table, table->read_set); > + > + se->new_lookup_keys(); > + > + while (!(source_exhausted= mrr_funcs.next(mrr_iter, &mrr_cur_range))) > + { > + char *cass_key; > + int cass_key_len; > + > + DBUG_ASSERT(mrr_cur_range.range_flag & EQ_RANGE); > + > + uchar *key= (uchar*)mrr_cur_range.start_key.key; > + key_len= mrr_cur_range.start_key.length; > + //key_len= calculate_key_len(table, active_index, key, keypart_map); // NEED THIS?? > + store_key_image_to_rec(table->field[0], (uchar*)key, key_len); > + > + rowkey_converter->mariadb_to_cassandra(&cass_key, &cass_key_len); > + > + // Primitive buffer control > + if (se->add_lookup_key(cass_key, cass_key_len) > > + THDVAR(table->in_use, multiget_batch_size)) > + break; I'd suggest to add a status variable to count how many times the buffer was refilled > + } > + > + dbug_tmp_restore_column_map(table->read_set, old_map); > + > + return se->multiget_slice(); > +} ... > +///////////////////////////////////////////////////////////////////////////// > +// Dummy implementations start > +///////////////////////////////////////////////////////////////////////////// > + > + > +int ha_cassandra::index_next(uchar *buf) Why did you do all these dummy methods? > +{ > + int rc; > + DBUG_ENTER("ha_cassandra::index_next"); > + rc= HA_ERR_WRONG_COMMAND; > + DBUG_RETURN(rc); > +} > + > + > +int ha_cassandra::index_prev(uchar *buf) > +{ > + int rc; > + DBUG_ENTER("ha_cassandra::index_prev"); > + rc= HA_ERR_WRONG_COMMAND; > + DBUG_RETURN(rc); > +} > + > + > +int ha_cassandra::index_first(uchar *buf) > +{ > + int rc; > + DBUG_ENTER("ha_cassandra::index_first"); > + rc= HA_ERR_WRONG_COMMAND; > + DBUG_RETURN(rc); > +} > + > +int ha_cassandra::index_last(uchar *buf) > +{ > + int rc; > + DBUG_ENTER("ha_cassandra::index_last"); > + rc= HA_ERR_WRONG_COMMAND; > + DBUG_RETURN(rc); > +} > + > + > +ha_rows ha_cassandra::records_in_range(uint inx, key_range *min_key, > + key_range *max_key) > +{ > + DBUG_ENTER("ha_cassandra::records_in_range"); > + //DBUG_RETURN(10); // low number to force index usage > + DBUG_RETURN(HA_POS_ERROR); > +} > + > + > +class Column_name_enumerator_impl : public Column_name_enumerator > +{ > + ha_cassandra *obj; > + uint idx; > +public: > + Column_name_enumerator_impl(ha_cassandra *obj_arg) : obj(obj_arg), idx(1) {} > + const char* get_next_name() > + { > + if (idx == obj->table->s->fields) > + return NULL; > + else > + return obj->table->field[idx++]->field_name; > + } > +}; > + > + > +int ha_cassandra::update_row(const uchar *old_data, uchar *new_data) > +{ > + DYNAMIC_ARRAY oldvals, oldnames, vals, names; > + String oldvalcol, valcol; > + char *oldfree_names= NULL, *free_names= NULL; > + my_bitmap_map *old_map; > + int res; > + DBUG_ENTER("ha_cassandra::update_row"); > + /* Currently, it is guaranteed that new_data == table->record[0] */ > + DBUG_ASSERT(new_data == table->record[0]); > + /* For now, just rewrite the full record */ > + se->clear_insert_buffer(); > + > + old_map= dbug_tmp_use_all_columns(table, table->read_set); > + > + char *old_key; > + int old_key_len; > + se->get_read_rowkey(&old_key, &old_key_len); > + > + /* Get the key we're going to write */ > + char *new_key; > + int new_key_len; > + if (rowkey_converter->mariadb_to_cassandra(&new_key, &new_key_len)) > + { > + my_error(ER_WARN_DATA_OUT_OF_RANGE, MYF(0), > + rowkey_converter->field->field_name, insert_lineno); > + dbug_tmp_restore_column_map(table->read_set, old_map); > + DBUG_RETURN(HA_ERR_AUTOINC_ERANGE); > + } > + > + /* > + Compare it to the key we've read. For all types that Cassandra supports, > + binary byte-wise comparison can be used > + */ > + bool new_primary_key; > + if (new_key_len != old_key_len || memcmp(old_key, new_key, new_key_len)) > + new_primary_key= true; > + else > + new_primary_key= false; > + > + if (dyncol_set) > + { > + Field *field= table->field[dyncol_field]; > + /* move to get old_data */ > + my_ptrdiff_t diff; > + diff= (my_ptrdiff_t) (old_data - new_data); > + field->move_field_offset(diff); // Points now at old_data > + if ((res= read_dyncol(&oldvals, &oldnames, &oldvalcol, &oldfree_names))) > + DBUG_RETURN(res); > + field->move_field_offset(-diff); // back to new_data > + if ((res= read_dyncol(&vals, &names, &valcol, &free_names))) > + { > + free_dynamic_row(&oldnames, &oldvals, oldfree_names); > + DBUG_RETURN(res); > + } > + } > + > + if (new_primary_key) > + { > + /* > + Primary key value changed. This is essentially a DELETE + INSERT. > + Add a DELETE operation into the batch > + */ > + Column_name_enumerator_impl name_enumerator(this); > + se->add_row_deletion(old_key, old_key_len, &name_enumerator, > + (LEX_STRING *)oldnames.buffer, > + (dyncol_set ? oldnames.elements : 0)); > + oldnames.elements= oldvals.elements= 0; // they will be deleted > + } > + > + se->start_row_insert(new_key, new_key_len); > + > + /* Convert other fields */ > + for (uint i= 1; i < table->s->fields; i++) I'd probably would've simply done delete+insert in all cases :) but as you've went to the troubles of updating column by column, it really make little sense to rewrite columns that didn't change. could you check table->write_set here and memcmp with the old column value before updating? > + { > + char *cass_data; > + int cass_data_len; > + if (dyncol_set && dyncol_field == i) > + { > + DBUG_ASSERT(field_converters[i] == NULL); > + if ((res= write_dynamic_row(&vals, &names))) > + goto err; > + } > + else > + { > + if (field_converters[i]->mariadb_to_cassandra(&cass_data, &cass_data_len)) > + { > + my_error(ER_WARN_DATA_OUT_OF_RANGE, MYF(0), > + field_converters[i]->field->field_name, insert_lineno); > + dbug_tmp_restore_column_map(table->read_set, old_map); > + DBUG_RETURN(HA_ERR_AUTOINC_ERANGE); > + } > + se->add_insert_column(field_converters[i]->field->field_name, 0, > + cass_data, cass_data_len); > + } > + } > + if (dyncol_set) > + { > + /* find removed fields */ > + uint i= 0, j= 0; > + LEX_STRING *onames= (LEX_STRING *)oldnames.buffer; > + LEX_STRING *nnames= (LEX_STRING *)names.buffer; > + /* both array are sorted */ > + for(; i < oldnames.elements; i++) > + { > + int scmp= 0; > + while (j < names.elements && > + (nnames[j].length < onames[i].length || > + (nnames[j].length == onames[i].length && > + (scmp= memcmp(nnames[j].str, onames[i].str, > + onames[i].length)) < 0))) > + j++; > + if (j < names.elements && > + nnames[j].length == onames[i].length && > + scmp == 0) > + j++; > + else > + se->add_insert_delete_column(onames[i].str, onames[i].length); > + } > + } > + > + dbug_tmp_restore_column_map(table->read_set, old_map); > + > + res= se->do_insert(); > + > + if (res) > + my_error(ER_INTERNAL_ERROR, MYF(0), se->error_str()); > + > +err: > + if (dyncol_set) > + { > + free_dynamic_row(&oldnames, &oldvals, oldfree_names); > + free_dynamic_row(&names, &vals, free_names); > + } > + > + DBUG_RETURN(res? HA_ERR_INTERNAL_ERROR: 0); > +} > + > + > +int ha_cassandra::extra(enum ha_extra_function operation) > +{ > + DBUG_ENTER("ha_cassandra::extra"); > + DBUG_RETURN(0); > +} please. why do you like dummy methods that much? :) > + > + > +/* The following function was copied from ha_blackhole::store_lock: */ > +THR_LOCK_DATA **ha_cassandra::store_lock(THD *thd, > + THR_LOCK_DATA **to, > + enum thr_lock_type lock_type) > +{ > + DBUG_ENTER("ha_cassandra::store_lock"); > + if (lock_type != TL_IGNORE && lock.type == TL_UNLOCK) > + { > + /* > + Here is where we get into the guts of a row level lock. > + If TL_UNLOCK is set > + If we are not doing a LOCK TABLE or DISCARD/IMPORT > + TABLESPACE, then allow multiple writers > + */ > + > + if ((lock_type >= TL_WRITE_CONCURRENT_INSERT && > + lock_type <= TL_WRITE) && !thd_in_lock_tables(thd) > + && !thd_tablespace_op(thd)) 1. tablespace op in cassandra? really? too much copy-paste is confusing. (here and in the comments too) 2. did you test LOCK TABLES? > + lock_type = TL_WRITE_ALLOW_WRITE; that makes all changes to cassanrda immediately visible by concurrently running threads. okay, if that's intentional, but it needs to be documented, as it not the SQL semantics. > + > + /* > + In queries of type INSERT INTO t1 SELECT ... FROM t2 ... > + MySQL would use the lock TL_READ_NO_INSERT on t2, and that > + would conflict with TL_WRITE_ALLOW_WRITE, blocking all inserts > + to t2. Convert the lock to a normal read lock to allow > + concurrent inserts to t2. > + */ > + > + if (lock_type == TL_READ_NO_INSERT && !thd_in_lock_tables(thd)) > + lock_type = TL_READ; this also breaks SBR for INSERT ... SELECT. same as above - okay if intentional, but should be thouroughly documented. > + > + lock.type= lock_type; > + } > + *to++= &lock; > + DBUG_RETURN(to); > +} > + > + > +int ha_cassandra::external_lock(THD *thd, int lock_type) > +{ > + DBUG_ENTER("ha_cassandra::external_lock"); > + DBUG_RETURN(0); > +} > + > +int ha_cassandra::delete_table(const char *name) > +{ > + DBUG_ENTER("ha_cassandra::delete_table"); > + /* > + Cassandra table is just a view. Dropping it doesn't affect the underlying > + column family. > + */ > + DBUG_RETURN(0); > +} more dummies... > + > + > +/** > + check_if_incompatible_data() called if ALTER TABLE can't detect otherwise > + if new and old definition are compatible > + > + @details If there are no other explicit signs like changed number of > + fields this function will be called by compare_tables() > + (sql/sql_tables.cc) to decide should we rewrite whole table or only .frm > + file. > + > +*/ > + > +bool ha_cassandra::check_if_incompatible_data(HA_CREATE_INFO *info, > + uint table_changes) > +{ > + DBUG_ENTER("ha_cassandra::check_if_incompatible_data"); > + /* Checked, we intend to have this empty for Cassandra SE. */ > + DBUG_RETURN(COMPATIBLE_DATA_YES); > +} > + > + > +///////////////////////////////////////////////////////////////////////////// > +// Dummy implementations end > +///////////////////////////////////////////////////////////////////////////// ::update_row and ::store_lock are hardly dummies :) > + > +static int show_cassandra_vars(THD *thd, SHOW_VAR *var, char *buff) > +{ > + cassandra_counters_copy= cassandra_counters; > + > + var->type= SHOW_ARRAY; > + var->value= (char *) &cassandra_status_variables; what's the point? If you don't do anything in this function, you can just as easily list all status variables below in the func_status[] array. > + return 0; > +} > + > + > +struct st_mysql_storage_engine cassandra_storage_engine= > +{ MYSQL_HANDLERTON_INTERFACE_VERSION }; > + > +static struct st_mysql_show_var func_status[]= > +{ > + {"Cassandra", (char *)show_cassandra_vars, SHOW_FUNC}, > + {0,0,SHOW_UNDEF} > +}; > + > +maria_declare_plugin(cassandra) > +{ > + MYSQL_STORAGE_ENGINE_PLUGIN, > + &cassandra_storage_engine, > + "CASSANDRA", > + "Monty Program Ab", > + "Cassandra storage engine", > + PLUGIN_LICENSE_GPL, > + cassandra_init_func, /* Plugin Init */ > + cassandra_done_func, /* Plugin Deinit */ > + 0x0001, /* version number (0.1) */ > + func_status, /* status variables */ > + cassandra_system_variables, /* system variables */ > + "0.1", /* string version */ > + MariaDB_PLUGIN_MATURITY_EXPERIMENTAL /* maturity */ > +} > +maria_declare_plugin_end; Regards, Sergei

2 3

Re: [Maria-developers] [Commits] Rev 3480: Post review changes in the interface part 1.
by Sergei Golubchik 19 Dec '12

19 Dec '12

Hi, Sanja! few preliminary comments: On Dec 19, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-10.0-cassandra/ > ------------------------------------------------------------ > revno: 3480 > revision-id: sanja(a)askmonty.org-20121219200839-oyxvfz7im9p7wqpw > parent: sanja(a)montyprogram.com-20121119121604-5h5tu0zn11em0sb3 > committer: sanja(a)askmonty.org > branch nick: work-maria-10.0-cassandra > timestamp: Wed 2012-12-19 22:08:39 +0200 > message: > Post review changes in the interface part 1. > === modified file 'mysys/ma_dyncol.c' > --- a/mysys/ma_dyncol.c 2012-09-28 12:27:16 +0000 > +++ b/mysys/ma_dyncol.c 2012-12-19 20:08:39 +0000 > @@ -26,6 +26,39 @@ > SUCH DAMAGE. > */ > > +/* > + Numeric format: > + =============== > + * Fixed header part > + 1 byte flags: > + 0,1 bits - <offset size> - 1 > + 2-7 bits - 0 > + 2 bytes column counter > + * Columns directory sorted by column number, each entry contains of: > + 2 bytes column number > + <offset size> bytes (1-4) combined offset from beginning of > + the data segment + 3 bit type > + * Data of above columns size of data and length depend on type > + > + Columns with names: > + =================== > + * Fixed header part > + 1 byte flags: > + 0,1 bits - <offset size> - 2 > + 2 bit - 1 (mens format with names) s/mens/means/g > + 3,4 bits - 01 (mens <names offset size> - 1, now 2 is only supported size) eh. I would simply assume that if the bit 2 is 1, this also means offset-2. Like, you have "old format" as above, and "new format" with names, support for recursion (4 bits per type, offset-2), etc. you only have 6 bits here, let's use them sparingly and keep things simple. > + 5-7 bits - 0 > + 2 bytes column counter > + * Variable header part > + <names offset size> (2) bytes size of stored names pool > + * Column directory sorted by names, each consists of > + <names offset size> (2) bytes offset of name > + <offset size> bytes (1-4)bytes combined offset from beginning of > + the data segment + 4 bit type > + * Names stored one after another > + * Data of above columns size of data and length depend on type > +*/ > + > #include "mysys_priv.h" > #include <m_string.h> > #include <ma_dyncol.h> > === modified file 'storage/cassandra/ha_cassandra.cc' > --- a/storage/cassandra/ha_cassandra.cc 2012-11-19 12:16:04 +0000 > +++ b/storage/cassandra/ha_cassandra.cc 2012-12-19 20:08:39 +0000 > @@ -893,9 +893,37 @@ public: > /** > Converting dynamic columns types to/from casandra types > */ > + > + > +/** > + Check and initialize (if it is needed) string MEM_ROOT > +*/ > +static void alloc_strings_memroot(MEM_ROOT *mem_root) > +{ > + if (mem_root->block_size == 0) write if (alloc_root_inited(mem_root)) > + { > + /* > + The mem_root used to allocate UUID (of length 36 + \0) so make > + appropriate allocated size > + */ > + init_alloc_root(mem_root, > + (36 + 1 + ALIGN_SIZE(sizeof(USED_MEM))) * 10 + > + ALLOC_ROOT_MIN_BLOCK_SIZE, > + (36 + 1 + ALIGN_SIZE(sizeof(USED_MEM))) * 10 + > + ALLOC_ROOT_MIN_BLOCK_SIZE); > + } > +} > + > +static void free_strings_memroot(MEM_ROOT *mem_root) > +{ > + if (mem_root->block_size != 0) and here > + free_root(mem_root, MYF(0)); > +} > + > @@ -1730,6 +1759,7 @@ int ha_cassandra::read_cassandra_columns > int res= 0; > ulong total_name_len= 0; > > + strings_root.block_size= 0; // indicates uninitialized MEM_ROOT Use clear_alloc_root(&strings_root); > /* > cassandra_to_mariadb() calls will use field->store(...) methods, which > require that the column is in the table->write_set Regards, Sergei

1 0

Re: [Maria-developers] (MDEV-3949) OTP authenticator plugins
by Roberto Spadim 19 Dec '12

19 Dec '12

hi sergei! sorry about wrong place, i'm new in mariadb and i bit lost, since i changed from mysql development and error correction to mysql user and now mariadb user i will add my email to list and start asking there about s/key, shure! if you could send and maybe put in mariadb by default could be very nice! i'm using some windows and some linux server, in this case i can't use only pam.d, and some linux servers don't give me a unix user to use it :/ that's why i'm trying to find a OTP solution to mariadb, and if not try to develop one (when i have time) well if you could send your solution we could develop some more ideas for sure, i'will not use it to make money, just to secure my self 2012/12/19 <serg(a)montyprogram.com> > ou ask a question - do it on the mailing list. Jira is for > bug reports and feature requests. > -- Roberto Spadim Spadim Technology / SPAEmpresarial

2 1

[Maria-developers] Patches in 10.0 missing in 10.0-base, causing buildbot failures
by Kristian Nielsen 19 Dec '12

19 Dec '12

Hi, I was looking at 10.0-base in Buildbot, in particular at the "update2" failures in all the .deb builds. I fixed one thing in Buildbot, but it still fails. I suspect it is because of these two (at least) commits: sergii(a)pisem.net-20121027121326-q87f5lt6ldxr1em6 sergii(a)pisem.net-20121020163522-uw2apo36c6nn6yi8 These were pushed to 10.0, while it seems they should instead have been pushed to 10.0-base (where they are missing). There are probably also other commits with same problem, eg. this one which is a 5.5 merge according to commit message: sergii(a)pisem.net-20121103112851-bvy1mlhirxbkedvf Any suggestions what to do to fix this and get 10.0-base working in Buildbot? - Kristian.

3 5

Re: [Maria-developers] [Commits] Rev 3449: MDEV-532: Fix some race conditions in test cases. in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Kristian Nielsen 19 Dec '12

19 Dec '12

Hi Serg, Just wanted to be sure you saw this. The async binlogging of the checkpoint event after MDEV-532 means a few test cases need to wait for the event to be logged, or they will get races causing failures with result difference in SHOW BINLOG EVENTS or similar. Typically after running FLUSH LOGS. I fixed the ones I found already. But it is possible that a few others will pop up, and it is useful that someone else than me knows how to fix (so I'm picking on you as the reviewer :-). As seen in the patch, it's just a matter of including the file include/wait_for_binlog_checkpoint.inc in the appropriate place. - Kristian. knielsen(a)knielsen-hq.org writes: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 3449 > revision-id: knielsen(a)knielsen-hq.org-20121217114911-m3qv20xmk7htleez > parent: igor(a)askmonty.org-20121217004919-xgon81zqncwimr3m > committer: knielsen(a)knielsen-hq.org > branch nick: mariadb-10.0-base > timestamp: Mon 2012-12-17 12:49:11 +0100 > message: > MDEV-532: Fix some race conditions in test cases. > > With MDEV-532, the binlog_checkpoint event is logged asynchronously > from a binlog background thread. This causes some sporadic failures > in some test cases whose output depends on order of events in > binlog. > > Fix using an include file that waits until the binlog checkpoint > event has been logged before proceeding with the test case. > === modified file 'mysql-test/extra/rpl_tests/rpl_insert_delayed.test' > --- a/mysql-test/extra/rpl_tests/rpl_insert_delayed.test 2012-09-22 14:11:40 +0000 > +++ b/mysql-test/extra/rpl_tests/rpl_insert_delayed.test 2012-12-17 11:49:11 +0000 > @@ -94,8 +94,10 @@ if (`SELECT @@global.binlog_format = 'S > #flush the logs before the test > connection slave; > FLUSH LOGS; > + source include/wait_for_binlog_checkpoint.inc; > connection master; > FLUSH LOGS; > + source include/wait_for_binlog_checkpoint.inc; > } > > CREATE TABLE t1(a int, UNIQUE(a)); > > === modified file 'mysql-test/extra/rpl_tests/rpl_log.test' > --- a/mysql-test/extra/rpl_tests/rpl_log.test 2011-10-19 19:45:18 +0000 > +++ b/mysql-test/extra/rpl_tests/rpl_log.test 2012-12-17 11:49:11 +0000 > @@ -43,6 +43,7 @@ let $binlog_limit= 1,4; > source include/show_binlog_events.inc; > let $binlog_limit=; > flush logs; > +--source include/wait_for_binlog_checkpoint.inc > > # We need an extra update before doing save_master_pos. > # Otherwise, an unlikely scenario may occur: > > === modified file 'mysql-test/extra/rpl_tests/rpl_show_relaylog_events.inc' > --- a/mysql-test/extra/rpl_tests/rpl_show_relaylog_events.inc 2011-01-13 14:31:37 +0000 > +++ b/mysql-test/extra/rpl_tests/rpl_show_relaylog_events.inc 2012-12-17 11:49:11 +0000 > @@ -41,8 +41,10 @@ INSERT INTO t1 VALUES (3); > # > > FLUSH LOGS; > +--source include/wait_for_binlog_checkpoint.inc > -- connection master > FLUSH LOGS; > +--source include/wait_for_binlog_checkpoint.inc > DROP TABLE t1; > > --let $is_relay_log= 0 > > === added file 'mysql-test/include/wait_for_binlog_checkpoint.inc' > --- a/mysql-test/include/wait_for_binlog_checkpoint.inc 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/include/wait_for_binlog_checkpoint.inc 2012-12-17 11:49:11 +0000 > @@ -0,0 +1,53 @@ > +# include/wait_for_binlog_checkpoint.inc > +# > +# SUMMARY > +# > +# Wait until binlog checkpoint has been logged for current binlog file. > +# This is useful to avoid races with output difference for binlog > +# checkpoints, as these are logged asynchronously from the binlog > +# background thread. > +# > +# USAGE: > +# > +# --source include/wait_for_binlog_checkpoint.inc > + > +let $_wait_count= 300; > + > +let $_found= 0; > + > +while ($_wait_count) > +{ > + dec $_wait_count; > + let $_cur_binlog= query_get_value(SHOW MASTER STATUS, File, 1); > + let $_more= 1; > + let $_row= 1; > + while ($_more) > + { > + let $_event= query_get_value(SHOW BINLOG EVENTS IN "$_cur_binlog", Event_type, $_row); > + if ($_event == "No such row") > + { > + let $_more= 0; > + } > + if ($_event == "Binlog_checkpoint") > + { > + let $_info= query_get_value(SHOW BINLOG EVENTS IN "$_cur_binlog", Info, $_row); > + if (`SELECT INSTR("$_info", "$_cur_binlog") != 0`) > + { > + let $_more= 0; > + let $_wait_count= 0; > + let $_found= 1; > + } > + } > + inc $_row; > + } > + if ($_wait_count) > + { > + real_sleep 0.1; > + } > +} > + > +if (!$_found) > +{ > + eval SHOW BINLOG EVENTS IN "$_cur_binlog"; > + --die ERROR: failed while waiting for binlog checkpoint $_cur_binlog > +} > > === modified file 'mysql-test/suite/innodb/t/binlog_consistent.test' > --- a/mysql-test/suite/innodb/t/binlog_consistent.test 2012-02-07 15:22:36 +0000 > +++ b/mysql-test/suite/innodb/t/binlog_consistent.test 2012-12-17 11:49:11 +0000 > @@ -72,6 +72,7 @@ connection con3; > --echo # Connection con3 > COMMIT; > FLUSH LOGS; > +--source include/wait_for_binlog_checkpoint.inc > > connection default; > --echo # Connection default > > === modified file 'mysql-test/suite/multi_source/multisource.test' > --- a/mysql-test/suite/multi_source/multisource.test 2012-10-04 20:36:17 +0000 > +++ b/mysql-test/suite/multi_source/multisource.test 2012-12-17 11:49:11 +0000 > @@ -169,6 +169,7 @@ select * from db2.t1; > > --connection master1 > flush logs; > +--source include/wait_for_binlog_checkpoint.inc > --save_master_pos > --connection slave > --sync_with_master 0, 'master1' > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] extended keys
by Igor Babaev 17 Dec '12

17 Dec '12

On 12/17/2012 06:43 AM, Rich Prohaska wrote: > What do I need to change in my storage engine for extended keys? > Where should I look? > Thanks. Hi, see the code /* Currently only InnoDB can use extended keys */ share->set_use_ext_keys_flag(legacy_db_type == DB_TYPE_INNODB); in open_binary_frm(), table.cc (mariadb) You have to change it. There is a similar code in mysql, also in open_binary_frm(),table.cc Regards, Igor. > > On Sun, Dec 16, 2012 at 10:58 PM, Igor Babaev <igor(a)askmonty.org> wrote: >> On 12/14/2012 11:33 AM, Rich Prohaska wrote: >>> MySQL 5.6.9 changed how the number of key parts are measured. They >>> replaced one simple counter (key_parts) with four counters. Glancing >>> at the code, i assume that this was done to support extended keys. If >>> so, the mariadb implementation of extended keys is a lot simpler. We >>> have to maintain the tokudb storage engine on both mysql and mariadb. >>> Is the mariadb design going to change to match the 5.6 implementation, >>> or are the designs now separate? >>> Thanks >>> Rich Prohaska >> >> Hi, >> >> MariaDB supports extended keys starting with first version of 5.5. >> See https://kb.askmonty.org/en/extended-keys/. >> >> See also blogs: >> >> http://igors-notes.blogspot.com/2011/12/3-way-join-that-touches-only-indexe… >> >> http://s.petrunia.net/blog/?p=74 >> >> The interesting thing is that Oracle essentially pulled MariaDB >> implementation for this feature into a mysql version that was in RC. >> Yet one hardly can find anything about the feature in the changelog or >> in the manual. >> >> Regards, >> Igor. >> >> >> >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~maria-developers >>> Post to : maria-developers(a)lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~maria-developers >>> More help : https://help.launchpad.net/ListHelp >>

1 0

Re: [Maria-developers] [Commits] Rev 3592: Test commit for diff options in file:///H:/bzr/5.5/
by Sergei Golubchik 17 Dec '12

17 Dec '12

Hi, Vladislav! On Dec 17, Vladislav Vaintroub wrote: > revno: 3592 > revision-id: wlad(a)montyprogram.com-20121217203725-f5e0mguvmjvb97su > committer: Vladislav Vaintroub <wlad(a)montyprogram.com> > branch nick: 5.5 > timestamp: Mon 2012-12-17 21:37:25 +0100 > message: > Test commit for diff options > === modified file 'sql/mysqld.cc' > --- a/sql/mysqld.cc 2012-11-22 09:19:31 +0000 > +++ b/sql/mysqld.cc 2012-12-17 20:37:25 +0000 > @@ -2441,21 +2441,6 @@ void dec_connection_count(THD *thd) The patch looks ok to me. But I'd prefer a better changeset comment :) Regards, Sergei

1 0

[Maria-developers] extended keys
by Rich Prohaska 17 Dec '12

17 Dec '12

MySQL 5.6.9 changed how the number of key parts are measured. They replaced one simple counter (key_parts) with four counters. Glancing at the code, i assume that this was done to support extended keys. If so, the mariadb implementation of extended keys is a lot simpler. We have to maintain the tokudb storage engine on both mysql and mariadb. Is the mariadb design going to change to match the 5.6 implementation, or are the designs now separate? Thanks Rich Prohaska

2 1

Re: [Maria-developers] [Commits] Rev 3435: MDEV-532: Async InnoDB commit checkpoint. in http://bazaar.launchpad.net/~maria-captains/maria/10.0
by Kristian Nielsen 14 Dec '12

14 Dec '12

Hi Serg, As we discussed under review of MDEV-232, here is a separate patch that makes InnoDB/XtraDB commit checkpointing be more asynchroneous. See MDEV-532 for further description of this task. I hope you will review this, at your convenience. - Kristian. knielsen(a)knielsen-hq.org writes: > At http://bazaar.launchpad.net/~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 3435 > revision-id: knielsen(a)knielsen-hq.org-20120914124453-zsap6hjclq3vrb6n > parent: knielsen(a)knielsen-hq.org-20120913123129-kaujy4cw0jc9o08k > committer: knielsen(a)knielsen-hq.org > branch nick: work-10.0-mdev225-181-232 > timestamp: Fri 2012-09-14 14:44:53 +0200 > message: > MDEV-532: Async InnoDB commit checkpoint. > > Make the commit checkpoint inside InnoDB be asynchroneous. > Implement a background thread in binlog to do the writing and flushing of > binlog checkpoint events to disk. > === modified file 'mysql-test/suite/binlog/r/binlog_checkpoint.result' > --- a/mysql-test/suite/binlog/r/binlog_checkpoint.result 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/r/binlog_checkpoint.result 2012-09-14 12:44:53 +0000 > @@ -70,8 +70,14 @@ show binlog events in 'master-bin.000003 > Log_name Pos Event_type Server_id End_log_pos Info > master-bin.000003 # Format_desc # # SERVER_VERSION, BINLOG_VERSION > master-bin.000003 # Binlog_checkpoint # # master-bin.000001 > +SET DEBUG_SYNC= "RESET"; > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > SET DEBUG_SYNC= "now SIGNAL con2_continue"; > con1 is still pending, no new binlog checkpoint should have been logged. > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > +SET DEBUG_SYNC= "RESET"; > show binlog events in 'master-bin.000003' from <binlog_start>; > Log_name Pos Event_type Server_id End_log_pos Info > master-bin.000003 # Format_desc # # SERVER_VERSION, BINLOG_VERSION > > === modified file 'mysql-test/suite/binlog/r/binlog_xa_recover.result' > --- a/mysql-test/suite/binlog/r/binlog_xa_recover.result 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/r/binlog_xa_recover.result 2012-09-14 12:44:53 +0000 > @@ -118,7 +118,11 @@ master-bin.00000<binlog_start> # Table_m > master-bin.00000<binlog_start> # Write_rows # # table_id: # flags: STMT_END_F > master-bin.00000<binlog_start> # Xid # # COMMIT /* XID */ > SET DEBUG_SYNC= "now SIGNAL con10_cont"; > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > SET DEBUG_SYNC= "now SIGNAL con12_cont"; > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > SET DEBUG_SYNC= "now SIGNAL con11_cont"; > Checking that master-bin.000004 is the last binlog checkpoint > show binlog events in 'master-bin.00000<binlog_start>' from <binlog_start>; > > === modified file 'mysql-test/suite/binlog/t/binlog_checkpoint.test' > --- a/mysql-test/suite/binlog/t/binlog_checkpoint.test 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/t/binlog_checkpoint.test 2012-09-14 12:44:53 +0000 > @@ -71,6 +71,12 @@ SET DEBUG_SYNC= "now WAIT_FOR con2_ready > --let $binlog_file= master-bin.000003 > --source include/show_binlog_events.inc > > +# We need to sync the test case with the background processing of the > +# commit checkpoint, otherwise we get nondeterministic results. > +SET DEBUG_SYNC= "RESET"; > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > + > SET DEBUG_SYNC= "now SIGNAL con2_continue"; > > connection con2; > @@ -78,6 +84,12 @@ reap; > > connection default; > --echo con1 is still pending, no new binlog checkpoint should have been logged. > +# Make sure commit checkpoint is processed before we check that no checkpoint > +# event has been binlogged. > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > +SET DEBUG_SYNC= "RESET"; > + > --let $binlog_file= master-bin.000003 > --source include/show_binlog_events.inc > > > === modified file 'mysql-test/suite/binlog/t/binlog_xa_recover.test' > --- a/mysql-test/suite/binlog/t/binlog_xa_recover.test 2012-09-13 12:31:29 +0000 > +++ b/mysql-test/suite/binlog/t/binlog_xa_recover.test 2012-09-14 12:44:53 +0000 > @@ -14,8 +14,24 @@ CREATE TABLE t1 (a INT PRIMARY KEY, b ME > # Insert some data to force a couple binlog rotations (3), so we get some > # normal binlog checkpoints before starting the test. > INSERT INTO t1 VALUES (100, REPEAT("x", 4100)); > +# Wait for the master-bin.000002 binlog checkpoint to appear. > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000002" > +--let $field= Info > +--let $condition= = "master-bin.000002" > +--source include/wait_show_condition.inc > INSERT INTO t1 VALUES (101, REPEAT("x", 4100)); > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000003" > +--let $field= Info > +--let $condition= = "master-bin.000003" > +--source include/wait_show_condition.inc > INSERT INTO t1 VALUES (102, REPEAT("x", 4100)); > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000004" > +--let $field= Info > +--let $condition= = "master-bin.000004" > +--source include/wait_show_condition.inc > > # Now start a bunch of transactions that span multiple binlog > # files. Leave then in the state prepared-but-not-committed in the engine > @@ -153,10 +169,19 @@ SET DEBUG_SYNC= "now SIGNAL con10_cont"; > connection con10; > reap; > connection default; > + > +# We need to sync the test case with the background processing of the > +# commit checkpoint, otherwise we get nondeterministic results. > +SET @old_dbug= @@global.DEBUG_DBUG; > +SET GLOBAL debug_dbug="+d,binlog_background_checkpoint_processed"; > + > SET DEBUG_SYNC= "now SIGNAL con12_cont"; > connection con12; > reap; > connection default; > +SET DEBUG_SYNC= "now WAIT_FOR binlog_background_checkpoint_processed"; > +SET GLOBAL debug_dbug= @old_dbug; > + > SET DEBUG_SYNC= "now SIGNAL con11_cont"; > connection con11; > reap; > @@ -210,7 +235,20 @@ RESET MASTER; > # crash recovery fails due to the error insert used for previous test. > INSERT INTO t1 VALUES (21, REPEAT("x", 4100)); > INSERT INTO t1 VALUES (22, REPEAT("x", 4100)); > +# Wait for the master-bin.000003 binlog checkpoint to appear. > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000003" > +--let $field= Info > +--let $condition= = "master-bin.000003" > +--source include/wait_show_condition.inc > INSERT INTO t1 VALUES (23, REPEAT("x", 4100)); > +# Wait for the last (master-bin.000004) binlog checkpoint to appear. > +--let $wait_for_all= 0 > +--let $show_statement= SHOW BINLOG EVENTS IN "master-bin.000004" > +--let $field= Info > +--let $condition= = "master-bin.000004" > +--source include/wait_show_condition.inc > + > --write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > wait-binlog_xa_recover.test > EOF > > === modified file 'mysql-test/suite/perfschema/r/all_instances.result' > --- a/mysql-test/suite/perfschema/r/all_instances.result 2012-06-22 09:46:28 +0000 > +++ b/mysql-test/suite/perfschema/r/all_instances.result 2012-09-14 12:44:53 +0000 > @@ -76,6 +76,7 @@ wait/synch/mutex/sql/Master_info::run_lo > wait/synch/mutex/sql/Master_info::sleep_lock > wait/synch/mutex/sql/MDL_map::mutex > wait/synch/mutex/sql/MDL_wait::LOCK_wait_status > +wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_binlog_thread > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_index > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_xid_list > wait/synch/mutex/sql/MYSQL_RELAY_LOG::LOCK_index > @@ -129,6 +130,8 @@ wait/synch/cond/sql/Master_info::sleep_c > wait/synch/cond/sql/Master_info::start_cond > wait/synch/cond/sql/Master_info::stop_cond > wait/synch/cond/sql/MDL_context::COND_wait_status > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread_end > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_queue_busy > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list > wait/synch/cond/sql/MYSQL_BIN_LOG::update_cond > > === modified file 'mysql-test/suite/perfschema/r/relaylog.result' > --- a/mysql-test/suite/perfschema/r/relaylog.result 2012-06-22 09:46:28 +0000 > +++ b/mysql-test/suite/perfschema/r/relaylog.result 2012-09-14 12:44:53 +0000 > @@ -56,8 +56,11 @@ where event_name like "%MYSQL_BIN_LOG%" > and event_name not like "%MYSQL_BIN_LOG::update_cond" > order by event_name; > EVENT_NAME COUNT_STAR > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread NONE > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread_end NONE > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_queue_busy NONE > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list NONE > +wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_binlog_thread MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_index MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_xid_list MANY > "Expect no slave relay log" > @@ -131,8 +134,11 @@ where event_name like "%MYSQL_BIN_LOG%" > and event_name not like "%MYSQL_BIN_LOG::update_cond" > order by event_name; > EVENT_NAME COUNT_STAR > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread MANY > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_binlog_thread_end NONE > wait/synch/cond/sql/MYSQL_BIN_LOG::COND_queue_busy NONE > -wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list NONE > +wait/synch/cond/sql/MYSQL_BIN_LOG::COND_xid_list MANY > +wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_binlog_thread MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_index MANY > wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_xid_list MANY > "Expect a slave relay log" > > === modified file 'sql/debug_sync.cc' > --- a/sql/debug_sync.cc 2012-03-28 17:26:00 +0000 > +++ b/sql/debug_sync.cc 2012-09-14 12:44:53 +0000 > @@ -984,6 +984,7 @@ static bool debug_sync_eval_action(THD * > DBUG_ENTER("debug_sync_eval_action"); > DBUG_ASSERT(thd); > DBUG_ASSERT(action_str); > + DBUG_PRINT("debug_sync", ("action_str='%s'", action_str)); > > /* > Get debug sync point name. Or a special command. > > === modified file 'sql/log.cc' > --- a/sql/log.cc 2012-09-13 12:31:29 +0000 > +++ b/sql/log.cc 2012-09-14 12:44:53 +0000 > @@ -53,6 +53,7 @@ > #include "rpl_handler.h" > #include "debug_sync.h" > #include "sql_show.h" > +#include "my_pthread.h" > > /* max size of the log message */ > #define MAX_LOG_BUFFER_SIZE 1024 > @@ -106,6 +107,14 @@ static SHOW_VAR binlog_status_vars_detai > {NullS, NullS, SHOW_LONG} > }; > > +/* Variables for the binlog background thread. */ > +static bool binlog_thread_started= false; > +static bool binlog_background_thread_stop= false; > +static MYSQL_BIN_LOG::xid_count_per_binlog * > + binlog_background_thread_queue= NULL; > + > +static bool start_binlog_background_thread(); > + > > /** > purge logs, master and slave sides both, related error code > @@ -2957,12 +2966,27 @@ void MYSQL_BIN_LOG::cleanup() > my_free(b); > } > > + /* Wait for the binlog thread to stop. */ > + if (!is_relay_log && binlog_thread_started) > + { > + mysql_mutex_lock(&LOCK_binlog_thread); > + binlog_background_thread_stop= true; > + mysql_cond_signal(&COND_binlog_thread); > + while (binlog_background_thread_stop) > + mysql_cond_wait(&COND_binlog_thread_end, &LOCK_binlog_thread); > + mysql_mutex_unlock(&LOCK_binlog_thread); > + binlog_thread_started= false; > + } > + > mysql_mutex_destroy(&LOCK_log); > mysql_mutex_destroy(&LOCK_index); > mysql_mutex_destroy(&LOCK_xid_list); > + mysql_mutex_destroy(&LOCK_binlog_thread); > mysql_cond_destroy(&update_cond); > mysql_cond_destroy(&COND_queue_busy); > mysql_cond_destroy(&COND_xid_list); > + mysql_cond_destroy(&COND_binlog_thread); > + mysql_cond_destroy(&COND_binlog_thread_end); > } > DBUG_VOID_RETURN; > } > @@ -2988,6 +3012,11 @@ void MYSQL_BIN_LOG::init_pthread_objects > mysql_cond_init(m_key_update_cond, &update_cond, 0); > mysql_cond_init(m_key_COND_queue_busy, &COND_queue_busy, 0); > mysql_cond_init(key_BINLOG_COND_xid_list, &COND_xid_list, 0); > + > + mysql_mutex_init(key_BINLOG_LOCK_binlog_thread, > + &LOCK_binlog_thread, MY_MUTEX_INIT_FAST); > + mysql_cond_init(key_BINLOG_COND_binlog_thread, &COND_binlog_thread, 0); > + mysql_cond_init(key_BINLOG_COND_binlog_thread_end, &COND_binlog_thread_end, 0); > } > > > @@ -3085,6 +3114,10 @@ bool MYSQL_BIN_LOG::open(const char *log > DBUG_ENTER("MYSQL_BIN_LOG::open"); > DBUG_PRINT("enter",("log_type: %d",(int) log_type_arg)); > > + if (!is_relay_log && !binlog_thread_started && > + start_binlog_background_thread()) > + DBUG_RETURN(1); > + > if (init_and_set_log_file_name(log_name, new_name, log_type_arg, > io_cache_type_arg)) > { > @@ -5540,11 +5573,7 @@ bool general_log_write(THD *thd, enum en > } > > > -/* > - I would like to make this function static, but this causes compiler warnings > - when it is declared as friend function in log.h. > -*/ > -void > +static void > binlog_checkpoint_callback(void *cookie) > { > MYSQL_BIN_LOG::xid_count_per_binlog *entry= > @@ -8116,9 +8145,128 @@ int TC_LOG_BINLOG::unlog(ulong cookie, m > void > TC_LOG_BINLOG::commit_checkpoint_notify(void *cookie) > { > - mark_xid_done(((xid_count_per_binlog *)cookie)->binlog_id, true); > + xid_count_per_binlog *entry= static_cast<xid_count_per_binlog *>(cookie); > + mysql_mutex_lock(&LOCK_binlog_thread); > + entry->next_in_queue= binlog_background_thread_queue; > + binlog_background_thread_queue= entry; > + mysql_cond_signal(&COND_binlog_thread); > + mysql_mutex_unlock(&LOCK_binlog_thread); > } > > +/* > + Binlog service thread. > + > + This thread is used to log binlog checkpoints in the background, rather than > + in the context of random storage engine threads that happen to call > + commit_checkpoint_notify_ha() and may not like the delays while syncing > + binlog to disk or may not be setup with all my_thread_init() and other > + necessary stuff. > + > + In the future, this thread could also be used to do log rotation in the > + background, which could elimiate all stalls around binlog rotations. > +*/ > +pthread_handler_t > +binlog_background_thread(void *arg __attribute__((unused))) > +{ > + bool stop; > + MYSQL_BIN_LOG::xid_count_per_binlog *queue, *next; > + THD *thd; > + > + my_thread_init(); > + thd= new THD; > + thd->system_thread= SYSTEM_THREAD_BINLOG_BACKGROUND; > + my_pthread_setspecific_ptr(THR_THD, thd); > + mysql_mutex_lock(&LOCK_thread_count); > + thd->thread_id= thread_id++; > + mysql_mutex_unlock(&LOCK_thread_count); > + > + for (;;) > + { > + /* > + Wait until there is something in the queue to process, or we are asked > + to shut down. > + */ > + thd_proc_info(thd, "Waiting for background binlog tasks"); > + mysql_mutex_lock(&mysql_bin_log.LOCK_binlog_thread); > + for (;;) > + { > + stop= binlog_background_thread_stop; > + queue= binlog_background_thread_queue; > + if (stop || queue) > + break; > + mysql_cond_wait(&mysql_bin_log.COND_binlog_thread, > + &mysql_bin_log.LOCK_binlog_thread); > + } > + /* Grab the queue, if any. */ > + binlog_background_thread_queue= NULL; > + mysql_mutex_unlock(&mysql_bin_log.LOCK_binlog_thread); > + > + /* Process any incoming commit_checkpoint_notify() calls. */ > + while (queue) > + { > + thd_proc_info(thd, "Processing binlog checkpoint notification"); > + /* Grab next pointer first, as mark_xid_done() may free the element. */ > + next= queue->next_in_queue; > + mysql_bin_log.mark_xid_done(queue->binlog_id, true); > + queue= next; > + > + DBUG_EXECUTE_IF("binlog_background_checkpoint_processed", > + DBUG_ASSERT(!debug_sync_set_action( > + thd, > + STRING_WITH_LEN("now SIGNAL binlog_background_checkpoint_processed"))); > + ); > + } > + > + if (stop) > + break; > + } > + > + thd_proc_info(thd, "Stopping binlog background thread"); > + > + mysql_mutex_lock(&LOCK_thread_count); > + delete thd; > + mysql_mutex_unlock(&LOCK_thread_count); > + > + my_thread_end(); > + > + /* Signal that we are (almost) stopped. */ > + mysql_mutex_lock(&mysql_bin_log.LOCK_binlog_thread); > + binlog_background_thread_stop= false; > + mysql_cond_signal(&mysql_bin_log.COND_binlog_thread_end); > + mysql_mutex_unlock(&mysql_bin_log.LOCK_binlog_thread); > + > + return 0; > +} > + > +#ifdef HAVE_PSI_INTERFACE > +static PSI_thread_key key_thread_binlog; > + > +static PSI_thread_info all_binlog_threads[]= > +{ > + { &key_thread_binlog, "binlog_background", PSI_FLAG_GLOBAL}, > +}; > +#endif /* HAVE_PSI_INTERFACE */ > + > +static bool > +start_binlog_background_thread() > +{ > + pthread_t th; > + > +#ifdef HAVE_PSI_INTERFACE > + if (PSI_server) > + PSI_server->register_thread("sql", all_binlog_threads, > + array_elements(all_binlog_threads)); > +#endif > + > + if (mysql_thread_create(key_thread_binlog, &th, NULL, > + binlog_background_thread, NULL)) > + return 1; > + > + binlog_thread_started= true; > + return 0; > +} > + > + > int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > IO_CACHE *first_log, > Format_description_log_event *fdle) > > === modified file 'sql/log.h' > --- a/sql/log.h 2012-09-13 12:31:29 +0000 > +++ b/sql/log.h 2012-09-14 12:44:53 +0000 > @@ -395,8 +395,6 @@ class MYSQL_QUERY_LOG: public MYSQL_LOG > #define BINLOG_COOKIE_IS_DUMMY(c) \ > ( ((ulong)(c)>>1) == BINLOG_COOKIE_DUMMY_ID ) > > -void binlog_checkpoint_callback(void *cookie); > - > class binlog_cache_mngr; > class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG > { > @@ -451,27 +449,6 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > }; > > /* > - A list of struct xid_count_per_binlog is used to keep track of how many > - XIDs are in prepared, but not committed, state in each binlog. And how > - many commit_checkpoint_request()'s are pending. > - > - When count drops to zero in a binlog after rotation, it means that there > - are no more XIDs in prepared state, so that binlog is no longer needed > - for XA crash recovery, and we can log a new binlog checkpoint event. > - > - The list is protected against simultaneous access from multiple > - threads by LOCK_xid_list. > - */ > - struct xid_count_per_binlog : public ilink { > - char *binlog_name; > - uint binlog_name_len; > - ulong binlog_id; > - /* Total prepared XIDs and pending checkpoint requests in this binlog. */ > - long xid_count; > - xid_count_per_binlog(); /* Give link error if constructor used. */ > - }; > - I_List<xid_count_per_binlog> binlog_xid_count_list; > - /* > When this is set, a RESET MASTER is in progress. > > Then we should not write any binlog checkpoints into the binlog (that > @@ -480,7 +457,6 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > checkpoint arrives - when all have arrived, RESET MASTER will complete. > */ > bool reset_master_pending; > - friend void binlog_checkpoint_callback(void *cookie); > > /* LOCK_log and LOCK_index are inited by init_pthread_objects() */ > mysql_mutex_t LOCK_index; > @@ -553,10 +529,35 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > int write_transaction_or_stmt(group_commit_entry *entry); > bool write_transaction_to_binlog_events(group_commit_entry *entry); > void trx_group_commit_leader(group_commit_entry *leader); > - void mark_xid_done(ulong cookie, bool write_checkpoint); > - void mark_xids_active(ulong cookie, uint xid_count); > > public: > + /* > + A list of struct xid_count_per_binlog is used to keep track of how many > + XIDs are in prepared, but not committed, state in each binlog. And how > + many commit_checkpoint_request()'s are pending. > + > + When count drops to zero in a binlog after rotation, it means that there > + are no more XIDs in prepared state, so that binlog is no longer needed > + for XA crash recovery, and we can log a new binlog checkpoint event. > + > + The list is protected against simultaneous access from multiple > + threads by LOCK_xid_list. > + */ > + struct xid_count_per_binlog : public ilink { > + char *binlog_name; > + uint binlog_name_len; > + ulong binlog_id; > + /* Total prepared XIDs and pending checkpoint requests in this binlog. */ > + long xid_count; > + /* For linking in requests to the binlog background thread. */ > + xid_count_per_binlog *next_in_queue; > + xid_count_per_binlog(); /* Give link error if constructor used. */ > + }; > + I_List<xid_count_per_binlog> binlog_xid_count_list; > + mysql_mutex_t LOCK_binlog_thread; > + mysql_cond_t COND_binlog_thread; > + mysql_cond_t COND_binlog_thread_end; > + > using MYSQL_LOG::generate_name; > using MYSQL_LOG::is_open; > > @@ -712,6 +713,8 @@ class MYSQL_BIN_LOG: public TC_LOG, priv > bool appendv(const char* buf,uint len,...); > bool append(Log_event* ev); > > + void mark_xids_active(ulong cookie, uint xid_count); > + void mark_xid_done(ulong cookie, bool write_checkpoint); > void make_log_name(char* buf, const char* log_ident); > bool is_active(const char* log_file_name); > bool can_purge_log(const char *log_file_name); > > === modified file 'sql/mysqld.cc' > --- a/sql/mysqld.cc 2012-09-13 12:31:29 +0000 > +++ b/sql/mysqld.cc 2012-09-14 12:44:53 +0000 > @@ -724,6 +724,7 @@ PSI_mutex_key key_LOCK_des_key_file; > #endif /* HAVE_OPENSSL */ > > PSI_mutex_key key_BINLOG_LOCK_index, key_BINLOG_LOCK_xid_list, > + key_BINLOG_LOCK_binlog_thread, > key_delayed_insert_mutex, key_hash_filo_lock, key_LOCK_active_mi, > key_LOCK_connection_count, key_LOCK_crypt, key_LOCK_delayed_create, > key_LOCK_delayed_insert, key_LOCK_delayed_status, key_LOCK_error_log, > @@ -766,6 +767,7 @@ static PSI_mutex_info all_server_mutexes > > { &key_BINLOG_LOCK_index, "MYSQL_BIN_LOG::LOCK_index", 0}, > { &key_BINLOG_LOCK_xid_list, "MYSQL_BIN_LOG::LOCK_xid_list", 0}, > + { &key_BINLOG_LOCK_binlog_thread, "MYSQL_BIN_LOG::LOCK_binlog_thread", 0}, > { &key_RELAYLOG_LOCK_index, "MYSQL_RELAY_LOG::LOCK_index", 0}, > { &key_delayed_insert_mutex, "Delayed_insert::mutex", 0}, > { &key_hash_filo_lock, "hash_filo::lock", 0}, > @@ -834,6 +836,7 @@ PSI_cond_key key_PAGE_cond, key_COND_act > #endif /* HAVE_MMAP */ > > PSI_cond_key key_BINLOG_COND_xid_list, key_BINLOG_update_cond, > + key_BINLOG_COND_binlog_thread, key_BINLOG_COND_binlog_thread_end, > key_COND_cache_status_changed, key_COND_manager, > key_COND_rpl_status, key_COND_server_started, > key_delayed_insert_cond, key_delayed_insert_cond_client, > @@ -863,6 +866,8 @@ static PSI_cond_info all_server_conds[]= > #endif /* HAVE_MMAP */ > { &key_BINLOG_COND_xid_list, "MYSQL_BIN_LOG::COND_xid_list", 0}, > { &key_BINLOG_update_cond, "MYSQL_BIN_LOG::update_cond", 0}, > + { &key_BINLOG_COND_binlog_thread, "MYSQL_BIN_LOG::COND_binlog_thread", 0}, > + { &key_BINLOG_COND_binlog_thread_end, "MYSQL_BIN_LOG::COND_binlog_thread_end", 0}, > { &key_BINLOG_COND_queue_busy, "MYSQL_BIN_LOG::COND_queue_busy", 0}, > { &key_RELAYLOG_update_cond, "MYSQL_RELAY_LOG::update_cond", 0}, > { &key_RELAYLOG_COND_queue_busy, "MYSQL_RELAY_LOG::COND_queue_busy", 0}, > > === modified file 'sql/mysqld.h' > --- a/sql/mysqld.h 2012-09-13 12:31:29 +0000 > +++ b/sql/mysqld.h 2012-09-14 12:44:53 +0000 > @@ -226,6 +226,7 @@ extern PSI_mutex_key key_LOCK_des_key_fi > #endif > > extern PSI_mutex_key key_BINLOG_LOCK_index, key_BINLOG_LOCK_xid_list, > + key_BINLOG_LOCK_binlog_thread, > key_delayed_insert_mutex, key_hash_filo_lock, key_LOCK_active_mi, > key_LOCK_connection_count, key_LOCK_crypt, key_LOCK_delayed_create, > key_LOCK_delayed_insert, key_LOCK_delayed_status, key_LOCK_error_log, > @@ -257,6 +258,7 @@ extern PSI_cond_key key_PAGE_cond, key_C > #endif /* HAVE_MMAP */ > > extern PSI_cond_key key_BINLOG_COND_xid_list, key_BINLOG_update_cond, > + key_BINLOG_COND_binlog_thread, key_BINLOG_COND_binlog_thread_end, > key_COND_cache_status_changed, key_COND_manager, > key_COND_rpl_status, key_COND_server_started, > key_delayed_insert_cond, key_delayed_insert_cond_client, > > === modified file 'sql/rpl_rli.cc' > --- a/sql/rpl_rli.cc 2012-09-13 12:31:29 +0000 > +++ b/sql/rpl_rli.cc 2012-09-14 12:44:53 +0000 > @@ -58,6 +58,7 @@ Relay_log_info::Relay_log_info(bool is_s > { > DBUG_ENTER("Relay_log_info::Relay_log_info"); > > + relay_log.is_relay_log= TRUE; > #ifdef HAVE_PSI_INTERFACE > relay_log.set_psi_keys(key_RELAYLOG_LOCK_index, > key_RELAYLOG_update_cond, > @@ -206,8 +207,6 @@ a file name for --relay-log-index option > name_warning_sent= 1; > } > > - rli->relay_log.is_relay_log= TRUE; > - > /* > note, that if open() fails, we'll still have index file open > but a destructor will take care of that > > === modified file 'sql/sql_class.h' > --- a/sql/sql_class.h 2012-09-13 12:31:29 +0000 > +++ b/sql/sql_class.h 2012-09-14 12:44:53 +0000 > @@ -1244,7 +1244,8 @@ enum enum_thread_type > SYSTEM_THREAD_SLAVE_SQL= 4, > SYSTEM_THREAD_NDBCLUSTER_BINLOG= 8, > SYSTEM_THREAD_EVENT_SCHEDULER= 16, > - SYSTEM_THREAD_EVENT_WORKER= 32 > + SYSTEM_THREAD_EVENT_WORKER= 32, > + SYSTEM_THREAD_BINLOG_BACKGROUND= 64 > }; > > inline char const * > > === modified file 'storage/innobase/handler/ha_innodb.cc' > --- a/storage/innobase/handler/ha_innodb.cc 2012-09-13 12:31:29 +0000 > +++ b/storage/innobase/handler/ha_innodb.cc 2012-09-14 12:44:53 +0000 > @@ -106,6 +106,7 @@ static ulong commit_threads = 0; > static mysql_mutex_t commit_threads_m; > static mysql_cond_t commit_cond; > static mysql_mutex_t commit_cond_m; > +static mysql_mutex_t pending_checkpoint_mutex; > static bool innodb_inited = 0; > > #define INSIDE_HA_INNOBASE_CC > @@ -222,11 +223,13 @@ static mysql_pfs_key_t innobase_share_mu > static mysql_pfs_key_t commit_threads_m_key; > static mysql_pfs_key_t commit_cond_mutex_key; > static mysql_pfs_key_t commit_cond_key; > +static mysql_pfs_key_t pending_checkpoint_mutex_key; > > static PSI_mutex_info all_pthread_mutexes[] = { > {&commit_threads_m_key, "commit_threads_m", 0}, > {&commit_cond_mutex_key, "commit_cond_mutex", 0}, > - {&innobase_share_mutex_key, "innobase_share_mutex", 0} > + {&innobase_share_mutex_key, "innobase_share_mutex", 0}, > + {&pending_checkpoint_mutex_key, "pending_checkpoint_mutex", 0} > }; > > static PSI_cond_info all_innodb_conds[] = { > @@ -2592,6 +2595,9 @@ innobase_init( > mysql_mutex_init(commit_cond_mutex_key, > &commit_cond_m, MY_MUTEX_INIT_FAST); > mysql_cond_init(commit_cond_key, &commit_cond, NULL); > + mysql_mutex_init(pending_checkpoint_mutex_key, > + &pending_checkpoint_mutex, > + MY_MUTEX_INIT_FAST); > innodb_inited= 1; > #ifdef MYSQL_DYNAMIC_PLUGIN > if (innobase_hton != p) { > @@ -2639,6 +2645,7 @@ innobase_end( > mysql_mutex_destroy(&commit_threads_m); > mysql_mutex_destroy(&commit_cond_m); > mysql_cond_destroy(&commit_cond); > + mysql_mutex_destroy(&pending_checkpoint_mutex); > } > > DBUG_RETURN(err); > @@ -3008,6 +3015,16 @@ innobase_rollback_trx( > DBUG_RETURN(convert_error_code_to_mysql(error, 0, NULL)); > } > > + > +struct pending_checkpoint { > + struct pending_checkpoint *next; > + handlerton *hton; > + void *cookie; > + ib_uint64_t lsn; > +}; > +static struct pending_checkpoint *pending_checkpoint_list; > +static struct pending_checkpoint *pending_checkpoint_list_end; > + > /*****************************************************************//** > Handle a commit checkpoint request from server layer. > We simply flush the redo log immediately and do the notify call.*/ > @@ -3017,8 +3034,113 @@ innobase_checkpoint_request( > handlerton *hton, > void *cookie) > { > - log_buffer_flush_to_disk(); > - commit_checkpoint_notify_ha(hton, cookie); > + ib_uint64_t lsn; > + ib_uint64_t flush_lsn; > + struct pending_checkpoint * entry; > + > + /* Do the allocation outside of lock to reduce contention. The normal > + case is that not everything is flushed, so we will need to enqueue. */ > + entry = static_cast<struct pending_checkpoint *> > + (my_malloc(sizeof(*entry), MYF(MY_WME))); > + if (!entry) { > + sql_print_error("Failed to allocate %u bytes." > + " Commit checkpoint will be skipped.", > + static_cast<unsigned>(sizeof(*entry))); > + return; > + } > + > + entry->next = NULL; > + entry->hton = hton; > + entry->cookie = cookie; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + lsn = log_get_lsn(); > + flush_lsn = log_get_flush_lsn(); > + if (lsn > flush_lsn) { > + /* Put the request in queue. > + When the log gets flushed past the lsn, we will remove the > + entry from the queue and notify the upper layer. */ > + entry->lsn = lsn; > + if (pending_checkpoint_list_end) { > + pending_checkpoint_list_end->next = entry; > + } else { > + pending_checkpoint_list = entry; > + } > + pending_checkpoint_list_end = entry; > + entry = NULL; > + } > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (entry) { > + /* We are already flushed. Notify the checkpoint immediately. */ > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + my_free(entry); > + } > +} > + > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +extern "C" UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn) /*!< in: LSN flushed to disk */ > +{ > + struct pending_checkpoint * pending; > + struct pending_checkpoint * entry; > + struct pending_checkpoint * last_ready; > + > + /* It is safe to do a quick check for NULL first without lock. > + Even if we should race, we will at most skip one checkpoint and > + take the next one, which is harmless. */ > + if (!pending_checkpoint_list) > + return; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + pending = pending_checkpoint_list; > + if (!pending) > + { > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + return; > + } > + > + last_ready = NULL; > + for (entry = pending; entry != NULL; entry = entry -> next) > + { > + if (entry->lsn > flush_lsn) > + break; > + last_ready = entry; > + } > + > + if (last_ready) > + { > + /* We found some pending checkpoints that are now flushed to > + disk. So remove them from the list. */ > + pending_checkpoint_list = entry; > + if (!entry) > + pending_checkpoint_list_end = NULL; > + } > + > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (!last_ready) > + return; > + > + /* Now that we have released the lock, notify upper layer about all > + commit checkpoints that have now completed. */ > + for (;;) { > + entry = pending; > + pending = pending->next; > + > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + > + my_free(entry); > + if (entry == last_ready) > + break; > + } > } > > /*****************************************************************//** > > === modified file 'storage/innobase/include/ha_prototypes.h' > --- a/storage/innobase/include/ha_prototypes.h 2011-04-26 17:55:52 +0000 > +++ b/storage/innobase/include/ha_prototypes.h 2012-09-14 12:44:53 +0000 > @@ -136,6 +136,17 @@ innobase_mysql_print_thd( > uint max_query_len); /*!< in: max query length to print, or 0 to > use the default max length */ > > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn); /*!< in: LSN flushed to disk */ > + > /**************************************************************//** > Converts a MySQL type to an InnoDB type. Note that this function returns > the 'mtype' of InnoDB. InnoDB differentiates between MySQL's old <= 4.1 > > === modified file 'storage/innobase/include/log0log.h' > --- a/storage/innobase/include/log0log.h 2012-06-07 13:44:26 +0000 > +++ b/storage/innobase/include/log0log.h 2012-09-14 12:44:53 +0000 > @@ -151,6 +151,13 @@ UNIV_INLINE > ib_uint64_t > log_get_lsn(void); > /*=============*/ > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void); > +/*=============*/ > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/innobase/include/log0log.ic' > --- a/storage/innobase/include/log0log.ic 2011-04-05 07:18:43 +0000 > +++ b/storage/innobase/include/log0log.ic 2012-09-14 12:44:53 +0000 > @@ -411,6 +411,25 @@ log_get_lsn(void) > return(lsn); > } > > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void) > +/*=============*/ > +{ > + ib_uint64_t lsn; > + > + mutex_enter(&(log_sys->mutex)); > + > + lsn = log_sys->flushed_to_disk_lsn; > + > + mutex_exit(&(log_sys->mutex)); > + > + return(lsn); > +} > + > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/innobase/log/log0log.c' > --- a/storage/innobase/log/log0log.c 2012-03-21 03:48:12 +0000 > +++ b/storage/innobase/log/log0log.c 2012-09-14 12:44:53 +0000 > @@ -1353,6 +1353,8 @@ log_write_up_to( > ulint loop_count = 0; > #endif /* UNIV_DEBUG */ > ulint unlock; > + ib_uint64_t write_lsn; > + ib_uint64_t flush_lsn; > > if (recv_no_ibuf_operations) { > /* Recovery is running and no operations on the log files are > @@ -1530,8 +1532,13 @@ log_write_up_to( > > log_flush_do_unlocks(unlock); > > + write_lsn = log_sys->write_lsn; > + flush_lsn = log_sys->flushed_to_disk_lsn; > + > mutex_exit(&(log_sys->mutex)); > > + innobase_mysql_log_notify(write_lsn, flush_lsn); > + > return; > > do_waits: > > === modified file 'storage/xtradb/handler/ha_innodb.cc' > --- a/storage/xtradb/handler/ha_innodb.cc 2012-09-13 12:31:29 +0000 > +++ b/storage/xtradb/handler/ha_innodb.cc 2012-09-14 12:44:53 +0000 > @@ -120,6 +120,7 @@ static ulong commit_threads = 0; > static mysql_mutex_t commit_threads_m; > static mysql_cond_t commit_cond; > static mysql_mutex_t commit_cond_m; > +static mysql_mutex_t pending_checkpoint_mutex; > static bool innodb_inited = 0; > > > @@ -253,11 +254,13 @@ static mysql_pfs_key_t innobase_share_mu > static mysql_pfs_key_t commit_threads_m_key; > static mysql_pfs_key_t commit_cond_mutex_key; > static mysql_pfs_key_t commit_cond_key; > +static mysql_pfs_key_t pending_checkpoint_mutex_key; > > static PSI_mutex_info all_pthread_mutexes[] = { > {&commit_threads_m_key, "commit_threads_m", 0}, > {&commit_cond_mutex_key, "commit_cond_mutex", 0}, > - {&innobase_share_mutex_key, "innobase_share_mutex", 0} > + {&innobase_share_mutex_key, "innobase_share_mutex", 0}, > + {&pending_checkpoint_mutex_key, "pending_checkpoint_mutex", 0} > }; > > static PSI_cond_info all_innodb_conds[] = { > @@ -3060,6 +3063,9 @@ innobase_init( > mysql_mutex_init(commit_cond_mutex_key, > &commit_cond_m, MY_MUTEX_INIT_FAST); > mysql_cond_init(commit_cond_key, &commit_cond, NULL); > + mysql_mutex_init(pending_checkpoint_mutex_key, > + &pending_checkpoint_mutex, > + MY_MUTEX_INIT_FAST); > innodb_inited= 1; > #ifdef MYSQL_DYNAMIC_PLUGIN > if (innobase_hton != p) { > @@ -3107,6 +3113,7 @@ innobase_end( > mysql_mutex_destroy(&commit_threads_m); > mysql_mutex_destroy(&commit_cond_m); > mysql_cond_destroy(&commit_cond); > + mysql_mutex_destroy(&pending_checkpoint_mutex); > } > > DBUG_RETURN(err); > @@ -3500,6 +3507,16 @@ innobase_rollback_trx( > DBUG_RETURN(convert_error_code_to_mysql(error, 0, NULL)); > } > > + > +struct pending_checkpoint { > + struct pending_checkpoint *next; > + handlerton *hton; > + void *cookie; > + ib_uint64_t lsn; > +}; > +static struct pending_checkpoint *pending_checkpoint_list; > +static struct pending_checkpoint *pending_checkpoint_list_end; > + > /*****************************************************************//** > Handle a commit checkpoint request from server layer. > We simply flush the redo log immediately and do the notify call.*/ > @@ -3509,8 +3526,113 @@ innobase_checkpoint_request( > handlerton *hton, > void *cookie) > { > - log_buffer_flush_to_disk(); > - commit_checkpoint_notify_ha(hton, cookie); > + ib_uint64_t lsn; > + ib_uint64_t flush_lsn; > + struct pending_checkpoint * entry; > + > + /* Do the allocation outside of lock to reduce contention. The normal > + case is that not everything is flushed, so we will need to enqueue. */ > + entry = static_cast<struct pending_checkpoint *> > + (my_malloc(sizeof(*entry), MYF(MY_WME))); > + if (!entry) { > + sql_print_error("Failed to allocate %u bytes." > + " Commit checkpoint will be skipped.", > + static_cast<unsigned>(sizeof(*entry))); > + return; > + } > + > + entry->next = NULL; > + entry->hton = hton; > + entry->cookie = cookie; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + lsn = log_get_lsn(); > + flush_lsn = log_get_flush_lsn(); > + if (lsn > flush_lsn) { > + /* Put the request in queue. > + When the log gets flushed past the lsn, we will remove the > + entry from the queue and notify the upper layer. */ > + entry->lsn = lsn; > + if (pending_checkpoint_list_end) { > + pending_checkpoint_list_end->next = entry; > + } else { > + pending_checkpoint_list = entry; > + } > + pending_checkpoint_list_end = entry; > + entry = NULL; > + } > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (entry) { > + /* We are already flushed. Notify the checkpoint immediately. */ > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + my_free(entry); > + } > +} > + > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +extern "C" UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn) /*!< in: LSN flushed to disk */ > +{ > + struct pending_checkpoint * pending; > + struct pending_checkpoint * entry; > + struct pending_checkpoint * last_ready; > + > + /* It is safe to do a quick check for NULL first without lock. > + Even if we should race, we will at most skip one checkpoint and > + take the next one, which is harmless. */ > + if (!pending_checkpoint_list) > + return; > + > + mysql_mutex_lock(&pending_checkpoint_mutex); > + pending = pending_checkpoint_list; > + if (!pending) > + { > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + return; > + } > + > + last_ready = NULL; > + for (entry = pending; entry != NULL; entry = entry -> next) > + { > + if (entry->lsn > flush_lsn) > + break; > + last_ready = entry; > + } > + > + if (last_ready) > + { > + /* We found some pending checkpoints that are now flushed to > + disk. So remove them from the list. */ > + pending_checkpoint_list = entry; > + if (!entry) > + pending_checkpoint_list_end = NULL; > + } > + > + mysql_mutex_unlock(&pending_checkpoint_mutex); > + > + if (!last_ready) > + return; > + > + /* Now that we have released the lock, notify upper layer about all > + commit checkpoints that have now completed. */ > + for (;;) { > + entry = pending; > + pending = pending->next; > + > + commit_checkpoint_notify_ha(entry->hton, entry->cookie); > + > + my_free(entry); > + if (entry == last_ready) > + break; > + } > } > > /*****************************************************************//** > > === modified file 'storage/xtradb/include/ha_prototypes.h' > --- a/storage/xtradb/include/ha_prototypes.h 2012-02-21 19:51:56 +0000 > +++ b/storage/xtradb/include/ha_prototypes.h 2012-09-14 12:44:53 +0000 > @@ -136,6 +136,17 @@ innobase_mysql_print_thd( > uint max_query_len); /*!< in: max query length to print, or 0 to > use the default max length */ > > +/*****************************************************************//** > +Log code calls this whenever log has been written and/or flushed up > +to a new position. We use this to notify upper layer of a new commit > +checkpoint when necessary.*/ > +UNIV_INTERN > +void > +innobase_mysql_log_notify( > +/*===============*/ > + ib_uint64_t write_lsn, /*!< in: LSN written to log file */ > + ib_uint64_t flush_lsn); /*!< in: LSN flushed to disk */ > + > /**************************************************************//** > Converts a MySQL type to an InnoDB type. Note that this function returns > the 'mtype' of InnoDB. InnoDB differentiates between MySQL's old <= 4.1 > > === modified file 'storage/xtradb/include/log0log.h' > --- a/storage/xtradb/include/log0log.h 2012-08-27 16:13:17 +0000 > +++ b/storage/xtradb/include/log0log.h 2012-09-14 12:44:53 +0000 > @@ -151,6 +151,13 @@ UNIV_INLINE > ib_uint64_t > log_get_lsn(void); > /*=============*/ > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void); > +/*=============*/ > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/xtradb/include/log0log.ic' > --- a/storage/xtradb/include/log0log.ic 2011-07-14 19:22:41 +0000 > +++ b/storage/xtradb/include/log0log.ic 2012-09-14 12:44:53 +0000 > @@ -411,6 +411,25 @@ log_get_lsn(void) > return(lsn); > } > > +/************************************************************//** > +Gets the last lsn that is fully flushed to disk. > +@return last flushed lsn */ > +UNIV_INLINE > +ib_uint64_t > +log_get_flush_lsn(void) > +/*=============*/ > +{ > + ib_uint64_t lsn; > + > + mutex_enter(&(log_sys->mutex)); > + > + lsn = log_sys->flushed_to_disk_lsn; > + > + mutex_exit(&(log_sys->mutex)); > + > + return(lsn); > +} > + > /**************************************************************** > Gets the log group capacity. It is OK to read the value without > holding log_sys->mutex because it is constant. > > === modified file 'storage/xtradb/log/log0log.c' > --- a/storage/xtradb/log/log0log.c 2012-08-27 16:13:17 +0000 > +++ b/storage/xtradb/log/log0log.c 2012-09-14 12:44:53 +0000 > @@ -1390,6 +1390,8 @@ log_write_up_to( > ulint loop_count = 0; > #endif /* UNIV_DEBUG */ > ulint unlock; > + ib_uint64_t write_lsn; > + ib_uint64_t flush_lsn; > > if (recv_no_ibuf_operations) { > /* Recovery is running and no operations on the log files are > @@ -1568,8 +1570,13 @@ log_write_up_to( > > log_flush_do_unlocks(unlock); > > + write_lsn = log_sys->write_lsn; > + flush_lsn = log_sys->flushed_to_disk_lsn; > + > mutex_exit(&(log_sys->mutex)); > > + innobase_mysql_log_notify(write_lsn, flush_lsn); > + > return; > > do_waits: > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 2

[Maria-developers] review of the MDEV-377 Name support for dynamic columns
by Sergei Golubchik 14 Dec '12

14 Dec '12

Hi, Sanja, See my review comments below: > === modified file 'include/ma_dyncol.h' > --- include/ma_dyncol.h 2011-09-22 09:04:00 +0000 > +++ include/ma_dyncol.h 2012-11-12 15:11:23 +0000 > @@ -39,6 +39,12 @@ > */ > #define MAX_DYNAMIC_COLUMN_LENGTH 0X1FFFFFFFL > > +/* > + Limits of implementation > +*/ > +#define MAX_NAME_LENGTH 255 If we store only offsets, this limit can go away > +#define MAX_TOTAL_NAME_LENGTH 65535 I'd suggest to store the offset length in the header, just as you do for data offsets. You might hard-code the fixed offset of 2, but if it's in the header, you can later change it, if needed. > + > /* NO and OK is the same used just to show semantics */ > #define ER_DYNCOL_NO ER_DYNCOL_OK > > @@ -81,6 +88,7 @@ struct st_dynamic_column_value > struct { > LEX_STRING value; > CHARSET_INFO *charset; > + my_bool nonfreeable; Yuck. That doesn't look correct. Dynamic column code does not allocate or frees values. The concept of "nonfreeable" values simply has no place in the dynamic column universe. I realize that ha_cassandra needs to know whether to free certain strings or not. But it's the internal problem of ha_cassandra, it has nothing to do with dynaic columns, besides the fact that these certain strings are sometines stored in the struct st_dynamic_column_value. > } string; > struct { > decimal_digit_t buffer[DECIMAL_BUFF_LENGTH]; > @@ -103,6 +111,20 @@ dynamic_column_create_many(DYNAMIC_COLUM > DYNAMIC_COLUMN_VALUE *values); > > enum enum_dyncol_func_result > +dynamic_column_create_many_fmt(DYNAMIC_COLUMN *str, > + uint column_count, > + uchar *column_keys, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool names); I think this should rather be dynamic_column_create_many_named(DYNAMIC_COLUMN *str, uint column_count, LEX_STRING *column_keys, DYNAMIC_COLUMN_VALUE *values); > +enum enum_dyncol_func_result > +dynamic_column_create_many_internal_fmt(DYNAMIC_COLUMN *str, > + uint column_count, > + void *column_keys, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool new_str, > + my_bool string_keys); *internal* should not be in include/, it's not part of the API > + > +enum enum_dyncol_func_result > dynamic_column_update(DYNAMIC_COLUMN *org, uint column_nr, > DYNAMIC_COLUMN_VALUE *value); > enum enum_dyncol_func_result > @@ -110,16 +132,30 @@ dynamic_column_update_many(DYNAMIC_COLUM > uint add_column_count, > uint *column_numbers, > DYNAMIC_COLUMN_VALUE *values); > +enum enum_dyncol_func_result > +dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str, > + uint add_column_count, > + void *column_keys, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool string_keys); same as above, make it dynamic_column_update_many_named > > enum enum_dyncol_func_result > dynamic_column_delete(DYNAMIC_COLUMN *org, uint column_nr); > > enum enum_dyncol_func_result > dynamic_column_exists(DYNAMIC_COLUMN *org, uint column_nr); > +enum enum_dyncol_func_result > +dynamic_column_exists_str(DYNAMIC_COLUMN *str, LEX_STRING *name); > +enum enum_dyncol_func_result > +dynamic_column_exists_fmt(DYNAMIC_COLUMN *str, void *key, my_bool string_keys); remove the second one, and rename the first s/_str/_named/ > > /* List of not NULL columns */ > enum enum_dyncol_func_result > dynamic_column_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); > +enum enum_dyncol_func_result > +dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr); > +enum enum_dyncol_func_result > +dynamic_column_list_fmt(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array, my_bool string_keys); Same. > > /* > if the column do not exists it is NULL > @@ -127,10 +163,36 @@ dynamic_column_list(DYNAMIC_COLUMN *org, > enum enum_dyncol_func_result > dynamic_column_get(DYNAMIC_COLUMN *org, uint column_nr, > DYNAMIC_COLUMN_VALUE *store_it_here); > +enum enum_dyncol_func_result > +dynamic_column_get_str(DYNAMIC_COLUMN *str, LEX_STRING *name, > + DYNAMIC_COLUMN_VALUE *store_it_here); s/_str/_named > + > +my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str); > + > +enum enum_dyncol_func_result > +dynamic_column_check(DYNAMIC_COLUMN *str); > + > +enum enum_dyncol_func_result > +dynamic_column_json(DYNAMIC_COLUMN *str, DYNAMIC_STRING *json); > > #define dynamic_column_initialize(A) memset((A), 0, sizeof(*(A))) > #define dynamic_column_column_free(V) dynstr_free(V) > > +/* conversion of values to 3 base types */ > +enum enum_dyncol_func_result > +dynamic_column_val_str(DYNAMIC_STRING *str, DYNAMIC_COLUMN_VALUE *val, > + CHARSET_INFO *cs, my_bool quote); instead of my_bool quote, better pass a char quote_char. when it's 0 - no quoting. Otherwise it can be " or ` To make it less error prone, you can create an enum NOT_QUOTED=0, QUOTED='`', ANSI_QUOTED='"' > +enum enum_dyncol_func_result > +dynamic_column_val_long(longlong *ll, DYNAMIC_COLUMN_VALUE *val); > +enum enum_dyncol_func_result > +dynamic_column_val_double(double *dbl, DYNAMIC_COLUMN_VALUE *val); > + > + > +enum enum_dyncol_func_result > +dynamic_column_vals(DYNAMIC_COLUMN *str, > + DYNAMIC_ARRAY *names, DYNAMIC_ARRAY *vals, > + char **free_names); Eh. may be dynamic_column_unpack() ? > + > /*************************************************************************** > Internal functions, don't use if you don't know what you are doing... > ***************************************************************************/ > > === modified file 'mysql-test/t/dyncol.test' > --- mysql-test/t/dyncol.test 2012-02-29 20:55:04 +0000 > +++ mysql-test/t/dyncol.test 2012-11-12 15:11:23 +0000 > @@ -550,3 +550,125 @@ select hex(COLUMN_CREATE(0, COLUMN_GET(@ > > select hex(COLUMN_CREATE(0, COLUMN_GET(COLUMN_CREATE(0, 0.0 as decimal), 0 as decimal))); > select hex(COLUMN_CREATE(0, 0.0 as decimal)); > + > +--echo # > +--echo # test of symbolic names > +--echo # > +--echo # creation test (names) > +set names utf8; > +select hex(column_create("адын", 1212)); > +select hex(column_create("1212", 1212)); > +select hex(column_create(1212, 2, "www", 3)); > +select hex(column_create("1212", 2, "www", 3)); > +select hex(column_create("1212", 2, 3, 3)); > +select hex(column_create("1212", 2, "адын", 1, 3, 3)); to test quoting, add a column with a ` (backtick) in the name and few other strange names - with a single quote, double quote, backslash, space, dot, comma, whatever > +set names default; ... > === modified file 'sql/sql_yacc.yy' > --- sql/sql_yacc.yy 2012-06-13 23:28:47 +0000 > +++ sql/sql_yacc.yy 2012-11-12 15:11:23 +0000 > @@ -8799,6 +8801,13 @@ function_call_nonkeyword: > MYSQL_YYABORT; > } > | > + COLUMN_CHECK_SYM '(' expr ')' > + { > + $$= new (YYTHD->mem_root) Item_func_dyncol_check($3); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + | This should be removed completely. See, for example, how EXP function is created (hint: in item_create.cc) and do the same > COLUMN_EXISTS_SYM '(' expr ',' expr ')' > { > $$= new (YYTHD->mem_root) Item_func_dyncol_exists($3, $5); > @@ -8820,6 +8829,13 @@ function_call_nonkeyword: > MYSQL_YYABORT; > } > | > + COLUMN_JSON_SYM '(' expr ')' > + { > + $$= create_func_dyncol_json(YYTHD, $3); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + | same. please, remove the dedicated rule for this and most of other COLUMN_xxx functions. > COLUMN_GET_SYM '(' expr ',' expr AS cast_type ')' > { > LEX *lex= Lex; > > === modified file 'mysys/ma_dyncol.c' > --- mysys/ma_dyncol.c 2011-11-22 17:04:38 +0000 > +++ mysys/ma_dyncol.c 2012-11-12 15:11:23 +0000 > @@ -37,19 +41,42 @@ > */ > /* mask to get above bits */ > #define DYNCOL_FLG_OFFSET 3 > +#define DYNCOL_FLG_NAMES 4 > /* All known flags mask */ > -#define DYNCOL_FLG_KNOWN 3 > +#define DYNCOL_FLG_KNOWN 7 > + > +/* formats */ > +#define DYNCOL_FMT_NUM 0 > +#define DYNCOL_FMT_STR 1 why not an enum? > > /* dynamic column size reserve */ > #define DYNCOL_SYZERESERVE 80 > > +#define DYNCOL_OFFSET_ERROR 0xffffffff > + > /* length of fixed string header 1 byte - flags, 2 bytes - columns counter */ > #define FIXED_HEADER_SIZE 3 > +/* > + length of fixed string header with names > + 1 byte - flags, 2 bytes - columns counter, 2 bytes - name pool size > +*/ > +#define FIXED_HEADER_SIZE_NM 5 > > #define COLUMN_NUMBER_SIZE 2 > +/* 1 byte name length + 2 bytes offset from the name pool */ > +#define COLUMN_NAMEPTR_SIZE 3 > > #define MAX_OFFSET_LENGTH 5 > > +#define DYNCOL_NUM_CHAR 6 > + > +my_bool dynamic_column_has_names(DYNAMIC_COLUMN *str) > +{ > + if (str->length < 1) > + return FALSE; > + return test(str->str[0] & DYNCOL_FLG_NAMES); > +} > + > static enum enum_dyncol_func_result > dynamic_column_time_store(DYNAMIC_COLUMN *str, > MYSQL_TIME *value); > @@ -62,6 +89,311 @@ dynamic_column_time_read_internal(DYNAMI > static enum enum_dyncol_func_result > dynamic_column_date_read_internal(DYNAMIC_COLUMN_VALUE *store_it_here, > uchar *data, size_t length); > +static enum enum_dyncol_func_result > +dynamic_column_get_internal(DYNAMIC_COLUMN *str, > + DYNAMIC_COLUMN_VALUE *store_it_here, > + uint num_key, LEX_STRING *str_key); > +static enum enum_dyncol_func_result > +dynamic_column_exists_internal(DYNAMIC_COLUMN *str, uint num_key, > + LEX_STRING *str_key); > +enum enum_dyncol_func_result > +dynamic_column_update_many_fmt(DYNAMIC_COLUMN *str, > + uint add_column_count, > + void *column_keys, > + DYNAMIC_COLUMN_VALUE *values, > + my_bool string_keys); > +static int plan_sort_num(const void *a, const void *b); > +static int plan_sort_str(const void *a, const void *b); > + > +/* > + Structure to hold information about dynamic columns record and > + iterate through it. > +*/ > + > +struct st_dyn_header > +{ > + uchar *header, *nmpool, *dtpool, *data_end; > + size_t offset_size; > + size_t entry_size; > + size_t header_size; > + size_t nmpool_size; > + size_t data_size; > + /* DYNCOL_FMT_NUM - numeric columns, DYNCOL_FMT_STR - column names */ > + uint format; > + uint column_count; > + > + uchar *entry, *data, *name; > + size_t offset; > + uint length; > + enum enum_dynamic_column_type type; > +}; > + > +typedef struct st_dyn_header DYN_HEADER; > + > +static inline my_bool read_fixed_header(DYN_HEADER *hdr, > + DYNAMIC_COLUMN *str); > +static void set_fixed_header(DYNAMIC_COLUMN *str, > + uint offset_size, > + uint column_count); > +static my_bool type_and_offset_store(uchar *place, size_t offset_size, > + DYNAMIC_COLUMN_TYPE type, > + size_t offset); > + > +/* > + Calculate entry size (E) and header size (H) by offset size (O) and column > + count (C) and fixed part of entry size (F). > +*/ > + > +#define calc_param(E,H,F,O,C) do { \ > + (*(E))= (O) + F; \ > + (*(H))= (*(E)) * (C); \ > +}while(0); > + > + > +/** > + Name pool size functions, for numeric format it is 0 > +*/ > + > +size_t name_size_num(void *keys __attribute__((unused)), > + uint i __attribute__((unused))) > +{ > + return 0; > +} > + > + > +/** > + Name pool size functions. > +*/ > +size_t name_size_str(void *keys, uint i) > +{ > + return ((LEX_STRING *) keys)[i].length; > +} > + > + > +/** > + Comparator function for references on column numbers for qsort > + (numeric format) > +*/ > + > +static int column_sort_num(const void *a, const void *b) > +{ > + return **((uint **)a) - **((uint **)b); > +} > + > + > +/** > + Comparator function for references on column numbers for qsort > + (names format) > +*/ > + > +static int column_sort_str(const void *a, const void *b) > +{ > + LEX_STRING *s1= *((LEX_STRING **)a); > + LEX_STRING *s2= *((LEX_STRING **)b); > + int rc= s1->length - s2->length; > + if (rc == 0) > + rc= memcmp((void *)s1->str, (void *)s2->str, (size_t) s1->length); > + return rc; > +} > + > + > +/** > + Check limit function (numeric format) > +*/ > + > +static my_bool check_limit_num(const void *val) > +{ > + return **((uint **)val) > UINT_MAX16; > +} > + > + > +/** > + Check limit function (names format) > +*/ > + > +static my_bool check_limit_str(const void *val) > +{ > + return (*((LEX_STRING **)val))->length > MAX_NAME_LENGTH; > +} > + > + > +/** > + Write numeric format static header part. > +*/ > + > +void set_fixed_header_num(DYNAMIC_COLUMN *str, DYN_HEADER *hdr) > +{ > + set_fixed_header(str, hdr->offset_size, hdr->column_count); > + hdr->header= (uchar *)str->str + FIXED_HEADER_SIZE; > + hdr->nmpool= hdr->dtpool= hdr->header + hdr->header_size; > +} > + > + > +/** > + Write names format static header part. > +*/ > + > +void set_fixed_header_str(DYNAMIC_COLUMN *str, DYN_HEADER *hdr) > +{ > + set_fixed_header(str, hdr->offset_size, hdr->column_count); There's no need to write the column count there, you save two bytes per record. a column count = (offset of the first name - size of the fized header) / (size of the fixed per column header) > + str->str[0]|= DYNCOL_FLG_NAMES; > + int2store(str->str + 3, hdr->nmpool_size); > + hdr->header= (uchar *)str->str + FIXED_HEADER_SIZE_NM; > + hdr->nmpool= hdr->header + hdr->header_size; > + hdr->dtpool= hdr->nmpool + hdr->nmpool_size; > +} > + > + > +/** > + Write numeric format header entry > + 2 bytes - column number > + 1-4 bytes - data offset combined with type > + > + @param hdr descriptor of dynamic column record > + @param column_key pointer to uint (column number) > + @param value value which will be written (only type used) > + @param offset offset of the data > +*/ > + > +my_bool put_header_entry_num(DYN_HEADER *hdr, this is and many other functions should be static > + void *column_key, > + DYNAMIC_COLUMN_VALUE *value, > + size_t offset) > +{ > + uint *column_number= (uint *)column_key; > + int2store(hdr->entry, *column_number); > + DBUG_ASSERT(hdr->nmpool_size == 0); > + if (type_and_offset_store(hdr->entry, hdr->offset_size, > + value->type, > + offset)) > + return TRUE; > + hdr->entry= hdr->entry + hdr->entry_size; > + return FALSE; > +} > + > + > +/** > + Write names format header entry > + 1 byte - name length > + 2 bytes - name offset in the name pool as discussed on the irc, please store either lengths or offsets, but not both > + 1-4 bytes - data offset combined with type > + > + @param hdr descriptor of dynamic column record > + @param column_key pointer to LEX_STRING (column name) > + @param value value which will be written (only type used) > + @param offset offset of the data > +*/ > + > +my_bool put_header_entry_str(DYN_HEADER *hdr, > + void *column_key, > + DYNAMIC_COLUMN_VALUE *value, > + size_t offset) > +{ > + LEX_STRING *column_name= (LEX_STRING *)column_key; > + DBUG_ASSERT(column_name->length <= MAX_NAME_LENGTH); > + hdr->entry[0]= column_name->length; > + DBUG_ASSERT(hdr->name - hdr->nmpool < (long) 0x10000L); > + int2store(hdr->entry + 1, hdr->name - hdr->nmpool); > + memcpy(hdr->name, column_name->str, column_name->length); > + DBUG_ASSERT(hdr->nmpool_size != 0 || column_name->length == 0); > + if (type_and_offset_store(hdr->entry + 1, hdr->offset_size, > + value->type, > + offset)) > + return TRUE; > + hdr->entry+= hdr->entry_size; > + hdr->name+= column_name->length; > + return FALSE; > +} > + > + > +/** > + Format descriptor, contain constants and function references for > + format processing > +*/ > + > +struct st_service_funcs > +{ > + /* size of fixed header */ > + uint fixed_hdr; > + /* size of fixed part of header entry */ > + uint fixed_hdr_entry; > + > + /*size of array element which stores keys */ > + uint key_size_in_array; > + > + size_t (*name_size) > + (void *, uint); > + int (*column_sort) > + (const void *a, const void *b); > + my_bool (*check_limit) > + (const void *val); > + void (*set_fixed_hdr) > + (DYNAMIC_COLUMN *str, DYN_HEADER *hdr); > + my_bool (*put_header_entry)(DYN_HEADER *hdr, > + void *column_key, > + DYNAMIC_COLUMN_VALUE *value, > + size_t offset); > + int (*plan_sort)(const void *a, const void *b); > +}; > + > + > +/** > + Actual our 2 format descriptors > +*/ > + > +static struct st_service_funcs fmt_data[2]= > +{ > + { > + FIXED_HEADER_SIZE, > + COLUMN_NUMBER_SIZE, > + sizeof(uint), > + &name_size_num, > + &column_sort_num, > + &check_limit_num, > + &set_fixed_header_num, > + &put_header_entry_num, > + &plan_sort_num > + }, > + { > + FIXED_HEADER_SIZE_NM, > + COLUMN_NAMEPTR_SIZE, > + sizeof(LEX_STRING), > + &name_size_str, > + &column_sort_str, > + &check_limit_str, > + &set_fixed_header_str, > + &put_header_entry_str, > + &plan_sort_str > + } > +}; > + > + > +/** > + Read dynamic column record header and fill the descriptor > + > + @param hdr dynamic columns record descriptor to fill > + @param str dynamic columns record > + > + @return ER_DYNCOL_* return code > +*/ > + > +enum enum_dyncol_func_result > +init_read_hdr(DYN_HEADER *hdr, DYNAMIC_COLUMN *str) > +{ > + if (read_fixed_header(hdr, str)) > + return ER_DYNCOL_FORMAT; > + hdr->header= (uchar*)str->str + fmt_data[hdr->format].fixed_hdr; > + calc_param(&hdr->entry_size, &hdr->header_size, > + fmt_data[hdr->format].fixed_hdr_entry, hdr->offset_size, > + hdr->column_count); > + hdr->nmpool= hdr->header + hdr->header_size; > + hdr->dtpool= hdr->nmpool + hdr->nmpool_size; > + hdr->data_size= str->length - fmt_data[hdr->format].fixed_hdr - > + hdr->header_size - hdr->nmpool_size; > + hdr->data_end= (uchar*)str->str + str->length; > + return ER_DYNCOL_OK; > +} > + > > /** > Initialize dynamic column string with (make it empty but correct format) > @@ -1640,83 +2132,763 @@ dynamic_column_list(DYNAMIC_COLUMN *str, > > > /** > + List not-null columns in the packed string (any format) > + > + @param str The packed string > + @param array_of_lexstr Where to put reference on created array > + > + @return ER_DYNCOL_* return code > +*/ > + > +enum enum_dyncol_func_result > +dynamic_column_list_str(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_lexstr) > +{ > + DYN_HEADER header; > + uchar *read; > + struct st_service_funcs *fmt; > + uint i; > + enum enum_dyncol_func_result rc; > + > + bzero(array_of_lexstr, sizeof(*array_of_lexstr)); /* In case of errors */ > + if (str->length == 0) > + return ER_DYNCOL_OK; /* no columns */ > + > + if ((rc= init_read_hdr(&header, str)) < 0) > + return rc; > + > + fmt= fmt_data + header.format; > + > + if (header.entry_size * header.column_count + fmt->fixed_hdr > > + str->length) > + return ER_DYNCOL_FORMAT; > + > + if (init_dynamic_array(array_of_lexstr, sizeof(LEX_STRING), > + header.column_count, 0)) > + return ER_DYNCOL_RESOURCE; > + > + for (i= 0, read= header.header; > + i < header.column_count; > + i++, read+= header.entry_size) > + { > + LEX_STRING tmp; > + if (header.format == DYNCOL_FMT_NUM) > + { > + uint nm= uint2korr(read); > + tmp.str= my_malloc(DYNCOL_NUM_CHAR, MYF(0)); ouch. do you really need to do header.column_count small mallocs here? > + if (!tmp.str) > + return ER_DYNCOL_RESOURCE; > + tmp.length= snprintf(tmp.str, DYNCOL_NUM_CHAR, "%u", nm); longlong2str or longlong10_to_str > + } > + else > + { > + tmp.length= read[0]; > + tmp.str= my_malloc(tmp.length + 1, MYF(0)); ouch. do you really need to do header.column_count small mallocs here? > + if(!tmp.str) > + return ER_DYNCOL_RESOURCE; > + memcpy(tmp.str, (const void *)header.nmpool + uint2korr(read + 1), > + tmp.length); > + tmp.str[tmp.length]= '\0'; // just for safety > + } > + /* Insert can't never fail as it's pre-allocated above */ > + (void) insert_dynamic(array_of_lexstr, (uchar *)&tmp); > + } > + return ER_DYNCOL_OK; > +} > + > +/** > Find the place of the column in the header or place where it should be put > > - @param num Number of the column > - @param header Pointer to the header > - @param entry_size Size of a header entry > - @param column_count Number of columns in the packed string > - @param entry Return pointer to the entry or next entry > + @param hdr descriptor of dynamic column record > + @param key Name or number of column to fetch > + (depends on string_key) > + @param string_key True if we gave pointer to LEX_STRING. > > @retval TRUE found > @retval FALSE pointer set to the next row > */ > > static my_bool > -find_place(uint num, uchar *header, size_t entry_size, > - uint column_count, uchar **entry) > +find_place(DYN_HEADER *hdr, void *key, my_bool string_keys) > { > uint mid, start, end, val; > int flag; > + LEX_STRING str; > + char buff[DYNCOL_NUM_CHAR]; > + my_bool need_conversion= ((string_keys ? DYNCOL_FMT_STR : DYNCOL_FMT_NUM) != > + hdr->format); > LINT_INIT(flag); /* 100 % safe */ UNINIT_VAR is preferable, when possible (but it doesn't work with structures) > + /* new format can't be numeric if the old one is names */ > + DBUG_ASSERT(string_keys || > + hdr->format == DYNCOL_FMT_NUM); > > start= 0; > - end= column_count -1; > + end= hdr->column_count -1; > mid= 1; > while (start != end) > { > - uint val; > - mid= (start + end) / 2; > - val= uint2korr(header + mid * entry_size); > - if ((flag= CMP_NUM(num, val)) <= 0) > - end= mid; > - else > - start= mid + 1; > + uint val; > + mid= (start + end) / 2; > + hdr->entry= hdr->header + mid * hdr->entry_size; > + if (!string_keys) > + { > + val= uint2korr(hdr->entry); > + flag= CMP_NUM(*((uint *)key), val); > + } > + else > + { > + if (need_conversion) > + { > + str.str= backwritenum(buff + sizeof(buff), uint2korr(hdr->entry)); > + str.length= (buff + sizeof(buff)) - str.str; > + } > + else > + { > + DBUG_ASSERT(hdr->format == DYNCOL_FMT_STR); > + str.length= hdr->entry[0]; > + str.str= (char *)hdr->nmpool + uint2korr(hdr->entry + 1); > + } > + flag= ((LEX_STRING *) key)->length - str.length; > + if (flag == 0) > + flag= memcmp(((LEX_STRING *) key)->str, str.str, str.length); > + } > + if (flag <= 0) > + end= mid; > + else > + start= mid + 1; > } > + hdr->entry= hdr->header + start * hdr->entry_size; > if (start != mid) > { > - val= uint2korr(header + start * entry_size); > - flag= CMP_NUM(num, val); > + if (!string_keys) > + { > + val= uint2korr(hdr->entry); > + flag= CMP_NUM(*((uint *)key), val); > + } > + else > + { > + if (need_conversion) > + { > + str.str= backwritenum(buff + sizeof(buff), uint2korr(hdr->entry)); > + str.length= (buff + sizeof(buff)) - str.str; > + } > + else > + { > + DBUG_ASSERT(hdr->format == DYNCOL_FMT_STR); > + str.length= hdr->entry[0]; > + str.str= (char*) hdr->nmpool + uint2korr(hdr->entry + 1); > + } > + flag= ((LEX_STRING *) key)->length - str.length; > + if (flag == 0) > + flag= memcmp(((LEX_STRING *) key)->str, str.str, str.length); > + } > } > - *entry= header + start * entry_size; > if (flag > 0) > - *entry+= entry_size; /* Point at next bigger key */ > + hdr->entry+= hdr->entry_size; /* Point at next bigger key */ > return flag == 0; > } > > > /* > - Description of plan of adding/removing/updating a packed string > + It is internal structure which describes plan of chenging the record s/chenging/changing/ what do you mean "a plan" ? > + of dynamic columns > */ > > typedef enum {PLAN_REPLACE, PLAN_ADD, PLAN_DELETE, PLAN_NOP} PLAN_ACT; > > struct st_plan { > DYNAMIC_COLUMN_VALUE *val; > - uint *num; > + void *key; > uchar *place; > size_t length; > - int hdelta, ddelta; > + int hdelta, ddelta, ndelta; > + uint mv_offset, mv_length, mv_end; > PLAN_ACT act; > }; > typedef struct st_plan PLAN; > > > -static int plan_sort(const void *a, const void *b) > +/** > + Sort function for plan by column number > +*/ > + > +static int plan_sort_num(const void *a, const void *b) > { > - return ((PLAN *)a)->num[0] - ((PLAN *)b)->num[0]; > + return *((uint *)((PLAN *)a)->key) - *((uint *)((PLAN *)b)->key); > +} > + > + > +/** > + Sort function for plan by column name > +*/ > + > +static int plan_sort_str(const void *a, const void *b) > +{ > + int res= (((LEX_STRING *)((PLAN *)a)->key)->length - > + ((LEX_STRING *)((PLAN *)b)->key)->length); > + if (res == 0) > + res= memcmp(((LEX_STRING *)((PLAN *)a)->key)->str, > + ((LEX_STRING *)((PLAN *)b)->key)->str, > + ((LEX_STRING *)((PLAN *)a)->key)->length); > + return res; > +} > + > +#define DELTA_CHECK(S, D, C) \ > + if ((S) == 0) \ > + (S)= (D); \ > + else if (((S) > 0 && (D) < 0) || \ > + ((S) < 0 && (D) > 0)) \ > + { \ > + (C)= TRUE; \ > + } > + > +/** > + Update dynamic column by copying in a new record (string). > + > + @param str Dynamic column record to change > + @param plan Plan of changing the record > + @param add_column_count number of records in the plan array. > + @param hdr descriptor of old dynamic column record > + @param new_hdr descriptor of new dynamic column record > + @param convert need conversion from numeric to names format > + > + @return ER_DYNCOL_* return code > +*/ > + > +enum enum_dyncol_func_result > +dynamic_column_update_copy(DYNAMIC_COLUMN *str, PLAN *plan, > + uint add_column_count, > + DYN_HEADER *hdr, DYN_HEADER *new_hdr, > + my_bool convert) > +{ > + DYNAMIC_COLUMN tmp; > + struct st_service_funcs *fmt= fmt_data + hdr->format, > + *new_fmt= fmt_data + new_hdr->format; > + uint i, j, k; > + size_t all_headers_size; > + > + if (dynamic_column_init_str(&tmp, > + (new_fmt->fixed_hdr + new_hdr->header_size + > + new_hdr->nmpool_size + > + new_hdr->data_size + DYNCOL_SYZERESERVE))) > + { > + return ER_DYNCOL_RESOURCE; > + } > + bzero(tmp.str, new_fmt->fixed_hdr); > + (*new_fmt->set_fixed_hdr)(&tmp, new_hdr); > + /* Adjust tmp to contain whole the future header */ > + tmp.length= new_fmt->fixed_hdr + new_hdr->header_size + new_hdr->nmpool_size; > + > + > + /* > + Copy data to the new string > + i= index in array of changes > + j= index in packed string header index > + */ > + new_hdr->entry= new_hdr->header; > + new_hdr->name= new_hdr->nmpool; > + all_headers_size= new_fmt->fixed_hdr + > + new_hdr->header_size + new_hdr->nmpool_size; > + for (i= 0, j= 0; i < add_column_count || j < hdr->column_count; i++) > + { > + size_t first_offset; > + uint start= j, end; > + LINT_INIT(first_offset); > + > + /* > + Search in i and j for the next column to add from i and where to > + add. > + */ > + > + while (i < add_column_count && plan[i].act == PLAN_NOP) > + i++; /* skip NOP */ > + > + if (i == add_column_count) > + j= end= hdr->column_count; > + else > + { > + /* > + old data portion. We don't need to check that j < column_count > + as plan[i].place is guaranteed to have a pointer inside the > + data. > + */ > + while (hdr->header + j * hdr->entry_size < plan[i].place) > + j++; > + end= j; > + if ((plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > + j++; /* data at 'j' will be removed */ > + } > + > + /* > + Adjust all headers since last loop. > + We have to do this as the offset for data has moved > + */ > + for (k= start; k < end; k++) > + { > + uchar *read= hdr->header + k * hdr->entry_size; > + void *key; > + LEX_STRING name; > + size_t offs; > + uint nm; > + DYNAMIC_COLUMN_TYPE tp; > + char buff[DYNCOL_NUM_CHAR]; > + > + if (hdr->format == DYNCOL_FMT_NUM) > + { > + if (convert) > + { > + name.str= backwritenum(buff + sizeof(buff), uint2korr(read)); > + name.length= (buff + sizeof(buff)) - name.str; > + key= &name; > + } > + else > + { > + nm= uint2korr(read); /* Column nummber */ > + key= &nm; > + } > + } > + else > + { > + name.length= read[0]; > + name.str= (char *) hdr->nmpool + uint2korr(read + 1); > + key= &name; > + } > + if (type_and_offset_read(&tp, &offs, > + read + fmt->fixed_hdr_entry, hdr->offset_size)) > + goto err; > + if (k == start) > + first_offset= offs; > + else if (offs < first_offset) > + goto err; > + > + offs+= plan[i].ddelta; > + { > + DYNAMIC_COLUMN_VALUE val; > + val.type= tp; // only the type used in the header > + if ((*new_fmt->put_header_entry)(new_hdr, key, &val, offs)) > + goto err; > + } > + } > + > + /* copy first the data that was not replaced in original packed data */ > + if (start < end) > + { > + size_t data_size; > + /* Add old data last in 'tmp' */ > + hdr->entry= hdr->header + start * hdr->entry_size; > + data_size= > + hdr_interval_length(hdr, hdr->header + end * hdr->entry_size); > + if (data_size == DYNCOL_OFFSET_ERROR || > + (long) data_size < 0 || > + data_size > hdr->data_size - first_offset) > + goto err; > + > + memcpy(tmp.str + tmp.length, (char *)hdr->dtpool + first_offset, > + data_size); > + tmp.length+= data_size; > + } > + > + /* new data adding */ > + if (i < add_column_count) > + { > + if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > + { > + if ((*new_fmt->put_header_entry)(new_hdr, plan[i].key, > + plan[i].val, > + tmp.length - all_headers_size)) > + goto err; > + data_store(&tmp, plan[i].val); /* Append new data */ > + } > + } > + } > + dynamic_column_column_free(str); > + *str= tmp; > + return ER_DYNCOL_OK; > +err: > + dynamic_column_column_free(&tmp); > + return ER_DYNCOL_FORMAT; > +} > + > +enum enum_dyncol_func_result > +dynamic_column_update_move_left(DYNAMIC_COLUMN *str, PLAN *plan, > + size_t offset_size, > + size_t entry_size, > + size_t header_size, > + size_t new_offset_size, > + size_t new_entry_size, > + size_t new_header_size, > + uint column_count, > + uint new_column_count, > + uint add_column_count, > + uchar *header_end, > + size_t max_offset) > +{ > + uchar *write; > + uchar *header_base= (uchar *)str->str + FIXED_HEADER_SIZE; > + uint i, j, k; > + size_t curr_offset; > + > + write= (uchar *)str->str + FIXED_HEADER_SIZE; > + set_fixed_header(str, new_offset_size, new_column_count); > + > + /* > + Move headers first. > + i= index in array of changes > + j= index in packed string header index > + */ > + for (curr_offset= 0, i= 0, j= 0; > + i < add_column_count || j < column_count; > + i++) > + { > + size_t first_offset; > + uint start= j, end; > + LINT_INIT(first_offset); > + > + /* > + Search in i and j for the next column to add from i and where to > + add. > + */ > + > + while (i < add_column_count && plan[i].act == PLAN_NOP) > + i++; /* skip NOP */ > + > + if (i == add_column_count) > + j= end= column_count; > + else > + { > + /* > + old data portion. We don't need to check that j < column_count > + as plan[i].place is guaranteed to have a pointer inside the > + data. > + */ > + while (header_base + j * entry_size < plan[i].place) > + j++; > + end= j; > + if ((plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > + j++; /* data at 'j' will be removed */ > + } > + plan[i].mv_end= end; > + > + { > + DYNAMIC_COLUMN_TYPE tp; > + if (type_and_offset_read(&tp, &first_offset, > + header_base + start * entry_size + > + COLUMN_NUMBER_SIZE, offset_size)) > + return ER_DYNCOL_FORMAT; > + } > + /* find data to be moved */ > + if (start < end) > + { > + size_t data_size= > + get_length_interval(header_base + start * entry_size, > + header_base + end * entry_size, > + header_end, offset_size, max_offset); > + if (data_size == DYNCOL_OFFSET_ERROR || > + (long) data_size < 0 || > + data_size > max_offset - first_offset) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + DBUG_ASSERT(curr_offset == first_offset + plan[i].ddelta); > + plan[i].mv_offset= first_offset; > + plan[i].mv_length= data_size; > + curr_offset+= data_size; > + } > + else > + { > + plan[i].mv_length= 0; > + plan[i].mv_offset= curr_offset; > + } > + > + if (plan[i].ddelta == 0 && offset_size == new_offset_size && > + plan[i].act != PLAN_DELETE) > + write+= entry_size * (end - start); > + else > + { > + /* > + Adjust all headers since last loop. > + We have to do this as the offset for data has moved > + */ > + for (k= start; k < end; k++) > + { > + uchar *read= header_base + k * entry_size; > + size_t offs; > + uint nm; > + DYNAMIC_COLUMN_TYPE tp; > + > + nm= uint2korr(read); /* Column nummber */ > + if (type_and_offset_read(&tp, &offs, read + COLUMN_NUMBER_SIZE, > + offset_size)) > + return ER_DYNCOL_FORMAT; > + > + if (k > start && offs < first_offset) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + > + offs+= plan[i].ddelta; > + int2store(write, nm); > + /* write rest of data at write + COLUMN_NUMBER_SIZE */ > + type_and_offset_store(write, new_offset_size, tp, offs); > + write+= new_entry_size; > + } > + } > + > + /* new data adding */ > + if (i < add_column_count) > + { > + if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > + { > + int2store(write, *((uint *)plan[i].key)); > + type_and_offset_store(write, new_offset_size, > + plan[i].val[0].type, > + curr_offset); > + write+= new_entry_size; > + curr_offset+= plan[i].length; > + } > + } > + } > + > + /* > + Move data. > + i= index in array of changes > + j= index in packed string header index > + */ > + str->length= (FIXED_HEADER_SIZE + new_header_size); > + for (i= 0, j= 0; > + i < add_column_count || j < column_count; > + i++) > + { > + uint start= j, end; > + > + /* > + Search in i and j for the next column to add from i and where to > + add. > + */ > + > + while (i < add_column_count && plan[i].act == PLAN_NOP) > + i++; /* skip NOP */ > + > + j= end= plan[i].mv_end; > + if (i != add_column_count && > + (plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > + j++; > + > + /* copy first the data that was not replaced in original packed data */ > + if (start < end && plan[i].mv_length) > + { > + memmove((header_base + new_header_size + > + plan[i].mv_offset + plan[i].ddelta), > + header_base + header_size + plan[i].mv_offset, > + plan[i].mv_length); > + } > + str->length+= plan[i].mv_length; > + > + /* new data adding */ > + if (i < add_column_count) > + { > + if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > + { > + data_store(str, plan[i].val); /* Append new data */ > + } > + } > + } > + return ER_DYNCOL_OK; > +} > + > +enum enum_dyncol_func_result > +dynamic_column_update_move_right(DYNAMIC_COLUMN *str, PLAN *plan, > + size_t offset_size, > + size_t entry_size, > + size_t header_size, > + size_t new_offset_size, > + size_t new_entry_size, > + size_t new_header_size, > + uint column_count, > + uint new_column_count, > + uint add_column_count, > + uchar *header_end, > + size_t max_offset) > +{ > + uchar *write; > + uchar *header_base= (uchar *)str->str + FIXED_HEADER_SIZE; > + uint i, j, k; > + size_t curr_offset; > + > + write= (uchar *)str->str + FIXED_HEADER_SIZE; > + set_fixed_header(str, new_offset_size, new_column_count); > + > + /* > + Move data first. > + i= index in array of changes > + j= index in packed string header index > + */ > + for (curr_offset= 0, i= 0, j= 0; > + i < add_column_count || j < column_count; > + i++) > + { > + size_t first_offset; > + uint start= j, end; > + LINT_INIT(first_offset); > + > + /* > + Search in i and j for the next column to add from i and where to > + add. > + */ > + > + while (i < add_column_count && plan[i].act == PLAN_NOP) > + i++; /* skip NOP */ > + > + if (i == add_column_count) > + j= end= column_count; > + else > + { > + /* > + old data portion. We don't need to check that j < column_count > + as plan[i].place is guaranteed to have a pointer inside the > + data. > + */ > + while (header_base + j * entry_size < plan[i].place) > + j++; > + end= j; > + if ((plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > + j++; /* data at 'j' will be removed */ > + } > + plan[i].mv_end= end; > + > + { > + DYNAMIC_COLUMN_TYPE tp; > + type_and_offset_read(&tp, &first_offset, > + header_base + start * entry_size + COLUMN_NUMBER_SIZE, offset_size); > + } > + /* find data to be moved */ > + if (start < end) > + { > + size_t data_size= > + get_length_interval(header_base + start * entry_size, > + header_base + end * entry_size, > + header_end, offset_size, max_offset); > + if (data_size == DYNCOL_OFFSET_ERROR || > + (long) data_size < 0 || > + data_size > max_offset - first_offset) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + DBUG_ASSERT(curr_offset == first_offset + plan[i].ddelta); > + plan[i].mv_offset= first_offset; > + plan[i].mv_length= data_size; > + curr_offset+= data_size; > + } > + else > + { > + plan[i].mv_length= 0; > + plan[i].mv_offset= curr_offset; > + } > + > + if (plan[i].ddelta == 0 && offset_size == new_offset_size && > + plan[i].act != PLAN_DELETE) > + write+= entry_size * (end - start); > + else > + { > + /* > + Adjust all headers since last loop. > + We have to do this as the offset for data has moved > + */ > + for (k= start; k < end; k++) > + { > + uchar *read= header_base + k * entry_size; > + size_t offs; > + uint nm; > + DYNAMIC_COLUMN_TYPE tp; > + > + nm= uint2korr(read); /* Column nummber */ > + type_and_offset_read(&tp, &offs, read + COLUMN_NUMBER_SIZE, offset_size); > + if (k > start && offs < first_offset) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + > + offs+= plan[i].ddelta; > + int2store(write, nm); > + /* write rest of data at write + COLUMN_NUMBER_SIZE */ > + if (type_and_offset_store(write, new_offset_size, tp, offs)) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + write+= new_entry_size; > + } > + } > + > + /* new data adding */ > + if (i < add_column_count) > + { > + if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > + { > + int2store(write, *((uint *)plan[i].key)); > + if (type_and_offset_store(write, new_offset_size, > + plan[i].val[0].type, > + curr_offset)) > + { > + str->length= 0; // just something valid > + return ER_DYNCOL_FORMAT; > + } > + write+= new_entry_size; > + curr_offset+= plan[i].length; > + } > + } > + } > + > + /* > + Move headers. > + i= index in array of changes > + j= index in packed string header index > + */ > + str->length= (FIXED_HEADER_SIZE + new_header_size); > + for (i= 0, j= 0; > + i < add_column_count || j < column_count; > + i++) > + { > + uint start= j, end; > + > + /* > + Search in i and j for the next column to add from i and where to > + add. > + */ > + > + while (i < add_column_count && plan[i].act == PLAN_NOP) > + i++; /* skip NOP */ > + > + j= end= plan[i].mv_end; > + if (i != add_column_count && > + (plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > + j++; > + > + /* copy first the data that was not replaced in original packed data */ > + if (start < end && plan[i].mv_length) > + { > + memmove((header_base + new_header_size + > + plan[i].mv_offset + plan[i].ddelta), > + header_base + header_size + plan[i].mv_offset, > + plan[i].mv_length); > + } > + str->length+= plan[i].mv_length; > + > + /* new data adding */ > + if (i < add_column_count) > + { > + if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > + { > + data_store(str, plan[i].val); /* Append new data */ > + } > + } > + } > + return ER_DYNCOL_OK; > } > > -#define DELTA_CHECK(S, D, C) \ > - if ((S) == 0) \ > - (S)= (D); \ > - else if (((S) > 0 && (D) < 0) || \ > - ((S) < 0 && (D) > 0)) \ > - { \ > - (C)= TRUE; \ > - break; \ > - } \ > - > > /** > Update the packed string with the given columns > @@ -1826,26 +3044,36 @@ dynamic_column_update_many(DYNAMIC_COLUM > > /* Set common variables for all plans */ > plan[i].ddelta= data_delta; > + plan[i].ndelta= name_delta; > /* get header delta in entries */ > plan[i].hdelta= header_delta; > plan[i].length= 0; /* Length if NULL */ > > - if (find_place(plan[i].num[0], > - (uchar *)str->str + FIXED_HEADER_SIZE, > - entry_size, column_count, &entry)) > + if (find_place(&header, plan[i].key, string_keys)) > { > - size_t entry_data_size; > + size_t entry_data_size, entry_name_size= 0; > > /* Data existed; We have to replace or delete it */ > > - entry_data_size= get_length(entry, header_end, > - offset_size, max_offset); > - if ((long) entry_data_size < 0) > + entry_data_size= hdr_interval_length(&header, header.entry + > + header.entry_size); > + if (entry_data_size == DYNCOL_OFFSET_ERROR || > + (long) entry_data_size < 0) > { > rc= ER_DYNCOL_FORMAT; > goto end; > } > > + //get_length(header.entry, header.dtpool, header.offset_size, > + //header.data_size); did you forget to delete this? > + if (new_header.format == DYNCOL_FMT_STR) > + { > + if (header.format == DYNCOL_FMT_STR) > + entry_name_size= header.entry[0]; > + else > + entry_name_size= numlen(uint2korr(header.entry)); > + } > + > if (plan[i].val->type == DYN_COL_NULL) > { > /* Inserting a NULL means delete the old data */ > @@ -1891,219 +3124,759 @@ dynamic_column_update_many(DYNAMIC_COLUM > goto end; > } > data_delta+= plan[i].length; > + if (new_header.format == DYNCOL_FMT_STR) > + name_delta+= ((LEX_STRING *)plan[i].key)->length; > } > } > - plan[i].place= entry; > + plan[i].place= header.entry; > } > plan[add_column_count].hdelta= header_delta; > plan[add_column_count].ddelta= data_delta; > - new_column_count= column_count + header_delta; > + plan[add_column_count].act= PLAN_NOP; > + plan[add_column_count].place= header.dtpool; > + > + new_header.column_count= header.column_count + header_delta; > > /* > Check if it is only "increasing" or only "decreasing" plan for (header > and data separately). > */ > - data_size= str->length - header_size - FIXED_HEADER_SIZE; > - new_data_size= data_size + data_delta; > - if ((new_offset_size= dynamic_column_offset_bytes(new_data_size)) >= > + new_header.data_size= header.data_size + data_delta; > + new_header.nmpool_size= new_header.nmpool_size + name_delta; > + DBUG_ASSERT(new_header.format != DYNCOL_FMT_NUM || > + new_header.nmpool_size == 0); > + if ((new_header.offset_size= > + dynamic_column_offset_bytes(new_header.data_size)) >= > MAX_OFFSET_LENGTH) > { > rc= ER_DYNCOL_LIMIT; > goto end; > } > > -#ifdef NOT_IMPLEMENTED > - /* if (new_offset_size != offset_size) then we have to rewrite header */ > - header_delta_sign= new_offset_size - offset_size; > + copy= ((header.format != new_header.format) || > + (new_header.format == DYNCOL_FMT_STR)); > + /* if (new_header.offset_size!=offset_size) then we have to rewrite header */ > + header_delta_sign= > + ((int)new_header.offset_size + new_fmt->fixed_hdr_entry) - > + ((int)header.offset_size + fmt->fixed_hdr_entry); > data_delta_sign= 0; > - for (i= 0; i < add_column_count; i++) > + // plan[add_column_count] contains last deltas. > + for (i= 0; i <= add_column_count && !copy; i++) > { > /* This is the check for increasing/decreasing */ > DELTA_CHECK(header_delta_sign, plan[i].hdelta, copy); > DELTA_CHECK(data_delta_sign, plan[i].ddelta, copy); > } > -#endif > - calc_param(&new_entry_size, &new_header_size, > - new_offset_size, new_column_count); > + calc_param(&new_header.entry_size, &new_header.header_size, > + new_fmt->fixed_hdr_entry, > + new_header.offset_size, new_header.column_count); > > /* > - The following code always make a copy. In future we can do a more > - optimized version when data is only increasing / decreasing. > + Need copy because: > + 1. Header/data parts moved in different directions. > + 2. There is no enough allocated space in the string. > + 3. Header and data moved in different directions. > */ > + if (copy || /*1*/ forgot to delete /*1*/ ? > + str->max_length < str->length + header_delta + data_delta || /*2*/ > + ((header_delta_sign < 0 && data_delta_sign > 0) || > + (header_delta_sign > 0 && data_delta_sign < 0))) /*3*/ > + rc= dynamic_column_update_copy(str, plan, add_column_count, > + &header, &new_header, > + convert); > + else > + if (header_delta_sign < 0) > + rc= dynamic_column_update_move_left(str, plan, header.offset_size, > + header.entry_size, > + header.header_size, > + new_header.offset_size, > + new_header.entry_size, > + new_header.header_size, > + header.column_count, > + new_header.column_count, > + add_column_count, header.dtpool, > + header.data_size); > + else > + /* > + rc= dynamic_column_update_move_right(str, plan, offset_size, > + entry_size, header_size, > + new_header.offset_size, > + new_header.entry_size, > + new_heder.header_size, column_count, > + new_header.column_count, > + add_column_count, header_end, > + header.data_size); > + */ > + rc= dynamic_column_update_copy(str, plan, add_column_count, > + &header, &new_header, > + convert); > +end: > + my_free(alloc_plan); > + return rc; > + > +create_new_string: > + /* There is no columns from before, so let's just add the new ones */ > + rc= ER_DYNCOL_OK; > + my_free(alloc_plan); > + if (not_null != 0) > + rc= dynamic_column_create_many_internal_fmt(str, add_column_count, > + (uint*)column_keys, values, > + str->str == NULL, > + string_keys); > + goto end; > +} > + > + > +/** > + Update the packed string with the given column > + > + @param str String where to write the data > + @param column_number Array of columns number > + @param values Array of columns values > + > + @return ER_DYNCOL_* return code > +*/ > + > + > +int dynamic_column_update(DYNAMIC_COLUMN *str, uint column_nr, > + DYNAMIC_COLUMN_VALUE *value) > +{ > + return dynamic_column_update_many(str, 1, &column_nr, value); > +} > + > + > +enum enum_dyncol_func_result > +dynamic_column_check(DYNAMIC_COLUMN *str) > +{ > + struct st_service_funcs *fmt; > + enum enum_dyncol_func_result rc= ER_DYNCOL_FORMAT; > + DYN_HEADER header; > + uint i; > + size_t data_offset= 0, name_offset= 0; > + size_t prev_data_offset= 0, prev_name_offset= 0; > + LEX_STRING name= {0,0}, prev_name= {0,0}; > + uint num= 0, prev_num= 0; > + void *key, *prev_key; > + enum enum_dynamic_column_type type= DYN_COL_NULL, prev_type= DYN_COL_NULL; > + > + DBUG_ENTER("dynamic_column_check"); > > - /*if (copy) */ > + if (str->length == 0) > { > - DYNAMIC_COLUMN tmp; > - uchar *header_base= (uchar *)str->str + FIXED_HEADER_SIZE, > - *write; > - if (dynamic_column_init_str(&tmp, > - (FIXED_HEADER_SIZE + new_header_size + > - new_data_size + DYNCOL_SYZERESERVE))) > - { > - rc= ER_DYNCOL_RESOURCE; > - goto end; > - } > - write= (uchar *)tmp.str + FIXED_HEADER_SIZE; > - /* Adjust tmp to contain whole the future header */ > - tmp.length= FIXED_HEADER_SIZE + new_header_size; > - set_fixed_header(&tmp, new_offset_size, new_column_count); > - data_delta= 0; > + DBUG_PRINT("info", ("empty string is OK")); > + DBUG_RETURN(ER_DYNCOL_OK); > + } > > - /* > - Copy data to the new string > - i= index in array of changes > - j= index in packed string header index > - */ > + bzero(&header, sizeof(header)); > > - for (i= 0, j= 0; i < add_column_count || j < column_count; i++) > - { > - size_t first_offset; > - uint start= j, end; > - LINT_INIT(first_offset); > + /* Check that header is OK */ > + if (read_fixed_header(&header, str)) > + { > + DBUG_PRINT("info", ("Reading fixed string header failed")); > + goto end; > + } > + fmt= fmt_data + header.format; > + calc_param(&header.entry_size, &header.header_size, > + fmt->fixed_hdr_entry, header.offset_size, > + header.column_count); > + /* headers are out of string length (no space for data and part of headers) */ > + if (fmt->fixed_hdr + header.header_size + header.nmpool_size > str->length) > + { > + DBUG_PRINT("info", ("Fixed header: %u Header size: %u " > + "Name pool size: %u but Strig length: %u", > + (uint)fmt->fixed_hdr, > + (uint)header.header_size, > + (uint)header.nmpool_size, > + (uint)str->length)); > + goto end; > + } > + header.header= (uchar*)str->str + fmt->fixed_hdr; > + header.nmpool= header.header + header.header_size; > + header.dtpool= header.nmpool + header.nmpool_size; > + header.data_size= str->length - fmt->fixed_hdr - > + header.header_size - header.nmpool_size; > > - /* > - Search in i and j for the next column to add from i and where to > - add. > - */ > + /* read and check headers */ > + if (header.format == DYNCOL_FMT_NUM) > + { > + key= &num; > + prev_key= &prev_num; > + } > + else > + { > + key= &name; > + prev_key= &prev_name; > + } > + for (i= 0, header.entry= header.header; > + i < header.column_count; > + i++, header.entry+= header.entry_size) > + { > > - while (i < add_column_count && plan[i].act == PLAN_NOP) > - i++; /* skip NOP */ > - if (i == add_column_count) > - j= end= column_count; > - else > + if (header.format == DYNCOL_FMT_NUM) > + { > + num= uint2korr(header.entry); > + } > + else > + { > + DBUG_ASSERT(header.format == DYNCOL_FMT_STR); > + name.length= header.entry[0]; > + name_offset= uint2korr(header.entry + 1); > + name.str= (char *)header.nmpool + name_offset; > + } > + if (type_and_offset_read(&type, &data_offset, > + header.entry + fmt->fixed_hdr_entry, > + header.offset_size)) > + goto end; > + > + DBUG_ASSERT(type != DYN_COL_NULL); > + if (data_offset > header.data_size) > + { > + DBUG_PRINT("info", ("Field order: %u Data offset: %u" > + " > Data pool size: %u", > + (uint)i, > + (uint)name_offset, > + (uint)header.nmpool_size)); > + goto end; > + } > + if (name_offset > header.nmpool_size) > + { > + DBUG_PRINT("info", ("Field order: %u Name offset: %u" > + " > Name pool size: %u", > + (uint)i, > + (uint)name_offset, > + (uint)header.nmpool_size)); > + goto end; > + } > + if (prev_type != DYN_COL_NULL) > + { > + /* It is not first entry */ > + if (prev_data_offset >= data_offset) > { > - /* > - old data portion. We don't need to check that j < column_count > - as plan[i].place is guaranteed to have a pointer inside the > - data. > - */ > - while (header_base + j * entry_size < plan[i].place) > - j++; > - end= j; > - if ((plan[i].act == PLAN_REPLACE || plan[i].act == PLAN_DELETE)) > - j++; /* data at 'j' will be removed */ > + DBUG_PRINT("info", ("Field order: %u Previous data offset: %u" > + " >= Current data offset: %u", > + (uint)i, > + (uint)prev_data_offset, > + (uint)data_offset)); > + goto end; > } > - > - if (plan[i].ddelta == 0 && offset_size == new_offset_size) > + if (prev_name_offset > name_offset) > { > - uchar *read= header_base + start * entry_size; > - DYNAMIC_COLUMN_TYPE tp; > - /* > - It's safe to copy the header unchanged. This is usually the > - case for the first header block before any changed data. > - */ > - if (start < end) /* Avoid memcpy with 0 */ > - { > - size_t length= entry_size * (end - start); > - memcpy(write, read, length); > - write+= length; > - } > - /* Read first_offset */ > - type_and_offset_read(&tp, &first_offset, read, offset_size); > + DBUG_PRINT("info", ("Field order: %u Previous name offset: %u" > + " > Current name offset: %u", > + (uint)i, > + (uint)prev_data_offset, > + (uint)data_offset)); > + goto end; > } > - else > + if ((*fmt->column_sort)(&prev_key, &key) >= 0) > { > - /* > - Adjust all headers since last loop. > - We have to do this as the offset for data has moved > - */ > - for (k= start; k < end; k++) > - { > - uchar *read= header_base + k * entry_size; > - size_t offs; > - uint nm; > - DYNAMIC_COLUMN_TYPE tp; > + DBUG_PRINT("info", ("Field order: %u Previous key >= Current key", > + (uint)i)); > + goto end; > + } > + } > + prev_num= num; > + prev_name= name; > + prev_data_offset= data_offset; > + prev_name_offset= name_offset; > + prev_type= type; > + } > + > + /* check data, which we can */ > + for (i= 0, header.entry= header.header; > + i < header.column_count; > + i++, header.entry+= header.entry_size) > + { > + DYNAMIC_COLUMN_VALUE store; > + // already checked by previouse pass > + type_and_offset_read(&header.type, &header.offset, > + header.entry + fmt->fixed_hdr_entry, > + header.offset_size); > + header.length= > + hdr_interval_length(&header, header.entry + header.entry_size); > + header.data= header.dtpool + header.offset; > + switch ((header.type)) { > + case DYN_COL_INT: > + rc= dynamic_column_sint_read(&store, header.data, header.length); > + break; > + case DYN_COL_UINT: > + rc= dynamic_column_uint_read(&store, header.data, header.length); > + break; > + case DYN_COL_DOUBLE: > + rc= dynamic_column_double_read(&store, header.data, header.length); > + break; > + case DYN_COL_STRING: > + rc= dynamic_column_string_read(&store, header.data, header.length); > + break; > + case DYN_COL_DECIMAL: > + rc= dynamic_column_decimal_read(&store, header.data, header.length); > + break; > + case DYN_COL_DATETIME: > + rc= dynamic_column_date_time_read(&store, header.data, > + header.length); > + break; > + case DYN_COL_DATE: > + rc= dynamic_column_date_read(&store, header.data, header.length); > + break; > + case DYN_COL_TIME: > + rc= dynamic_column_time_read(&store, header.data, header.length); > + break; > + case DYN_COL_NULL: > + default: > + rc= ER_DYNCOL_FORMAT; > + goto end; > + } > + if (rc != ER_DYNCOL_OK) > + { > + DBUG_ASSERT(rc < 0); > + DBUG_PRINT("info", ("Field order: %u Can't read data: %i", > + (uint)i, (int) rc)); > + goto end; > + } > + } > > - nm= uint2korr(read); /* Column nummber */ > - type_and_offset_read(&tp, &offs, read, offset_size); > - if (k == start) > - first_offset= offs; > - else if (offs < first_offset) > + rc= ER_DYNCOL_OK; > +end: > + DBUG_RETURN(rc); > +} > + > + > +enum enum_dyncol_func_result > +dynamic_column_val_str(DYNAMIC_STRING *str, DYNAMIC_COLUMN_VALUE *val, > + CHARSET_INFO *cs, my_bool quote) > +{ > + char buff[40]; > + int len; > + switch (val->type) { > + case DYN_COL_INT: > + len= snprintf(buff, sizeof(buff), "%lld", val->x.long_value); > + if (dynstr_append_mem(str, buff, len)) > + return ER_DYNCOL_RESOURCE; > + break; > + case DYN_COL_UINT: > + len= snprintf(buff, sizeof(buff), "%llu", val->x.ulong_value); > + if (dynstr_append_mem(str, buff, len)) > + return ER_DYNCOL_RESOURCE; > + break; > + case DYN_COL_DOUBLE: > + len= snprintf(buff, sizeof(buff), "%lg", val->x.double_value); > + if (dynstr_realloc(str, len + (quote ? 2 : 0))) > + return ER_DYNCOL_RESOURCE; > + if (quote) > + str->str[str->length++]= '"'; > + dynstr_append_mem(str, buff, len); > + if (quote) > + str->str[str->length++]= '"'; > + break; > + case DYN_COL_STRING: > + { > + char *alloc= NULL; > + char *from= val->x.string.value.str; > + uint bufflen; > + my_bool conv= !my_charset_same(val->x.string.charset, cs); > + my_bool rc; > + len= val->x.string.value.length; > + bufflen= (len * (conv ? cs->mbmaxlen : 1)); > + if (dynstr_realloc(str, bufflen)) > + return ER_DYNCOL_RESOURCE; > + > + // guaranty UTF-8 string for value > + if (!my_charset_same(val->x.string.charset, cs)) > + { > + uint dummy_errors; > + if (!quote) > { > - dynamic_column_column_free(&tmp); > - rc= ER_DYNCOL_FORMAT; > - goto end; > + /* convert to the destination */ > + str->length+= copy_and_convert_extended(str->str, bufflen, > + cs, > + from, len, > + val->x.string.charset, > + &dummy_errors); > + return ER_DYNCOL_OK; > } > - > - offs+= plan[i].ddelta; > - int2store(write, nm); > - /* write rest of data at write + COLUMN_NUMBER_SIZE */ > - type_and_offset_store(write, new_offset_size, tp, offs); > - write+= new_entry_size; > + if ((alloc= (char *)my_malloc(bufflen, MYF(0)))) > + { > + len= > + copy_and_convert_extended(alloc, bufflen, cs, > + from, len, val->x.string.charset, > + &dummy_errors); > + from= alloc; > + } > + else > + return ER_DYNCOL_RESOURCE; > } > + if (quote) > + rc= dynstr_append_quoted(str, from, len); > + else > + rc= dynstr_append_mem(str, from, len); > + if (alloc) > + my_free(alloc); > + if (rc) > + return ER_DYNCOL_RESOURCE; > + break; > } > + case DYN_COL_DECIMAL: > + len= sizeof(buff); > + decimal2string(&val->x.decimal.value, buff, &len, > + 0, val->x.decimal.value.frac, > + '0'); > + if (dynstr_append_mem(str, buff, len)) > + return ER_DYNCOL_RESOURCE; > + break; > + case DYN_COL_DATETIME: > + case DYN_COL_DATE: > + case DYN_COL_TIME: > + len= my_TIME_to_str(&val->x.time_value, buff, AUTO_SEC_PART_DIGITS); > + if (dynstr_realloc(str, len + (quote ? 2 : 0))) > + return ER_DYNCOL_RESOURCE; > + if (quote) > + str->str[str->length++]= '"'; > + dynstr_append_mem(str, buff, len); > + if (quote) > + str->str[str->length++]= '"'; > + break; > + case DYN_COL_NULL: > + if (dynstr_append_mem(str, "null", 4)) > + return ER_DYNCOL_RESOURCE; > + break; > + default: > + return(ER_DYNCOL_FORMAT); > + } > + return(ER_DYNCOL_OK); > +} > > - /* copy first the data that was not replaced in original packed data */ > - if (start < end) > + > +enum enum_dyncol_func_result > +dynamic_column_val_long(longlong *ll, DYNAMIC_COLUMN_VALUE *val) > +{ > + enum enum_dyncol_func_result rc= ER_DYNCOL_OK; > + *ll= 0; > + switch (val->type) { > + case DYN_COL_INT: > + *ll= val->x.long_value; > + break; > + case DYN_COL_UINT: > + *ll= (longlong)val->x.ulong_value; > + if (val->x.ulong_value > ULONGLONG_MAX) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_DOUBLE: > + *ll= (longlong)val->x.double_value; > + if (((double) *ll) != val->x.double_value) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_STRING: > { > - /* Add old data last in 'tmp' */ > - size_t data_size= > - get_length_interval(header_base + start * entry_size, > - header_base + end * entry_size, > - header_end, offset_size, max_offset); > - if ((long) data_size < 0 || > - data_size > max_offset - first_offset) > + longlong i= 0, sign= 1; > + char *src= val->x.string.value.str; > + uint len= val->x.string.value.length; > + > + while (len && my_isspace(&my_charset_latin1, *src)) src++,len--; > + > + if (len) > { > - dynamic_column_column_free(&tmp); > - rc= ER_DYNCOL_FORMAT; > - goto end; > + if (*src == '-') > + { > + sign= -1; > + src++; > + } else if (*src == '-') > + src++; > + while(len && my_isdigit(&my_charset_latin1, *src)) > + { > + i= i * 10 + (*src - '0'); > + src++; > + } > } > - > - memcpy(tmp.str + tmp.length, (char *)header_end + first_offset, > - data_size); > - tmp.length+= data_size; > + else > + rc= ER_DYNCOL_TRUNCATED; > + if (len) > + rc= ER_DYNCOL_TRUNCATED; > + *ll= i * sign; > + break; > } > + case DYN_COL_DECIMAL: > + if (decimal2longlong(&val->x.decimal.value, ll) != E_DEC_OK) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_DATETIME: > + *ll= (val->x.time_value.year * 10000000000L + > + val->x.time_value.month * 100000000L + > + val->x.time_value.day * 1000000 + > + val->x.time_value.hour * 10000 + > + val->x.time_value.minute * 100 + > + val->x.time_value.second) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_DATE: > + *ll= (val->x.time_value.year * 10000 + > + val->x.time_value.month * 100 + > + val->x.time_value.day) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_TIME: > + *ll= (val->x.time_value.hour * 10000 + > + val->x.time_value.minute * 100 + > + val->x.time_value.second) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_NULL: > + rc= ER_DYNCOL_TRUNCATED; > + break; > + default: > + return(ER_DYNCOL_FORMAT); > + } > + return(rc); > +} > + > > - /* new data adding */ > - if (i < add_column_count) > +enum enum_dyncol_func_result > +dynamic_column_val_double(double *dbl, DYNAMIC_COLUMN_VALUE *val) > +{ > + enum enum_dyncol_func_result rc= ER_DYNCOL_OK; > + *dbl= 0; > + switch (val->type) { > + case DYN_COL_INT: > + *dbl= (double)val->x.long_value; > + if (((longlong) *dbl) != val->x.long_value) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_UINT: > + *dbl= (double)val->x.ulong_value; > + if (((ulonglong) *dbl) != val->x.ulong_value) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_DOUBLE: > + *dbl= val->x.double_value; > + break; > + case DYN_COL_STRING: > { > - if( plan[i].act == PLAN_ADD || plan[i].act == PLAN_REPLACE) > - { > - int2store(write, plan[i].num[0]); > - type_and_offset_store(write, new_offset_size, > - plan[i].val[0].type, > - tmp.length - > - (FIXED_HEADER_SIZE + new_header_size)); > - write+= new_entry_size; > - data_store(&tmp, plan[i].val); /* Append new data */ > - } > - data_delta= plan[i].ddelta; > + char *str, *end; > + if ((str= malloc(val->x.string.value.length + 1))) > + return ER_DYNCOL_RESOURCE; > + memcpy(str, val->x.string.value.str, val->x.string.value.length); > + str[val->x.string.value.length]= '\0'; > + *dbl= strtod(str, &end); > + if (*end != '\0') > + rc= ER_DYNCOL_TRUNCATED; > } > - } > - dynamic_column_column_free(str); > - *str= tmp; > + case DYN_COL_DECIMAL: > + if (decimal2double(&val->x.decimal.value, dbl) != E_DEC_OK) > + rc= ER_DYNCOL_TRUNCATED; > + break; > + case DYN_COL_DATETIME: > + *dbl= (double)(val->x.time_value.year * 10000000000L + > + val->x.time_value.month * 100000000L + > + val->x.time_value.day * 1000000 + > + val->x.time_value.hour * 10000 + > + val->x.time_value.minute * 100 + > + val->x.time_value.second) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_DATE: > + *dbl= (double)(val->x.time_value.year * 10000 + > + val->x.time_value.month * 100 + > + val->x.time_value.day) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_TIME: > + *dbl= (double)(val->x.time_value.hour * 10000 + > + val->x.time_value.minute * 100 + > + val->x.time_value.second) * > + (val->x.time_value.neg ? -1 : 1); > + break; > + case DYN_COL_NULL: > + rc= ER_DYNCOL_TRUNCATED; > + break; > + default: > + return(ER_DYNCOL_FORMAT); > } > + return(rc); > +} > > - rc= ER_DYNCOL_OK; > > -end: > - my_free(plan); > - return rc; > +/** > + Convert to JSON > > -create_new_string: > - /* There is no columns from before, so let's just add the new ones */ > - rc= ER_DYNCOL_OK; > - if (not_null != 0) > - rc= dynamic_column_create_many_internal(str, add_column_count, > - column_numbers, values, > - str->str == NULL); > - goto end; > + @param str The packed string > + @param json Where to put json result > + > + @return ER_DYNCOL_* return code > +*/ > + > +enum enum_dyncol_func_result > +dynamic_column_json(DYNAMIC_COLUMN *str, DYNAMIC_STRING *json) > +{ > + DYN_HEADER header; > + uint i; > + enum enum_dyncol_func_result rc; > + > + bzero(json, sizeof(DYNAMIC_STRING)); /* In case of errors */ > + if (str->length == 0) > + return ER_DYNCOL_OK; /* no columns */ > + > + if ((rc= init_read_hdr(&header, str)) < 0) > + return rc; > + > + if (header.entry_size * header.column_count + FIXED_HEADER_SIZE > > + str->length) > + return ER_DYNCOL_FORMAT; > + > + if (init_dynamic_string(json, NULL, str->length * 2, 100)) > + return ER_DYNCOL_RESOURCE; > + > + if (dynstr_append_mem(json, "[", 1)) > + return ER_DYNCOL_RESOURCE; > + rc= ER_DYNCOL_RESOURCE; > + for (i= 0, header.entry= header.header; > + i < header.column_count; > + i++, header.entry+= header.entry_size) > + { > + DYNAMIC_COLUMN_VALUE val; > + if (i != 0 && dynstr_append_mem(json, ",", 1)) > + goto err; > + header.length= > + hdr_interval_length(&header, header.entry + header.entry_size); > + header.data= header.dtpool + header.offset; > + /* > + Check that the found data is withing the ranges. This can happen if > + we get data with wrong offsets. > + */ > + if (header.length == DYNCOL_OFFSET_ERROR || > + header.length > INT_MAX || header.offset > header.data_size) > + { > + rc= ER_DYNCOL_FORMAT; > + goto err; > + } > + if ((rc= dynamic_column_get_value(&header, &val)) < 0 || > + dynstr_append_mem(json, "{", 1)) > + goto err; > + if (header.format == DYNCOL_FMT_NUM) > + { > + uint nm= uint2korr(header.entry); > + if (dynstr_realloc(json, DYNCOL_NUM_CHAR + 3)) > + goto err; > + json->str[json->length++]= '"'; > + json->length+= (snprintf(json->str + json->length, > + DYNCOL_NUM_CHAR, "%u", nm)); > + } > + else > + { > + uint len= header.entry[0]; > + if (dynstr_realloc(json, len + 3)) > + goto err; > + json->str[json->length++]= '"'; > + memcpy(json->str + json->length, (const void *)header.nmpool + > + uint2korr(header.entry + 1), len); > + json->length+= len; > + } > + json->str[json->length++]= '"'; > + json->str[json->length++]= ':'; > + if ((rc= dynamic_column_val_str(json, &val, > + &my_charset_utf8_general_ci, TRUE)) < 0 || > + dynstr_append_mem(json, "}", 1)) > + goto err; > + } > + if (dynstr_append_mem(json, "]", 1)) > + return ER_DYNCOL_RESOURCE; > + return ER_DYNCOL_OK; > + > +err: > + json->length= 0; > + return rc; > } > > > /** > - Update the packed string with the given column > + Convert to DYNAMIC_COLUMN_VALUE values and names (LEX_STING) dynamic array > > - @param str String where to write the data > - @param column_number Array of columns number > - @param values Array of columns values > + @param str The packed string > + @param names Where to put names > + @param vals Where to put values > + @param free_names pointer to free names buffer if there is it. > > @return ER_DYNCOL_* return code > */ > > - > -int dynamic_column_update(DYNAMIC_COLUMN *str, uint column_nr, > - DYNAMIC_COLUMN_VALUE *value) > +enum enum_dyncol_func_result > +dynamic_column_vals(DYNAMIC_COLUMN *str, > + DYNAMIC_ARRAY *names, DYNAMIC_ARRAY *vals, > + char **free_names) > { > - return dynamic_column_update_many(str, 1, &column_nr, value); > + DYN_HEADER header; > + char *nm; > + uint i; > + enum enum_dyncol_func_result rc; > + > + *free_names= 0; > + bzero(names, sizeof(DYNAMIC_ARRAY)); /* In case of errors */ > + bzero(vals, sizeof(DYNAMIC_ARRAY)); /* In case of errors */ > + if (str->length == 0) > + return ER_DYNCOL_OK; /* no columns */ > + > + if ((rc= init_read_hdr(&header, str)) < 0) > + return rc; > + > + if (header.entry_size * header.column_count + FIXED_HEADER_SIZE > > + str->length) > + return ER_DYNCOL_FORMAT; > + > + if (init_dynamic_array(names, sizeof(LEX_STRING), > + header.column_count, 0) || > + init_dynamic_array(vals, sizeof(DYNAMIC_COLUMN_VALUE), > + header.column_count, 0) || > + (header.format == DYNCOL_FMT_NUM && > + !(*free_names= (char *)malloc(DYNCOL_NUM_CHAR * header.column_count)))) why do you need a special malloc() for names, you can put them in the buffer of 'names' dynarray. it'll be a lot more convenient for the caller (no free_names to care about). > + { > + rc= ER_DYNCOL_RESOURCE; > + goto err; > + } > + nm= *free_names; > + > + for (i= 0, header.entry= header.header; > + i < header.column_count; > + i++, header.entry+= header.entry_size) > + { > + DYNAMIC_COLUMN_VALUE val; > + LEX_STRING name; > + header.length= > + hdr_interval_length(&header, header.entry + header.entry_size); > + header.data= header.dtpool + header.offset; > + /* > + Check that the found data is withing the ranges. This can happen if > + we get data with wrong offsets. > + */ > + if (header.length == DYNCOL_OFFSET_ERROR || > + header.length > INT_MAX || header.offset > header.data_size) > + { > + rc= ER_DYNCOL_FORMAT; > + goto err; > + } > + if ((rc= dynamic_column_get_value(&header, &val)) < 0) > + goto err; > + > + if (header.format == DYNCOL_FMT_NUM) > + { > + uint num= uint2korr(header.entry); > + name.str= nm; > + name.length= snprintf(nm, DYNCOL_NUM_CHAR, "%u", num); > + nm+= name.length + 1; > + } > + else > + { > + name.length= header.entry[0]; > + name.str= (char *)header.nmpool + uint2korr(header.entry + 1); > + } > + /* following is preallocated and so do not fail */ > + (void) insert_dynamic(names, (uchar *)&name); > + (void) insert_dynamic(vals, (uchar *)&val); > + } > + return ER_DYNCOL_OK; > + > +err: > + delete_dynamic(names); > + delete_dynamic(vals); > + if (*free_names) > + my_free(*free_names); > + *free_names= 0; > + return rc; > } > Regards, Sergei

1 0

[Maria-developers] a mysterious (to me) block of code in the middle of ha_recover
by Rich Prohaska 13 Dec '12

13 Dec '12

Hello, The ha_recover function in MySQL 5.5.28 has a block of code in the middle of it labelled with WILL_BE_DELETED_LATER that gets compiled into mysqld. The ha_recover function in Mariadb 5.5.28a does not have this code in it. It appears that the code forces prepared transactions to be rolled back in MySQL even when mysqld is past the recover phase. Does anyone know what this code does? Does anyone know why this code exists in MySQL but not in Mariadb? Thanks

2 1

[Maria-developers] a mysterious (to me) block of code in the middle of ha_recover
by Rich Prohaska 13 Dec '12

13 Dec '12

Hello, The ha_recover function in MySQL 5.5.28 has a block of code in the middle of it labelled with WILL_BE_DELETED_LATER that gets compiled into mysqld. The ha_recover function in Mariadb 5.5.28a does not have this code in it. It appears that the code forces prepared transactions to be rolled back in MySQL. Does anyone know what this code does? Does anyone know why this code exists in MySQL but not in Mariadb? Thanks Rich Prohaska

1 0

[Maria-developers] release day of week for security releases
by Daniel Black 07 Dec '12

07 Dec '12

Hi, Thanks for the latest releases with security fixes. While I appreciate that all of the development of these security fixes was in public (without mentioning it was a security fix - well at least the remote code exec), I'm wondering if security releases could occur on a weekday where sysadmins need not forsake part of their weekend to correct a public vulnerability. Just my thoughts and preferences. I appreciate others may consider things different. It also appears that the fedora 17 mariadb galera updates are only partially pushed. Maybe its just my setup after switching from non-galera repo. $ sudo yum update [sudo] password for dan: Loaded plugins: langpacks, presto, priorities, refresh-packagekit, security 38 packages excluded due to repository priority protections Resolving Dependencies --> Running transaction check ---> Package MariaDB-client.x86_64 0:5.5.25-1 will be updated ---> Package MariaDB-client.x86_64 0:5.5.27-1 will be an update ---> Package MariaDB-common.x86_64 0:5.5.25-1 will be updated ---> Package MariaDB-common.x86_64 0:5.5.27-1 will be an update ---> Package MariaDB-server.x86_64 0:5.5.25-1 will be updated ---> Package MariaDB-server.x86_64 0:5.5.27-1 will be obsoleting ---> Package mysql.x86_64 0:5.5.28-1.fc17 will be obsoleted --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================== Installing: MariaDB-server x86_64 5.5.27-1 mariadb 33 M replacing mysql.x86_64 5.5.28-1.fc17 Updating: MariaDB-client x86_64 5.5.27-1 mariadb 8.6 M MariaDB-common x86_64 5.5.27-1 mariadb 23 k Transaction Summary ============================================================================================================================================================== Install 1 Package Upgrade 2 Packages Total size: 42 M Total download size: 8.6 M Is this ok [y/N]: y Downloading Packages: Setting up and reading Presto delta metadata MariaDB-5.5.27-fedora17-x86_64 FAILED HTTP Error 404 - Not Found : http://yum.mariadb.org/5.5-galera/fedora17-amd64/rpms/MariaDB-5.5.27-fedora… http://yum.mariadb.org/5.5-galera/fedora17-amd64/rpms/MariaDB-5.5.27-fedora…: [Errno 14] HTTP Error 404 - Not Found : http://yum.mariadb.org/5.5-galera/fedora17-amd64/rpms/MariaDB-5.5.27-fedora… /etc/yum.repos.d/mariadb.repo : [mariadb] name = MariaDB baseurl = http://yum.mariadb.org/5.5-galera/fedora17-amd64 gpgcheck=1 enabled=1 -- Daniel Black, Engineer @ Open Query (http://openquery.com) Remote expertise & maintenance for MySQL/MariaDB server environments.

2 2

[Maria-developers] [[Bug] skip-character-set-client-handshake]
by spumer-tm＠yandex.ru 07 Dec '12

07 Dec '12

>Description: <I'm using MariaDB with my gameservers. Server works with db through sourcemod. In my db i'am using utf8 encoding, and all works fine on MariaDB 5.3.8 with params: init-connect = "SET NAMES 'utf8' COLLATE 'utf8_general_ci'" skip-character-set-client-handshake But after upgrade to 5.5.27 i found a 'bug'. When i'm send query from my game server, over time character_set_connection, character_set_client and character_set_results set to latin1. After enable log i saw: 88 Connect login@domain as anonymous on sourcebans 88 Query SET NAMES 'utf8' COLLATE 'utf8_general_ci' 88 Query SET NAMES latin1 88 Query SHOW VARIABLES LIKE '%character_set%' But my request is only show vars. I assumed that sourcemod is guilty. But skip-character-set-client-handshake is set! I disable it, and try again. At this time i send SET collation_connection = utf8_general_ci and SET NAMES utf8 after my query. But over time it set to latin1 again. No errosr in error.log Sorry for my english.> >How-To-Repeat: <I don't know how to repeat it manually (without sourcemod app)> >Fix: <It works fine in previous versions. Just not accept NAMES from client.> >Submitter-Id: <submitter ID> >Originator: spumer >Organization: <russian game servers> >MySQL support: [ email support ] >Synopsis: <skip-character-set-client-handshake not work correctly> >Severity: >Priority: >Category: mysql >Class: >Release: mysql-5.5.28a (MariaDB Server) >C compiler: gcc-4.5.3 (Gentoo 4.5.3-r1 p1.0, pie-0.4.5) 4.5.3 >C++ compiler: g++-4.5.3(Gentoo 4.5.3-r1 p1.0, pie-0.4.5) 4.5.3 >Environment: <VMware ESXi 5, Gentoo, Glibc 2.15> System: Linux localhost 3.6.1-gentoo #1 SMP Mon Oct 8 06:00:23 YEKT 2012 x86_64 Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz GenuineIntel GNU/Linux Some paths: /usr/bin/perl /usr/bin/make /usr/bin/gmake /usr/bin/gcc /usr/bin/cc GCC: Используются внутренние спецификации. COLLECT_GCC=/usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3/gcc COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/4.5.3/lto-wrapper Целевая архитектура: x86_64-pc-linux-gnu Параметры конфигурации: /var/tmp/portage/sys-devel/gcc-4.5.3-r1/work/gcc-4.5.3/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/4.5.3 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.5.3/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.5.3 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.5.3/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.5.3/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.5.3/include/g++-v4 --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --disable-altivec --disable-fixed-point --without-ppl --without-cloog --disable-lto --enable-nls --without-included-gettext --with-system-zlib --disable-werror --enable-secureplt --enable-multilib --enable-libmudflap --disable-libssp --enable-libgomp --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/4.5.3/python --enable-checking=release --disable-libgcj --enable-languages=c,c++,fortran --enable-shared --enable-threads=posix --enable-__cxa_a! texit --enable-clocale=gnu --enable-targets=all --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.5.3-r1 p1.0, pie-0.4.5' Модель многопоточности: posix gcc версия 4.5.3 (Gentoo 4.5.3-r1 p1.0, pie-0.4.5) Compilation info (call): CC='/usr/bin/gcc' CFLAGS='-Wall -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' CXX='/usr/bin/c++' CXXFLAGS='-Wall -Wno-unused-parameter -fno-implicit-templates -fno-exceptions -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' LDFLAGS='' ASFLAGS='' Compilation info (used): CC='/usr/bin/gcc' CFLAGS='-Wall -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' CXX='/usr/bin/c++' CXXFLAGS='-Wall -Wno-unused-parameter -fno-implicit-templates -fno-exceptions -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -DMY_PTHREAD_FASTMUTEX=1' LDFLAGS='' ASFLAGS='' LIBC: lrwxrwxrwx 1 root root 12 дек. 2 12:43 /lib/libc.so.6 -> libc-2.15.so -rwxr-xr-x 1 root root 1734576 дек. 2 12:42 /lib/libc-2.15.so -rw-r--r-- 1 root root 4594916 дек. 2 12:42 /usr/lib/libc.a -rwxr-xr-x 1 root root 253 дек. 2 12:42 /usr/lib/libc.so Perl: This is perl 5, version 12, subversion 4 (v5.12.4) built for x86_64-linux-thread-multi

2 1

[Maria-developers] test case for mdev 3885 is named incorrectly
by Rich Prohaska 06 Dec '12

06 Dec '12

revision 3584 named the test case for mdev 3885 "test_mdev3855" (NOTE 3855).

2 1

Re: [Maria-developers] How to get JDBC launchpad branch to trigger build in buildbot?
by Kristian Nielsen 03 Dec '12

03 Dec '12

"Vladislav Vaintroub" <wlad(a)montyprogram.com> writes: > there is a change in the Launchpad repository . I have done some things > that I hoped would help, but it looks like it does not help. > I subscribed Buildbot user to the changes in branch > https://code.launchpad.net/~maria-captains/mariadb-java-client/trunk It seems to be subscribed as "Branch attribute notifications only". Check the wiki page, it has to be "branch revision notifications only" (and it's probably not very useful to send review requests to Buildbot :-) > + "lp:~maria-captains/mariadb-java-client/trunk" : "jdbc", As explained on the wiki page, the branch should be on Launchpad under some name lp:~maria-captains/maria/XXX (for some XXX) for this to work. > +f_win_jdbc = factory.BuildFactory() > +f_win32_debug.addStep(ShellCommand( ^^^^^^^^^^^^^ copy-paste error? "f_win_jdbc" ? > +f_win_rqg.addStep(ShellCommand( ^^^^^^^^^ "f_win_jdbc" ? > + name = "bzr_checkout", > + command=["dojob", "bzr" ,"checkout", "-r", WithProperties("%(revision)s"), WithProperties("lp:~maria-captains/maria/%(branch)s"), WithProperties("c:\\buildbot\\%(buildername)s\\build")], Did you mean "lp:~maria-captains/mariadb-java-client/%(branch)s" here? Or did you mean lp:~maria-captains/maria/jdbc in the branch declaration? > + WithProperties("cd c:\\buildbot\\%(buildername)s\\build && C:\\buildscripts\\build_jdbc") Did you consider putting this script into the source tree instead? That makes it easier when you have to modify the script along with the code in the source three: old branches continue to use the old version of the script with the old source version, new branches can use the new script with the new source changes. (But it's up to you what works best in this particular case). > + "nextBuild": myNextBuild, The myNextBuild() function selects among mariadb branches, that's probably not very useful for a jdbc builder? > see. Maybe you can have a quick look, if you got time? Would be a great > help. Ok, I will go through it all and try to get everything working. It's probably better to add a separate configuration (branch map, schedulers, builders) for the jdbc driver to not get it mixed up with the mariadb source builds. - Kristian.

2 2

[Maria-developers] API Documentaion for Replication Features
by Gordan Bobic 03 Dec '12

03 Dec '12

Hi, I have a need for replicating data from multiple masters (independent data going to different tables). I have looked into the following with no success: Tungsten: Slow, CPU hungry, outrageously bloated, and unacceptably unreliable (replication will stall silently at times, and the only way to resume it is to offline and online the replication service). mysql-replication-listener: Just about got it to work with local binlogs, but haven't managed to get it to connect via TCP to a MySQL server. So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: 1) Slave TCP connections to the master and binlog fetching 2) Binlog parsing 3) Config file parsing In the longer term I'd like to write a patch that adds multiple-master functionality into MySQL itself. For configuring it, I was thinking about the following WRT configuration in my.cnf with multiple section instances, e.g.: [replicator] server-id=12345 master-host=server1 master-port=3306 master-user=user1 master-password=password1 replicate-do-table=db1.table1 [replicator] server-id=12345 master-host=server2 master-port=3306 master-user=user2 master-password=password2 replicate-do-table=db2.table2 etc. Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand. TIA. Gordan

4 9

[Maria-developers] MariaDB 5.5.28a release prep
by Sergei Golubchik 28 Nov '12

28 Nov '12

Preliminary release notes and changelog pages have been created for 5.5.28a https://kb.askmonty.org/en/mariadb-5528a-release-notes/ https://kb.askmonty.org/en/mariadb-5528a-changelog/ Please let me know if there is anything in particular which should be mentioned in the release notes. Thanks, Sergei _______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers(a)lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp

1 0

[Maria-developers] Review of mdev-452
by Michael Widenius 28 Nov '12

28 Nov '12

Hi! Here is the review of https://mariadb.atlassian.net/browse/MDEV-452 (Add full support for auto-initialized/updated timestamp and datetime) === modified file 'mysql-test/r/log_tables.result' --- mysql-test/r/log_tables.result 2011-10-19 19:45:18 +0000 +++ mysql-test/r/log_tables.result 2012-10-17 12:43:56 +0000 @@ -53,7 +53,7 @@ ERROR HY000: You can't use locks with lo show create table mysql.general_log; Table Create Table general_log CREATE TABLE `general_log` ( - `event_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + `event_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), Why does all log tables now have CURRENT_TIMESTAMP(6) instead of CURRENT_TIMESTAMP ? This would cause a problem for someone that updates to MySQL 10.0 and then want to go back... Many of the other result file changes comes from the same change for general_log and slow_log. Is this also the case for MySQL 5.6 or only for MariaDB? Answer from Timour on IRC: Timour> This comes from MySQL 5.6 and we need to stay compatbile. === modified file 'sql/event_db_repository.cc' --- sql/event_db_repository.cc 2012-09-04 16:26:30 +0000 +++ sql/event_db_repository.cc 2012-10-17 12:43:56 +0000 @@ -829,9 +829,6 @@ Event_db_repository::update_event(THD *t (int) table->field[ET_FIELD_ON_COMPLETION]->val_int())) goto end; - /* Don't update create on row update. */ - table->timestamp_field_type= TIMESTAMP_NO_AUTO_SET; - Why don't we anymore need to tell the rest of the code that the automatic fields shouldn't be updated from this function? Timour> As I remember, the main change compared to the old design is Timour> that we no longer update timestamps inside the handler Timour> So the place where timestamp updates are triggered is such Timour> that such updates will no longer be triggered in these cases. Yes, mysql_event_fill_row() doesn't call fill_record_n_invoke_before_triggers(). It instead has own code that probably not calling setting of any default fields. This does however mean that one can never get automatic timestamps with events. === modified file 'sql/field.cc' --- sql/field.cc 2012-09-08 22:22:06 +0000 +++ sql/field.cc 2012-10-17 12:43:56 +0000 @@ -4419,10 +4419,12 @@ Field_timestamp::Field_timestamp(uchar * { /* For 4.0 MYD and 4.0 InnoDB compatibility */ flags|= UNSIGNED_FLAG | BINARY_FLAG; - if (unireg_check != NONE && !share->timestamp_field) + if (unireg_check != NONE) { - /* This timestamp has auto-update */ - share->timestamp_field= this; + /* + This TIMESTAMP column is hereby quietly assumed to have an insert or + update default function. + */ We need to here add a comment what the TIMESTAMP_FLAG now means, as it has lost it's orginal meaning. Add to the above comment: /* We mark the flag with TIMESTAMP_FLAG to indicate to the client that this field will be automaticly updated on insert */ flags|= TIMESTAMP_FLAG; if (unireg_check != TIMESTAMP_DN_FIELD) flags|= ON_UPDATE_NOW_FLAG; Please also fix the function comment that refers to TABLE::timestamp_field <cut> +void Field_timestamp::set_explicit_default(Item *value) +{ + if (value && + ((value->type() == Item::DEFAULT_VALUE_ITEM && + !((Item_default_value*)value)->arg) || + (!maybe_null() && value->is_null()))) + return; + flags|= HAS_EXPLICIT_DEFAULT; +} The above function confused me greatly (mostly because of names) You should not have to check for value != 0. An item should never be 0. I noticed that some code called set_explicit_default() with NULL, but it would be better to for this add a general item function: field->set_explicit_default(NULL) -> field->mark_that_field_has_been_given_a_value() Where: Field::mark_that_field_has_been_given_a_value() { flags|= HAS_EXPLICIT_VALUE; } (Note name change for HAS_EXPLICIT_DEFAULT) This can be inline and we will win some function calls. The set_explicit_default() function also needs a comment as it's very confusing. To clarify the code, lets first add a function: /* Returns 1 if the default returns the default value for a column */ Item_default_value::default_column() { return arg != 0; } And then original function would look like: /** Mark columns that has been given a value with HAS_EXPLICIT_VALUE. For timestamp columns, the only case where a column is not marked with been given a value are: - It's explicitely assigned with DEFAULT - We assign NULL to a timestamp field that is defined as NOT NULL. This is how MySQL has worked since it's start. */ void Field_timestamp::set_explicit_default(Item *value) { if (((value->type() == Item::DEFAULT_VALUE_ITEM && !(Item_default_value*) value)->default_column()) || (!maybe_null() && value->is_null())) return; mark_that_field_has_been_given_a_value(); } <cut> + +int Field_datetime::set_time() +{ + THD *thd= current_thd; current_thd -> table->in_use; + MYSQL_TIME now_time; + thd->variables.time_zone->gmt_sec_to_TIME(&now_time, thd->query_start()); + now_time.second_part= thd->query_start_sec_part(); + set_notnull(); + store_TIME(&now_time); + thd->time_zone_used= 1; + return 0; +} @@ -8857,16 +8866,37 @@ bool Create_field::init(THD *thd, char * { uint sign_len, allowed_type_modifier= 0; ulong max_field_charlength= MAX_FIELD_CHARLENGTH; + const bool on_update_is_function= + (fld_on_update_value != NULL && + fld_on_update_value->type() == Item::FUNC_ITEM); DBUG_ENTER("Create_field::init()"); field= 0; field_name= fld_name; - def= fld_default_value; flags= fld_type_modifier; option_list= create_opt; - unireg_check= (fld_type_modifier & AUTO_INCREMENT_FLAG ? - Field::NEXT_NUMBER : Field::NONE); + + if (fld_default_value != NULL && fld_default_value->type() == Item::FUNC_ITEM) + { + /* There is a function default for insertions. */ + def= NULL; + unireg_check= on_update_is_function ? + Field::TIMESTAMP_DNUN_FIELD : // for insertions and for updates. + Field::TIMESTAMP_DN_FIELD; // only for insertions. Add braces around the on_update_is... to get better indentation. + } + else + { + /* No function default for insertions. Either NULL or a constant. */ + def= fld_default_value; + if (on_update_is_function) + unireg_check= Field::TIMESTAMP_UN_FIELD; // function default for updates + else + unireg_check= (fld_type_modifier & AUTO_INCREMENT_FLAG) != 0 ? + Field::NEXT_NUMBER : // Automatic increment. + Field::NONE; Add () around expression. + } + <cut> + + +/** + Mark the field as having an explicit default value. + + @param value if available, the value that the field is being set to + + @note + Fields that have an explicit default value should not be updated + automatically via the DEFAULT or ON UPDATE functions. The functions + that deal with data change functionality (INSERT/UPDATE/LOAD), + determine if there is an explicit value for each field before performing + the data change, and call this method to mark the field. + + If the 'value' parameter is NULL, then the field is marked unconditionally + as having an explicit value. If 'value' is not NULL, then it can be further + analyzed to check if it really should count as a value. +*/ + +void Field::set_explicit_default(Item *value) +{ /* Return if value is DEFAULT */ + if (value && value->type() == Item::DEFAULT_VALUE_ITEM && + !((Item_default_value*)value)->arg) + return; + flags|= HAS_EXPLICIT_DEFAULT; +} Remove checking if value is null. flags|= HAS_EXPLICIT_DEFAULT -> flags|= HAS_EXPLICIT_VALUE === modified file 'sql/field.h' @@ -1239,12 +1270,26 @@ class Field_timestamp :public Field_str virtual int set_time(); <cut> + virtual void set_explicit_default(Item *value); + virtual int evaluate_insert_default_function() + { + int res= 0; + if (has_insert_default_function()) + res= set_time(); + return res; + } + virtual int evaluate_update_default_function() + { + int res= 0; + if (has_update_default_function()) + res= set_time(); + return res; + } /* Get TIMESTAMP field value as seconds since begging of Unix Epoch */ virtual my_time_t get_timestamp(ulong *sec_part) const; virtual void store_TIME(my_time_t timestamp, ulong sec_part) @@ -1252,7 +1297,6 @@ class Field_timestamp :public Field_str int4store(ptr,timestamp); } bool get_date(MYSQL_TIME *ltime, ulonglong fuzzydate); - timestamp_auto_set_type get_auto_set_type() const; uchar *pack(uchar *to, const uchar *from, uint max_length __attribute__((unused))) { @@ -1503,6 +1547,28 @@ class Field_datetime :public Field_tempo void sql_type(String &str) const; bool zero_pack() const { return 1; } bool get_date(MYSQL_TIME *ltime, ulonglong fuzzydate); + virtual int set_time(); + virtual void set_default() + { + if (has_insert_default_function()) + set_time(); + else + Field::set_default(); + } + virtual int evaluate_insert_default_function() + { + int res= 0; + if (has_insert_default_function()) + res= set_time(); + return res; + } + virtual int evaluate_update_default_function() + { + int res= 0; + if (has_update_default_function()) + res= set_time(); + return res; + } It's a bit of a shame that we need to have the exact same functions twice? One way to solve it would be to add these functions to Field_str, as both Field_timestamp() and Field_datetime() depends on this class. === modified file 'sql/sql_base.cc' <cut> static bool -fill_record(THD * thd, List<Item> &fields, List<Item> &values, +fill_record(THD * thd, TABLE *table_arg, List<Item> &fields, List<Item> &values, bool ignore_errors) { Why this change (was not needed for this functionality). timour> I think I had a different implementation before, then I kept this timrou> change, because it looked simpler. bool -fill_record_n_invoke_before_triggers(THD *thd, List<Item> &fields, +fill_record_n_invoke_before_triggers(THD *thd, TABLE *table, List<Item> &fields, List<Item> &values, bool ignore_errors, - Table_triggers_list *triggers, enum trg_event_type event) { bool result; - result= (fill_record(thd, fields, values, ignore_errors) || + Table_triggers_list *triggers= table->triggers; + result= (fill_record(thd, table, fields, values, ignore_errors) || (triggers && triggers->process_triggers(thd, event, TRG_ACTION_BEFORE, TRUE))); /* @@ -8925,7 +8924,6 @@ fill_record_n_invoke_before_triggers(THD */ if (!result && triggers) { - TABLE *table= 0; List_iterator_fast<Item> f(fields); Item *fld; Item_field *item_field; @@ -8933,9 +8931,7 @@ fill_record_n_invoke_before_triggers(THD { fld= (Item_field*)f++; item_field= fld->filed_for_view_update(); - if (item_field && item_field->field && - (table= item_field->field->table) && - table->vfield) + if (item_field && item_field->field && table && table->vfield) result= update_virtual_fields(thd, table, TRUE); Add: DBUG_ASSERT(table == item_field->field->table) Note that if we can only have one table here, than we could also replace: if (!result && triggers) with: if (!result && triggers && table) As the loop will not do anything if table == 0. <cut> bool -fill_record_n_invoke_before_triggers(THD *thd, Field **ptr, +fill_record_n_invoke_before_triggers(THD *thd, TABLE *table, Field **ptr, List<Item> &values, bool ignore_errors, - Table_triggers_list *triggers, enum trg_event_type event) { bool result; - result= (fill_record(thd, ptr, values, ignore_errors, FALSE) || + Table_triggers_list *triggers= table->triggers; + result= (fill_record(thd, table, ptr, values, ignore_errors, FALSE) || (triggers && triggers->process_triggers(thd, event, TRG_ACTION_BEFORE, TRUE))); /* @@ -9071,7 +9064,6 @@ fill_record_n_invoke_before_triggers(THD */ if (!result && triggers && *ptr) { - TABLE *table= (*ptr)->table; Add DBUG_ASSERT(table == (*ptr)->table); if (table->vfield) result= update_virtual_fields(thd, table, TRUE); } <cut> === modified file 'sql/sql_insert.cc' @@ -1696,13 +1660,32 @@ int write_record(THD *thd, TABLE *table, restore_record(table,record[1]); DBUG_ASSERT(info->update_fields->elements == info->update_values->elements); - if (fill_record_n_invoke_before_triggers(thd, *info->update_fields, + if (fill_record_n_invoke_before_triggers(thd, table, *info->update_fields, *info->update_values, info->ignore, - table->triggers, TRG_EVENT_UPDATE)) goto before_trg_err; + bool different_records= (!records_are_comparable(table) || + compare_record(table)); + /* + Default fields must be updated before checking view updateability. + This branch of INSERT is executed only when a UNIQUE key was violated + with the ON DUPLICATE KEY UPDATE option. In this case the INSERT + operation is transformed to an UPDATE, and the default fields must + be updated as if this is an UPDATE. + */ + if (different_records && table->default_field) + { + bool res; + enum_sql_command cmd= thd->lex->sql_command; + thd->lex->sql_command= SQLCOM_UPDATE; + res= table->update_default_fields(); + thd->lex->sql_command= cmd; + if (res) + goto err; + } Another way to do the above nicer would be to give thd->lex->sql_command as an argument to update_default_fields(). Another option would be to have: update_defaults_fields(table) { return update_defaults_fields(table, table->in_use->lex->cmd); } And use the later in this case. However this is not that critical to fix... <cut> + Field **field,**org_field, *found_next_number_field, **dfield_ptr; You need to do dfield_ptr= 0 to avoid a compiler warning. @@ -2390,6 +2370,12 @@ TABLE *Delayed_insert::get_local_table(T bitmap= (uchar*) (field + share->fields + 1); copy->record[0]= (bitmap + share->column_bitmap_size*3); memcpy((char*) copy->record[0], (char*) table->record[0], share->reclength); + if (share->default_fields) + { + copy->default_field= (Field**) client_thd->alloc((share->default_fields+1)* + sizeof(Field**)); You should check for error condition here. + dfield_ptr= copy->default_field; + } /* Make a copy of all fields. The copied fields need to point into the copied record. This is done @@ -2407,18 +2393,19 @@ TABLE *Delayed_insert::get_local_table(T (*field)->move_field_offset(adjust_ptrs); // Point at copy->record[0] if (*org_field == found_next_number_field) (*field)->table->found_next_number_field= *field; + if (share->default_fields && + ((*org_field)->has_insert_default_function() || + (*org_field)->has_update_default_function())) + { + /* Put the newly copied field into the set of default fields. */ + *dfield_ptr= *field; + (*dfield_ptr)->unireg_check= (*org_field)->unireg_check; It's strange that new_field() resets unireg_check. Don't know why this is done and we should at some point fix that it's not reset and see what happens. + dfield_ptr++; + } } *field=0; <cut> @@ -850,7 +837,7 @@ read_fixed_length(THD *thd, COPY_INFO &i ER(ER_WARN_TOO_FEW_RECORDS), thd->warning_info->current_row_for_warning()); if (!field->maybe_null() && field->type() == FIELD_TYPE_TIMESTAMP) - ((Field_timestamp*) field)->set_time(); + field->set_time(); Shouldn't the above be done now by testing for if the field has an insert_default_function? @@ -994,10 +982,11 @@ read_sep_field(THD *thd, COPY_INFO &info DBUG_RETURN(1); } field->set_null(); + field->set_explicit_default(NULL); if (!field->maybe_null()) { if (field->type() == MYSQL_TYPE_TIMESTAMP) - ((Field_timestamp*) field)->set_time(); + field->set_time(); shouldn't we use an insert_default_function here too ? else if (field != table->next_number_field) field->set_warning(MYSQL_ERROR::WARN_LEVEL_WARN, ER_WARN_NULL_TO_NOTNULL, 1); @@ -1066,7 +1056,7 @@ read_sep_field(THD *thd, COPY_INFO &info DBUG_RETURN(1); } if (!field->maybe_null() && field->type() == FIELD_TYPE_TIMESTAMP) - ((Field_timestamp*) field)->set_time(); + field->set_time(); shouldn't we use an insert_default_function here too ? /* TODO: We probably should not throw warning for each field. But how about intention to always have the same number @@ -1206,7 +1196,7 @@ read_xml_field(THD *thd, COPY_INFO &info if (!field->maybe_null()) { if (field->type() == FIELD_TYPE_TIMESTAMP) - ((Field_timestamp *) field)->set_time(); + field->set_time(); shouldn't we use an insert_default_function here too ? === modified file 'sql/sql_parse.cc' --- sql/sql_parse.cc 2012-10-06 07:30:52 +0000 +++ sql/sql_parse.cc 2012-10-18 12:57:12 +0000 @@ -269,12 +269,14 @@ void init_update_queries(void) CF_CAN_GENERATE_ROW_EVENTS; sql_command_flags[SQLCOM_CREATE_INDEX]= CF_CHANGES_DATA | CF_AUTO_COMMIT_TRANS; sql_command_flags[SQLCOM_ALTER_TABLE]= CF_CHANGES_DATA | CF_WRITE_LOGS_COMMAND | - CF_AUTO_COMMIT_TRANS | CF_REPORT_PROGRESS ; + CF_AUTO_COMMIT_TRANS | CF_REPORT_PROGRESS | + CF_INSERTS_DATA; Should we really have CF_INSERTS_DATA here? We should not update any timestamps for ALTER TABLE? === modified file 'sql/table.cc' @@ -2472,7 +2471,7 @@ int open_table_from_share(THD *thd, TABL } /* - Process virtual columns, if any. + Process virtual and default columns, if any. */ if (!share->vfields) outparam->vfield= NULL; @@ -2484,10 +2483,26 @@ int open_table_from_share(THD *thd, TABL goto err; outparam->vfield= vfield_ptr; + } + + if (!share->default_fields) + outparam->default_field= NULL; You don't need to set default_fields to NULL as outparam is zeroed in beginning of the function. (Same is true for outparam->vfield) <cut> Regards, Monty

1 0

[Maria-developers] EXISTS-to-IN can work anywhere?
by Sergei Petrunia 26 Nov '12

26 Nov '12

Hello, Currently, EXISTS-to-IN rewrite only works on WHERE's top level, or under NOT, where the rewrite doesn't change semantics wrt NULL values. It seems, it is possible to make it work in other contexts, too. We need to convert it into a "NULL-ignoring IN", a special kind of IN predicate. For normal IN: NULL IN (select that returns something) --> NULL NULL IN (select that returns nothing) --> FALSE $value IN (select that doesn't return $value) --> FALSE $value IN (select that doesn't return $value but returns NULL) --> NULL null-ignoring-IN should return FALSE in all of the above cases. In fact, the optimizer is perfectly capable of doing this. What we need is just to disable special handling for NULLs. Let's discuss this tomorrow (Tuesday). BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB 5.1.66 release prep
by Sergei Golubchik 26 Nov '12

26 Nov '12

Preliminary release notes and changelog pages have been created for 5.1.66, 5.2.13, 5.3.11. https://kb.askmonty.org/en/mariadb-5166-release-notes/ https://kb.askmonty.org/en/mariadb-5166-changelog/ https://kb.askmonty.org/en/mariadb-5213-release-notes/ https://kb.askmonty.org/en/mariadb-5213-changelog/ https://kb.askmonty.org/en/mariadb-5311-release-notes/ https://kb.askmonty.org/en/mariadb-5311-changelog/ Please let me know if there is anything in particular which should be mentioned in the release notes. Thanks, Sergei

1 0

[Maria-developers] EXISTS-to-IN is ok to push
by Sergei Petrunia 23 Nov '12

23 Nov '12

Hi Sanja, I've made some fixes to comments in lp:~maria-captains/maria/10.0-exists2in. With that, it is ok to push EXISTS-to-IN into 10.0. Sorry for the delay. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] crash in TC_LOG_MMAP::log_one_transaction in maria-10.0.0
by Rich Prohaska 19 Nov '12

19 Nov '12

Hello, I am having problems with the shinny new mariadb-10.0.0. I have a simple test case that replaces a row in innodb and tokudb and then commits. Eventually, mysqld hits this assert: mysqld: /home/prohaska/maria10-build/mariadb-10.0.0/sql/log.cc:7547: int TC_LOG_MMAP::log_one_transaction(my_xid): Assertion `p->ptr < p->end' failed. The test works great when the binlog is enabled. The test crashes when the binlog is OFF. So, maybe the problem is isolated to the TC_LOG_MMAP logic. I noticed that the TC_LOG_MMAP::unlog function is quite different than the mariadb-5.5 version. What is the best way to get this fixed? Would it help if i made the test case available? Thanks Rich Prohaska Here is the stack trace: Program received signal SIGABRT, Aborted. 0x0000003006a32885 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003006a32885 in raise () from /lib64/libc.so.6 #1 0x0000003006a34065 in abort () from /lib64/libc.so.6 #2 0x0000003006a2b9fe in __assert_fail_base () from /lib64/libc.so.6 #3 0x0000003006a2bac0 in __assert_fail () from /lib64/libc.so.6 #4 0x00000000008862bf in TC_LOG_MMAP::log_one_transaction (this=0x1541f00 <tc_log_mmap>, xid=142018) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/log.cc:7547 #5 0x0000000000885442 in TC_LOG_MMAP::log_and_order (this=0x1541f00 <tc_log_mmap>, thd=0x27f0000, xid=142018, all=true, need_prepare_ordered=false, need_commit_ordered=true) at /home/prohaska/maria10-\ build/mariadb-10.0.0/sql/log.cc:7180 #6 0x00000000007b819f in ha_commit_trans (thd=0x27f0000, all=true) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/handler.cc:1329 #7 0x0000000000710307 in trans_commit (thd=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/transaction.cc:213 #8 0x000000000060a2df in mysql_execute_command (thd=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:4084 #9 0x000000000060f1a7 in mysql_parse (thd=0x27f0000, rawbuf=0x7fff6c0050b8 "commit", length=6, parser_state=0x7fffe4112660) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:6056 #10 0x00000000006029a3 in dispatch_command (command=COM_QUERY, thd=0x27f0000, packet=0x27fa341 "commit", packet_length=6) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:1216 #11 0x0000000000601bdb in do_command (thd=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_parse.cc:945 #12 0x00000000006ff6d1 in do_handle_one_connection (thd_arg=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_connect.cc:1254 #13 0x00000000006ff179 in handle_one_connection (arg=0x27f0000) at /home/prohaska/maria10-build/mariadb-10.0.0/sql/sql_connect.cc:1168 #14 0x0000003006e077f1 in start_thread () from /lib64/libpthread.so.0 #15 0x0000003006ae592d in clone () from /lib64/libc.so.6

3 9

Re: [Maria-developers] Fwd: [Commits] Rev 3479: MDEV-3858 Change JOIN_TAB::records_read from ha_rows to double in file:///home/tsk/mprog/src/10.0-md3858/
by Sergei Petrunia 15 Nov '12

15 Nov '12

Hi Timour, Ok to push. On Thu, Nov 15, 2012 at 12:56:59PM +0200, Timour Katchaounov wrote: > Sergey, > > Could you please review this trivial task, shouldn't take more than 10 min. > > Timour > > ------------------------------------------------------------ > revno: 3479 > revision-id: timour(a)askmonty.org-20121115105450-zkiva73utg8pg49j > parent: wlad(a)montyprogram.com-20121111095852-1jsqvi3zcdml80lt > fixes bug: https://mariadb.atlassian.net/browse/MDEV-3858 > committer: timour(a)askmonty.org > branch nick: 10.0-md3858 > timestamp: Thu 2012-11-15 12:54:50 +0200 > message: > MDEV-3858 Change JOIN_TAB::records_read from ha_rows to double > > Currently JOIN_TAB::records_read is of type ha_rows. > This is an integer type, which prevents proper selectivity and rows estimates. > > > === modified file 'mysql-test/r/subselect_sj_mat.result' > --- a/mysql-test/r/subselect_sj_mat.result 2012-07-25 16:53:49 +0000 > +++ b/mysql-test/r/subselect_sj_mat.result 2012-11-15 10:54:50 +0000 > @@ -94,7 +94,7 @@ a1 a2 > explain extended > select * from t1i where a1 in (select b1 from t2i where b1 > '0'); > id select_type table type possible_keys key key_len ref rows filtered Extra > -1 PRIMARY t2i index it2i1,it2i3 it2i1 # NULL 5 40.00 Using where; Using index; LooseScan > +1 PRIMARY t2i index it2i1,it2i3 it2i1 # NULL 5 50.00 Using where; Using index; LooseScan > 1 PRIMARY t1i ref _it1_idx _it1_idx # _ref_ 1 100.00 > Warnings: > Note 1003 select `test`.`t1i`.`a1` AS `a1`,`test`.`t1i`.`a2` AS `a2` from `test`.`t1i` semi join (`test`.`t2i`) where ((`test`.`t1i`.`a1` = `test`.`t2i`.`b1`) and (`test`.`t2i`.`b1` > '0')) > @@ -117,7 +117,7 @@ a1 a2 > explain extended > select * from t1i where (a1, a2) in (select b1, b2 from t2i where b1 > '0'); > id select_type table type possible_keys key key_len ref rows filtered Extra > -1 PRIMARY t2i index it2i1,it2i2,it2i3 it2i3 # NULL 5 40.00 Using where; Using index; LooseScan > +1 PRIMARY t2i index it2i1,it2i2,it2i3 it2i3 # NULL 5 50.00 Using where; Using index; LooseScan > 1 PRIMARY t1i ref _it1_idx _it1_idx # _ref_ 1 100.00 > Warnings: > Note 1003 select `test`.`t1i`.`a1` AS `a1`,`test`.`t1i`.`a2` AS `a2` from `test`.`t1i` semi join (`test`.`t2i`) where ((`test`.`t1i`.`a2` = `test`.`t2i`.`b2`) and (`test`.`t1i`.`a1` = `test`.`t2i`.`b1`) and (`test`.`t2i`.`b1` > '0')) > @@ -319,7 +319,7 @@ where (a1, a2) in (select b1, b2 from t2 > (a1, a2) in (select c1, c2 from t3i > where (c1, c2) in (select b1, b2 from t2i where b2 > '0')); > id select_type table type possible_keys key key_len ref rows filtered Extra > -1 PRIMARY t2i index it2i1,it2i2,it2i3 # # # 5 40.00 # > +1 PRIMARY t2i index it2i1,it2i2,it2i3 # # # 5 50.00 # > 1 PRIMARY t1i ref it1i1,it1i2,it1i3 # # # 1 100.00 # > 1 PRIMARY t3i ref it3i1,it3i2,it3i3 # # # 1 100.00 # > 1 PRIMARY t2i ref it2i1,it2i2,it2i3 # # # 2 100.00 # > @@ -407,7 +407,7 @@ id select_type table type possible_keys > 2 MATERIALIZED t2 ALL NULL # # # 5 100.00 # > 4 MATERIALIZED t3 ALL NULL # # # 4 100.00 # > 3 MATERIALIZED t3 ALL NULL # # # 4 100.00 # > -7 UNION t2i index it2i1,it2i2,it2i3 # # # 5 40.00 # > +7 UNION t2i index it2i1,it2i2,it2i3 # # # 5 50.00 # > 7 UNION t1i ref it1i1,it1i2,it1i3 # # # 1 100.00 # > 7 UNION t3i ref it3i1,it3i2,it3i3 # # # 1 100.00 # > 7 UNION t2i ref it2i1,it2i2,it2i3 # # # 2 100.00 # > > === modified file 'sql/item_subselect.cc' > --- a/sql/item_subselect.cc 2012-11-03 11:28:51 +0000 > +++ b/sql/item_subselect.cc 2012-11-15 10:54:50 +0000 > @@ -4553,7 +4553,7 @@ double get_fanout_with_deps(JOIN *join, > !tab->emb_sj_nest && > tab->records_read != 0) > { > - fanout *= rows2double(tab->records_read); > + fanout *= tab->records_read; > } > } > return fanout; > > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2012-11-04 15:32:32 +0000 > +++ b/sql/sql_select.cc 2012-11-15 10:54:50 +0000 > @@ -6789,7 +6789,7 @@ void JOIN::get_prefix_cost_and_fanout(ui > > double JOIN::get_examined_rows() > { > - ha_rows examined_rows; > + double examined_rows; > double prev_fanout= 1; > JOIN_TAB *tab= first_breadth_first_tab(this, WALK_OPTIMIZATION_TABS); > JOIN_TAB *prev_tab= tab; > @@ -6799,7 +6799,7 @@ double JOIN::get_examined_rows() > while ((tab= next_breadth_first_tab(this, WALK_OPTIMIZATION_TABS, tab))) > { > prev_fanout *= prev_tab->records_read; > - examined_rows+= (ha_rows) (tab->get_examined_rows() * prev_fanout); > + examined_rows+= tab->get_examined_rows() * prev_fanout; > prev_tab= tab; > } > return examined_rows; > @@ -7704,7 +7704,8 @@ get_best_combination(JOIN *join) > sub-order > */ > SJ_MATERIALIZATION_INFO *sjm= cur_pos->table->emb_sj_nest->sj_mat_info; > - j->records= j->records_read= (ha_rows)(sjm->is_sj_scan? sjm->rows : 1); > + j->records_read= sjm->is_sj_scan? sjm->rows : 1; > + j->records= (ha_rows) j->records_read; > JOIN_TAB *jt; > JOIN_TAB_RANGE *jt_range; > if (!(jt= (JOIN_TAB*)join->thd->alloc(sizeof(JOIN_TAB)*sjm->tables)) || > @@ -7766,7 +7767,7 @@ get_best_combination(JOIN *join) > Save records_read in JOIN_TAB so that select_describe()/etc don't have > to access join->best_positions[]. > */ > - j->records_read= (ha_rows)join->best_positions[tablenr].records_read; > + j->records_read= join->best_positions[tablenr].records_read; > join->map2table[j->table->tablenr]= j; > > /* If we've reached the end of sjm nest, switch back to main sequence */ > @@ -10460,7 +10461,7 @@ double JOIN_TAB::scan_time() > > ha_rows JOIN_TAB::get_examined_rows() > { > - ha_rows examined_rows; > + double examined_rows; > > if (select && select->quick && use_quick != 2) > examined_rows= select->quick->records; > @@ -10490,7 +10491,7 @@ ha_rows JOIN_TAB::get_examined_rows() > } > } > else > - examined_rows= (ha_rows) records_read; > + examined_rows= records_read; > > return examined_rows; > } > @@ -21972,7 +21973,7 @@ int JOIN::print_explain(select_result_si > } > else > { > - ha_rows examined_rows= tab->get_examined_rows(); > + double examined_rows= tab->get_examined_rows(); > > item_list.push_back(new Item_int((longlong) (ulonglong) examined_rows, > MY_INT64_NUM_DECIMAL_DIGITS)); > @@ -21982,7 +21983,7 @@ int JOIN::print_explain(select_result_si > { > float f= 0.0; > if (examined_rows) > - f= (float) (100.0 * tab->records_read / examined_rows); > + f= (100.0 * (float)tab->records_read) / examined_rows; > set_if_smaller(f, 100.0); > item_list.push_back(new Item_float(f, 2)); > } > > === modified file 'sql/sql_select.h' > --- a/sql/sql_select.h 2012-11-04 15:32:32 +0000 > +++ b/sql/sql_select.h 2012-11-15 10:54:50 +0000 > @@ -280,8 +280,8 @@ typedef struct st_join_table { > */ > double read_time; > > - /* psergey-todo: make the below have type double, like POSITION::records_read? */ > - ha_rows records_read; > + /* Copy of POSITION::records_read, set by get_best_combination() */ > + double records_read; > > /* Startup cost for execution */ > double startup_cost; > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] The patch for the task mdev-539
by Sergei Golubchik 13 Nov '12

13 Nov '12

Hi, Igor! Sorry for a delay :( Please, find my review attached. On Sep 27, Igor Babaev wrote: > === modified file 'mysql-test/r/alter_table.result' > --- a/mysql-test/r/alter_table.result 2012-05-21 13:30:25 +0000 > +++ b/mysql-test/r/alter_table.result 2012-09-27 21:29:26 +0000 > @@ -1340,3 +1340,203 @@ > execute stmt1; > deallocate prepare stmt1; > drop table t2; > +# > +# mdev-539: fast build of unique/primary indexes for MyISAM > +# I'd suggest to have these test cases in a separate file, say, alter_table_539.test But it's just a siggestion, do as you prefer. > +DROP DATABASE IF EXISTS dbt3_s001; > +CREATE DATABASE dbt3_s001; > +use dbt3_s001; > +drop index `primary` on customer; > +show create table customer; > +Table Create Table > +customer CREATE TABLE `customer` ( > + `c_custkey` int(11) NOT NULL, > + `c_name` varchar(25) DEFAULT NULL, > + `c_address` varchar(40) DEFAULT NULL, > + `c_nationkey` int(11) DEFAULT NULL, > + `c_phone` char(15) DEFAULT NULL, > + `c_acctbal` double DEFAULT NULL, > + `c_mktsegment` char(10) DEFAULT NULL, > + `c_comment` varchar(117) DEFAULT NULL, > + KEY `i_c_nationkey` (`c_nationkey`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +alter table customer add primary key (c_custkey); > +show create table customer; > +Table Create Table > +customer CREATE TABLE `customer` ( > + `c_custkey` int(11) NOT NULL, > + `c_name` varchar(25) DEFAULT NULL, > + `c_address` varchar(40) DEFAULT NULL, > + `c_nationkey` int(11) DEFAULT NULL, > + `c_phone` char(15) DEFAULT NULL, > + `c_acctbal` double DEFAULT NULL, > + `c_mktsegment` char(10) DEFAULT NULL, > + `c_comment` varchar(117) DEFAULT NULL, > + PRIMARY KEY (`c_custkey`), > + KEY `i_c_nationkey` (`c_nationkey`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +drop index `primary` on customer; > +insert into customer values > +(3,'Customer#00000000303','MG9kdTD2WBHm',1,'11-719-748-3364',7498.12,'AUTOMOBILE','special packages wake. slyly reg'); > +alter table customer add primary key (c_custkey); please add a test case for alter ignore table > +ERROR 23000: Duplicate entry '3' for key 'PRIMARY' > +show create table customer; > +Table Create Table > +customer CREATE TABLE `customer` ( ... > === modified file 'sql/ha_partition.cc' > --- a/sql/ha_partition.cc 2012-08-31 21:54:54 +0000 > +++ b/sql/ha_partition.cc 2012-09-27 21:29:26 +0000 > @@ -6862,6 +6862,8 @@ > > flags_to_return= ht->alter_table_flags(flags); > flags_to_return|= m_file[0]->alter_table_flags(flags); > + flags_to_return&= > + ~(HA_FAST_ADD_INDEX | HA_FAST_ADD_UNIQUE_INDEX | HA_FAST_ADD_PK_INDEX); You don't need these flags - see below > > /* > If one partition fails we must be able to revert the change for the other, > > === modified file 'sql/sql_table.cc' > --- a/sql/sql_table.cc 2012-09-02 02:41:38 +0000 > +++ b/sql/sql_table.cc 2012-09-27 21:29:26 +0000 > @@ -6743,6 +6745,11 @@ > my_error(ER_LOCK_WAIT_TIMEOUT, MYF(0)); > goto err_new_table_cleanup; > }); > + > + if (!ignore && > + test(alter_flags & (HA_FAST_ADD_UNIQUE_INDEX | HA_FAST_ADD_PK_INDEX))) > + new_table->file->extra(HA_EXTRA_CREATE_UNIQUE_INDEX_BY_SORT); What's the point? extra() functions were always *optional* in the storage engine API, any storage engine is allowed to ignore any HA_EXTRA_* requests, or all of them. It means, you don't need to introduce HA_FAST_whatever flags, and don't need to check them. Just write if (!ignore) new_table->file->extra(HA_EXTRA_CREATE_UNIQUE_INDEX_BY_SORT); and let the engine ignore it, if it wishes to do so. as a small bonus, you won't need to move alter_flags variable around, it can stay where it was, inside the if() block. > + > error= copy_data_between_tables(thd, table, new_table, > alter_info->create_list, ignore, > order_num, order, &copied, &deleted, > > === modified file 'storage/maria/ha_maria.cc' > --- a/storage/maria/ha_maria.cc 2012-08-14 16:59:28 +0000 > +++ b/storage/maria/ha_maria.cc 2012-09-27 21:29:26 +0000 > @@ -1633,6 +1633,9 @@ please, use the recipe from https://kb.askmonty.org/en/how-to-get-more-out-of-bzr-when-working-on-maria… it's much more difficult to review patches without sufficient context (that is, without function names on every chunk) > param->testflag|= T_REP_BY_SORT; > error= maria_repair_by_sort(param, file, fixed_name, > test(param->testflag & T_QUICK)); > + if (error && share->state.dupp_key != MARIA_MAX_KEY) > + print_keydup_error(share->state.dupp_key, > + ER(ER_DUP_ENTRY_WITH_KEY_NAME)); > } > } > else > > === modified file 'storage/myisam/ha_myisam.cc' > --- a/storage/myisam/ha_myisam.cc 2012-07-18 18:40:15 +0000 > +++ b/storage/myisam/ha_myisam.cc 2012-09-27 21:29:26 +0000 > @@ -1130,6 +1130,9 @@ > thd_proc_info(thd, "Repair by sorting"); > error = mi_repair_by_sort(&param, file, fixed_name, > test(param.testflag & T_QUICK)); > + if (error && share->state.dupp_key != MAX_KEY) > + print_keydup_error(share->state.dupp_key, > + ER(ER_DUP_ENTRY_WITH_KEY_NAME)); I don't understand it. print_keydup_error() takes the value to print from table->record[[0]. But - in the case of mi_repair_by_sort - how does the value end up here? What code puts the conflicting key value into table->record[0] ? > } > } > else > > === modified file 'storage/myisam/mi_check.c' > --- a/storage/myisam/mi_check.c 2012-04-07 13:58:46 +0000 > +++ b/storage/myisam/mi_check.c 2012-09-27 21:29:26 +0000 > @@ -2408,12 +2411,22 @@ > (my_bool) (!(param->testflag & T_VERBOSE)), > param->sort_buffer_length)) > { > + > + if (sort_param.sort_info->dupp) > + { > + share->state.dupp_key= sort_param.key; > + keydup_error= 1; > + } > + if (!(keydup_error && param->suppress_keydup_error_handling)) 1. please use param->testflag for flags, instead of a new special field 2. I don't see why are you doing it here, normally the error is issued below, as "Duplicate key for record at %10s against...." I'd expect to see any new code that deals with duplicate key errors at the same place where the old code is > + { > param->retry_repair= 1; > if (! param->error_printed) > mi_check_print_error(param, "Couldn't fix table with create_index_by_sort(). Error: %d", > my_errno); > + } > goto err; > } > + > /* No need to calculate checksum again. */ > sort_param.calc_checksum= 0; > free_root(&sort_param.wordroot, MYF(0)); > @@ -2533,18 +2546,27 @@ > } > if (got_error) > { > + if (!(keydup_error && param->suppress_keydup_error_handling)) > + { > if (! param->error_printed) > mi_check_print_error(param,"%d when fixing table",my_errno); You could simply set param->error_printed when you get dplicate key error - it'll avoid all further errors, you won't need to wrap them all in if()'s > + } > if (new_file >= 0) > { > (void) mysql_file_close(new_file, MYF(0)); > (void) mysql_file_delete(mi_key_file_datatmp, > param->temp_filename, MYF(MY_WME)); > + if (!(keydup_error && param->suppress_keydup_error_handling)) > + { Why do you need this if() ? > if (info->dfile == new_file) /* Retry with key cache */ > if (unlikely(mi_open_datafile(info, share, name, -1))) > param->retry_repair= 0; /* Safety */ > + } > } > + if (!(keydup_error && param->suppress_keydup_error_handling)) > + { > mi_mark_crashed_on_repair(info); > + } Why do you need to avoid mi_mark_crashed_on_repair() ? > if (killed_ptr(param)) > param->retry_repair= 0; /* No use to retry repair */ > } > @@ -4682,8 +4708,9 @@ > (!rows || rows >= MI_MIN_ROWS_TO_DISABLE_INDEXES)); > for (i=0 ; i < share->base.keys ; i++,key++) > { > - if (!(key->flag & (HA_NOSAME | HA_SPATIAL | HA_AUTO_KEY)) && > - ! mi_too_big_key_for_sort(key,rows) && info->s->base.auto_key != i+1) > + if (!(key->flag & (HA_SPATIAL | HA_AUTO_KEY)) && > + ! mi_too_big_key_for_sort(key,rows) && info->s->base.auto_key != i+1 && > + (!(key->flag & HA_NOSAME) || info->create_unique_index_by_sort)) Now, this directly contradicts the function name and the function comment. Could you please rename the function and update the comment? And instead of checking info->create_unique_index_by_sort I'd rather pass it down as an argument: void mi_disable_indexes(MI_INFO *info, ha_rows rows, my_bool only_non_unique) { > { > mi_clear_key_active(share->state.key_map, i); > info->update|= HA_STATE_CHANGED; > > === modified file 'storage/myisam/mi_extra.c' > --- a/storage/myisam/mi_extra.c 2012-07-31 17:29:07 +0000 > +++ b/storage/myisam/mi_extra.c 2012-09-27 21:29:26 +0000 > @@ -365,6 +365,9 @@ > case HA_EXTRA_CHANGE_KEY_TO_DUP: > mi_extra_keyflag(info, function); > break; > + case HA_EXTRA_CREATE_UNIQUE_INDEX_BY_SORT: > + info->create_unique_index_by_sort= 1; > + break; Two comments here: 1. I don't like the name "create unique index by sort", because ALTER TABLE itself does not create any indexes by sort or anything it simply tells the engine to optimize a bulk insertion. and it's myisam that does repair-by-sort. I mean, a better name would be a high-level name that does not imply myisam-specific details. something like HA_EXTRA_BULK_INSERT_DUPLICATES_UNLIKELY which tells the engine to optimize the following bulk insert for the case when unique key conflicts are unlikely to happen. (in myisam case it means to build unique indexes by sort, right) 2. I've discussed this with Monty and we both didn't like using extra() here, even to preserve the API compatibility. Monty thinks there's no need to bother, and we can change start_bulk_insert() any way we want. But to make it better for third-party engines, I'd suggest something like class handler { ... virtual void start_bulk_insert(ha_rows rows) {} virtual void start_bulk_insert(ha_rows rows, ulonglong flags) { start_bulk_insert(rows); } ... } that is, you introduce the new function, and keep the old one as a fallback. This way it stays compatible, and old engines won't need any changes. But myisam/maria will implement the new start_bulk_insert variant, with two arguments, and will recognize the flag HA_BULK_INSERT_DUPLICATES_UNLIKELY And no extra() is needed. > case HA_EXTRA_MMAP: > #ifdef HAVE_MMAP > mysql_mutex_lock(&share->intern_lock); > Regards, Sergei

1 0

[Maria-developers] MariaDB 5.3.10 release prep
by Daniel Bartholomew 12 Nov '12

12 Nov '12

Preliminary release notes and changelog pages have been created for 5.3.10. https://kb.askmonty.org/en/mariadb-5310-release-notes/ https://kb.askmonty.org/en/mariadb-5310-changelog/ Please let me know if there is anything in particular which should be mentioned in the release notes. Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

[Maria-developers] MariaDB 10.0.0 Alpha Release Prep
by Daniel Bartholomew 12 Nov '12

12 Nov '12

All, The files for this release have been uploaded to the mirrors and we're just waiting until Monday morning, U.S. Eastern time, to announce the release. The preliminary release notes and changelog pages are here: - https://kb.askmonty.org/en/mariadb-1000-release-notes/ - https://kb.askmonty.org/en/mariadb-1000-changelog/ The release notes page is pretty sparse. Its content is from the list of "New already done Features" on the "What is MariaDB 10.0?" page. Please send me additions and corrections ASAP (or make the changes yourself). Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

Re: [Maria-developers] [Commits] Rev 3569: MDEV-3830 - fix build on Intel compiler in file:///H:/bzr/5.5/
by Kristian Nielsen 03 Nov '12

03 Nov '12

Vladislav Vaintroub <wlad(a)montyprogram.com> writes: > === modified file 'mysys/my_context.c' > --- a/mysys/my_context.c 2012-10-31 11:47:25 +0000 > +++ b/mysys/my_context.c 2012-11-02 23:31:50 +0000 > @@ -206,7 +206,7 @@ > ( > "movq %%rsp, (%[save])\n\t" > "movq %[stack], %%rsp\n\t" > -#if __GNUC__ >= 4 && __GNUC_MINOR__ >= 4 > +#if __GNUC__ >= 4 && __GNUC_MINOR__ >= 4 && !defined(__INTEL_COMPILER) There are two places this is done (one for 32-bit and one for 64-bit). You probably need the !defined(__INTEL_COMPILER) on both. - Kristian.

1 0

Re: [Maria-developers] [Commits] Rev 3589: Fixed bug mdev-585 (LP bug #637962) in file:///home/igor/maria/maria-5.3-mdev585/
by Sergei Petrunia 02 Nov '12

02 Nov '12

Hello Igor, The fix seems to be ok to push. Which version do you want to push it into? This is not a crash, should we really push it into 5.3, or to 5.5? On Thu, Nov 01, 2012 at 02:54:34PM -0700, Igor Babaev wrote: > At file:///home/igor/maria/maria-5.3-mdev585/ > > ------------------------------------------------------------ > revno: 3589 > revision-id: igor(a)askmonty.org-20121101215433-ctfm3zfhc6uhuy3i > parent: psergey(a)askmonty.org-20121031053425-kfn0l7yj6z7jtnqw > committer: Igor Babaev <igor(a)askmonty.org> > branch nick: maria-5.3-mdev585 > timestamp: Thu 2012-11-01 14:54:33 -0700 > message: > Fixed bug mdev-585 (LP bug #637962) > If, when executing a query with ORDER BY col LIMIT n, the optimizer chose > an index-merge scan to access the table containing col while there existed > an index defined over col then optimizer did not consider the possibility > of using an alternative range scan by this index to avoid filesort. This > could cause a performance degradation if the optimizer flag index_merge was > set up to 'on'. > === modified file 'mysql-test/r/range_vs_index_merge.result' > --- a/mysql-test/r/range_vs_index_merge.result 2012-03-21 00:03:28 +0000 > +++ b/mysql-test/r/range_vs_index_merge.result 2012-11-01 21:54:33 +0000 > @@ -1221,6 +1221,150 @@ > Seattle USA 563374 > Caracas VEN 1975294 > set optimizer_switch=@save_optimizer_switch; > +# > +# Bug mdev-585: range vs index-merge with ORDER BY ... LIMIT n > +# (LP bug #637962) > +# > +DROP INDEX CountryPopulation ON City; > +DROP INDEX CountryName ON City; > +DROP INDEX CityName on City; > +CREATE INDEX Name ON City(Name); > +CREATE INDEX Population ON City(Population); > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City index_merge Country,Name,Population Name,Country 35,3 NULL # Using sort_union(Name,Country); Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000); > +ID Name Country Population > +384 Cabo Frio BRA 119503 > +387 Camaragibe BRA 118968 > +403 Catanduva BRA 107761 > +412 Cachoeirinha BRA 103240 > +508 Watford GBR 113080 > +509 Ipswich GBR 114000 > +510 Slough GBR 112000 > +511 Exeter GBR 111000 > +512 Cheltenham GBR 106000 > +513 Gloucester GBR 107000 > +514 Saint Helens GBR 106293 > +515 Sutton Coldfield GBR 106001 > +516 York GBR 104425 > +517 Oldham GBR 103931 > +518 Basildon GBR 100924 > +519 Worthing GBR 100000 > +635 Mallawi EGY 119283 > +636 Bilbays EGY 113608 > +637 Mit Ghamr EGY 101801 > +638 al-Arish EGY 100447 > +701 Tarragona ESP 113016 > +702 Lleida (Lérida) ESP 112207 > +703 Jaén ESP 109247 > +704 Ourense (Orense) ESP 109120 > +705 Mataró ESP 104095 > +706 Algeciras ESP 103106 > +707 Marbella ESP 101144 > +759 Gonder ETH 112249 > +869 Cabuyao PHL 106630 > +870 Calapan PHL 105910 > +873 Cauayan PHL 103952 > +903 Serekunda GMB 102600 > +909 Sohumi GEO 111700 > +913 Tema GHA 109975 > +914 Sekondi-Takoradi GHA 103653 > +924 Villa Nueva GTM 101295 > +1844 Cape Breton CAN 114733 > +1847 Cambridge CAN 109186 > +2406 Herakleion GRC 116178 > +2407 Kallithea GRC 114233 > +2408 Larisa GRC 113090 > +2908 Cajamarca PER 108009 > +3002 Besançon FRA 117733 > +3003 Caen FRA 113987 > +3004 Orléans FRA 113126 > +3005 Mulhouse FRA 110359 > +3006 Rouen FRA 106592 > +3007 Boulogne-Billancourt FRA 106367 > +3008 Perpignan FRA 105115 > +3009 Nancy FRA 103605 > +3411 Ceyhan TUR 102412 > +3567 Carúpano VEN 119639 > +3568 Catia La Mar VEN 117012 > +3571 Calabozo VEN 107146 > +3786 Cam Ranh VNM 114041 > +3792 Tartu EST 101246 > +4002 Carrollton USA 109576 > +4027 Cape Coral USA 102286 > +4032 Cambridge USA 101355 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 2 > +Handler_read_next 385 > +Handler_read_prev 0 > +Handler_read_rnd 377 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City range Country,Name,Population Population 4 NULL # Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +ID Name Country Population > +519 Worthing GBR 100000 > +638 al-Arish EGY 100447 > +518 Basildon GBR 100924 > +707 Marbella ESP 101144 > +3792 Tartu EST 101246 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 1 > +Handler_read_next 59 > +Handler_read_prev 0 > +Handler_read_rnd 0 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +set optimizer_switch='index_merge=off'; > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City range Country,Name,Population Population 4 NULL # Using index condition; Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +ID Name Country Population > +519 Worthing GBR 100000 > +638 al-Arish EGY 100447 > +518 Basildon GBR 100924 > +707 Marbella ESP 101144 > +3792 Tartu EST 101246 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 1 > +Handler_read_next 59 > +Handler_read_prev 0 > +Handler_read_rnd 0 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +set optimizer_switch=@save_optimizer_switch; > DROP DATABASE world; > use test; > CREATE TABLE t1 ( > > === modified file 'mysql-test/r/range_vs_index_merge_innodb.result' > --- a/mysql-test/r/range_vs_index_merge_innodb.result 2012-03-21 00:03:28 +0000 > +++ b/mysql-test/r/range_vs_index_merge_innodb.result 2012-11-01 21:54:33 +0000 > @@ -1222,6 +1222,150 @@ > Seattle USA 563374 > Caracas VEN 1975294 > set optimizer_switch=@save_optimizer_switch; > +# > +# Bug mdev-585: range vs index-merge with ORDER BY ... LIMIT n > +# (LP bug #637962) > +# > +DROP INDEX CountryPopulation ON City; > +DROP INDEX CountryName ON City; > +DROP INDEX CityName on City; > +CREATE INDEX Name ON City(Name); > +CREATE INDEX Population ON City(Population); > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City index_merge Country,Name,Population Name,Country 35,3 NULL # Using sort_union(Name,Country); Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000); > +ID Name Country Population > +384 Cabo Frio BRA 119503 > +387 Camaragibe BRA 118968 > +403 Catanduva BRA 107761 > +412 Cachoeirinha BRA 103240 > +508 Watford GBR 113080 > +509 Ipswich GBR 114000 > +510 Slough GBR 112000 > +511 Exeter GBR 111000 > +512 Cheltenham GBR 106000 > +513 Gloucester GBR 107000 > +514 Saint Helens GBR 106293 > +515 Sutton Coldfield GBR 106001 > +516 York GBR 104425 > +517 Oldham GBR 103931 > +518 Basildon GBR 100924 > +519 Worthing GBR 100000 > +635 Mallawi EGY 119283 > +636 Bilbays EGY 113608 > +637 Mit Ghamr EGY 101801 > +638 al-Arish EGY 100447 > +701 Tarragona ESP 113016 > +702 Lleida (Lérida) ESP 112207 > +703 Jaén ESP 109247 > +704 Ourense (Orense) ESP 109120 > +705 Mataró ESP 104095 > +706 Algeciras ESP 103106 > +707 Marbella ESP 101144 > +759 Gonder ETH 112249 > +869 Cabuyao PHL 106630 > +870 Calapan PHL 105910 > +873 Cauayan PHL 103952 > +903 Serekunda GMB 102600 > +909 Sohumi GEO 111700 > +913 Tema GHA 109975 > +914 Sekondi-Takoradi GHA 103653 > +924 Villa Nueva GTM 101295 > +1844 Cape Breton CAN 114733 > +1847 Cambridge CAN 109186 > +2406 Herakleion GRC 116178 > +2407 Kallithea GRC 114233 > +2408 Larisa GRC 113090 > +2908 Cajamarca PER 108009 > +3002 Besançon FRA 117733 > +3003 Caen FRA 113987 > +3004 Orléans FRA 113126 > +3005 Mulhouse FRA 110359 > +3006 Rouen FRA 106592 > +3007 Boulogne-Billancourt FRA 106367 > +3008 Perpignan FRA 105115 > +3009 Nancy FRA 103605 > +3411 Ceyhan TUR 102412 > +3567 Carúpano VEN 119639 > +3568 Catia La Mar VEN 117012 > +3571 Calabozo VEN 107146 > +3786 Cam Ranh VNM 114041 > +3792 Tartu EST 101246 > +4002 Carrollton USA 109576 > +4027 Cape Coral USA 102286 > +4032 Cambridge USA 101355 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 2 > +Handler_read_next 385 > +Handler_read_prev 0 > +Handler_read_rnd 377 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City range Country,Name,Population Population 4 NULL # Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +ID Name Country Population > +519 Worthing GBR 100000 > +638 al-Arish EGY 100447 > +518 Basildon GBR 100924 > +707 Marbella ESP 101144 > +3792 Tartu EST 101246 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 1 > +Handler_read_next 59 > +Handler_read_prev 0 > +Handler_read_rnd 0 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +set optimizer_switch='index_merge=off'; > +EXPLAIN > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE City range Country,Name,Population Population 4 NULL # Using index condition; Using where > +FLUSH STATUS; > +SELECT * FROM City > +WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > +AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +ID Name Country Population > +519 Worthing GBR 100000 > +638 al-Arish EGY 100447 > +518 Basildon GBR 100924 > +707 Marbella ESP 101144 > +3792 Tartu EST 101246 > +SHOW STATUS LIKE 'Handler_read_%'; > +Variable_name Value > +Handler_read_first 0 > +Handler_read_key 1 > +Handler_read_next 59 > +Handler_read_prev 0 > +Handler_read_rnd 0 > +Handler_read_rnd_deleted 0 > +Handler_read_rnd_next 0 > +set optimizer_switch=@save_optimizer_switch; > DROP DATABASE world; > use test; > CREATE TABLE t1 ( > > === modified file 'mysql-test/t/range_vs_index_merge.test' > --- a/mysql-test/t/range_vs_index_merge.test 2012-03-21 00:03:28 +0000 > +++ b/mysql-test/t/range_vs_index_merge.test 2012-11-01 21:54:33 +0000 > @@ -675,6 +675,64 @@ > $cond; > > set optimizer_switch=@save_optimizer_switch; > + > +--echo # > +--echo # Bug mdev-585: range vs index-merge with ORDER BY ... LIMIT n > +--echo # (LP bug #637962) > +--echo # > + > +DROP INDEX CountryPopulation ON City; > +DROP INDEX CountryName ON City; > +DROP INDEX CityName on City; > + > +CREATE INDEX Name ON City(Name); > +CREATE INDEX Population ON City(Population); > + > + > +--replace_column 9 # > +EXPLAIN > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000); > +FLUSH STATUS; > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000); > +SHOW STATUS LIKE 'Handler_read_%'; > + > + > +--replace_column 9 # > +EXPLAIN > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > + > +FLUSH STATUS; > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +SHOW STATUS LIKE 'Handler_read_%'; > + > + > +set optimizer_switch='index_merge=off'; > + > +--replace_column 9 # > +EXPLAIN > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > + > +FLUSH STATUS; > +SELECT * FROM City > + WHERE ((Name > 'Ca' AND Name < 'Cf') OR (Country > 'E' AND Country < 'H')) > + AND (Population >= 100000 AND Population < 120000) > +ORDER BY Population LIMIT 5; > +SHOW STATUS LIKE 'Handler_read_%'; > + > +set optimizer_switch=@save_optimizer_switch; > > > DROP DATABASE world; > > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2012-10-10 05:21:22 +0000 > +++ b/sql/sql_select.cc 2012-11-01 21:54:33 +0000 > @@ -18079,15 +18079,18 @@ > */ > > if (quick_type == QUICK_SELECT_I::QS_TYPE_INDEX_MERGE || > - quick_type == QUICK_SELECT_I::QS_TYPE_INDEX_INTERSECT || > + quick_type == QUICK_SELECT_I::QS_TYPE_INDEX_INTERSECT || > quick_type == QUICK_SELECT_I::QS_TYPE_ROR_UNION || > quick_type == QUICK_SELECT_I::QS_TYPE_ROR_INTERSECT) > - goto use_filesort; > - ref_key= select->quick->index; > - ref_key_parts= select->quick->used_key_parts; > + ref_key= MAX_KEY; > + else > + { > + ref_key= select->quick->index; > + ref_key_parts= select->quick->used_key_parts; > + } > } > > - if (ref_key >= 0) > + if (ref_key >= 0 && ref_key != MAX_KEY) > { > /* > We come here when there is a REF key. > @@ -18229,7 +18232,8 @@ > else > keys= usable_keys; > > - if (ref_key >= 0 && table->covering_keys.is_set(ref_key)) > + if (ref_key >= 0 && ref_key != MAX_KEY && > + table->covering_keys.is_set(ref_key)) > ref_key_quick_rows= table->quick_rows[ref_key]; > > read_time= join->best_positions[tablenr].read_time; > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] Rev 3176: MDEV-616 fix (MySQL fix accepted)
by Sergei Golubchik 31 Oct '12

31 Oct '12

Hi, Sanja! On Oct 09, sanja(a)montyprogram.com wrote: > revno: 3176 > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.2-MDEV-616 > timestamp: Tue 2012-10-09 17:36:02 +0300 > message: > MDEV-616 fix (MySQL fix accepted) Looks ok, as far as I understand. Regards, Sergei

1 0

[Maria-developers] Fwd: possible bug in MySQL 5.6.7-rc slave replication crash safety?
by Zardosht Kasheff 24 Oct '12

24 Oct '12

Hello, Does Maria 10.0 implement crash safe replication that is found in MySQL 5.6.7? If so, does this issue exist there? Thanks -Zardosht ---------- Forwarded message ---------- From: Zardosht Kasheff <zardosht(a)gmail.com> Date: Wed, Oct 17, 2012 at 8:59 AM Subject: possible bug in MySQL 5.6.7-rc slave replication crash safety? To: internals(a)lists.mysql.com Hello all, We downloaded MySQL 5.6.7-rc to see slave replication crash safety in action. A simple test did not seem to work, as described in http://bugs.mysql.com/bug.php?id=67246. Does anyone have insight into the problem? Perhaps we screwed something up? Thanks -Zardosht

2 8

Re: [Maria-developers] Please look at and advise about MDEV-3801
by Igor Babaev 22 Oct '12

22 Oct '12

On 10/22/2012 12:53 AM, Timour Katchaounov wrote: > Igor, > > Great, the patch indeed fixes the bug both on 5.3 and 5.5, however the > reduced test case I produced still takes 21 MB, and I was not successful > in reducing it to a size that can be added to the regression test suite. > I already spent too much in reducing the test case. Would you agree that > I push your patch without a test case? I think you still have to add the test case case (in a separate file) that is run with the --big option. Regards, Igor. > > Timour > > > On 20.10.2012 08:56, Igor Babaev wrote: >> On 10/19/2012 12:43 PM, Timour Katchaounov wrote: >>> Hi Igor, >>> >>> Could you please look at this bug >>> https://mariadb.atlassian.net/browse/MDEV-3801, >>> and read my last comment. The bug may be (just a guess ATM) related to >>> index >>> creation on the temporary table created for the derived table. Since >>> this is your >>> code, I'd like you to tell me if you have seen anything similar before, >>> or if >>> you have any hints. >>> >>> If not, I will continue with the bug and will trace the cause for the >>> wrong key information. >>> >>> Thanks, >>> Timour >> >> Timour, >> >> The following patch fixes this problem >> >> Regards, >> Igor. >> >> === modified file 'sql/sql_select.cc' >> --- sql/sql_select.cc 2012-10-10 05:21:22 +0000 >> +++ sql/sql_select.cc 2012-10-20 05:49:13 +0000 >> @@ -8981,7 +8981,7 @@ void JOIN::drop_unused_derived_keys() >> JOIN_TAB *tab; >> for (tab= first_linear_tab(this, WITHOUT_CONST_TABLES); >> tab; >> - tab= next_linear_tab(this, tab, WITHOUT_BUSH_ROOTS)) >> + tab= next_linear_tab(this, tab, WITH_BUSH_ROOTS)) >> { >> >> TABLE *table=tab->table; >> >> MariaDB [test]> set @@tmp_table_size=16384; >> ERROR 2006 (HY000): MySQL server has gone away >> No connection. Trying to reconnect... >> Connection id: 1 >> Current database: test >> >> Query OK, 0 rows affected (5.48 sec) >> >> MariaDB [test]> set @@tmp_table_size=16384; >> Query OK, 0 rows affected (0.00 sec) >> >> MariaDB [test]> EXPLAIN SELECT t1.deal_id FROM table_e AS t1 JOIN >> table_d AS t10 ON (t1.deal_id = t10.deal_id) WHERE t1.deal_id IN >> (SELECT * FROM (SELECT DISTINCT t1.deal_id FROM table_e AS >> t1 JOIN table_d AS t2 ON (t1.deal_id = t2.deal_id) >> JOIN table_c_s AS t3 ON (t2.deal_id = t3.deal_id) AND (t2.feid = >> t3.feid) JOIN table_f AS t4 ON t3.exchange = t4.exchange) >> tU) limit 10; >> +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ >> >> | id | select_type | table | type | possible_keys >> | key | key_len | ref | >> rows | Extra | >> +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ >> >> | 1 | PRIMARY | <subquery2> | ALL | distinct_key >> | NULL | NULL | NULL | >> 8488 | | >> | 1 | PRIMARY | t1 | eq_ref | PRIMARY >> | PRIMARY | 4 | tU.deal_id | >> 1 | Using index | >> | 1 | PRIMARY | t10 | ref | PRIMARY,full_id >> | PRIMARY | 4 | tU.deal_id | >> 1 | Using index | >> | 2 | MATERIALIZED | <derived3> | ALL | NULL >> | NULL | NULL | NULL | >> 8488 | | >> | 3 | DERIVED | t4 | system | PRIMARY >> | NULL | NULL | NULL | >> 1 | Using temporary | >> | 3 | DERIVED | t3 | ref | >> PRIMARY,feid,exchange,deal_feid,feid_date | exchange | 3 | const >> | 8488 | | >> | 3 | DERIVED | t1 | eq_ref | PRIMARY >> | PRIMARY | 4 | test.t3.deal_id | >> 1 | Using index | >> | 3 | DERIVED | t2 | eq_ref | PRIMARY,full_id >> | PRIMARY | 8 | test.t3.deal_id,test.t3.feid | >> 1 | Using index; Distinct | >> +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ >> >> 8 rows in set (4.22 sec) >> >> MariaDB [test]> SELECT t1.deal_id FROM table_e AS t1 JOIN table_d AS t10 >> ON (t1.deal_id = t10.deal_id) WHERE t1.deal_id IN (SELECT * FROM >> (SELECT DISTINCT t1.deal_id FROM table_e AS t1 >> JOIN table_d AS t2 ON (t1.deal_id = t2.deal_id) JOIN >> table_c_s AS t3 ON (t2.deal_id = t3.deal_id) AND (t2.feid = t3.feid) >> JOIN table_f AS t4 ON t3.exchange = t4.exchange) tU) limit >> 10; >> +---------+ >> | deal_id | >> +---------+ >> | 30 | >> | 82 | >> | 83 | >> | 108 | >> | 155 | >> | 156 | >> | 162 | >> | 184 | >> | 186 | >> | 195 | >> +---------+ >> 10 rows in set (10.62 sec) >>

1 1

Re: [Maria-developers] Please look at and advise about MDEV-3801
by Igor Babaev 20 Oct '12

20 Oct '12

On 10/19/2012 12:43 PM, Timour Katchaounov wrote: > Hi Igor, > > Could you please look at this bug > https://mariadb.atlassian.net/browse/MDEV-3801, > and read my last comment. The bug may be (just a guess ATM) related to > index > creation on the temporary table created for the derived table. Since > this is your > code, I'd like you to tell me if you have seen anything similar before, > or if > you have any hints. > > If not, I will continue with the bug and will trace the cause for the > wrong key information. > > Thanks, > Timour Timour, The following patch fixes this problem Regards, Igor. === modified file 'sql/sql_select.cc' --- sql/sql_select.cc 2012-10-10 05:21:22 +0000 +++ sql/sql_select.cc 2012-10-20 05:49:13 +0000 @@ -8981,7 +8981,7 @@ void JOIN::drop_unused_derived_keys() JOIN_TAB *tab; for (tab= first_linear_tab(this, WITHOUT_CONST_TABLES); tab; - tab= next_linear_tab(this, tab, WITHOUT_BUSH_ROOTS)) + tab= next_linear_tab(this, tab, WITH_BUSH_ROOTS)) { TABLE *table=tab->table; MariaDB [test]> set @@tmp_table_size=16384; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... Connection id: 1 Current database: test Query OK, 0 rows affected (5.48 sec) MariaDB [test]> set @@tmp_table_size=16384; Query OK, 0 rows affected (0.00 sec) MariaDB [test]> EXPLAIN SELECT t1.deal_id FROM table_e AS t1 JOIN table_d AS t10 ON (t1.deal_id = t10.deal_id) WHERE t1.deal_id IN (SELECT * FROM (SELECT DISTINCT t1.deal_id FROM table_e AS t1 JOIN table_d AS t2 ON (t1.deal_id = t2.deal_id) JOIN table_c_s AS t3 ON (t2.deal_id = t3.deal_id) AND (t2.feid = t3.feid) JOIN table_f AS t4 ON t3.exchange = t4.exchange) tU) limit 10; +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ | 1 | PRIMARY | <subquery2> | ALL | distinct_key | NULL | NULL | NULL | 8488 | | | 1 | PRIMARY | t1 | eq_ref | PRIMARY | PRIMARY | 4 | tU.deal_id | 1 | Using index | | 1 | PRIMARY | t10 | ref | PRIMARY,full_id | PRIMARY | 4 | tU.deal_id | 1 | Using index | | 2 | MATERIALIZED | <derived3> | ALL | NULL | NULL | NULL | NULL | 8488 | | | 3 | DERIVED | t4 | system | PRIMARY | NULL | NULL | NULL | 1 | Using temporary | | 3 | DERIVED | t3 | ref | PRIMARY,feid,exchange,deal_feid,feid_date | exchange | 3 | const | 8488 | | | 3 | DERIVED | t1 | eq_ref | PRIMARY | PRIMARY | 4 | test.t3.deal_id | 1 | Using index | | 3 | DERIVED | t2 | eq_ref | PRIMARY,full_id | PRIMARY | 8 | test.t3.deal_id,test.t3.feid | 1 | Using index; Distinct | +----+--------------+-------------+--------+-------------------------------------------+----------+---------+------------------------------+------+-----------------------+ 8 rows in set (4.22 sec) MariaDB [test]> SELECT t1.deal_id FROM table_e AS t1 JOIN table_d AS t10 ON (t1.deal_id = t10.deal_id) WHERE t1.deal_id IN (SELECT * FROM (SELECT DISTINCT t1.deal_id FROM table_e AS t1 JOIN table_d AS t2 ON (t1.deal_id = t2.deal_id) JOIN table_c_s AS t3 ON (t2.deal_id = t3.deal_id) AND (t2.feid = t3.feid) JOIN table_f AS t4 ON t3.exchange = t4.exchange) tU) limit 10; +---------+ | deal_id | +---------+ | 30 | | 82 | | 83 | | 108 | | 155 | | 156 | | 162 | | 184 | | 186 | | 195 | +---------+ 10 rows in set (10.62 sec)

1 0

[Maria-developers] my_apc.cc and new innodb
by Sergei Golubchik 19 Oct '12

19 Oct '12

Hi. There's some issue we need to discuss. I've discovered it when merging your APC code into 10.0-monty, but the problem is simple enough and could've been found earlier too. In the thd_killed() you do: 3809 extern "C" int thd_killed(const MYSQL_THD thd) 3810 { 3811 if (!thd) 3812 thd= current_thd; 3813 3814 Apc_target *apc_target= (Apc_target*)&thd->apc_target; 3815 if (apc_target->have_apc_requests()) 3816 apc_target->process_apc_requests(); 3817 3818 if (!(thd->killed & KILL_HARD_BIT)) 3819 return 0; 3820 return thd->killed; 3821 } But there's no rule that thd_killed() cannot be called with thd different from current_thd. That is, with thd of *another thread*. Of course, when it happens, the caller is responsible for making sure that the owner thread of that thd isn't going to use it concurrently. And that's exactly what happens in the new InnoDB. Now, show explain wasn't designed to work in this situation, it created items (which are created using sql_alloc, that uses mem_root of the current thread, and added to the free item list of the current thd). As a short-term fix, I've added a protection to thd_killed() to not call apc functions if thd != current_thd. But I don't like it, I think a better solution would be to switch thds, like thread pool does, and process apc requests anyway. What do you think? Regards, Sergei

1 0

[Maria-developers] libmysqlclient16 for MariaDB 5.5?
by Jim Nelin 19 Oct '12

19 Oct '12

Hi guys! I just recently joined this team/list, so I'm going to be polite and introduce myself. My name is Jim, work mainly as a server technician and developer and I've been using MariaDB for the past 2 years or so. Just recently i upgraded one of my older MySQL 5.1 nodes to MariaDB 5.5 - which comes with a lot of nice new features (I've previously only used MariaDB 5.1 and 5.2) I mainly use Debian nodes (stable/squeeze) for my and my clients projects. One thing i discovered shortly after the first upgrade was that there was no libmysqlclient16 (or libmariadbclient16) for the 5.5 release of MariaDB - only the new libmysqlclient18. This causes my applications to still use the older MySQL 5.1 release of libmysqlclient16 from the stable debian repo. As a lot of packages in Debian Squeeze (and squeeze-backports) still depends on libmysqlclient16 (php5-mysql, postfix-mysql, dovecot-common to mention a few.) It still works without any problems, with the stable/squeeze 5.1 mysqlclientlib16 (afaik anyway), but seems a bit risky to use an older client-lib against a newer server (I'm not aware of the changes in the client lib between 5.1 and 5.5 tho). The work-around this (to make it "a bit" better) was to manually install the .deb packages from MariaDB 5.3 which DO contain libmysqlclient16 (libmysqlclient16-5.3.9-mariadb117~squeeze). This still isn't "perfect" in my opinion - i would love to have a real lib(mysql|mariadb)client16 for the 5.5 release on Debian Squeeze. This is my current "setup" is the following: http://paste.debian.net/hidden/7c8219e0/ (basicly the client16-lib from 5.3 against a 5.5 server). I understand that it won't be any performance increases or similar due to older applications not using the new features in libmysqlclient18 or mDB 5.5. But it would help me to get rid of the warning below in phpMyAdmin and also make me feel more safe with the setup - a more consistent setup. "Your PHP MySQL library version 5.3.9-MariaDB differs from your MySQL server version 5.5.27. This may cause unpredictable behavior." So, the questions is - why is there isn't any libmysqlclient16 for MariaDB 5.5? And it possible to add it to the repos? If not - why? Can this be solved with a virtual package or wrapper-lib of some kind against libmariadbclient18? Please correct me if I've got something wrong. Thanks in advance, Best Regards -- Jim Nelin | +46 (0)720 454 909 Jine | http://jine.se | jim(a)jine.se

2 1

[Maria-developers] Fwd: XA problems in mysql-5.6.7-rc
by Rich Prohaska 19 Oct '12

19 Oct '12

Hello, I have been debugging a deadlock problem in the 2PC logging code in MySQL 5.6.7-rc. Luckily, Sergei pointed me to a bug fix. I integrated this fix into MySQL 5.6.7, and with a couple of tweaks, the deadlock problem is fixed. I verified the fix with helgrind, since helgrind reported a lock ordering violation in the original 5.6.7 code. My tweaks are: 1. Move the mutex unlock to the right place in the TC_LOG_MMAP::log_xid function. WRT the main maria trunk, move line 7087 of log.c to after the break, since the code expects the page lock to be held after the for loop. 2. helgrind reported a race on tc_log_cur_pages_used, so I used statistic_increment on line 6983. Also need to build with SAFE_STATISTICS so that the statistics counters are atomically modified. ---------- Forwarded message ---------- From: Sergei Golubchik <serg(a)askmonty.org> Date: Thu, Oct 11, 2012 at 4:04 PM Subject: Re: XA problems in mysql-5.6.7-rc To: Rich Prohaska <prohaska(a)tokutek.com> Cc: internals(a)lists.mysql.com Hi, Rich! On Oct 11, Rich Prohaska wrote: > Hello Sergei, > Can you give me a pointer to a source files, or a source control changeset? This is the bug report: https://bugs.launchpad.net/maria/+bug/578117 And this is the patch: http://bazaar.launchpad.net/~maria-captains/maria/5.1/revision/2881 > On Thu, Oct 11, 2012 at 3:54 PM, Sergei Golubchik <serg(a)askmonty.org> wrote: > > On Oct 11, Rich Prohaska wrote: > >> Hello All, > >> > >> TokuDB supports XA and runs in both MariaDB and MySQL. It works in > >> MariaDB without any patches. Unfortunately, there are several XA bugs > >> related to multiple XA storage engines in MySQL 5.6.7. It would be > >> great if these bugs were fixed in MySQL. One way to test these > >> changes is to include another example storage engine that supports XA. > >> This storage engine does not even need to work; it just needs to > >> export the prepare method in the handlerton interface. > >> > >> There is a block of code in handler.cc surrounded by > >> WILL_BE_DELETED_LATER. If I disable the debug assert, what else do I > >> need to worry about? > > > > There is a possible deadlock in the MySQL XA code. I don't know how > > probable it is to get it in practice, we've found it with our mutex > > deadlock detector (MariaDB only). I had to rework XA code in MariaDB > > quite a bit to get rid of the deadlock. > > Regards, Sergei

1 0

[Maria-developers] MariaDB and spatial indexes
by Vangelis Katsikaros 16 Oct '12

16 Oct '12

Hi I completed my MSc thesis, that was performed with the co-supervision of Sergei Golubchik from Monty Program AB. It's titled "Towards the universal spatial data model based indexing and its implementation in MySQL" which translates into doing a literature overview for R-tree based spatial indexes, GiST trees (Generalized Search Trees) and implementing them in MyISAM. The work is in progress! In case you are interested in the algorithm details of these indexes: * Thesis text: http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6394 * Current patches: https://www.dropbox.com/s/bw2cq2525zypi6m/patches.tar.gz Many many thanks to Sergei for all his help and support :) Regards Vangelis

1 0

[Maria-developers] FYI: SHOW EXPLAIN pushed to 10.0-base
by Sergei Petrunia 06 Oct '12

06 Oct '12

Hello, FYI, SHOW EXPLAIN has been merged and pushed to 10.0-base. It was not pushed into 5.5. Just in case, the final 5.5-based tree with SHOW EXPLAIN patch can be found here: https://code.launchpad.net/~maria-captains/maria/5.5-show-explain BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Global transaction ID
by Kristian Nielsen 05 Oct '12

05 Oct '12

Hi Kristian, Monty asked me to connect with you regarding global transaction IDs in MariaDB. I reviewed some time ago the MySQL 5.6 global transaction ID design and implementation. Unfortunately I really did not like what I found, to the point that I decided this was not something we would want to have or support in MariaDB (see https://lists.launchpad.net/maria-developers/msg04837.html for an ealier mail on this). Both you and Giuseppe also had critisisms about it. Instead I came up with a (in my opinion) better design. I have now written this up in (great) detail here: https://mariadb.atlassian.net/browse/MDEV-26 It would be great if you (/your team) could comment on this and how it relates to what you need or any comments/suggestions etc. I know it is a long text, and a complex problem, so also feel free to just ask me for any clarifications or explanations you would like, even without digging through the whole text. The basic idea is to keep the existing concept of a stream of replication events and a slave's current position within this stream, but in a way that is uniform across the whole multi-level replication topology (global transaction ID), and that facilitates simple scripting of automated promotion of a new master. I have thought for a long time over this problem and how to best approach it and make it work well both with existing replication usage and with new features like multi-source and parallel replication. It is hard to get right, but I am fairly pleased with what I have ended up with so far. I am hopeful that this will be a good way forward, but I very much welcome any comments or alternate suggestions. Regards, - (the other) Kristian.

2 2

[Maria-developers] MariaDB Optimization for MySQL Users
by Daniel Bartholomew 04 Oct '12

04 Oct '12

MariaDB contains many new options and optimizations which, for compatibility or other reasons, are not enabled in the default install. Enabling them helps you gain extra performance from the same hardware when upgrading from MySQL to MariaDB. Monty thought this would be a good article for the Knowledgebase and I agree. I've created a stub article for this using a few short items Monty provided (just to get the ball rolling), and I am now soliciting suggestions from any and all of you so I can expand it. https://kb.askmonty.org/en/mariadb-optimization-for-mysql-users/ Please send me your MariaDB-only tips and recommended settings and other things people should enable or tweak after upgrading to MariaDB and I'll add them to the page (or feel free to do it yourself). Thanks! -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

Re: [Maria-developers] multi-source replication, first (untested) version
by Kristian Nielsen 02 Oct '12

02 Oct '12

Michael Widenius <monty(a)askmonty.org> writes: > I have now got the first version of multi source patch ported to > MariadB. As requested, I did a quick read through of the patch. I took the patch from your 10.0-mdev253 tree, this revision: revid:monty@askmonty.org-20120930233044-e0m6u6t9q3jeqx03 Below are my comments. Overall, it was nice to see the relatively modest changes needed to get this working. - Kristian. ----------------------------------------------------------------------- > === modified file 'sql/sys_vars.cc' > --- sql/sys_vars.cc 2012-09-22 12:30:24 +0000 > +++ sql/sys_vars.cc 2012-10-01 09:55:23 +0000 > +static Sys_var_multi_source_ulong > +Sys_slave_skip_counter("sql_slave_skip_counter", > + "Skip the next N events from the master log", > + SESSION_VAR(slave_skip_counter), > + NO_CMD_LINE, > + offsetof(Master_info, rli.slave_skip_counter), This use of offsetof() on a C++ class (non-POD) is incorrect. Even if it works for you with your current version of GCC and current compiler options, it is likely to break at some point on some future compiler. And it will be hell to track down. Look, I understand that you do not like some of the ways that the C++ language is defined, like here that offsetof() is not allowed on non-POD data structures. But we have to live with it. Surely, it is better to have a standard language that compiler writers and compiler users agree on how works - even if that language is less convenient than what we would like. So let us please change this to be correct C++. I can change it for you if you want. > === modified file 'sql/slave.cc' > --- sql/slave.cc 2012-09-13 12:31:29 +0000 > +++ sql/slave.cc 2012-10-01 09:55:23 +0000 > +bool show_all_master_info(THD* thd) > +{ > + uint i, elements; > + Master_info **tmp; > + DBUG_ENTER("show_master_info"); > + > + if (send_show_master_info_header(thd, active_mi, 1)) > + DBUG_RETURN(TRUE); > + > + if (!(elements= master_info_index->master_info_hash.records)) > + goto end; > + > + /* > + Sort lines to get them into a predicted order > + (needed for test cases and to not confuse users) > + */ > + if (!(tmp= (Master_info**) thd->alloc(sizeof(Master_info*) * elements))) > + DBUG_RETURN(TRUE); > + > + for (i= 0; i < elements; i++) > + { > + tmp[i]= (Master_info *) my_hash_element(&master_info_index-> > + master_info_hash, i); > + } Isn't there a race condition here? What happens if the number of elements changes between looking up number of elements and the loop? If there is a lock protecting this, please add a safe_mutex_assert_owner() and/or a comment. The call of show_all_master_info() is like this: mysql_mutex_lock(&LOCK_active_mi); if (lex->verbose) res= show_all_master_info(thd); Does LOCK_active_mi protect all changes to multi-master state? If so, I think it should be renamed - "active_mi" is rather confusing in this case. > === modified file 'sql/sql_class.h' > --- sql/sql_class.h 2012-09-22 14:11:40 +0000 > +++ sql/sql_class.h 2012-10-01 09:55:23 +0000 > + /** > + Place holders to store Multi-source variables in sys_var.cc during > + update and show of variables. > + */ > + ulong slave_skip_counter; > + ulong max_relay_log_size; > + /* Names. These will be allocated in buffers in thd */ > + LEX_STRING default_master_connection; > + > + LEX_STRING connection_name; /* If slave */ > + char default_master_connection_buff[MAX_CONNECTION_NAME+1]; Is there not some other way to do this? THD seems to have become some kind of garbage dump for all kinds of special-purpose data. This is not good, it pollutes the CPU caches for all threads, and generally makes everything a mess :-/ I would like to see us move towards having less things in THD that are not relevant to all threads, not more. Could we add a single void * somewhere in THD for special-purpose threads? And then replication threads could use that for their special-purpose stuff. Or maybe sub-class THD. Then later we could start moving other stuff out of THD that is not relevant to all threads... > === modified file 'sql/rpl_rli.h' > --- sql/rpl_rli.h 2012-08-27 16:13:17 +0000 > +++ sql/rpl_rli.h 2012-10-01 09:55:23 +0000 > + volatile ulong slave_skip_counter; /* Must be ulong */ Such comments are extremely unhelpful. Instead explain why it must be ulong. > === modified file 'sql/item_create.cc' > --- sql/item_create.cc 2012-01-13 14:50:02 +0000 > +++ sql/item_create.cc 2012-10-01 09:55:23 +0000 > @@ -4393,27 +4393,36 @@ Create_func_master_pos_wait::create_nati > + thd->lex->safe_to_cache_query= 0; > case 3: > thd->lex->safe_to_cache_query= 0; > + case 4: > + thd->lex->safe_to_cache_query= 0; Isn't it redundant to set safe_to_cache_query twice here? > === modified file 'sql/rpl_mi.cc' > --- sql/rpl_mi.cc 2012-03-27 23:04:46 +0000 > +++ sql/rpl_mi.cc 2012-10-01 09:55:23 +0000 > + /* Store connection name and lower case connection name */ > + connection_name.length= cmp_connection_name.length= > + connection_name_arg->length; > + if ((connection_name.str= (char*) my_malloc(connection_name_arg->length*2+2, > + MYF(MY_WME)))) > + { > + cmp_connection_name.str= (connection_name.str + > + connection_name_arg->length+1); > + strmake(connection_name.str, connection_name_arg->str, > + connection_name.length); > + memcpy(cmp_connection_name.str, connection_name_arg->str, > + connection_name.length+1); > + my_casedn_str(system_charset_info, cmp_connection_name.str); > + } Is it really correct to ignore out-of-memory error here, and just do nothing? Ah, I think this is handled by checking Master_info::error() in other places? If so, then please add a comment here explaining this. > +/** > + Create a log file with a signed suffix. I do not know what is meant with the term "signed suffix". This is not something that people will understand. Does it mean "Create a log file name by appending a dash "-" followed by a suffix" ? Also rename the file create_signed_file_name() - again, the "signed" is confusing. For example "create_augmented_file_name", or "create_name_with_dash_and_suffix" or something. > + It's valid if it's a valid system name, is less than > + MAX_CONNECTION_NAME. "It is valid if it is a valid system name of length less than MAX_CONNECTION_NAME". > === modified file 'client/mysqltest.cc' > --- client/mysqltest.cc 2012-08-31 21:54:54 +0000 > +++ client/mysqltest.cc 2012-10-01 09:55:23 +0000 > + if (buff) > + my_free(start); Better use my_free(buff) here. > === modified file 'sql/rpl_mi.h' > --- sql/rpl_mi.h 2012-03-27 23:04:46 +0000 > +++ sql/rpl_mi.h 2012-10-01 09:55:23 +0000 > + char master_log_name[FN_REFLEN+6]; /* Place for multi-*/ "place for" -> "room for". > === modified file 'sql/mysqld.cc' > --- sql/mysqld.cc 2012-09-22 14:11:40 +0000 > +++ sql/mysqld.cc 2012-10-01 09:55:23 +0000 > + /* > + We must have LOCK_open before LOCK_global_system_variables because > + LOCK_open is hold while sql_plugin.c::intern_sys_var_ptr() is called. "is hold" -> "is held" > === modified file 'sql/item_create.cc' > --- sql/item_create.cc 2012-01-13 14:50:02 +0000 > +++ sql/item_create.cc 2012-10-01 09:55:23 +0000 > @@ -4393,27 +4393,36 @@ Create_func_master_pos_wait::create_nati > if (item_list != NULL) > arg_count= item_list->elements; > > + if (arg_count < 2 || arg_count >4) Missing space before "4"

2 1

[Maria-developers] Json Explain
by Peter Laursen 30 Sep '12

30 Sep '12

Is Explain output in Json format planned in MariaDB (refer http://glebshchepa.blogspot.in/2012/04/optimizer-new-explain-formatjson.html) ? If so when? and will the implementation details and output format be fully identical to Oracle/MySQL 5.6? (I did not find it mentioned here: http://monty-says.blogspot.dk/2012/09/now-is-good-time-to-be-part-of-future… http://blog.mariadb.org/mariadb-directions/ .. sorry if I overlooked it somewhere) -- Peter -- Webyog

4 8

Re: [Maria-developers] [Commits] Rev 3578: MDEV-495 backport --ignore-db-dir
by Sergei Golubchik 25 Sep '12

25 Sep '12

Hi, Holyfoot! On Sep 25, holyfoot(a)askmonty.org wrote: > revno: 3578 > revision-id: holyfoot(a)askmonty.org-20120925094024-yv0wpbl6k7b6bgg8 > parent: sanja(a)montyprogram.com-20120920094859-lcrxe8lr1mtg8h10 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev495 > timestamp: Tue 2012-09-25 14:40:24 +0500 > message: > MDEV-495 backport --ignore-db-dir. > The feature was backported from MySQL 5.6. > Some code was added to make commands as > SELECT * FROM ignored_db.t1; > CALL ignored_db.proc(); > USE ignored_db; > to take that option into account. Just a few comments: 1. add a test case for CREATE DATABASE, CREATE DATABASE ignored_db should fail. 2. I'd prefer to get an error "db not found" instead of "incorrect db name", is it difficult to do? 3. add a test for the following: lost+found is ignored, but CREATE DATABASE `lost+found` should succeed! (indeed, it creates a directory lost@002bfound, which is not ignored), and later SELECT, CALL, USE, SHOW DATABASES, should show that new db. > === added file 'mysql-test/r/ignore_db_dirs_basic.result' > --- a/mysql-test/r/ignore_db_dirs_basic.result 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/r/ignore_db_dirs_basic.result 2012-09-25 09:40:24 +0000 > @@ -0,0 +1,28 @@ > +select @@ignore_db_dirs; > +@@ignore_db_dirs > +e,lost+found,.mysqlgui,ignored_db > +# Check that SHOW DATABASES ignores all directories from > +# @@ignore_db_dirs and all directories with names starting > +# with '.' > +SHOW DATABASES; > +Database > +information_schema > +#mysql50#.otherdir > +mtr > +mysql > +test > +USE ignored_db; > +ERROR 42000: Incorrect database name 'ignored_db' > +SELECT * FROM ignored_db.t1; > +ERROR 42000: Incorrect database name 'ignored_db' > +CALL ignored_db.p1(); > +ERROR 42000: Incorrect database name 'ignored_db' > +SELECT COUNT(*) FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME='ignored_db'; > +COUNT(*) > +1 > +SET @@global.ignore_db_dirs = 'aha'; > +ERROR HY000: Variable 'ignore_db_dirs' is a read only variable > +SET @@local.ignore_db_dirs = 'aha'; > +ERROR HY000: Variable 'ignore_db_dirs' is a read only variable > +SET @@ignore_db_dirs = 'aha'; > +ERROR HY000: Variable 'ignore_db_dirs' is a read only variable When merging to 5.5 you'll need to move this test to sys_vars suite (otherwise sys_vars.all_vars will fail) > === modified file 'sql/sql_show.cc' > --- a/sql/sql_show.cc 2012-08-24 13:39:34 +0000 > +++ b/sql/sql_show.cc 2012-09-25 09:40:24 +0000 > @@ -379,6 +379,239 @@ bool mysqld_show_privileges(THD *thd) > } > > > +/** Hash of LEX_STRINGs used to search for ignored db directories. */ > +static HASH ignore_db_dirs_hash; > + > +/** > + An array of LEX_STRING pointers to collect the options at > + option parsing time. > +*/ > +static DYNAMIC_ARRAY ignore_db_dirs_array; I don't think that sql_show.cc is where all this code really belongs, but ok, as you like. Regards, Sergei

1 0

[Maria-developers] FYI: More ha_cassandra benchmark data
by Sergei Petrunia 24 Sep '12

24 Sep '12

Hi! For the record, posting some ha_cassandra benchmarks: <contents> Run #1: re-run previous benchmarks Run #2: evaluate max. network performance Run #3: Load two dbt3 databases at once Run #4: load two datasets into different nodes Results Conclusions </contents> === Run #1: re-run previous benchmarks === My results with - m1.large mysqld node (mysqld uses optimized binary) - Cassandra on 2*m1.large nodes - Loading data with the standard LOAD DATA INFILE command. - %CPU mysqld =14 sar -n DEV 10: 03:04:41 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:04:51 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 03:04:51 PM eth0 1575.20 156.60 81.47 4458.13 0.00 0.00 0.00 On cassandra node: - jsvc %CPU 70-170... 02:51:00 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 02:51:05 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:51:05 PM eth0 8325.20 7333.80 3376.25 1696.74 0.00 0.00 0.00 Total load time: time: 19 min. === Run #2: evaluate max. network performance === for comparison, sending /dev/zero on one host to /dev/null on another with nc: 03:39:24 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:39:34 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 03:39:34 PM eth0 16756.70 3688.50 851.91 87477.61 0.00 0.00 0.00 sending in both directions: SQL: 03:45:36 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:45:46 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 03:45:46 PM eth0 63308.20 4473.50 73002.80 81484.77 0.00 0.00 0.00 Cassandra: 03:47:32 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:47:42 PM lo 0.20 0.20 0.01 0.01 0.00 0.00 0.00 03:47:42 PM eth0 60741.50 15241.70 84110.07 70712.32 0.00 0.00 0.00 === Run #3: Load two dbt3 databases at once === - Loading two DBT-3 databases at once. - The databases have their own key spaces, and mysql tables - Still, both loaders connect to the same cassandra host. Cassandra host: 04:27:57 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 04:28:07 PM eth0 9979.10 8984.00 6006.77 2621.08 0.00 0.00 0.00 04:29:07 PM eth0 9181.80 7315.00 6416.25 2686.83 0.00 0.00 0.00 04:29:17 PM eth0 10277.00 9629.60 6047.00 2672.02 0.00 0.00 0.00 04:30:23 PM CPU %user %nice %system %iowait %steal %idle 04:30:33 PM all 34.56 24.80 24.71 0.18 0.05 15.69 04:30:53 PM all 32.08 11.20 24.25 1.31 0.08 31.07 04:31:03 PM all 35.59 22.55 22.28 0.27 0.00 19.31 it has periodical slow-downs like this: 04:32:38 PM all 53.25 1.30 19.05 3.03 0.00 23.38 04:32:39 PM all 65.84 10.40 21.78 0.00 0.00 1.98 04:32:40 PM all 37.14 33.81 18.10 0.95 0.00 10.00 04:32:41 PM all 42.65 25.98 26.96 0.00 0.00 4.41 04:32:42 PM all 33.17 39.42 19.23 0.00 0.00 8.17 04:32:43 PM all 36.89 36.89 19.90 0.00 0.00 6.31 04:32:44 PM all 39.81 35.55 13.74 0.00 0.00 10.90 04:32:45 PM all 34.45 33.97 22.01 0.00 0.00 9.57 04:32:46 PM all 35.58 36.06 21.15 0.00 0.00 7.21 04:32:47 PM all 31.05 35.16 15.07 0.00 0.00 18.72 04:32:48 PM all 41.31 26.29 20.66 0.00 0.00 11.74 04:32:49 PM all 39.02 32.20 25.37 0.00 0.00 3.41 04:32:50 PM all 26.64 34.50 13.97 0.00 0.00 24.89 04:32:51 PM all 56.93 27.72 12.87 0.00 0.00 2.48 04:32:52 PM all 71.14 15.42 11.44 0.00 0.00 1.99 04:32:53 PM all 64.22 8.82 23.04 0.00 0.00 3.92 04:32:54 PM all 76.73 1.98 19.31 0.00 0.00 1.98 04:32:55 PM all 61.24 0.00 30.14 0.96 0.00 7.66 04:32:56 PM all 29.20 0.00 32.40 0.00 0.00 38.40 04:32:57 PM all 30.29 0.00 29.88 0.00 0.00 39.83 04:32:58 PM all 23.19 0.00 20.29 0.00 0.36 56.16 04:32:59 PM all 23.34 0.00 16.38 0.00 0.35 59.93 04:33:00 PM all 20.07 0.00 21.90 0.00 0.00 58.03 04:33:01 PM all 24.20 0.00 18.86 0.00 0.00 56.94 04:33:02 PM all 20.07 0.00 22.18 0.00 0.70 57.04 Is this a compaction? mysqld host: 04:29:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 04:29:01 PM eth0 1416.20 247.90 75.43 6578.11 0.00 0.00 0.00 04:29:11 PM eth0 1175.00 199.60 62.49 5292.67 0.00 0.00 0.00 04:29:21 PM eth0 1074.60 186.90 57.21 4952.19 0.00 0.00 0.00 04:30:30 PM CPU %user %nice %system %iowait %steal %idle 04:30:40 PM all 7.77 0.00 0.56 0.00 0.05 91.62 04:30:50 PM all 8.20 0.00 0.60 0.00 0.30 90.89 04:31:00 PM all 8.01 0.00 0.66 0.00 0.05 91.28 Conclusions: - mysqld is still idle most of the time. - Cassandra is busier. Sometimes, it is 1014/Sep/12 12:50 PM0% busy. - network use betwen mysqld and cassandra is 2x more, which is expected Run time: 24 minutes for both (compare to 19 minutes) === Run #4: load two datasets into different nodes === ==== Network ==== SQL node: 05:44:21 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 05:44:41 PM eth0 1178.10 210.10 62.81 5510.03 0.00 0.00 0.00 05:44:51 PM eth0 1062.10 200.90 57.53 5152.63 0.00 0.00 0.00 05:45:01 PM eth0 965.80 194.00 52.53 5012.62 0.00 0.00 0.00 Cassandra1: 05:44:19 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 05:44:49 PM eth0 10159.00 8659.20 4443.26 1798.75 0.00 0.00 0.00 05:44:59 PM eth0 9018.20 7750.10 3982.21 1578.30 0.00 0.00 0.00 05:45:09 PM eth0 8741.30 7521.70 3921.69 1541.55 0.00 0.00 0.00 Cassandra2: 05:44:56 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 05:45:06 PM eth0 9305.60 7906.00 4121.10 1622.92 0.00 0.00 0.00 05:45:16 PM eth0 9186.50 7828.80 4258.12 1681.78 0.00 0.00 0.00 05:45:26 PM eth0 9675.70 8283.10 4306.89 1698.37 0.00 0.00 0.00 ==== CPU ==== SQL node: 05:46:45 PM CPU %user %nice %system %iowait %steal %idle 05:46:55 PM all 8.39 0.00 0.70 0.00 0.40 90.51 05:47:05 PM all 7.65 0.00 0.74 0.00 0.70 90.91 05:47:15 PM all 8.76 0.00 0.64 0.00 0.89 89.70 cassandra 1: 05:47:01 PM CPU %user %nice %system %iowait %steal %idle 05:47:11 PM all 27.28 28.24 27.55 0.00 0.05 16.89 05:47:21 PM all 31.45 24.44 27.56 1.38 0.05 15.13 05:47:31 PM all 29.46 19.60 23.19 2.35 0.04 25.36 05:47:41 PM all 38.47 0.04 14.21 0.15 0.07 47.06 05:47:51 PM all 31.04 9.92 17.97 0.39 0.08 40.61 05:48:01 PM all 24.73 28.07 21.77 0.26 0.04 25.12 05:48:11 PM all 19.79 11.67 19.23 1.83 0.07 47.41 05:48:21 PM all 20.29 0.00 11.07 0.06 0.13 68.44 05:48:31 PM all 24.08 0.00 11.96 0.13 0.13 63.69 Cassandra 2: 05:46:56 PM CPU %user %nice %system %iowait %steal %idle 05:47:06 PM all 31.97 27.14 26.91 0.09 0.05 13.85 05:47:16 PM all 32.79 20.43 23.39 2.34 0.04 21.01 05:47:26 PM all 26.43 27.07 27.29 0.14 0.09 18.98 05:47:36 PM all 17.72 8.59 12.00 2.97 0.21 58.52 05:47:46 PM all 13.63 0.00 6.77 0.06 0.30 79.23 05:47:56 PM all 27.19 3.92 8.52 0.11 0.25 60.01 05:48:06 PM all 19.94 27.83 18.47 0.13 0.08 33.54 05:48:16 PM all 13.69 16.01 14.24 2.69 0.15 53.22 05:48:26 PM all 18.13 0.00 9.57 0.25 0.38 71.66 === Results === The first client failed with ERROR 1928 (HY000) at line 145: Internal error: 'TimedOutException: Default TException.' real 4m3.463s user 0m0.008s sys 0m0.000s The second completed after: real 19m57.875s (which matches load time for loading one dataset). === Conclusions === - need to fix the Timeout error (filed as MDEV-535) - CPU time spent by mysqld is negligible now (I don't know whether this comes from using faster CPU for SQL node, or using a release build of mariadb, or both) - A confirmation of previous findings: if we use only one connection, the speed at which we can insert the data is very limited (see: with two connections, we have inserted 2x data in 20% more time. This is where we used two connections to connect to the *same* node) - I am not sure whether I should just start using multiple connections, or also automatically route the data (both writes and reads) to the right node? - need to check out Hector client. It seems to be able to connect to multiple nodes of Cassandra cluster. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Updates to mytop (from MariaDB 5.5 distribution)
by Michael Widenius 22 Sep '12

22 Sep '12

Hi! I have done some extensions to mytop as part of porting, fixing and enhancing it for MariaDB. I have now merged my changes with 1.91 and thought it was time to share the changes with you. Sorry for not doing it sooner! I also have added Jean Weisbuch patches for --full-queries and reading of .my.cnf Here is a diff of my changes. Hope you find some of them useful! If you have any questions, feel free to ask! Best regards, Monty Creator of MySQL and MariaDB

1 0

[Maria-developers] Cassandra SE: problems including it into release
by Sergei Petrunia 21 Sep '12

21 Sep '12

Hello, I've figured there may be an additional challenge in getting Cassandra storage engine into a *MariaDB release*. The problem is packaging. <contents> Cassandra SE packaging problems Need Thrift library Need recent gcc How did Sphinx SE manage to avoid a similar mess? Possible way out </contents> == Cassandra SE packaging problems == === Need recent gcc === C++ code generated by Thrift uses boost, and so one needs a recent gcc to compile it with MySQL. We have the same issue with oqgraph. See MDEV-501 for details. I am not aware of crashing problems that ograph+boost combination has. Still, recent-gcc requirement may limit the range of platforms we can build for. For example, I never managed to compile on Ubuntu 9.04. === Need Thrift library === Cassandra SE uses Thrift protocol to talk to server. In order to do this, it needs to link againist libthrift.so. There are no Thrift packages in either Fedora or Ubuntu (actually I haven't seen Thrift packages for any OS). Thrift homepage only offers source tarballs. The source tarball uses automake and compiles into 1. thrift compiler binary (we dont actually need it. Source code generated by thrift needs to be modified anyway.) 2. thrift libraries for a dozen of languages, including libthrift.so, which we do need. == How did Sphinx SE manage to avoid a similar mess? == I've asked myself a question how do we manage to include Sphinx SE without having to mess with Sphinx packages. The answer is: Sphinx SE doesn't need any sphinx libraries, either at compile time, or at runtime. It implements its client protocol fully, except for INSERT commands, for which it uses the available mysql client library to talk to Sphinx server using mysql client-server protocol. == An unlikely but possible way out == Cassandra 1.2 (the next version that is promised to be released "in Q4 2012" will introduce native network protocol. A week ago they've pushed first documentation draft. The native network protocol should allow to do everything that Thrift API allows, and more. There are no C++ clients yet, though. If I am faced with a choice betwen 1. implementing a relatively simple network protocol 2. Making Thrift packages for various platforms (and potentially convincing downstream maintainers to accept them) then I will definitely pick #1 (I have implemented network protocols in the past). There is a big risk, though: native protocol support is very new in Cassandra, they pushed it about a week ago. It was pushed into their "trunk" which is under development. I don't know if the code in their trunk is sufficiently stable for us to develop againist. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

3 3

Re: [Maria-developers] New to MariaDB
by Giacomo Picchiarelli 21 Sep '12

21 Sep '12

Thanks! At beginning Il'll try to do some benchmarks! On 13/09/2012 22:11, Michael Widenius wrote: > Hi! > >>>>>> "Sergei" == Sergei Golubchik <serg(a)askmonty.org> writes: > Sergei> Hi, Giacomo! > Sergei> On Aug 11, Giacomo Picchiarelli wrote: >>> Hi, >>> >>> I'm new to MariaDB and I'm reading the knowledge base. >>> I'm interested in studying Aria Engine (and contribute, if I can :) ), >>> are there tasks for newbie developers? > Sergei> I'm afraid, there are no tasks in the Aria engine right now :( > > Sergei> If you're interested to look at other tasks - here is the list of task, > Sergei> feel free to pick anything from it: > > Sergei> https://mariadb.atlassian.net/secure/IssueNavigator.jspa?requestId=11300 > > Actually, there are some open bugs on Aria: > > https://mariadb.atlassian.net/browse/MDEV-333 > > You can go to: > https://mariadb.atlassian.net/secure/QuickSearch.jspa > And search for "Aria" > > Same goes for lanchpad: > > https://bugs.launchpad.net/maria?field.searchtext=aria&search=Search&field.… > > Note that many of the bugs in launchpad for Aria are already fixed. > However it would be quite valuable if you could help us go trough > them and mark which still are open and try to fix them yourself. > > Another thing that can be done on Aria is to run some benchmarks on it > and try to find ways to improve the performance on it. > > Regards, > Michael Widenius > -- Giacomo Picchiarelli Skype: g.picchiarelli

2 1

Re: [Maria-developers] [Commits] Rev 3577: MDEV-521 fix. in file:///home/bell/maria/bzr/work-maria-5.3-MDEV-521/
by Sergei Petrunia 20 Sep '12

20 Sep '12

Hi Sanja, Ok to push. On Thu, Sep 20, 2012 at 12:49:03PM +0300, sanja(a)montyprogram.com wrote: > At file:///home/bell/maria/bzr/work-maria-5.3-MDEV-521/ > > ------------------------------------------------------------ > revno: 3577 > revision-id: sanja(a)montyprogram.com-20120920094859-lcrxe8lr1mtg8h10 > parent: timour(a)askmonty.org-20120917081346-nqd08d0v27w8zp5c > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.3-MDEV-521 > timestamp: Thu 2012-09-20 12:48:59 +0300 > message: > MDEV-521 fix. > > After pullout item during single row subselect transformation it should be fixed properly. > === modified file 'mysql-test/r/subselect.result' > --- a/mysql-test/r/subselect.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect.result 2012-09-20 09:48:59 +0000 > @@ -6163,5 +6163,20 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > > === modified file 'mysql-test/r/subselect_no_mat.result' > --- a/mysql-test/r/subselect_no_mat.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_mat.result 2012-09-20 09:48:59 +0000 > @@ -6162,6 +6162,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set optimizer_switch=default; > > === modified file 'mysql-test/r/subselect_no_opts.result' > --- a/mysql-test/r/subselect_no_opts.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_opts.result 2012-09-20 09:48:59 +0000 > @@ -6158,6 +6158,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set @optimizer_switch_for_subselect_test=null; > > === modified file 'mysql-test/r/subselect_no_scache.result' > --- a/mysql-test/r/subselect_no_scache.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_scache.result 2012-09-20 09:48:59 +0000 > @@ -6169,6 +6169,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set optimizer_switch=default; > > === modified file 'mysql-test/r/subselect_no_semijoin.result' > --- a/mysql-test/r/subselect_no_semijoin.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_semijoin.result 2012-09-20 09:48:59 +0000 > @@ -6158,6 +6158,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set @optimizer_switch_for_subselect_test=null; > > === modified file 'mysql-test/t/subselect.test' > --- a/mysql-test/t/subselect.test 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/t/subselect.test 2012-09-20 09:48:59 +0000 > @@ -5229,5 +5229,18 @@ set optimizer_switch=@mdev367_optimizer_ > > DROP TABLE t1; > > +--echo # > +--echo # MDEV-521 single value subselect transformation problem > +--echo # > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > + > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > + > +# Cleanup > +DROP TABLE t1; > + > --echo # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > > === modified file 'sql/item_subselect.cc' > --- a/sql/item_subselect.cc 2012-09-17 08:13:46 +0000 > +++ b/sql/item_subselect.cc 2012-09-20 09:48:59 +0000 > @@ -1007,11 +1007,9 @@ Item_singlerow_subselect::select_transfo > } > substitution= select_lex->item_list.head(); > /* > - as far as we moved content to upper level, field which depend of > - 'upper' select is not really dependent => we remove this dependence > + as far as we moved content to upper level we have to fix dependences & Co > */ > - substitution->walk(&Item::remove_dependence_processor, 0, > - (uchar *) select_lex->outer_select()); > + substitution->fix_after_pullout(select_lex->outer_select(), &substitution); > } > DBUG_RETURN(false); > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] Server-id patch
by Kristian Nielsen 18 Sep '12

18 Sep '12

Robert Hodges <robert.hodges(a)continuent.com> writes: > I think we need to take this out to one of the email threads. I hope other > people would be interested in these problems. Agree, I've quoted liberally and Cc:'ed maria-developers@ > On Aug 31, 2012, at 6:14 AM PDT, Kristian Nielsen wrote: >> Robert Hodges <robert.hodges(a)continuent.com> writes: >> >>> 1.) The server-id of which I was speaking is a very simple patch to allow us >>> to do the following client command: 'set session server-id=X' where X is >>> whatever we choose. Currently MySQL replication sets the server ID from an >>> internal structure (THD as I recall), which means that applications like >>> Tungsten cannot get the server-id into the binlog when applying updates on a >>> slave. Server-id is essential to prevent replication loops. Monty and I >>> discussed this feature briefly over dinner in April. I hacked in support >>> using a UDF about a year ago but it would be better to have this become a >>> dynamic session variable. >> Yes, it does not seem too hard to do (though as always some care is needed to >> introduce it in a clean way). >> >> I created an issue for this with some descriptions: >> >> https://mariadb.atlassian.net/browse/MDEV-500 > SET SESSION pseudo_server_id=236; > > Then the server_id 236 would to go into the binlog. This seems like a good > idea and avoids accidentally reseting the server_id, which is a pretty > significant variable. Seems there is hope we can have this in 10.0. >>> 2.) I agree with your criticisms of the MySQL 5.6 replication design. It's >>> surprisingly complex and looks as if it will have a lot of bugs. I felt at >> >> Yes. They need this complexity because their parallel replication (which is >> similar to Tungsten parallel replication I think?) can arbitrarily reorder >> transactions in a slave binlog compared to the order in the master binlog. > We just use the sequential log. The slaves are complex but it does not > pollute the log. In fact, everything up to the point of applying to the > slave is purely sequential. I think their design got this wrong. Well, the MySQL --log-slave-updates work by generating completely new events on the slave as the received events are executed by the SQL thread(s). Do you mean that Tungsten does not do this, but uses the original binlog with original events from the master? How is multi-master (one slave replicating from several masters) handled (if it is handled) ? >>> the time that the better approach on transaction IDs would be to put enough >>> information into logs to enable applications to compute a vector clock on >>> the stream of binlog events coming from multiple servers. This is a >> >> Interesting, I though about a concept of "vector clock" before, but did not >> know the terminology. >> >> Aha, so your point is that even though we will apply changes from multiple >> upstream master servers in parallel on a single slave, with a vector clock, we >> can still order them on a slave according to which transactions on one server >> where visible to which transactions on the other. And stuff like >> that. Interesting! >> >> But note that to be truly "theoretically sound", all queries (ie. SELECT) as >> well as all applications that interact directly or indirectly with multiple >> servers need to be included in the vector clock. So it is not without >> complexity. > Are you thinking about holding locks? For practical purposes it's enough to > do writes only. My point is that you have a global ID that is partially > ordered by originating server. Vector clocks are the traditional way of > handling that. This is the basis for most mechanisms of conflict > resolution--you check to see if the clocks are ordered and if they are not > (because two thing happened at the same time, hence are not comparable), it > signals a conflict that must be reconciled. Well, I was mostly just reacting to the phrase "truly theoretically sound". If readers are not considered in the vector clock, then they are able to observe different states of the database at different nodes in the cluster. But as you say, that is probably acceptable. >> This does not immediately apply to the MySQL design, as they work with a >> single master (applying in parallel transactions in different databases), and >> as they arbitrarily reorder between different databases, without any attempt >> to maintain inter-database consistency. But of course, my point is that this >> design is fundamentally wrong. >> >>> theoretically sound approach to enable both synchronous and asynchronous >>> multi-master replication. (Galera incidentally uses approximately this >>> approach for sync replication as you may be aware.) If you give us server >>> IDs we don't need anything else. Oops, I forgot. Making the binlog truly >>> crash safe in every possible case is also very helpful. >> >> I think it is getting closer to being truly crash safe - can you mention the >> situations where it is not? And btw., are you interested in the recent testing >> we (especially Elena) made for testing such crash safety, and the fixes made >> for it? We found some fairly interesting (and scary) stuff... > Actually your (MP's?) fix for group commit is the most important > improvement. Can we turn sync_binlog back on without killing performance? Yes, well at least performance is substantially improved with sync_binlog=1. With a parallel load you can see improvements of 10x or more, and even single-threaded load is improved in 10.0. >> I have been thinking about this for some time. Basically, the correct approach >> is to make the binlog into a true transaction log using standard textbook >> techniques. Pre-allocate the log, make it page based (this will also improve >> performance). Do ping-pong writes at the end so that we can recover even from >> torn pages. Stuff like that. Would be cool to do, but so far I have not really >> seen much interest for something like this. > It would be a lot of work. If group commit is fixed, then most on-premise > people can solve enough problems with RAID and BBU that they don't need much > more. Right, agree, I thought you had something more subtle than sync_binlog=0 in mind. (BTW, I fixed a couple binlog crash recovery bugs recently in MariaDB, which also affect MySQL). >>> 3.) I read the email you cited. I need some time to study it in order to >>> comment intelligently. The most immediate feedback is that I doubt you want >>> to make assumptions about how slaves will apply transactions. The role of >>> the master is to provide local transactions in serial order. Looking across >>> a set of masters you then have a set of easily identifiable streams of >>> partially ordered transactions. That in turn provides a mathematically >>> tractable model for reasoning about things like update order, dependencies, >>> and potential conflicts. Any metadata you can supply about dependencies, >>> e.g., which schema(s) transactions actually touch is useful but may or may >>> not be relevant on slaves depending on the application. It's up to slaves >>> to make decisions on reordering. >> >> But think about a three-level multi-master topology. Servers A and B are >> masters for server C, which is itself a master for D. The DBA/application >> developer will declare that transactions on A and B are independent, so C can >> apply them in any order, and in parallel. Clearly on D we will want to also >> apply them in parallel, and the only way we can do that is to use the >> different server ids A and B in the stream of events in the binlog of C. >> >> So now consider a simple two-level topology with just master C and slave >> D. Why not still let C put different "server_id"s in different events, as a >> way to tell server D that events are independent? For example, if the DBA runs >> a long-running ALTER TABLE, he can ensure that no transactions touch this >> table while it runs, and just SET SESSION SERVER_ID to something else, and it >> can run in parallel on D without delaying normal transactions. >> >> Or think about MySQL 5.6 style parallel replication, where the DBA declares >> that on server A, transactions against different databases are to be >> considered independent and applied in parallel on the slave. While on server B >> such transactions are not independent. Now we want to setup slave C to >> replicate from both A and B. Clearly, C is the wrong place to configure >> whether transactions in different databases are to be considered independent - >> this is a property of how the transactions are generated, not where they are >> replicated. But if we just let the server_id of transactions on master A be a >> hash of the affected database, then MySQL 5.6 style parallel replication just >> falls out of multi-master for tree, without any of their complexity. > I have not ever seen this topology arise in practice within MySQL, as people > tend to keep things pretty homogeneous. It seems to arise mostly in data > warehouse loading where multiple applications load data into the same store > for reporting. These are more heterogeneous but arallel information from > MySQL is of limited value here--data warehouses in Vertica or Hadoop don't > have constraints and you can just jam data into them in parallel. Ok. >> It is just a matter of not artificially restricting transactions generated on >> a given server to have just one server_id value. Which is in fact exactly what >> you asked me about originally with SET SESSION SERVER_ID :) >> >> On the other hand, the slave is free to try to discover more opportunities for >> independence / parallelism if it can, and use them. But automatically >> discovering opportunities for parallelism on the slave is a very hard problem >> in general, as I am sure you agree ... > Meanwhile, I don't like the idea of using server_id in the way you describe > because it overloads the meaning. Server_id just tells us which server > actually committed the transaction in the first place. The reason we need > the session variable was to be able to signal this properly. I was not > intending to use it more broadly. Ok, I've noted your concern. It would be interesting to learn how you use the knowledge that one mysqld instance can have at most one server id in generated transactions? > More generally, I believe that if you want to allow programmers to declare > that things are independent it would be best to add SQL-level features. For > instance, Facebook has a patch that prevents transactions from spanning > schemas. This is a very useful feature, because it ensures that all updates > in one schema are independent from others. That in turn means you can > parallelize without being afraid of deadlocks (a big problem in this space) > or constraint violations. > > You could generalize this into the notion of a shard, which is a scope for > independent transactions and presumably also referential integrity. This > would be a very useful feature for SQL databases. I don't know if you have > read it but Pat Helland has a great paper on this topic called "Life beyond > Distributed Transactions: an Apostate’s Opinion." It was visionary at the > time (and still is) and has never been properly implemented in relational > DBMS to my knowledge. Well, independent schemas are nice, but not enough. There are many applications that need the slave to apply transactions in parallel even within a single table. - Kristian.

1 0

[Maria-developers] Review/help needed with MDEV-532
by Kristian Nielsen 14 Sep '12

14 Sep '12

Hi Serg! I've pushed the MDEV-181/MDEV-225/MDEV-232 to 10.0-base, thanks for review! As you remember, we discussed extending it with a separate patch for doing the checkpoint inside InnoDB in an asynchroneous manner. I have created MDEV-532 for this. It was not hard to create the patch, and it seems to work. But I could use some help with one last test failure that I am somewhat mystified by: rpl.rpl_binlog_index 'row' w7 [ fail ] Found warnings/errors in server log file! Test ended at 2012-09-13 15:44:01 line 120913 15:43:48 [ERROR] mysqld: Table './test/t1' is marked as crashed and should be repaired 120913 15:43:48 [Warning] Checking table: './test/t1' 120913 15:43:48 [ERROR] test.t1: 1 client is using or hasn't closed the table properly I assume this is related to how I create a background thread for the binlog, something that I do wrong or fail to do, so any suggestions on where to look would be appreciated. I have pushed the code here: lp:~maria-captains/maria/10.0-knielsen - Kristian.

1 1

[Maria-developers] New to MariaDB
by Giacomo Picchiarelli 13 Sep '12

13 Sep '12

Hi, I'm new to MariaDB and I'm reading the knowledge base. I'm interested in studying Aria Engine (and contribute, if I can :) ), are there tasks for newbie developers? Thank you. Regards. -- Giacomo Picchiarelli <gpicchiarelli(a)gmail.com> Skype: g.picchiarelli

2 1

Re: [Maria-developers] IDE debugging Mariadb linux build
by md＠rpzdesign.com 10 Sep '12

10 Sep '12

Timour: I respect your technical position, but I do not get my paychecks from monty. My total budgeted time to deal with "issues" related to Aria is exactly zero, so I am not going to try to ramp up learning hardware GDB breakpoints, etc. My broadcast was to enable the many other non-montyprogram developers a chance to load the project into the IDE/debugger so they can tweak or modify something and then contribute back into the community. They likely won't contribute if the only thing they can use is VI and GDB command line. Fortunately, Monty has provided good tested code over the years and that is why Maria will succeed, I do not have to do everything myself, but when I run into a problem, I have a chance at fixing it. Netbeans makes it easy to use the CmakeLists.txt file to load the project. Basically, open source gives me all the power and not ALL the responsibility, bloody magic really. Keep up the good work, marco On 9/10/2012 1:26 AM, Timour Katchaounov wrote: > Hi Marco, > >> I can report to all the developers for MariaDB on linux >> that Netbeans 7.1.2 is working VERY well on Fedora 14 - 64 bit. > > Netbeans has worked well for MySQL already few years ago, I tried it. > The main problem for me with both Netbeans and Eclipse was that both > have limited GDB support. I couldn't find a way to set hardware > breakpoints, neither to inspect easily arrays. AFAIR it was not possible > to evaluate expressions within the debugger, not sure about conditional > breakpoints. > > All the above features are indispensable when working on server bugs, or > analyzing how things work. If you have found a way to overcome these > limitations, then these IDEs might be useful for server development. > > > Timour > > >> >> Vladislav indicated he used only Windows Visual studio and OSX Xcode. >> >> It has a great Find It Grep support and the debugger is easy to set >> breakpoints and run the code. >> >> Marco >> >> On 9/7/2012 3:55 AM, Vladislav Vaintroub wrote: >>> >>>> -----Original Message----- >>>> From: maria-developers- >>>> bounces+wlad=montyprogram.com(a)lists.launchpad.net [mailto:maria- >>>> developers-bounces+wlad=montyprogram.com(a)lists.launchpad.net] On >>>> Behalf Of md(a)rpzdesign.com >>>> Sent: Freitag, 7. September 2012 03:00 >>>> To: maria-developers(a)lists.launchpad.net >>>> Subject: [Maria-developers] IDE debugging Mariadb linux build >>>> >>>> I am using cmake -G"CodeBlocks - Unix Makefiles" to load the 5.5.25 >>>> project into CodeBlocks for building and editing. >>>> >>>> Does anybody use KDevelop4 or Codeblocks to build mariadb? >>> I used KDevelop some time ago (half a year or so), and Codeblocks >>> long-long >>> ago (still with vanilla MySQL) >>> >>>> If so, how do you set the parameters up so you can run the server in >>>> gdb debug? >>> Debugging and building are slightly different things. CMake does not >>> have >>> builtin support for IDE-debugging, so you need to use a semi-manual >>> method >>> . >>> >>> I found the below method useful >>> 1. Switch to build directory >>> 2. cd mysql-test >>> 3. perl mysql-test-run.pl --start 1st --manual-debug >>> >>> This will create database directory, config file, and write path for >>> mysqld >>> executable together working directory and args to the console. You can >>> copy/paste it into IDE, where debug parameters for executable are >>> stored (I >>> do not know btw where KDevelop4 has parameters for debugging, I just >>> describe the process I used in Visual Studio and Xcode ) >>> >>>> Is there a way to run mysqld without mysql_safe monitor daemon in >>>> front >>>> of it? >>> Yes, for example the one above. >>> >>> >>> >>> >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers(a)lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp >

1 0

Re: [Maria-developers] [knielsen@knielsen-hq.org] [Commits] Rev 3459: MDEV-232: Remove one fsync() from commit phase. in http://bazaar.launchpad.net/~maria-captains/maria/5.5
by Kristian Nielsen 09 Sep '12

09 Sep '12

Sergei Golubchik <serg(a)askmonty.org> writes: > Hi, Kristian! > > On Jun 22, Kristian Nielsen wrote: >> Can you review below patch for mdev-232 (At your convenience, not urgent)? > > Here's the review. Mostly questions, I have to confess :) Thanks a lot for your review! I've tried to answer all your questions below, and for many I have tried to clarify the patch, as noted. Do you want to see the changes for when I implement async commit_checkpoint_request() in InnoDB/XtraDB, before I push it to 10.0? >> === modified file 'sql/slave.cc' >> --- a/sql/slave.cc 2012-06-22 09:40:40 +0000 >> +++ b/sql/slave.cc 2012-06-22 10:20:29 +0000 >> @@ -1751,13 +1751,12 @@ when it try to get the value of TIME_ZON >> /* Announce MariaDB slave capabilities. */ >> DBUG_EXECUTE_IF("simulate_slave_capability_none", goto after_set_capability;); >> { >> - const char *q= >> - DBUG_EVALUATE_IF("simulate_slave_capability_old_53", >> - "SET @mariadb_slave_capability=" >> - STRINGIFY_ARG(MARIA_SLAVE_CAPABILITY_ANNOTATE), >> - "SET @mariadb_slave_capability=" >> - STRINGIFY_ARG(MARIA_SLAVE_CAPABILITY_MINE)); >> - if (mysql_real_query(mysql, q, strlen(q))) >> + int rc= DBUG_EVALUATE_IF("simulate_slave_capability_old_53", >> + mysql_real_query(mysql, STRING_WITH_LEN("SET @mariadb_slave_capability=" >> + STRINGIFY_ARG(MARIA_SLAVE_CAPABILITY_ANNOTATE))), >> + mysql_real_query(mysql, STRING_WITH_LEN("SET @mariadb_slave_capability=" >> + STRINGIFY_ARG(MARIA_SLAVE_CAPABILITY_MINE)))); >> + if (rc) > > Why? Monty asked me to not use strlen() which I had introduced due to a request from you in review of an earlier worklog. And you wrote that you did not care much one way or the other (neither do I), so I reverted the change back to the original without strlen(). (Maybe I should have done it as a separate changeset, but I took this short cut since MDEV 225/181/232 are kind of tied together and will likely be pushed together.) >> === modified file 'storage/xtradb/handler/ha_innodb.cc' >> --- a/storage/xtradb/handler/ha_innodb.cc 2012-06-15 12:54:23 +0000 >> +++ b/storage/xtradb/handler/ha_innodb.cc 2012-06-22 10:20:29 +0000 >> @@ -485,8 +486,11 @@ static MYSQL_THDVAR_ULONG(lock_wait_time >> static MYSQL_THDVAR_ULONG(flush_log_at_trx_commit, PLUGIN_VAR_OPCMDARG, >> "Set to 0 (write and flush once per second)," >> " 1 (write and flush at each commit)" >> - " or 2 (write at commit, flush once per second).", >> - NULL, NULL, 1, 0, 2, 0); >> + " or 2 (write at commit, flush once per second)." >> + " Can also be set to 3 to write and flush at each commit, even when this is" >> + " unnecessary because of binlog crash recovery; should not normally be used" >> + " due to performance degration, but provided for completeness.", > > would you like to improve the description on =1 value? For example, > 1 (write and flush on prepare or on commit) > 3 (write and flush both on prepare and on commit, usually it is redundant) Yes, good idea. Here is my suggestion: "Controls the durability/speed trade-off for commits." " Set to 0 (write and flush redo log to disk only once per second)," " 1 (flush to disk at each commit)," " 2 (write to kernel at commit but flush to disk only once per second)" " or 3 (flush to disk twice per commit, usually redundant)." " 1 and 3 guarantees that after a crash, committed transactions will" " not be lost and will be consistent with the binlog and other transactional" " engines. 2 can get inconsistent and lose transactions if there is a" " power failure or kernel crash but not if mysqld crashes. 0 has no" " guarantees in case of crash. 0 and 2 can be faster than 1 or 3.", >> /*****************************************************************//** >> +Handle a commit checkpoint request from server layer. >> +We simply flush the redo log immediately and do the notify call.*/ >> +static >> +void >> +innobase_checkpoint_request( >> + handlerton *hton, >> + void *cookie) >> +{ >> + log_buffer_flush_to_disk(); >> + ha_commit_checkpoint_notify(hton, cookie); >> +} >> + >> +/*****************************************************************//** > > What about making innobase_checkpoint_request() implementation that flushed > in the background? At least to serve as an example of the correct usage of your > API, if not for performance. And to test your code for cases when the notification > isn't coming directly from the checkpoint_request method. > > it looks like it's very easy to make innodb to invoke ha_commit_checkpoint_notify > from the background flusher thread (the one that syncs once a second). > you only need to store the cookie in the log_sys, and invoke > ha_commit_checkpoint_notify after the flush when this stored cookie isn't NULL. Yes, I would like to do this, I will try. It might even help performance, I have seen several blog posts mentioning problems with stalls at binlog rotation, and this might help. As you say, this should be easy. I need to store a list of cookies and their associated LSN (not just a single cookie), but this is infrequently executed so just a simple linked list with malloc/free of elements will work fine. >> +void >> +ha_commit_checkpoint_notify(handlerton *hton, void *cookie) > > are there other ha_xxx methods that a storage engine calls? > > typically server calls methods, like, ha_commit_trans, and they invoke storage > engine methods. Not the other way around. It is exactly for this reason I used names > like "trans_register_ha" - inverse of ha_register_trans - to emphasize that this > method is invoked by the engine to call into the server, not by the server to > call into the engine. Ah, thanks, good point. I suggest "commit_checkpoint_notify_ha()", what do you think? >> === modified file 'sql/log.h' >> --- a/sql/log.h 2012-06-22 09:46:28 +0000 >> +++ b/sql/log.h 2012-06-22 10:20:29 +0000 >> @@ -359,9 +373,14 @@ class MYSQL_QUERY_LOG: public MYSQL_LOG >> Ids start from BINLOG_COOKIE_START; the value BINLOG_COOKIE_DUMMY is special >> meaning "no binlog" (we cannot use zero as that is reserved for error return >> from log_and_order). >> + The low-order bit is reserved for an error flag. >> */ >> -#define BINLOG_COOKIE_DUMMY 1 >> -#define BINLOG_COOKIE_START 2 >> +#define BINLOG_COOKIE_ERROR_FLAG 1 >> +#define BINLOG_COOKIE_ID_INCREMENT (BINLOG_COOKIE_ERROR_FLAG*2) >> +#define BINLOG_COOKIE_DUMMY (BINLOG_COOKIE_ID_INCREMENT*1) > > Is that right? BINLOG_COOKIE_DUMMY == BINLOG_COOKIE_ID_INCREMENT ? It is "right", but I agree it is incomprehensible. I will improve it. The thing is, I need to store a number of different values in the "cookie" used with log_and_order() and unlog(): - 0, which is the error return value from log_and_order() - A pair (binlog_id, error_flag) - A special dummy binlog_id meaning that nothing was logged to the binlog. I will change it to this, what do you think? : /* We assign each binlog file an internal ID, used to identify them for unlog(). The IDs start from 0 and increment for each new binlog created. In unlog() we need to know the ID of the binlog file that the corresponding transaction was written into. We also need a special value for a corner case where there is no corresponding binlog id (since nothing was logged), as well as an error flag to mark that unlog() must return failure. We use the following macros to pack all of this information into the single ulong available with log_and_order() / unlog(). Note that we cannot use the value 0, as that is reserved as error return value from log_and_order(). */ #define BINLOG_COOKIE_ERROR_RETURN 0 #define BINLOG_COOKIE_DUMMY_ID 1 #define BINLOG_COOKIE_BASE 2 #define BINLOG_COOKIE_DUMMY(error_flag) \ ( (BINLOG_COOKIE_DUMMY_ID<<1) | ((error_flag)&1) ) #define BINLOG_COOKIE_MAKE(id, error_flag) \ ( (((id)+BINLOG_COOKIE_BASE)<<1) | ((error_flag)&1) ) #define BINLOG_COOKIE_GET_ERROR_FLAG(c) ((c) & 1) #define BINLOG_COOKIE_GET_ID(c) ( ((ulong)(c)>>1) - BINLOG_COOKIE_BASE ) #define BINLOG_COOKIE_IS_DUMMY(c) \ ( ((ulong)(c)>>1) == BINLOG_COOKIE_DUMMY_ID ) Then the code in log.cc can use these macros, and not have to depend on strange details on how to pack different values into a single ulong. I think the resulting code is much clearer, now BINLOG_COOKIE_* is only used in log_and_order() and unlog() functions, thanks! >> class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG >> @@ -464,15 +492,6 @@ class MYSQL_BIN_LOG: public TC_LOG, priv >> uint file_id; >> uint open_count; // For replication >> int readers_count; >> - bool need_start_event; >> - /* >> - no_auto_events means we don't want any of these automatic events : >> - Start/Rotate/Stop. That is, in 4.x when we rotate a relay log, we don't >> - want a Rotate_log event to be written to the relay log. When we start a >> - relay log etc. So in 4.x this is 1 for relay logs, 0 for binlogs. >> - In 5.0 it's 0 for relay logs too! >> - */ >> - bool no_auto_events; > > Why? Because it's some old code which isn't used nowadays? Yes. This was dead code. >> === modified file 'sql/log.cc' >> --- a/sql/log.cc 2012-06-22 09:46:28 +0000 >> +++ b/sql/log.cc 2012-06-22 10:20:29 +0000 >> @@ -3597,12 +3623,40 @@ bool MYSQL_BIN_LOG::reset_logs(THD* thd) >> + /* >> We are going to nuke all binary log files. >> - So first wait until all pending binlog checkpoints have completed. >> + Without binlog, we cannot XA recover prepared-but-not-committed >> + transactions in engines. So force a commit checkpoint first. >> + >> + Note that we take and immediately release LOCK_commit_ordered. This has >> + the effect to ensure that any on-going group commit (in >> + trx_group_commit_leader()) has completed before we request the checkpoint, >> + due to the chaining of LOCK_log and LOCK_commit_ordered in that function. >> + (We are holding LOCK_log, so no new group commit can start). >> + >> + Without this, it is possible (though perhaps unlikely) that the RESET >> + MASTER could run in-between the write to the binlog and the >> + commit_ordered() in the engine of some transaction, and then a crash >> + later would leave such transaction not recoverable. >> */ >> + mysql_mutex_lock(&LOCK_commit_ordered); >> + mysql_mutex_unlock(&LOCK_commit_ordered); >> + >> + mark_xids_active(current_binlog_id, 1); >> + do_checkpoint_request(current_binlog_id); > > why do_checkpoint_request ? There may be transactions in the current binlog file that are not yet flushed to disk in the engine. If we proceed in RESET MASTER to delete the binlog file, and then crash, then we could lose such transactions. So we request a checkpoint for the current binlog file (we already did so for any previous binlog files). And then wait for this and any other pending checkpoint to complete. Only then can we safely delete binlog files. >> @@ -3731,6 +3780,31 @@ bool MYSQL_BIN_LOG::reset_logs(THD* thd) >> + /* >> + Remove all entries in the xid_count list except the last. >> + Normally we will just be deleting all the entries that we waited for to >> + drop to zero above. But if we fail during RESET MASTER for some reason >> + then we will not have created any new log file, and we may keep the last >> + of the old entries. > > I'm not sure I understand this comment. > What's the point of keeping the last of the old entries, if old binlog > files are deleted and binary logging is probably stopped (when RESET MASTER fails > it's kinda serious)? The code in sql/log.cc assumes that binlog_xid_count_list is never empty. So even in the error case we must be sure to leave at least one entry in there. So the comment is to explain why we do not simply delete everything in the list at the start, and then insert the new entry on top of that. If we get an error half-way, error handling is just easier by doing the deletion at the end, where we know what we can delete and what not. It is true that RESET MASTER failure is quite serious, and if we cannot open a new binlog after deleting the old ones we are indeed in trouble. And there are very likely legacy bugs lurking in this error case. But with this code, at least no new error handling issues are introduced, and the rest of the code is clean and can rely on having at least one entry in the list. > If some engine confirms that everything below a certain cookie of > a particular binlog_id is flushed. Does it automatically mean that > all previous binlog_id's are flushed too in this engine? Yes. This is the relevant comment in handler.h: The TC will periodically (eg. once per binlog rotation) call commit_checkpoint_request(). When this happens, the engine must arrange for all transaction that have completed commit_ordered() to be durably flushed to disk (this does not include transactions that might be in the middle of executing commit_ordered()). When such flush has completed, the engine must call commit_checkpoint_notify_ha(), passing back the opaque "cookie". commit_ordered() calls complete in the same order as transactions are written in the binlog. And we request a checkpoint for a binlog file only when all transactions in it have completed commit_ordered(). So when an engine has flushed all transactions that completed commit_ordered(), this includes everything in the given binlog, as well as anything in previous binlogs. >> @@ -5493,7 +5656,27 @@ int MYSQL_BIN_LOG::rotate(bool force_rot >> if (!write_incident_already_locked(current_thd)) >> flush_and_sync(0); >> >> - *check_purge= true; >> + /* >> + We failed to rotate - so we have to decrement the xid_count back that >> + we incremented before attempting the rotate. >> + */ > > why you don't use mark_xid_done() here? We cannot call mark_xid_done() here, as we are holding LOCK_log - and mark_xid_done() also takes (sometimes) LOCK_log. >> @@ -5920,22 +6115,29 @@ MYSQL_BIN_LOG::write_binlog_checkpoint_e >> */ >> if (!ev.write(&log_file) && !flush_and_sync(0)) >> { >> - bool check_purge= false; >> signal_update(); >> - rotate(false, &check_purge); >> - if (check_purge) >> - purge(); > > Why did you remove rotate/purge from here? Binlog rotation triggers the writing of checkpoint event in the first place. So it does not seem right that writing checkpoint event can trigger binlog rotation. This could in principle lead to infinite rotation of new binlogs, though one would need a _very_ small max binlog size to trigger that. >> +int TC_LOG_MMAP::unlog(ulong cookie, my_xid xid) >> +{ >> + pending_cookies *full_buffer= NULL; >> + DBUG_ASSERT(*(my_xid *)(data+cookie) == xid); >> + >> + /* >> + Do not delete the entry immediately, as there may be participating storage >> + engines which implement commit_checkpoint_request(), and thus have not yet >> + flushed the commit durably to disk. >> + >> + Instead put it in a queue - and periodically, we will request a checkpoint >> + from all engines and delete a whole batch at once. >> + */ >> + mysql_mutex_lock(&LOCK_pending_checkpoint); >> + if (pending_checkpoint == NULL) >> + { >> + uint32 size= sizeof(*pending_checkpoint); >> + if (!(pending_checkpoint= >> + (pending_cookies *)my_malloc(size, MYF(MY_ZEROFILL)))) >> + { >> + my_error(ER_OUTOFMEMORY, MYF(0), size); >> + mysql_mutex_unlock(&LOCK_pending_checkpoint); >> + return 1; >> + } >> + pending_checkpoint->cookies[pending_checkpoint->count++]= cookie; > > Hm, should this be under if (pending_checkpoint == NULL) ? No :-( That seems completely, utterly broken :-( So this means that I must have not tested this code at all. Which makes me sad. Of course, with PBXT dead, I am not sure how it is even possible to test it... ? Thanks for spotting. >> + /* >> + We do an extra increment and notify here - this ensures that >> + things work also if there are no engines at all that support >> + commit_checkpoint_request. >> + */ >> + ++full_buffer->pending_count; >> + ha_commit_checkpoint_request(full_buffer, mmap_do_checkpoint_callback); >> + commit_checkpoint_notify(full_buffer); > > I'm sorry, I don't understand how you handle the case when some engines don't > support commit_checkpoint_request If an engine supports commit_checkpoint_request, then ha_commit_checkpoint_request() will ask the engine to do a checkpoint; and pending_count will be incremented for each such engine to ensure that we wait for each engine to finish the checkpoint before we delete any entries from the on-disk log. If an engine does not support commit_checkpoint_request, then we will not do anything with that engine before deleting entries. But such engines are required to make their transactions durable already in their commit() method, so deletion is always safe. If _no_ engines support ha_commit_checkpoint_request then we would not get any callback to commit_checkpoint_notify() - so we add one extra such a call ourselves, at the end. Also, the increment of pending_count and decrement in the extra commit_checkpoint_notify() call protects against this race: - We call engine A's commit_checkpoint_request() - Before we have time to call engine B's commit_checkpoint_request(), engine A does commit_checkpoint_notify(), so pending_count drops to zero and we delete entries even though B has not had a chance to do a checkpoint yet - We call engine B's commit_checkpoint_request with a void *cookie pointing to freed memory. Hope this clarifies things. >> @@ -7869,6 +8196,11 @@ int TC_LOG_BINLOG::recover(LOG_INFO *lin >> if (!binlog_checkpoint_found) >> break; >> first_round= false; >> + DBUG_EXECUTE_IF("xa_recover_expect_master_bin_000004", >> + if (0 != strcmp("./master-bin.000004", binlog_checkpoint_name) && >> + 0 != strcmp(".\\master-bin.000004", binlog_checkpoint_name)) > > Are you sure the name is always lowercase even on case-insensitive > filesystems? I'd use strcasecmp to avoid the need to think about it :) > Something like > > if (0 != strcasecmp("master-bin.000004", binlog_checkpoint_name+2)) > > (right, unsafe if strlen(binlog_checkpoint_name) < 2, but it's just a test case) I don't understand - this has nothing to do with file system case sensitivity. The string is generated inside log.cc based on what mysql-test-run.pl passes to mysqld for the --log-bin option, I don't see how this could change case. And I thought it was clearer to explicitly show the two possible cases for the full file name (otherwise the offset 2 in the comparison is completely magic). But as you said, this is just a test case, I'm happy to change it if you insists (I just did not understand the reason to change it). BTW, as I remember this was for something that was quite hard to test in mysql-test-run. Since recovery happens early in the server startup, before anything mysqltest can do. So I could not think of anything better than to inject an error insert with a configuration option. - Kristian.

4 9

[Maria-developers] MariaDB 5.5.27 Release Prep
by Daniel Bartholomew 07 Sep '12

07 Sep '12

All, I've prepped the release for MariaDB 5.5.27. We're now waiting for the mirrors to seed the files before turning on the downloads page and sending out announcements. Missing at the moment are our Windows builds because of an issue with the Windows build server. Hopefully those will be ready soon. Draft Release Notes and Changelog are located here: http://kb.askmonty.org/en/mariadb-5527-release-notes/ http://kb.askmonty.org/en/mariadb-5527-changelog/ If there is anything of particular note which should be mentioned in the release notes, please let me know. Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

[Maria-developers] EXPLAIN ANALYZE
by Sergei Petrunia 03 Sep '12

03 Sep '12

Hello Igor, I've got EXPLAIN ANALYZE to work for the first queries, see details at http://kb.askmonty.org/en/explain-analyze/ I need input re whether the shown information is what you needed for 10.0 works. Also, we need to decide what should be shown in these columns for join buffering/subqueries (let's discuss this on the optimizer call tomorrow). BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MDEV-406: EXPLAIN ANALYZE or a different syntax?
by sergii＠pisem.net 03 Sep '12

03 Sep '12

Hi. MDEV-406. The chosen syntax is "EXPLAIN ANALYZE". Pro: 1. PostgreSQL uses this syntax. Contra: 1. Full name is EXPLAIN ANALYZE SELECT. But statements are at most two words in MySQL: SHOW PROCESSLIST, CREATE TABLE, etc. Even the task description talks about EXPLAIN ANALYZE. Having EXPLAIN SELECT and (in the future) EXPLAIN UPDATE and EXPLAIN DELETE, EXPLAIN ANALYZE looks like it's EXPLAIN ANALYZE TABLE. But it is not. 2. In SQL the stament is a verb. That's the spirit of the standard. SELECT something. INSERT something. CREATE something, SHOW... We've deviated from it with the HANDLER command, unfortunately. Still, EXPLAIN ANALYZE SELECT is very unnatural English. EXPLAIN SELECT is better, because "SELECT" is perceived as a noun, as "SELECT statement". So EXPLAIN SELECT means "explain the SELECT statement". But EXPLAIN ANALYZE SELECT just doesn't map into English at all. Possible variants (keep in mind that instead of SELECT one might later have UPDATE and DELETE): EXPLAIN ANALYZED SELECT ... ANALYZE SELECT .... The last syntax is stragely consistent with EXPLAIN: We used to have both EXPLAIN table_name and EXPLAIN SELECT ... So, extending ANALYZE TABLE table_name to ANALYZE SELECT ... follows this weird historical pattern. (which is, of course, no argument, just a funny coincidence) Regards, Sergei

1 0

[Maria-developers] Please test: Cassandra batched-inserts, Batched Key Access
by Sergei Petrunia 31 Aug '12

31 Aug '12

Hi Elena, I've pushed "initial" support for batched INSERT and BKA into 5.5-cassandra, and updated http://kb.askmonty.org/en/cassandra-storage-engine to cover batching. Could you please test it? Cassandra uses "hashed" mode with BKA (see docs), which haven't been used for a while (previous user was Falcon storage engine), so it is possible that something malfunctions in BKA. Names/meaning of control variables and status counter names are preliminary and may change in the future. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] [Commits] Rev 3565: MDEV-492: fixed incorrect error check. in file:///home/bell/maria/bzr/work-maria-5.3-MDEV-492/
by Sergei Golubchik 29 Aug '12

29 Aug '12

Hi, Oleksandr! On Aug 29, Oleksandr Byelkin wrote: > >> === modified file 'sql/sql_update.cc' > >> --- a/sql/sql_update.cc 2012-04-26 17:21:37 +0000 > >> +++ b/sql/sql_update.cc 2012-08-27 14:29:26 +0000 > >> @@ -872,7 +872,7 @@ int mysql_update(THD *thd, > >> id= thd->arg_of_last_insert_id_function ? > >> thd->first_successful_insert_id_in_prev_stmt : 0; > >> > >> - if (error < 0) > >> + if (error < 0 && (!thd->is_error())) > >> { > >> char buff[MYSQL_ERRMSG_SIZE]; > >> my_snprintf(buff, sizeof(buff), ER(ER_UPDATE_INFO), (ulong) found, (ulong) updated, > > Nope, this is not good. > > The bug happens because the error from remove_eq_conds() is ignored. > > You need to catch and process this error as soon as possible. > > While in your patch you let the update work all the way till the end. > > > When "update works all the way" it catched inside update loops, but was > that if update never make any update it ignores errors. > > So actually it should be my fix or both fixes, because errors > (especially such unexpected as "out of resources" could happened almost > anywhere, so issuing OK to the client without checking error state is a > mistake. I know. Before sending a review I actually tried to get a crash by generating an error in different places of the UPDATE. E.g. in the loop: INSERT t1 VALUES (1,2); UPDATE .... or in fill_record(): UPDATE t1 SET a=COLUMN_CREATE(...); or with double my_error(): ... WHERE COLUMN_CREATE(...) OR COLUMN_CREATE(...); all these cases were correctly taken care of, no crash, no ok packet. So, I think, it's enough to fix only that early failure that you have found. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3565: MDEV-492: fixed incorrect error check. in file:///home/bell/maria/bzr/work-maria-5.3-MDEV-492/
by Sergei Golubchik 28 Aug '12

28 Aug '12

Hi, Sanja! On Aug 27, sanja(a)askmonty.org wrote: > message: > MDEV-492: fixed incorrect error check. > === modified file 'mysql-test/t/errors.test' > --- a/mysql-test/t/errors.test 2012-03-12 07:56:56 +0000 > +++ b/mysql-test/t/errors.test 2012-08-27 14:29:26 +0000 > @@ -85,3 +85,12 @@ INSERT INTO t2 VALUES (1,0) ON DUPLICATE > INSERT INTO t2(a,b) VALUES (1,0) ON DUPLICATE KEY UPDATE > b=(SELECT VALUES(a)+2 FROM t1); > DROP TABLE t1, t2; > + > +--echo # > +--echo # MDEV-492: incorrect error check before sending OK in mysql_update > +--echo # > +CREATE TABLE t1 (a CHAR(3), b BLOB); > +--error ER_DYN_COL_DATA > +UPDATE t1 SET a = 'new' > +WHERE COLUMN_CREATE( 1, 'v', 1, 'w' ) IS NULL; > +drop table t1; > > === modified file 'sql/sql_update.cc' > --- a/sql/sql_update.cc 2012-04-26 17:21:37 +0000 > +++ b/sql/sql_update.cc 2012-08-27 14:29:26 +0000 > @@ -872,7 +872,7 @@ int mysql_update(THD *thd, > id= thd->arg_of_last_insert_id_function ? > thd->first_successful_insert_id_in_prev_stmt : 0; > > - if (error < 0) > + if (error < 0 && (!thd->is_error())) > { > char buff[MYSQL_ERRMSG_SIZE]; > my_snprintf(buff, sizeof(buff), ER(ER_UPDATE_INFO), (ulong) found, (ulong) updated, Nope, this is not good. The bug happens because the error from remove_eq_conds() is ignored. You need to catch and process this error as soon as possible. While in your patch you let the update work all the way till the end. This patch is better, it aborts much earlier: === modified file 'sql/sql_update.cc' --- sql/sql_update.cc 2012-08-09 15:22:00 +0000 +++ sql/sql_update.cc 2012-08-28 19:05:33 +0000 @@ -406,7 +406,7 @@ int mysql_update(THD *thd, table->file->info(HA_STATUS_VARIABLE | HA_STATUS_NO_LOCK); select= make_select(table, 0, 0, conds, 0, &error); - if (error || !limit || + if (error || !limit || thd->is_error() || (select && select->check_quick(thd, safe_update, limit))) { delete select; Regards, Sergei

1 0

Re: [Maria-developers] [Commits] Rev 3353: Performed re-factoring and re-structuring of the code for mwl#248: in file:///home/igor/maria/maria-5.5-mwl248-changes/
by Sergei Golubchik 28 Aug '12

28 Aug '12

Hi, Davi! On Jul 31, Davi Arnaut wrote: > Also, I would like to point out that my main argument a few e-mails > ago is that you cannot memset/bzero a non-POD type in C++ without > occurring in undefined/implementation-defined behavior, which was > ignored. This significantly lowers my confidence in MariaDB being a > high quality code base (save for some developers who tend to get it). :) After your first email I've checked that code. Perhaps I've missed something, but it did look like all bzero-ed structures in the igor's patch were of POD types. Regards, Sergei

2 3

[Maria-developers] MariaDB 5.3.8 Release Prep
by Daniel Bartholomew 28 Aug '12

28 Aug '12

All, The MariaDB 5.3.8 release is almost ready. The files have been uploaded and should be mirrored soon. I expect to send out the release announcement at around 10am U.S. Eastern time tomorrow (28 Aug 2012). As usual, I've created preliminary Release Notes and Changelog pages: http://kb.askmonty.org/en/mariadb-538-release-notes/ http://kb.askmonty.org/en/mariadb-538-changelog/ Please let me know ASAP if there are any particularly noteworthy items in this release which I should mention in the Release Notes. Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

[Maria-developers] table_share->field vs table->field
by Markus Pilman 22 Aug '12

22 Aug '12

Hi all, I ran into a strange problem: I am working on a storage engine for MariaDB. To access field information I usually do as written in the documentation: for (Field** field = table->field; *field; ++field) { /* some code here */ } This works fine in most cases (for writing - read handler::write_row - it seems to work always) but sometimes (I think always in the index_next method) accessing **(table->field) results in a segmentation fault (but *(table->field) is not null, but something like 0x5e6f4c6d4578244d). Strange enough, this error does not happen always. In the debugger I saw, that table_share->field points to the correct location there so I replaces table->field in the above loop with table_share->field. But this also seems to not work - it now crashes on other queries (also on updates, but after restarting MariaDB it also crashes on a full table scan). So my question: how can I access the fields in a reliable manner? I need to do that to set/get the BLOB/TEXT fields. We have a method in our handler called set_blobs that resets the pointer in the record to the blobs, which looks like the following: void ha_kvstore::set_blobs(uchar *buf, kvid key, btree::buffer &raw_data) { auto current_read_pos = raw_data.const_ptr() + table_share->reclength; for (Field** field = table->field; *field; ++field) { if (!((*field)->flags & BLOB_FLAG)) continue; (*field)->table = table; Field_blob* blob = *((Field_blob**)field); uint32 packlength = blob->pack_length_no_ptr(); bmove((*field)->ptr + packlength, &current_read_pos, sizeof(char*)); current_read_pos += blob->get_length(); } } The storage engine always crashes in this code. What are we doing wrong? Any help is appreciated! Best Markus

2 2

[Maria-developers] Introduction / Os X Build Host
by Pablito 21 Aug '12

21 Aug '12

Hello I'm Pablito Vicente, I'm from Argentina, have been using Mysql for 12 years now as a web developer ( PHP mostly ). During the weekend I was trying to download a package for Mac and I notice brew was the only choice, that I used and worked but I would like to know if I can help generating builds for the latest versions and I will try to also investigate how to create packages so people can download a DMG image for easier installation. I will follow the instructions on: https://kb.askmonty.org/en/buildbot-setup-buildbot-setup-for-macosx/ to set it up but I was wondering if this will work when the computers are turned on only for this task. I don't usually leave computers on unless that's required. I could offer builds for Mountain Lion and Snow Leopard, though I think they're probably the same. well let me know if this can be of any help. I might send some emails for some help as I've been away from sys admin tasks for years. warm regards, Pablito

1 0

Re: [Maria-developers] Fwd: [Commits] Rev 3498: Fix bug mdev-447: Wrong output from the EXPLAIN command of the test case for lp bug #714999 in file:///home/tsk/mprog/src/5.5-md447/
by Sergei Petrunia 21 Aug '12

21 Aug '12

Hi Timour Ok to push. On Mon, Aug 20, 2012 at 02:47:35PM +0300, Timour Katchaounov wrote: > Sergey, > > Could you please review this simple patch. > It is a backport of several lines from your code. > > Timour > > ------------------------------------------------------------ > revno: 3498 > revision-id: timour(a)askmonty.org-20120820114533-hznbt0z7ikdxoawq > parent: knielsen(a)knielsen-hq.org-20120817123528-eggusbbjukhxshvs > committer: timour(a)askmonty.org > branch nick: 5.5-md447 > timestamp: Mon 2012-08-20 14:45:33 +0300 > message: > Fix bug mdev-447: Wrong output from the EXPLAIN command of the test case for lp bug #714999 > > The fix backports from MWL#182: Explain running statements the logic that > saves the original JOIN_TAB array of a query plan after optimization. This > array is later used during EXPLAIN to iterate over the original JOIN plan > nodes in the cases when this plan could be changed by early subquery > execution during the optimization phase of the outer query. > > > === modified file 'mysql-test/r/subselect.result' > --- a/mysql-test/r/subselect.result 2012-07-25 16:41:48 +0000 > +++ b/mysql-test/r/subselect.result 2012-08-20 11:45:33 +0000 > @@ -4196,7 +4196,7 @@ INSERT INTO t1 VALUES (1,1),(2,1); > EXPLAIN SELECT 1 FROM t1 WHERE a = (SELECT COUNT(*) FROM t1 GROUP BY b); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ref a a 5 const 1 Using where; Using index > -2 SUBQUERY internal_tmp_table ALL group_key NULL NULL NULL 1 Using temporary; Using filesort > +2 SUBQUERY t1 ALL NULL NULL NULL NULL 2 Using temporary; Using filesort > DROP TABLE t1; > CREATE TABLE t1 (id int NOT NULL, st CHAR(2), INDEX idx(id)); > INSERT INTO t1 VALUES > > === modified file 'mysql-test/r/subselect_innodb.result' > --- a/mysql-test/r/subselect_innodb.result 2012-06-06 19:26:40 +0000 > +++ b/mysql-test/r/subselect_innodb.result 2012-08-20 11:45:33 +0000 > @@ -333,7 +333,7 @@ WHERE (SELECT DISTINCT b FROM t3) > 0); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 const PRIMARY PRIMARY 4 const 1 Using where; Using index > 2 SUBQUERY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE > -3 SUBQUERY internal_tmp_table ALL group_key NULL NULL NULL 0 Using temporary > +3 SUBQUERY t3 ALL NULL NULL NULL NULL 1 Using temporary > SELECT * > FROM t1 > WHERE t1.a = ( > @@ -386,7 +386,7 @@ select 1 from t1 where 1 like (select 1 > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 1 > 2 SUBQUERY t1 ALL NULL NULL NULL NULL 1 > -3 SUBQUERY internal_tmp_table ALL group_key NULL NULL NULL 1 Using temporary; Using filesort > +3 SUBQUERY t1 ALL NULL NULL NULL NULL 1 Using temporary; Using filesort > select 1 from t1 where 1 like (select 1 from t1 where 1 <=> (select 1 from t1 group by a1)); > 1 > 1 > > === modified file 'mysql-test/r/subselect_mat_cost_bugs.result' > --- a/mysql-test/r/subselect_mat_cost_bugs.result 2012-07-25 16:41:48 +0000 > +++ b/mysql-test/r/subselect_mat_cost_bugs.result 2012-08-20 11:45:33 +0000 > @@ -148,7 +148,7 @@ FROM t2 GROUP BY f1 > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE noticed after reading const tables > 2 SUBQUERY t1 system NULL NULL NULL NULL 1 > -3 SUBQUERY internal_tmp_table ALL group_key NULL NULL NULL 1 Using temporary; Using filesort > +3 SUBQUERY t2 ALL NULL NULL NULL NULL 2 Using temporary; Using filesort > drop table t1, t2, t3; > # > # LP BUG#715034 Item_sum_distinct::clear(): Assertion `tree != 0' failed > > === modified file 'mysql-test/r/subselect_no_mat.result' > --- a/mysql-test/r/subselect_no_mat.result 2012-07-25 16:41:48 +0000 > +++ b/mysql-test/r/subselect_no_mat.result 2012-08-20 11:45:33 +0000 > @@ -4200,7 +4200,7 @@ INSERT INTO t1 VALUES (1,1),(2,1); > EXPLAIN SELECT 1 FROM t1 WHERE a = (SELECT COUNT(*) FROM t1 GROUP BY b); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ref a a 5 const 1 Using where; Using index > -2 SUBQUERY internal_tmp_table ALL group_key NULL NULL NULL 1 Using temporary; Using filesort > +2 SUBQUERY t1 ALL NULL NULL NULL NULL 2 Using temporary; Using filesort > DROP TABLE t1; > CREATE TABLE t1 (id int NOT NULL, st CHAR(2), INDEX idx(id)); > INSERT INTO t1 VALUES > > === modified file 'mysql-test/r/subselect_no_opts.result' > --- a/mysql-test/r/subselect_no_opts.result 2012-07-25 16:41:48 +0000 > +++ b/mysql-test/r/subselect_no_opts.result 2012-08-20 11:45:33 +0000 > @@ -4196,7 +4196,7 @@ INSERT INTO t1 VALUES (1,1),(2,1); > EXPLAIN SELECT 1 FROM t1 WHERE a = (SELECT COUNT(*) FROM t1 GROUP BY b); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ref a a 5 const 1 Using where; Using index > -2 SUBQUERY internal_tmp_table ALL group_key NULL NULL NULL 1 Using temporary; Using filesort > +2 SUBQUERY t1 ALL NULL NULL NULL NULL 2 Using temporary; Using filesort > DROP TABLE t1; > CREATE TABLE t1 (id int NOT NULL, st CHAR(2), INDEX idx(id)); > INSERT INTO t1 VALUES > > === modified file 'mysql-test/r/subselect_no_scache.result' > --- a/mysql-test/r/subselect_no_scache.result 2012-07-25 16:41:48 +0000 > +++ b/mysql-test/r/subselect_no_scache.result 2012-08-20 11:45:33 +0000 > @@ -4202,7 +4202,7 @@ INSERT INTO t1 VALUES (1,1),(2,1); > EXPLAIN SELECT 1 FROM t1 WHERE a = (SELECT COUNT(*) FROM t1 GROUP BY b); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ref a a 5 const 1 Using where; Using index > -2 SUBQUERY internal_tmp_table ALL group_key NULL NULL NULL 1 Using temporary; Using filesort > +2 SUBQUERY t1 ALL NULL NULL NULL NULL 2 Using temporary; Using filesort > DROP TABLE t1; > CREATE TABLE t1 (id int NOT NULL, st CHAR(2), INDEX idx(id)); > INSERT INTO t1 VALUES > > === modified file 'mysql-test/r/subselect_no_semijoin.result' > --- a/mysql-test/r/subselect_no_semijoin.result 2012-07-25 16:41:48 +0000 > +++ b/mysql-test/r/subselect_no_semijoin.result 2012-08-20 11:45:33 +0000 > @@ -4196,7 +4196,7 @@ INSERT INTO t1 VALUES (1,1),(2,1); > EXPLAIN SELECT 1 FROM t1 WHERE a = (SELECT COUNT(*) FROM t1 GROUP BY b); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ref a a 5 const 1 Using where; Using index > -2 SUBQUERY internal_tmp_table ALL group_key NULL NULL NULL 1 Using temporary; Using filesort > +2 SUBQUERY t1 ALL NULL NULL NULL NULL 2 Using temporary; Using filesort > DROP TABLE t1; > CREATE TABLE t1 (id int NOT NULL, st CHAR(2), INDEX idx(id)); > INSERT INTO t1 VALUES > > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2012-08-15 06:34:18 +0000 > +++ b/sql/sql_select.cc 2012-08-20 11:45:33 +0000 > @@ -271,8 +271,11 @@ Item_equal *find_item_equal(COND_EQUAL * > bool *inherited_fl); > JOIN_TAB *first_depth_first_tab(JOIN* join); > JOIN_TAB *next_depth_first_tab(JOIN* join, JOIN_TAB* tab); > -JOIN_TAB *first_breadth_first_tab(JOIN *join); > -JOIN_TAB *next_breadth_first_tab(JOIN *join, JOIN_TAB *tab); > + > +enum enum_exec_or_opt {WALK_OPTIMIZATION_TABS , WALK_EXECUTION_TABS}; > +JOIN_TAB *first_breadth_first_tab(JOIN *join, enum enum_exec_or_opt tabs_kind); > +JOIN_TAB *next_breadth_first_tab(JOIN *join, enum enum_exec_or_opt tabs_kind, > + JOIN_TAB *tab); > > /** > This handles SELECT with and without UNION. > @@ -6649,12 +6652,12 @@ double JOIN::get_examined_rows() > { > ha_rows examined_rows; > double prev_fanout= 1; > - JOIN_TAB *tab= first_breadth_first_tab(this); > + JOIN_TAB *tab= first_breadth_first_tab(this, WALK_OPTIMIZATION_TABS); > JOIN_TAB *prev_tab= tab; > > examined_rows= tab->get_examined_rows(); > > - while ((tab= next_breadth_first_tab(this, tab))) > + while ((tab= next_breadth_first_tab(this, WALK_OPTIMIZATION_TABS, tab))) > { > prev_fanout *= prev_tab->records_read; > examined_rows+= (ha_rows) (tab->get_examined_rows() * prev_fanout); > @@ -7269,23 +7272,30 @@ prev_record_reads(POSITION *positions, u > Enumerate join tabs in breadth-first fashion, including const tables. > */ > > -JOIN_TAB *first_breadth_first_tab(JOIN *join) > +JOIN_TAB *first_breadth_first_tab(JOIN *join, enum enum_exec_or_opt tabs_kind) > { > - return join->join_tab; /* There's always one (i.e. first) table */ > + /* There's always one (i.e. first) table */ > + return (tabs_kind == WALK_EXECUTION_TABS)? join->join_tab: > + join->table_access_tabs; > } > > > -JOIN_TAB *next_breadth_first_tab(JOIN *join, JOIN_TAB *tab) > +JOIN_TAB *next_breadth_first_tab(JOIN *join, enum enum_exec_or_opt tabs_kind, > + JOIN_TAB *tab) > { > + JOIN_TAB* const first_top_tab= first_breadth_first_tab(join, tabs_kind); > + const uint n_top_tabs_count= (tabs_kind == WALK_EXECUTION_TABS)? > + join->top_join_tab_count: > + join->top_table_access_tabs_count; > if (!tab->bush_root_tab) > { > /* We're at top level. Get the next top-level tab */ > tab++; > - if (tab < join->join_tab + join->top_join_tab_count) > + if (tab < first_top_tab + n_top_tabs_count) > return tab; > > /* No more top-level tabs. Switch to enumerating SJM nest children */ > - tab= join->join_tab; > + tab= first_top_tab; > } > else > { > @@ -7309,7 +7319,7 @@ JOIN_TAB *next_breadth_first_tab(JOIN *j > Ok, "tab" points to a top-level table, and we need to find the next SJM > nest and enter it. > */ > - for (; tab < join->join_tab + join->top_join_tab_count; tab++) > + for (; tab < first_top_tab + n_top_tabs_count; tab++) > { > if (tab->bush_children) > return tab->bush_children->start; > @@ -7333,7 +7343,7 @@ JOIN_TAB *first_top_level_tab(JOIN *join > > JOIN_TAB *next_top_level_tab(JOIN *join, JOIN_TAB *tab) > { > - tab= next_breadth_first_tab(join, tab); > + tab= next_breadth_first_tab(join, WALK_EXECUTION_TABS, tab); > if (tab && tab->bush_root_tab) > tab= NULL; > return tab; > @@ -7633,6 +7643,12 @@ get_best_combination(JOIN *join) > > join->top_join_tab_count= join->join_tab_ranges.head()->end - > join->join_tab_ranges.head()->start; > + /* > + Save pointers to select join tabs for SHOW EXPLAIN > + */ > + join->table_access_tabs= join->join_tab; > + join->top_table_access_tabs_count= join->top_join_tab_count; > + > update_depend_map(join); > DBUG_RETURN(0); > } > @@ -21389,8 +21405,8 @@ static void select_describe(JOIN *join, > bool printing_materialize_nest= FALSE; > uint select_id= join->select_lex->select_number; > > - for (JOIN_TAB *tab= first_breadth_first_tab(join); tab; > - tab= next_breadth_first_tab(join, tab)) > + for (JOIN_TAB *tab= first_breadth_first_tab(join, WALK_OPTIMIZATION_TABS); tab; > + tab= next_breadth_first_tab(join, WALK_OPTIMIZATION_TABS, tab)) > { > if (tab->bush_root_tab) > { > @@ -21473,16 +21489,8 @@ static void select_describe(JOIN *join, > else > { > TABLE_LIST *real_table= table->pos_in_table_list; > - /* > - Internal temporary tables have no corresponding table reference > - object. Such a table may appear in EXPLAIN when a subquery that needs > - a temporary table has been executed, and JOIN::exec replaced the > - original JOIN with a plan to access the data in the temp table > - (made by JOIN::make_simple_join). > - */ > - const char *tab_name= real_table ? real_table->alias : > - "internal_tmp_table"; > - item_list.push_back(new Item_string(tab_name, strlen(tab_name), cs)); > + item_list.push_back(new Item_string(real_table->alias, > + strlen(real_table->alias), cs)); > } > /* "partitions" column */ > if (join->thd->lex->describe & DESCRIBE_PARTITIONS) > > === modified file 'sql/sql_select.h' > --- a/sql/sql_select.h 2012-06-06 19:26:40 +0000 > +++ b/sql/sql_select.h 2012-08-20 11:45:33 +0000 > @@ -897,6 +897,19 @@ class JOIN :public Sql_alloc > > public: > JOIN_TAB *join_tab, **best_ref; > + > + /* > + For "Using temporary+Using filesort" queries, JOIN::join_tab can point to > + either: > + 1. array of join tabs describing how to run the select, or > + 2. array of single join tab describing read from the temporary table. > + > + SHOW EXPLAIN code needs to read/show #1. This is why two next members are > + there for saving it. > + */ > + JOIN_TAB *table_access_tabs; > + uint top_table_access_tabs_count; > + > JOIN_TAB **map2table; ///< mapping between table indexes and JOIN_TABs > JOIN_TAB *join_tab_save; ///< saved join_tab for subquery reexecution > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] Rev 3353: Performed re-factoring and re-structuring of the code for mwl#248: in file:///home/igor/maria/maria-5.5-mwl248-changes/
by Kristian Nielsen 13 Aug '12

13 Aug '12

Davi Arnaut <davi(a)twitter.com> writes: > On Wed, Aug 1, 2012 at 12:46 AM, Kristian Nielsen > <knielsen(a)knielsen-hq.org> wrote: >> Davi Arnaut <davi(a)twitter.com> writes: >> >>> On Thu, Jul 26, 2012 at 5:50 PM, Igor Babaev <igor(a)askmonty.org> wrote: >>>> + bzero(column_stats, sizeof(Column_statistics) * cnt); >>> >>> That's not allowed for non-POD types. >> >> Can you elaborate on exactly when bzero() (or memset() or similar) is allowed >> or not allowed? Ok, so I got time to research this a bit more. The class in question is like this, the original has more members but this is the essential structure: class Column_statistics { private: static const uint Scale_factor_nulls_ratio= 100000; public: Field *min_value; private: ulong nulls_ratio; public: void set_all_nulls() { ... } }; The question is whether it is ok to allocate this with alloc_root() or similar, and initialise it with bzero(pointer, sizeof(Column_statistics)). (The issue of bzero vs. memset is a separate one, not discussed in this mail). Davi points out that this is not a POD. This is because a POD must be a C++ aggregate, and aggregates can not have private members. Unfortunately, it seems the question of whether bzero is allowed for a non-POD, is more difficult to answer. If we look to the standards, the answer is no. It is not well-defined to bzero() any data except integers. This is what the standard says. However, this is unhelpful for the case at hand, as bzero to initialise a simple struct is a well-established practice in systems programming and commonly accepted to be valid. It seems generally accepted that bzero() of a POD is acceptable. Since the only thing that makes Column_statistics not a POD is the presence of private: members, the question becomes: Should bzero initialisation of classes with private members be disallowed? I could not find any definitive reference to support such disallowance. I am not sure why private members would be disallowed in aggretate, and hence POD. My guess is because aggregates may be initialised with an initialiser list, which makes no sense for private member. Or maybe compilers could re-arrance private members to be placed after public members. Or compilers could optimise based on knowledge that no external code can modify a private member (so it does not need to be spilled over a call to pthread_mutex_lock() or similar). But none of this would give a problem with bzero-initialisation after alloc. So I do not know of a language standard/compiler argument to disallow bzero-initialisation of non-POD. ---- But I would still strongly advice to be very careful with such code, and I think Davi's original concern is similar to mine. The problem is that Column_statistics already looks a lot like a nicely encapsulated proper C++ class. It has private members, accessors, it even defines some constants. One could easily imagine that in the future, there will be the temptation to extend it even more so, like add a virtual function - and then we suddenly have a big problem. bzero() is most certainly _NOT_ ok for such class. So I would personally stay well away from this grey area, and rather - _either_ use only a simple struct with no member functions or private or anything like that - and use bzero/memcpy and such. - _or_ do a full class, but then only initialise it with proper call of contructors (whether explicitly or implicitly defined). That makes it much clearer whether a given struct class is one or the other. But I suppose the optimiser code already heavily uses a different strategy. At the least, the declaration of class Column_statistics should have a comment saying that it is initialised with bzero(), so it can not use features not compatible with this. The nice thing about Davi's suggested rule "no bzero of non-POD" is that it is simple, easy to remember, and easy to state. Which is the primary point of coding style guidelines in the first place: make the rules clear, so there is less wasted time discussing what they are. Hope this helps, - Kristian.

1 1

Re: [Maria-developers] [Commits] Rev 3353: Performed re-factoring and re-structuring of the code for mwl#248: in file:///home/igor/maria/maria-5.5-mwl248-changes/
by Michael Widenius 13 Aug '12

13 Aug '12

Hi! >>>>> "Davi" == Davi Arnaut <davi(a)twitter.com> writes: <cut> Davi> It's not well defined, that's one of the reasons why it was Davi> deprecated. It does not even exist anymore in some systems, Davi> POSIX.1-2008 removed the specification of bzero(). It's well defined in the sence that it works everywhere (at least for MySQL code). >> Better to always use the same code to reset memory than doing it many >> different ways. In MySQL code bzero() has been the offical and >> recommended way to reset memory. Davi> Not anymore, Oracle MySQL has addressed this issue. In MariaDB code that is still not the case. Personally, I find bzero() easier to read, instead of memset() as there is no extra parameter and it's also easier grep for it. I assume you agree that for all particular purposes, there is no reason to not use bzero() in MariaDB code? Regards, Monty

2 1

[Maria-developers] Thoughts on Global Transaction ID, parallel slave, and multi-source replication
by Kristian Nielsen 10 Aug '12

10 Aug '12

This is a writeup of some thoughts I have on getting Global Transaction ID in MariaDB, as well as some related thoughts on parallel slave and multi-source replication. MySQL 5.6 has global transaction ID now, and I looked into it in some details. Unfortunately, I _really_ did not like what I saw. There are two main problems, in my opinion: 1. The basic design is bad, leading to an excessively complicated implementation. 2. The implementation looks very incomplete/immature, to the point that I am amazed that anyone would push code like this into a main tree. Point 2 means that I really do not want to maintain this piece of **** in MariaDB, though we can hope that things will improve in subsequent releases. Point 1 leads me to suggest that instead of merging the MySQL 5.6 implementation, we could do our own implementation, using a design that is both much simpler, and at the same time more flexible that what is in MySQL 5.6. I will elaborate further on these points in the rest of the mail. ---- With respect to point 2, let me quote this little gem from the MySQL 5.6 code: /* Apparently this code is not being called. We need to investigate if this is a bug or this code is not necessary. /Alfranio */ DBUG_ASSERT(0); /NOTREACHED/ Now, certainly, there are dark corners of the MySQL replication code, and adding a comment/assert like this could be justified if a developer encounters it and finds it may be dead code. But this is _new_ code, added in a new 5.6 release! So the authors of the new feature, by their own statement, have no idea what the new code they are pushing into the main tree is actually doing. That is just insane. I have no idea what the replication team at MySQL is up to these days pulling stunts like this, I have to confess I am disappointed. I found many similar examples in the 5.6 code. I wrote up a more detailed analysis in TODO-171, where I elaborate more on the problems I see with the MySQL 5.6 Global Transaction ID code: https://mariadb.atlassian.net/browse/TODO-171 When the authors do not understand the code themselves, it really does not seem attractive for us to try to support and bugfix it. I also found a lot of what I think are left-over, unsused parts of an even more complicated design that was apparently abandoned (or postponed?), but left in the code, which does not make supporting this any easier. ---- To explain point 1, I first need to mention the MySQL 5.6 feature of parallel slave apply, as I believe that a major factor in the design of MySQL Global Transaction ID has been the need to interoperate with that feature. The MySQL parallel replication slave feature works so that transactions on tables in different databases can be run in parallel with each other (incidentally, there is some indication that the MySQL replication team considers this *the* meaning of parallel slave, which is sad, as it is just one piece of the general problem of replication slave scalability - it helps in some workloads, but it is useless for others). So with parallel slave, transactions on the master can be executed in a different order on the slave. This means that the crucial notion of a slaves current position in the replication stream becomes more complicated - it is not just a single point in the master binlog, rather every slave replication thread can be at a different point in the master binlog. MySQL 5.6 solves this by introducing "Gtid sets". A Gtid is a MySQL 5.6 global transaction ID - a unique, 24-byte identifier for every transaction. The state of a replication slave at any given moment is then the set Gtids of all transactions ever executed on that slave server This notion of having a set of all transactions ever executed really complicates the design. There is a need to represent such a set as basically intervals of "holes" in the Gtid sequence, to avoid the memory representation of such sets growing without bound over time. This in turns introduces the requirement that there can _never_ be holes in the sequence of Gtids generated on a master - if there ever is introduced a hole, slaves will wait around for ever waiting for a transaction containing the missing Gtid. I think recovering from errors with such a concept will be forever a nightmare. I think this basic fundament of the design is the main source of my dislike for the MySQL global transaction ID. It sounds nice on a first look, but once you start to think deeper on the problem it just does not seem to work well. ---- So I think I have a better idea of how to make Global Transaction ID work well with parallel slave based on executing transactions in different databases out-of-order. The MySQL Gtid is defined as a pair of server-id and sequence number. Every master server has a single server-id (despite no less than 128 bits being allocated for this - anyone planning a replication hierarchy with 2**128 servers?). The master generates a single stream of events in the binlog, though streams from other servers may be interleaved with it in case of multi-master. It is then the task of a slave to try to split up the single master stream in to multiple streams that can be executed in parallel. I think this is the root of the problem. The slave is the wrong place to do this splitting! It is on the master that we know how to interpret transactions (ie. that transactions in different databases are / are not independent), so doing this on the slave is harder. And two slaves can in general split differently, which makes it harder for a slave deeper in the replication hierarchy to switch to a new master (that master being itself a slave of a higher-up master), leading to the need for a complex set-of-all-executed-transactions concept. Suppose we instead make the decision on the _master_ on how to split up the replication stream for parallel apply. Instead of a single server-id for the master, we can have multiple - let us call them "Source ID". On the _master_, we will decide that two transactions in different databases are independent, and assign them different Source ID. The slave then receives multiple replication streams, and is free to execute each stream in parallel - or not, at the discretion of the DBA. Transactions _within_ a single stream are always executed in-order, so the slave has no need to remember any sets of all transactions ever executed - all it needs to remember is the sequence number within each stream it has seen - ie. last sequence number for each Source ID. This is a simple and obvious generalisation of the old concept of current binlog position. Downstream slaves see the exact same binlog sequences, each with transactions in the same order (though sequences may be interleaved differently) - so changing masters is simpler, all servers in the hierarchy have the same view. So to implement the same functionality as MySQL 5.6 parallel replication, all we need is on the master to generate Source ID as a hash of the database name (just as MySQL 5.6 assigns transactions to slave threads based on such hash of the database name). And implement a facility on the slave such that transactions with different Source ID can be executed in parallel by multiple slave threads. And now, since we decide on the master, we can provide much more flexibility for the application. We can have _some_ databases db1,db2 be executed in parallel on the slave - while other databases db3,db4 are considered dependent, and not modified out-of-order on the slave. Or the application can declare that two transactions are independent within the same database or even within the same table; for example log record inserts into a table can often be allowed to happen in different order on slaves. Or the DBA can assign a special Source ID to a long-runnign ALTER TABLE, and such ALTER TABLE can then execute independently on slaves to not stall the normal load. All we need is to provide a simple facility (eg. session variable) that applications/users can use to specify Source ID. And then the next step is multi-source replication, which is the much-request feature that one slave can replicate from two or more different masters at the same time. The different masters will just supply different Source IDs, so apart from the need to maintain multiple TCP connections, one to each master, from the rest of the code's point of view multi-source replication is no different, it is just more independent replication streams. So my suggestion is: rather than struggle to try and merge and support the MySQL 5.6 Global Transaction ID and parallel slave, let us instead implement something like the above idea, with multiple Source IDs on the master, and parallel execution of distinct replication streams on the slave. I believe this will be no more work, and the end result will be both simpler and more flexible. Thanks, - Kristian.

4 9

[Maria-developers] Cost analysis: checking if join optimization costs match the reality.
by Sergei Petrunia 10 Aug '12

10 Aug '12

Hi! I've ran this experiment: I took the scalar subquery out of the query and then tried to compare optimizer costs with execution times. == The query == explain extended select sql_calc_found_rows s_name, s_address from nation straight_join supplier where s_suppkey in (select ps_suppkey from partsupp where ps_partkey in (select p_partkey from part where p_name like 'forest%') ) and s_nationkey = n_nationkey and n_name = 'CANADA' order by s_name limit 10; +--+-----------+--------+------++-------------+-------+-------------------+----+---------------------------------------------------------+ |id|select_type|table |type ||key |key_len|ref |rows|Extra | +--+-----------+--------+------++-------------+-------+-------------------+----+---------------------------------------------------------+ | 1|PRIMARY |nation |ref ||n_name |26 |const | 1|Using where; Using index; Using temporary; Using filesort| | 1|PRIMARY |supplier|ref ||i_s_nationkey|5 |nation.n_nationkey | 207| | | 1|PRIMARY |partsupp|ref ||i_ps_suppkey |4 |supplier.s_suppkey | 50|Using index | | 1|PRIMARY |part |eq_ref||PRIMARY |4 |partsupp.ps_partkey| 1|Using where; FirstMatch(supplier) | +--+-----------+--------+------++-------------+-------+-------------------+----+---------------------------------------------------------+ +--+------------+-----------+------++-------------+-------+------------------+----+---------------------------------------------------------+ |id|select_type |table |type ||key |key_len|ref |rows|Extra | +--+------------+-----------+------++-------------+-------+------------------+----+---------------------------------------------------------+ | 1|PRIMARY |nation |ref ||n_name |26 |const | 1|Using where; Using index; Using temporary; Using filesort| | 1|PRIMARY |supplier |ref ||i_s_nationkey|5 |nation.n_nationkey| 207| | | 1|PRIMARY |<subquery2>|eq_ref||distinct_key |4 |func | 1| | | 2|MATERIALIZED|part |range ||p_name |58 |NULL |2387|Using where; Using index | | 2|MATERIALIZED|partsupp |ref ||i_ps_partkey |4 |part.p_partkey | 2|Using index | +--+------------+-----------+------++-------------+-------+------------------+----+---------------------------------------------------------+ === Execution times === FirstMatch: 0.20 - 0.30 sec. Materialization: 0.20 sec - 0.30 sec run times for both fluctuate between 0.20 and 0.30 sec, FirstMatch seems to be somewhat slower (dunno if the difference is statistically meaningful) == Optimizer costs === FirstMatch: 11029.4 Materialization: 3594.5 === Read record counters === (source data is plan counters provided below) FirstMatch: 43185 Materialization: 12354 + 9K temptable writes + 400 reads === Conclusions === - optimizer thinks FirstMatch is 3x more expensive. - In reality it is *somewhat* expensive. - Record counters say FirstMatch is 3.5x .. 1.9x more expensive, depending on how we treat temptable writes/reads ==> Overall, I see no big discrepancies. ==> Also, I have also discovered that JOIN::get_prefix_cost_and_fanout() forgets to add record_count/TIME_FOR_COMPARE, which causes (slightly) different cost for SJ-Materialization. (I have noted the difference in some previous email) === Firstmatch plan counters === MariaDB [dbt3sf1]> show table_statistics; +--------------+------------+-----------+--------------+-------------------------+ | Table_schema | Table_name | Rows_read | Rows_changed | Rows_changed_x_#indexes | +--------------+------------+-----------+--------------+-------------------------+ | dbt3sf1 | nation | 1 | 0 | 0 | | dbt3sf1 | part | 21386 | 0 | 0 | | dbt3sf1 | partsupp | 21386 | 0 | 0 | | dbt3sf1 | supplier | 412 | 0 | 0 | +--------------+------------+-----------+--------------+-------------------------+ 4 rows in set (0.00 sec) MariaDB [dbt3sf1]> show index_statistics; +--------------+------------+---------------+-----------+ | Table_schema | Table_name | Index_name | Rows_read | +--------------+------------+---------------+-----------+ | dbt3sf1 | nation | n_name | 1 | | dbt3sf1 | part | PRIMARY | 21386 | | dbt3sf1 | supplier | i_s_nationkey | 412 | | dbt3sf1 | partsupp | i_ps_suppkey | 21386 | +--------------+------------+---------------+-----------+ Total= 43185 rows read === Materialization plan counters === MariaDB [dbt3sf1]> show index_statistics; +--------------+------------+---------------+-----------+ | Table_schema | Table_name | Index_name | Rows_read | +--------------+------------+---------------+-----------+ | dbt3sf1 | nation | n_name | 1 | | dbt3sf1 | part | p_name | 2389 | | dbt3sf1 | supplier | i_s_nationkey | 412 | | dbt3sf1 | partsupp | i_ps_partkey | 9552 | +--------------+------------+---------------+-----------+ 4 rows in set (0.00 sec) MariaDB [dbt3sf1]> show table_statistics; +--------------+------------+-----------+--------------+-------------------------+ | Table_schema | Table_name | Rows_read | Rows_changed | Rows_changed_x_#indexes | +--------------+------------+-----------+--------------+-------------------------+ | dbt3sf1 | nation | 1 | 0 | 0 | | dbt3sf1 | part | 2389 | 0 | 0 | | dbt3sf1 | partsupp | 9552 | 0 | 0 | | dbt3sf1 | supplier | 412 | 0 | 0 | +--------------+------------+-----------+--------------+-------------------------+ 4 rows in set (0.00 sec) Total = 12354 rows read + 9K temptable writes + 400 reads from temptable. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 1