- developers - lists.mariadb.org

[Maria-developers] MDEV-15740 (InnoDB lost Durability due to incorrect fix of MDEV-11937)
by Marko Mäkelä 02 Nov '18

02 Nov '18

Hi Kristian, Could you please take a look at this InnoDB regression in MariaDB 10.2 and later: https://jira.mariadb.org/browse/MDEV-15740 Because of this bug, we can no longer write tests that would ensure that certain changes are present in the redo log, before killing and restarting the server. This already hurt me a bit when testing MDEV-15562 (instant DROP COLUMN) in 10.4. I had to add extra sleeps to test cases in order to actually test redo log roll-forward followed by rollback. -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation

2 3

Re: [Maria-developers] [Commits] 737de39: Issue #885: Can't locate lib/mtr_process.pl in @INC on Ubuntu 18.04.1
by Kristian Nielsen 02 Nov '18

02 Nov '18

Hi Sergey, psergey(a)askmonty.org (Sergei Petrunia) writes: > committer: Sergei Petrunia > branch nick: mysql-5.6-rocksdb-spetrunia > Issue #885: Can't locate lib/mtr_process.pl in @INC on Ubuntu 18.04.1 > > Add "." to @INC array like upstream does. Just want to make sure you are aware of this: https://github.com/MariaDB/server/commit/d185f1d68bb1f37bea10d8ac6188e5a04f… I see the branch nick is mysql-5.6-<something>, so maybe this is like this for legacy reasons, which is fine. But this is basically the wrong fix. Clearly sometimes in ancient history, someone did not understand Perl `require` semantics, and ended up with a horrible mess. For example, if someone ever finds they need `require mtr_misc.pl` in a third file, they will find it will not work. Anyway, just a pointer in case you were not aware of a small land-mine hidden under your feet ;-) - Kristian.

2 1

Re: [Maria-developers] f95616dc477: MDEV-17401: LOAD DATA from very big file into MyISAM table results in EOF error and corrupt index
by Sergei Golubchik 31 Oct '18

31 Oct '18

Hi, Oleksandr! ok to push! On Oct 30, Oleksandr Byelkin wrote: > revision-id: f95616dc4774328381044b53263d6a23ae19bfb4 (mariadb-5.5.61-39-gf95616dc477) > parent(s): 893ebb739e24637985892da1555f2e6f1a8ac5b6 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-10-30 19:36:17 +0100 > message: > > MDEV-17401: LOAD DATA from very big file into MyISAM table results in EOF error and corrupt index > > my_read fixed as in higher versions. > my_pread made as my_read aware of partial read of huge chunks of files > MY_FULL_IO enabled for file operations > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] c29706f: MDEV-17286 Decimal field multiplication bug.
by Sergei Golubchik 30 Oct '18

30 Oct '18

Hi, Alexey! On Oct 30, Alexey Botchkov wrote: > > What if both arguments will be zeros, but have intg=DIG_PER_DEC1-1. > > Won't the result have more that DIG_PER_DEC1 zeros? > > Leading zero 'digits' are truncated, so we never get more than DIG_PER_DEC > there. Actually we don't need to do that check at all (intg <= > DIG_PER_DEC1), > i removed it already. Put it in an assert please. I find asserts quite useful as a documentation tool :) Then ok to push! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] c29706f: MDEV-17286 Decimal field multiplication bug.
by Sergei Golubchik 30 Oct '18

30 Oct '18

Hi, Alexey! On Oct 24, Alexey Botchkov wrote: > revision-id: c29706f7560148488791acca3c92dde1b188334e (mariadb-5.5.61-32-gc29706f) > parent(s): e31e697f17f79ffa6913499e7e2d29866f24b475 > committer: Alexey Botchkov > timestamp: 2018-10-24 01:26:49 +0400 > message: > > MDEV-17286 Decimal field multiplication bug. > > We should clear trailing zeroes in frac part. Otherwise > that tail is growing quickly and forces unnecessary truncating of > arguments. This looks good, thanks. Please add a test for create select, as discussed. And fix the bug number in the commit comment. See one question below. > @@ -2109,6 +2104,14 @@ int decimal_mul(const decimal_t *from1, const decimal_t *from2, decimal_t *to) > for (; d_to_move--; cur_d++, buf1++) > *cur_d= *buf1; > } > + > + /* Now we have to check for -0.000 case */ > + if (to->sign && to->intg <= DIG_PER_DEC1 && to->frac == 0 && > + to->buf[0] == 0) What if both arguments will be zeros, but have intg=DIG_PER_DEC1-1. Won't the result have more that DIG_PER_DEC1 zeros? > + { > + /* We got decimal zero */ > + decimal_make_zero(to); > + } > return error; > } > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] c29706f: MDEV-17286 Decimal field multiplication bug.
by Sergei Golubchik 29 Oct '18

29 Oct '18

Hi, Alexey! On Oct 24, Alexey Botchkov wrote: > revision-id: c29706f7560148488791acca3c92dde1b188334e (mariadb-5.5.61-32-gc29706f) > parent(s): e31e697f17f79ffa6913499e7e2d29866f24b475 > committer: Alexey Botchkov > timestamp: 2018-10-24 01:26:49 +0400 > message: > > MDEV-17286 Decimal field multiplication bug. it's MDEV-17256 :) > > We should clear trailing zeroes in frac part. Otherwise > that tail is growing quickly and forces unnecessary truncating of > arguments. I don't understand this, sorry. Could you elaborate, please? I understand the "tail is growing quickly" part. How does it "force unnecessary truncating of arguments"? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-12321 authentication plugin: SET PASSWORD support
by Sergei Golubchik 27 Oct '18

27 Oct '18

Hi, Please review commits for MDEV-12321 "authentication plugin: SET PASSWORD support" They're now in the bb-10.4-serg branch, the last seven commits: ========================= commit 62e8340f513 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Wed Oct 17 12:48:13 2018 +0200 MDEV-12321 authentication plugin: SET PASSWORD support Support SET PASSWORD for authentication plugins. Authentication plugin API is extended with two optional methods: * hash_password() is used to compute a password hash (or digest) from the plain-text password. This digest will be stored in mysql.user table * preprocess_hash() is used to convert this digest into some memory representation that can be later used to authenticate a user. Build-in plugins convert the hash from hexadecimal or base64 to binary, to avoid doing it on every authentication attempt. Note a change in behavior: when loading privileges (on startup or on FLUSH PRIVILEGES) an account with an unknown plugin was loaded with a warning (e.g. "Plugin 'foo' is not loaded"). But such an account could not be used for authentication until the plugin is installed. Now an account like that will not be loaded at all (with a warning, still). Indeed, without plugin's preprocess_hash() method the server cannot know how to load an account. Thus, if a new authentication plugin is installed run-time, one might need FLUSH PRIVILEGES to activate all existing accounts that were using this new plugin. commit 8266ca26f27 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Mon Oct 15 01:39:03 2018 +0200 misc cleanups * remove dead code (from .yy) * remove redundant commands from the test * extract common code into a reusable function (get_auth_plugin, push_new_user) * rename update_user_table->update_user_table_password * simplify acl_update_role * don't strdup a string that's already in a memroot (in ACL_ROLE::ACL_ROLE(ACL_USER*)) * create parent_grantee and role_grants dynamic arrays with size 0. to avoid any memory allocations when roles aren't used. commit fd0bcb5e791 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Sun Oct 14 13:52:52 2018 +0200 Use mysql.user.authentication_string for password Don't distinguish between a "password hash" and "authentication string" anymore. Now both are stored in mysql.user.authentication_string, both are handled identically internally. A "password hash" is just how some particular plugins interpret authentication string. Set mysql.user.plugin even if there is no password. The server will use mysql_native_password plugin in these cases, let's make it expicit. Remove LEX_USER::pwhash. commit 15f30100e0d Author: Sergei Golubchik <serg(a)mariadb.org> Date: Sat Oct 13 18:32:05 2018 +0200 cleanup: sql_acl.cc remove fix_plugin_ptr() it was doing two my_strcasecmp() unconditionally, to optimize away one conditional my_strcasecmp() later. commit df9d95d2a85 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Sat Oct 13 11:30:39 2018 +0200 cleanup: sql_acl.cc remove username=NULL Some parts of sql_acl.cc historically assumed that empty username is represented by username=NULL, other parts used username="" for that. And most of the code wasn't sure and checked both (like in `if (!user || !user[0])`). Change it to use an empty string everywhere. commit 8b3cbc1e469 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Fri Oct 12 18:48:15 2018 +0200 cleanup: sql_acl.cc password->LEX_CSTRING commit 2fc7a20da6a Author: Sergei Golubchik <serg(a)mariadb.org> Date: Fri Oct 12 19:24:28 2018 +0200 cleanup: safe_lexcstrdup_root() ========================= I've split somewhat independent changes into separate commits, but, of course, if you'd like to review one big patch you can `git diff` them all together. Now, the syntax problem. Old MySQL syntax (for the last ~20 years) was (1) GRANT ... TO user IDENTIFIED BY 'plain-text password' (2) GRANT ... TO user IDENTIFIED BY PASSWORD 'password hash' (3) SET PASSWORD = 'password hash' (4) SET PASSWORD = PASSWORD('plain-text password') (5) SET PASSWORD = OLD_PASSWORD('plain-text password') here, syntax (1) and (4) were forcing mysql_native_password authentication, (5) was forcing mysql_old_password, and (2) and (3) were auto-detecting, based on the hash length. Later MariaDB and MySQL added support for pluggable authentication with the syntax (6) GRANT ... TO user IDENTIFIED VIA plugin AS 'password hash' MySQL 5.7 added support for specifying plain-text password for plugins using the syntax (7) GRANT ... TO user IDENTIFIED WITH plugin BY 'plain-text password' I don't quite like it because there's no logical reason why "BY" means a plain-text password, while "AS" means a hash. Also we support "USING" instead of "AS", which also means a hash. One can easily get lost in all these USING/AS/BY and what special semantics each of them has. The syntax I've implemented is based on SET PASSWORD: (8) GRANT ... TO user IDENTIFIED VIA plugin AS PASSWORD('plain-text password') This is quite intuitive and pretends that there's a sql function PASSWORD() which returns a hash and it's used as an expression where a hash is anyway expected. Same works in SET PASSWORD too, obviously. A PASSWORD() function actually exists and returns the password hash. The problem here is that PASSWORD() function becomes quite magical. It returns a different password, depending on what plugin the user is using. One can still do SELECT PASSWORD("foo"), OLD_PASSWORD("foo"), but they'll return values for mysql_native_password and mysql_old_password as before. In the context of SET PASSWORD or GRANT (or CREATE/ALTER USER) it becomes context dependent, it's a bit difficult to swallow. Another approach would be not to pretend it's a function. Say (9) GRANT ... TO user IDENTIFIED VIA plugin AS PASSWORD 'plain-text password' SET PASSWORD = PASSWORD 'plain-text password' but, unfortunately, it is exactly backwards from the historical behaviour of (1) and (2). All in all I'm leaning towards (8), but I'm not quite happy with it :( One way to solve it could be to extend PASSWORD() function to allow a second argument, plugin name, like in SELECT PASSWORD("foo", "ed25519") Yet another way could be to remove SQL-level functions PASSWORD() and OLD_PASSWORD(). That would be my favorite, they always were nothing but trouble. But I wouldn't risk doing it now :) Regards, Sergei

2 3

Re: [Maria-developers] cc5c19ae523: MDEV-17401: LOAD DATA from very big file into MyISAM table results in EOF error and corrupt index
by Sergei Golubchik 24 Oct '18

24 Oct '18

Hi, Oleksandr! On Oct 12, Oleksandr Byelkin wrote: > revision-id: cc5c19ae5233ba90de086de76043774ae6c78cd7 (mariadb-5.5.61-30-gcc5c19ae523) > parent(s): acf8fc1ff8a7b2d49e25279670b04b8eb096ce0c > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-10-12 09:07:05 +0200 > message: > > MDEV-17401: LOAD DATA from very big file into MyISAM table results in EOF error and corrupt index > > my_read fixed as in higher versions. > my_pread made as my_read aware of partial read of huge chunks of files > MY_FULL_IO enabled for file operations > > --- > mysys/mf_iocache.c | 4 ++++ > mysys/my_pread.c | 25 ++++++++++++++++++------- > mysys/my_read.c | 52 ++++++++++++++++++++++++++++------------------------ > 3 files changed, 50 insertions(+), 31 deletions(-) > > diff --git a/mysys/mf_iocache.c b/mysys/mf_iocache.c > index bd71e2ae527..9a051cf6a9e 100644 > --- a/mysys/mf_iocache.c > +++ b/mysys/mf_iocache.c > @@ -282,6 +282,10 @@ int init_io_cache(IO_CACHE *info, File file, size_t cachesize, > } > info->inited=info->aio_result.pending=0; > #endif > + if (type == READ_CACHE || type == WRITE_CACHE || type == SEQ_READ_APPEND) > + info->myflags|= MY_FULL_IO; > + else > + info->myflags&= ~MY_FULL_IO; 1. that's a bit redundant, it'd be enough to do it in init_io_cache. but ok. 2. why you didn't remove MY_FULL_IO setting from reinit_io_cache? > DBUG_RETURN(0); > } /* init_io_cache */ > > diff --git a/mysys/my_pread.c b/mysys/my_pread.c > index 51b4c5f5599..5cbd6b4316b 100644 > --- a/mysys/my_pread.c > +++ b/mysys/my_pread.c > @@ -47,8 +47,7 @@ > size_t my_pread(File Filedes, uchar *Buffer, size_t Count, my_off_t offset, > myf MyFlags) > { > - size_t readbytes; > - int error= 0; > + size_t readbytes, save_count= 0; > > DBUG_ENTER("my_pread"); > > @@ -66,11 +65,10 @@ size_t my_pread(File Filedes, uchar *Buffer, size_t Count, my_off_t offset, > #else > readbytes= pread(Filedes, Buffer, Count, offset); > #endif > - error = (readbytes != Count); > > - if (error) > + if (readbytes != Count) > { > - my_errno= errno ? errno : -1; > + my_errno= errno; > if (errno == 0 || (readbytes != (size_t) -1 && > (MyFlags & (MY_NABP | MY_FNABP)))) > my_errno= HA_ERR_FILE_TOO_SHORT; > @@ -82,6 +80,17 @@ size_t my_pread(File Filedes, uchar *Buffer, size_t Count, my_off_t offset, > (int) readbytes)); > continue; /* Interrupted */ > } > + > + /* Do a read retry if we didn't get enough data on first read */ > + if (readbytes != (size_t) -1 && readbytes != 0 && > + (MyFlags & MY_FULL_IO)) > + { > + Buffer+= readbytes; > + Count-= readbytes; > + save_count+= readbytes; > + continue; eh, did you test your patch? I don't see how it could work without offset+= readbytes; > + } > + > if (MyFlags & (MY_WME | MY_FAE | MY_FNABP)) > { > if (readbytes == (size_t) -1) > @@ -97,8 +106,10 @@ size_t my_pread(File Filedes, uchar *Buffer, size_t Count, my_off_t offset, > DBUG_RETURN(MY_FILE_ERROR); /* Return with error */ > } > if (MyFlags & (MY_NABP | MY_FNABP)) > - DBUG_RETURN(0); /* Read went ok; Return 0 */ > - DBUG_RETURN(readbytes); /* purecov: inspected */ > + readbytes= 0; /* Read went ok; Return 0 */ > + else > + readbytes+= save_count; > + DBUG_RETURN(readbytes); > } > } /* my_pread */ > > diff --git a/mysys/my_read.c b/mysys/my_read.c > index 883a1c5fdc7..47f2d725f65 100644 > --- a/mysys/my_read.c > +++ b/mysys/my_read.c > @@ -35,17 +35,16 @@ this is copied from 10.1 verbatim, I presume Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] c850799967c: MDEV-10963 Fragmented BINLOG query
by Sergei Golubchik 23 Oct '18

23 Oct '18

Hi, Andrei! First, when I run it I got: Warning: 112 bytes lost at 0x7f212f26a670, allocated by T@0 at sql/sql_binlog.cc:68, sql/sql_binlog.cc:181, sql/sql_parse.cc:5625, sql/sql_parse.cc:7465, sql/sql_parse.cc:1497, sql/sql_parse.cc:1124, sql/sql_connect.cc:1330, sql/sql_connect.cc:1243 so you have a memory leak with your thd->lex->comment.str Second, BINGLOG CONCAT() is totally possible, inventing a new keyword defeats the whole point of using a familiar and intuitive syntax. I've attached a simple patch that implements it. On the other hand, I just thought that CONCAT() syntax might be not so intuitive after all, as - strictly speaking - a result of CONCAT cannot be larger than max_allowed_packet. So it might be confusing to use CONCAT here. May be just use your idea of BINLOG @a, @b; with no CONCAT or anything else at all? Also, I'm thinking whether BINLOG (in this last form) could automatically clear its variables @a and @b. Without an explicit SET @a=NULL. It's a bit weird to have such a side effect. But it'll help to save memory - the event size it over 3GB, right? So you'll have it split in two variables. Then you create a concatenated copy (it's over 3GB more), then you decode it (already more than 9GB in memory), and then it's executed where copies of huge blob values are created again, may be more than once. Freeing @a and @b and the concatenated copy as soon as possible will save over 6GB in RAM usage. Note, we cannot auto-reset variables in the syntax with CONCAT(), DEFRAGMENT() or whatever, because functions cannot do it to their arguments. On Oct 18, Andrei Elkin wrote: > revision-id: c850799967c561d08242987269d94af6ae4c7c5e (mariadb-10.1.35-59-gc850799967c) > parent(s): d3a8b5aa9cee24a6397662e30df2e915f45460e0 > author: Andrei Elkin <andrei.elkin(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2018-09-21 18:48:06 +0300 > message: > > MDEV-10963 Fragmented BINLOG query > > The problem was originally stated in > http://bugs.mysql.com/bug.php?id=82212 > The size of an base64-encoded Rows_log_event exceeds its > vanilla byte representation in 4/3 times. > When a binlogged event size is about 1GB mysqlbinlog generates > a BINLOG query that can't be send out due to its size. > > It is fixed with fragmenting the BINLOG argument C-string into > (approximate) halves when the base64 encoded event is over 1GB size. > The mysqlbinlog in such case puts out > > SET @binlog_fragment_0='base64-encoded-fragment_0'; > SET @binlog_fragment_1='base64-encoded-fragment_1'; > BINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_1); > > to represent a big BINLOG 'base64-encoded-"total"'. > Two more statements are composed to promptly release memory > SET @binlog_fragment_0=NULL; > SET @binlog_fragment_1=NULL; > > The 2 fragments are enough, though the client and server still may > need to tweak their @@max_allowed_packet to satisfy to the fragment > size (which they would have to do anyway with greater number of > fragments, should that be desired). > > On the lower level the following changes are made: > > Log_event::print_base64() > remains to call encoder and store the encoded data into a cache but > now *without* doing any formatting. The latter is left for time > when the cache is copied to an output file (e.g mysqlbinlog output). > No formatting behavior is also reflected by the change in the meaning > of the last argument which specifies whether to cache the encoded data. > > my_b_copy_to_file() > is turned into my_b_copy_to_file_frag() > which accepts format specifier arguments to build a proper syntax BINLOG > query in both the fragmented (n_frag > 1) and non-fragmented (n_frag == 1) > cases. > > Rows_log_event::print_helper() > Takes decision whether to fragment, prepares respective format specifiers > and invokes the cache-to-file copying function, which is now > > copy_cache_frag_to_file_and_reinit() > replaces original copy_event_cache_to_file_and_reinit() to pass > extra arguments to my_b_copy_to_file() successor of > > my_b_copy_to_file_frag() > replaces the former pure copier not to also conduct wrapping the encoded > data per format specifiers. > With its 'n_frag' argument as 1 > and the rest or args NULL works as the original function. > > diff --git a/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row_frag.test b/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row_frag.test > new file mode 100644 > index 00000000000..5318f8acec6 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row_frag.test > @@ -0,0 +1,43 @@ > +--source include/have_debug.inc > +--source include/have_log_bin.inc > +--source include/have_binlog_format_row.inc > + > +--let $MYSQLD_DATADIR= `select @@datadir` > +--let $max_size=1024 > + > +CREATE TABLE t (a TEXT); > +# events of interest are guaranteed to stay in 000001 log > +RESET MASTER; > +--eval INSERT INTO t SET a=repeat('a', $max_size) > +SELECT a from t into @a; > +FLUSH LOGS; > +DELETE FROM t; > + > +--exec $MYSQL_BINLOG --debug-binlog-row-event-max-encoded-size=256 $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/mysqlbinlog.sql > + > +--exec $MYSQL test < $MYSQLTEST_VARDIR/tmp/mysqlbinlog.sql how do you know the event was split, if you aren't looking into mysqlbinlog.sql file? may be your code doesn't work at all, or it was removed by some incorrect merge? the test will pass anyway... > + > +SELECT a LIKE @a as 'true' FROM t; > +SELECT @binlog_fragment_0, @binlog_fragment_1 as 'NULL'; > + > +# improper syntax error > +--echo BINLOG number-of-fragments must be exactly two > +--error ER_PARSE_ERROR > +BINLOG DEFRAGMENT(@binlog_fragment); > +--error ER_PARSE_ERROR > +BINLOG DEFRAGMENT(@binlog_fragment, @binlog_fragment, @binlog_fragment); > + > +# corrupted fragments error check (to the expected error code notice, > +# the same error code occurs in a similar unfragmented case) > +SET @binlog_fragment_0='012345'; > +SET @binlog_fragment_1='012345'; > +--error ER_SYNTAX_ERROR > +BINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_1); > + > +# Not existing fragment is not allowed > +SET @binlog_fragment_0='012345'; > +--error ER_BASE64_DECODE_ERROR > +BINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_not_exist); > + > +--echo # Cleanup > +DROP TABLE t; > diff --git a/mysys/mf_iocache2.c b/mysys/mf_iocache2.c > index 2499094037d..6b7ff8a7568 100644 > --- a/mysys/mf_iocache2.c > +++ b/mysys/mf_iocache2.c > @@ -22,13 +22,23 @@ > #include <stdarg.h> > #include <m_ctype.h> > > -/* > +/** > Copy contents of an IO_CACHE to a file. > > SYNOPSIS > - my_b_copy_to_file() > - cache IO_CACHE to copy from > - file File to copy to > + my_b_copy_to_file_frag > + > + cache IO_CACHE to copy from > + file File to copy to > + n_frag # of fragments > + > + Other arguments represent format strings to enable wrapping > + of the fragments and their total, including > + > + before_frag before a fragment > + after_frag after a fragment > + after_last_frag after all the fragments > + final_per_frag in the end per each fragment if you edit a comment, change it to Doxygen syntax. Especially, as you start it with Doxygen's /**, it is simply invalid to have SYNOPSIS and DESCRIPTION inside. > > DESCRIPTION > Copy the contents of the cache to the file. The cache will be > @@ -38,33 +48,120 @@ > If a failure to write fully occurs, the cache is only copied > partially. > > - TODO > - Make this function solid by handling partial reads from the cache > - in a correct manner: it should be atomic. > + The copying is made in so many steps as the number of fragments as > + specified by the parameter 'n_frag'. Each step is wrapped with > + writing to the file 'before_frag' and 'after_frag' formated > + strings, unless the parameters are NULL. In the end, optionally, > + first 'after_last_frag' string is appended to 'file' followed by > + 'final_per_frag' per each fragment. > + final item. > > RETURN VALUE > 0 All OK > 1 An error occurred > + > + TODO > + Make this function solid by handling partial reads from the cache > + in a correct manner: it should be atomic. > */ > int > -my_b_copy_to_file(IO_CACHE *cache, FILE *file) > +my_b_copy_to_file_frag(IO_CACHE *cache, FILE *file, > + uint n_frag, > + const char* before_frag, > + const char* after_frag, > + const char* after_last, > + const char* final_per_frag, > + char* buf) > { > - size_t bytes_in_cache; > - DBUG_ENTER("my_b_copy_to_file"); > + size_t bytes_in_cache; // block, may have short size in the last one > + size_t written_off_last_block; // consumed part of the block by last fragment > + size_t total_size= my_b_tell(cache); > + size_t frag_size= total_size / n_frag + 1; > + size_t total_written= 0; > + size_t frag_written; // bytes collected in the current fragment > + uint i; > + > + DBUG_ENTER("my_b_copy_to_file_frag"); > + > + DBUG_ASSERT(cache->type == WRITE_CACHE); > > /* Reinit the cache to read from the beginning of the cache */ > if (reinit_io_cache(cache, READ_CACHE, 0L, FALSE, FALSE)) > DBUG_RETURN(1); > - bytes_in_cache= my_b_bytes_in_cache(cache); > - do > - { > - if (my_fwrite(file, cache->read_pos, bytes_in_cache, > - MYF(MY_WME | MY_NABP)) == (size_t) -1) > - DBUG_RETURN(1); > - } while ((bytes_in_cache= my_b_fill(cache))); > - if(cache->error == -1) > - DBUG_RETURN(1); > - DBUG_RETURN(0); > + > + for (i= 0, written_off_last_block= 0, bytes_in_cache= my_b_bytes_in_cache(cache); > + i < n_frag; > + i++, total_written += frag_written) > + { > + frag_written= 0; > + if (before_frag) > + { > + sprintf(buf, before_frag, i); > + my_fwrite(file, (uchar*) buf, strlen(buf), MYF(MY_WME | MY_NABP)); > + } > + do > + { > + /* > + Either the current block is the last (L) in making the > + current fragment (and possibly has some extra not to fit (LG) into > + the fragment), or (I) the current (whole then) block is > + intermediate. > + */ > + size_t block_to_write= (frag_written + bytes_in_cache >= frag_size) ? > + frag_size - frag_written : bytes_in_cache; > + > + DBUG_ASSERT(n_frag != 1 || > + (block_to_write == bytes_in_cache && > + written_off_last_block == 0)); > + > + if (my_fwrite(file, cache->read_pos + written_off_last_block, > + block_to_write, > + MYF(MY_WME | MY_NABP)) == (size_t) -1) > + /* no cache->error is set here */ > + DBUG_RETURN(1); > + > + frag_written += block_to_write; > + if (frag_written == frag_size) // (L) > + { > + DBUG_ASSERT(block_to_write <= bytes_in_cache); > + written_off_last_block= block_to_write; > + bytes_in_cache -= written_off_last_block; // (LG) when bytes>0 > + /* > + Nothing should be left in cache at the end of the > + last fragment construction. > + */ > + DBUG_ASSERT(i != n_frag - 1 || bytes_in_cache == 0); > + > + break; > + } > + else > + { > + written_off_last_block= 0; // (I) > + } > + } while ((bytes_in_cache= my_b_fill(cache))); > + > + if (after_frag) > + { > + sprintf(buf, after_frag, NULL); > + my_fwrite(file, (uchar*) buf, strlen(buf), MYF(MY_WME | MY_NABP)); > + } > + } > + > + DBUG_ASSERT(total_written == total_size); // output == input > + > + if (after_last) > + { > + sprintf(buf, after_last, n_frag); > + my_fwrite(file, (uchar*) buf, strlen(buf), MYF(MY_WME | MY_NABP)); > + } > + > + for (i= 0; final_per_frag && i < n_frag ; i++) > + { > + sprintf(buf, final_per_frag, i); > + my_fwrite(file, (uchar*) buf, strlen(buf), MYF(MY_WME | MY_NABP)); > + } > + > + DBUG_RETURN(cache->error == -1); this is a big and very specialized function with non-standard indentation. Please, move it from mysys to mysqlbinlog.cc and fix the indentation. > } > > > diff --git a/sql/log_event.cc b/sql/log_event.cc > index e1912ad4620..0c39200148f 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -10474,12 +10482,108 @@ void Rows_log_event::pack_info(Protocol *protocol) > #endif > > #ifdef MYSQL_CLIENT > +void copy_cache_to_file_wrapped(FILE *file, > + PRINT_EVENT_INFO *print_event_info, > + IO_CACHE *body, > + bool do_print_encoded) > +{ > + uint n_frag= 1; > + const char* before_frag= NULL; > + char* after_frag= NULL; > + char* after_last= NULL; > + char* final_per_frag= NULL; > + /* > + 2 fragments can always represent near 1GB row-based > + base64-encoded event as two strings each of size less than > + max(max_allowed_packet). Greater number of fragments does not > + save from potential need to tweak (increase) @@max_allowed_packet > + before to process the fragments. So 2 is safe and enough. > + */ > + const char fmt_last_frag2[]= > + "\nBINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_1)%s\n"; > + const char fmt_last_per_frag[]= "\nSET @binlog_fragment_%%d = NULL%s\n"; this is completely unnecessary "generalization", print simply "SET @binlog_fragment_0=NULL,binlog_fragment_1=NULL%s\n" in particular, as your "BINLOG DEFRAGMENT" statement is not generalized. (or to the cleanup inside BINLOG, as discussed above). > + const char fmt_before_frag[]= "\nSET @binlog_fragment_%d ='\n"; > + /* > + Buffer to pass to copy_cache_frag_to_file_and_reinit to > + compute formatted strings according to specifiers. > + The sizes may depend on an actual fragment number size in terms of decimal > + signs so its maximum is estimated (not precisely yet safely) below. > + */ > + char buf[(sizeof(fmt_last_frag2) + sizeof(fmt_last_per_frag)) > + + ((sizeof(n_frag) * 8)/3 + 1) // decimal index > + + sizeof(print_event_info->delimiter) + 3]; // delim, \n and 0. > + > + if (do_print_encoded) > + { > + after_frag= (char*) my_malloc(sizeof(buf), MYF(MY_WME)); > + sprintf(after_frag, "'%s\n", print_event_info->delimiter); > + if (my_b_tell(body) > opt_binlog_rows_event_max_encoded_size) > + n_frag= 2; > + if (n_frag > 1) > + { > + before_frag= fmt_before_frag; > + after_last= (char*) my_malloc(sizeof(buf), MYF(MY_WME)); > + sprintf(after_last, fmt_last_frag2, (char*) print_event_info->delimiter); > + final_per_frag= (char*) my_malloc(sizeof(buf), MYF(MY_WME)); > + sprintf(final_per_frag, fmt_last_per_frag, > + (char*) print_event_info->delimiter); > + } > + else > + { > + before_frag= "\nBINLOG '\n"; > + } > + } > + if (copy_cache_frag_to_file_and_reinit(body, file, n_frag, > + before_frag, after_frag, > + after_last, final_per_frag, buf)) 1. get rid of this copy_cache_frag_to_file_and_reinit function, it's not helping. 2. instead of pushing all this complex logic of printing fragments with before and after and in-between and start and final, it'd be *much* simpler to do it here and just extend my_b_copy_to_file() to print at most N bytes. > + { > + body->error= -1; > + goto err; > + } > + > +err: > + my_free(after_frag); > + my_free(after_last); > + my_free(final_per_frag); > +} > + > +/* > + The function invokes base64 encoder to run on the current > + event string and store the result into two caches. > + When the event ends the current statement the caches are is copied into > + the argument file. > + Copying is also concerned how to wrap the event, specifically to produce > + a valid SQL syntax. > + When the encoded data size is within max(MAX_ALLOWED_PACKET) > + a regular BINLOG query is composed. Otherwise it is build as fragmented > + > + SET @binlog_fragment_0='...'; > + SET @binlog_fragment_1='...'; > + BINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_1); > + > + where fragments are represented by a sequence of "indexed" user > + variables. > + Two more statements are composed as well > + > + SET @binlog_fragment_0=NULL; > + SET @binlog_fragment_1=NULL; > + > + to promptly release memory. > + > + NOTE. > + If any changes made don't forget to duplicate them to > + Old_rows_log_event as long as it's supported. > +*/ > void Rows_log_event::print_helper(FILE *file, > PRINT_EVENT_INFO *print_event_info, > char const *const name) > { > IO_CACHE *const head= &print_event_info->head_cache; > IO_CACHE *const body= &print_event_info->body_cache; > + bool do_print_encoded= > + print_event_info->base64_output_mode != BASE64_OUTPUT_DECODE_ROWS && > + !print_event_info->short_form; > + > if (!print_event_info->short_form) > { > bool const last_stmt_event= get_flags(STMT_END_F); > diff --git a/sql/sql_binlog.cc b/sql/sql_binlog.cc > index 91cf038907e..b15d72e036a 100644 > --- a/sql/sql_binlog.cc > +++ b/sql/sql_binlog.cc > @@ -28,6 +28,73 @@ > // START_EVENT_V3, > // Log_event_type, > // Log_event > + > +/* > + Copy fragments into the standard placeholder thd->lex->comment.str. > + > + Compute the size of the (still) encoded total, > + allocate and then copy fragments one after another. > + The size can exceed max(max_allowed_packet) which is not a > + problem as no String instance is created off this char array. > + > + Return 0 at success, -1 otherwise. > +*/ > +int binlog_defragment(THD *thd) > +{ > + LEX_STRING *curr_frag_name; > + > + thd->lex->comment.length= 0; > + thd->lex->comment.str= NULL; > + /* compute the total size */ > + for (uint i= 0; i < thd->lex->fragmented_binlog_event.n_frag; i++) > + { > + user_var_entry *entry; > + > + curr_frag_name= &thd->lex->fragmented_binlog_event.frag_name[i]; > + entry= > + (user_var_entry*) my_hash_search(&thd->user_vars, > + (uchar*) curr_frag_name->str, > + curr_frag_name->length); > + if (!entry || entry->type != STRING_RESULT) > + { > + my_printf_error(ER_BASE64_DECODE_ERROR, > + "%s: BINLOG fragment user " > + "variable '%s' has unexpectedly no value", MYF(0), > + ER_THD(thd, ER_BASE64_DECODE_ERROR), curr_frag_name->str); > + return -1; > + } > + thd->lex->comment.length += entry->length; > + } > + thd->lex->comment.str= // to be freed by the caller > + (char *) my_malloc(thd->lex->comment.length, MYF(MY_WME)); > + if (!thd->lex->comment.str) > + { > + my_error(ER_OUTOFMEMORY, MYF(ME_FATALERROR), 1); > + return -1; > + } > + > + /* fragments are merged into allocated buf */ > + size_t gathered_length= 0; > + for (uint i= 0; i < thd->lex->fragmented_binlog_event.n_frag; i++) > + { > + user_var_entry *entry; > + > + curr_frag_name= &thd->lex->fragmented_binlog_event.frag_name[i]; > + entry= > + (user_var_entry*) my_hash_search(&thd->user_vars, > + (uchar*) curr_frag_name->str, > + curr_frag_name->length); > + memcpy(thd->lex->comment.str + gathered_length, > + entry->value, entry->length); > + > + gathered_length += entry->length; > + } > + DBUG_ASSERT(gathered_length == thd->lex->comment.length); ok. 1. BINLOG uses LEX::comment to avoid increasing sizeof(LEX) for a rarely used statement. So don't increase it for a much-more-rarely used event overlow case. That is, remove fragmented_binlog_event. You can use a pair of LEX_STRING's in LEX (e.g. comment/ident) or (less hackish) List<LEX_STRING>, which are plenty in LEX. 2. Don't look up variables twice, remember them in the first loop. You can unroll it too, there're only two variables anyway. E.g. the second loop can be replaced by just memcpy(thd->lex->comment.str, var[0]->value, var[0]->length); memcpy(thd->lex->comment.str + var[0]->length, var[1]->value, var[1]->length); or even shorter, if you start your function with LEX_STRING *out= &thd->lex->comment; 3. Not "unexpectedly no value". ER_WRONG_TYPE_FOR_VAR looks more appropriate there. > + > + return 0; > +} > + > + > /** > Execute a BINLOG statement. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 4

[Maria-developers] MDEV-13912
by Michal Schorm 22 Oct '18

22 Oct '18

Hello, can I get some status update on MDEV-13912 please? It is still an issue for the customer on my side and I'd love to see fixed it in 5.5.62 release. Michal -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

2 2

Re: [Maria-developers] cf9bafc: MDEV-14815 - Server crash or AddressSanitizer errors or valgrind warnings
by Sergei Golubchik 19 Oct '18

19 Oct '18

Hi, Sergey! On Oct 19, Sergey Vojtovich wrote: > revision-id: cf9bafc8cb0943c914d118525888140ea15e6b55 (mariadb-10.0.36-50-gcf9bafc) > parent(s): 8e716138cef2d9be603cdf71701d82bcb72dfd69 > author: Sergey Vojtovich > committer: Sergey Vojtovich > timestamp: 2018-10-18 23:50:50 +0400 > message: > > MDEV-14815 - Server crash or AddressSanitizer errors or valgrind warnings > in thr_lock / has_old_lock upon FLUSH TABLES > > Explicit partition access of partitioned MEMORY table under LOCK TABLES > may cause subsequent statements to crash the server, deadlock, trigger > valgrind warnings or ASAN errors. Freed memory was being used due to > incorrect cleanup. > > At least MyISAM and InnoDB don't seem to be affected, since their > THR_LOCK structures don't survive FLUSH TABLES. MEMORY keeps table shared > data (including THR_LOCK) even if there're no open instances. > > There's partition_info::lock_partitions bitmap, which holds bits of > partitions allowed to be accessed after pruning. This bitmap is > updated for each individual statement. > > This bitmap was abused in ha_partition::store_lock() such that when we > need to unlock a table, locked by LOCK TABLES, only locks for partitions > that were accessed by previous statement were released. > > Eventually FLUSH TABLES frees THR_LOCK_DATA objects, which are still > linked into THR_LOCK lists. When such THR_LOCK gets reused we end up with > freed memory access. > > Fixed by using ha_partition::m_locked_partitions bitmap similarly to > ha_partition::external_lock(). Thanks, good comment. > Some unused code removed. Generally, it's better to do that in a separate commit. It's very easy to do with git citool, and doesn't requite any advance planning, you just edit whatever you like and then run 'git citool' twice, selecting what lines you want to commit each time. > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 0488ebf..02de92a 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -3907,9 +3907,14 @@ THR_LOCK_DATA **ha_partition::store_lock(THD *thd, > } > else > { > - for (i= bitmap_get_first_set(&(m_part_info->lock_partitions)); > + MY_BITMAP *used_partitions= lock_type == TL_UNLOCK || > + lock_type == TL_IGNORE ? why TL_IGNORE too? external_lock only checks for TL_UNLOCK. > + &m_locked_partitions : > + &m_part_info->lock_partitions; > + > + for (i= bitmap_get_first_set(used_partitions); > i < m_tot_parts; > - i= bitmap_get_next_set(&m_part_info->lock_partitions, i)) > + i= bitmap_get_next_set(used_partitions, i)) > { > DBUG_PRINT("info", ("store lock %d iteration", i)); > to= m_file[i]->store_lock(thd, to, lock_type); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] 7ce7184cf0e: MDEV-17133 dump thread reads from a past position
by Sergei Golubchik 19 Oct '18

19 Oct '18

Hi, Andrei! ok to push. But please add a meaningful commit comment. Thanks! On Oct 19, Andrei Elkin wrote: > revision-id: 7ce7184cf0e354a1c9114bb31df1fb63b1fbbcc7 (mariadb-10.1.35-59-g7ce7184cf0e) > parent(s): d3a8b5aa9cee24a6397662e30df2e915f45460e0 > author: Andrei Elkin <andrei.elkin(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2018-09-18 23:21:18 +0300 > message: > > MDEV-17133 dump thread reads from a past position > > --- > mysys/mf_iocache.c | 8 +++-- > unittest/sql/mf_iocache-t.cc | 74 +++++++++++++++++++++++++++++++++++++++++++- > 2 files changed, 79 insertions(+), 3 deletions(-) > > diff --git a/mysys/mf_iocache.c b/mysys/mf_iocache.c > index 56b1ae3fc6e..8f829a8c9c2 100644 > --- a/mysys/mf_iocache.c > +++ b/mysys/mf_iocache.c > @@ -563,7 +563,7 @@ int _my_b_write(IO_CACHE *info, const uchar *Buffer, size_t Count) > > int _my_b_cache_read(IO_CACHE *info, uchar *Buffer, size_t Count) > { > - size_t length, diff_length, left_length= 0, max_length; > + size_t length= 0, diff_length, left_length= 0, max_length; > my_off_t pos_in_file; > DBUG_ENTER("_my_b_cache_read"); > > @@ -668,7 +668,11 @@ int _my_b_cache_read(IO_CACHE *info, uchar *Buffer, size_t Count) > else > { > info->error= 0; > - DBUG_RETURN(0); /* EOF */ > + if (length == 0) > + { > + DBUG_RETURN(0); // EOF > + } > + length= 0; > } > } > else if ((length= mysql_file_read(info->file,info->buffer, max_length, Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Correct parsing of BINLOG 'XXX' with comment inside
by Sachin Setia 19 Oct '18

19 Oct '18

Hi Bar, Currently if we have generated the sql file using mysqlbinlog -v and if we have big statement binlog , then mysqlbinlog add comment in between of Binlog 'XXX' statement , but unfortunately base64_decode does not understands it, and it through error since # is not a base64 char This patches solves this. There are 2 patches one for 5.5 and 2nd for 10.0 and above Please review it.

4 7

Re: [Maria-developers] 9b453c1ebf4: MDEV-10963 Fragmented BINLOG query
by Sergei Golubchik 18 Oct '18

18 Oct '18

Hi, Andrei! On Sep 07, andrei.elkin(a)pp.inet.fi wrote: > revision-id: 9b453c1ebf47a20ea1436de3284f8810d0e64c4c (mariadb-10.1.34-8-g9b453c1ebf4) > parent(s): c09a8b5b36edb494e2bcc93074c06e26cd9f2b92 > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-09-07 20:36:16 +0300 > message: > > MDEV-10963 Fragmented BINLOG query > > The problem was originally stated in > http://bugs.mysql.com/bug.php?id=82212 > The size of an base64-encoded Rows_log_event exceeds its > vanilla byte representation in 4/3 times. > When a binlogged event size is about 1GB mysqlbinlog generates > a BINLOG query that can't be send out due to its size. > > It is fixed with fragmenting the BINLOG argument C-string into > (approximate) halves when the base64 encoded event is over 1GB size. > The mysqlbinlog in such case puts out > > SET @binlog_fragment_0='base64-encoded-fragment_0'; > SET @binlog_fragment_1='base64-encoded-fragment_1'; > BINLOG 2, 'binlog_fragment'; No, that's totally ridiculous. A statement that internally walks the list of user variables and collects values from variables named according to a specific pattern, seriously? Please, make it BINLOG CONCAT(@binlog_fragment_0, @binlog_fragment_1) that'll work with no questions asked, everybody understands what it means. The parser doesn't need to accept an arbitrary expression here, it'd be simpler and safer to hard-code the syntax as above. A couple of minor generic comments below, but I've stopped reviewing, because the code will change anyway. > to represent a big BINLOG 'base64-encoded-"total"'. > Two more statements are composed to promptly release memory > SET @binlog_fragment_0=NULL; > SET @binlog_fragment_1=NULL; > > diff --git a/client/mysqlbinlog.cc b/client/mysqlbinlog.cc > index 9753125dd67..f0689631cfb 100644 > --- a/client/mysqlbinlog.cc > +++ b/client/mysqlbinlog.cc > @@ -56,7 +56,13 @@ Rpl_filter *binlog_filter= 0; > > #define BIN_LOG_HEADER_SIZE 4 > #define PROBE_HEADER_LEN (EVENT_LEN_OFFSET+4) > - > +/* > + 2 fragments can always represent near 1GB row-based base64-encoded event as > + two strings each of size less than max(max_allowed_packet). > + Bigger number of fragments does not safe from potential need to tune (increase) > + @@max_allowed_packet before to process the fragments. So 2 is safe and enough. > +*/ > +#define BINLOG_ROWS_EVENT_ENCODED_FRAGMENTS 2 > > #define CLIENT_CAPABILITIES (CLIENT_LONG_PASSWORD | CLIENT_LONG_FLAG | CLIENT_LOCAL_FILES) > > @@ -71,6 +77,9 @@ ulong bytes_sent = 0L, bytes_received = 0L; > ulong mysqld_net_retry_count = 10L; > ulong open_files_limit; > ulong opt_binlog_rows_event_max_size; > +#ifndef DBUG_OFF > +ulong opt_binlog_rows_event_max_encoded_size; > +#endif Try to avoid unnecessary ifdefs. Here you could've defined the variable unconditionally and suppressed the warning with __attribute__((unused)), of you could've used it unconditionally as a splitting threshold below and only hidden the command line option. > uint test_flags = 0; > static uint opt_protocol= 0; > static FILE *result_file; > @@ -813,7 +822,14 @@ write_event_header_and_base64(Log_event *ev, FILE *result_file, > > /* Write header and base64 output to cache */ > ev->print_header(head, print_event_info, FALSE); > - ev->print_base64(body, print_event_info, FALSE); > + > + /* the assert states the only current use case for the function */ > + DBUG_ASSERT(print_event_info->base64_output_mode == > + BASE64_OUTPUT_ALWAYS); 1. don't document the obvious 2. don't wrap the line if it fits in 80-char line > + > + ev->print_base64(body, print_event_info, > + print_event_info->base64_output_mode != > + BASE64_OUTPUT_DECODE_ROWS); > > /* Read data from cache and write to result file */ > if (copy_event_cache_to_file_and_reinit(head, result_file) || > @@ -1472,6 +1490,15 @@ that may lead to an endless loop.", > "This value must be a multiple of 256.", > &opt_binlog_rows_event_max_size, &opt_binlog_rows_event_max_size, 0, > GET_ULONG, REQUIRED_ARG, UINT_MAX, 256, ULONG_MAX, 0, 256, 0}, > +#ifndef DBUG_OFF > + {"binlog-row-event-max-encoded-size", 0, all debug command-line options must start with "debug-", that is, either rename it to "debug-binlog-row-event-max-encoded-size" or make it non-debug. > + "The maximum size of base64-encoded rows-event in one BINLOG pseudo-query " > + "instance. When the computed actual size exceeds the limit " > + "the BINLOG's argument string is fragmented in two.", > + &opt_binlog_rows_event_max_encoded_size, > + &opt_binlog_rows_event_max_encoded_size, 0, > + GET_ULONG, REQUIRED_ARG, UINT_MAX/4, 256, ULONG_MAX, 0, 256, 0}, > +#endif > {"verify-binlog-checksum", 'c', "Verify checksum binlog events.", > (uchar**) &opt_verify_binlog_checksum, (uchar**) &opt_verify_binlog_checksum, > 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] [Commits] 2f4a0c5be2c: Fix accumulation of old rows in mysql.gtid_slave_pos
by Kristian Nielsen 13 Oct '18

13 Oct '18

Hi Andrei! Can you review this patch? This fixes a rather serious bug in parallel replication that was reported on maria-discuss@ here: https://lists.launchpad.net/maria-discuss/msg05253.html The user ended up with millions of rows in mysql.gtid_slave_pos, which is quite bad. I think this should go in 10.1. The bug is probably also present in 10.0, but it will be much harder to trigger since 10.0 does not have optimistic parallel replication. So leaving it unfixed in 10.0 seems ok. I plan to manually merge it to 10.2 and 10.3 as well, since the merge to 10.3 is not entirely trivial, due to interaction with @@gtid_pos_auto_engines. Github branch here: https://github.com/knielsen/server/commits/gtid_table_garbage_rows - Kristian. Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > revision-id: 2f4a0c5be2c5d5153c4253a49ba8820ab333a9a0 (mariadb-10.1.35-71-g2f4a0c5be2c) > parent(s): 1fc5a6f30c3a9c047dcf9a36b00026d98f286f6b > author: Kristian Nielsen > committer: Kristian Nielsen > timestamp: 2018-10-07 18:59:52 +0200 > message: > > Fix accumulation of old rows in mysql.gtid_slave_pos > > This would happen especially in optimistic parallel replication, where there > is a good chance that a transaction will be rolled back (due to conflicts) > after it has executed record_gtid(). If the transaction did any deletions of > old rows as part of record_gtid(), those deletions will be undone as well. > And the code did not properly ensure that the deletions would be re-tried. > > This patch makes record_gtid() remember the list of deletions done as part > of a transaction. Then in rpl_slave_state::update() when the changes have > been committed, we discard the list. However, in case of error and rollback, > in cleanup_context() we will instead put the list back into > rpl_global_gtid_slave_state so that the deletions will be re-tried later. > > Probably fixes part of the cause of MDEV-12147 as well. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > > --- > .../suite/rpl/r/rpl_parallel_optimistic.result | 6 ++ > .../suite/rpl/t/rpl_parallel_optimistic.test | 17 +++++ > sql/log_event.cc | 6 +- > sql/rpl_gtid.cc | 64 ++++++++++++------ > sql/rpl_gtid.h | 2 +- > sql/rpl_rli.cc | 79 ++++++++++++++++++++++ > sql/rpl_rli.h | 11 +++ > 7 files changed, 161 insertions(+), 24 deletions(-) > > diff --git a/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result b/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > index 3cd4f8231bf..99bd8562ffe 100644 > --- a/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > +++ b/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > @@ -571,4 +571,10 @@ SET GLOBAL slave_parallel_mode=@old_parallel_mode; > SET GLOBAL slave_parallel_threads=@old_parallel_threads; > include/start_slave.inc > DROP TABLE t1, t2, t3; > +include/save_master_gtid.inc > +include/sync_with_master_gtid.inc > +Check that no more than the expected last two GTIDs are in mysql.gtid_slave_pos > +select count(*) from mysql.gtid_slave_pos order by domain_id, sub_id; > +count(*) > +2 > include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test b/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > index 9f6669279db..3867a3fdf3a 100644 > --- a/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > +++ b/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > @@ -549,5 +549,22 @@ SET GLOBAL slave_parallel_threads=@old_parallel_threads; > > --connection server_1 > DROP TABLE t1, t2, t3; > +--source include/save_master_gtid.inc > + > +--connection server_2 > +--source include/sync_with_master_gtid.inc > +# Check for left-over rows in table mysql.gtid_slave_pos (MDEV-12147). > +# > +# There was a bug when a transaction got a conflict and was rolled back. It > +# might have also handled deletion of some old rows, and these deletions would > +# then also be rolled back. And since the deletes were never re-tried, old no > +# longer needed rows would accumulate in the table without limit. > +# > +# The earlier part of this test file have plenty of transactions being rolled > +# back. But the last DROP TABLE statement runs on its own and should never > +# conflict, thus at this point the mysql.gtid_slave_pos table should be clean. > +--echo Check that no more than the expected last two GTIDs are in mysql.gtid_slave_pos > +select count(*) from mysql.gtid_slave_pos order by domain_id, sub_id; > > +--connection server_1 > --source include/rpl_end.inc > diff --git a/sql/log_event.cc b/sql/log_event.cc > index e1912ad4620..e07b7002398 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -4429,7 +4429,7 @@ int Query_log_event::do_apply_event(rpl_group_info *rgi, > > gtid= rgi->current_gtid; > if (rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, > - true, false)) > + rgi, false)) > { > int errcode= thd->get_stmt_da()->sql_errno(); > if (!is_parallel_retry_error(rgi, errcode)) > @@ -7132,7 +7132,7 @@ Gtid_list_log_event::do_apply_event(rpl_group_info *rgi) > { > if ((ret= rpl_global_gtid_slave_state->record_gtid(thd, &list[i], > sub_id_list[i], > - false, false))) > + NULL, false))) > return ret; > rpl_global_gtid_slave_state->update_state_hash(sub_id_list[i], &list[i], > NULL); > @@ -7639,7 +7639,7 @@ int Xid_log_event::do_apply_event(rpl_group_info *rgi) > rgi->gtid_pending= false; > > gtid= rgi->current_gtid; > - err= rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, true, > + err= rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, rgi, > false); > if (err) > { > diff --git a/sql/rpl_gtid.cc b/sql/rpl_gtid.cc > index 7b1acf17ef5..94944b5b3e5 100644 > --- a/sql/rpl_gtid.cc > +++ b/sql/rpl_gtid.cc > @@ -77,7 +77,7 @@ rpl_slave_state::record_and_update_gtid(THD *thd, rpl_group_info *rgi) > rgi->gtid_pending= false; > if (rgi->gtid_ignore_duplicate_state!=rpl_group_info::GTID_DUPLICATE_IGNORE) > { > - if (record_gtid(thd, &rgi->current_gtid, sub_id, false, false)) > + if (record_gtid(thd, &rgi->current_gtid, sub_id, NULL, false)) > DBUG_RETURN(1); > update_state_hash(sub_id, &rgi->current_gtid, rgi); > } > @@ -328,6 +328,8 @@ rpl_slave_state::update(uint32 domain_id, uint32 server_id, uint64 sub_id, > } > } > rgi->gtid_ignore_duplicate_state= rpl_group_info::GTID_DUPLICATE_NULL; > + > + rgi->pending_gtid_deletes_clear(); > } > > if (!(list_elem= (list_element *)my_malloc(sizeof(*list_elem), MYF(MY_WME)))) > @@ -377,15 +379,24 @@ int > rpl_slave_state::put_back_list(uint32 domain_id, list_element *list) > { > element *e; > + int err= 0; > + > + mysql_mutex_lock(&LOCK_slave_state); > if (!(e= (element *)my_hash_search(&hash, (const uchar *)&domain_id, 0))) > - return 1; > + { > + err= 1; > + goto end; > + } > while (list) > { > list_element *next= list->next; > e->add(list); > list= next; > } > - return 0; > + > +end: > + mysql_mutex_unlock(&LOCK_slave_state); > + return err; > } > > > @@ -468,12 +479,12 @@ gtid_check_rpl_slave_state_table(TABLE *table) > /* > Write a gtid to the replication slave state table. > > - Do it as part of the transaction, to get slave crash safety, or as a separate > - transaction if !in_transaction (eg. MyISAM or DDL). > - > gtid The global transaction id for this event group. > sub_id Value allocated within the sub_id when the event group was > read (sub_id must be consistent with commit order in master binlog). > + rgi rpl_group_info context, if we are recording the gtid transactionally > + as part of replicating a transactional event. NULL if called from > + outside of a replicated transaction. > > Note that caller must later ensure that the new gtid and sub_id is inserted > into the appropriate HASH element with rpl_slave_state.add(), so that it can > @@ -481,13 +492,13 @@ gtid_check_rpl_slave_state_table(TABLE *table) > */ > int > rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > - bool in_transaction, bool in_statement) > + rpl_group_info *rgi, bool in_statement) > { > TABLE_LIST tlist; > int err= 0; > bool table_opened= false; > TABLE *table; > - list_element *elist= 0, *next; > + list_element *elist= 0, *cur, *next; > element *elem; > ulonglong thd_saved_option= thd->variables.option_bits; > Query_tables_list lex_backup; > @@ -558,7 +569,7 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > thd->wsrep_ignore_table= true; > #endif > > - if (!in_transaction) > + if (!rgi) > { > DBUG_PRINT("info", ("resetting OPTION_BEGIN")); > thd->variables.option_bits&= > @@ -601,9 +612,9 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > if ((elist= elem->grab_list()) != NULL) > { > /* Delete any old stuff, but keep around the most recent one. */ > - list_element *cur= elist; > - uint64 best_sub_id= cur->sub_id; > + uint64 best_sub_id= elist->sub_id; > list_element **best_ptr_ptr= &elist; > + cur= elist; > while ((next= cur->next)) > { > if (next->sub_id > best_sub_id) > @@ -636,7 +647,8 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > table->file->print_error(err, MYF(0)); > goto end; > } > - while (elist) > + cur = elist; > + while (cur) > { > uchar key_buffer[4+8]; > > @@ -646,9 +658,9 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > /* `break' does not work inside DBUG_EXECUTE_IF */ > goto dbug_break; }); > > - next= elist->next; > + next= cur->next; > > - table->field[1]->store(elist->sub_id, true); > + table->field[1]->store(cur->sub_id, true); > /* domain_id is already set in table->record[0] from write_row() above. */ > key_copy(key_buffer, table->record[0], &table->key_info[0], 0, false); > if (table->file->ha_index_read_map(table->record[1], key_buffer, > @@ -662,8 +674,7 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > not want to endlessly error on the same element in case of table > corruption or such. > */ > - my_free(elist); > - elist= next; > + cur= next; > if (err) > break; > } > @@ -686,18 +697,31 @@ IF_DBUG(dbug_break:, ) > */ > if (elist) > { > - mysql_mutex_lock(&LOCK_slave_state); > put_back_list(gtid->domain_id, elist); > - mysql_mutex_unlock(&LOCK_slave_state); > + elist = 0; > } > > ha_rollback_trans(thd, FALSE); > } > close_thread_tables(thd); > - if (in_transaction) > + if (rgi) > + { > thd->mdl_context.release_statement_locks(); > + /* > + Save the list of old gtid entries we deleted. If this transaction > + fails later for some reason and is rolled back, the deletion of those > + entries will be rolled back as well, and we will need to put them back > + on the to-be-deleted list so we can re-do the deletion. Otherwise > + redundant rows in mysql.gtid_slave_pos may accumulate if transactions > + are rolled back and retried after record_gtid(). > + */ > + rgi->pending_gtid_deletes_save(gtid->domain_id, elist); > + } > else > + { > thd->mdl_context.release_transactional_locks(); > + rpl_group_info::pending_gtid_deletes_free(elist); > + } > } > thd->lex->restore_backup_query_tables_list(&lex_backup); > thd->variables.option_bits= thd_saved_option; > @@ -1080,7 +1104,7 @@ rpl_slave_state::load(THD *thd, char *state_from_master, size_t len, > > if (gtid_parser_helper(&state_from_master, end, &gtid) || > !(sub_id= next_sub_id(gtid.domain_id)) || > - record_gtid(thd, &gtid, sub_id, false, in_statement) || > + record_gtid(thd, &gtid, sub_id, NULL, in_statement) || > update(gtid.domain_id, gtid.server_id, sub_id, gtid.seq_no, NULL)) > return 1; > if (state_from_master == end) > diff --git a/sql/rpl_gtid.h b/sql/rpl_gtid.h > index 79d566bddbf..7bd639b768f 100644 > --- a/sql/rpl_gtid.h > +++ b/sql/rpl_gtid.h > @@ -182,7 +182,7 @@ struct rpl_slave_state > uint64 seq_no, rpl_group_info *rgi); > int truncate_state_table(THD *thd); > int record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > - bool in_transaction, bool in_statement); > + rpl_group_info *rgi, bool in_statement); > uint64 next_sub_id(uint32 domain_id); > int iterate(int (*cb)(rpl_gtid *, void *), void *data, > rpl_gtid *extra_gtids, uint32 num_extra, > diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc > index 64a1b535307..b35130c1505 100644 > --- a/sql/rpl_rli.cc > +++ b/sql/rpl_rli.cc > @@ -1680,6 +1680,7 @@ rpl_group_info::reinit(Relay_log_info *rli) > long_find_row_note_printed= false; > did_mark_start_commit= false; > gtid_ev_flags2= 0; > + pending_gtid_delete_list= NULL; > last_master_timestamp = 0; > gtid_ignore_duplicate_state= GTID_DUPLICATE_NULL; > speculation= SPECULATE_NO; > @@ -1804,6 +1805,12 @@ void rpl_group_info::cleanup_context(THD *thd, bool error) > erroneously update the GTID position. > */ > gtid_pending= false; > + > + /* > + Rollback will have undone any deletions of old rows we might have made > + in mysql.gtid_slave_pos. Put those rows back on the list to be deleted. > + */ > + pending_gtid_deletes_put_back(); > } > m_table_map.clear_tables(); > slave_close_thread_tables(thd); > @@ -2027,6 +2034,78 @@ rpl_group_info::unmark_start_commit() > } > > > +/* > + When record_gtid() has deleted any old rows from the table > + mysql.gtid_slave_pos as part of a replicated transaction, save the list of > + rows deleted here. > + > + If later the transaction fails (eg. optimistic parallel replication), the > + deletes will be undone when the transaction is rolled back. Then we can > + put back the list of rows into the rpl_global_gtid_slave_state, so that > + we can re-do the deletes and avoid accumulating old rows in the table. > +*/ > +void > +rpl_group_info::pending_gtid_deletes_save(uint32 domain_id, > + rpl_slave_state::list_element *list) > +{ > + /* > + We should never get to a state where we try to save a new pending list of > + gtid deletes while we still have an old one. But make sure we handle it > + anyway just in case, so we avoid leaving stray entries in the > + mysql.gtid_slave_pos table. > + */ > + DBUG_ASSERT(!pending_gtid_delete_list); > + if (unlikely(pending_gtid_delete_list)) > + pending_gtid_deletes_put_back(); > + > + pending_gtid_delete_list= list; > + pending_gtid_delete_list_domain= domain_id; > +} > + > + > +/* > + Take the list recorded by pending_gtid_deletes_save() and put it back into > + rpl_global_gtid_slave_state. This is needed if deletion of the rows was > + rolled back due to transaction failure. > +*/ > +void > +rpl_group_info::pending_gtid_deletes_put_back() > +{ > + if (pending_gtid_delete_list) > + { > + rpl_global_gtid_slave_state->put_back_list(pending_gtid_delete_list_domain, > + pending_gtid_delete_list); > + pending_gtid_delete_list= NULL; > + } > +} > + > + > +/* > + Free the list recorded by pending_gtid_deletes_save(). Done when the deletes > + in the list have been permanently committed. > +*/ > +void > +rpl_group_info::pending_gtid_deletes_clear() > +{ > + pending_gtid_deletes_free(pending_gtid_delete_list); > + pending_gtid_delete_list= NULL; > +} > + > + > +void > +rpl_group_info::pending_gtid_deletes_free(rpl_slave_state::list_element *list) > +{ > + rpl_slave_state::list_element *next; > + > + while (list) > + { > + next= list->next; > + my_free(list); > + list= next; > + } > +} > + > + > rpl_sql_thread_info::rpl_sql_thread_info(Rpl_filter *filter) > : rpl_filter(filter) > { > diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h > index 74d5b6fe416..b40a34a54e6 100644 > --- a/sql/rpl_rli.h > +++ b/sql/rpl_rli.h > @@ -676,6 +676,11 @@ struct rpl_group_info > /* Needs room for "Gtid D-S-N\x00". */ > char gtid_info_buf[5+10+1+10+1+20+1]; > > + /* List of not yet committed deletions in mysql.gtid_slave_pos. */ > + rpl_slave_state::list_element *pending_gtid_delete_list; > + /* Domain associated with pending_gtid_delete_list. */ > + uint32 pending_gtid_delete_list_domain; > + > /* > The timestamp, from the master, of the commit event. > Used to do delayed update of rli->last_master_timestamp, for getting > @@ -817,6 +822,12 @@ struct rpl_group_info > char *gtid_info(); > void unmark_start_commit(); > > + static void pending_gtid_deletes_free(rpl_slave_state::list_element *list); > + void pending_gtid_deletes_save(uint32 domain_id, > + rpl_slave_state::list_element *list); > + void pending_gtid_deletes_put_back(); > + void pending_gtid_deletes_clear(); > + > time_t get_row_stmt_start_timestamp() > { > return row_stmt_start_timestamp; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 5

[Maria-developers] SP memory usage
by jerome brauge 12 Oct '18

12 Oct '18

Hello Alexander, As stored procedures seems to be very memory aggressive, I found two possible optimizations to avoid having three strings containing the stored procedure code in memory. Probably, same optimizations may be applied to packages. I haven't add any tests because I think everything is covered by the existing tests. What do you think about ? In a second time, I think working on SP cache. Currently it' s a simple hash table and all the hash table is flushed as soon as the threshold (stored_program_cache) is reached. It's not very good. We could : - Use a more intelligent cache (LFU ?) - Add some performance counters Regards. Ce message et les pi?ces jointes sont confidentiels et ?tablis ? l'attention exclusive de ses destinataires. Toute utilisation ou diffusion, m?me partielle, non autoris?e est interdite. Tout message ?lectronique est susceptible d'alt?ration; CEGID d?cline donc toute responsabilit? au titre de ce message. Si vous n'?tes pas le destinataire de ce message, merci de le d?truire et d'avertir l'exp?diteur. This message and any attachments are confidential and intended solely for the addressees. Any unauthorized use or disclosure, either whole or partial is prohibited. E-mails are susceptible to alteration; CEGID shall therefore not be liable for the content of this message. If you are not the intended recipient of this message, please delete it and notify the sender.

1 0

[Maria-developers] [beginner] how can I assign an existing ticket to myself?
by Takashi Sasaki 12 Oct '18

12 Oct '18

Hello, I wanted to contribute something to MariaDB, so I signed up to Jira, but how can I assign an existing ticket to myself? [programming experience]: I have been programming for over 10 years but recently using Java, Python, Node.js at work and C ++ is a little played a few years ago. [knowledge of the MariaDB source] I just started reading the code recently and I am almost an amateur. [how much you know about using MySQL/MariaDB] I use MySQL for WordPress backend as a user and I use Oracle for work, so I have knowledge of general RDBMS, but I do not have deep knowledge specific to the products. Thanks, Takashi Sasaki

3 2

Re: [Maria-developers] [Commits] 1edec241473: MDEV-17137: Syntax errors with VIEW using MEDIAN
by Sergey Petrunia 11 Oct '18

11 Oct '18

Hi Varun, On Wed, Sep 12, 2018 at 06:13:23PM +0530, Varun wrote: > revision-id: 1edec241473a122553fa9a27be995f3aef52654e (mariadb-10.3.7-171-g1edec241473) > parent(s): 5a1868b58d26b286b6ad433096e7184895953311 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-09-12 18:10:16 +0530 > message: > > MDEV-17137: Syntax errors with VIEW using MEDIAN > > The syntax error happened because we had not implemented a different print for > percentile functions. The syntax is a bit different when we use percentile functions > as window functions in comparision to normal window functions. > Implemented a seperate print function for percentile functions > ... > diff --git a/sql/item_windowfunc.h b/sql/item_windowfunc.h > index b3e23748246..268182a2788 100644 > --- a/sql/item_windowfunc.h > +++ b/sql/item_windowfunc.h > @@ -1311,6 +1311,7 @@ class Item_window_func : public Item_func_or_sum > bool resolve_window_name(THD *thd); > > void print(String *str, enum_query_type query_type); > + void print_percentile_functions(String *str, enum_query_type query_type); > * Please make this function private * I think "print_percentile_functions" is a bad name as it implies multiple functions are printed. In fact, only one function is printed. How about "print_for_percentile_functions"? Ok to push after the above is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] bb2db687a05: 26 aug (MDEV-371 Unique indexes for blobs)
by Sergei Golubchik 10 Oct '18

10 Oct '18

Hi, Sachin! Could you please explain what you store in the frm file and how you modify keyinfo/keypart structures in memory? Needs to be tested with partitioned tables. Particularly with MDEV-14005. Tests were great and thourough, good job. It'd be useful to have at least some tests for innodb too, though. See other comments below. On Oct 08, Sachin Setiya wrote: > diff --git a/mysql-test/main/long_unique.result b/mysql-test/main/long_unique.result > new file mode 100644 > index 00000000000..1a06c230c72 > --- /dev/null > +++ b/mysql-test/main/long_unique.result > @@ -0,0 +1,1391 @@ > +#Structure of tests > +#First we will check all option for > +#table containing single unique column > +#table containing keys like unique(a,b,c,d) etc > +#then table containing 2 blob unique etc > +set @allowed_packet= @@max_allowed_packet; > +#table with single long blob column; > +create table t1(a blob unique); > +insert into t1 values(1),(2),(3),(56),('sachin'),('maria'),(123456789034567891),(null),(null),(123456789034567890); > +#table structure; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table t1 > +Non_unique 0 > +Key_name a > +Seq_in_index 1 > +Column_name a > +Collation A > +Cardinality NULL > +Sub_part NULL > +Packed NULL > +Null YES > +Index_type HASH_INDEX I'm not sure about it. One cannot write UNIQUE (a) USING HASH_INDEX in CREATE TABLE, right? May be just report HASH here? (and fix the parser to behave accordingly) > +Comment > +Index_comment > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 10 Deleted blocks: 0 > +Recordlength: 20 > + > +table description: > +Key Start Len Index Type > +1 12 8 multip. ulonglong NULL good idea > +select * from information_schema.columns where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +def test t1 a 1 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob UNI select,insert,update,references NEVER NULL > +select * from information_schema.statistics where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME NON_UNIQUE INDEX_SCHEMA INDEX_NAME SEQ_IN_INDEX COLUMN_NAME COLLATION CARDINALITY SUB_PART PACKED NULLABLE INDEX_TYPE COMMENT INDEX_COMMENT > +def test t1 0 test a 1 a A NULL NULL NULL YES HASH_INDEX > +select * from information_schema.key_column_usage where table_schema= 'test' and table_name= 't1'; > +CONSTRAINT_CATALOG def > +CONSTRAINT_SCHEMA test > +CONSTRAINT_NAME a > +TABLE_CATALOG def > +TABLE_SCHEMA test > +TABLE_NAME t1 > +COLUMN_NAME a > +ORDINAL_POSITION 1 > +POSITION_IN_UNIQUE_CONSTRAINT NULL > +REFERENCED_TABLE_SCHEMA NULL > +REFERENCED_TABLE_NAME NULL > +REFERENCED_COLUMN_NAME NULL > +# table select we should not be able to see db_row_hash_column; > +select * from t1; > +a > +1 > +2 > +3 > +56 > +sachin > +maria > +123456789034567891 > +NULL > +NULL > +123456789034567890 > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +#duplicate entry test; > +insert into t1 values(2); > +ERROR 23000: Duplicate entry '2' for key 'a' > +insert into t1 values('sachin'); > +ERROR 23000: Duplicate entry 'sachin' for key 'a' > +insert into t1 values(123456789034567891); > +ERROR 23000: Duplicate entry '123456789034567891' for key 'a' looks great :) > +select * from t1; > +a > +1 > +2 > +3 > +56 > +sachin > +maria > +123456789034567891 > +NULL > +NULL > +123456789034567890 > +insert into t1 values(11),(22),(33); > +insert into t1 values(12),(22); > +ERROR 23000: Duplicate entry '22' for key 'a' > +select * from t1; > +a > +1 > +2 > +3 > +56 > +sachin > +maria > +123456789034567891 > +NULL > +NULL > +123456789034567890 > +11 > +22 > +33 > +12 > +insert into t1 values(repeat('s',4000*10)),(repeat('s',4001*10)); > +insert into t1 values(repeat('m',4000*10)),(repeat('m',4000*10)); > +ERROR 23000: Duplicate entry 'mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm' for key 'a' > +insert into t1 values(repeat('m',4001)),(repeat('m',4002)); > +truncate table t1; > +insert into t1 values(1),(2),(3),(4),(5),(8),(7); > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 7 Deleted blocks: 0 > +Recordlength: 20 > + > +table description: > +Key Start Len Index Type > +1 12 8 multip. ulonglong NULL > +#now some alter commands; > +alter table t1 add column b int; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b int(11) YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)) I think there shouldn't be (65535) here > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(1,2); > +ERROR 23000: Duplicate entry '1' for key 'a' > +insert into t1 values(2,2); > +ERROR 23000: Duplicate entry '2' for key 'a' > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +#now try to change db_row_hash_1 column; > +alter table t1 drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 add column d int , add column e int , drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 modify column db_row_hash_1 int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column a int , add column b int, modify column db_row_hash_1 int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 change column db_row_hash_1 dsds int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column asd int, change column db_row_hash_1 dsds int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 drop column b , add column c int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +#now add some column with name db_row_hash; > +alter table t1 add column db_row_hash_1 int unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(45,1,55),(46,1,55); > +ERROR 23000: Duplicate entry '55' for key 'db_row_hash_1' > +alter table t1 add column db_row_hash_2 int, add column db_row_hash_3 int; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +c int(11) YES NULL > +db_row_hash_1 int(11) YES UNI NULL > +db_row_hash_2 int(11) YES NULL > +db_row_hash_3 int(11) YES NULL > +#this should also drop the unique index ; > +alter table t1 drop column a; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +#add column with unique index on blob ; > +alter table t1 add column a blob unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `a` blob DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +# try to change the blob unique column name; > +#this will change index to b tree; > +alter table t1 modify column a int ; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +alter table t1 add column clm blob unique; > +#try changing the name ; > +alter table t1 change column clm clm_changed blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + `clm_changed` blob DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `clm` (`clm_changed`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +t1 0 clm 1 clm_changed A NULL NULL NULL YES HASH_INDEX > +#now drop the unique key; > +alter table t1 drop key clm; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + `clm_changed` blob DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +drop table t1; > +create table t1 (a TEXT CHARSET latin1 COLLATE latin1_german2_ci unique); > +desc t1; > +Field Type Null Key Default Extra > +a text YES UNI NULL > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH_INDEX > +insert into t1 values ('ae'); > +insert into t1 values ('AE'); > +ERROR 23000: Duplicate entry 'AE' for key 'a' > +insert into t1 values ('Ä'); this should've failed, shouldn't it? > +drop table t1; > +create table t1 (a int primary key, b blob unique); > +desc t1; > +Field Type Null Key Default Extra > +a int(11) NO PRI NULL > +b blob YES UNI NULL > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 PRIMARY 1 a A 0 NULL NULL BTREE > +t1 0 b 1 b A NULL NULL NULL YES HASH_INDEX > +insert into t1 values(1,1),(2,2),(3,3); > +insert into t1 values(1,1); > +ERROR 23000: Duplicate entry '1' for key 'b' > +insert into t1 values(7,1); > +ERROR 23000: Duplicate entry '1' for key 'b' > +drop table t1; > +#table with multiple long blob column and varchar text column ; > +create table t1(a blob unique, b int , c blob unique , d text unique , e varchar(3000) unique); > +insert into t1 values(1,2,3,4,5),(2,11,22,33,44),(3111,222,333,444,555),(5611,2222,3333,4444,5555), > +('sachin',341,'fdf','gfgfgfg','hghgr'),('maria',345,'frter','dasd','utyuty'), > +(123456789034567891,353534,53453453453456,64565464564564,45435345345345), > +(123456789034567890,43545,657567567567,78967657567567,657567567567567676); > +#table structure; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b int(11) YES NULL > +c blob YES UNI NULL > +d text YES UNI NULL > +e varchar(3000) YES UNI NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `c` (`c`), > + UNIQUE KEY `d` (`d`), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH_INDEX > +t1 0 c 1 c A NULL NULL NULL YES HASH_INDEX > +t1 0 d 1 d A NULL NULL NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 8 Deleted blocks: 0 > +Recordlength: 3072 > + > +table description: > +Key Start Len Index Type > +1 3063 8 multip. ulonglong NULL > +2 3055 8 multip. ulonglong NULL > +3 3047 8 multip. ulonglong NULL > +4 3039 8 multip. ulonglong prefix NULL why "prefix" here? how is it different from others? > +select * from information_schema.columns where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +def test t1 a 1 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob UNI select,insert,update,references NEVER NULL > +def test t1 b 2 NULL YES int NULL NULL 10 0 NULL NULL NULL int(11) select,insert,update,references NEVER NULL > +def test t1 c 3 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob UNI select,insert,update,references NEVER NULL > +def test t1 d 4 NULL YES text 65535 65535 NULL NULL NULL latin1 latin1_swedish_ci text UNI select,insert,update,references NEVER NULL > +def test t1 e 5 NULL YES varchar 3000 3000 NULL NULL NULL latin1 latin1_swedish_ci varchar(3000) UNI select,insert,update,references NEVER NULL > +select * from information_schema.statistics where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME NON_UNIQUE INDEX_SCHEMA INDEX_NAME SEQ_IN_INDEX COLUMN_NAME COLLATION CARDINALITY SUB_PART PACKED NULLABLE INDEX_TYPE COMMENT INDEX_COMMENT > +def test t1 0 test a 1 a A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test c 1 c A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test d 1 d A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test e 1 e A NULL NULL NULL YES HASH_INDEX > +select * from information_schema.key_column_usage where table_schema= 'test' and table_name= 't1'; > +CONSTRAINT_CATALOG CONSTRAINT_SCHEMA CONSTRAINT_NAME TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION POSITION_IN_UNIQUE_CONSTRAINT REFERENCED_TABLE_SCHEMA REFERENCED_TABLE_NAME REFERENCED_COLUMN_NAME > +def test a def test t1 a 1 NULL NULL NULL NULL > +def test c def test t1 c 1 NULL NULL NULL NULL > +def test d def test t1 d 1 NULL NULL NULL NULL > +def test e def test t1 e 1 NULL NULL NULL NULL > +#table select we should not be able to see db_row_hash_1 column; > +select * from t1; > +a b c d e > +1 2 3 4 5 > +2 11 22 33 44 > +3111 222 333 444 555 > +5611 2222 3333 4444 5555 > +sachin 341 fdf gfgfgfg hghgr > +maria 345 frter dasd utyuty > +123456789034567891 353534 53453453453456 64565464564564 45435345345345 > +123456789034567890 43545 657567567567 78967657567567 657567567567567676 > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +select db_row_hash_2 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_2' in 'field list' > +select db_row_hash_3 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_3' in 'field list' > +#duplicate entry test; > +insert into t1 values(21,2,3,42,51); > +ERROR 23000: Duplicate entry '3' for key 'c' > +insert into t1 values('sachin',null,null,null,null); > +ERROR 23000: Duplicate entry 'sachin' for key 'a' > +insert into t1 values(1234567890345671890,4353451,6575675675617,789676575675617,657567567567567676); > +ERROR 23000: Duplicate entry '657567567567567676' for key 'e' > +select * from t1; > +a b c d e > +1 2 3 4 5 > +2 11 22 33 44 > +3111 222 333 444 555 > +5611 2222 3333 4444 5555 > +sachin 341 fdf gfgfgfg hghgr > +maria 345 frter dasd utyuty > +123456789034567891 353534 53453453453456 64565464564564 45435345345345 > +123456789034567890 43545 657567567567 78967657567567 657567567567567676 > +insert into t1 values(repeat('s',4000*10),100,repeat('s',4000*10),repeat('s',4000*10), > +repeat('s',400)),(repeat('s',4001*10),1000,repeat('s',4001*10),repeat('s',4001*10), > +repeat('s',2995)); > +insert into t1 values(repeat('m',4000*11),10,repeat('s',4000*11),repeat('s',4000*11),repeat('s',2995)); > +ERROR 23000: Duplicate entry 'ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss' for key 'e' > +truncate table t1; > +insert into t1 values(1,2,3,4,5),(2,11,22,33,44),(3111,222,333,444,555),(5611,2222,3333,4444,5555); > +#now some alter commands; > +alter table t1 add column f int; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b int(11) YES NULL > +c blob YES UNI NULL > +d text YES UNI NULL > +e varchar(3000) YES UNI NULL > +f int(11) YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `c` (`c`(65535)), > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +#unique key should not break; > +insert into t1 values(1,2,3,4,5,6); > +ERROR 23000: Duplicate entry '1' for key 'a' > +select db_row_hash_1 , db_row_hash_2, db_row_hash_3 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +#now try to change db_row_hash_1 column; > +alter table t1 drop column db_row_hash_1, drop column db_row_hash_2, drop column db_row_hash_3; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 add column dg int , add column ef int , drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 modify column db_row_hash_1 int , modify column db_row_hash_2 int, modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column ar int , add column rb int, modify column db_row_hash_1 int , modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 change column db_row_hash_1 dsds int , change column db_row_hash_2 dfdf int , change column db_row_hash_3 gdfg int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column asd int, drop column a, change column db_row_hash_1 dsds int, change db_row_hash_3 fdfdfd int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 drop column b , add column g int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `c` (`c`(65535)), > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +#now add some column with name db_row_hash; > +alter table t1 add column db_row_hash_1 int unique; > +alter table t1 add column db_row_hash_2 int unique; > +alter table t1 add column db_row_hash_3 int unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `c` (`c`(65535)), > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `db_row_hash_3` (`db_row_hash_3`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +alter table t1 add column db_row_hash_7 int, add column db_row_hash_5 int , add column db_row_hash_4 int ; > +alter table t1 drop column db_row_hash_7,drop column db_row_hash_3, drop column db_row_hash_4; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +c blob YES UNI NULL > +d text YES UNI NULL > +e varchar(3000) YES UNI NULL > +f int(11) YES NULL > +g int(11) YES NULL > +db_row_hash_1 int(11) YES UNI NULL > +db_row_hash_2 int(11) YES UNI NULL > +db_row_hash_5 int(11) YES NULL > +#this show now break anything; "this should not" ? > +insert into t1 values(1,2,3,4,5,6,23,5,6); > +ERROR 23000: Duplicate entry '1' for key 'a' > +#this should also drop the unique index; > +alter table t1 drop column a, drop column c; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +#add column with unique index on blob; > +alter table t1 add column a blob unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` blob DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES HASH_INDEX > +#try to change the blob unique column name; > +#this will change index to b tree; > +alter table t1 modify column a int , modify column e int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +alter table t1 add column clm1 blob unique,add column clm2 blob unique; > +#try changing the name; > +alter table t1 change column clm1 clm_changed1 blob, change column clm2 clm_changed2 blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + `clm_changed1` blob DEFAULT NULL, > + `clm_changed2` blob DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `clm1` (`clm_changed1`), > + UNIQUE KEY `clm2` (`clm_changed2`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +t1 0 clm1 1 clm_changed1 A NULL NULL NULL YES HASH_INDEX > +t1 0 clm2 1 clm_changed2 A NULL NULL NULL YES HASH_INDEX > +#now drop the unique key; > +alter table t1 drop key clm1, drop key clm2; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + `clm_changed1` blob DEFAULT NULL, > + `clm_changed2` blob DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +drop table t1; > +#now the table with key on multiple columns; the ultimate test; > +create table t1(a blob, b int , c varchar(2000) , d text , e varchar(3000) , f longblob , g int , h text , > +unique(a,b,c), unique(c,d,e),unique(e,f,g,h), unique(b,d,g,h)); > +insert into t1 values(1,1,1,1,1,1,1,1),(2,2,2,2,2,2,2,2),(3,3,3,3,3,3,3,3),(4,4,4,4,4,4,4,4),(5,5,5,5,5,5,5,5), > +('maria',6,'maria','maria','maria','maria',6,'maria'),('mariadb',7,'mariadb','mariadb','mariadb','mariadb',8,'mariadb') > +,(null,null,null,null,null,null,null,null),(null,null,null,null,null,null,null,null); > +#table structure; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES MUL NULL > +b int(11) YES MUL NULL > +c varchar(2000) YES MUL NULL > +d text YES NULL > +e varchar(3000) YES MUL NULL > +f longblob YES NULL > +g int(11) YES NULL > +h text YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`,`e`), > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`), > + UNIQUE KEY `b` (`b`,`d`,`g`,`h`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH_INDEX > +t1 0 a 2 b A NULL NULL NULL YES HASH_INDEX > +t1 0 a 3 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 1 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 NULL NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 1 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 2 f A 0 NULL NULL YES HASH_INDEX > +t1 0 e 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 e 4 h A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 NULL NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 NULL NULL YES HASH_INDEX > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 9 Deleted blocks: 0 > +Recordlength: 5092 > + > +table description: > +Key Start Len Index Type > +1 5081 8 multip. ulonglong prefix NULL > +2 5073 8 multip. ulonglong prefix NULL > +3 5065 8 multip. ulonglong prefix NULL > +4 5057 8 multip. ulonglong NULL > +select * from information_schema.columns where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +def test t1 a 1 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob MUL select,insert,update,references NEVER NULL > +def test t1 b 2 NULL YES int NULL NULL 10 0 NULL NULL NULL int(11) MUL select,insert,update,references NEVER NULL > +def test t1 c 3 NULL YES varchar 2000 2000 NULL NULL NULL latin1 latin1_swedish_ci varchar(2000) MUL select,insert,update,references NEVER NULL > +def test t1 d 4 NULL YES text 65535 65535 NULL NULL NULL latin1 latin1_swedish_ci text select,insert,update,references NEVER NULL > +def test t1 e 5 NULL YES varchar 3000 3000 NULL NULL NULL latin1 latin1_swedish_ci varchar(3000) MUL select,insert,update,references NEVER NULL > +def test t1 f 6 NULL YES longblob 4294967295 4294967295 NULL NULL NULL NULL NULL longblob select,insert,update,references NEVER NULL > +def test t1 g 7 NULL YES int NULL NULL 10 0 NULL NULL NULL int(11) select,insert,update,references NEVER NULL > +def test t1 h 8 NULL YES text 65535 65535 NULL NULL NULL latin1 latin1_swedish_ci text select,insert,update,references NEVER NULL > +select * from information_schema.statistics where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME NON_UNIQUE INDEX_SCHEMA INDEX_NAME SEQ_IN_INDEX COLUMN_NAME COLLATION CARDINALITY SUB_PART PACKED NULLABLE INDEX_TYPE COMMENT INDEX_COMMENT > +def test t1 0 test a 1 a A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test a 2 b A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test a 3 c A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test c 1 c A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test c 2 d A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test c 3 e A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test e 1 e A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test e 2 f A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test e 3 g A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test e 4 h A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test b 1 b A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test b 2 d A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test b 3 g A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test b 4 h A 0 NULL NULL YES HASH_INDEX > +select * from information_schema.key_column_usage where table_schema= 'test' and table_name= 't1'; > +CONSTRAINT_CATALOG CONSTRAINT_SCHEMA CONSTRAINT_NAME TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION POSITION_IN_UNIQUE_CONSTRAINT REFERENCED_TABLE_SCHEMA REFERENCED_TABLE_NAME REFERENCED_COLUMN_NAME > +def test a def test t1 a 1 NULL NULL NULL NULL > +def test a def test t1 b 2 NULL NULL NULL NULL > +def test a def test t1 c 3 NULL NULL NULL NULL > +def test c def test t1 c 1 NULL NULL NULL NULL > +def test c def test t1 d 2 NULL NULL NULL NULL > +def test c def test t1 e 3 NULL NULL NULL NULL > +def test e def test t1 e 1 NULL NULL NULL NULL > +def test e def test t1 f 2 NULL NULL NULL NULL > +def test e def test t1 g 3 NULL NULL NULL NULL > +def test e def test t1 h 4 NULL NULL NULL NULL > +def test b def test t1 b 1 NULL NULL NULL NULL > +def test b def test t1 d 2 NULL NULL NULL NULL > +def test b def test t1 g 3 NULL NULL NULL NULL > +def test b def test t1 h 4 NULL NULL NULL NULL > +# table select we should not be able to see db_row_hash_1 column; > +select * from t1; > +a b c d e f g h > +1 1 1 1 1 1 1 1 > +2 2 2 2 2 2 2 2 > +3 3 3 3 3 3 3 3 > +4 4 4 4 4 4 4 4 > +5 5 5 5 5 5 5 5 > +maria 6 maria maria maria maria 6 maria > +mariadb 7 mariadb mariadb mariadb mariadb 8 mariadb > +NULL NULL NULL NULL NULL NULL NULL NULL > +NULL NULL NULL NULL NULL NULL NULL NULL > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +select db_row_hash_2 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_2' in 'field list' > +select db_row_hash_3 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_3' in 'field list' > +#duplicate entry test; > +#duplicate keys entry; > +insert into t1 values(1,1,1,0,0,0,0,0); > +ERROR 23000: Duplicate entry '1-1-1' for key 'a' > +insert into t1 values(0,0,1,1,1,0,0,0); > +ERROR 23000: Duplicate entry '1-1-1' for key 'c' > +insert into t1 values(0,0,0,0,1,1,1,1); > +ERROR 23000: Duplicate entry '1-1-1-1' for key 'e' > +insert into t1 values(1,1,1,1,1,0,0,0); > +ERROR 23000: Duplicate entry '1-1-1' for key 'a' > +insert into t1 values(0,0,0,0,1,1,1,1); > +ERROR 23000: Duplicate entry '1-1-1-1' for key 'e' > +insert into t1 values(1,1,1,1,1,1,1,1); > +ERROR 23000: Duplicate entry '1-1-1' for key 'a' > +select db_row_hash_1,db_row_hash_2,db_row_hash_3,db_row_hash_4,db_row_hash_5 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +alter table t1 drop column db_row_hash_1, drop column db_row_hash_2, drop column db_row_hash_3; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 add column dg int , add column ef int , drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 modify column db_row_hash_1 int , modify column db_row_hash_2 int, modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column ar int , add column rb int, modify column db_row_hash_1 int , modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 change column db_row_hash_1 dsds int , change column db_row_hash_2 dfdf int , change column db_row_hash_3 gdfg int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column asd int, drop column a, change column db_row_hash_1 dsds int, change db_row_hash_3 fdfdfd int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`,`e`), > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`), > + UNIQUE KEY `b` (`b`,`d`,`g`,`h`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +# add column named db_row_hash_*; > +alter table t1 add column db_row_hash_7 int , add column db_row_hash_5 int, > +add column db_row_hash_1 int, add column db_row_hash_2 int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + `db_row_hash_7` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 2 b A NULL NULL NULL YES HASH_INDEX > +t1 0 a 3 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 1 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 1 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 2 f A 0 NULL NULL YES HASH_INDEX > +t1 0 e 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 e 4 h A 0 65535 NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +alter table t1 drop column db_row_hash_7 , drop column db_row_hash_5 , > +drop column db_row_hash_1, drop column db_row_hash_2 ; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 2 b A NULL NULL NULL YES HASH_INDEX > +t1 0 a 3 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 1 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 1 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 2 f A 0 NULL NULL YES HASH_INDEX > +t1 0 e 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 e 4 h A 0 65535 NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +#try to change column names; > +alter table t1 change column a aa blob , change column b bb blob , change column d dd blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `aa` blob DEFAULT NULL, > + `bb` blob DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `dd` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `a` (`aa`(65535),`bb`,`c`), > + UNIQUE KEY `c` (`c`,`dd`(65535),`e`), > + UNIQUE KEY `b` (`bb`,`dd`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 e 2 f A NULL NULL NULL YES HASH_INDEX > +t1 0 e 3 g A NULL NULL NULL YES HASH_INDEX > +t1 0 e 4 h A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 1 aa A 0 65535 NULL YES HASH_INDEX > +t1 0 a 2 bb A 0 NULL NULL YES HASH_INDEX > +t1 0 a 3 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 1 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 2 dd A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 bb A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 dd A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +alter table t1 change column aa a blob , change column bb b blob , change column dd d blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `a` (`a`(65535),`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 e 2 f A NULL NULL NULL YES HASH_INDEX > +t1 0 e 3 g A NULL NULL NULL YES HASH_INDEX > +t1 0 e 4 h A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 1 a A 0 65535 NULL YES HASH_INDEX > +t1 0 a 2 b A 0 NULL NULL YES HASH_INDEX > +t1 0 a 3 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 1 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +#now we will change the data type to int and varchar limit so that we no longer require hash_index; > +#on key a_b_c; > +alter table t1 modify column a varchar(20) , modify column b varchar(20) , modify column c varchar(20); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` varchar(20) DEFAULT NULL, > + `b` varchar(20) DEFAULT NULL, > + `c` varchar(20) DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 e 2 f A NULL NULL NULL YES HASH_INDEX > +t1 0 e 3 g A NULL NULL NULL YES HASH_INDEX > +t1 0 e 4 h A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +t1 0 a 2 b A NULL NULL NULL YES BTREE > +t1 0 a 3 c A 0 NULL NULL YES BTREE > +t1 0 c 1 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +#change it back; > +alter table t1 modify column a blob , modify column b blob , modify column c blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 e 2 f A NULL NULL NULL YES HASH_INDEX > +t1 0 e 3 g A NULL NULL NULL YES HASH_INDEX > +t1 0 e 4 h A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 1 a A 0 NULL NULL YES HASH_INDEX > +t1 0 a 2 b A 0 NULL NULL YES HASH_INDEX > +t1 0 a 3 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 1 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +#try to delete blob column in unique; > +truncate table t1; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 e 2 f A NULL NULL NULL YES HASH_INDEX > +t1 0 e 3 g A NULL NULL NULL YES HASH_INDEX > +t1 0 e 4 h A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 1 a A 0 NULL NULL YES HASH_INDEX > +t1 0 a 2 b A 0 NULL NULL YES HASH_INDEX > +t1 0 a 3 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 1 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +#now try to delete keys; > +alter table t1 drop key c, drop key e; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`b`(65535),`c`(65535)), > + UNIQUE KEY `b` (`b`(65535),`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 2 b A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 3 c A 0 65535 NULL YES HASH_INDEX > +t1 0 b 1 b A 0 65535 NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +drop table t1; > +#now alter table containing some data basically some tests with ignore; > +create table t1 (a blob); > +insert into t1 values(1),(2),(3); > +#normal alter table; > +alter table t1 add unique key(a); > +alter table t1 drop key a; > +truncate table t1; > +insert into t1 values(1),(1),(2),(2),(3); > +alter table t1 add unique key(a); > +ERROR 23000: Duplicate entry '1' for key 'a' > +alter ignore table t1 add unique key(a); > +select * from t1; > +a > +1 > +2 > +3 > +insert into t1 values(1); > +ERROR 23000: Duplicate entry '1' for key 'a' > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH_INDEX > +drop table t1; > +#Now with multiple keys; > +create table t1(a blob , b blob, c blob , d blob , e int); > +insert into t1 values (1,1,1,1,1); > +insert into t1 values (1,1,1,1,1); > +insert into t1 values (2,1,1,1,1); > +insert into t1 values (2,2,2,2,2); > +insert into t1 values (3,3,4,4,4); > +insert into t1 values (4,4,4,4,4); > +alter table t1 add unique key(a,c), add unique key(b,d), add unique key(e); > +ERROR 23000: Duplicate entry '1-1' for key 'a' > +alter ignore table t1 add unique key(a,c), add unique key(b,d), add unique key(e); > +select * from t1; > +a b c d e > +1 1 1 1 1 > +2 2 2 2 2 > +3 3 4 4 4 > +insert into t1 values (1,12,1,13,14); > +ERROR 23000: Duplicate entry '1-1' for key 'a' > +insert into t1 values (12,1,14,1,14); > +ERROR 23000: Duplicate entry '1-1' for key 'b' > +insert into t1 values (13,12,13,14,4); > +ERROR 23000: Duplicate entry '4' for key 'e' > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`c`(65535)), > + UNIQUE KEY `b` (`b`(65535),`d`(65535)), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 2 c A NULL 65535 NULL YES HASH_INDEX > +t1 0 b 1 b A NULL 65535 NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A 0 NULL NULL YES BTREE > +drop table t1; > +#visibility of db_row_hash > +create table t1 (a blob unique , b blob unique); > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b blob YES UNI NULL > +insert into t1 values(1,19); > +insert into t1 values(2,29); > +insert into t1 values(3,39); > +insert into t1 values(4,49); > +create table t2 (DB_ROW_HASH_1 int, DB_ROW_HASH_2 int); > +insert into t2 values(11,1); > +insert into t2 values(22,2); > +insert into t2 values(33,3); > +insert into t2 values(44,4); > +select * from t1; > +a b > +1 19 > +2 29 > +3 39 > +4 49 > +select * from t2; > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +22 2 > +33 3 > +44 4 > +select DB_ROW_HASH_1, DB_ROW_HASH_2 from t1; > +ERROR 42S22: Unknown column 'DB_ROW_HASH_1' in 'field list' > +#bug > +select DB_ROW_HASH_1, DB_ROW_HASH_2 from t1,t2; > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +11 1 > +11 1 > +11 1 > +22 2 > +22 2 > +22 2 > +22 2 > +33 3 > +33 3 > +33 3 > +33 3 > +44 4 > +44 4 > +44 4 > +44 4 > +select * from t1 where DB_ROW_HASH_1 in (select DB_ROW_HASH_1 from t2); > +ERROR 42S22: Unknown column 'DB_ROW_HASH_1' in 'IN/ALL/ANY subquery' > +select DB_ROW_HASH_1, DB_ROW_HASH_2 from t1,t2 where DB_ROW_HASH_1 in (select DB_ROW_HASH_1 from t2); > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +22 2 > +33 3 > +44 4 > +11 1 > +22 2 > +33 3 > +44 4 > +11 1 > +22 2 > +33 3 > +44 4 > +11 1 > +22 2 > +33 3 > +44 4 > +select * from t2 where DB_ROW_HASH_1 in (select DB_ROW_HASH_1 from t1); > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +22 2 > +33 3 > +44 4 > +select DB_ROW_HASH_1 from t1,t2 where t1.DB_ROW_HASH_1 = t2.DB_ROW_HASH_2; > +ERROR 42S22: Unknown column 't1.DB_ROW_HASH_1' in 'where clause' > +select DB_ROW_HASH_1 from t1 inner join t2 on t1.a = t2.DB_ROW_HASH_2; > +DB_ROW_HASH_1 > +11 > +22 > +33 > +44 > +drop table t1,t2; > +#very long blob entry; > +SET @@GLOBAL.max_allowed_packet=67108864; > +connect 'newcon', localhost, root,,; > +connection newcon; > +show variables like 'max_allowed_packet'; > +Variable_name Value > +max_allowed_packet 67108864 > +create table t1(a longblob unique, b longblob , c longblob , unique(b,c)); > +desc t1; > +Field Type Null Key Default Extra > +a longblob YES UNI NULL > +b longblob YES MUL NULL > +c longblob YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` longblob DEFAULT NULL, > + `b` longblob DEFAULT NULL, > + `c` longblob DEFAULT NULL, > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `b` (`b`,`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH_INDEX > +t1 0 b 1 b A NULL NULL NULL YES HASH_INDEX > +t1 0 b 2 c A 0 NULL NULL YES HASH_INDEX > +insert into t1 values(concat(repeat('sachin',10000000),'1'),concat(repeat('sachin',10000000),'1'), > +concat(repeat('sachin',10000000),'1')); > +insert into t1 values(concat(repeat('sachin',10000000),'2'),concat(repeat('sachin',10000000),'2'), > +concat(repeat('sachin',10000000),'1')); > +insert into t1 values(concat(repeat('sachin',10000000),'2'),concat(repeat('sachin',10000000),'2'), > +concat(repeat('sachin',10000000),'4')); > +ERROR 23000: Duplicate entry 'sachinsachinsachinsachinsachinsachinsachinsachinsachinsachinsach' for key 'a' > +insert into t1 values(concat(repeat('sachin',10000000),'3'),concat(repeat('sachin',10000000),'1'), > +concat(repeat('sachin',10000000),'1')); > +ERROR 23000: Duplicate entry 'sachinsachinsachinsachinsachinsachinsachinsachinsachinsachinsach' for key 'b' > +drop table t1; > +#long key unique with different key length > +create table t1(a blob, unique(a(3000))); > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 3000 NULL YES HASH_INDEX > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + UNIQUE KEY `a` (`a`(3000)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 value(concat(repeat('s',3000),'1')); > +insert into t1 value(concat(repeat('s',3000),'2')); > +ERROR 23000: Duplicate entry 'ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss' for key 'a' > +insert into t1 value(concat(repeat('a',3000),'2')); > +drop table t1; > +create table t1(a varchar(4000), b longblob , c varchar(5000), d longblob, > +unique(a(3500), b(1000000)), unique(c(4500), d(10000000))); > +desc t1; > +Field Type Null Key Default Extra > +a varchar(4000) YES MUL NULL > +b longblob YES NULL > +c varchar(5000) YES MUL NULL > +d longblob YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` varchar(4000) DEFAULT NULL, > + `b` longblob DEFAULT NULL, > + `c` varchar(5000) DEFAULT NULL, > + `d` longblob DEFAULT NULL, > + UNIQUE KEY `a` (`a`(3500),`b`), > + UNIQUE KEY `c` (`c`(4500),`d`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 3500 NULL YES HASH_INDEX > +t1 0 a 2 b A NULL NULL NULL YES HASH_INDEX > +t1 0 c 1 c A 0 4500 NULL YES HASH_INDEX > +t1 0 c 2 d A 0 NULL NULL YES HASH_INDEX > +drop table t1; > +disconnect newcon; > +connection default; > +SET @@GLOBAL.max_allowed_packet=4194304; > +#ext bug > +create table t1(a int primary key, b blob unique, c int, d blob , index(c)); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) NOT NULL, > + `b` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + `d` blob DEFAULT NULL, > + PRIMARY KEY (`a`), > + UNIQUE KEY `b` (`b`), > + KEY `c` (`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(1,23,1,33); > +insert into t1 values(2,23,1,33); > +ERROR 23000: Duplicate entry '23' for key 'b' > +drop table t1; > +create table t2 (a blob unique , c int , index(c)); > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `a` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`), > + KEY `c` (`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t2 values(1,1); > +insert into t2 values(2,1); > +drop table t2; > +#not null test //todo solve warnings > +create table t1(a blob unique not null); > +desc t1; > +Field Type Null Key Default Extra > +a blob NO PRI NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob NOT NULL, > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(1); > +insert into t1 values(3); > +insert into t1 values(1); > +ERROR 23000: Duplicate entry '1' for key 'a' > +drop table t1; > +create table t1(a int primary key, b blob unique , c blob unique not null); > +insert into t1 values(1,1,1); > +insert into t1 values(2,1,2); > +ERROR 23000: Duplicate entry '1' for key 'b' > +insert into t1 values(3,3,1); > +ERROR 23000: Duplicate entry '1' for key 'c' > +drop table t1; > +create table t1 (a blob unique not null, b blob not null, c blob not null, unique(b,c)); > +desc t1; > +Field Type Null Key Default Extra > +a blob NO PRI NULL > +b blob NO MUL NULL > +c blob NO NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob NOT NULL, > + `b` blob NOT NULL, > + `c` blob NOT NULL, > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `b` (`b`,`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values (1, 2, 3); > +insert into t1 values (2, 1, 3); > +insert into t1 values (2, 1, 3); > +ERROR 23000: Duplicate entry '2' for key 'a' > +drop table t1; > +set @@GLOBAL.max_allowed_packet= @allowed_packet; > diff --git a/sql/field.h b/sql/field.h > index b6f28808e2e..13eea28fe8a 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -539,6 +539,7 @@ class Virtual_column_info: public Sql_alloc > Item *expr; > LEX_CSTRING name; /* Name of constraint */ > uint flags; > + LEX_CSTRING hash_expr; It is not a good idea to put your hash-unique-constraint specific data into a common Virtual_column_info. Luckily, you don't need it. Just like "current_timestamp()" expression is generated on the fly as needed in parse_vcol_defs(), you can generate the hash expression there too, no need to do it in TABLE_SHARE::init_from_binary_frm_image(). So, just remove hash_expr. > > Virtual_column_info() > : vcol_type((enum_vcol_info_type)VCOL_TYPE_NONE), > @@ -681,7 +688,7 @@ class Field: public Value_source > GEOM_MULTIPOINT = 4, GEOM_MULTILINESTRING = 5, GEOM_MULTIPOLYGON = 6, > GEOM_GEOMETRYCOLLECTION = 7 > }; > - enum imagetype { itRAW, itMBR}; > + enum imagetype { itRAW, itMBR, itHASH}; itHASH isn't used anywhere in your patch :) > > utype unireg_check; > uint32 field_length; // Length of field > diff --git a/sql/field.cc b/sql/field.cc > index dc854826ed6..f75f52141e3 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -9822,7 +9822,7 @@ int Field_bit::key_cmp(const uchar *str, uint length) > } > > > -int Field_bit::cmp_offset(uint row_offset) > +int Field_bit::cmp_offset(long row_offset) may be my_ptrdiff_t ? > { > if (bit_len) > { > diff --git a/sql/item_func.h b/sql/item_func.h > index 3b6cb4ceeac..0391de0c5ad 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -803,6 +803,19 @@ class Item_long_func: public Item_int_func > }; > > > +class Item_func_hash: public Item_int_func > +{ > +public: > + Item_func_hash(THD *thd, List<Item> &item): Item_int_func(thd, item) > + {} > + longlong val_int(); > + void fix_length_and_dec(); > + const Type_handler *type_handler() const { return &type_handler_long; } > + Item *get_copy(THD *thd) > + { return get_item_copy<Item_func_hash>(thd, this); } > + const char *func_name() const { return "HASH"; } is it a user visible function? Can one write SELECT HASH(a) FROM t1 ? looks like yes, so 1. please add tests for it 2. we need to document what hash function, exactly, is used. 3. May be better to call it MARIADB_HASH? to highlight its internal-use nature and a home-baked hash function? And to reduce a chance for a collision with user defined functions. 4. An even better approach would be not to create any user visible function at all. Currently unpack_vcol_info_from_frm() calls the parser and then initializes vcol_info and calls fix_fields. If you split it in two you can do `new Item_func_hash` manually and skip the first part of unpack_vcol_info_from_frm(). And you won't need HASH to be user-visible. As a minor optimization, the same can be done for "current_timestamp()" > +}; > + > class Item_longlong_func: public Item_int_func > { > public: > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 4923c628bf9..c620ea23165 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -2300,7 +2300,7 @@ int show_create_table(THD *thd, TABLE_LIST *table_list, String *packet, > */ > packet->append(STRING_WITH_LEN("PRIMARY KEY")); > } > - else if (key_info->flags & HA_NOSAME) > + else if (key_info->flags & HA_NOSAME || key_info->flags & HA_LONG_UNIQUE_HASH) Is it needed? As far as I can see, you always set HA_NOSAME whenever you set HA_LONG_UNIQUE_HASH. > packet->append(STRING_WITH_LEN("UNIQUE KEY ")); > else if (key_info->flags & HA_FULLTEXT) > packet->append(STRING_WITH_LEN("FULLTEXT KEY ")); > diff --git a/include/my_base.h b/include/my_base.h > index c36072c0bfa..ba17f8d80dc 100644 > --- a/include/my_base.h > +++ b/include/my_base.h > @@ -290,6 +291,11 @@ enum ha_base_keytype { > #define HA_KEY_HAS_PART_KEY_SEG 65536 > /* Internal Flag Can be calcaluted */ > #define HA_INVISIBLE_KEY 2<<18 > +/* > + Some more flags for keys these are not stored in > + frm it is calculated on the fly in init_from_binary_frm_image > +*/ 1. I don't think you need to make the same comment more verbose every time you repeat it :) Just say, like #define HA_INVISIBLE_KEY 2<<18 /* this is calculated too */ #define HA_LONG_UNIQUE_HASH 2<<19 /* this is calculated too */ 2. Do you need a separate flag for that, just HA_INVISIBLE_KEY isn't enough? > +#define HA_LONG_UNIQUE_HASH 2<<19 > /* Automatic bits in key-flag */ > > #define HA_SPACE_PACK_USED 4 /* Test for if SPACE_PACK used */ > @@ -317,6 +323,13 @@ enum ha_base_keytype { > #define HA_BIT_PART 1024 > #define HA_CAN_MEMCMP 2048 /* internal, never stored in frm */ > > +/* > + Used for key parts whole length is greater then > file->max_key_part_length > + Only used for HA_LONG_UNIQUE_HASH keys > +*/ //TODO a better name ?? > +#define HA_HASH_KEY_PART_FLAG 4096 > +/* Field need to be frees externally */ > +#define HA_FIELD_EX_FREED 8192 Okay, this is a weird name, agree. But it is not used anywere, so there's an easy solution :) > /* optionbits for database */ > #define HA_OPTION_PACK_RECORD 1U > #define HA_OPTION_PACK_KEYS 2U > diff --git a/sql/table.h b/sql/table.h > index 785fd9f3427..9ed7fcd7beb 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -346,9 +346,39 @@ enum field_visibility_t { > INVISIBLE_FULL > }; > > -#define INVISIBLE_MAX_BITS 3 > +#define INVISIBLE_MAX_BITS 3 > +/* We will store the info into 3rd bit if field is hash field */ > +#define HASH_FIELD_MASK 15 > +#define HASH_FIELD_MASK_SHIFT 4 > +#define HA_HASH_FIELD_LENGTH 8 > +#define HA_HASH_KEY_LENGTH_WITHOUT_NULL 8 > +#define HA_HASH_KEY_LENGTH_WITH_NULL 9 > +//TODO is this correct ? how about size of char ptr on 32/16 bit machine? I don't understand that, sorry > +#define HA_HASH_KEY_PART_LENGTH 4 + 8 // 4 for length , 8 for portable size of char ptr you almost always should use parentheses in macro expressions > > +const LEX_CSTRING ha_hash_str {STRING_WITH_LEN("HASH")}; ??? you create a copy of ha_hash_str symbol in every .o file that included table.h? > > + > +int find_field_pos_in_hash(Item *hash_item, const char * field_name); > + > +int fields_in_hash_str(Item *hash_item); > + > +Field * field_ptr_in_hash_str(Item *hash_item, int index); > + > +int get_key_part_length(KEY *keyinfo, int index); > + > +void calc_hash_for_unique(ulong &nr1, ulong &nr2, String *str); > + > +void create_update_handler(THD *thd, TABLE *table); > + > +void delete_update_handler(THD *thd, TABLE *table); > + > +void setup_table_hash(TABLE *table); > + > +void re_setup_table(TABLE *table); > + > +int get_hash_key(THD *thd, TABLE *table, handler *h, uint key_index, uchar *rec_buf, > + uchar *key_buff); > /** > Category of table found in the table share. > */ > @@ -1094,6 +1124,17 @@ struct TABLE > THD *in_use; /* Which thread uses this */ > > uchar *record[3]; /* Pointer to records */ > + /* record buf to resolve hash collisions for long UNIQUE constraints */ > + uchar *check_unique_buf; > + handler *update_handler; /* Handler used in case of update */ > + /* > + In the case of write row for long unique we are unable of find > + Whick key is voilated. Because we in case of duplicate hash we never reach > + handler write_row function. So print_error will always print that > + key 0 is voilated. We store which key is voilated in this variable > + by default this should be initialized to -1 typos :) "unable to find", "which", "violated" > + */ > + int dupp_hash_key; > uchar *write_row_record; /* Used as optimisation in > THD::write_row */ > uchar *insert_values; /* used by INSERT ... UPDATE */ > @@ -2898,6 +2939,7 @@ void append_unescaped(String *res, const char *pos, size_t length); > void prepare_frm_header(THD *thd, uint reclength, uchar *fileinfo, > HA_CREATE_INFO *create_info, uint keys, KEY *key_info); > const char *fn_frm_ext(const char *name); > +void calc_hash_for_unique(ulong &nr1, ulong &nr2, String *str); you have this prototype twice in table.h > > /* Check that the integer is in the internal */ > static inline int set_zone(int nr,int min_zone,int max_zone) > diff --git a/sql/table.cc b/sql/table.cc > index 73b1a6bd9b2..8cd4db2844d 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -747,7 +748,13 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > if (i == 0) > { > ext_key_parts+= (share->use_ext_keys ? first_keyinfo->user_defined_key_parts*(keys-1) : 0); > + /* > + Some keys can be HA_LONG_UNIQUE_HASH , but we do not know at this point , > + how many ?, but will always be less than or equal to total num of > + keys. Each HA_LONG_UNIQUE_HASH key require one extra key_part in which > + it stored hash. On safe side we will allocate memory for each key. > + */ > - n_length=keys * sizeof(KEY) + ext_key_parts * sizeof(KEY_PART_INFO); > + n_length=keys * sizeof(KEY) + (ext_key_parts +keys) * sizeof(KEY_PART_INFO); Hmm, why wouldn't you store the number of HA_KEY_ALG_LONG_HASH keys in EXTRA2_LONG_UNIQUES ? Then you'll know it here. > if (!(keyinfo= (KEY*) alloc_root(&share->mem_root, > n_length + len))) > return 1; > @@ -798,6 +805,14 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > } > key_part->store_length=key_part->length; > } > + if (keyinfo->algorithm == HA_KEY_ALG_LONG_HASH) > + { > + keyinfo->flags|= HA_LONG_UNIQUE_HASH | HA_NOSAME; > + keyinfo->key_length= 0; > + share->ext_key_parts++; > + // This empty key_part for storing Hash > + key_part++; > + } so, you write keyinfo's for HA_LONG_UNIQUE_HASH keys into the frm? why don't you write keysegs for them? I mean, it is not very logical. I'd thought you won't write either keyinfos or keysegs. Or, okay, you could write both. But only keyinfos and no keysegs? That's strange > > /* > Add primary key to end of extended keys for non unique keys for > @@ -1143,13 +1159,21 @@ bool parse_vcol_defs(THD *thd, MEM_ROOT *mem_root, TABLE *table, > pos+= expr_length; > } > > - /* Now, initialize CURRENT_TIMESTAMP fields */ > + /* Now, initialize CURRENT_TIMESTAMP and UNIQUE_INDEX_HASH_FIELD fields */ > for (field_ptr= table->field; *field_ptr; field_ptr++) > { > Field *field= *field_ptr; > - if (field->has_default_now_unireg_check()) > + if (field->vcol_info && (length = field->vcol_info->hash_expr.length)) > { > expr_str.length(parse_vcol_keyword.length); > + expr_str.append((char*)field->vcol_info->hash_expr.str, length); > + vcol= unpack_vcol_info_from_frm(thd, mem_root, table, &expr_str, > + &(field->vcol_info), error_reported); see above about not going through the parser for hash keys. > + *(vfield_ptr++)= *field_ptr; > + > + } > + if (field->has_default_now_unireg_check()) > + { > expr_str.append(STRING_WITH_LEN("current_timestamp(")); > expr_str.append_ulonglong(field->decimals()); > expr_str.append(')'); > @@ -2106,7 +2132,8 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > uchar flags= *extra2_field_flags++; > if (flags & VERS_OPTIMIZED_UPDATE) > reg_field->flags|= VERS_UPDATE_UNVERSIONED_FLAG; > - > + if (flags & EXTRA2_LONG_UNIQUE_HASH_FIELD) > + reg_field->flags|= LONG_UNIQUE_HASH_FIELD; so, you write LONG_UNIQUE_HASH_FIELD fields to frm too. Why? > reg_field->invisible= f_visibility(flags); > } > if (reg_field->invisible == INVISIBLE_USER) > @@ -2350,6 +2438,8 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > key_part= keyinfo->key_part; > uint key_parts= share->use_ext_keys ? keyinfo->ext_key_parts : > keyinfo->user_defined_key_parts; > + if (keyinfo->flags & HA_LONG_UNIQUE_HASH) > + key_parts++; key_parts++ ? Doesn't your HA_LONG_UNIQUE_HASH key have only one part, always? > for (i=0; i < key_parts; key_part++, i++) > { > Field *field; > @@ -2363,7 +2453,16 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > > field= key_part->field= share->field[key_part->fieldnr-1]; > key_part->type= field->key_type(); > + if (keyinfo->flags & HA_LONG_UNIQUE_HASH > + &&(key_part->length > handler_file->max_key_part_length() > + || key_part->length == 0)) 1. fix the spacing and the indentation here, please 2. what should happen if HA_LONG_UNIQUE_HASH flag is set, but the key_part->length is small ? > + { > + key_part->key_part_flag= HA_HASH_KEY_PART_FLAG; > + key_part->store_length= HA_HASH_KEY_PART_LENGTH; > + } > + /* Invisible Full is currently used by long uniques */ > - if (field->invisible > INVISIBLE_USER && !field->vers_sys_field()) > + if ((field->invisible == INVISIBLE_USER || > + field->invisible == INVISIBLE_SYSTEM )&& !field->vers_sys_field()) why is this change? > keyinfo->flags |= HA_INVISIBLE_KEY; > if (field->null_ptr) > { > @@ -2428,7 +2527,8 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > field->part_of_sortkey= share->keys_in_use; > } > } > - if (field->key_length() != key_part->length) > + if (field->key_length() != key_part->length && > + !(keyinfo->flags & HA_LONG_UNIQUE_HASH)) why is that? > { > #ifndef TO_BE_DELETED_ON_PRODUCTION > if (field->type() == MYSQL_TYPE_NEWDECIMAL) > @@ -2470,7 +2570,8 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > if (!(key_part->key_part_flag & (HA_BLOB_PART | HA_VAR_LENGTH_PART | > HA_BIT_PART)) && > key_part->type != HA_KEYTYPE_FLOAT && > - key_part->type == HA_KEYTYPE_DOUBLE) > + key_part->type == HA_KEYTYPE_DOUBLE && > + !(keyinfo->flags & HA_LONG_UNIQUE_HASH)) why is that? > key_part->key_part_flag|= HA_CAN_MEMCMP; > } > keyinfo->usable_key_parts= usable_parts; // Filesort > @@ -8346,6 +8454,377 @@ double KEY::actual_rec_per_key(uint i) > } > > > +/* > + find out the field positoin in hash_str() > + position starts from 0 > + else return -1; > +*/ > +int find_field_pos_in_hash(Item *hash_item, const char * field_name) > +{ > + Item_func_or_sum * temp= static_cast<Item_func_or_sum *>(hash_item); > + Item_args * t_item= static_cast<Item_args *>(temp); > + uint arg_count= t_item->argument_count(); > + Item ** arguments= t_item->arguments(); > + Field * t_field; > + > + for (uint j=0; j < arg_count; j++) > + { > + DBUG_ASSERT(arguments[j]->type() == Item::FIELD_ITEM || > + arguments[j]->type() == Item::FUNC_ITEM); > + if (arguments[j]->type() == Item::FIELD_ITEM) > + { > + t_field= static_cast<Item_field *>(arguments[j])->field; > + } > + else > + { > + Item_func_left *fnc= static_cast<Item_func_left *>(arguments[j]); > + t_field= static_cast<Item_field *>(fnc->arguments()[0])->field; > + } > + if (!my_strcasecmp(system_charset_info, t_field->field_name.str, field_name)) > + return j; > + } > + return -1; > +} > + > +/* > + find total number of field in hash_str > +*/ > +int fields_in_hash_str(Item * hash_item) can be static > +{ > + Item_func_or_sum * temp= static_cast<Item_func_or_sum *>(hash_item); > + Item_args * t_item= static_cast<Item_args *>(temp); > + return t_item->argument_count(); > +} > + > +/* > + Returns fields ptr given by hash_str index > + Index starts from 0 > +*/ > +Field * field_ptr_in_hash_str(Item *hash_item, int index) > +{ > + Item_func_or_sum * temp= static_cast<Item_func_or_sum *>(hash_item); > + Item_args * t_item= static_cast<Item_args *>(temp); > + return static_cast<Item_field *>(t_item->arguments()[index])->field; > +} > + > +//NO longer Needed remove it, then :) > +int get_key_part_length(KEY *keyinfo, int index) > +{ > + DBUG_ASSERT(keyinfo->flags & HA_LONG_UNIQUE_HASH); > + TABLE *tbl= keyinfo->table; > + Item *h_item= keyinfo->key_part->field->vcol_info->expr; > + Field *fld= field_ptr_in_hash_str(h_item, index); > + if (!index) > + return fld->pack_length()+HA_HASH_KEY_LENGTH_WITH_NULL+1; > + return fld->pack_length()+1; > +} > +/** > + @brief clone of current handler. > + Creates a clone of handler used in update for > + unique hash key. > + @param thd Thread Object > + @param table Table Object > + @return handler object > +*/ > +void create_update_handler(THD *thd, TABLE *table) better, clone_handler_for_update > +{ > + handler *update_handler= NULL; > + for (uint i= 0; i < table->s->keys; i++) indentation > + { > + if (table->key_info[i].flags & HA_LONG_UNIQUE_HASH) > + { > + update_handler= table->file->clone(table->s->normalized_path.str, > + thd->mem_root); > + update_handler->ha_external_lock(thd, F_RDLCK); > + table->update_handler= update_handler; why do you store it in TABLE? you can just keep it in a local variable in mysql_update, you don't need it outside of mysql_update anyway. (and multi-update) also, you shouldn't need to scan all table keys here. use some flag or property in TABLE_SCHEMA to check quickly whether the table has these lock unique keys. > + return; > + } > + } > + return; > +} > + > +/** > + @brief Deletes update handler object > + @param thd Thread Object > + @param table Table Object > +*/ > +void delete_update_handler(THD *thd, TABLE *table) > +{ > + if (table->update_handler) > + { > + table->update_handler->ha_external_lock(thd, F_UNLCK); > + table->update_handler->ha_close(); > + delete table->update_handler; > + table->update_handler= NULL; > + } > +} > +/** > + @brief This function makes table object with > + long unique keys ready for storage engine. > + It makes key_part of HA_LONG_UNIQUE_HASH point to > + hash key_part. > + @param table Table object > + */ > +void setup_table_hash(TABLE *table) this and other functions could be methods in TABLE > +{ > + /* > + Extra parts of long unique key which are used only at server level > + for example in key unique(a, b, c) //a b c are blob > + extra_key_part_hash is 3 > + */ > + uint extra_key_part_hash= 0; > + uint hash_parts= 0; > + KEY *s_keyinfo= table->s->key_info; > + KEY *keyinfo= table->key_info; > + /* > + Sometime s_keyinfo or key_info can be null. So > + two different loop for keyinfo and s_keyinfo > + reference test case:- main.subselect_sj2 how they could be null here? > + */ > + > + if (keyinfo) > + { > + for (uint i= 0; i < table->s->keys; i++, keyinfo++) > + { > + if (keyinfo->flags & HA_LONG_UNIQUE_HASH) > + { > + DBUG_ASSERT(keyinfo->user_defined_key_parts == > + keyinfo->ext_key_parts); > + keyinfo->flags&= ~(HA_NOSAME | HA_LONG_UNIQUE_HASH); > + keyinfo->algorithm= HA_KEY_ALG_UNDEF; > + extra_key_part_hash+= keyinfo->ext_key_parts; > + hash_parts++; > + keyinfo->key_part= keyinfo->key_part+ keyinfo->ext_key_parts; > + keyinfo->user_defined_key_parts= keyinfo->usable_key_parts= > + keyinfo->ext_key_parts= 1; > + keyinfo->key_length= keyinfo->key_part->store_length; > + } > + } > + table->s->key_parts-= extra_key_part_hash; > + table->s->key_parts+= hash_parts; > + table->s->ext_key_parts-= extra_key_part_hash; I don't understand what you're doing here. Could you add a comment, explaning the resulting structure of keys and keysegs and what's where? > + } > + if (s_keyinfo) > + { > + for (uint i= 0; i < table->s->keys; i++, s_keyinfo++) > + { > + if (s_keyinfo->flags & HA_LONG_UNIQUE_HASH) > + { > + DBUG_ASSERT(s_keyinfo->user_defined_key_parts == > + s_keyinfo->ext_key_parts); > + s_keyinfo->flags&= ~(HA_NOSAME | HA_LONG_UNIQUE_HASH); > + s_keyinfo->algorithm= HA_KEY_ALG_BTREE; > + extra_key_part_hash+= s_keyinfo->ext_key_parts; > + s_keyinfo->key_part= s_keyinfo->key_part+ s_keyinfo->ext_key_parts; > + s_keyinfo->user_defined_key_parts= s_keyinfo->usable_key_parts= > + s_keyinfo->ext_key_parts= 1; > + s_keyinfo->key_length= s_keyinfo->key_part->store_length; > + } > + } > + if (!keyinfo) > + { > + table->s->key_parts-= extra_key_part_hash; > + table->s->key_parts+= hash_parts; > + table->s->ext_key_parts-= extra_key_part_hash; > + } > + } > +} > + > +/** > + @brief Revert the effect of setup_table_hash > + @param table Table Object > + */ > +void re_setup_table(TABLE *table) > +{ > + //extra key parts excluding hash , which needs to be added in keyparts > + uint extra_key_parts_ex_hash= 0; > + uint extra_hash_parts= 0; // this var for share->extra_hash_parts > + KEY *s_keyinfo= table->s->key_info; > + KEY *keyinfo= table->key_info; > + /* > + Sometime s_keyinfo can be null so > + two different loop for keyinfo and s_keyinfo > + ref test case:- main.subselect_sj2 > + */ > + if (keyinfo) > + { > + for (uint i= 0; i < table->s->keys; i++, keyinfo++) > + { > + if (keyinfo->user_defined_key_parts == 1 && > + keyinfo->key_part->field->flags & LONG_UNIQUE_HASH_FIELD) > + { > + keyinfo->flags|= (HA_NOSAME | HA_LONG_UNIQUE_HASH); > + keyinfo->algorithm= HA_KEY_ALG_LONG_HASH; > + /* Sometimes it can happen, that we does not parsed hash_str. > + Like when this function is called in ha_create. So we will > + Use field from table->field rather then share->field*/ > + Item *h_item= table->field[keyinfo->key_part->fieldnr - 1]-> > + vcol_info->expr; > + uint hash_parts= fields_in_hash_str(h_item); > + keyinfo->key_part= keyinfo->key_part- hash_parts; > + keyinfo->user_defined_key_parts= keyinfo->usable_key_parts= > + keyinfo->ext_key_parts= hash_parts; > + extra_key_parts_ex_hash+= hash_parts; > + extra_hash_parts++; > + keyinfo->key_length= -1; > + } > + } > + table->s->key_parts-= extra_hash_parts; > + table->s->key_parts+= extra_key_parts_ex_hash; > + table->s->ext_key_parts+= extra_key_parts_ex_hash + extra_hash_parts; > + } > + if (s_keyinfo) > + { > + for (uint i= 0; i < table->s->keys; i++, s_keyinfo++) > + { > + if (s_keyinfo->user_defined_key_parts == 1 && > + s_keyinfo->key_part->field->flags & LONG_UNIQUE_HASH_FIELD) > + { > + s_keyinfo->flags|= (HA_NOSAME | HA_LONG_UNIQUE_HASH); > + s_keyinfo->algorithm= HA_KEY_ALG_LONG_HASH; > + extra_hash_parts++; > + /* Sometimes it can happen, that we does not parsed hash_str. > + Like when this function is called in ha_create. So we will > + Use field from table->field rather then share->field*/ > + Item *h_item= table->field[s_keyinfo->key_part->fieldnr - 1]-> > + vcol_info->expr; > + uint hash_parts= fields_in_hash_str(h_item); > + s_keyinfo->key_part= s_keyinfo->key_part- hash_parts; > + s_keyinfo->user_defined_key_parts= s_keyinfo->usable_key_parts= > + s_keyinfo->ext_key_parts= hash_parts; > + extra_key_parts_ex_hash+= hash_parts; > + s_keyinfo->key_length= -1; > + } > + } > + if (!keyinfo) > + { > + table->s->key_parts-= extra_hash_parts; > + table->s->key_parts+= extra_key_parts_ex_hash; > + table->s->ext_key_parts+= extra_key_parts_ex_hash + extra_hash_parts; > + } > + } > +} > + > +/** > + @brief set_hash > + @param table > + @param key_index > + @param key_buff > + > +int get_hash_key(THD *thd,TABLE *table, handler *h, uint key_index, > + uchar * rec_buff, uchar *key_buff) unused? > +{ > + KEY *keyinfo= &table->key_info[key_index]; > + DBUG_ASSERT(keyinfo->flags & HA_LONG_UNIQUE_HASH); > + KEY_PART_INFO *temp_key_part= table->key_info[key_index].key_part; > + Field *fld[keyinfo->user_defined_key_parts]; > + Field *t_field; > + uchar hash_buff[9]; > + ulong nr1= 1, nr2= 4; > + String *str1, temp1; > + String *str2, temp2; > + bool is_null= false; > + bool is_same= true; > + bool is_index_inited= h->inited; > + /* difference between field->ptr and start of rec_buff * > + long diff = rec_buff- table->record[0]; > + int result= 0; > + for (uint i=0; i < keyinfo->user_defined_key_parts; i++, temp_key_part++) > + { > + uint maybe_null= MY_TEST(temp_key_part->null_bit); > + fld[i]= temp_key_part->field->new_key_field(thd->mem_root, table, > + key_buff + maybe_null, 12, > + maybe_null?key_buff:0, 1, > + temp_key_part->key_part_flag > + & HA_HASH_KEY_PART_FLAG); > + if (fld[i]->is_real_null()) > + { > + is_null= true; > + break; > + } > + str1= fld[i]->val_str(&temp1); > + calc_hash_for_unique(nr1, nr2, str1); > + key_buff+= temp_key_part->store_length; > + } > + if (is_null && !(keyinfo->flags & HA_NULL_PART_KEY)) > + return HA_ERR_KEY_NOT_FOUND; > + if (keyinfo->flags & HA_NULL_PART_KEY) > + { > + hash_buff[0]= is_null; > + int8store(hash_buff + 1, nr1); > + } > + else > + int8store(hash_buff, nr1); > + if (!is_index_inited) > + result= h->ha_index_init(key_index, 0); > + if (result) > + return result; > + > + setup_table_hash(table); > + result= h->ha_index_read_map(rec_buff, hash_buff, HA_WHOLE_KEY, > + HA_READ_KEY_EXACT); > + re_setup_table(table); > + if (!result) > + { > + for (uint i=0; i < keyinfo->user_defined_key_parts; i++) > + { > + t_field= keyinfo->key_part[i].field; > + t_field->move_field(t_field->ptr+diff, > + t_field->null_ptr+diff, t_field->null_bit); > + } > + do > + { > + re_setup_table(table); > + is_same= true; > + for (uint i=0; i < keyinfo->user_defined_key_parts; i++) > + { > + t_field= keyinfo->key_part[i].field; > + if (fld[i]->is_real_null() && t_field->is_real_null()) > + continue; > + if (!fld[i]->is_real_null() && !t_field->is_real_null()) > + { > + str1= t_field->val_str(&temp1); > + str2= fld[i]->val_str(&temp2); > + if (my_strcasecmp(str1->charset(), str1->c_ptr_safe(), > + str2->c_ptr_safe())) > + { > + is_same= false; > + break; > + } > + } > + else > + { > + is_same= false; > + break; > + } > + } > + setup_table_hash(table); > + } > + while (!is_same && !(result= h->ha_index_next_same(rec_buff, hash_buff, > + keyinfo->key_length))); > + for (uint i=0; i < keyinfo->user_defined_key_parts; i++) > + { > + t_field= keyinfo->key_part[i].field; > + t_field->move_field(t_field->ptr-diff, > + t_field->null_ptr-diff, t_field->null_bit); > + } > + } > + if (!is_index_inited) > + h->ha_index_end(); > + re_setup_table(table); > + for (uint i=0; i < keyinfo->user_defined_key_parts; i++) > + { > + if (keyinfo->key_part[i].key_part_flag & HA_FIELD_EX_FREED) > + { > + Field_blob *blb= static_cast<Field_blob *>(fld[i]); > + uchar * addr; > + blb->get_ptr(&addr); > + my_free(addr); > + } > + } > + return result; > +} > +*/ > LEX_CSTRING *fk_option_name(enum_fk_option opt) > { > static LEX_CSTRING names[]= > @@ -8814,3 +9293,15 @@ bool TABLE::export_structure(THD *thd, Row_definition_list *defs) > } > return false; > } > + > +void calc_hash_for_unique(ulong &nr1, ulong &nr2, String *str) > +{ > + CHARSET_INFO *cs; > + uchar l[4]; > + int4store(l, str->length()); > + cs= &my_charset_bin; > + cs->coll->hash_sort(cs, l, sizeof(l), &nr1, &nr2); > + cs= str->charset(); > + cs->coll->hash_sort(cs, (uchar *)str->ptr(), str->length(), &nr1, &nr2); > + sql_print_information("setiya %lu, %s", nr1, str->ptr()); really? :) > +} > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 4ac8b102d79..f42687c0377 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -3343,6 +3343,87 @@ mysql_add_invisible_index(THD *thd, List<Key> *key_list, > key_list->push_back(key, thd->mem_root); > return key; > } > +/** > + Add hidden level 3 hash field to table in case of long s/hidden level 3/fully invisible/ > + unique column > + @param thd Thread Context. > + @param create_list List of table fields. > + @param cs Field Charset > + @param key_info Whole Keys buffer > + @param key_index Index of current key 1. instead of key_info and key_index, you can just take a key_info of the current key here (that is key_info + key_index). 2. what is the "current key"? Is it the unique key over blobs that you create a hash field for? please clarify the comment. > +*/ > + > +static void add_hash_field(THD * thd, List<Create_field> *create_list, > + CHARSET_INFO *cs, KEY *key_info, int key_index) > +{ > + List_iterator<Create_field> it(*create_list); > + Create_field *dup_field, *cf= new (thd->mem_root) Create_field(); > + cf->flags|= UNSIGNED_FLAG | LONG_UNIQUE_HASH_FIELD; > + cf->charset= cs; > + cf->decimals= 0; > + cf->length= cf->char_length= cf->pack_length= HA_HASH_FIELD_LENGTH; > + cf->invisible= INVISIBLE_FULL; > + cf->pack_flag|= FIELDFLAG_MAYBE_NULL; > + uint num= 1; > + char *temp_name= (char *)thd->alloc(30); > + my_snprintf(temp_name, 30, "DB_ROW_HASH_%u", num); please, give names for 30 and for the whole my_snprintf - macros, functions, whatever. And better use a LEX_STRING for it, for example: #define LONG_HASH_FIELD_NAME_LENGTH 30 static inline make_long_hash_field_name(LEX_STRING buf, uint num) { buf->length= my_snprintf(buf->str, LONG_HASH_FIELD_NAME_LENGTH, "DB_ROW_HASH_%u", num); } > + /* > + Check for collusions collisions :) > + */ > + while ((dup_field= it++)) > + { > + if (!my_strcasecmp(system_charset_info, temp_name, dup_field->field_name.str)) > + { > + num++; > + my_snprintf(temp_name, 30, "DB_ROW_HASH_%u", num); > + it.rewind(); > + } > + } > + it.rewind(); > + cf->field_name.str= temp_name; > + cf->field_name.length= strlen(temp_name); you won't need strlen here, if you use LEX_STRING above > + cf->set_handler(&type_handler_longlong); > + /* > + We have added db_row_hash field in starting of > + fields array , So we have to change key_part > + field index > + for (int i= 0; i <= key_index; i++, key_info++) > + { > + KEY_PART_INFO *info= key_info->key_part; > + for (uint j= 0; j < key_info->user_defined_key_parts; j++, info++) > + { > + info->fieldnr+= 1; > + info->offset+= HA_HASH_FIELD_LENGTH; > + } > + }*/ Forgot to remove it? > + key_info[key_index].flags|= HA_NOSAME; > + key_info[key_index].algorithm= HA_KEY_ALG_LONG_HASH; > + it.rewind(); > + uint record_offset= 0; > + while ((dup_field= it++)) > + { > + dup_field->offset= record_offset; > + if (dup_field->stored_in_db()) > + record_offset+= dup_field->pack_length; Why do you change all field offsets? You only need to put your field last, that's all. Like while ((dup_field= it++)) set_if_bigger(record_offset, dup_field->offset + dup_field->pack_length) > + } > + cf->offset= record_offset; > + /* > + it.rewind(); > + while ((sql_field= it++)) > + { > + if (!sql_field->stored_in_db()) > + { > + sql_field->offset= record_offset; > + record_offset+= sql_field->pack_length; > + } > + } why is that? > + */ > + /* hash column should be fully hidden */ > + //prepare_create_field(cf, NULL, 0); Forgot to remove it? > + create_list->push_back(cf,thd->mem_root); > +} > + > + > /* > Preparation for table creation > > @@ -3868,6 +3951,8 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > } > > cols2.rewind(); > + key_part_info->fieldnr= field; > + key_part_info->offset= (uint16) sql_field->offset; why did you need to move that? > if (key->type == Key::FULLTEXT) > { > if ((sql_field->real_field_type() != MYSQL_TYPE_STRING && > @@ -3922,8 +4007,19 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > column->length= MAX_LEN_GEOM_POINT_FIELD; > if (!column->length) > { > - my_error(ER_BLOB_KEY_WITHOUT_LENGTH, MYF(0), column->field_name.str); > - DBUG_RETURN(TRUE); > + if (key->type == Key::PRIMARY) > + { > + my_error(ER_BLOB_KEY_WITHOUT_LENGTH, MYF(0), column->field_name.str); > + DBUG_RETURN(TRUE); > + } > + else if (!is_hash_field_added) > + { > + add_hash_field(thd, &alter_info->create_list, > + create_info->default_table_charset, > + *key_info_buffer, key_number); > + column->length= 0; why column->length= 0 ? > + is_hash_field_added= true; > + } > } > } > #ifdef HAVE_SPATIAL > @@ -4062,11 +4159,29 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > } > else > { > - my_error(ER_TOO_LONG_KEY, MYF(0), key_part_length); > - DBUG_RETURN(TRUE); > - } > + if(key->type == Key::UNIQUE) > + { > + if (!is_hash_field_added) > + { > + add_hash_field(thd, &alter_info->create_list, > + create_info->default_table_charset, > + *key_info_buffer, key_number); > + is_hash_field_added= true; instead of is_hash_field_added and many add_hash_field() here and there, I'd rather rename the variable to hash_field_needed. And only do hash_field_needed= true. And at the end if (hash_field_needed) add_hash_field(...) > + } > + } > + else > + { > + my_error(ER_TOO_LONG_KEY, MYF(0), key_part_length); any test cases for this error? > + DBUG_RETURN(TRUE); > + } > + } > } > - key_part_info->length= (uint16) key_part_length; > + /* We can not store key_part_length more then 2^16 - 1 in frm > + So we will simply make it zero */ Not really. If someone explicitly asks for a long prefix, it should be an error. Like CREATE TABLE (a blob, UNIQUE (a(65537)); -- this is an error > + if (is_hash_field_added && key_part_length > (2<<16) - 1) > + key_part_info->length= 0; > + else > + key_part_info->length= (uint16) key_part_length; > /* Use packed keys for long strings on the first column */ > if (!((*db_options) & HA_OPTION_NO_PACK_KEYS) && > !((create_info->table_options & HA_OPTION_NO_PACK_KEYS)) && > @@ -4122,12 +4237,37 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > if (key->type == Key::UNIQUE && !(key_info->flags & HA_NULL_PART_KEY)) > unique_key=1; > key_info->key_length=(uint16) key_length; > - if (key_length > max_key_length && key->type != Key::FULLTEXT) > + if (key_length > max_key_length && key->type != Key::FULLTEXT && > + !is_hash_field_added) > { > my_error(ER_TOO_LONG_KEY,MYF(0),max_key_length); > DBUG_RETURN(TRUE); > } > > + if (is_hash_field_added) > + { > + if (key_info->flags & HA_NULL_PART_KEY) > + null_fields++; > + else > + { > + uint elements= alter_info->create_list.elements; > + Create_field *hash_fld= static_cast<Create_field *>(alter_info-> > + create_list.elem(elements -1 )); > + hash_fld->flags|= NOT_NULL_FLAG; > + hash_fld->pack_flag&= ~FIELDFLAG_MAYBE_NULL; > + /* > + Althought we do not need default value anywhere in code , but if we create > + table with non null long columns , then at the time of insert we get warning. > + So default value is used so solve this warning. > + Virtual_column_info *default_value= new (thd->mem_root) Virtual_column_info(); > + char * def_str= (char *)alloc_root(thd->mem_root, 2); > + strncpy(def_str, "0", 1); > + default_value->expr_str.str= def_str; > + default_value->expr_str.length= 1; > + default_value->expr_item= new (thd->mem_root) Item_int(thd,0); > + hash_fld->default_value= default_value; */ > + } Forgot to remove it? > + } > if (validate_comment_length(thd, &key->key_create_info.comment, > INDEX_COMMENT_MAXLEN, > ER_TOO_LONG_INDEX_COMMENT, > @@ -8328,6 +8468,11 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > enum Key::Keytype key_type; > LEX_CSTRING tmp_name; > bzero((char*) &key_create_info, sizeof(key_create_info)); > + if (key_info->flags & HA_LONG_UNIQUE_HASH) > + { > + key_info->flags&= ~(HA_LONG_UNIQUE_HASH); > + key_info->algorithm= HA_KEY_ALG_UNDEF; > + } Why? > key_create_info.algorithm= key_info->algorithm; > /* > We copy block size directly as some engines, like Area, sets this > diff --git a/sql/handler.cc b/sql/handler.cc > index b77b2a3fa2c..cda53dfdc87 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -6179,6 +6185,185 @@ int handler::ha_reset() > DBUG_RETURN(reset()); > } > > +static int check_duplicate_long_entry_key(TABLE *table, handler *h, uchar *new_rec, > + uint key_no) > +{ > + Field *hash_field; > + int result, error= 0; > + if (!(table->key_info[key_no].user_defined_key_parts == 1 > + && table->key_info[key_no].key_part->field->flags & LONG_UNIQUE_HASH_FIELD )) > + return 0; What if LONG_UNIQUE_HASH_FIELD is set but user_defined_key_parts != 1? > + hash_field= table->key_info[key_no].key_part->field; > + DBUG_ASSERT((table->key_info[key_no].flags & HA_NULL_PART_KEY && > + table->key_info[key_no].key_length == HA_HASH_KEY_LENGTH_WITH_NULL) > + || table->key_info[key_no].key_length == HA_HASH_KEY_LENGTH_WITHOUT_NULL); > + uchar ptr[HA_HASH_KEY_LENGTH_WITH_NULL]; > + > + if (hash_field->is_real_null()) > + return 0; > + > + key_copy(ptr, new_rec, &table->key_info[key_no], > + table->key_info[key_no].key_length, false); good, use existing function, no need to reinvent the wheel > + > + if (!table->check_unique_buf) > + table->check_unique_buf= (uchar *)alloc_root(&table->mem_root, > + table->s->reclength*sizeof(uchar)); > + > + result= h->ha_index_init(key_no, 0); > + if (result) > + return result; > + result= h->ha_index_read_map(table->check_unique_buf, > + ptr, HA_WHOLE_KEY, HA_READ_KEY_EXACT); > + if (!result) > + { > + bool is_same; > + do > + { > + Item_func_or_sum * temp= static_cast<Item_func_or_sum *>(hash_field-> > + vcol_info->expr); > + Item_args * t_item= static_cast<Item_args *>(temp); > + uint arg_count= t_item->argument_count(); > + Item ** arguments= t_item->arguments(); > + long diff= table->check_unique_buf - new_rec; > + Field * t_field; > + is_same= true; > + for (uint j=0; j < arg_count; j++) > + { > + DBUG_ASSERT(arguments[j]->type() == Item::FIELD_ITEM || > + // this one for left(fld_name,length) > + arguments[j]->type() == Item::FUNC_ITEM); > + if (arguments[j]->type() == Item::FIELD_ITEM) > + { > + t_field= static_cast<Item_field *>(arguments[j])->field; > + if (t_field->cmp_offset(diff)) > + is_same= false; > + } > + else > + { > + Item_func_left *fnc= static_cast<Item_func_left *>(arguments[j]); > + DBUG_ASSERT(!my_strcasecmp(system_charset_info, "left", fnc->func_name())); > + //item_data= fnc->val_str(&tmp1); > + DBUG_ASSERT(fnc->arguments()[0]->type() == Item::FIELD_ITEM); > + t_field= static_cast<Item_field *>(fnc->arguments()[0])->field; > + // field_data= t_field->val_str(&tmp2); > + // if (my_strnncoll(t_field->charset(),(const uchar *)item_data->ptr(), > + // item_data->length(), > + // (const uchar *)field_data.ptr(), > + // item_data->length())) > + // return 0; > + uint length= fnc->arguments()[1]->val_int(); > + if (t_field->cmp_max(t_field->ptr, t_field->ptr + diff, length)) > + is_same= false; > + } > + } you don't need all that code, use key_cmp() or key_cmp_if_same() from key.cc > + } > + while (!is_same && !(result= table->file->ha_index_next_same(table->check_unique_buf, > + ptr, table->key_info[key_no].key_length))); > + if (is_same) > + { > + table->dupp_hash_key= key_no; > + error= HA_ERR_FOUND_DUPP_KEY; > + goto exit; > + } > + else > + goto exit; > + } > + if (result == HA_ERR_LOCK_WAIT_TIMEOUT) > + { > + table->dupp_hash_key= key_no; > + //TODO check if this is the only case > + error= HA_ERR_FOUND_DUPP_KEY; Why? > + } > + exit: > + h->ha_index_end(); > + return error; > +} > +/** @brief > + check whether inserted/updated records breaks the > + unique constraint on long columns. > + In the case of update we just need to check the specic key > + reason for that is consider case > + create table t1(a blob , b blob , x blob , y blob ,unique(a,b) > + ,unique(x,y)) > + and update statement like this > + update t1 set a=23+a; in this case if we try to scan for > + whole keys in table then index scan on x_y will return 0 > + because data is same so in the case of update we take > + key as a parameter in normal insert key should be -1 > + @returns 0 if no duplicate else returns error > + */ > +static int check_duplicate_long_entries(TABLE *table, handler *h, uchar *new_rec) > +{ > + table->dupp_hash_key= -1; > + int result; > + for (uint i= 0; i < table->s->keys; i++) > + { > + if ((result= check_duplicate_long_entry_key(table, h, new_rec, i))) > + return result; > + } > + return 0; > +} > + > +/** @brief > + check whether updated records breaks the > + unique constraint on long columns. > + @returns 0 if no duplicate else returns error > + */ > +static int check_duplicate_long_entries_update(TABLE *table, handler *h, uchar *new_rec) > +{ > + Field **f, *field; > + Item *h_item; > + int error= 0; > + bool is_update_handler_null= false; > + /* > + Here we are comparing whether new record and old record are same > + with respect to fields in hash_str > + */ > + long reclength= table->record[1]-table->record[0]; > + for (uint i= 0; i < table->s->keys; i++) > + { > + if (table->key_info[i].user_defined_key_parts == 1 && > + table->key_info[i].key_part->field->flags & LONG_UNIQUE_HASH_FIELD) > + { > + /* > + Currently mysql_update is pacthed so that it will automatically set the > + Update handler and then free it but ha_update_row is used in many function ( > + like in case of reinsert) Instead of patching them all here we check is > + update_handler is null then set it And then set it null again > + */ > + if (!table->update_handler) > + { > + create_update_handler(current_thd, table); > + is_update_handler_null= true; > + } > + h_item= table->key_info[i].key_part->field->vcol_info->expr; > + for (f= table->field; f && (field= *f); f++) > + { > + if ( find_field_pos_in_hash(h_item, field->field_name.str) != -1) > + { > + /* Compare fields if they are different then check for duplicates*/ > + if(field->cmp_binary_offset(reclength)) > + { > + if((error= check_duplicate_long_entry_key(table, table->update_handler, > + new_rec, i))) > + goto exit; > + /* > + break beacuse check_duplicate_long_entrie_key will > + take care of remaning fields > + */ > + break; > + } > + } > + } > + } > + } > + exit: > + if (is_update_handler_null) > + { > + delete_update_handler(current_thd, table); > + } > + return error; > +} > > int handler::ha_write_row(uchar *buf) > { > @@ -6189,14 +6374,21 @@ int handler::ha_write_row(uchar *buf) > DBUG_ENTER("handler::ha_write_row"); > DEBUG_SYNC_C("ha_write_row_start"); > > + setup_table_hash(table); No-no. You cannot modify table structure back and forth for *every inserted row*. It's ok to do it once, when a table is opened. But not for every row, > MYSQL_INSERT_ROW_START(table_share->db.str, table_share->table_name.str); > mark_trx_read_write(); > increment_statistics(&SSV::ha_write_count); > > + if ((error= check_duplicate_long_entries(table, table->file, buf))) > + { > + re_setup_table(table); > + DBUG_RETURN(error); > + } > TABLE_IO_WAIT(tracker, m_psi, PSI_TABLE_WRITE_ROW, MAX_KEY, 0, > { error= write_row(buf); }) > > MYSQL_INSERT_ROW_DONE(error); > + re_setup_table(table); > if (likely(!error) && !row_already_logged) > { > rows_changed++; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-17359
by jerome brauge 09 Oct '18

09 Oct '18

Hello Alexander, Can you review this patch for MDEV-17359 ? Best regards. Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de ses destinataires. Toute utilisation ou diffusion, même partielle, non autorisée est interdite. Tout message électronique est susceptible d'altération; CEGID décline donc toute responsabilité au titre de ce message. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'avertir l'expéditeur. This message and any attachments are confidential and intended solely for the addressees. Any unauthorized use or disclosure, either whole or partial is prohibited. E-mails are susceptible to alteration; CEGID shall therefore not be liable for the content of this message. If you are not the intended recipient of this message, please delete it and notify the sender.

2 9

Re: [Maria-developers] d1ccc60: MDEV-14005 Remove need for Partition Key to be part of Primary Key.
by Sergei Golubchik 05 Oct '18

05 Oct '18

Hi, Alexey! Some minor comments, questions, see below... On Aug 21, Alexey Botchkov wrote: > revision-id: d1ccc60360c79cd456abbebafc1c80c8043b7dce (mariadb-10.3.6-106-gd1ccc60) > parent(s): ead9a34a3e934f607c2ea7a6c68f7f6d9d29b5bd > committer: Alexey Botchkov > timestamp: 2018-08-21 14:47:45 +0400 > message: > > MDEV-14005 Remove need for Partition Key to be part of Primary Key. > > The limitation was removed, so now we check all the partition > for unique key duplicates in these cases. > > diff --git a/mysql-test/main/partition_unique.result b/mysql-test/main/partition_unique.result > new file mode 100644 > index 0000000..17a9550 > --- /dev/null > +++ b/mysql-test/main/partition_unique.result > @@ -0,0 +1,51 @@ > +CREATE TABLE t1 ( > +id int(10) NOT NULL, > +status varchar(1) DEFAULT NULL, > +PRIMARY KEY (`id`) > +) > +PARTITION BY LIST COLUMNS(`status`) > +( > +PARTITION `a` VALUES IN ('A'), > +PARTITION `b` VALUES IN ('B'), > +PARTITION `c` VALUES IN ('C'), > +PARTITION `d` DEFAULT); > +INSERT INTO t1 VALUES (4, 'A'); > +INSERT INTO t1 VALUES (6, 'A'); > +INSERT INTO t1 VALUES (4, 'C'); > +ERROR 23000: Duplicate entry '4' for key 'PRIMARY' > +INSERT INTO t1 VALUES (5, 'C'); > +UPDATE t1 SET id=4 WHERE id=5; > +ERROR 23000: Duplicate entry '4' for key 'PRIMARY' > +UPDATE t1 SET id=4 WHERE id=5 AND status='C'; > +ERROR 23000: Duplicate entry '4' for key 'PRIMARY' > +UPDATE t1 SET id=6 WHERE id=4 AND status='A'; > +ERROR 23000: Duplicate entry '6' for key 'PRIMARY' > +select * from t1; > +id status > +4 A > +6 A > +5 C > +connect con1,localhost,root,,test; > +connect con2,localhost,root,,test; > +connection con1; > +SET DEBUG_SYNC= 'berfore_part_unique_check SIGNAL bpu_hit WAIT_FOR bpu_flushed'; may be s/berfore/before/ ? :) > +INSERT INTO t1 VALUES(7, 'A'); > +connection con2; > +SET DEBUG_SYNC= 'now WAIT_FOR bpu_hit'; > +INSERT INTO t1 VALUES(7, 'C'); > +connection default; > +SET DEBUG_SYNC= 'now SIGNAL bpu_flushed'; > +connection con1; > +connection con2; > +ERROR 23000: Duplicate entry '7' for key 'PRIMARY' > +disconnect con1; > +disconnect con2; > +connection default; > +select * from t1; > +id status > +4 A > +6 A > +7 A > +5 C > +DROP TABLE t1; > +set debug_sync= "RESET"; > diff --git a/mysql-test/main/partition_unique.test b/mysql-test/main/partition_unique.test > new file mode 100644 > index 0000000..f292359 > --- /dev/null > +++ b/mysql-test/main/partition_unique.test > @@ -0,0 +1,58 @@ > +--source include/have_innodb.inc > +--source include/have_partition.inc > + > +CREATE TABLE t1 ( > + id int(10) NOT NULL, > + status varchar(1) DEFAULT NULL, > + PRIMARY KEY (`id`) > + ) > + PARTITION BY LIST COLUMNS(`status`) > + ( > + PARTITION `a` VALUES IN ('A'), > + PARTITION `b` VALUES IN ('B'), > + PARTITION `c` VALUES IN ('C'), > + PARTITION `d` DEFAULT); you don't seem to test innodb, this looks like a partitioned myisam table > + > +INSERT INTO t1 VALUES (4, 'A'); > +INSERT INTO t1 VALUES (6, 'A'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES (4, 'C'); > +INSERT INTO t1 VALUES (5, 'C'); > +--error ER_DUP_ENTRY > +UPDATE t1 SET id=4 WHERE id=5; > +--error ER_DUP_ENTRY > +UPDATE t1 SET id=4 WHERE id=5 AND status='C'; > +--error ER_DUP_ENTRY > +UPDATE t1 SET id=6 WHERE id=4 AND status='A'; > +select * from t1; > + > +connect (con1,localhost,root,,test); > +connect (con2,localhost,root,,test); > +connection con1; > +SET DEBUG_SYNC= 'berfore_part_unique_check SIGNAL bpu_hit WAIT_FOR bpu_flushed'; > +send INSERT INTO t1 VALUES(7, 'A'); > + > +connection con2; > +SET DEBUG_SYNC= 'now WAIT_FOR bpu_hit'; > +send INSERT INTO t1 VALUES(7, 'C'); add also a test that you can insert some other value in 'A' and 'C' partitions. That is, partitions aren't completely locked. > + > +connection default; > +SET DEBUG_SYNC= 'now SIGNAL bpu_flushed'; > + > +connection con1; > +--reap > +connection con2; > +--error ER_DUP_ENTRY > +--reap > + > +disconnect con1; > +disconnect con2; > + > +connection default; > + > +select * from t1; > + > +DROP TABLE t1; > + > +set debug_sync= "RESET"; > + > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 262e791..da07075 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -368,6 +368,7 @@ void ha_partition::init_handler_variables() > part_share= NULL; > m_new_partitions_share_refs.empty(); > m_part_ids_sorted_by_num_of_records= NULL; > + m_cu_errkey= (uint) -1; rather confusing name. Could you de-confuse it? Why not simply m_errkey? or m_err_dupkey, or whatever and why did you make it unsigned? you need to cast in every second occurence of it :) > m_partitions_to_open= NULL; > > m_range_info= NULL; > @@ -4198,6 +4199,95 @@ void ha_partition::try_semi_consistent_read(bool yes) > } > > > +int ha_partition::check_files_for_key(uchar *key, int n_key, > + int part_begin, int part_end, > + int part_to_skip, > + int *res) > +{ > + DBUG_ASSERT(inited == NONE || > + (inited == RND && !m_scan_value)); > + > + for (int n=part_begin; n < part_end; n++) > + { > + handler *f= m_file[n]; > + init_stat sav_inited; > + > + if ((int) n == part_to_skip) > + continue; > + > + if ((sav_inited= f->inited) == RND) > + f->ha_rnd_end(); When can this happen? > + > + *res= f->index_read_idx_map(m_part_info->table->record[0], > + n_key, key, HA_WHOLE_KEY, HA_READ_KEY_EXACT); > + > + f->ha_end_keyread(); > + > + if (sav_inited == RND) > + { > + int ires= f->ha_rnd_init(FALSE); > + if (ires) > + *res= ires; > + } > + > + if (*res == HA_ERR_KEY_NOT_FOUND) > + continue; > + > + if (*res == 0) > + *res= HA_ERR_FOUND_DUPP_KEY; > + > + m_last_part= n; > + m_cu_errkey= n_key; > + return 1; > + } > + > + *res= 0; > + return 0; > +} > + > + > +int ha_partition::check_uniques_insert(uchar *buf, > + int part_begin, int part_end, > + KEY **dup_key, > + int *res) > +{ > + uint n_key; > + > + for (n_key=0; n_key < m_part_info->n_uniques_to_check; n_key++) > + { > + uchar *cbuf= m_part_info->unique_key_buf[0]; > + KEY *ckey= m_part_info->uniques_to_check[n_key]; > + uint n; > + > + for (n=0; n < ckey->user_defined_key_parts; n++) > + { can you use key_copy() here? > + const KEY_PART_INFO *cpart= ckey->key_part + n; > + uint maybe_null= MY_TEST(cpart->null_bit); > + Field *f= cpart->field; > + my_ptrdiff_t ofs= buf-table->record[0]; > + > + f->move_field_offset(ofs); > + if (maybe_null) > + cbuf[0]= f->is_null(); > + f->get_key_image(cbuf+maybe_null, cpart->length, Field::itRAW); > + cbuf+= cpart->store_length; > + f->move_field_offset(-ofs); > + } > + > + if (check_files_for_key(m_part_info->unique_key_buf[0], > + table->key_info - ckey, > + part_begin, part_end, -1, res)) > + { > + *dup_key= ckey; > + return 1; > + } > + } > + > + *res= 0; > + return 0; > +} > + > + > /**************************************************************************** > MODULE change record > ****************************************************************************/ > @@ -4307,7 +4398,41 @@ int ha_partition::write_row(uchar * buf) > start_part_bulk_insert(thd, part_id); > > tmp_disable_binlog(thd); /* Do not replicate the low-level changes. */ > + > + /* > + Check unique keys (if there is any) for duplications in > + partitions 0 .. inserted partition, then > + do the write_row then check the unique in > + partitions inserted partition +1 .. m_tot_parts. > + We do so to keep the order of locking always same to > + avoid deadlocks. > + */ > + if (table->part_info->n_uniques_to_check && > + check_uniques_insert(buf, 0, part_id, &dup_key, &error)) > + { > + goto exit; > + } > + > error= m_file[part_id]->ha_write_row(buf); > + > + DEBUG_SYNC(thd, "berfore_part_unique_check"); > + > + if (!error && table->part_info->n_uniques_to_check && > + check_uniques_insert(buf, part_id+1, m_tot_parts, &dup_key, &error)) > + { > + /* > + Errors here are ignored, as the error already found. > + and i don't have a good idea what to do if things go > + wrong here. > + */ > + if (!(m_file[part_id]->ha_index_read_idx_map(table->record[1], > + dup_key - table->key_info, table->part_info->unique_key_buf[0], > + HA_WHOLE_KEY, HA_READ_KEY_EXACT))) 1. why do you need ha_index_read_idx_map here? you've just did ha_write_row, isn't it enough? 2. if the table is transational and no IGNORE was used, you don't need to delete anything, the statement will be rolled back anyway. an easy optimization. > + { > + (void) m_file[part_id]->ha_delete_row(buf); > + } > + } > + > if (have_auto_increment && !table->s->next_number_keypart) > set_auto_increment_if_higher(table->next_number_field); > reenable_binlog(thd); > @@ -4317,6 +4442,59 @@ int ha_partition::write_row(uchar * buf) > table->auto_increment_field_not_null= saved_auto_inc_field_not_null; > DBUG_RETURN(error); > } > + > + > +int ha_partition::check_uniques_update(const uchar *old_data, > + const uchar *new_data, > + int new_part_id, int *res) > +{ > + uint n_key; > + > + for (n_key=0; n_key < m_part_info->n_uniques_to_check; n_key++) > + { > + uchar *buf0= m_part_info->unique_key_buf[0]; > + uchar *buf1= m_part_info->unique_key_buf[1]; > + KEY *ckey= m_part_info->uniques_to_check[n_key]; > + uint n; > + > + for (n=0; n < ckey->user_defined_key_parts; n++) > + { > + const KEY_PART_INFO *cpart= ckey->key_part + n; > + uint maybe_null= MY_TEST(cpart->null_bit); > + Field *f= cpart->field; > + my_ptrdiff_t dif; > + > + dif= old_data - table->record[0]; > + f->move_field_offset(dif); > + if (maybe_null) > + buf0[0]= f->is_null(); > + f->get_key_image(buf0+maybe_null, cpart->length, Field::itRAW); > + buf0+= cpart->store_length; > + f->move_field_offset(-dif); > + > + dif= new_data - table->record[0]; > + f->move_field_offset(dif); > + if (maybe_null) > + buf1[0]= f->is_null(); > + f->get_key_image(buf1+maybe_null, cpart->length, Field::itRAW); > + buf1+= cpart->store_length; > + f->move_field_offset(-dif); > + } > + > + if (memcmp(m_part_info->unique_key_buf[0], m_part_info->unique_key_buf[1], > + buf0 - m_part_info->unique_key_buf[0]) == 0) > + { > + /* Key did not change. */ > + continue; > + } > + > + if (check_files_for_key(m_part_info->unique_key_buf[1], > + table->key_info - ckey, 0, m_tot_parts, new_part_id, res)) How does that work? I'd expect you to do like you did for write_row: - check from 0 to new_part_id, skipping old_part_id, - insert the row - check from new_part_id+1 to m_last_part, skipping old_part_id. > + return 1; > + } > + > + return 0; > +} > > > /* > @@ -5754,11 +5935,14 @@ int ha_partition::index_read_idx_map(uchar *buf, uint index, > get_partition_set(table, buf, index, &m_start_key, &m_part_spec); > > /* > - We have either found exactly 1 partition > + If there is no 'unbound' unique keys where not all keyparts > + are partition definition fields, > + we have either found exactly 1 partition I don't understand. It's HA_READ_KEY_EXACT, so it can never find more than one row in a unique key, and one row can never be in more than one partition. Or do you mean unique key over nullable column that can have many null values? Then mention it in the comment, please. > (in which case start_part == end_part) > - or no matching partitions (start_part > end_part) > + or no matching partitions (start_part > end_part), > */ > - DBUG_ASSERT(m_part_spec.start_part >= m_part_spec.end_part); > + DBUG_ASSERT(m_part_spec.start_part >= m_part_spec.end_part || > + m_part_info->n_uniques_to_check); > /* The start part is must be marked as used. */ > DBUG_ASSERT(m_part_spec.start_part > m_part_spec.end_part || > bitmap_is_set(&(m_part_info->read_partitions), > @@ -8315,15 +8499,20 @@ int ha_partition::info(uint flag) > { > handler *file= m_file[m_last_part]; > DBUG_PRINT("info", ("info: HA_STATUS_ERRKEY")); > - /* > - This flag is used to get index number of the unique index that > - reported duplicate key > - We will report the errkey on the last handler used and ignore the rest > - Note: all engines does not support HA_STATUS_ERRKEY, so set errkey. > - */ > - file->errkey= errkey; > - file->info(HA_STATUS_ERRKEY | no_lock_flag); > - errkey= file->errkey; > + if ((int) m_cu_errkey >= 0) > + errkey= m_cu_errkey; > + else > + { > + /* > + This flag is used to get index number of the unique index that > + reported duplicate key > + We will report the errkey on the last handler used and ignore the rest > + Note: all engines does not support HA_STATUS_ERRKEY, so set errkey. I don't understand what this means (I know it's an old comment, but still) Is it "not all engines support HA_STATUS_ERRKEY"? If yes, please change it to say that. > + */ > + file->errkey= errkey; > + file->info(HA_STATUS_ERRKEY | no_lock_flag); > + errkey= file->errkey; > + } > } > if (flag & HA_STATUS_TIME) > { > diff --git a/sql/partition_info.h b/sql/partition_info.h > index e00a2c4..eeac0d9 100644 > --- a/sql/partition_info.h > +++ b/sql/partition_info.h > @@ -283,6 +283,14 @@ class partition_info : public Sql_alloc > bool is_auto_partitioned; > bool has_null_value; > bool column_list; // COLUMNS PARTITIONING, 5.5+ > + /* > + Unique keys that don't have all the partitioning fields in them > + need to be checked when INSERT/UPDATE. > + So they are collected here. > + */ > + KEY **uniques_to_check; > + uint n_uniques_to_check; > + uchar *unique_key_buf[2]; Is it safe? Isn't partition_info shared between many handlers so that every handler can only use it read-only? it's not quite clear to me. it seems that every TABLE has its own partition_info. But cloned handlers all share the same partition_info. > > partition_info() > : get_partition_id(NULL), get_part_partition_id(NULL), > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 3133b94..6cd46dd 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -988,15 +945,15 @@ static bool check_unique_keys(TABLE *table) > check_fields_in_PF(table->part_info->full_part_field_array, > &all_fields, &some_fields); > clear_indicator_in_key_fields(table->key_info+i); > - if (unlikely(!all_fields)) > + if (!all_fields) > { > - my_error(ER_UNIQUE_KEY_NEED_ALL_FIELDS_IN_PF,MYF(0),"UNIQUE INDEX"); you can rename ER_UNIQUE_KEY_NEED_ALL_FIELDS_IN_PF to ER_UNUSED with the next free number > - result= TRUE; > - break; > + ++keys_found; > + if (table->part_info->n_uniques_to_check >= keys_found) > + table->part_info->uniques_to_check[keys_found-1]= table->key_info+i; > } > } > } > - DBUG_RETURN(result); > + DBUG_RETURN(keys_found); > } > > > @@ -2798,6 +2765,47 @@ bool partition_key_modified(TABLE *table, const MY_BITMAP *fields) > } > > > + /* > + Check if there are unique keys that not entirely 'inside' the partition > + key and if any fields of such keys are modified. > + If it's so, it usually means we have to use tamporary storage for records "temporary" > + handling the UPDATE command. > + > + SYNOPSIS please, use doxygen format for new functions > + partition_unique_modified > + table TABLE object for which partition fields are set-up > + fields Bitmap representing fields to be modified > + > + RETURN VALUES > + TRUE Need special handling of UPDATE > + FALSE Normal UPDATE handling is ok > +*/ > + > +bool partition_unique_modified(TABLE *table, const MY_BITMAP *fields) > +{ > + partition_info *part_info= table->part_info; > + DBUG_ENTER("partition_unique_modified"); > + > + if (!part_info) > + DBUG_RETURN(FALSE); > + > + if (part_info->uniques_to_check == 0) > + DBUG_RETURN(FALSE); > + > + for (uint n_key=0; n_key < part_info->n_uniques_to_check; n_key++) > + { > + KEY *ckey= part_info->uniques_to_check[n_key]; > + for (uint n_part=0; n_part < ckey->user_defined_key_parts; n_part++) > + { > + if (bitmap_is_set(fields, ckey->key_part[n_part].field->field_index)) > + DBUG_RETURN(TRUE); > + } > + } > + > + DBUG_RETURN(FALSE); > +} > + > + > /* > A function to handle correct handling of NULL values in partition > functions. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 0f97f6b8398: MDEV-17346 parallel slave start and stop races to workers disappeared
by Kristian Nielsen 05 Oct '18

05 Oct '18

Hi Andrei! andrei.elkin(a)pp.inet.fi writes: > revision-id: 0f97f6b8398054ccb0507fbacc76c9deeddd47a4 (mariadb-10.1.35-71-g0f97f6b8398) > author: Andrei Elkin > timestamp: 2018-10-03 15:42:12 +0300 > message: > > MDEV-17346 parallel slave start and stop races to workers disappeared Ooh, that's a nice catch. The code around slave start/stop really is terribly tricky, glad that you were able to sort this one out. Patch looks good to push. What about test case? I assume this was caught from occasional random failures in buildbot? That should be ok as test case for this bug, I think. It can be a _lot_ of work to write reliable tests for stuff like this, and the start/stop slave is stressed quite a bit by the existing replication tests. > The bug appears as a slave SQL thread hanging in > rpl_parallel_thread_pool::get_thread() while there are no slave worker > threads to awake it. > > The hang could occur at parallel slave worker pool activation by a > "new" started SQL thread when the pool was concurrently deactivated by > being terminated "old" SQL thread. At reading the current pool size > the SQL thread did not employ necessary protection designed by > MDEV-9573. The pool can't be deactivated when there is an active slave > but the "new" slave might set its active status too late while seeing > the pool still non-empty. The end product of four computaional events > > Old_slave:any_slave_sql_running() => 0 > New_slave:slave_running= "active" > New_slave:global_rpl_thread_pool.size > 0 => true > Old_slave:global_rpl_thread_pool.size := 0 > > could led to the observed hang. The new SQL thread proceeds to scheduling > having all workers gone. > > Fixed with making the SQL thread at the pool activation first > to grab the same lock as potential deactivator also does prior > to destroy the pool. At first I did not see how this fixes the problem. But it is because of the extra check inside rpl_parallel_change_thread_count(): if (!new_count && !force) { if (any_slave_sql_running()) { pool_mark_not_busy(pool); return 0; // Ok to not resize pool } } So one possibility is that New_slave has set running=active when Old_slave hits that test. Then Old_slave will leave the pool intact thanks to this test. The other possibility is that New_slave has not yet set running=active at this point, so Old_slave can empty the pool. But then Old_slave already marked the pool busy. So the patch will ensure that New_slave will see that the pool has been emptied, and will know to re-populate it before starting. Seems solid. Thanks for fixing! - Kristian. > --- > sql/rpl_parallel.cc | 25 ++++++++++++++++++++++--- > 1 file changed, 22 insertions(+), 3 deletions(-) > > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 35cddee6d4d..8fef2d66635 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1617,13 +1617,32 @@ int rpl_parallel_resize_pool_if_no_slaves(void) > } > > > +/** > + Pool activation is preceeded by taking a "lock" of pool_mark_busy > + which guarantees the number of running slaves drops to zero atomicly > + with the number of pool workers. > + This resolves race between the function caller thread and one > + that may be attempting to deactivate the pool. > +*/ > int > rpl_parallel_activate_pool(rpl_parallel_thread_pool *pool) > { > + int rc= 0; > + > + if ((rc= pool_mark_busy(pool, current_thd))) > + return rc; // killed > + > if (!pool->count) > - return rpl_parallel_change_thread_count(pool, opt_slave_parallel_threads, > - 0); > - return 0; > + { > + pool_mark_not_busy(pool); > + rc= rpl_parallel_change_thread_count(pool, opt_slave_parallel_threads, > + 0); > + } > + else > + { > + pool_mark_not_busy(pool); > + } > + return rc; > }

1 0

Re: [Maria-developers] c45b4a7: MDEV-17200 - pthread_detach called for already detached threads
by Sergei Golubchik 05 Oct '18

05 Oct '18

Hi, Sergey! ok to push! On Sep 21, Sergey Vojtovich wrote: > revision-id: c45b4a774b6d1404a080a1c1759f780fa78f223b (mariadb-10.1.35-65-gc45b4a7) > parent(s): 82524239c48a2603f115a6117e061751f6427d08 > author: Sergey Vojtovich > committer: Sergey Vojtovich > timestamp: 2018-09-21 16:24:23 +0400 > message: > > MDEV-17200 - pthread_detach called for already detached threads > > pthread_detach_this_thread() was intended to be defined to something > meaningful only on some ancient unixes, which don't have > pthread_attr_setdetachstate() defined. Otherwise, on normal unixes, > threads are created detached in the first place. > > This was broken in 0f01bf267680244ec488adaf65a42838756ed48e so that > we started calling pthread_detach() for already detached threads. > Intention was to detach aria checkpoint thread. > > However in 87007dc2f71634cc460271eb277ad851ec69c04b aria service threads > were made joinable with appropriate handling, which makes breaking > revision unneccessary. > > Revert remnants of 0f01bf267680244ec488adaf65a42838756ed48e, so that > pthread_detach_this_thread() is meaningful only on some ancient unixes > again. > > --- > include/my_pthread.h | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/include/my_pthread.h b/include/my_pthread.h > index 6b830ca..ae2f912 100644 > --- a/include/my_pthread.h > +++ b/include/my_pthread.h > @@ -184,7 +184,7 @@ int pthread_cancel(pthread_t thread); > #define pthread_key(T,V) pthread_key_t V > #define my_pthread_getspecific_ptr(T,V) my_pthread_getspecific(T,(V)) > #define my_pthread_setspecific_ptr(T,V) pthread_setspecific(T,(void*) (V)) > -#define pthread_detach_this_thread() { pthread_t tmp=pthread_self() ; pthread_detach(tmp); } > +#define pthread_detach_this_thread() > #define pthread_handler_t EXTERNC void * > typedef void *(* pthread_handler)(void *); > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 0f97f6b8398: MDEV-17346 parallel slave start and stop races to workers disappeared
by andrei.elkin＠pp.inet.fi 03 Oct '18

03 Oct '18

Kristian, hello. Perhaps you could review this worker pool patch though it actually follows up changes made by MDEV-9573, by Monty. Cheers, Andrei > revision-id: 0f97f6b8398054ccb0507fbacc76c9deeddd47a4 (mariadb-10.1.35-71-g0f97f6b8398) > parent(s): 1fc5a6f30c3a9c047dcf9a36b00026d98f286f6b > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-10-03 15:42:12 +0300 > message: > > MDEV-17346 parallel slave start and stop races to workers disappeared > > The bug appears as a slave SQL thread hanging in > rpl_parallel_thread_pool::get_thread() while there are no slave worker > threads to awake it. > > The hang could occur at parallel slave worker pool activation by a > "new" started SQL thread when the pool was concurrently deactivated by > being terminated "old" SQL thread. At reading the current pool size > the SQL thread did not employ necessary protection designed by > MDEV-9573. The pool can't be deactivated when there is an active slave > but the "new" slave might set its active status too late while seeing > the pool still non-empty. The end product of four computaional events > > Old_slave:any_slave_sql_running() => 0 > New_slave:slave_running= "active" > New_slave:global_rpl_thread_pool.size > 0 => true > Old_slave:global_rpl_thread_pool.size := 0 > > could led to the observed hang. The new SQL thread proceeds to scheduling > having all workers gone. > > Fixed with making the SQL thread at the pool activation first > to grab the same lock as potential deactivator also does prior > to destroy the pool. > > --- > sql/rpl_parallel.cc | 25 ++++++++++++++++++++++--- > 1 file changed, 22 insertions(+), 3 deletions(-) > > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 35cddee6d4d..8fef2d66635 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1617,13 +1617,32 @@ int rpl_parallel_resize_pool_if_no_slaves(void) > } > > > +/** > + Pool activation is preceeded by taking a "lock" of pool_mark_busy > + which guarantees the number of running slaves drops to zero atomicly > + with the number of pool workers. > + This resolves race between the function caller thread and one > + that may be attempting to deactivate the pool. > +*/ > int > rpl_parallel_activate_pool(rpl_parallel_thread_pool *pool) > { > + int rc= 0; > + > + if ((rc= pool_mark_busy(pool, current_thd))) > + return rc; // killed > + > if (!pool->count) > - return rpl_parallel_change_thread_count(pool, opt_slave_parallel_threads, > - 0); > - return 0; > + { > + pool_mark_not_busy(pool); > + rc= rpl_parallel_change_thread_count(pool, opt_slave_parallel_threads, > + 0); > + } > + else > + { > + pool_mark_not_busy(pool); > + } > + return rc; > } > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Track of memory consumption.
by jerome brauge 02 Oct '18

02 Oct '18

Hi, I'd like to know what is the best way to track memory consumption /leak in mariadb. Our application use intensively stored procedure and with only 35 connection we have to restart mariadb each 1.5 hours because it use 15Go of memory very fast. Mysqladmin doesn't show excessive memory consumption : | Max_used_connections | 42 | | Memory_used | 2422894288 | | Memory_used_initial | 291786624 | But [root@vmdevmdb ~]# ps -up 20039 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND mysql 20039 10.5 73.8 14574168 12000852 ? Ssl 14:30 11:19 /usr/sbin/mysqld --> 12Go of resident memory in only 11 minutes of cpu. Best regards Jérôme. Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de ses destinataires. Toute utilisation ou diffusion, même partielle, non autorisée est interdite. Tout message électronique est susceptible d'altération; CEGID décline donc toute responsabilité au titre de ce message. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'avertir l'expéditeur. This message and any attachments are confidential and intended solely for the addressees. Any unauthorized use or disclosure, either whole or partial is prohibited. E-mails are susceptible to alteration; CEGID shall therefore not be liable for the content of this message. If you are not the intended recipient of this message, please delete it and notify the sender.

1 0

[Maria-developers] CONC-312; caching_sha2_plugin
by Michal Schorm 02 Oct '18

02 Oct '18

Hello, Can I get some explanation, why was https://jira.mariadb.org/browse/CONC-312 closed as WON'T FIX ? -- MySQL 8.0 implemented "caching_sha2_password" as its default authentification plugin / method. MariaDB doesn't have support for it so mariadb-connector-c cannot be used for MySQL 8. That makes problems for nearly every Linux OS, beacuse they chose MariaDB over MySQL as the default implemetation. I'd like to know the MariaDB stance to it. Do you see it as an issue as well? Are there any plans to deal with it? How? Michal Schorm -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

2 2

Re: [Maria-developers] e11e0e6c8b9: MDEV-17167 - InnoDB: Failing assertion: table->get_ref_count() == 0 upon
by Sergey Vojtovich 24 Sep '18

24 Sep '18

Hi Sergei, On Mon, Sep 24, 2018 at 03:02:40PM +0200, Sergei Golubchik wrote: > Hi, Sergey! > > On Sep 12, Sergey Vojtovich wrote: > > revision-id: e11e0e6c8b9a85ca7f16c6b9b28fb505d464a96e (mariadb-10.3.7-186-ge11e0e6c8b9) > > parent(s): f2f661b848c02ebd7c444ba645b26416d3e4d2ad > > author: Sergey Vojtovich > > committer: Sergey Vojtovich > > timestamp: 2018-09-12 16:36:45 +0400 > > message: > > > > MDEV-17167 - InnoDB: Failing assertion: table->get_ref_count() == 0 upon > > truncating a temporary table > > > > TRUNCATE expects only one TABLE instance (which is used by TRUNCATE > > itself) to be open. However this requirement wasn't enforced after > > "MDEV-5535: Cannot reopen temporary table". > > > > Fixed by closing unused table instances before performing TRUNCATE. > > > diff --git a/sql/sql_class.h b/sql/sql_class.h > > index 6256522bf5c..e7dcd997e05 100644 > > --- a/sql/sql_class.h > > +++ b/sql/sql_class.h > > @@ -4612,6 +4612,7 @@ class THD :public Statement, > > > > TMP_TABLE_SHARE* save_tmp_table_share(TABLE *table); > > void restore_tmp_table_share(TMP_TABLE_SHARE *share); > > + void close_unused_temporary_table_instances(const TABLE_LIST *tl); > > > > private: > > /* Whether a lock has been acquired? */ > > diff --git a/sql/sql_truncate.cc b/sql/sql_truncate.cc > > index 201825d4593..695fcb538f9 100644 > > --- a/sql/sql_truncate.cc > > +++ b/sql/sql_truncate.cc > > @@ -401,6 +401,8 @@ bool Sql_cmd_truncate_table::truncate_table(THD *thd, TABLE_LIST *table_ref) > > /* In RBR, the statement is not binlogged if the table is temporary. */ > > binlog_stmt= !thd->is_current_stmt_binlog_format_row(); > > > > + thd->close_unused_temporary_table_instances(table_ref); > > Is it only TRUNCATE that needs it? What about ALTER? REPAIR? Other > similar commands? Right, do you think it is worth to exapnd scope of this bug and look for other possible issues? > > +*/ > > + > > +void THD::close_unused_temporary_table_instances(const TABLE_LIST *tl) > > +{ > > + TMP_TABLE_SHARE *share= find_tmp_table_share(tl); > > + > > + if (share) > > + { > > + Share_free_tables::List purge_tables; > > + All_share_tables_list::Iterator tables_it(share->all_tmp_tables); > > + > > + while (TABLE *table= tables_it++) > > + { > > + if (table->query_id == 0) > > + purge_tables.push_front(table); > > + } > > + > > + while (TABLE *table= purge_tables.pop_front()) > > + { > > + share->all_tmp_tables.remove(table); > > + free_temporary_table(table); > > + } > > Why are you doing it in two loops? Because free_temporary_table() > invalidates the iterator? Not free_temporary_table(), but rather share->all_tmp_tables.remove(table). OTOH it seems to be safe to remove current element with it++, but not with ++it. I'll try to remove extra loop if it works. Thanks, Sergey

2 1

[Maria-developers] unsubscribe
by Valentin Rakush 17 Sep '18

17 Sep '18

-- Best Regards, Valentin Rakush.

1 0

[Maria-developers] Compiling default MariaDB with OpenSSL in Windows
by Braiam Peguero 14 Sep '18

14 Sep '18

Hi, Following Georg Richter recommendations for compiling MariaDB on Windows 64bit, with OpenSSL using the following cmake commands: cmake .. -DWITH_SSL=system -G "Visual Studio 15 2017 Win64" cmake --build . --config relwithdebinfo --target package However, several recipes fail to build, for the following errors: "builddir\mysys_ssl\mysys_ssl.vcxproj" (default target) (8) -> (ClCompile target) -> c:\users\braiam\src\mariadb-10.3.9\mysys_ssl\my_crypt.cc(42): error C3861: 'bzero': identifier not found [C:\Users\br aiam\src\mariadb-10.3.9\bld\mysys_ssl\mysys_ssl.vcxproj] and "builddir\package.vcxproj" (default target) (1) -> "builddir\ALL_BUILD.vcxproj" (default target) (3) -> "builddir\libmariadb\sha256_password.vcxproj" (default target) (205) -> (Link target) -> sha256_pw.obj : error LNK2019: unresolved external symbol BIO_free referenced in function auth_sha256_client [C:\User s\braiam\src\mariadb-10.3.9\bld\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol BIO_new_mem_buf referenced in function auth_sha256_client [ builddir\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol RSA_size referenced in function auth_sha256_client [C:\User s\braiam\src\mariadb-10.3.9\bld\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol RSA_public_encrypt referenced in function auth_sha256_clien t [builddir\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol RSA_free referenced in function auth_sha256_client [C:\User s\braiam\src\mariadb-10.3.9\bld\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol PEM_read_bio_RSA_PUBKEY referenced in function auth_sha256_ client [builddir\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol ERR_clear_error referenced in function auth_sha256_client [ builddir\libmariadb\sha256_password.vcxproj] builddir\libmariadb\RelWithDebInfo\sha256_password.dll : fatal error LNK1120: 7 unresol ved externals [builddir\libmariadb\sha256_password.vcxproj] Using the OpenSSL 1.1.0i 64bit binaries from Shinning Light Productions. Using the 1.0.2p version produce even more linking errors. I also used FireDaemon but failed. Is there a recommended Windows binary for compiling? Or do I need to compile my own openssl? -- Braiam Peguero

2 3

Re: [Maria-developers] [Commits] 3c7e2540f63: MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion
by Sergey Petrunia 12 Sep '18

12 Sep '18

Hi Varun, Looking at the testcase name, I have found Serg's patch for MDEV-4285. His patch skips sorting if the keys have zero length. He does it at a late stage, in Filesort_buffer::sort_buffer(). His patch doesn't cover the example of MDEV-17020, because the testcase in MDEV-17020 uses LIMIT and so uses a Priority Queue (and not Filesort_buffer::sort_buffer which uses qsort). However, your patch doesn't cover all cases either. There are other ways to produce zero-length keys. I have found this one: SELECT * FROM t1 ORDER BY 'foo', substring(pk, 0) LIMIT 2; it crashes in the same way as the test for MDEV-17020 used to do. Possible options: 1. Find all Items that might produce zero-length keys. It's difficult not to miss anything. 2. Instead of #1, add a universal logic "if sort key length is 0, don't sort". Is this doable? 3. Extend the Serg's approach. We cannot use his code directly ("dont sort"), because Priority Queue API doesn't easily allow it, but perhaps we could add a comparison function that will allow to compare zero-length keys? Or make them 1-byte long with a dummy value (do we need the sorting to be stable??) On Fri, Aug 31, 2018 at 12:18:08AM +0530, Varun wrote: > revision-id: 3c7e2540f63c6801a0d0a27fbea7c934bb61221f (mariadb-10.0.36-13-g3c7e2540f63) > parent(s): b3c320bb0b93e516cda4db277cfa3efeef48c988 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-08-31 00:14:20 +0530 > message: > > MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion > > We have a case here where the the entire sort key is empty and so sorting is not needed. > The crash happens because filesort expects the sort key to have length > 0. > The case here is like ORDER BY CONVERT(a,char(0)), which if evaluated is a constant so > can be removed from the ORDER BY clause > > --- > mysql-test/r/order_by_zerolength-4285.result | 28 +++++++++++++++++----------- > mysql-test/t/order_by_zerolength-4285.test | 13 +++++++++++++ > sql/item_timefunc.h | 6 ++++++ > 3 files changed, 36 insertions(+), 11 deletions(-) > > diff --git a/mysql-test/r/order_by_zerolength-4285.result b/mysql-test/r/order_by_zerolength-4285.result > index f60ce7d90c7..f77b284df3c 100644 > --- a/mysql-test/r/order_by_zerolength-4285.result > +++ b/mysql-test/r/order_by_zerolength-4285.result > @@ -12,15 +12,21 @@ pk > 8 > 9 > 10 > -Warnings: > -Warning 1292 Truncated incorrect CHAR(0) value: '1' > -Warning 1292 Truncated incorrect CHAR(0) value: '2' > -Warning 1292 Truncated incorrect CHAR(0) value: '3' > -Warning 1292 Truncated incorrect CHAR(0) value: '4' > -Warning 1292 Truncated incorrect CHAR(0) value: '5' > -Warning 1292 Truncated incorrect CHAR(0) value: '6' > -Warning 1292 Truncated incorrect CHAR(0) value: '7' > -Warning 1292 Truncated incorrect CHAR(0) value: '8' > -Warning 1292 Truncated incorrect CHAR(0) value: '9' > -Warning 1292 Truncated incorrect CHAR(0) value: '10' > +drop table t1; > +# > +# MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion > +# > +set @save_sql_mode= @@sql_mode; > +SET @@sql_mode= ''; > +CREATE TABLE t1 (pk INT PRIMARY KEY); > +INSERT INTO t1 VALUES (1),(2); > +explain > +SELECT * FROM t1 ORDER BY 'foo', CONVERT(pk, CHAR(0)) LIMIT 2; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index NULL PRIMARY 4 NULL 2 Using index > +SELECT * FROM t1 ORDER BY 'foo', Cast(pk as CHAR(0)) LIMIT 2; > +pk > +1 > +2 > +set @@sql_mode= @save_sql_mode; > drop table t1; > diff --git a/mysql-test/t/order_by_zerolength-4285.test b/mysql-test/t/order_by_zerolength-4285.test > index 2fb58edd36d..f03d528320c 100644 > --- a/mysql-test/t/order_by_zerolength-4285.test > +++ b/mysql-test/t/order_by_zerolength-4285.test > @@ -6,3 +6,16 @@ insert into t1 values (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); > select * from t1 order by now(), cast(pk as char(0)); > drop table t1; > > +--echo # > +--echo # MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion > +--echo # > + > +set @save_sql_mode= @@sql_mode; > +SET @@sql_mode= ''; > +CREATE TABLE t1 (pk INT PRIMARY KEY); > +INSERT INTO t1 VALUES (1),(2); > +explain > +SELECT * FROM t1 ORDER BY 'foo', CONVERT(pk, CHAR(0)) LIMIT 2; > +SELECT * FROM t1 ORDER BY 'foo', Cast(pk as CHAR(0)) LIMIT 2; > +set @@sql_mode= @save_sql_mode; > +drop table t1; > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index 80b98758fb7..461c3feed8c 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -903,6 +903,12 @@ class Item_char_typecast :public Item_str_func > String *val_str(String *a); > void fix_length_and_dec(); > void print(String *str, enum_query_type query_type); > + table_map used_tables() const > + { > + if (!cast_length) > + return (table_map) 0L; > + return Item_str_func::used_tables(); > + } > }; > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 17540ad: MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>.
by Sergei Golubchik 04 Sep '18

04 Sep '18

Hi, Alexey! On Sep 03, Alexey Botchkov wrote: > revision-id: 17540adc625ff4ac0f00aa10796cb6e763838d18 (mariadb-10.1.35-25-g17540ad) > parent(s): 34e026f35a7fbeeaec412547ee207c880944ae68 > committer: Alexey Botchkov > timestamp: 2018-09-03 16:49:46 +0400 > message: > > MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>. > > LOCK view WRITE shouldn't block FLUSH view. > So we set the view's mdl_request type to it's tables. > > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 6bd6b6a..d7a2a6b 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1534,8 +1534,7 @@ bool mysql_make_view(THD *thd, TABLE_SHARE *share, TABLE_LIST *table, > for (tbl= view_main_select_tables; tbl; tbl= tbl->next_local) > { > tbl->lock_type= table->lock_type; > - tbl->mdl_request.set_type((tbl->lock_type >= TL_WRITE_ALLOW_WRITE) ? > - MDL_SHARED_WRITE : MDL_SHARED_READ); > + tbl->mdl_request.set_type(table->mdl_request.type); > } I agree it's looks like a proper fix. But I'm a bit cautious about possible side effects. A more conservative fix would be only change the lock level for TL_NO_READ_WRITE, like tbl->lock_type == TL_NO_READ_WRITE ? TL_NO_READ_WRITE : tbl->lock_type >= TL_WRITE_ALLOW_WRITE ? MDL_SHARED_WRITE : MDL_SHARED_READ (or with an if(), for readability). But I don't see whether it'll make any difference as compared to your fix. Please, run all tests locally, and I mean *all*, big, ps, embedded. And see if you can think of any cases where this change would cause a change in behavior. After that - ok to push! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] 2 pull requests
by Ming Lin 04 Sep '18

04 Sep '18

Hi Sergey, Could you take a look at these 2 PRs? MDEV-16768: https://github.com/MariaDB/server/pull/837 You approved this one, but not merged yet. MDEV-15816: https://github.com/MariaDB/server/pull/849 This one was already merged by facebook. Thanks, Ming

1 1

[Maria-developers] Binlog handling of faulty DDL query
by Sachin Setiya 03 Sep '18

03 Sep '18

Hi Everyone! Suppose this case CREATE USER user1@localhost IDENTIFIED BY 'BsG9#9.cem#!85', user2@localhost IDENTIFIED BY 'x'; user2 has too short passowrd which will give error (if we use security plugin) IN the case of *mariadb* we will create user1 and wont create user2 In the case of *mysql (8.0)* we will return error and we will not create any user Our approach create a lots of problem for binary log and replication(MDEV-14716, MDEV-16728) Because in Query_log_event::write() we write the whole query not the parts of it. So we have 2 options to solve this either write modified query into binlog or do what mysql does. So what you think , which approach we should use ? Or there is another better way ? -- Regards sachin

5 6

Re: [Maria-developers] 63ad6a9: MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>.
by Sergei Golubchik 02 Sep '18

02 Sep '18

Hi, Alexey! On Sep 02, Alexey Botchkov wrote: > revision-id: 63ad6a9e1a33ddd5547767b2894e09ae66196f69 (mariadb-10.1.35-23-g63ad6a9) > parent(s): 288212f489c8cd88c4cc98f8aecc3366c85a90be > committer: Alexey Botchkov > timestamp: 2018-09-02 09:24:33 +0400 > message: > > MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>. > > Check if the argument of the FLUSH TABLE is a VIEW and handle it > accordingly. > > diff --git a/mysql-test/r/flush.result b/mysql-test/r/flush.result > index b643510..2d7b81b 100644 > --- a/mysql-test/r/flush.result > +++ b/mysql-test/r/flush.result > @@ -496,3 +496,27 @@ flush relay logs,relay logs; > ERROR HY000: Incorrect usage of FLUSH and RELAY LOGS > flush slave,slave; > ERROR HY000: Incorrect usage of FLUSH and SLAVE > +# > +# MDEV-15890 Strange error message if you try to > +# FLUSH TABLES <view> after LOCK TABLES <view>. > +# > +CREATE TABLE t1 (qty INT, price INT); > +CREATE VIEW v1 AS SELECT qty, price, qty*price AS value FROM t1; > +LOCK TABLES v1 READ; > +FLUSH TABLES v1; > +ERROR HY000: Table 't1' was locked with a READ lock and can't be updated > +UNLOCK TABLES; > +LOCK TABLES v1 WRITE; > +FLUSH TABLES v1; > +ERROR HY000: Table 't1' was locked with a READ lock and can't be updated Why is that? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MariaDB v10 on AIX
by REIX, Tony 31 Aug '18

31 Aug '18

Hi, We are porting MariaDB v10(.3.7/9) on AIX now. First, I'd like to know if there are people porting (or who has ported) MariaDB on AIX in this mailing list. Second, we have found some issue dealing with: AIX, -O0, and inline, which leads to a symbol missing at link time: my_ulonglong2double . It deals with the code below. Since we are beginners with MariaDB, I'd like to have some comments and/or recommendations about how it should be fixed. By using "static __inline" instead of "inline" below, and adding it too to the ma_global.h definition of my_ulonglong2double() , we are able to fix it. However, probably that using: #define my_ulonglong2double(nr) ((double) nr) would be enough. Or adding my_port.c to the MYSYS_SOURCES variable in mysys/CMakeLists.txt would help: IF (AIX) SET (MYSYS_SOURCES ${MYSYS_SOURCES} my_port.c) ENDIF() however, that does not work, and I do not see my_port.c being added to the list of files to be added to mysys. Any idea why "IF (AIX)" does not work? (I'm also a CMake beginner). Thanks/Regards, Tony Reix mysys/my_port.c: #include "mysys_priv.h" #ifdef _AIX double my_ulonglong2double(unsigned long long nr) { return (double) nr; } mysys/mysys_priv.h: #include <my_global.h> mysys/CMakeLists.txt : SET(MYSYS_SOURCES ........ No my_port.c !! include/my_global.h: #ifdef _AIX /* By soren(a)t.dk */ #define ulonglong2double(A) my_ulonglong2double(A) C_MODE_START inline double my_ulonglong2double(unsigned long long A) { return (double)A; } C_MODE_END libmariadb/include/ma_global.h: #ifdef _AIX /* By soren(a)t.dk */ #define ulonglong2double(A) my_ulonglong2double(A) #ifdef __cplusplus extern "C" { #endif double my_ulonglong2double(unsigned long long A); déclaration externe #ifdef __cplusplus Cordialement, Tony Reix tony.reix(a)atos.net ATOS / Bull SAS ATOS Expert IBM Coop Architect & Technical Leader Office : +33 (0) 4 76 29 72 67 1 rue de Provence - 38432 Échirolles - France www.atos.net<https://mail.ad.bull.net/owa/redir.aspx?C=PvphmPvCZkGrAgHVnWGsdMcDKgzl_dEIs…>

2 1

Re: [Maria-developers] 44692f17a62: MDEV-15436: If log_bin and log_bin_index is different SST with rsync fails.
by Sergei Golubchik 31 Aug '18

31 Aug '18

Hi, Jan! On Aug 31, serg(a)mariadb.org wrote: > revision-id: 44692f17a621d7f0261d84b97a7334352463038c (mariadb-10.2.16-6-g44692f17a62) > parent(s): 7d0d934ca642e485b2c008727dc20c83e26cce10 > author: Jan Lindström > committer: Jan Lindström > timestamp: 2018-06-26 12:56:19 +0300 > message: > > MDEV-15436: If log_bin and log_bin_index is different SST with rsync fails. > > Problem was that in SST log_bin_index name and directory was not > handled and passed to rsync SST script. > > mysql-test/suite/galera/r/galera_sst_rsync2.result | 396 +++++++++++++++++++++ > mysql-test/suite/galera/t/galera_sst_rsync2.cnf | 15 + > mysql-test/suite/galera/t/galera_sst_rsync2.test | 12 + I didn't review the test, but I presume you've verified that it fails without the bug fix. > diff --git a/scripts/wsrep_sst_rsync.sh b/scripts/wsrep_sst_rsync.sh > index 05c99880d75..6886d2ce1e1 100644 > --- a/scripts/wsrep_sst_rsync.sh > +++ b/scripts/wsrep_sst_rsync.sh > @@ -132,6 +132,12 @@ then > BINLOG_FILENAME=$(basename $WSREP_SST_OPT_BINLOG) > fi > > +if ! [ -z $WSREP_SST_OPT_BINLOG_INDEX ] > +then > + BINLOG_INDEX_DIRNAME=$(dirname $WSREP_SST_OPT_BINLOG_INDEX) > + BINLOG_INDEX_FILENAME=$(basename $WSREP_SST_OPT_BINLOG_INDEX) > +fi > + > WSREP_LOG_DIR=${WSREP_LOG_DIR:-""} > # if WSREP_LOG_DIR env. variable is not set, try to get it from my.cnf > if [ -z "$WSREP_LOG_DIR" ]; then > @@ -204,12 +210,20 @@ then > OLD_PWD="$(pwd)" > cd $BINLOG_DIRNAME > > - binlog_files_full=$(tail -n $BINLOG_N_FILES ${BINLOG_FILENAME}.index) > + if ! [ -z $WSREP_SST_OPT_BINLOG_INDEX ] first, I think it'd be simpler to do, like if [ -z $WSREP_SST_OPT_BINLOG_INDEX ] then WSREP_SST_OPT_BINLOG_INDEX=$WSREP_SST_OPT_BINLOG.index fi it's enough to do it once before setting BINLOG_INDEX_DIRNAME and BINLOG_INDEX_FILENAME and you woudn't need if's here and below. second, you apparently are trying to take into account that binlog index may be a different directory. Please use this configuration in your test case, put binlog index in a different directory. otherwise ok. please, push after adjusting the test case to put binlog index in a different directory. > + binlog_files_full=$(tail -n $BINLOG_N_FILES ${BINLOG_FILENAME}.index) > + then > + cd $BINLOG_INDEX_DIRNAME > + binlog_files_full=$(tail -n $BINLOG_N_FILES ${BINLOG_INDEX_FILENAME}.index) > + fi > + > + cd $BINLOG_DIRNAME > binlog_files="" > for ii in $binlog_files_full > do > binlog_files="$binlog_files $(basename $ii)" > done > + > if ! [ -z "$binlog_files" ] > then > wsrep_log_info "Preparing binlog files for transfer:" > @@ -391,7 +405,11 @@ EOF > tar -xvf $BINLOG_TAR_FILE >&2 > for ii in $(ls -1 ${BINLOG_FILENAME}.*) > do > - echo ${BINLOG_DIRNAME}/${ii} >> ${BINLOG_FILENAME}.index > + if ! [ -z $WSREP_SST_OPT_BINLOG_INDEX ] > + echo ${BINLOG_DIRNAME}/${ii} >> ${BINLOG_FILENAME}.index > + then > + echo ${BINLOG_DIRNAME}/${ii} >> ${BINLOG_INDEX_DIRNAME}/${BINLOG_INDEX_FILENAME}.index > + fi > done > fi > cd "$OLD_PWD" Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 6be8bd6: MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>.
by Sergei Golubchik 31 Aug '18

31 Aug '18

Hi, Alexey! On Jun 24, Alexey Botchkov wrote: > revision-id: 6be8bd6ee05c6f18b6a6a0375f9bc2b68ef6c43e (mariadb-10.3.7-59-g6be8bd6) > parent(s): 1abd877e2df9e83bc1c2f5195796f427a35bd3f1 > committer: Alexey Botchkov > timestamp: 2018-06-24 23:12:20 +0400 > message: > > MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>. > > Check if the argument of the FLUSH TABLE is a VIEW and handle it > accordingly. Looks great! But please add more tests. A test with a write lock, and tests with locking the view and flushing the table or vice versa. After that ok to push! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] 10.2 and merges (e.g. TokuDB)
by Phil Sweeney 28 Aug '18

28 Aug '18

Hi there, It seems 10.2.6 is still shipping with TokuDB from PS 5.6.35-80 (Feb 8th 2017). I can't see a jira issue for a 10.2 merge since 10.2.4. Perhaps intentional to help stabilise 10.2 for release? 10.0.30 also shipped with the same Feb version (in March). 10.1.23 however was released in May and still has this version from Feb, which seems unfortunate, but I know it's because it has to flow through 10.0 first. I see 10.0.31 is due to get TokuDB from PS 5.6.36-82 And that should flow into 10.1.24 soon too. However, 10.2.7 isn't due until July, so by then TokuDB version will be 5 months old in 10.2 I'm waiting on a fairly important row count fix that should appear in the next Percona Server release. When it does, if the current merging strategy continues, do I have to first wait for 10.0.32 to get it in late July, then 10.1.26 in August, then 10.2 after that? Or with 10.0, 10.1 and 10.2 all being supported and needing merges, is there a new strategy on the cards to shorten this lag? Thanks! Phil

3 7

[Maria-developers] MDEV-371 Long unique Index
by Sachin Setiya 28 Aug '18

28 Aug '18

Hi Serg! Please review the patch for Unique Index For Long (MDEV-371) (It is in bb-10.4-blob-index branch) Architecture:- We will create a virtual column DB_ROW_HASH_X (X is any number which makes this field name unique) for each long unique column which will store hash. At the time of ha_write_row we will check for hash collusions if we find same hash then we will retrieve the blob and then compare byte by byte. key_info for long unique have one extra key_part which will point to hash field. For example unique(a,b) key will have 3 key_parts :- a ,b DB_ROW_HASH_X But on server layer we can just see a,b parts on storage layer we can just see DB_ROW_HASH_X key_part. I have created 2 function setup_table_hash and resetup_table which helps in this transition. If user specify length of blob then we use LEFT function to trim the value. -- Regards Sachin Setiya Software Engineer at MariaDB

1 0

Re: [Maria-developers] [Commits] f522fb9: MDEV-16934 Query with very large IN clause lists runs slowly
by Sergey Petrunia 17 Aug '18

17 Aug '18

Hello Igor, I am debugging with the patch and I see that PARAM::range_count is not reset after the eq_ranges_exceeds_limit() call. This causes different kinds of undesired effects. For example, here create table ten(a int); insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table t1 (pk int, kp1 int, kp2 int, kp3 int, col1 int, key(kp1,kp2,kp3), primary key(pk)); insert into t1 select A.a + 10*B.a+100* C.a, A.a, B.a , C.a, 1234 from ten A, ten B, ten C; alter table t1 add key3 int, add key2 int; update t1 set key3=kp1; update t1 set key2=kp1; alter table t1 add key(key2), add key(key3); set eq_range_index_dive_limit=10; explain format=json select 1 from t1 force index (key2, key3) where key3=1 or key2=1; index_merge will use Sort-Union instead of Union. I assume we are targeting only IN (...) cases? Both our and MySQL's fixes will still do many records_in_range calls for queries like keypart1 IN (1,2,3,...) AND keypart2 IN (1,2,3,...) AND keypart3 >= 10 (because here the ranges will not be equality ranges). I assume this is outside of the scope of this fix? If so, then the patch is ok to push after PARAM::range_count issue is resolved. On Wed, Aug 15, 2018 at 07:48:30PM -0700, IgorBabaev wrote: > revision-id: f522fb9f63f88195bccbbc0326a4a318cc2d3db0 (mariadb-10.2.16-50-gf522fb9) > parent(s): 3c141e319ab29afdfe843553da385fe5d981906e > author: Igor Babaev > committer: Igor Babaev > timestamp: 2018-08-15 19:48:30 -0700 > message: > > MDEV-16934 Query with very large IN clause lists runs slowly > > This patch introduces support for the system variable eq_range_index_dive_limit > that existed in MySQL starting from 5.6. The variable sets a limit for > index dives into equality ranges. Index dives are performed by optimizer > to estimate the number of rows in range scans. Index dives usually provide > good estimate but they are pretty expensive. To estimate the number of rows > in equality ranges statistical data on indexes can be employed. Its usage gives > not so good estimates but it's cheap. So if the number of equality dives > required by an index scan exceeds the set limit no dives for equality > ranges are performed by the optimizer for this index. > > As the new system variable is introduced in a stable version the default > value for it is set to a special value meaning there is no limit for the number > of index dives performed by the optimizer. > > The patch partially uses the MySQL code for WL 5957 > 'Statistics-based Range optimization for many ranges'. > > --- > mysql-test/r/mysqld--help.result | 6 +++ > mysql-test/r/range.result | 41 +++++++++++++++++ > mysql-test/r/range_mrr_icp.result | 41 +++++++++++++++++ > .../sys_vars/r/sysvars_server_embedded.result | 14 ++++++ > .../sys_vars/r/sysvars_server_notembedded.result | 14 ++++++ > mysql-test/t/range.test | 33 ++++++++++++++ > sql/multi_range_read.cc | 11 +++++ > sql/opt_range.cc | 27 +++++++++++ > sql/opt_range.h | 4 +- > sql/opt_range_mrr.cc | 53 +++++++++++++++------- > sql/sql_class.h | 1 + > sql/sql_statistics.h | 2 +- > sql/sys_vars.cc | 10 ++++ > 13 files changed, 238 insertions(+), 19 deletions(-) > > diff --git a/mysql-test/r/mysqld--help.result b/mysql-test/r/mysqld--help.result > index 3b7d5e2..51ece33 100644 > --- a/mysql-test/r/mysqld--help.result > +++ b/mysql-test/r/mysqld--help.result > @@ -197,6 +197,11 @@ The following specify which files/extra groups are read (specified before remain > cache, etc) > --enforce-storage-engine=name > Force the use of a storage engine for new tables > + --eq-range-index-dive-limit=# > + The optimizer will use existing index statistics instead > + of doing index dives for equality ranges if the number of > + equality ranges for the index is larger than or equal to > + this number. If set to 0, index dives are always used. > --event-scheduler[=name] > Enable the event scheduler. Possible values are ON, OFF, > and DISABLED (keep the event scheduler completely > @@ -1259,6 +1264,7 @@ encrypt-binlog FALSE > encrypt-tmp-disk-tables FALSE > encrypt-tmp-files FALSE > enforce-storage-engine (No default value) > +eq-range-index-dive-limit 0 > event-scheduler OFF > expensive-subquery-limit 100 > expire-logs-days 0 > diff --git a/mysql-test/r/range.result b/mysql-test/r/range.result > index 3a71d08..0c6be5e 100644 > --- a/mysql-test/r/range.result > +++ b/mysql-test/r/range.result > @@ -3002,5 +3002,46 @@ deallocate prepare stmt; > set optimizer_switch=@save_optimizer_switch; > drop table t1,t2,t3; > # > +# MDEV-16934: using system variable eq_range_index_dive_limit > +# to reduce the number of index dives > +# > +create table t1 (a int, b varchar(31), index idx(a)); > +insert into t1 values > +(7,'xxxx'), (1,'yy'), (3,'aaa'), (1,'bbb'), (2,'zz'), > +(4,'vvvvv'), (7,'ddd'), (9,'zzzzz'), (1,'cc'), (5,'ffff'); > +insert into t1 select a+10, concat(b,'zz') from t1; > +insert into t1 select a+15, concat(b,'yy') from t1; > +insert into t1 select a+100, concat(b,'xx') from t1; > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status OK > +select cast(count(a)/count(distinct a) as unsigned) as rec_per_key from t1; > +rec_per_key > +2 > +set eq_range_index_dive_limit=0; > +explain select * from t1 where a in (8, 15, 31, 1, 9); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range idx idx 5 NULL 7 Using index condition > +select * from t1 where a in (8, 15, 31, 1, 9); > +a b > +1 yy > +1 bbb > +1 cc > +9 zzzzz > +15 ffffzz > +set eq_range_index_dive_limit=2; > +explain select * from t1 where a in (8, 15, 31, 1, 9); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range idx idx 5 NULL 10 Using index condition > +select * from t1 where a in (8, 15, 31, 1, 9); > +a b > +1 yy > +1 bbb > +1 cc > +9 zzzzz > +15 ffffzz > +set eq_range_index_dive_limit=default; > +drop table t1; > +# > # End of 10.2 tests > # > diff --git a/mysql-test/r/range_mrr_icp.result b/mysql-test/r/range_mrr_icp.result > index 799a299..93f414f 100644 > --- a/mysql-test/r/range_mrr_icp.result > +++ b/mysql-test/r/range_mrr_icp.result > @@ -3014,6 +3014,47 @@ deallocate prepare stmt; > set optimizer_switch=@save_optimizer_switch; > drop table t1,t2,t3; > # > +# MDEV-16934: using system variable eq_range_index_dive_limit > +# to reduce the number of index dives > +# > +create table t1 (a int, b varchar(31), index idx(a)); > +insert into t1 values > +(7,'xxxx'), (1,'yy'), (3,'aaa'), (1,'bbb'), (2,'zz'), > +(4,'vvvvv'), (7,'ddd'), (9,'zzzzz'), (1,'cc'), (5,'ffff'); > +insert into t1 select a+10, concat(b,'zz') from t1; > +insert into t1 select a+15, concat(b,'yy') from t1; > +insert into t1 select a+100, concat(b,'xx') from t1; > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status OK > +select cast(count(a)/count(distinct a) as unsigned) as rec_per_key from t1; > +rec_per_key > +2 > +set eq_range_index_dive_limit=0; > +explain select * from t1 where a in (8, 15, 31, 1, 9); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range idx idx 5 NULL 7 Using index condition; Rowid-ordered scan > +select * from t1 where a in (8, 15, 31, 1, 9); > +a b > +1 yy > +1 bbb > +9 zzzzz > +1 cc > +15 ffffzz > +set eq_range_index_dive_limit=2; > +explain select * from t1 where a in (8, 15, 31, 1, 9); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range idx idx 5 NULL 10 Using index condition; Rowid-ordered scan > +select * from t1 where a in (8, 15, 31, 1, 9); > +a b > +1 yy > +1 bbb > +9 zzzzz > +1 cc > +15 ffffzz > +set eq_range_index_dive_limit=default; > +drop table t1; > +# > # End of 10.2 tests > # > set optimizer_switch=@mrr_icp_extra_tmp; > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > index f7a5cf5..9d7fa10 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > @@ -765,6 +765,20 @@ NUMERIC_BLOCK_SIZE NULL > ENUM_VALUE_LIST NULL > READ_ONLY NO > COMMAND_LINE_ARGUMENT NULL > +VARIABLE_NAME EQ_RANGE_INDEX_DIVE_LIMIT > +SESSION_VALUE 0 > +GLOBAL_VALUE 0 > +GLOBAL_VALUE_ORIGIN COMPILE-TIME > +DEFAULT_VALUE 0 > +VARIABLE_SCOPE SESSION > +VARIABLE_TYPE INT UNSIGNED > +VARIABLE_COMMENT The optimizer will use existing index statistics instead of doing index dives for equality ranges if the number of equality ranges for the index is larger than or equal to this number. If set to 0, index dives are always used. > +NUMERIC_MIN_VALUE 0 > +NUMERIC_MAX_VALUE 4294967295 > +NUMERIC_BLOCK_SIZE 1 > +ENUM_VALUE_LIST NULL > +READ_ONLY NO > +COMMAND_LINE_ARGUMENT REQUIRED > VARIABLE_NAME ERROR_COUNT > SESSION_VALUE 0 > GLOBAL_VALUE NULL > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > index 034195e..cada139 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > @@ -779,6 +779,20 @@ NUMERIC_BLOCK_SIZE NULL > ENUM_VALUE_LIST NULL > READ_ONLY NO > COMMAND_LINE_ARGUMENT NULL > +VARIABLE_NAME EQ_RANGE_INDEX_DIVE_LIMIT > +SESSION_VALUE 0 > +GLOBAL_VALUE 0 > +GLOBAL_VALUE_ORIGIN COMPILE-TIME > +DEFAULT_VALUE 0 > +VARIABLE_SCOPE SESSION > +VARIABLE_TYPE INT UNSIGNED > +VARIABLE_COMMENT The optimizer will use existing index statistics instead of doing index dives for equality ranges if the number of equality ranges for the index is larger than or equal to this number. If set to 0, index dives are always used. > +NUMERIC_MIN_VALUE 0 > +NUMERIC_MAX_VALUE 4294967295 > +NUMERIC_BLOCK_SIZE 1 > +ENUM_VALUE_LIST NULL > +READ_ONLY NO > +COMMAND_LINE_ARGUMENT REQUIRED > VARIABLE_NAME ERROR_COUNT > SESSION_VALUE 0 > GLOBAL_VALUE NULL > diff --git a/mysql-test/t/range.test b/mysql-test/t/range.test > index ab95180..ed68cfb 100644 > --- a/mysql-test/t/range.test > +++ b/mysql-test/t/range.test > @@ -2046,6 +2046,39 @@ set optimizer_switch=@save_optimizer_switch; > drop table t1,t2,t3; > > --echo # > +--echo # MDEV-16934: using system variable eq_range_index_dive_limit > +--echo # to reduce the number of index dives > +--echo # > + > +create table t1 (a int, b varchar(31), index idx(a)); > + > +insert into t1 values > + (7,'xxxx'), (1,'yy'), (3,'aaa'), (1,'bbb'), (2,'zz'), > + (4,'vvvvv'), (7,'ddd'), (9,'zzzzz'), (1,'cc'), (5,'ffff'); > +insert into t1 select a+10, concat(b,'zz') from t1; > +insert into t1 select a+15, concat(b,'yy') from t1; > +insert into t1 select a+100, concat(b,'xx') from t1; > + > +analyze table t1; > + > +select cast(count(a)/count(distinct a) as unsigned) as rec_per_key from t1; > + > +let $q= > +select * from t1 where a in (8, 15, 31, 1, 9); > + > +set eq_range_index_dive_limit=0; > +eval explain $q; > +eval $q; > + > +set eq_range_index_dive_limit=2; > +eval explain $q; > +eval $q; > + > +set eq_range_index_dive_limit=default; > + > +drop table t1; > + > +--echo # > --echo # End of 10.2 tests > --echo # > > diff --git a/sql/multi_range_read.cc b/sql/multi_range_read.cc > index 50918d8..94a96c2 100644 > --- a/sql/multi_range_read.cc > +++ b/sql/multi_range_read.cc > @@ -17,6 +17,7 @@ > #include <my_bit.h> > #include "sql_select.h" > #include "key.h" > +#include "sql_statistics.h" > > /**************************************************************************** > * Default MRR implementation (MRR to non-MRR converter) > @@ -67,6 +68,9 @@ handler::multi_range_read_info_const(uint keyno, RANGE_SEQ_IF *seq, > /* Default MRR implementation doesn't need buffer */ > *bufsz= 0; > > + bool use_statistics_for_eq_range= eq_ranges_exceeds_limit(seq, > + seq_init_param); > + > seq_it= seq->init(seq_init_param, n_ranges, *flags); > while (!seq->next(seq_it, &range)) > { > @@ -87,8 +91,15 @@ handler::multi_range_read_info_const(uint keyno, RANGE_SEQ_IF *seq, > min_endp= range.start_key.length? &range.start_key : NULL; > max_endp= range.end_key.length? &range.end_key : NULL; > } > + int keyparts_used= my_count_bits(range.start_key.keypart_map); > if ((range.range_flag & UNIQUE_RANGE) && !(range.range_flag & NULL_RANGE)) > rows= 1; /* there can be at most one row */ > + else if (use_statistics_for_eq_range && > + !(range.range_flag & NULL_RANGE) && > + (range.range_flag & EQ_RANGE) && > + table->key_info[keyno].actual_rec_per_key(keyparts_used - 1) > 0.5) > + rows= > + (ha_rows) table->key_info[keyno].actual_rec_per_key(keyparts_used - 1); > else > { > if (HA_POS_ERROR == (rows= this->records_in_range(keyno, min_endp, > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index b011c01..c9c37a2 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -14616,6 +14616,33 @@ void QUICK_GROUP_MIN_MAX_SELECT::add_keys_and_lengths(String *key_names, > } > > > +/* Check whether the number for equality ranges exceeds the set threshold */ > + > +bool eq_ranges_exceeds_limit(RANGE_SEQ_IF *seq, void *seq_init_param) > +{ > + KEY_MULTI_RANGE range; > + range_seq_t seq_it; > + uint count = 0; > + PARAM *param= ((SEL_ARG_RANGE_SEQ*) seq_init_param)->param; > + uint limit= param->thd->variables.eq_range_index_dive_limit; > + > + if (limit == 0) > + { > + /* 'Statistics instead of index dives' feature is turned off */ > + return false; > + } > + seq_it= seq->init(seq_init_param, 0, 0); > + while (!seq->next(seq_it, &range)) > + { > + if ((range.range_flag & EQ_RANGE) && !(range.range_flag & NULL_RANGE)) > + { > + if (++count >= limit) > + return true; > + } > + } > + return false; > +} > + > #ifndef DBUG_OFF > > static void print_sel_tree(PARAM *param, SEL_TREE *tree, key_map *tree_map, > diff --git a/sql/opt_range.h b/sql/opt_range.h > index c1f7079..6698c98 100644 > --- a/sql/opt_range.h > +++ b/sql/opt_range.h > @@ -242,7 +242,7 @@ class SEL_ARG :public Sql_alloc > Number of children of this element in the RB-tree, plus 1 for this > element itself. > */ > - uint16 elements; > + uint32 elements; > /* > Valid only for elements which are RB-tree roots: Number of times this > RB-tree is referred to (it is referred by SEL_ARG::next_key_part or by > @@ -1664,6 +1664,8 @@ SQL_SELECT *make_select(TABLE *head, table_map const_tables, > > bool calculate_cond_selectivity_for_table(THD *thd, TABLE *table, Item **cond); > > +bool eq_ranges_exceeds_limit(RANGE_SEQ_IF *seq, void *seq_init_param); > + > #ifdef WITH_PARTITION_STORAGE_ENGINE > bool prune_partitions(THD *thd, TABLE *table, Item *pprune_cond); > #endif > diff --git a/sql/opt_range_mrr.cc b/sql/opt_range_mrr.cc > index ace6208..7f80a3e 100644 > --- a/sql/opt_range_mrr.cc > +++ b/sql/opt_range_mrr.cc > @@ -272,25 +272,44 @@ bool sel_arg_range_seq_next(range_seq_t rseq, KEY_MULTI_RANGE *range) > key_info= NULL; > else > key_info= &seq->param->table->key_info[seq->real_keyno]; > - > + > /* > - Conditions below: > - (1) - range analysis is used for estimating condition selectivity > - (2) - This is a unique key, and we have conditions for all its > - user-defined key parts. > - (3) - The table uses extended keys, this key covers all components, > - and we have conditions for all key parts. > + This is an equality range (keypart_0=X and ... and keypart_n=Z) if > + (1) - There are no flags indicating open range (e.g., > + "keypart_x > y") or GIS. > + (2) - The lower bound and the upper bound of the range has the > + same value (min_key == max_key). > */ > - if (!(cur->min_key_flag & ~NULL_RANGE) && !cur->max_key_flag && > - (!key_info || // (1) > - ((uint)key_tree->part+1 == key_info->user_defined_key_parts && // (2) > - key_info->flags & HA_NOSAME) || // (2) > - ((key_info->flags & HA_EXT_NOSAME) && // (3) > - (uint)key_tree->part+1 == key_info->ext_key_parts) // (3) > - ) && > - range->start_key.length == range->end_key.length && > - !memcmp(seq->param->min_key,seq->param->max_key,range->start_key.length)) > - range->range_flag= UNIQUE_RANGE | (cur->min_key_flag & NULL_RANGE); > + const uint is_open_range = > + (NO_MIN_RANGE | NO_MAX_RANGE | NEAR_MIN | NEAR_MAX | GEOM_FLAG); > + const bool is_eq_range_pred = > + !(cur->min_key_flag & is_open_range) && // (1) > + !(cur->max_key_flag & is_open_range) && // (1) > + range->start_key.length == range->end_key.length && // (2) > + !memcmp(seq->param->min_key, seq->param->max_key, // (2) > + range->start_key.length); > + > + if (is_eq_range_pred) > + { > + range->range_flag = EQ_RANGE; > + > + /* > + Conditions below: > + (1) - Range analysis is used for estimating condition selectivity > + (2) - This is a unique key, and we have conditions for all its > + user-defined key parts. > + (3) - The table uses extended keys, this key covers all components, > + and we have conditions for all key parts. > + */ > + if ( > + !key_info || // (1) > + ((uint)key_tree->part+1 == key_info->user_defined_key_parts && // (2) > + key_info->flags & HA_NOSAME) || // (2) > + ((key_info->flags & HA_EXT_NOSAME) && // (3) > + (uint)key_tree->part+1 == key_info->ext_key_parts) // (3) > + ) > + range->range_flag |= UNIQUE_RANGE | (cur->min_key_flag & NULL_RANGE); > + } > > if (seq->param->is_ror_scan) > { > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 6663d9e..d2f00b2d 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -569,6 +569,7 @@ typedef struct system_variables > ha_rows max_join_size; > ha_rows expensive_subquery_limit; > ulong auto_increment_increment, auto_increment_offset; > + uint eq_range_index_dive_limit; > ulong lock_wait_timeout; > ulong join_cache_level; > ulong max_allowed_packet; > diff --git a/sql/sql_statistics.h b/sql/sql_statistics.h > index f465838..4ff85cd 100644 > --- a/sql/sql_statistics.h > +++ b/sql/sql_statistics.h > @@ -21,7 +21,7 @@ enum enum_use_stat_tables_mode > { > NEVER, > COMPLEMENTARY, > - PEFERABLY, > + PREFERABLY, > } Use_stat_tables_mode; > > typedef > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index e6aec47..650b17a 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -2578,6 +2578,16 @@ static Sys_var_ulong Sys_div_precincrement( > SESSION_VAR(div_precincrement), CMD_LINE(REQUIRED_ARG), > VALID_RANGE(0, DECIMAL_MAX_SCALE), DEFAULT(4), BLOCK_SIZE(1)); > > +static Sys_var_uint Sys_eq_range_index_dive_limit( > + "eq_range_index_dive_limit", > + "The optimizer will use existing index statistics instead of " > + "doing index dives for equality ranges if the number of equality " > + "ranges for the index is larger than or equal to this number. " > + "If set to 0, index dives are always used.", > + SESSION_VAR(eq_range_index_dive_limit), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, UINT_MAX32), DEFAULT(0), > + BLOCK_SIZE(1)); > + > static Sys_var_ulong Sys_range_alloc_block_size( > "range_alloc_block_size", > "Allocation block size for storing ranges during optimization", > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] adaafcf5184: MDEV-16930: Crash when VALUES in derived table contains expressions
by Sergei Golubchik 16 Aug '18

16 Aug '18

Hi, Oleksandr! On Aug 16, Oleksandr Byelkin wrote: > revision-id: adaafcf5184375369c0c4b3ccb1e82e846024ef8 (mariadb-10.3.7-137-gadaafcf5184) > parent(s): 8716bb3b72b7f5fed69b6dde413c2138f6d175b2 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-08-16 13:55:36 +0200 > message: > > MDEV-16930: Crash when VALUES in derived table contains expressions > > Give names to the value constructor columns as in SELECT-list. > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index f915895a260..20cea4959eb 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -13306,15 +13306,20 @@ opt_values: > ; > > values: > - values ',' expr_or_default > + values ',' remember_name expr_or_default remember_end > { > - if (unlikely(Lex->insert_list->push_back($3, thd->mem_root))) > + if (unlikely(Lex->insert_list->push_back($4, thd->mem_root))) > MYSQL_YYABORT; > + // give some name in case of using in table value constuctor (TVC) > + $4->set_name(thd, $3, (uint) ($5 - $3), thd->charset()); > + > } > - | expr_or_default > + | remember_name expr_or_default remember_end > { > - if (unlikely(Lex->insert_list->push_back($1, thd->mem_root))) > + if (unlikely(Lex->insert_list->push_back($2, thd->mem_root))) > MYSQL_YYABORT; > + // give some name in case of using in table value constuctor (TVC) > + $2->set_name(thd, $1, (uint) ($3 - $1), thd->charset()); > } > ; Could you please create a special rule for that? Like named_value: remember_name expr_or_default remember_end { if (unlikely(Lex->insert_list->push_back($2, thd->mem_root))) MYSQL_YYABORT; // give some name in case of using in table value constuctor (TVC) $2->set_name(thd, $1, (uint) ($3 - $1), thd->charset()); } values: values ',' named_value | named_value ; if it'll work, that is Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 7fc57734575: MDEV-6439: Server crashes in Explain_union::print_explain with explain in slow log, tis620 charset
by Oleksandr Byelkin 15 Aug '18

15 Aug '18

Hi! See small comment inline. (fix and OK to push) Am 15.08.2018 um 13:36 schrieb Sergey Petrunia: > MDEV-6439: Server crashes in Explain_union::print_explain with explain in slow log, tis620 charset > > Item_subselect::is_expensive() used to return FALSE (Inexpensive) whenever > it saw that one of SELECTs in the Subquery's UNION is degenerate. It > ignored the fact that other parts of the UNION might not be inexpensive, > including the case where pther parts of the UNION have no query plan yet. > > For a subquery in form col >= ANY (SELECT 'foo' UNION SELECT 'bar') > this would cause the query to be considered inexpensive when there is > no query plan for the second part of the UNION, which in turn would > cause the SELECT 'foo' to compute and free itself while still inside > JOIN::optimize for that SELECT (See MDEV comment for full description). > > diff --git a/mysql-test/r/subselect_extra_no_semijoin.result b/mysql-test/r/subselect_extra_no_semijoin.result > index 79bca38..200682b 100644 > --- a/mysql-test/r/subselect_extra_no_semijoin.result > +++ b/mysql-test/r/subselect_extra_no_semijoin.result > @@ -482,3 +482,22 @@ DROP TABLE t1,t2; > set optimizer_switch= @tmp_subselect_extra_derived; > set optimizer_switch= @subselect_extra_no_sj_tmp; > set @optimizer_switch_for_subselect_extra_test=null; > +# > +# MDEV-6439: Server crashes in Explain_union::print_explain with explain in slow log, tis620 charset > +# > +SET NAMES tis620; > +set @tmp= @@global.slow_query_log; > +SET GLOBAL slow_query_log = 1; > +SET long_query_time = 0.000001; > +SET log_slow_verbosity = 'explain'; > +CREATE TABLE t1 (a VARCHAR(3)) ENGINE=MyISAM; > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo'); > +a > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo' UNION SELECT 'bar' ); > +ERROR HY000: Illegal mix of collations (tis620_thai_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation '<=' > +create table t2 (b int); > +insert into t2 values (1),(2),(3); > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo' FROM t2); > +ERROR HY000: Illegal mix of collations (tis620_thai_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation '<=' > +drop table t1,t2; > +SET GLOBAL slow_query_log=@tmp; > diff --git a/mysql-test/t/subselect_extra_no_semijoin.test b/mysql-test/t/subselect_extra_no_semijoin.test > index 8aba3dd..41593fb 100644 > --- a/mysql-test/t/subselect_extra_no_semijoin.test > +++ b/mysql-test/t/subselect_extra_no_semijoin.test > @@ -6,4 +6,34 @@ set @optimizer_switch_for_subselect_extra_test='semijoin=off,firstmatch=off,loo > > set optimizer_switch= @subselect_extra_no_sj_tmp; > > -set @optimizer_switch_for_subselect_extra_test=null; > \ No newline at end of file > +set @optimizer_switch_for_subselect_extra_test=null; > + > +--echo # > +--echo # MDEV-6439: Server crashes in Explain_union::print_explain with explain in slow log, tis620 charset > +--echo # > + > +## Using a separate client connection is easier than restoring state > +connect(con1,localhost,root,,); > + > +SET NAMES tis620; > +set @tmp= @@global.slow_query_log; > +SET GLOBAL slow_query_log = 1; > +SET long_query_time = 0.000001; > +SET log_slow_verbosity = 'explain'; > + please remove above space at the end of line and push. > +CREATE TABLE t1 (a VARCHAR(3)) ENGINE=MyISAM; > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo'); > +--error ER_CANT_AGGREGATE_2COLLATIONS > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo' UNION SELECT 'bar' ); > + > +create table t2 (b int); > +insert into t2 values (1),(2),(3); > + > +--error ER_CANT_AGGREGATE_2COLLATIONS > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo' FROM t2); > + > +drop table t1,t2; > +SET GLOBAL slow_query_log=@tmp; > +disconnect con1; > +connection default; > + > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index a8dfdff..14b2ccd 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -575,7 +575,7 @@ bool Item_subselect::is_expensive() > */ > if (cur_join->optimized && > (cur_join->zero_result_cause || !cur_join->tables_list)) > - return false; > + continue; > > /* > If a subquery is not optimized we cannot estimate its cost. A subquery is

1 0

[Maria-developers] MDEV-16768 and MDEV-16703 patch review
by Ming Lin 10 Aug '18

10 Aug '18

Hi Sergey, Could you help to review below 2 patches please? MDEV-16768: https://github.com/MariaDB/server/pull/837 MDEV-16703: https://github.com/MariaDB/server/pull/839 Thanks, Ming

1 0

Re: [Maria-developers] 6df4e4a855f: MDEV-16217: Assertion `!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))' failed in Field_num::get_date
by Sergei Golubchik 09 Aug '18

09 Aug '18

Hi, Oleksandr! See questions below. There are few changes that I'm not quite sure about. On Aug 08, Oleksandr Byelkin wrote: > revision-id: 6df4e4a855fe223988c12681f8caec6c49b2f629 (mariadb-10.2.16-66-g6df4e4a855f) > parent(s): 4ddcb4eb46c62cf459936554d43351db740ba14d > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-08-08 12:28:15 +0200 > message: > > MDEV-16217: Assertion `!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))' failed in Field_num::get_date > > - clean up DEFAULT() to work only with default value and correctly print > itself. > - fix of DBUG_ASSERT about fields read/write > - fix of firld marking for write based really on the thd->mark_used_columns flag > > --- > mysql-test/r/func_default.result | 2 +- > mysql-test/r/func_time.result | 28 ++++++++++++++++++++++++++++ > mysql-test/t/func_time.test | 21 +++++++++++++++++++++ > sql/field.cc | 21 +++++++++++++++++++-- > sql/field.h | 1 + > sql/item.cc | 25 ++++++++++++++++++++++++- > sql/item.h | 5 +++++ > sql/sql_base.cc | 2 +- > 8 files changed, 100 insertions(+), 5 deletions(-) > > diff --git a/mysql-test/r/func_default.result b/mysql-test/r/func_default.result > index 535be10da86..1f331af287c 100644 > --- a/mysql-test/r/func_default.result > +++ b/mysql-test/r/func_default.result > @@ -8,7 +8,7 @@ explain extended select default(str), default(strnull), default(intg), default(r > id select_type table type possible_keys key key_len ref rows filtered Extra > 1 SIMPLE t1 system NULL NULL NULL NULL 1 100.00 > Warnings: > -Note 1003 select default('') AS `default(str)`,default('') AS `default(strnull)`,default(0) AS `default(intg)`,default(0) AS `default(rel)` from dual > +Note 1003 select default(`str`) AS `default(str)`,default(`strnull`) AS `default(strnull)`,default(`intg`) AS `default(intg)`,default(`rel`) AS `default(rel)` from dual This looks rather fishy. There can be no columns str and strnull in dual. Please add a test case with a view, to make sure you only get this effect (column names and dual) in explain extended (where it doesn't matter), but not in a view (where it does). > select * from t1 where str <> default(str); > str strnull intg rel > 0 0 > diff --git a/mysql-test/r/func_time.result b/mysql-test/r/func_time.result > index 55c2c93eb56..51f51820efc 100644 > --- a/mysql-test/r/func_time.result > +++ b/mysql-test/r/func_time.result > @@ -3260,3 +3260,31 @@ DROP TABLE t1,t2; > # > # End of 10.1 tests > # > +# > +# MDEV-16217: Assertion `!table || (!table->read_set || > +# bitmap_is_set(table->read_set, field_index))' > +# failed in Field_num::get_date > +# > +CREATE TABLE t1 (pk int default 0, a1 date); > +INSERT INTO t1 VALUES (1,'1900-01-01'),(2,NULL),(3,NULL),(4,NULL); > +CREATE VIEW v1 AS > +SELECT t1.pk AS pk, t1.a1 AS a1 FROM t1; > +SELECT a1 BETWEEN (('2018-08-24')) AND (DEFAULT(pk)) FROM v1; > +a1 BETWEEN (('2018-08-24')) AND (DEFAULT(pk)) > +0 > +NULL > +NULL > +NULL > +SELECT a1 BETWEEN (('2018-08-24')) AND (~ DEFAULT(pk)) FROM v1; > +a1 BETWEEN (('2018-08-24')) AND (~ DEFAULT(pk)) > +0 > +NULL > +NULL > +NULL > +Warnings: > +Warning 1292 Incorrect datetime value: '18446744073709551615' > +DROP VIEW v1; > +DROP TABLE t1; > +# > +# End of 10.2 tests > +# > diff --git a/mysql-test/t/func_time.test b/mysql-test/t/func_time.test > index 3503b6d5bc6..c8859feee93 100644 > --- a/mysql-test/t/func_time.test > +++ b/mysql-test/t/func_time.test > @@ -1857,3 +1857,24 @@ DROP TABLE t1,t2; > --echo # > --echo # End of 10.1 tests > --echo # > + > +--echo # > +--echo # MDEV-16217: Assertion `!table || (!table->read_set || > +--echo # bitmap_is_set(table->read_set, field_index))' > +--echo # failed in Field_num::get_date > +--echo # > +CREATE TABLE t1 (pk int default 0, a1 date); > +INSERT INTO t1 VALUES (1,'1900-01-01'),(2,NULL),(3,NULL),(4,NULL); > + > +CREATE VIEW v1 AS > +SELECT t1.pk AS pk, t1.a1 AS a1 FROM t1; > + > +SELECT a1 BETWEEN (('2018-08-24')) AND (DEFAULT(pk)) FROM v1; > +SELECT a1 BETWEEN (('2018-08-24')) AND (~ DEFAULT(pk)) FROM v1; > + > +DROP VIEW v1; > +DROP TABLE t1; > + > +--echo # > +--echo # End of 10.2 tests > +--echo # > diff --git a/sql/field.cc b/sql/field.cc > index 9ca9663f066..e12adbf47b3 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -70,8 +70,25 @@ const char field_separator=','; > #define BLOB_PACK_LENGTH_TO_MAX_LENGH(arg) \ > ((ulong) ((1LL << MY_MIN(arg, 4) * 8) - 1)) > > -#define ASSERT_COLUMN_MARKED_FOR_READ DBUG_ASSERT(!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))) > -#define ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED DBUG_ASSERT(is_stat_field || !table || (!table->write_set || bitmap_is_set(table->write_set, field_index) || (table->vcol_set && bitmap_is_set(table->vcol_set, field_index)))) > +// Column marked for read or the field set to read out or record[0] or [1] > +#define ASSERT_COLUMN_MARKED_FOR_READ \ > + DBUG_ASSERT(!table || \ > + (!table->read_set || \ > + bitmap_is_set(table->read_set, field_index) || \ > + (!(ptr >= table->record[0] && \ > + ptr < table->record[0] + table->s->reclength) && \ > + !(ptr >= table->record[1] && \ > + ptr < table->record[1] + table->s->reclength)))) why did you add record[1] to the assert? I think, having just record[0] would've been enough > + > +#define ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED \ > + DBUG_ASSERT(is_stat_field || !table || \ > + (!table->write_set || \ > + bitmap_is_set(table->write_set, field_index) || \ > + (!(ptr >= table->record[0] && \ > + ptr < table->record[0] + table->s->reclength) && \ > + !(ptr >= table->record[1] && \ > + ptr < table->record[1] + table->s->reclength))) || \ > + (table->vcol_set && bitmap_is_set(table->vcol_set, field_index))) > > #define FLAGSTR(S,F) ((S) & (F) ? #F " " : "") > > diff --git a/sql/field.h b/sql/field.h > index 22c276478b6..55c3ed4c4bd 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -630,6 +630,7 @@ class Virtual_column_info: public Sql_alloc > bool utf8; /* Already in utf8 */ > Item *expr; > LEX_STRING name; /* Name of constraint */ > + /* see VCOL_* (VCOL_FIELD_REF, ...) */ > uint flags; > > Virtual_column_info() > diff --git a/sql/item.cc b/sql/item.cc > index 58d2b7dbfc0..91fe7ff03aa 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -8917,8 +8917,19 @@ bool Item_default_value::fix_fields(THD *thd, Item **items) > fixed= 1; > return FALSE; > } > + > + /* > + DEFAULT() do not need table field so should not ask handler to bring > + field value (mark column for read) > + */ > + enum_mark_columns save_mark_used_columns= thd->mark_used_columns; > + thd->mark_used_columns= MARK_COLUMNS_NONE; > if (!arg->fixed && arg->fix_fields(thd, &arg)) > + { > + thd->mark_used_columns= save_mark_used_columns; > goto error; > + } > + thd->mark_used_columns= save_mark_used_columns; Hmm, I thought that the field wasn't marked in read_set, and that failed the assert. Now you're saying that it was? > real_arg= arg->real_item(); > @@ -8938,12 +8949,16 @@ bool Item_default_value::fix_fields(THD *thd, Item **items) > goto error; > memcpy((void *)def_field, (void *)field_arg->field, > field_arg->field->size_of()); > - IF_DBUG(def_field->is_stat_field=1,); // a hack to fool ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED ah, good. thanks! > + // If non-constant default value expression > if (def_field->default_value && def_field->default_value->flags) > { > uchar *newptr= (uchar*) thd->alloc(1+def_field->pack_length()); > if (!newptr) > goto error; > + /* > + Even if DEFAULT() do not read tables fields, the default value > + expression can do it. > + */ > fix_session_vcol_expr_for_read(thd, def_field, def_field->default_value); > if (thd->mark_used_columns != MARK_COLUMNS_NONE) > def_field->default_value->expr->walk(&Item::register_field_in_read_map, 1, 0); better call update_used_tables() from here. > @@ -8970,6 +8985,14 @@ void Item_default_value::print(String *str, enum_query_type query_type) > return; > } > str->append(STRING_WITH_LEN("default(")); > + /* > + We take DEFAULT from a field so do not need it value in case of const > + tables but its name so we set QT_NO_DATA_EXPANSION (as we print for > + table definition, also we do not need table and database name) > + */ > + query_type= (enum_query_type) (query_type | QT_NO_DATA_EXPANSION | > + QT_ITEM_IDENT_SKIP_DB_NAMES | > + QT_ITEM_IDENT_SKIP_TABLE_NAMES); in a view definition you might need table and database names. For example, a view that joins two tables from different databases, and they both have a column named `a`. And the query uses DEFAULT(db1.t1.a). > arg->print(str, query_type); > str->append(')'); > } > diff --git a/sql/item.h b/sql/item.h > index 8fad8dadf22..d252a256f76 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -5316,6 +5316,11 @@ class Item_default_value : public Item_field > return false; > } > table_map used_tables() const; > + virtual void update_used_tables() > + { > + if (field && field->default_value) > + field->default_value->expr->walk(&Item::register_field_in_read_map, 1, 0); > + } > Field *get_tmp_table_field() { return 0; } > Item *get_tmp_table_item(THD *thd) { return this; } > Item_field *field_for_view_update() { return 0; } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] Spelling errors in MariaDB sources
by Otto Kekäläinen 31 Jul '18

31 Jul '18

Hello! I recall I've made several big patches on MariaDB that fixes spelling errors. Somehow the same spelling errors seem to creep back in every now and then. I notice them while uploading stuff to Debian and using the Debian QA tool "Lintian". I haven't done git blame and looked deeply in it. I suspect it is related to how we merge old branches to new branches and occasionally loose stuff. Would somebody like to look into this? It could be an relatively easy task for some new contributor? $ grep spelling-error lintian-22e620c534.log I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_chk noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_chk thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_chk theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_dump_log noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_dump_log thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_dump_log theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_ftdump noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_ftdump thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_ftdump theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_pack noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_pack thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_pack theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_read_log noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_read_log thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_read_log theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisam_ftdump noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisam_ftdump thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisam_ftdump theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamchk noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamchk thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamchk theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamlog noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamlog thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamlog theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisampack noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisampack thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisampack theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_federated.so Allows to Allows one to I: mariadb-server-10.1: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_federatedx.so Allows to Allows one to I: mariadb-server-10.1: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_innodb.so allow to allow one to I: libmariadbd18: spelling-error-in-binary usr/lib/x86_64-linux-gnu/libmysqld.so.18 extentions extensions I: libmariadbd18: spelling-error-in-binary usr/lib/x86_64-linux-gnu/libmysqld.so.18 noone no one I: libmariadbd18: spelling-error-in-binary usr/lib/x86_64-linux-gnu/libmysqld.so.18 thats that's I: libmariadbd18: spelling-error-in-binary usr/lib/x86_64-linux-gnu/libmysqld.so.18 theres there's I: libmariadbd18: spelling-error-in-binary usr/lib/x86_64-linux-gnu/libmysqld.so.18 allow to allow one to I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup yuR your I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup occured occurred I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup accessable accessible I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup extentions extensions I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup reseting resetting I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup succeded succeeded I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup settting setting I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup postion position I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup noone no one I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup thats that's I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup theres there's I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup allow to allow one to I: mariadb-client-10.1: spelling-error-in-manpage usr/share/man/man1/innotop.1.gz Additonal Additional I: mariadb-plugin-tokudb: spelling-error-in-binary usr/bin/tokuftdump dont don't I: mariadb-plugin-tokudb: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_tokudb.so dont don't I: mariadb-client-core-10.1: spelling-error-in-binary usr/bin/mysql_embedded extentions extensions I: mariadb-client-core-10.1: spelling-error-in-binary usr/bin/mysql_embedded noone no one I: mariadb-client-core-10.1: spelling-error-in-binary usr/bin/mysql_embedded thats that's I: mariadb-client-core-10.1: spelling-error-in-binary usr/bin/mysql_embedded theres there's I: mariadb-client-core-10.1: spelling-error-in-binary usr/bin/mysql_embedded allow to allow one to I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld extentions extensions I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld reseting resetting I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld succeded succeeded I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld settting setting I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld postion position I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld noone no one I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld thats that's I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld theres there's I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld allow to allow one to I: mariadb-plugin-connect: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_connect.so Mutiple Multiple I: mariadb-plugin-connect: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_connect.so Openning Opening

1 0

[Maria-developers] Missing man pages
by Otto Kekäläinen 31 Jul '18

31 Jul '18

Hello Ian and others! While working on uploading latest MariaDB 10.1 into Debian I noticed it complains about quite a few missing man pages: $ grep manpage lintian-22e620c534.log W: mariadb-server-10.1: binary-without-manpage usr/bin/wsrep_sst_mariabackup W: mariadb-server-10.1: binary-without-manpage usr/bin/wsrep_sst_rsync_wan I: mariadb-client-10.1: spelling-error-in-manpage usr/share/man/man1/innotop.1.gz Additonal Additional W: mariadb-client-10.1: binary-without-manpage usr/bin/mariabackup W: mariadb-client-10.1: binary-without-manpage usr/bin/mbstream W: mariadb-client-10.1: binary-without-manpage usr/bin/mytop W: mariadb-plugin-tokudb: binary-without-manpage usr/bin/tokuft_logprint W: mariadb-client-core-10.1: binary-without-manpage usr/bin/mysql_embedded Could you Ian check if we have these on some branch already and make sure they and up an all the same branches and releases as the binaries themselves? And those who make new binaries, please at least write a stub for your new binary. It makes downstream work much faster when the man page in the source explains directly what the new binaries are and hints how they should be packaged. - Otto

1 0

[Maria-developers] Please review MDEV-16584 SP with a cursor inside a loop wastes THD memory aggressively
by Alexander Barkov 30 Jul '18

30 Jul '18

Hello Sanja, Please review a patch for MDEV-16584. Thanks!

2 1

Re: [Maria-developers] [Commits] 108af298032: MDEV-15454: Nested SELECT IN returns wrong results
by Sergey Petrunia 25 Jul '18

25 Jul '18

On Fri, May 11, 2018 at 03:08:29AM +0530, Varun wrote: > revision-id: 108af298032cb4cea316a08c304c306268a2badf (mariadb-5.5.56-213-g108af298032) > parent(s): 318097bb8f6e12c546b5dcd287416158209dbb39 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-05-11 02:56:33 +0530 > message: > > MDEV-15454: Nested SELECT IN returns wrong results > > In this case we are setting the field Item_func_eq::in_eqaulity_no for the semi-join equalities. > This helps us to remove these equalites as the inner tables are not available during parent select execution > while the outer tables are not available during materialization phase. > We only have it set for the equalites for the fields involved with the IN subquery > and reset it for the equalities which do not belong to the IN subquery. > > For example in case of nested IN subqueries: > > SELECT t1.a FROM t1 WHERE t1.a IN > (SELECT t2.a FROM t2 where t2.b IN > (select t3.b from t3 where t3.c=27 )) > > there are two equalites involving the fields of the IN subquery > > 1) t2.b = t3.b : the field Item_func_eq::in_eqaulity_no is set when we merge the grandchild select into the child select > 2) t1.a = t2.a : the field Item_func_eq::in_eqaulity_no is set when we merge the child select into the parent select > > But when we perform case 2) we should ensure that we reset the equalities in the child's WHERE clause. > > --- > mysql-test/r/subselect_sj_mat.result | 46 +++++++++++++++++++++++++ > mysql-test/t/subselect_sj_mat.test | 49 ++++++++++++++++++++++++++ > sql/opt_subselect.cc | 66 ++++++++++++++++++++++++++++++++++++ > 3 files changed, 161 insertions(+) > ... > diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc > index a7edd64e68b..31213a5cf1d 100644 > --- a/sql/opt_subselect.cc > +++ b/sql/opt_subselect.cc > @@ -1481,6 +1543,7 @@ static int subq_sj_candidate_cmp(Item_in_subselect* el1, Item_in_subselect* el2, > TRUE Out of memory error > */ > > + > static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect *subq_pred) > { > SELECT_LEX *parent_lex= parent_join->select_lex; > @@ -1713,6 +1776,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect *subq_pred) > > if (subq_pred->left_expr->cols() == 1) > { > + reset_equality_number_for_subq_conds(sj_nest->sj_on_expr); > /* add left = select_list_element */ > nested_join->sj_outer_expr_list.push_back(&subq_pred->left_expr); > /* > @@ -1735,6 +1799,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect *subq_pred) > } > else if (subq_pred->left_expr->type() == Item::ROW_ITEM) > { > + reset_equality_number_for_subq_conds(sj_nest->sj_on_expr); > /* > disassemple left expression and add > left1 = select_list_element1 and left2 = select_list_element2 ... > @@ -1759,6 +1824,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect *subq_pred) > } > else > { > + reset_equality_number_for_subq_conds(sj_nest->sj_on_expr); > /* > add row operation > left = (select_list_element1, select_list_element2, ...) So, the code here is if (...) { variant1; } else if (...) { variant2; } else { variant3; } and there is the same call added to all three: > + reset_equality_number_for_subq_conds(sj_nest->sj_on_expr); Can you move it out of the "if" and make it one call instead of three? Otherwise, the patch is ok, ok to push after the above is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] d277c4682e4: MDEV-16751: Server crashes in st_join_table::cleanup or TABLE_LIST::is_with_table_recursive_reference
by Sergey Petrunia 24 Jul '18

24 Jul '18

On Sun, Jul 22, 2018 at 03:56:37PM +0530, Varun wrote: > revision-id: d277c4682e4c6bb0e708296e6ae520c149dd78f4 (mariadb-5.5.60-47-gd277c4682e4) > parent(s): 9cea4ccf12cb6e8746b9b440d9c62408a9ef04af > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-07-22 15:56:25 +0530 > message: > > MDEV-16751: Server crashes in st_join_table::cleanup or TABLE_LIST::is_with_table_recursive_reference > with join_cache_level>2 > > During muliple equality propagation for a query in which we have an IN subquery, the items in the select list of the > subquery may not be part of the multiple equality because there might be another occurence of the same field in the > where clause of the subquery. > So we keyuse_is_valid_for_access_in_chosen_plan function which expects the items in the select list of the subquery to > be same to the ones in the multiple equality (through these multiple equalities we create keyuse array). > The solution would be that we expect the same field not the same Item because when we have SEMI JOIN MATERIALIZATION SCAN, > we use copy back technique to copies back the materialised table fields to the original fields of the base tables. > > --- > mysql-test/r/subselect_mat.result | 35 +++++++++++++++++++++++++++++++++++ > mysql-test/r/subselect_sj_mat.result | 35 +++++++++++++++++++++++++++++++++++ > mysql-test/t/subselect_sj_mat.test | 25 +++++++++++++++++++++++++ > sql/sql_select.cc | 4 +++- > 4 files changed, 98 insertions(+), 1 deletion(-) ... > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 6a64a0e9952..af6f63a55f1 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -7438,7 +7438,9 @@ bool JOIN_TAB::keyuse_is_valid_for_access_in_chosen_plan(JOIN *join, > st_select_lex *sjm_sel= emb_sj_nest->sj_subq_pred->unit->first_select(); > for (uint i= 0; i < sjm_sel->item_list.elements; i++) > { > - if (sjm_sel->ref_pointer_array[i] == keyuse->val) > + DBUG_ASSERT(keyuse->val->type() == Item::FIELD_ITEM); ^^(1) > + Field *field = ((Item_field*)sjm_sel->ref_pointer_array[i])->field; ^^(2) > + if (field->eq(((Item_field*)keyuse->val)->field)) > return true; > } > return false; As for typecast (2): As far as I understand, it is valid because of the following: - We get to this point in this function only when Semi-Join-Materialization-Scan is used. - SJM-Scan is only used when the subquery's select list has Item_field objects (This is checked in subquery_types_allow_materialization()). * Please add a comment in subquery_types_allow_materialization() about this function relying on that logic. * Please add here an assert that sjm_sel->ref_pointer_array[i] is an Item_field. The logic is complex and can be potentially broken. As for typecast (1): What if keyuse->val is not an Item_field? If we get to this point in this function, it means it's an expression that only depends on the SJ-Inner table. This cannot just happen due to limitations in subquery_types_allow_materialization(), and I wasn't able to construct an example where it would happen (tried playing with multiple-equalities, but substitute_for_best_equal_item is called after get_best_combination(), tried playing with constant tables but then SJ-Materialization is not used, etc. ) I think we will be safer by changing the assert to be this check: if (keyuse->val->type() == Item::FIELD_ITEM) { ... everything else } The idea is that a KEYUSE that satisfies the requirement will still be available, and we just want to filter out any extra KEYUSE that might pop up due to some optimization. what do you think? Ok to push after the above is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Updated Gtid_slave_pos of Untracked domain creates skipped events
by Sachin Setiya 23 Jul '18

23 Jul '18

Hi Andrei, Kristian! Problem:- This issue is regarding Mdev-9107 , where we have 3 master master with do_domain_ids of replication channel master domain-id . , When one replication channel is stopped and that master is updated , slave server updates the gtid slave_pos from second replication channel. And then when replication channel is brought up again then since we have updated the gtid_slave_pos we skip that event. I have created a abstract test case for this probem. (3) ^ ^ / \ (2)----->(1) 3 and 1 is configured with log slave updates. 3 has 2 replication channel m2_s3(do_domain_id=2 ), m1_s3(do_domain_id=1) 1 has one replication channel m2_s1(do_domain_id=2) I have tried with master_use_gtid == slave_pos and current_pos both give same result. When I stop replication channel m2_s3 and then write one-one event in (2) and (1) server. Server 3 will update its gtid_slave_pos and it will have latest gtid event seq no from domain id 2 and 1. When replication channel m2_s3 is bought up again since gtid_slave_pos of (3) is equal to (2) so we actually skipped the event generated on server 2. (I believe network lag between 2-->3 will have similar effect) (I have attached the test cases) Possible Solution. May be we should have one more option in master_use_gtid = binlog_state ? which will compare its binlog_state to arriving event gtid no ? (It is different from current_pos , current_pos will select the maximum seq_no between binlog_state and slave_pos) -- Regards Sachin Setiya Software Engineer at MariaDB

2 5

[Maria-developers] Please review a patch for MDEV-15758 Split Item_bool_func::get_mm_leaf() into virtual methods in Field and Type_handler
by Alexander Barkov 20 Jul '18

20 Jul '18

Hello Vicentiu, Please review my patch for MDEV-15758. It also fixes this bug: MDEV-15759 Expect "Impossible WHERE" for indexed_int_column=out_of_range_int_constant Thanks!

3 4

[Maria-developers] To MDEV-6608 Slave loses master binlog filename
by andrei.elkin＠pp.inet.fi 13 Jul '18

13 Jul '18

Hello, Kristian. I read through your analysis of the case to agree with the plan and its motivation: MDEV-6608> Generally, it is a very bad idea to destroy data during crash recovery ... try an approach that leaves the crashed binlog file intact, and instead ensures that the crashed binlog can be properly handled by the rest of the code. So in case of a partially written event group (transaction) the short (xid-less therefore) transaction is to be rolled back on the master. So why won't we properly enclose the group in the binlog at the end of recovery which means to append ROLLBACK Query-log-event, naturally before that the last event of the group needs sanity check? Was that that you had in mind actually? Cheers, Andrei

2 1

[Maria-developers] MDEV-13118 Wrong results with LOWER and UPPER and subquery
by Alexander Barkov 12 Jul '18

12 Jul '18

Hello Sergei, Please review a patch for MDEV-13118. Thanks!

2 4

[Maria-developers] Selecting Blob by parts (suppose select 10 to 100 char from blob )
by Sachin Setiya 12 Jul '18

12 Jul '18

Hello everyone! On optimizer call (11-7-2018) it was discussed , instead of selecting the whole blob , sometime we just need to select the subpart of it. And we should extend handler/engine api to facilitate this. Although this looks like a interesting idea , Serg (and I ) did not understand why we need this , what is use case of it. Some use case which I think can be like optimizing substring SELECT SUBSTRING(blob_col, 4,4) FROM t1; And we can push down this SUBSTRING condition directly to storage engine. But apart from this what use case this have ? -- Regards Sachin Setiya Software Engineer at MariaDB

1 0

[Maria-developers] What can MariaDB developers learn from Debian and Ubuntu Quality Assurance systems?
by Otto Kekäläinen 11 Jul '18

11 Jul '18

Hello! I had the presentation with the title in the subject line at the developers unconference some weeks ago. The slides are available here: https://mariadb.org/wp-content/uploads/2018/07/2018-06-27-mariadb-dev-learn… I recommend all developers skim though them. Debian has some really nice QA systems. All presentations from the event can be found at: https://mariadb.org/tampere-mariadb-developers-unconference-reportback-pres…

1 0

Re: [Maria-developers] 1a2748c: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 10 Jul '18

10 Jul '18

Hi, Alexey! On Jul 10, Alexey Botchkov wrote: > > > Did you test that it works? > Of course i tested the mysql_install_db script a lot. > Didn't test the .rpm and .deb though. As far as i can see they plainly run > mysql_install_db --user=.. under root. could you please test with rpm and deb too? should only take 10-20 minutes in a buildbot VM. ok to push after that (if all goes well :)) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 1a2748c: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 09 Jul '18

09 Jul '18

Hi, Alexey! On Jul 09, Alexey Botchkov wrote: > revision-id: 1a2748ceb0e30770eebc97d82d7baa885b332c49 (mariadb-10.3.6-45-g1a2748c) > parent(s): aa01f51bdef9cc38d8e0a75ea9e2788651e41d16 > committer: Alexey Botchkov > timestamp: 2018-07-09 13:20:42 +0400 > message: > > MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server. > > Scripts added to set safe permissions for the auth_pam_tool and it's > directory. > > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > index ad7c028..fd26228 100644 > --- a/scripts/mysql_install_db.sh > +++ b/scripts/mysql_install_db.sh > @@ -308,6 +308,7 @@ then > srcpkgdatadir="$srcdir/scripts" > buildpkgdatadir="$builddir/scripts" > plugindir="$builddir/plugin/auth_socket" > + pamtooldir="$builddir/plugin/auth_pam" > elif test -n "$basedir" > then > bindir="$basedir/bin" # only used in the help text > @@ -337,6 +338,7 @@ then > exit 1 > fi > plugindir=`find_in_dirs --dir auth_socket.so $basedir/lib*/plugin $basedir/lib*/mysql/plugin` > + pamtooldir=`find_in_dirs --dir auth_pam.so $basedir/lib*/plugin $basedir/lib*/mysql/plugin` Why not just pamtooldir=$plugindir ? > else > basedir="@prefix@" > bindir="@bindir@" > @@ -445,6 +448,13 @@ done > > if test -n "$user" > then > + chown $user "$pamtooldir/auth_pam_tool_dir" > + if test $? -ne 0 > + then > + echo "Cannot change ownership of the '$pamtooldir\auth_pam_tool_dir' directory" > + echo " to the '$user' user. Check that you have the necessary permissions and try again." > + exit 1 > + fi Looks ok. Did you test that it works? I mean, a test woud be to install from a tarball and run mysql_install_db or install from rpm or deb (they run the install script automatically) and then check that all permissions are correct. Also, I suspect that at additional chown root "$pamtooldir/auth_pam_tool_dir/auth_pam_tool could also be useful here. But not when the script is running in the builddir. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 765ba2a: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 06 Jul '18

06 Jul '18

Hi, Alexey! On Jul 05, Alexey Botchkov wrote: > revision-id: 765ba2ac76dab984183bf829dc3407713a4d5d9b (mariadb-10.3.6-40-g765ba2a) > parent(s): 7e704a2308e25953b5f8fb154eb325df3e25c2ec > committer: Alexey Botchkov > timestamp: 2018-07-05 17:00:47 +0400 > message: > > MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server. > > Proper access permissions for the auth_pam_tool_dir and auth_pam_tool. > > diff --git a/plugin/auth_pam/CMakeLists.txt b/plugin/auth_pam/CMakeLists.txt > index 4943d57..b9313de 100644 > --- a/plugin/auth_pam/CMakeLists.txt > +++ b/plugin/auth_pam/CMakeLists.txt > @@ -11,7 +11,13 @@ IF(HAVE_PAM_APPL_H) > ADD_DEFINITIONS(-D_GNU_SOURCE) > MYSQL_ADD_PLUGIN(auth_pam_v1 auth_pam_v1.c LINK_LIBRARIES pam MODULE_ONLY) > MYSQL_ADD_PLUGIN(auth_pam auth_pam.c LINK_LIBRARIES pam dl MODULE_ONLY) > - MYSQL_ADD_EXECUTABLE(auth_pam_tool auth_pam_tool.c DESTINATION ${INSTALL_PLUGINDIR}/auth_pam_tool_dir COMPONENT Server) > + MYSQL_ADD_EXECUTABLE(auth_pam_tool auth_pam_tool.c DESTINATION ${INSTALL_PLUGINDIR}/auth_pam_tool_dir COMPONENT Server) > TARGET_LINK_LIBRARIES(auth_pam_tool pam) > + INSTALL(CODE "EXECUTE_PROCESS( > + COMMAND chmod u=rwx,g=,o= auth_pam_tool_dir > + COMMAND chmod u=rwx,g=rx,o=rx auth_pam_tool_dir/auth_pam_tool > + COMMAND chmod +s auth_pam_tool_dir/auth_pam_tool > + WORKING_DIRECTORY \$ENV{DESTDIR}\${CMAKE_INSTALL_PREFIX}/${INSTALL_PLUGINDIR}/)" > + COMPONENT Server) I think it's generally ok. Two comments: minor: you can combine two chmods on auth_pam_tool in one, like u=rwxs major: you still need to make auth_pam_tool_dir to be owned by mysql user. I'm afraid the only way to do it is from a post-install scriptlet or from mysql_install_db. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] af15686: MDEV-16242: MyRocks: parallel slave on a table without PK can stop with ER_KEY_NOT_FOUND
by Kristian Nielsen 05 Jul '18

05 Jul '18

psergey(a)askmonty.org (Sergei Petrunia) writes: > MDEV-16242: MyRocks: parallel slave on a table without PK can stop with ER_KEY_NOT_FOUND > > DRAFT: If RBR event applier uses a secondary key or a full table scan > to locate a row, force waiting for prior commit to complete. I think if you wait_for_prior_commit() also for secondary key case, you should definitely restrict it to be only for conservative parallel replication, not for optimistic/agressive. Using a secondary key seems generally fine (as opposed to full table scan), and this problem should affect only conservative mode, no need to penalise optimistic/aggressive. (But maybe that is what is meant with "DRAFT"). - Kristian.

2 2

Re: [Maria-developers] b88e98df9da: C++11 experiment: check if all builders support it
by Sergei Golubchik 04 Jul '18

04 Jul '18

Hi, Sergey! > Current status is: all staging builders pass, except for Ubuntu Precise. But we no longer build on precise, do we? Any followup MDEVs after this one is pushed? Like migrate atomics ops to C++11 or something? > revision-id: b88e98df9daa586bd6ed1f5908ededaa65a6e464 (mariadb-10.3.6-16-gb88e98df9da) > parent(s): 9988a423d229730b252942b7fa066d1591b7eca9 > author: Sergey Vojtovich > committer: Sergey Vojtovich > timestamp: 2018-06-04 12:41:18 +0400 > message: > > C++11 experiment: check if all builders support it > > diff --git a/CMakeLists.txt b/CMakeLists.txt > index 76b0817f8c7..efeae3d1ca5 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -89,6 +89,14 @@ ELSE() > ENDIF() > PROJECT(${MYSQL_PROJECT_NAME}) > > +IF(CMAKE_VERSION VERSION_LESS "3.1") > + IF(CMAKE_CXX_COMPILER_ID STREQUAL "GNU") > + SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11") Why gnu++11 and not c++11 ? And why not for clang? > + ENDIF() > +ELSE() > + SET(CMAKE_CXX_STANDARD 11) I'd expect this to mean c++11, not gnu++11. > +ENDIF() > + > SET(CPACK_PACKAGE_NAME "MariaDB") > SET(CPACK_PACKAGE_DESCRIPTION_SUMMARY "MariaDB: a very fast and robust SQL database server") > SET(CPACK_PACKAGE_URL "http://mariadb.org") > diff --git a/storage/tokudb/PerconaFT/portability/toku_crash.cc b/storage/tokudb/PerconaFT/portability/toku_crash.cc > index 0af85342a99..297cc29d9ca 100644 > --- a/storage/tokudb/PerconaFT/portability/toku_crash.cc > +++ b/storage/tokudb/PerconaFT/portability/toku_crash.cc > @@ -70,7 +70,7 @@ run_gdb(pid_t parent_pid, const char *gdb_path) { > "-ex", "thread apply all bt", > "-ex", "thread apply all bt full", > exe_buf, pid_buf, > - NULL); > + (char*) NULL); Why's that? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

3 3

[Maria-developers] MDEV-13995 MAX(timestamp) returns a wrong result near DST change
by Alexander Barkov 03 Jul '18

03 Jul '18

Hi Sergei, Please review a patch for MDEV-13995. Thanks!

1 0

[Maria-developers] FOR loop documentation
by Alexander Barkov 03 Jul '18

03 Jul '18

Hi Ian, can you please document the FOR loop? I've created an MDEV for it: MDEV-16674 Document FOR loop with links to all relevant tasks. Thanks!

1 0

Re: [Maria-developers] ed2dcf0: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 01 Jul '18

01 Jul '18

Hi, Alexey! Looks good, few comments/suggestions below. That and my reply to your email (about tests and plugin name). Just push when you're ready, no need to put it In-Review again. Thanks! On Jul 01, Alexey Botchkov wrote: > revision-id: ed2dcf00fd26e363f34638a841ff17a111b7e324 (mariadb-10.3.6-33-ged2dcf0) > parent(s): 4b0cedf82d8d8ba582648dcb4a2620c146862a43 > committer: Alexey Botchkov > timestamp: 2018-07-01 12:23:54 +0400 > message: > > MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server. > > The new 'safe' version of the PAM plugin. > It works using the auth_pam_tool executable. That isolates possible > crashes in pam modules and, since we can set the +s bit for the tool, > we can normally run modules that read protected information. > > diff --git a/plugin/auth_pam/CMakeLists.txt b/plugin/auth_pam/CMakeLists.txt > index 5131752..9c0859c 100644 > --- a/plugin/auth_pam/CMakeLists.txt > +++ b/plugin/auth_pam/CMakeLists.txt > @@ -8,6 +8,10 @@ IF(HAVE_PAM_APPL_H) > IF(HAVE_STRNDUP) > ADD_DEFINITIONS(-DHAVE_STRNDUP) > ENDIF(HAVE_STRNDUP) > + ADD_DEFINITIONS(-D_GNU_SOURCE) > MYSQL_ADD_PLUGIN(auth_pam auth_pam.c LINK_LIBRARIES pam MODULE_ONLY) > + MYSQL_ADD_PLUGIN(auth_pam_safe auth_pam_safe.c LINK_LIBRARIES pam dl MODULE_ONLY) perhaps, rename to auth_pam_new.c (and auth_pam_new.so)? > + MYSQL_ADD_EXECUTABLE(auth_pam_tool auth_pam_tool.c DESTINATION ${INSTALL_PLUGINDIR}/auth_pam_tool_dir COMPONENT Server) > + TARGET_LINK_LIBRARIES(auth_pam_tool pam) > ENDIF(HAVE_PAM_APPL_H) > > diff --git a/plugin/auth_pam/auth_pam.c b/plugin/auth_pam/auth_pam.c > index ffc3d6f..c785ba4 100644 > --- a/plugin/auth_pam/auth_pam.c > +++ b/plugin/auth_pam/auth_pam.c > @@ -1,5 +1,5 @@ > /* > - Copyright (c) 2011, 2012, Monty Program Ab > + Copyright (c) 2018 MariaDB Corporation should be "2011, 2018" not just "2018" > This program is free software; you can redistribute it and/or modify > it under the terms of the GNU General Public License as published by > static struct st_mysql_auth info = > diff --git a/plugin/auth_pam/auth_pam_base.c b/plugin/auth_pam/auth_pam_base.c > new file mode 100644 > index 0000000..30b4e01 > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_base.c > @@ -0,0 +1,171 @@ > +/* > + Copyright (c) 2018 MariaDB Corporation > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111-1301 USA */ > + > +// struct param { > + // unsigned char buf[10240], *ptr; > + // MYSQL_PLUGIN_VIO *vio; > +// }; > +//static int write_packet(struct param *param, const unsigned char *buf, > + //int buf_len) > +//static int read_packet(struct param *param, unsigned char **pkt) forgot to remove these comments? > + > +#include <stdio.h> > +#include <string.h> > +#include <security/pam_appl.h> > +#include <security/pam_modules.h> > + > diff --git a/plugin/auth_pam/auth_pam_safe.c b/plugin/auth_pam/auth_pam_safe.c > new file mode 100644 > index 0000000..9dac90d > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_safe.c > @@ -0,0 +1,235 @@ ... > + PAM_DEBUG((stderr, "PAM: Child process prepares pipes.\n")); > + > + if (close(p_to_c[1]) < 0 || > + close(c_to_p[0]) < 0 || > + dup2(p_to_c[0], 0) < 0 || /* Parent's pipe to STDIN. */ > + dup2(c_to_p[1], 1) < 0) /* Sandbox's pipe to STDOUT. */ > + { > + exit(-1); > + } > + > + PAM_DEBUG((stderr, "PAM: check tool directory: %s, %s.\n", > + opt_plugin_dir, tool_name)); > + plugin_dir_len= strlen(opt_plugin_dir); > + if (plugin_dir_len + tool_name_len + 2 > sizeof(toolpath)) > + { > + /* Tool path too long. */ > + exit(-1); > + } > + > + memcpy(toolpath, opt_plugin_dir, plugin_dir_len); > + if (plugin_dir_len && toolpath[plugin_dir_len-1] != FN_LIBCHAR) > + toolpath[plugin_dir_len++]= FN_LIBCHAR; > + memcpy(toolpath+plugin_dir_len, tool_name, tool_name_len+1); check the length before copying? like, toolpath+plugin_dir_len+tool_name_len+1 < sizeof(toolpath) ? or simply strnxfrm(toolpath, sizeof(toolpath), opt_plugin_dir, "/", tool_name, NULL); > + > + PAM_DEBUG((stderr, "PAM: execute pam sandbox [%s].\n", toolpath)); > + (void) execl(toolpath, toolpath, NULL); > + PAM_DEBUG((stderr, "PAM: exec() failed.\n")); > + exit(-1); > + } ... > +static char use_cleartext_plugin; > +static MYSQL_SYSVAR_BOOL(use_cleartext_plugin, use_cleartext_plugin, > + PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY, > + "Use mysql_cleartext_plugin on the client side instead of the dialog " > + "plugin. This may be needed for compatibility reasons, but it only " > + "supports simple PAM policies that don't require anything besides " > + "a password", NULL, NULL, 0); > + > +#ifndef DBUG_OFF > +static MYSQL_SYSVAR_BOOL(debug, pam_debug, PLUGIN_VAR_OPCMDARG, > + "Log all PAM activity", NULL, NULL, 0); > +#endif still a lot of similar code in auth_pam.c and auth_pam_safe.c. could be moved to auth_pam_common.c? > + > + > diff --git a/plugin/auth_pam/auth_pam_tool.c b/plugin/auth_pam/auth_pam_tool.c > new file mode 100644 > index 0000000..5f19d29 > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_tool.c > @@ -0,0 +1,132 @@ > + > +int main(int argc, char **argv) > +{ > + struct param param; > + MYSQL_SERVER_AUTH_INFO info; > + unsigned char field; > + int res; > + char a_buf[MYSQL_USERNAME_LENGTH + 1 + 1024]; > + > + if (read(0, &field, 1) < 1) > + return -1; > +#ifndef DBUG_OFF > + pam_debug= field; > +#endif > + > + PAM_DEBUG((stderr, "PAM: sandbox started [%s].\n", argv[0])); > + > + info.user_name= a_buf; > + if ((res= read_string(0, info.user_name, sizeof(a_buf))) < 0) > + return -1; > + PAM_DEBUG((stderr, "PAM: sandbox username [%s].\n", info.user_name)); > + > +#ifndef DBUG_OFF > + /* Produce the crash for testing purposes. */ > + if ((res == 14) && > + memcmp(info.user_name, "crash_pam_tool", 14) == 0) > + { > + info.user_name= 0; > + memcpy(info.user_name, a_buf, sizeof(a_buf)); > + return -1; > + } now I think it's better to put it into pam_mariadb_mtr.c, sorry :( then it doesn't need to be in debug-only builds > +#endif > + > + info.auth_string= info.user_name + res + 1; > + if (read_string(0, info.auth_string, sizeof(a_buf) - 1 - res) < 0) > + return -1; > + > + PAM_DEBUG((stderr, "PAM: sandbox auth string [%s].\n", info.auth_string)); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 84c9ec5: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 01 Jul '18

01 Jul '18

Hi, Alexey! On Jun 30, Alexey Botchkov wrote: > > not sure about calling it "safe". I think it's more of a side > > effect, the main feature it that it works, while old pam plugin > > simply doesn't :) unless mysqld is run as root. > > > > and not totally sure about calling it also pam. it means one won't > > be able to load it and the old pam plugin at the same time. I > > suspect it'll still ok and benefits overweight it. > > Well i'm open to any ideas here :) > We can swap 'safe' with somethin else. 'box' for instance. > Or we can instead rename the original version like 'fast' or 'old'. > Finally we can build only one version of the plugin. The 'safe' > whenever possible. I'd rather rename the old version to (old) or (requires root) or something. > I also have a question about the testing. > Now pam*.test-s rely on some mariadb_mtr setup for PAM. How can i see what > is in the expected pam.d configuration file, and what pam modules are used? See plugin/auth_pam/testing/pam_mariadb_mtr.c Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] FYI: 10.3 master branch head builds OK Travis-CI
by Otto Kekäläinen 30 Jun '18

30 Jun '18

Hello! I am happy to announce that of this morning, Travis-CI for MariaDB 10.3 shows all green again! [1] Developers: please rebase your 10.3 branches on top of the latest 10.3 to get the Travis-CI benefits, and please don't break Travis. Last time it was all green in January it took only a few weeks until it was broken again. We don't have protected branches enabled yet, so we rely on all developers to be diligent and not push changes that break the all-green-status on Travis-CI for 10.3. Otto [1] https://travis-ci.org/MariaDB/server/builds [2} https://mariadb.org/protected-branches-ensuring-code-quality-git/

1 0

[Maria-developers] MariaDB 10.3.7 and GSSAPI : Issue
by REIX, Tony 28 Jun '18

28 Jun '18

Hi, We are porting MariaDB (10.3.7 now) on AIX. On some machines we have the following error message when running: + cmake . --trace --debug-trycompile --check-system-vars -GUnix Makefiles -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DINSTALL_LAYOUT=RPM -DLOG_LOCATION=/opt/freeware/mariadb/mariadb.log -DPID_FILE_DIR=%{_rundir}/mariadb -DNICE_PROJECT_NAME=MariaDB -DCMAKE_INSTALL_PREFIX=/opt/freeware -DCMAKE_AR=/usr/bin/ar -DCMAKE_SYSTEM_NAME=AIX -DINSTALL_SYSCONFDIR=/opt/freeware/etc -DINSTALL_SYSCONF2DIR=/opt/freeware/etc/my.cnf.d -DINSTALL_INCLUDEDIR=include/mysql -DINSTALL_INFODIR=share/info -DINSTALL_LIBDIR=/opt/freeware/lib64 -DINSTALL_MANDIR=share/man -DINSTALL_MYSQLSHAREDIR=share/mariadb -DINSTALL_PLUGINDIR=/opt/freeware/lib64/mariadb/plugin -DINSTALL_MYSQLTESTDIR=share/mysql-test -DINSTALL_SBINDIR=libexec -DINSTALL_SCRIPTDIR=bin -DINSTALL_SQLBENCHDIR=share -DINSTALL_SUPPORTFILESDIR=share/mariadb -DMYSQL_DATADIR=/opt/freeware/var/lib/mysql -DMYSQL_UNIX_ADDR=/var/lib64/mysql/mysql.sock -DTMPDIR=/var/tmp -DENABLED_LOCAL_INFILE=ON -DSECURITY_HARDENED=ON -DENABLE_DTRACE=ON -DWITH_EMBEDDED_SERVER=ON -DWITH_MARIABACKUP=ON -DWITH_UNIT_TESTS=ON -DCONC_WITH_SSL=ON -DWITH_SSL=system -DWITH_ZLIB=system -DWITH_JEMALLOC=no -DPLUGIN_MROONGA=DYNAMIC -DPLUGIN_OQGRAPH=DYNAMIC -DPLUGIN_CRACKLIB_PASSWORD_CHECK=DYNAMIC -DPLUGIN_ROCKSDB=DYNAMIC -DPLUGIN_SPHINX=DYNAMIC -DPLUGIN_TOKUDB=DYNAMIC -DPLUGIN_CONNECT=DYNAMIC -DCONNECT_WITH_MONGO=OFF -DCONNECT_WITH_JDBC=OFF -DCMAKE_CROSSCOMPILING=FALSE -DWITH_VALGRIND=OFF -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN=OFF -DWITH_INNODB_EXTRA_DEBUG=ON ...... /opt/freeware/share/cmake/Modules/CheckSymbolExists.cmake(87): try_compile(HAVE_KRB5_FREE_UNPARSED_NAME ${CMAKE_BINARY_DIR} /opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/CMakeFiles/CMakeTmp/CheckSymbolExists.cxx COMPILE_DEFINITIONS ${CMAKE_REQUIRED_DEFINITIONS} ${CHECK_SYMBOL_EXISTS_LIBS} CMAKE_FLAGS -DCOMPILE_DEFINITIONS:STRING=${MACRO_CHECK_SYMBOL_EXISTS_FLAGS} ${CMAKE_SYMBOL_EXISTS_INCLUDES} OUTPUT_VARIABLE OUTPUT ) CMake Error at /opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/CMakeFiles/CMakeTmp/CMakeLists.txt:14 (add_executable): Target "cmTC_3efe0" links to target "-L/opt/freeware/lib -blibpath:/opt/freeware/lib::/usr/lib:/lib -I/usr/include -I/opt/freeware/include -L/opt/freeware/lib -blibpath:/opt/freeware/lib:/usr/lib:/lib -bmaxdata:0x80000000 -brtl -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lpthreads" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? CMake Error at /opt/freeware/share/cmake/Modules/CheckSymbolExists.cmake:87 (try_compile): Failed to generate test project build system. Call Stack (most recent call first): /opt/freeware/share/cmake/Modules/CheckCXXSymbolExists.cmake:39 (__CHECK_SYMBOL_EXISTS_IMPL) plugin/auth_gssapi/CMakeLists.txt:24 (CHECK_CXX_SYMBOL_EXISTS) In file: mariadb-10.3.7/64bit/CMakeFiles/CMakeTmp/CMakeLists.txt , there is: cmake_minimum_required(VERSION 3.11.4.0) set(CMAKE_MODULE_PATH "/opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/cmake;/opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/plugin/auth_gssapi/cmake") project(CMAKE_TRY_COMPILE CXX) set(CMAKE_VERBOSE_MAKEFILE 1) set(CMAKE_CXX_FLAGS " -pie -fPIC -fPIC -fno-rtti") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${COMPILE_DEFINITIONS}") set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${EXE_LINKER_FLAGS}") include_directories(${INCLUDE_DIRECTORIES}) set(CMAKE_SUPPRESS_REGENERATION 1) link_directories(${LINK_DIRECTORIES}) add_definitions(-DPACKAGE=test -D_LARGEFILE_SOURCE -D_LARGE_FILES -D_FILE_OFFSET_BITS=64 -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS) cmake_policy(SET CMP0065 OLD) set(CMAKE_RUNTIME_OUTPUT_DIRECTORY "/opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/CMakeFiles/CMakeTmp") add_executable(cmTC_3efe0 "/opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/CMakeFiles/CMakeTmp/CheckSymbolExists.cxx") target_link_libraries(cmTC_3efe0 "-L/opt/freeware/lib -blibpath:/opt/freeware/lib::/usr/lib:/lib -I/usr/include -I/opt/freeware/include -L/opt/freeware/lib -blibpath:/opt/freeware/lib:/usr/lib:/lib -bmaxdata:0x80000000 -brtl -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lpthreads" ) Talking with CMake guys, they said: > I suspect mariadb's source is setting CMAKE_REQUIRED_LIBRARIES > to a space-separated value instead of a ;-separated value. > That's broken here: https://github.com/MariaDB/server/blob/ed0b84a0270bd99b001dd00654875d26e29b…<https://github.com/MariaDB/server/blob/ed0b84a0270bd99b001dd00654875d26e29b…> Do you have an idea about what is broken and what should be changed? either on AIX or for all target? Thanks/Regards, Cordialement, Tony Reix ATOS / Bull SAS ATOS Expert IBM Coop Architect & Technical Leader Office : +33 (0) 4 76 29 72 67 1 rue de Provence - 38432 Échirolles - France www.atos.net<https://mail.ad.bull.net/owa/redir.aspx?C=PvphmPvCZkGrAgHVnWGsdMcDKgzl_dEIs…>

2 3

[Maria-developers] Dead code in class sp_head
by Alexander Barkov 26 Jun '18

26 Jun '18

Hi Aleksey, I found some dead code in sp_head that seem to have appeared in your patch for "IB: 0.2 part IV". If you don't mind, I'll remove it. Please have a look into the attached patch. Thanks.

2 1

Re: [Maria-developers] [Commits] MDEV-16166 RBR breaks with HA_ERR_KEY_NOT_FOUND upon DELETE
by Alexey Botchkov 25 Jun '18

25 Jun '18

Ok to push. 05.06.2018 13:45, Sachin Setiya wrote: > commit bde95bc70d0ac011f55f1b3b55fee1c9276859da > Author: sachin <sachin.setiya(a)maridb.com > <mailto:sachin.setiya@maridb.com>> > Date: Tue Jun 5 15:14:19 2018 +0530 > > MDEV-16166 RBR breaks with HA_ERR_KEY_NOT_FOUND upon DELETE from > table... > with spatial index > So the issue is since it is spatial index , at the time of > searching index > for key (Rows_log_event::find_row) we use wrong field image we use > Field::itRAW while we should be using Field::itMBR > > diff --git a/mysql-test/suite/rpl/r/rpl_row_spatial.result > b/mysql-test/suite/rpl/r/rpl_row_spatial.result > new file mode 100644 > index 00000000000..8f546fc479e > --- /dev/null > +++ b/mysql-test/suite/rpl/r/rpl_row_spatial.result > @@ -0,0 +1,14 @@ > +include/master-slave.inc > +[connection master] > +CREATE TABLE t1 (g POINT NOT NULL, SPATIAL INDEX(g)); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(1 1)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(2 1)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(1 2)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(2 2)')); > +DELETE FROM t1 where MBREqual(g, ST_GEOMFROMTEXT('Point(1 2)')); > +select count(*) from t1; > +count(*) > +3 > +DELETE FROM t1; > +drop table t1; > +include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_row_spatial.test > b/mysql-test/suite/rpl/t/rpl_row_spatial.test > new file mode 100644 > index 00000000000..00c3dd7c54d > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_row_spatial.test > @@ -0,0 +1,17 @@ > +--source include/have_binlog_format_row.inc > +--source include/master-slave.inc > + > +CREATE TABLE t1 (g POINT NOT NULL, SPATIAL INDEX(g)); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(1 1)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(2 1)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(1 2)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(2 2)')); > +DELETE FROM t1 where MBREqual(g, ST_GEOMFROMTEXT('Point(1 2)')); > + > +--sync_slave_with_master > +select count(*) from t1; > + > +--connection master > +DELETE FROM t1; > +drop table t1; > +--source include/rpl_end.inc > diff --git a/sql/key.cc b/sql/key.cc > index 700bf6a05a6..7e5a3309b10 100644 > --- a/sql/key.cc > +++ b/sql/key.cc > @@ -145,7 +145,8 @@ void key_copy(uchar *to_key, uchar *from_record, > KEY *key_info, > { > key_length-= HA_KEY_BLOB_LENGTH; > length= min(key_length, key_part->length); > - uint bytes= key_part->field->get_key_image(to_key, length, > Field::itRAW); > + uint bytes= key_part->field->get_key_image(to_key, length, > +key_info->flags & HA_SPATIAL ? Field::itMBR : Field::itRAW); > if (with_zerofill && bytes < length) > bzero((char*) to_key + bytes, length - bytes); > to_key+= HA_KEY_BLOB_LENGTH; > > > -- > Regards > Sachin Setiya > Software Engineer at MariaDB > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] MDEV-12574 MAX(old_decimal) produces a column of the old DECIMAL type
by Alexander Barkov 24 Jun '18

24 Jun '18

Hi Sergei, Please review a patch for MDEV-12574. I suggest we push it into 10.2 (not into 10.3). Thanks!

2 1

Re: [Maria-developers] 37b432c1d59: MDEV-15242 Poor RBR update performance with partitioned tables
by Sergei Golubchik 22 Jun '18

22 Jun '18

Hi, andrei.elkin! On Jun 19, andrei.elkin(a)pp.inet.fi wrote: > revision-id: 37b432c1d59e66b742f97a280cae7784d85e62ec (mariadb-10.0.35-41-g37b432c1d59) > parent(s): d2e1ed8b936a78ceeec0e64231997d7ed18a4daf > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-06-19 18:14:47 +0300 > message: > > MDEV-15242 Poor RBR update performance with partitioned tables > > Observed and described > partitioned engine execution time difference > between master and slave was caused by excessive invocation > of base_engine::rnd_init which was done also for partitions > uninvolved into Rows-event operation. > The bug's slave slowdown therefore scales with the number of partitions. Ok to push > Fixed with applying an upstream patch. > > References: > ---------- > https://bugs.mysql.com/bug.php?id=73648 > Bug#25687813 REPLICATION REGRESSION WITH RBR AND PARTITIONED TABLES > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MariaDB Developers Unconference next week
by Otto Kekäläinen 22 Jun '18

22 Jun '18

Hello everybody! If you plan to attend the MariaDB Developers Unconference and related activities next week,* make sure you have signed up <https://www.meetup.com/MariaDB-Developers/events/249024232/> to the meetup* . We will soon be posting important information to attendees and only people who have signed up will receive it. Info: - www.meetup.com/MariaDB-Developers/events/249024232/ <https://www.meetup.com/MariaDB-Developers/events/249024232/> - mariadb.org/2018-2-developers-unconference-in-finland/ - mariadb.org/travel-tips-to-tampere-finland/

1 0

Re: [Maria-developers] 84c9ec5: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 22 Jun '18

22 Jun '18

Hi, Alexey! Just a couple of comments (args[] vs stdio and how to prevent brute-forcing), see below. Otherwise looks pretty good. On Jun 06, Alexey Botchkov wrote: > revision-id: 84c9ec5552bf14bda826912fb045ae36f34a4490 (mariadb-10.3.6-9-g84c9ec5) > parent(s): 637af7838316a49267e55cde7cf96e23f63e5c9d > committer: Alexey Botchkov > timestamp: 2018-06-06 14:13:07 +0400 > message: > > MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server. > > Crash-safe version of the PAM plugin added. It runs the auth_pam_tool > application that actually calls the PAM modules. > > diff --git a/plugin/auth_pam/auth_pam_base.c b/plugin/auth_pam/auth_pam_base.c > new file mode 100644 > index 0000000..885e6f0 > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_base.c > @@ -0,0 +1,173 @@ > +/* > + Copyright (c) 2011, 2012, Monty Program Ab could be updated... > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111-1301 USA */ > + > +// struct param { > + // unsigned char buf[10240], *ptr; > + // MYSQL_PLUGIN_VIO *vio; > +// }; > +//static int write_packet(struct param *param, const unsigned char *buf, > + //int buf_len) > +//static int read_packet(struct param *param, unsigned char **pkt) > + > +#define _GNU_SOURCE 1 /* for strndup */ > + > +#include <stdio.h> > +#include <string.h> > +#include <security/pam_appl.h> > +#include <security/pam_modules.h> > + > +/* It least solaris doesn't have strndup */ > + > +#ifndef HAVE_STRNDUP > +char *strndup(const char *from, size_t length) > +{ > + char *ptr; > + size_t max_length= strlen(from); > + if (length > max_length) > + length= max_length; > + if ((ptr= (char*) malloc(length+1)) != 0) > + { > + memcpy((char*) ptr, (char*) from, length); > + ptr[length]=0; > + } > + return ptr; > +} > +#endif > + > +#ifndef DBUG_OFF > +static char pam_debug = 0; > +#define PAM_DEBUG(X) do { if (pam_debug) { fprintf X; } } while(0) > +#else > +#define PAM_DEBUG(X) /* no-op */ > +#endif > + > +static int conv(int n, const struct pam_message **msg, > + struct pam_response **resp, void *data) > +{ > + struct param *param = (struct param *)data; > + unsigned char *end = param->buf + sizeof(param->buf) - 1; > + int i; > + > + *resp = 0; > + > + for (i = 0; i < n; i++) > + { > + /* if there's a message - append it to the buffer */ > + if (msg[i]->msg) > + { > + int len = strlen(msg[i]->msg); > + if (len > end - param->ptr) > + len = end - param->ptr; > + if (len > 0) > + { > + memcpy(param->ptr, msg[i]->msg, len); > + param->ptr+= len; > + *(param->ptr)++ = '\n'; > + } > + } > + /* if the message style is *_PROMPT_*, meaning PAM asks a question, > + send the accumulated text to the client, read the reply */ > + if (msg[i]->msg_style == PAM_PROMPT_ECHO_OFF || > + msg[i]->msg_style == PAM_PROMPT_ECHO_ON) > + { > + int pkt_len; > + unsigned char *pkt; > + > + /* allocate the response array. > + freeing it is the responsibility of the caller */ > + if (*resp == 0) > + { > + *resp = calloc(sizeof(struct pam_response), n); > + if (*resp == 0) > + return PAM_BUF_ERR; > + } > + > + /* dialog plugin interprets the first byte of the packet > + as the magic number. > + 2 means "read the input with the echo enabled" > + 4 means "password-like input, echo disabled" > + C'est la vie. */ > + param->buf[0] = msg[i]->msg_style == PAM_PROMPT_ECHO_ON ? 2 : 4; > + PAM_DEBUG((stderr, "PAM: conv: send(%.*s)\n", > + (int)(param->ptr - param->buf - 1), param->buf)); > + if (write_packet(param, param->buf, param->ptr - param->buf - 1)) > + return PAM_CONV_ERR; > + > + pkt_len = read_packet(param, &pkt); > + if (pkt_len < 0) > + { > + PAM_DEBUG((stderr, "PAM: conv: recv() ERROR\n")); > + return PAM_CONV_ERR; > + } > + PAM_DEBUG((stderr, "PAM: conv: recv(%.*s)\n", pkt_len, pkt)); > + /* allocate and copy the reply to the response array */ > + if (!((*resp)[i].resp= strndup((char*) pkt, pkt_len))) > + return PAM_CONV_ERR; > + param->ptr = param->buf + 1; > + } > + } > + return PAM_SUCCESS; > +} > + > +#define DO(X) if ((status = (X)) != PAM_SUCCESS) goto end > + > +#if defined(SOLARIS) || defined(__sun) > +typedef void** pam_get_item_3_arg; > +#else > +typedef const void** pam_get_item_3_arg; > +#endif > + > +static int pam_auth_base(struct param *param, MYSQL_SERVER_AUTH_INFO *info) > +{ > + pam_handle_t *pamh = NULL; > + int status; > + const char *new_username= NULL; > + /* The following is written in such a way to make also solaris happy */ > + struct pam_conv pam_start_arg = { &conv, (char*) param }; > + > + /* > + get the service name, as specified in > + > + CREATE USER ... IDENTIFIED WITH pam AS "service" > + */ > + const char *service = info->auth_string && info->auth_string[0] > + ? info->auth_string : "mysql"; > + > + param->ptr = param->buf + 1; > + > + PAM_DEBUG((stderr, "PAM: pam_start(%s, %s)\n", service, info->user_name)); > + DO( pam_start(service, info->user_name, &pam_start_arg, &pamh) ); > + > + PAM_DEBUG((stderr, "PAM: pam_authenticate(0)\n")); > + DO( pam_authenticate (pamh, 0) ); > + > + PAM_DEBUG((stderr, "PAM: pam_acct_mgmt(0)\n")); > + DO( pam_acct_mgmt(pamh, 0) ); > + > + PAM_DEBUG((stderr, "PAM: pam_get_item(PAM_USER)\n")); > + DO( pam_get_item(pamh, PAM_USER, (pam_get_item_3_arg) &new_username) ); > + > + if (new_username && strcmp(new_username, info->user_name)) > + strncpy(info->authenticated_as, new_username, > + sizeof(info->authenticated_as)); > + info->authenticated_as[sizeof(info->authenticated_as)-1]= 0; > + > +end: > + pam_end(pamh, status); > + PAM_DEBUG((stderr, "PAM: status = %d user = %s\n", status, info->authenticated_as)); > + return status == PAM_SUCCESS ? CR_OK : CR_ERROR; > +} > + > diff --git a/plugin/auth_pam/auth_pam_safe.c b/plugin/auth_pam/auth_pam_safe.c > new file mode 100644 > index 0000000..28c10b8 > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_safe.c > @@ -0,0 +1,235 @@ > +/* > + Copyright (c) 2011, 2012, Monty Program Ab could be updated... > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111-1301 USA */ > + > +#define _GNU_SOURCE 1 /* for strndup */ > + > +#include <unistd.h> > +#include <string.h> > +#include <mysql/plugin_auth.h> > +#include "auth_pam_tool.h" > +#include <my_global.h> > + > +#ifndef DBUG_OFF > +static char pam_debug = 0; > +#define PAM_DEBUG(X) do { if (pam_debug) { fprintf X; } } while(0) > +#else > +#define PAM_DEBUG(X) /* no-op */ > +#endif > + > +static char *opt_plugin_dir; /* To be dynamically linked. */ > +static const char *tool_name= "auth_pam_tool"; > +static const int tool_name_len= 13; > + > +static int pam_auth(MYSQL_PLUGIN_VIO *vio, MYSQL_SERVER_AUTH_INFO *info) > +{ > + int p_to_c[2], c_to_p[2]; /* Parent-to-child and child-to-parent pipes. */ > + pid_t proc_id; > + int result= CR_ERROR;; double semicolon :) > + > + PAM_DEBUG((stderr, "PAM: opening pipes.\n")); > + if (pipe(p_to_c) < 0 || pipe(c_to_p) < 0) > + { > + /* Error creating pipes. */ > + return CR_ERROR; > + } > + PAM_DEBUG((stderr, "PAM: forking.\n")); > + if ((proc_id= fork()) < 0) > + { > + /* Error forking. */ > + close(p_to_c[0]); > + close(c_to_p[1]); > + goto error_ret; > + } > + > + if (proc_id == 0) > + { > + /* The 'sandbox' process started. */ > + const char *arg[5]; > + char toolpath[FN_REFLEN]; > + size_t plugin_dir_len; > + > + PAM_DEBUG((stderr, "PAM: Child process prepares pipes.\n")); > + > + if (close(p_to_c[1]) < 0 || > + close(c_to_p[0]) < 0 || > + dup2(p_to_c[0], 0) < 0 || /* Parent's pipe to STDIN. */ > + dup2(c_to_p[1], 1) < 0) /* Sandbox's pipe to STDOUT. */ > + { > + exit(-1); > + } > + > + PAM_DEBUG((stderr, "PAM: check tool directory: %s, %s.\n", > + opt_plugin_dir, tool_name)); > + plugin_dir_len= strlen(opt_plugin_dir); > + if (plugin_dir_len + tool_name_len + 2 > sizeof(toolpath)) > + { > + /* Tool path too long. */ > + exit(-1); > + } > + > + memcpy(toolpath, opt_plugin_dir, plugin_dir_len); > + if (plugin_dir_len && toolpath[plugin_dir_len-1] != FN_LIBCHAR) > + toolpath[plugin_dir_len++]= FN_LIBCHAR; > + memcpy(toolpath+plugin_dir_len, tool_name, tool_name_len+1); > + > + arg[0]= toolpath; > + arg[1]= info->user_name; > + arg[2]= info->auth_string; > + if (pam_debug) > + { > + arg[3]= "DEBUG"; > + arg[4]= 0; > + } > + else > + arg[3]= 0; I'd rather passed that via the pipe, like, in the first three lines. so that it wouldn't be seen in `ps` output. May be it's harmless, but better to avoid this argument altogether. Just do, like fprintf(p_to_c[0], "%s\n%s\n%d\n", info->user_name, info->auth_string, pam_debug); and in the tool, fscanf("%s\n%s\n%d\n"); Hmm. Taking into account that the tool is installed SUID, you need to take extra care to avoid buffer overflows, assuming someone can invoke the tool directly. So, a little bit more complex than fscanf("%s\n%s\n%d\n"), sorry > + > + PAM_DEBUG((stderr, "PAM: execute pam sandbox:[%s] [%s] [%s] [%s].\n", > + arg[0], arg[1], arg[2], > + arg[3] ? arg[3] : "No debug")); > + (void) execv(arg[0], (char **) arg); > + PAM_DEBUG((stderr, "PAM: exec() failed.\n")); > + exit(-1); > + } > + > + /* Parent process continues. */ > + > + PAM_DEBUG((stderr, "PAM: parent continues.\n")); > + if (close(p_to_c[0]) < 0 || > + close(c_to_p[1]) < 0) > + goto error_ret; > + > + for (;;) > + { > + unsigned char field; > + > + PAM_DEBUG((stderr, "PAM: listening to the sandbox.\n")); > + if (read(c_to_p[0], &field, 1) < 1) > + { > + PAM_DEBUG((stderr, "PAM: read failed.\n")); > + goto error_ret; > + } > + > + if (field == AP_EOF) > + { > + PAM_DEBUG((stderr, "PAM: auth OK returned.\n")); > + break; > + } > + > + switch (field) > + { > + case AP_AUTHENTICATED_AS: > + PAM_DEBUG((stderr, "PAM: reading authenticated_as string.\n")); > + if (read_string(c_to_p[0], info->authenticated_as, > + sizeof(info->authenticated_as)) < 0) > + goto error_ret; > + break; > + > + case AP_CONV: > + { > + unsigned char buf[10240]; > + int buf_len; > + unsigned char *pkt; > + > + PAM_DEBUG((stderr, "PAM: getting CONV string.\n")); > + if ((buf_len= read_string(c_to_p[0], (char *) buf, sizeof(buf))) < 0) > + goto error_ret; > + > + PAM_DEBUG((stderr, "PAM: sending CONV string.\n")); > + if (vio->write_packet(vio, buf, buf_len)) > + goto error_ret; > + > + PAM_DEBUG((stderr, "PAM: reading CONV answer.\n")); > + if ((buf_len= vio->read_packet(vio, &pkt)) < 0) > + goto error_ret; > + > + PAM_DEBUG((stderr, "PAM: answering CONV.\n")); > + if (write_string(p_to_c[1], pkt, buf_len)) > + goto error_ret; > + } > + break; > + > + default: > + PAM_DEBUG((stderr, "PAM: unknown sandbox field.\n")); > + goto error_ret; > + } > + } > + result= CR_OK; > + > +error_ret: > + close(p_to_c[1]); > + close(c_to_p[0]); > + > + PAM_DEBUG((stderr, "PAM: auth result %d.\n", result)); > + return result; > +} > + > +static struct st_mysql_auth info = > +{ > + MYSQL_AUTHENTICATION_INTERFACE_VERSION, > + "dialog", > + pam_auth > +}; > + > +static char use_cleartext_plugin; > +static MYSQL_SYSVAR_BOOL(use_cleartext_plugin, use_cleartext_plugin, > + PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY, > + "Use mysql_cleartext_plugin on the client side instead of the dialog " > + "plugin. This may be needed for compatibility reasons, but it only " > + "supports simple PAM policies that don't require anything besides " > + "a password", NULL, NULL, 0); > + > +#ifndef DBUG_OFF > +static MYSQL_SYSVAR_BOOL(debug, pam_debug, PLUGIN_VAR_OPCMDARG, > + "Log all PAM activity", NULL, NULL, 0); > +#endif > + > + > +static struct st_mysql_sys_var* vars[] = { > + MYSQL_SYSVAR(use_cleartext_plugin), > +#ifndef DBUG_OFF > + MYSQL_SYSVAR(debug), > +#endif > + NULL > +}; > + > + > +static int init(void *p __attribute__((unused))) > +{ > + if (use_cleartext_plugin) > + info.client_auth_plugin= "mysql_clear_password"; > + if (!(opt_plugin_dir= dlsym(RTLD_DEFAULT, "opt_plugin_dir"))) > + return 1; > + > + return 0; > +} > + > +maria_declare_plugin(pam) > +{ > + MYSQL_AUTHENTICATION_PLUGIN, > + &info, > + "pam", > + "MariaDB Corp", > + "PAM based authentication (safe)", not sure about calling it "safe". I think it's more of a side effect, the main feature it that it works, while old pam plugin simply doesn't :) unless mysqld is run as root. and not totally sure about calling it also pam. it means one won't be able to load it and the old pam plugin at the same time. I suspect it'll still ok and benefits overweight it. > + PLUGIN_LICENSE_GPL, > + init, > + NULL, > + 0x0100, > + NULL, > + vars, > + "1.0", > + MariaDB_PLUGIN_MATURITY_STABLE Let's start from beta and change it to stable later :) > +} > +maria_declare_plugin_end; > diff --git a/plugin/auth_pam/auth_pam_tool.c b/plugin/auth_pam/auth_pam_tool.c > new file mode 100644 > index 0000000..e4addd3 > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_tool.c > @@ -0,0 +1,115 @@ > +/* > + Copyright (c) 2011, 2012, Monty Program Ab could be updated... > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111-1301 USA */ > + > +#define _GNU_SOURCE 1 /* for strndup */ you could just set _GNU_SOURCE once in CMakeLists.txt instead of doing it in every file > + > +#include <stdlib.h> > +#include <unistd.h> > +#include <mysql/plugin_auth_common.h> > + > +struct param { > + unsigned char buf[10240], *ptr; > +}; > + > + > +#include "auth_pam_tool.h" > + > + > +static int write_packet(struct param *param __attribute__((unused)), > + const unsigned char *buf, int buf_len) > +{ > + unsigned char b= AP_CONV; > + return write(1, &b, 1) < 1 || > + write_string(1, buf, buf_len); > +} > + > + > +static int read_packet(struct param *param, unsigned char **pkt) > +{ > + *pkt= (unsigned char *) param->buf; > + return read_string(0, (char *) param->buf, (int) sizeof(param->buf)) - 1; > +} > + > + > +typedef struct st_mysql_server_auth_info > +{ > + /** > + User name as sent by the client and shown in USER(). > + NULL if the client packet with the user name was not received yet. > + */ > + char *user_name; > + > + /** > + A corresponding column value from the mysql.user table for the > + matching account name > + */ > + char *auth_string; > + > + /** > + Matching account name as found in the mysql.user table. > + A plugin can override it with another name that will be > + used by MySQL for authorization, and shown in CURRENT_USER() > + */ > + char authenticated_as[MYSQL_USERNAME_LENGTH+1]; > +} MYSQL_SERVER_AUTH_INFO; > + > + > +#include "auth_pam_base.c" > + > + > +int main(int argc, char **argv) > +{ > + struct param param; > + MYSQL_SERVER_AUTH_INFO info; > + unsigned char field; > + int res; > + > + if (argc < 3) > + return -1; > + > + info.user_name= argv[1]; > + info.auth_string= argv[2]; > + > + if (argc >= 4) > + pam_debug= 1; As we discussed, there should be some ways to prevent it being used to brute-force passwords. It'll be suid root, so it cannot be made executable only by "mysql" user. I see two approaches here: 1. getuid() and check that the user name is "mysql" 2. getppid() and check that it matches the value in a pid_file second is heavier (one might need to read my.cnf to find the location of a pid_file). first hard-codes the user name. oh, a third approach. create a new directory under plugin-dir, say auth_tool with permissions r-x------ and owned by "mysql". Put the actual executable there, so nobody besides "mysql" user will be able to get to it. Yes, this seems the easiest, fastest, and most flexible too :) > + > + PAM_DEBUG((stderr, "PAM: sandbox started [%s] [%s] [%s] [%s].\n", argv[0], > + info.user_name, info.auth_string, > + pam_debug ? argv[3] : "No debug")); Hmm, where will it go? messages written by the tool to its stderr > + > + if ((res= pam_auth_base(&param, &info)) != CR_OK) > + { > + PAM_DEBUG((stderr, "PAM: auth failed, sandbox closed.\n")); > + return -1; > + } > + > + if (info.authenticated_as[0]) > + { > + PAM_DEBUG((stderr, "PAM: send authenticated_as field.\n")); > + field= AP_AUTHENTICATED_AS; > + if (write(1, &field, 1) < 1 || > + write_string(1, (unsigned char *) info.authenticated_as, > + strlen(info.authenticated_as))) > + return -1; > + } > + > + PAM_DEBUG((stderr, "PAM: send OK result.\n")); > + field= AP_EOF; > + write(1, &field, 1); > + > + PAM_DEBUG((stderr, "PAM: sandbox closed.\n")); > + return 0; > +} Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 980a5c31880: MDEV-14572: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status upon EXPLAIN UPDATE in PS
by Sergei Golubchik 19 Jun '18

19 Jun '18

Hi, Oleksandr! On Jan 28, Oleksandr Byelkin wrote: > Hi, Sergei! > > Flag in SELECT_LEX options dropped at the end of execution (in our case > prepare). All commands where EXPLAIN is possible restore the flag in main > SELECT_LEX (in mysql_explain_union()) except UPDATE (because it do > not use mysql_explain_union() > for explain). Without the flag it first try to send OK (as part of update) > then EOF as part of EXPLAIN. Ok. 1. Why does it not use mysql_explain_union() ? 2. Why EXPLAIN flag is need to be removed and restored anyway? 3. Please append the following to the commit comment: " Without it, mysql_update() didn't know that there will be EXPLAIN result set and was sending OK at the end of the update, which conflicted with the EOF sent later by EXPLAIN. " > On Sun, Jan 28, 2018 at 2:45 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > > > Hi, Oleksandr! > > > > On Dec 22, Oleksandr Byelkin wrote: > > > revision-id: 980a5c318803d6aa7355e019869597976dcf19bf (mariadb-10.0.33-37-g980a5c31880) > > > parent(s): 042f763268c0f209e7c12e0a6a72bb5d204dfe29 > > > author: Oleksandr Byelkin > > > committer: Oleksandr Byelkin > > > timestamp: 2017-12-22 14:00:10 +0100 > > > message: > > > > > > MDEV-14572: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status upon EXPLAIN UPDATE in PS > > > > > > Restore EXPAIN flag in SELECT_LEX before execution multi-update by flag in LEX > > > (the same but in other way made before INSERT/DELETE/SELECT) > > > > Please explain what was the reason for the bug, why the status was set > > twice, and how your change fixes it. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] 398ea6a: MDEV-11917 enum/set command-line options aren't respecting max-* settings.
by Sergei Golubchik 18 Jun '18

18 Jun '18

Hi, Alexey! On Mar 13, Alexey Botchkov wrote: > revision-id: 398ea6ab012938d08126c8d0e0291000d06f6cd6 (mariadb-10.2.13-43-g398ea6a) > parent(s): 9d95b8665a3d87ab857e77e220effc64454eb881 > committer: Alexey Botchkov > timestamp: 2018-03-13 17:13:27 +0400 > message: > > MDEV-11917 enum/set command-line options aren't respecting max-* settings. > > Code added to make maximum- working for Sys_var_enum > and Sys_var_set. > > diff --git a/sql/set_var.cc b/sql/set_var.cc > index e96e636..32e52a9 100644 > --- a/sql/set_var.cc > +++ b/sql/set_var.cc > @@ -452,6 +452,22 @@ void sys_var::do_deprecated_warning(THD *thd) > > @retval true on error, false otherwise (warning or ok) > */ > + > + > +bool throw_bounds_warning(THD *thd, const char *name,const char *v) > +{ > + if (thd->variables.sql_mode & MODE_STRICT_ALL_TABLES) > + { > + my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), name, v); This is rather illogical, setting a variable doesn't touch any tables, so MODE_STRICT_ALL_TABLES should not affect it. Also, the error message could be improved. But it's how it was, and it's not subject of this bugfix, I agree. > + return true; > + } > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_TRUNCATED_WRONG_VALUE, > + ER_THD(thd, ER_TRUNCATED_WRONG_VALUE), name, v); > + return false; > +} > + > + > bool throw_bounds_warning(THD *thd, const char *name, > bool fixed, bool is_unsigned, longlong v) > { > @@ -464,14 +480,7 @@ bool throw_bounds_warning(THD *thd, const char *name, > else > llstr(v, buf); > > - if (thd->variables.sql_mode & MODE_STRICT_ALL_TABLES) > - { > - my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), name, buf); > - return true; > - } > - push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > - ER_TRUNCATED_WRONG_VALUE, > - ER_THD(thd, ER_TRUNCATED_WRONG_VALUE), name, buf); ah, good. only one place to change when we fix the above. thanks. > + return throw_bounds_warning(thd, name, buf); > } > return false; > } > diff --git a/sql/sys_vars.ic b/sql/sys_vars.ic > index bc913b1..adcda5d 100644 > --- a/sql/sys_vars.ic > +++ b/sql/sys_vars.ic all ok, looks good. Just one comment. Please make max_var_ptr() logic to be in one place only. If C++ cannot do it, then, in the worst case, make it a macro, like #define define_max_var_ptr(TYPE) \ TYPE *max_var_ptr() { return scope() == SESSION ? (TYPE*)(((uchar*)&max_system_variables) + offset) : 0; } after that ok to push, thanks! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] f3dda8e: MDEV-15357 Wrong result with join_cache_level=4, BNLH join
by Sergei Golubchik 15 Jun '18

15 Jun '18

Hi, Igor! On Apr 30, Igor Babaev wrote: > revision-id: f3dda8e2002dc2ad630c07bda0d8c00be7269907 (mariadb-10.2.14-72-gf3dda8e) > parent(s): b4c5e4a717e3ce2c2d434106cc74417fe9a1d3dc > author: Igor Babaev > committer: Igor Babaev > timestamp: 2018-04-30 21:26:38 -0700 > message: > > MDEV-15357 Wrong result with join_cache_level=4, BNLH join > > This bug was introduced by the architectural changes of the patch > for MDEV-11640. The patch moved initialization of join caches after > the call of make_join_readinfo(). As any failure to initialize > a join cache caused denial of its usage the execution code for > the query had to be revised. This revision required rolling back many > actions taken by make_join_readinfo(). It was not done in the patch. > As a result if a denial of join cache happened some join conditions > could be lost. This was exactly the cause of wrong results in the > bug's reported test case. > > Thus the introduced architectural change is not valid and it would be > better to roll it back. At the same time two new methods > adjust_read_set_for_vcol_keyread() and get_covering_index_for_scan() > were added to the class JOIN_TAB to resolve the problems of MDEV-11640. > > diff --git a/sql/sql_join_cache.cc b/sql/sql_join_cache.cc > index 3612cb6..242cb91 100644 > --- a/sql/sql_join_cache.cc > +++ b/sql/sql_join_cache.cc > @@ -226,6 +226,16 @@ void JOIN_CACHE::calc_record_fields() > flag_fields+= MY_TEST(tab->table->maybe_null); > fields+= tab->used_fields; > blobs+= tab->used_blobs; > + if (tab->type == JT_ALL || tab->type == JT_HASH) > + { > + uint idx= tab->get_covering_index_for_scan(); as you moved JOIN_CACHE initialization back, I suspect that this calc_record_fields() now happens too early for get_covering_index_for_scan() to be useful. And why only JT_ALL and JT_HASH? > + if (idx != MAX_KEY) > + { > + fields-= bitmap_bits_set(tab->table->read_set); > + tab->adjust_read_set_for_vcol_keyread(idx); > + fields+= bitmap_bits_set(tab->table->read_set); > + } > + } > } > if ((with_match_flag= join_tab->use_match_flag())) > flag_fields++; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 30ef26f..e5b75ec 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -11404,6 +11375,61 @@ void JOIN_TAB::remove_redundant_bnl_scan_conds() > } > > > +uint JOIN_TAB::get_covering_index_for_scan() > +{ > + int idx= MAX_KEY; > + if (table->no_keyread) > + return idx; > + if (select && select->quick && > + select->quick->index != MAX_KEY && //not index_merge > + table->covering_keys.is_set(select->quick->index)) > + idx= select->quick->index; > + else if (!table->covering_keys.is_clear_all() && > + !(select && select->quick)) > + { // Only read index tree > + if (loosescan_match_tab) > + idx= loosescan_key; > + else > + { > +#ifdef BAD_OPTIMIZATION > + /* > + It has turned out that the below change, while speeding things > + up for disk-bound loads, slows them down for cases when the data > + is in disk cache (see BUG#35850): > + See bug #26447: "Using the clustered index for a table scan > + is always faster than using a secondary index". > + */ > + if (table->s->primary_key != MAX_KEY && > + table->file->primary_key_is_clustered()) > + idx= table->s->primary_key; > + else > +#endif > + idx= find_shortest_key(table, & table->covering_keys); > + } > + } > + return idx; > +} > + > + > +void JOIN_TAB::adjust_read_set_for_vcol_keyread(uint keyread_idx) > +{ > + if (!table->vcol_set || bitmap_is_clear_all(table->vcol_set)) > + return; > + > + MY_BITMAP *keyread_set= &table->cond_set; // is free for use at call normally one uses tmp_set for this, not hijacks cond_set > + bitmap_clear_all(keyread_set); > + table->mark_columns_used_by_index(keyread_idx, keyread_set); > + bitmap_intersect(keyread_set, table->read_set); > + bitmap_intersect(keyread_set, table->vcol_set); this looks wrong, you probably should do keyread_set INTERSECT (read_set UNION vcol_set) > + if (!bitmap_is_clear_all(keyread_set)) > + { > + bitmap_clear_all(keyread_set); > + table->mark_columns_used_by_index(keyread_idx, keyread_set); > + bitmap_intersect(table->read_set, keyread_set); > + } I don't understand that. 1. What if bitmap_is_clear_all(keyread_set) - what does it mean? 2. Why do you recalculate keyread_set, why not simply copy keyread_set into read_set ? > +} > + > + > /* > Plan refinement stage: do various setup things for the executor > > @@ -11526,7 +11552,10 @@ make_join_readinfo(JOIN *join, ulonglong options, uint no_jbuf_after) > tab->read_first_record= tab->type == JT_SYSTEM ? join_read_system > : join_read_const; > if (table->covering_keys.is_set(tab->ref.key) && !table->no_keyread) > + { > + tab->adjust_read_set_for_vcol_keyread(tab->ref.key); > table->file->ha_start_keyread(tab->ref.key); better combine both into inline void JOIN_TAB::start_keyread() { adjust_read_set_for_vcol_keyread(ref.key); tab->table->file->ha_start_keyread(ref.key); } or even JOIN_TAB::start_keyread_if_needed() putting the if() inside too > + } > else if ((!jcl || jcl > 4) && !tab->ref.is_access_triggered()) > push_index_cond(tab, tab->ref.key); > break; > @@ -11606,35 +11641,19 @@ make_join_readinfo(JOIN *join, ulonglong options, uint no_jbuf_after) > } > } > } > - if (!table->no_keyread) > + uint idx= tab->get_covering_index_for_scan(); > + if (idx != MAX_KEY) > { > if (tab->select && tab->select->quick && > tab->select->quick->index != MAX_KEY && //not index_merge > table->covering_keys.is_set(tab->select->quick->index)) > - table->file->ha_start_keyread(tab->select->quick->index); > + table->file->ha_start_keyread(idx); why do you need an if() here - you already checked that in get_covering_index_for_scan(). I generally don't like this code at all. In JOIN_CACHE you simply use tab->get_covering_index_for_scan(), but here you use it *plus* a bunch of if()s afterwards. It surely creates an impression that JOIN_CACHE and make_join_readinfo may have a different idea about whether keyread is used. Ideally here should be just if (idx != MAX_KEY) tab->start_keyread(idx); and even then it would be kinda questionable (because preconditions might change between two get_covering_index_for_scan() calls). > else if (!table->covering_keys.is_clear_all() && > !(tab->select && tab->select->quick)) > - { // Only read index tree > - if (tab->loosescan_match_tab) > - tab->index= tab->loosescan_key; > - else > - { > -#ifdef BAD_OPTIMIZATION > - /* > - It has turned out that the below change, while speeding things > - up for disk-bound loads, slows them down for cases when the data > - is in disk cache (see BUG#35850): > - See bug #26447: "Using the clustered index for a table scan > - is always faster than using a secondary index". > - */ > - if (table->s->primary_key != MAX_KEY && > - table->file->primary_key_is_clustered()) > - tab->index= table->s->primary_key; > - else > -#endif > - tab->index=find_shortest_key(table, & table->covering_keys); > - } > + { // Only read index tree > + tab->index= idx; > tab->read_first_record= join_read_first; > + table->file->ha_start_keyread(tab->index); > /* Read with index_first / index_next */ > tab->type= tab->type == JT_ALL ? JT_NEXT : JT_HASH_NEXT; > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-6111: Optimizer trace: let's get rid of "optimizer_trace_enabled"
by Sergey Petrunia 15 Jun '18

15 Jun '18

Hi Zhzhzoo, I see that the current patch uses @@optimizer_trace_enabled variable to control it. Meanwhile, in MySQL the name of the variable is "optimizer_trace". It is a flag set: mysql> select @@optimizer_trace; +-------------------------+ | @@optimizer_trace | +-------------------------+ | enabled=on,one_line=off | +-------------------------+ 1 row in set (0.00 sec) but one can set it to 1 or 0 to just set enabled on/off: mysql> set @@optimizer_trace=1; Query OK, 0 rows affected (0.00 sec) mysql> set @@optimizer_trace=0; Query OK, 0 rows affected (0.00 sec) This obviously carries some legacy but I think we should have it the same way, in order to be compatible with MySQL where possible. Please use the same name/values as MySQL does. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-6111: Optimizer trace: on the first output
by Sergey Petrunia 15 Jun '18

15 Jun '18

On Tue, Jun 12, 2018 at 07:40:08AM +0800, zhzhzoo zhang wrote: > I've done > condition processing and some range analysis. > and for the test input ( > https://dev.mysql.com/doc/internals/en/tracing-example.html) It now > outputs something like > (not very readable from the email so I've made two pastes, from the current tree and from MySQL 8): https://gist.github.com/spetrunia/cb4a1a382a9e81a60491d524dbfc08f6 https://gist.github.com/spetrunia/62347cc3bf26e003909a32b378b6684f > ```{ > "join_optimization": { > "select #": 1, > "steps": { > "inner": { > "select #": 1, Two 'select #' :1 seem to be redundant. There are also two nested "steps" elements which seem redundant as well. > "steps": [ > { > "condition_processing": { > "condition": "WHERE", > "original_condition": "`test`.`alias1`.`pk` <> 0 and > `test`.`alias2`.`pk` = `test`.`alias1`.`col_int_key`", > "steps": [ > { > "transformation": "equality_propagation", > "subselect_evaluation": [], > "resulting_condition": "`test`.`alias1`.`pk` <> 0 and > multiple equal(`test`.`alias2`.`pk`, `test`.`alias1`.`col_int_key`)???" What are the "???" at the end? Please get them removed. also, if you check EXPLAIN FORMAT=JSON output, it prints conditions without overquoting, like so: "attached_condition": "alias1.pk <> 0 and alias1.col_int_key is not null" This is done by using QT_EXPLAIN parameter: item->print(&str, QT_EXPLAIN); Let's print them this way here as well. > }, > { > "transformation": "constant_propagation", > "subselect_evaluation": [], > "resulting_condition": "`test`.`alias1`.`pk` <> 0 and > multiple equal(`test`.`alias2`.`pk`, `test`.`alias1`.`col_int_key`)???" > }, BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 46e0c8a9219: MDEV-11071: Assertion `thd->transaction.stmt.is_empty()' failed in Locked_tables_list::unlock_locked_table
by Sergei Golubchik 14 Jun '18

14 Jun '18

Hi, Oleksandr! Looks good, but please see a few comments below. On May 08, Oleksandr Byelkin wrote: > revision-id: 46e0c8a921978e7b95adf3f17f3c85b18d9f9ef6 (mariadb-10.2.14-78-g46e0c8a9219) > parent(s): 9bcd0f5fea8ca26742b10d37b95a966c69909ff1 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-05-08 15:26:26 +0200 > message: > > MDEV-11071: Assertion `thd->transaction.stmt.is_empty()' failed in Locked_tables_list::unlock_locked_table > > fix_length_and_dec now return result (error/OK) > diff --git a/sql/item.h b/sql/item.h > index 8921ee76f6a..e9713730a1c 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -4210,7 +4210,7 @@ class Item_func_or_sum: public Item_result_field, > also to make printing of items inherited from Item_sum uniform. > */ > virtual const char *func_name() const= 0; > - virtual void fix_length_and_dec()= 0; > + virtual bool fix_length_and_dec()= 0; 1. I wonder, if you change fix_length_and_dec to void in one of the derived classes, will you get a compiler warning for that? 2. may be also __attribute__ ((warn_unused_result)) here? I don't know how it'll work on virtual methods, may be it won't. both questions are for the use case "what if we merge some fix_length_and_dec related changes from 10.1, will the compiler tell us to update them?" > bool const_item() const { return const_item_cache; } > table_map used_tables() const { return used_tables_cache; } > Item* build_clone(THD *thd, MEM_ROOT *mem_root); > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index 44cc4f3cae9..46cf412e229 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -535,8 +535,9 @@ void Item_bool_rowready_func2::fix_length_and_dec() > we have to check for out of memory conditions here > */ > if (!args[0] || !args[1]) > - return; > - setup_args_and_comparator(current_thd, &cmp); > + return FALSE; > + bool res= setup_args_and_comparator(current_thd, &cmp); > + return res; not that it matters much (no need to change), but why didn't you return setup_args_and_comparator(current_thd, &cmp); directly? > } > > > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index de1b27cff1a..5efb98bc81d 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -911,10 +911,12 @@ class Item_func_strcmp :public Item_int_func > longlong val_int(); > uint decimal_precision() const { return 1; } > const char *func_name() const { return "strcmp"; } > - void fix_length_and_dec() > + bool fix_length_and_dec() > { > - agg_arg_charsets_for_comparison(cmp_collation, args, 2); > + if (agg_arg_charsets_for_comparison(cmp_collation, args, 2)) > + return TRUE; > fix_char_length(2); // returns "1" or "0" or "-1" this comment is kinda weird. remove? > + return FALSE; > } > Item *get_copy(THD *thd, MEM_ROOT *mem_root) > { return get_item_copy<Item_func_strcmp>(thd, mem_root, this); } > @@ -1987,10 +1997,10 @@ class Item_func_like :public Item_bool_func2 > const char *func_name() const { return "like"; } > enum precedence precedence() const { return CMP_PRECEDENCE; } > bool fix_fields(THD *thd, Item **ref); > - void fix_length_and_dec() > + bool fix_length_and_dec() > { > max_length= 1; > - agg_arg_charsets_for_comparison(cmp_collation, args, 2); should this one also be __attribute__ ((warn_unused_result)) ? > + return agg_arg_charsets_for_comparison(cmp_collation, args, 2); > } > void cleanup(); > > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index 4214980c36c..ca49e06a9e7 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -904,9 +905,10 @@ Item::Type Item_subselect::type() const > } > > > -void Item_subselect::fix_length_and_dec() > +bool Item_subselect::fix_length_and_dec() > { > engine->fix_length_and_dec(0); what about subselect_engine::fix_length_and_dec() ? should return bool too, I guess, right? > + return FALSE; > } > > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 9e7973b745c..f3cb85f01d3 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -4894,7 +4894,13 @@ int create_table_impl(THD *thd, > file= mysql_create_frm_image(thd, orig_db, orig_table_name, create_info, > alter_info, create_table_mode, key_info, > key_count, frm); > - if (!file) > + /* > + We have to check thd->is_error() here because it can be set by > + Item::val* for example, and before it will be cought accidentally by > + Item_func::fix_fields() of the next call. Now we removed the check > + from Item_func::fix_fields() > + */ this is a very confusing comment, don't write in comments that "it used to be this way, but now we've changed it and it is that way" and I still don't understand why do you need to check for thd->is_error() here > + if (!file || thd->is_error()) > goto err; > if (rea_create_table(thd, frm, path, db, table_name, create_info, > file, frm_only)) > @@ -7377,7 +7383,8 @@ static bool mysql_inplace_alter_table(THD *thd, > */ > if (mysql_rename_table(db_type, alter_ctx->new_db, alter_ctx->tmp_name, > alter_ctx->db, alter_ctx->alias, > - FN_FROM_IS_TMP | NO_HA_TABLE)) > + FN_FROM_IS_TMP | NO_HA_TABLE) || > + thd->is_error()) and here > { > // Since changes were done in-place, we can't revert them. > (void) quick_rm_table(thd, db_type, Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 6

Re: [Maria-developers] 0918ae42a83: MDEV-14014 Multi-Slave Replication Fail: bogus data in log event
by Sergei Golubchik 13 Jun '18

13 Jun '18

Hi, Andrei! On Jun 13, andrei.elkin(a)pp.inet.fi wrote: > revision-id: 0918ae42a8394b6faf28f67b70550894851be3bb (mariadb-10.1.33-36-g0918ae42a83) > parent(s): 26be5072429082e634b8fc102609370975443439 > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-06-13 16:27:00 +0300 > message: > > MDEV-14014 Multi-Slave Replication Fail: bogus data in log event > > MDEV-7257 made a dump thread to read from binlog concurrently with > writers as long as the read bytes are below a water-mark > (MYSQL_BIN_LOG::binlog_end_pos). However it appeared to be possible a > dump thread reader reach out for bytes past the water mark through a > feature of IO_CACHE that fills in the internal buffer and while doing > so it could read what the reader is not supposed to see (the bytes > above MYSQL_BIN_LOG::binlog_end_pos). > > The issue is fixed with constraining the IO_CACHE buffer fill to respect > the watermark. > An added test simulates potentially unconstrained buffer fill and an > assert guards this is not the case anymore. Looks good, thanks! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] unbundle RocksDB
by Michal Schorm 13 Jun '18

13 Jun '18

Hello, Since we have a rocksdb package ready in Fedora, I tried to unbundle it, using the system package instead. I wasn't successful though. I haven't found any CMake argument which seemed related and from the build log it seems, the code tries to link against the static libraries in any case. Before I'll start to dig further, I'm stopping by on this list, making sure it is possible. - is it currently supported? - have I overlooked any related CMake argument? Links: MariaDB in Fedora: https://src.fedoraproject.org/rpms/mariadb/tree/master -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

3 3

Re: [Maria-developers] d9913834ceb: MDEV-14014 Multi-Slave Replication Fail: bogus data in log event
by Sergei Golubchik 13 Jun '18

13 Jun '18

Hi, andrei.elkin! On May 30, andrei.elkin(a)pp.inet.fi wrote: > revision-id: d9913834cebe4ed8494b3d76762cd882788a8fd5 (mariadb-10.1.33-28-gd9913834ceb) > parent(s): c1698e8dc50f0c342c1a6886426ba1d43396cd1e > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-05-30 20:06:44 +0300 > message: > > MDEV-14014 Multi-Slave Replication Fail: bogus data in log event > > MDEV-7257 made a dump thread to read from binlog concurrently with > writers as long as the read bytes are below a water-mark > (MYSQL_BIN_LOG::binlog_end_pos). However it appeared to be possible a > dump thread reader reach out for bytes past the water mark through a > feature of IO_CACHE that fills in the internal buffer and while doing > so it could read what the reader is not supposed to see (the bytes > above MYSQL_BIN_LOG::binlog_end_pos). > > The issue is fixed with constraining the IO_CACHE buffer fill to respect > the watermark. > An added test simulates potentially unconstrained buffer fill and an > assert guards this is not the case anymore. What about a much simpler one-liner fix: --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -2549,6 +2549,7 @@ static int send_events(binlog_send_info *info, IO_CACHE* return 1; info->last_pos= linfo->pos; + log->end_of_file= end_pos; error= Log_event::read_log_event(log, packet, info->fdev, opt_master_verify_checksum ? info->current_checksum_alg : BINLOG_CHECKSUM_ALG_OFF); It doesn't change IO_CACHE structure and doesn't add any overhead to the IO_CACHE. Did you already test that your commit actually fixes the issue? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] 20f3479: MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>.
by Sergei Golubchik 12 Jun '18

12 Jun '18

Hi, Alexey! On Jun 11, Alexey Botchkov wrote: > revision-id: 20f347974f6a9b4e2c65a5ca79aefb8e118ba6cd (mariadb-10.1.33-33-g20f3479) > parent(s): 3627dd7f6a1cf5bd7151ff23290d64cb6dffea90 > committer: Alexey Botchkov > timestamp: 2018-06-11 14:58:46 +0400 > message: > > MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>. > > The reload_acl_and_cache() now looks into the VIEW definition > to check the involved tables if the locked_tables_mode. That's a pretty neat solution :) But it seems that - contrary to what I expected - that MariaDB actually distinguishes between locking a view a locking all its tables. Consider this test case: ---- create table t1 (a int); create view v1 as select * from t1; lock table v1 write; insert v1 values (1); select * from t1; unlock tables; drop view v1; drop table t1; ---- this works and inserts 1 just fine. If you change the lock table statement to lock t1, it'll with Table 'v1' was not locked with LOCK TABLES It'd be great if FLUSH TABLES will exhibit the same behavior, it needs to know if the view is locked, not only if all underlying tables are locked. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Maria-discuss] MariaDB crashes because of "long semaphore wait" after migrating from 10.1 to 10.3
by Vicențiu Ciorbaru 12 Jun '18

12 Jun '18

Hi! I have cc'ed the maria-developers mailing list as this is potentially a dev issue and also the devs responsible for Windows and InnoDB development. There might be some people on vacation right now, but this should get looked into in a week or two at most. Let's find a solution for your problem. :) Vicentiu On Sun, 10 Jun 2018, 19:11 TD, <tdcf(a)gmx.net> wrote: > Hi, > > I have a Windows server that has been running MariaDB 10.1 successfully > for over a year. The server remains mostly idle for long times with some > read access, but occasionally there are transactions that add data (about > 500k rows per commit). There can be up to 10 such transactions (one per > database) at the same time and during those times the server is under quite > some load (the code processing the data resides on the same server as the > database). We're using an 8 core AMD Ryzen, 32 GB RAM and a Samsung SSD. > > Like I said, this worked fine with MariaDB 10.1 all the time. > > We're currently in the process of setting up a new server (specs similar > to the ones above) and because MariaDB 10.3 has been released recently, I > was curious and installed it. The database now constantly shuts down > because of "[FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We > intentionally crash the server because it appears to be hung." whenever > there is one of the load situations described above. If I install MariaDB > 10.1 on the new server, it runs as fine as the old one. The crashes also > happen when I start with a clean database (no data migrated from 10.1 > databases). > > I'm not a huge MariaDB config pro, as it's only a tool to store the data, > it just needs to run with reasonable performance and has always worked > until now. As far as I know and can judge from comments in the file, the > my.ini in use is (mostly?) based on my-innodb-heavy-4G.cnf, but coworkers > might have made modifications. I've attached the contents below. > > I don't know how I can debug this issue further. Windows event viewer > contains several log warnings with the text "InnoDB: A long semaphore > wait:" but no further information why this wait happens. I couldn't find > any other error logs. I guess it is some configuration error (I read about > MariaDB switching from XtraDB to InnoDB, so I guess that the issue might be > related). > > Migrating to 10.3 is not an absolute must, but it would be nice to be able > to fix the problem. > > Any help would be greatly appreciated and if you need more information, > I'll gladly try to provide it. > > Thanks, > Tom. > > my.ini: > [mysqld] > max_connections = 100 > max_allowed_packet = 16M > binlog_cache_size = 1M > max_heap_table_size = 64M > read_buffer_size = 2M > read_rnd_buffer_size = 16M > sort_buffer_size = 8M > join_buffer_size = 8M > thread_cache_size = 10 > query_cache_size = 64M > query_cache_limit = 2M > thread_stack = 240K > tmp_table_size = 64M > slow_query_log > expire_logs_days = 14 > log-bin = mysql-bin > character-set-server=utf8 > collation-server=utf8_general_ci > > #*** MyISAM Specific options > key_buffer_size = 32M > bulk_insert_buffer_size = 64M > myisam_sort_buffer_size = 128M > myisam_max_sort_file_size = 10G > myisam_repair_threads = 1 > myisam_recover > > # *** INNODB Specific options *** > #innodb_fatal_semaphore_wait_threshold = 6000 #tried that, didn't help > #innodb_adaptive_hash_index = 0 #tried that (suggested in some stack > overflow post, didn't help) > innodb_buffer_pool_size = 2G > innodb_data_file_path = ibdata1:10M:autoextend > innodb_write_io_threads = 8 > innodb_read_io_threads = 8 > innodb_thread_concurrency = 16 > innodb_log_buffer_size = 8M > innodb_log_file_size = 256M > innodb_log_files_in_group = 3 > innodb_max_dirty_pages_pct = 10 > innodb_io_capacity = 2000 > innodb_lock_wait_timeout = 120 > > [mysql] > no-auto-rehash > > [myisamchk] > key_buffer_size = 512M > sort_buffer_size = 512M > read_buffer = 8M > write_buffer = 8M > > [mysqld_safe] > open-files-limit = 8192 > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp >

3 2

[Maria-developers] MariaDB Developers Unconference in a few weeks!
by Otto Kekäläinen 08 Jun '18

08 Jun '18

Hello everybody! The next developers unconference will be in a few weeks: https://mariadb.org/2018-2-developers-unconference-in-finland/ There are still open slots so we encourage you to sign up now. https://www.meetup.com/MariaDB-Developers/events/249024232/ We have plenty of break-out rooms this time and there is space, so we also encourage you to suggest sessions/discussions you would like to have to ian(a)mariadb.org. Around mid-summer is a great time to visit Finland. The nights that don't get dark are worth experiencing! Getting to the venue in Tampere., Finlnd is also very easy. https://mariadb.org/travel-tips-to-tampere-finland/#why-visit Hope to see you there! - Otto

1 0

Re: [Maria-developers] [Commits] 56d339111a4: MDEV-16374: Filtered shows 0 for materilization scan for a semi join, which makes optimizer always picks
by Sergey Petrunia 05 Jun '18

05 Jun '18

Hi Varun, On Mon, Jun 04, 2018 at 12:34:49PM +0530, Varun wrote: > revision-id: 56d339111a4e5c868ffa3b1f2538c5f101f67767 (mariadb-10.0.30-366-g56d339111a4) > parent(s): a61724a3ca187f6eb44fc67948acb76a94efa783 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-06-04 12:26:56 +0530 > message: > > MDEV-16374: Filtered shows 0 for materilization scan for a semi join, which makes optimizer always picks > materialization scan over materialization lookup > > For non-mergeable semi-joins we don't store the estimates of the IN subquery in table->file->stats.records. > In the function TABLE_LIST::fetch_number_of_rows, we store the number of rows in the tables > (estimates in case of derived table/views). > Currently we don't store the estimates for non-mergeable semi-joins, which leads to a problem of selecting > materialization scan over materialization lookup. > Fixed this by storing these estimated appropriately > * This is a change in the optimizer. Should we really do it in 10.0? In my opinion, this should go into 10.3 or 10.4. Let's discuss this, and I think Igor may have something to say, too. * The patch updates selectivity.test but not selectivity_innodb.result (which includes selectivity.test). * Please see my last comments at https://jira.mariadb.org/browse/MDEV-16374 . Here is an alternative patch (it passes the testsuite, and produces the same result for selectivity* tes): diff --git a/sql/sql_select.cc b/sql/sql_select.cc index 1285835..884997c 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -6602,7 +6602,7 @@ double matching_candidates_in_table(JOIN_TAB *s, bool with_found_constraint, { TABLE *table= s->table; double sel= table->cond_selectivity; - double table_records= (double)table->stat_records(); + double table_records= s->records; // psergey-10 dbl_records= table_records * sel; return dbl_records; } What do you think about it? I would like to get Igor's opinion also because I'm not certain which member variable should be changed here. (Why does derived table modify table->stats.records while JTBM semi-joins modify JOIN_TAB::records? These seem like two solutions for the same problem?) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Proposal for Negative Grants Project
by Rutuja Surve 04 Jun '18

04 Jun '18

Hello, Please find the proposal for the Negative Grants project attached along with this e-mail. It would be great to hear your opinions and suggestions on the approach proposed here, before delving into its implementation. Some context about the project: *The current privilege system allows access in this manner:GRANT SELECT ON some_database.* TO a_user@%;Any revoke would revoke access from the entire database. We want to support a revoke that would disable select on a specific table, like:REVOKE SELECT ON some_database.secret_table FROM a_user@%;Reference:[1] https://jira.mariadb.org/browse/MDEV-14443 <https://jira.mariadb.org/browse/MDEV-14443>[2] https://mariadb.com/kb/en/library/google-summer-of-code-2018/ <https://mariadb.com/kb/en/library/google-summer-of-code-2018/>Thanks,Rutuja*

3 6

[Maria-developers] Conservative parallel slave is "too optimistic" for certain engines
by Sergey Petrunia 01 Jun '18

01 Jun '18

Hello, This comes from MDEV-16242. == Symptoms == When one runs a parallel slave (mode=conservative) and replicates DML for MyRocks table without a Primary Key, replication may stop with a ER_KEY_NOT_FOUND error. This may happen even if the queries were run on the master sequentially. == A detail about conservative replication == Suppose the master runs these two statements, sequentially: INSERT INTO t1 VALUES (5),(6),(7); DELETE FROM t1; Parallel slave may schedule the INSERT to Thread1, and the DELETE to Thread2. In conservative parallel replication, the DELETE "will wait" to be executed after the INSERT. One may expect that "will wait" here means that DELETE execution does not start until the INSERT has committed. But it's more subtle than that: actually, DELETE execution will start as soon as the INSERT is ready to do the prepare/commit steps. I assume this was done to increase the parallelism: initial phases of DELETE can be ran in parallel with commit/prepare steps of the INSERT. This is safe to do: 1. The INSERT has acquired locks for all rows it is about to touch. DELETE will not be able to prevent INSERT from committing. 2. The DELETE starts its execution on a database that doesn't include the results of the INSERT (the INSERT has not committed yet!). But this is fine, because the DELETE locks the rows it is about to modify. If it attempts to access a row that the INSERT is about to insert, it will block on a lock until the INSERT finishes. == Applying this to MyRocks == The critical part is #2. It works if the DELETE command will acquire row locks for rows it about to DELETE. This is normally true: If the storage engine supports gap locks, attempting to read a locked gap will cause DELETE to wait for INSERT If the storage engine doesn't support gap locks, it will use Row-Based-Replication. RBR's Delete_rows_event includes Primary Key value of the row that should be deleted. Doing a point lookup on PK will cause the DELETE to block on the row lock. The remaining case is - storage engine that doesn't support gap locks - Row-based-replication of table without primary key In this case, finding the row to delete is done by scanning the table until we find the row where all columns match. The scan does not see the rows that the INSERT is about to INSERT, and DELETE stops with an error, "we could not find the row to delete". == Possible solutions == 1. Implement gap locking in MyRocks. This will take some time. 2. Change parallel slave to wait *for commit*. This should only be done if tables that are updated do not support Gap Locking. This is hard, it will require making risky changes on the SQL layer. 3. Disallow parallel slave execution for tables without Gap Locking and without primary key. (Looks most promising but I am not sure if it is sufficient). BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 2

[Maria-developers] Test failure overview for Debian/Ubuntu on 10.3 master branch (June 1st 2018 status report)
by Otto Kekäläinen 01 Jun '18

01 Jun '18

Hello! I reported here about the test failure status in early April and here is a follow-up. My old emails can be found here: https://lists.launchpad.net/maria-developers/msg11171.html https://lists.launchpad.net/maria-developers/msg11170.html If one looks at the picture from April and compares to the status today, the situation is a bit sad as we have even less green than before: https://lists.launchpad.net/maria-developers/pngcpxTB4wjTW.png Current issues we have in addition to the usual mtr test failures are: - On multiple bionic builds the upgrade fails to run with no apparent reason. The builder script does not seem to find any previous installation of anything, so that probably is missing from the images: + dpkg -l + grep -iE 'maria|mysql|galera' + true Thus the build fails. Example log: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-bionic-amd64/bui lds/196/steps/upgrade2/logs/stdio - On bionic and xenial aarch64 mysqld fails to start. In syslog we can see May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 0 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGN ORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION',' NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_O PTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANS I','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES' ,'STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES',' ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_N OT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH','EMPT Y_STRING_IS_NULL','SIMULTANEOUS_ASSIGNMENT'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGN ORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION',' NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_O PTIONS','NO_TABLE_OPTIONS','NO May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: _FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALU May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 0 [ERROR] mysqld: Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler. May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 6 [ERROR] InnoDB: Column last_update in table `mysql`.`innodb_table_stats` is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch). May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 6 [ERROR] InnoDB: Fetch of persistent statistics requested for table `mysql`.`gtid_slave_pos` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. http://buildbot.askmonty.org/buildbot/builders/kvm-deb-xenial-aarch64/b uilds/277/steps/upgrade1/logs/syslog - stretch amd64 fails to compile due to librocksdblib.a(compaction_job.cc.o): In function `rocksdb::ZSTD_TrainDictionary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long)': ./builddir/storage/rocksdb/./storage/rocksdb/rocksdb/util/compression.h :812: undefined reference to `ZDICT_trainFromBuffer' ./builddir/storage/rocksdb/./storage/rocksdb/rocksdb/util/compression.h :813: undefined reference to `ZDICT_isError' collect2: error: ld returned 1 exit status http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/bu ilds/3986/steps/compile/logs/stdio - jessie amd64 builds to compile due to /usr/bin/ld.bfd.real: /tmp/ccOIwjFo.ltrans0.ltrans.o: relocation R_X86_64_PC32 against symbol `toku_product_name_strings' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld.bfd.real: final link failed: Bad value collect2: error: ld returned 1 exit status http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/bui lds/4298/steps/compile/logs/stdio This -fPIC issue might be related to a change in hardening flags I "forward-ported" from Debian and I will look into it. All the other issues are things I cannot do anything about.. :( On the bright side I can report that there are at least no build regressions on Launchpad with current 10.3: https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.3/+build s?build_text=&build_state=all - Otto

1 0

[Maria-developers] With_sum_func_cache
by Alexander Barkov 31 May '18

31 May '18

Hello Sanja, I recently worked on MDEV-16309 and had hard time to find which Item classes in the hierarchy: - have always with_sum_func==true - have always with_sum_func==false - have variable with_sum_func To make it sure, before actually working on MDEV-16309, I had to create a separate working tree and did with Item::with_sum_func the same change that we previously did for Item::with_subselect in: MDEV-14517 Cleanup for Item::with_subselect and Item::has_subquery() (which you reviewed) - I find the code easier to read this way (instead of searching for all possible assignments, one can search who overrides the method) - Also, some Item can become smaller in size. It's pity to throw the patch away. So perhaps we could just apply it. Can you please have a look? Thanks. I the meanwhile I'll create a new MDEV for it (with a similar description to MDEV-14517)

2 2

[Maria-developers] Open and close a TABLE_LIST at random place in code
by Eugene Kosov 31 May '18

31 May '18

Hello. I need to see a table structure while executing some query (DELETE in my case, but there will be more query types in future). I have a TABLE_LIST which came from a parser. Now I'm opening it with open_table_only_view_structure(), examine table structure and close it, trying to prevent any side effects. And there is a lot of them. Current code looks like this: if (open_tables_only_view_structure(thd, table_list, false)) { return true; } // examine table structure close_mysql_tables(thd); for (Sroutine_hash_entry *e = thd->lex->sroutines_list.first;e; e=e->next) e->mdl_request.ticket= NULL; for (TABLE_LIST *it= table_list; it; it= it->next_global) { it->table= NULL; it->reinit_before_use(thd); //it->next_global= NULL; it->derived_result= NULL; it->derived= NULL; it->view= NULL; it->view_tables= NULL; it->view_sctx= NULL; it->file_version= 0; it->mariadb_version= 0; it->updatable_view= 0; it->view_suid= 0; it->derived_type= 0; it->open_type= OT_BASE_ONLY; it->view_creation_ctx= 0; // it->set_table_ref_id(TABLE_REF_NULL, 0); } Is there a way to close a TABLE_LIST gracefully without touching a lot of it's fields? Or even maybe it's possible to open a table in a simpler manner? -- Eugene

2 3

Re: [Maria-developers] [Commits] de722585c1b: MDEV-16225: wrong resultset from query with semijoin=on
by Sergey Petrunia 27 May '18

27 May '18

Hi Varun, On Thu, May 24, 2018 at 03:17:02AM +0530, Varun wrote: > revision-id: de722585c1b2ad2af99bd1749e8566ea77838764 (mariadb-5.5.56-219-gde722585c1b) > parent(s): 1ada4afb0a51f7283b6187a95019ec2cb80c8a0b > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-05-24 03:09:42 +0530 > message: > > MDEV-16225: wrong resultset from query with semijoin=on > > For non-semi-join subquery optimization we do a cost based decision between > Materialisation and IN -> EXIST transformation. The issue in this case is that for IN->EXIST transformation > we run JOIN::reoptimize with the IN->EXISt conditions and we come up with a new query plan. But when we compare > the cost with Materialization, we make the decision to chose Materialization so we need to restore the query plan > for Materilization. The restoring of query plan is not handled correctly. > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index d6d269a700f..b8d43b9797e 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -23805,6 +23805,22 @@ void JOIN::restore_query_plan(Join_plan_state *restore_from) > } > > } > + else > + { > + /* > + After running JOIN::reoptimize for IN->EXISTS > + transformation we might add elements to the keyuse array, > + The original query plan was that the keyuse array had no elements, so we > + should restore that state here. > + */ > + while(keyuse.elements) > + delete_dynamic_element(&keyuse, 0); I'm wondering, why do we need the above loop and not just delete everything with delete_dynamic(). But if I don't miss anything, this is not significant. Ok to push either the above variant or one with delete_dynamic(). > + for (uint i= 0; i < table_count; i++) > + { > + join_tab[i].keyuse= NULL; > + join_tab[i].checked_keys.clear_all(); > + } > + } > memcpy((uchar*) best_positions, (uchar*) restore_from->best_positions, > sizeof(POSITION) * (table_count + 1)); > /* Restore SJM nests */ > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Maria-discuss] Query ambiguity
by Karthick Subramanian 17 May '18

17 May '18

Check whether you really need a comma here: *metrics_5.id <http://metrics_5.id/>,* *(SELECT * Also in your screen shot, AS 'LT CPC`, there is no comma and you use derived table in SELECT and trying to use JOIN in the FROM using this. Sorry to say this, I think you can either break this logic into small simple chunks and use SP to derive the logic in case you are not much comfortable writing a complex SQL. I think there is lot of syntax errors on it. On Thu, May 17, 2018 at 10:05 AM, Dev C <chauhan.devsur(a)gmail.com> wrote: > Here is the full code and picture on what is required. > > SELECT > campaigns_5.name AS `Campaign Name`, > adgroups_5.name AS `Adgroup Name`, > STNegData1W.query, > keywords_5.matchType AS `Match Type`, > Sum(metrics_5.clicks) AS `LT Clicks`, > Sum(metrics_5.costs) AS `LT Spend`, > Sum(metrics_5.orders) AS `LT Orders`, > metrics_5.sales AS `LT Sales`, > If(Sum(metrics_5.sales)=0,0,Sum(metrics_5.costs)/Sum(metrics_5.sales)) > AS `LT ACOS`, > If(Sum(clicks)=0,0,Sum(costs)/Sum(clicks)) AS `LT CPC` > FROM (((keywords_5 INNER JOIN campaigns_5 ON keywords_5.campaignId = > campaigns_5.campaignId) INNER JOIN adgroups_5 ON keywords_5.adGroupId = > adgroups_5.adGroupId) INNER JOIN STNegData1W ON keywords_5.keywordId = > STNegData1W.keywordId) INNER JOIN metrics_5 ON STNegData1W.metricId = *metrics_5.id > <http://metrics_5.id>,* > *(SELECT * > STNegData4W.query, > STNegData4W.metricId, > STNegData4W.keywordId, > metrics_5.impressions > FROM > (SELECT searchterm_5.query, searchterm_5.metricId, > searchterm_5.keywordId, Sum(metrics_5.orders) AS `4W Orders` > FROM searchterm_5 INNER JOIN metrics_5 ON searchterm_5.metricId = > metrics_5.id > GROUP BY searchterm_5.query, searchterm_5.metricId, > searchterm_5.keywordId, metrics_5.created_at > HAVING (((Sum(metrics_5.orders))=0) AND ((metrics_5.created_at) > > DATE_SUB('2018-05-10',INTERVAL 28 DAY)))) AS STNegData4W > INNER JOIN > metrics_5 ON STNegData4W.metricId = metrics_5.id > GROUP BY STNegData4W.query, STNegData4W.metricId, STNegData4W.keywordId, > metrics_5.impressions, metrics_5.created_at > HAVING (((metrics_5.impressions)>0) AND ((metrics_5.created_at)> > DATE_SUB('2018-05-10', INTERVAL 7 DAY)))) AS STNegData1W > GROUP BY campaigns_5.name, adgroups_5.name, STNegData1W.query, > keywords_5.matchType, metrics_5.sales, campaigns_5.user_market_id, Right( > campaigns_5.name,4) > HAVING (((Sum(metrics_5.clicks))>5) AND ((campaigns_5.user_market_id)=12) > AND ((Right(campaigns_5.name,4))="PFUS")) > > On Thu, May 17, 2018 at 4:06 PM, Pantelis Theodosiou <ypercube(a)gmail.com> > wrote: > >> What Rhys said, please post queries as text, not as images. >> >> The issue is likely due to the way you structured your query. You have >> put a derived table in a random place (in the SELECT list). >> >> Please try to read about CTEs. It will help you design those complex >> queries better and make them more readable and easy to edit: >> https://mariadb.com/kb/en/library/with/ >> >> Best regards >> >> Pantelis Theodosiou >> >> On Thu, May 17, 2018 at 11:23 AM, <Rhys.Campbell(a)swisscom.com> wrote: >> >>> You're not making a lot of sense here. Please explain the "ambiguity". >>> That's a fairly complex query so you should include as text. >>> >>> >>> >>> "Does not work" <- In the image. It helps if you include error message. >>> >>> >>> >>> *From:* Maria-discuss [mailto:maria-discuss-bounces+rhys.campbell= >>> swisscom.com(a)lists.launchpad.net] *On Behalf Of *Dev C >>> *Sent:* 17 May 2018 06:41 >>> *To:* maria-discuss(a)lists.launchpad.net; Maria Developers < >>> maria-developers(a)lists.launchpad.net> >>> *Subject:* [Maria-discuss] Query ambiguity >>> >>> >>> >>> Hello >>> >>> >>> >>> I have query ambiguity as attached. Please help me on how to solve that. >>> >>> >>> >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~maria-discuss >>> Post to : maria-discuss(a)lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~maria-discuss >>> More help : https://help.launchpad.net/ListHelp >>> >>> >> > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp > >

1 0

Re: [Maria-developers] [Maria-discuss] Query ambiguity
by Pantelis Theodosiou 17 May '18

17 May '18

What Rhys said, please post queries as text, not as images. The issue is likely due to the way you structured your query. You have put a derived table in a random place (in the SELECT list). Please try to read about CTEs. It will help you design those complex queries better and make them more readable and easy to edit: https://mariadb.com/kb/en/library/with/ Best regards Pantelis Theodosiou On Thu, May 17, 2018 at 11:23 AM, <Rhys.Campbell(a)swisscom.com> wrote: > You're not making a lot of sense here. Please explain the "ambiguity". > That's a fairly complex query so you should include as text. > > > > "Does not work" <- In the image. It helps if you include error message. > > > > *From:* Maria-discuss [mailto:maria-discuss-bounces+rhys.campbell= > swisscom.com(a)lists.launchpad.net] *On Behalf Of *Dev C > *Sent:* 17 May 2018 06:41 > *To:* maria-discuss(a)lists.launchpad.net; Maria Developers < > maria-developers(a)lists.launchpad.net> > *Subject:* [Maria-discuss] Query ambiguity > > > > Hello > > > > I have query ambiguity as attached. Please help me on how to solve that. > > > > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp > >

2 1

Re: [Maria-developers] [Maria-discuss] Interval in MYsql query
by Peter Laursen 16 May '18

16 May '18

Use the DATE_ADD( ) or DATE_SUB() function with INTERVAL as explained here: https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html#functi… -- Peter. On Wed, May 16, 2018 at 4:07 PM, Dev C <chauhan.devsur(a)gmail.com> wrote: > Hello > > If I use below query > (metrics_5.created_at)>'5/10/2018'- 28 ) > it gives me some result > but if i use below query > (metrics_5.created_at)>'5/10/2018'- INTERVAL 28 DAY ) > > It does not give me any result, any one know why ? > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp > >

1 0

[Maria-developers] [question] password input length
by Michal Schorm 16 May '18

16 May '18

Hello everyone, 1) I found out, that passwords longer than 79 characters aren't accepted via password prompt. Is that expected? Morie info and reproducer in RH bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1566179 -- 2) Another password length cap at 213 characters can be found when using sha256_password authentification plugin. Is that expected too? Morie info and reproducer in RH bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1566219 -- 3) Different theme: why aren't tests for mariadb-connector-odbc, that can be found on GitHub, packged to the source tarball ? -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

2 2

Re: [Maria-developers] a639eff: MDEV-15813 ASAN use-after-poison in hp_hashnr upon HANDLER READ on a versioned HEAP table.
by Sergei Golubchik 16 May '18

16 May '18

Hi, Alexey! On May 16, Alexey Botchkov wrote: > revision-id: a639eff5940ed4147709511aeaee922fa7095284 (mariadb-10.3.6-193-ga639eff) > parent(s): cd15e764a8a4f3e799f1efaede413c95e3233df6 > committer: Alexey Botchkov > timestamp: 2018-05-16 09:44:22 +0400 > message: > > MDEV-15813 ASAN use-after-poison in hp_hashnr upon HANDLER READ on a versioned HEAP table. > > Check index capabilities before executing HANDLER READ command. > > diff --git a/sql/sql_handler.cc b/sql/sql_handler.cc > index 187a746..02d5bbf 100644 > --- a/sql/sql_handler.cc > +++ b/sql/sql_handler.cc > @@ -660,6 +660,18 @@ mysql_ha_fix_cond_and_key(SQL_HANDLER *handler, > Item *item; > key_part_map keypart_map; > uint key_len; > + const KEY *c_key= table->s->key_info + handler->keyno; > + > + if ((c_key->flags & HA_SPATIAL) || > + c_key->algorithm == HA_KEY_ALG_FULLTEXT || > + (ha_rkey_mode != HA_READ_KEY_EXACT && > + (table->file->index_flags(handler->keyno, 0, TRUE) & > + (HA_READ_NEXT | HA_READ_PREV | HA_READ_RANGE)) == 0)) This is strange. Do you mean that SPATIAL and FULLTEXT indexes have HA_READ_NEXT or HA_READ_PREV or HA_READ_RANGE? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-16094 Crash when using AS OF with a stored function
by Alexander Barkov 15 May '18

15 May '18

Hi Sergei, Please review a patch for MDEV-16094. Thanks!

2 2

Re: [Maria-developers] review of MDEV-16101 ADD PARTITION on table partitioned by list does not work …
by Jacob Mathew 14 May '18

14 May '18

Hi Holyfoot, Your fix creates more lists, all of them containing one element, but it does only one reorganization. My fix does a reorganization whenever a 33rd element is processed but creates less lists. I agree that reducing the number of reorganizations to one is a better way to fix the problem. So I will use your fix. Thanks, Jacob Jacob B. Mathew Spider and Server Developer MariaDB Corporation +1 408 655 8999 (mobile) jacob.b.mathew (Skype) jacob.mathew(a)MariaDB.com On Sun, May 13, 2018 at 2:36 PM, Alexey Botchkov <holyfoot(a)mariadb.com> wrote: > Hi, Jacob. > > Your fix: > https://github.com/MariaDB/server/commit/85bbea8dbb9c01defa993e44a5b16c > 43ef851fbd > > I don't like the way your fix work. > So after we get the 32-th item we do the reorganize_into_single_ > field_col_val() > trick that turns one element with 32 values into 32 elements with a single > value. > Then as we get 33-th element, we again put it into that first element, and > do so for > the next 31 element then call the reorganize_ again. > > I think we should just keep the 'one-value' list after the 'reorganize_' > So my fix would be like this: > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -1996,9 +1996,11 @@ part_column_list_val *partition_info::add_column_value(THD > *thd) > into the structure used for 1 column. After this we call > ourselves recursively which should always succeed. > */ > + num_columns= curr_list_object; > if (!reorganize_into_single_field_col_val(thd)) > { > - DBUG_RETURN(add_column_value(thd)); > + if (!init_column_part(thd)) > + DBUG_RETURN(add_column_value(thd)); > } > > > Best regards. > HF > >

1 0

Re: [Maria-developers] 5651e72: MDEV-15465 Server crash or ASAN heap-use-after-free in Item_func_match::cleanup upon using FT search with partitioning.
by Sergei Golubchik 14 May '18

14 May '18

Hi, Alexey! On May 14, Alexey Botchkov wrote: > Hi Sergei, > > I was wrong - this fix is only needed in 10.3, since the FT keys > are available for PARTITION tables. > So going to push it to the 10.3. Is it ok with you? Yes, thanks! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 4861977ce76: MDEV-14695: Assertion `n < m_size' failed in Bounds_checked_array<Element_type>::operator
by Vicențiu Ciorbaru 14 May '18

14 May '18

Hi Varun! I will take a second look at this but a first minor comment is to not save two characters by shortening fields to flds, it makes reading unnatural. Vicentiu On Sat, 12 May 2018 at 21:45 Varun <varunraiko1803(a)gmail.com> wrote: > revision-id: 4861977ce76761fefb9779a774984b6afa818f2e > (mariadb-10.2.5-636-g4861977ce76) > parent(s): 8b26fea83572cf5c043721b7835c3828937f9c27 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-05-13 00:09:15 +0530 > message: > > MDEV-14695: Assertion `n < m_size' failed in > Bounds_checked_array<Element_type>::operator > > In this issue we hit the assert because we are adding addition fields to > the field JOIN::all_fields list. This > is done because HEAP tables can't index BIT fields so we need to use an > additional hidden field for grouping because later it will be > converted to a LONG field. Original field will remain of the BIT type and > will be returned. This happens when we convert DISTINCT to > GROUP BY. > > The solution is to take into account the number of such hidden fields that > would be added to the field JOIN::all_fields list while calculating > the size of the ref_pointer_array. > > --- > sql/sql_base.cc | 9 ++++++--- > sql/sql_base.h | 4 ++-- > sql/sql_delete.cc | 3 ++- > sql/sql_lex.cc | 8 +++++++- > sql/sql_lex.h | 5 +++++ > sql/sql_select.cc | 4 +++- > sql/sql_union.cc | 1 + > 7 files changed, 26 insertions(+), 8 deletions(-) > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 95ff41cb6f4..81ccfe5e514 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -6919,7 +6919,7 @@ static bool setup_natural_join_row_types(THD *thd, > > int setup_wild(THD *thd, TABLE_LIST *tables, List<Item> &fields, > List<Item> *sum_func_list, > - uint wild_num) > + uint wild_num, uint *hidden_bit_flds) > { > if (!wild_num) > return(0); > @@ -6960,7 +6960,7 @@ int setup_wild(THD *thd, TABLE_LIST *tables, > List<Item> &fields, > else if (insert_fields(thd, ((Item_field*) item)->context, > ((Item_field*) item)->db_name, > ((Item_field*) item)->table_name, &it, > - any_privileges)) > + any_privileges, hidden_bit_flds)) > { > if (arena) > thd->restore_active_arena(arena, &backup); > @@ -7425,7 +7425,7 @@ bool get_key_map_from_key_list(key_map *map, TABLE > *table, > bool > insert_fields(THD *thd, Name_resolution_context *context, const char > *db_name, > const char *table_name, List_iterator<Item> *it, > - bool any_privileges) > + bool any_privileges, uint *hidden_bit_flds) > { > Field_iterator_table_ref field_iterator; > bool found; > @@ -7545,6 +7545,9 @@ insert_fields(THD *thd, Name_resolution_context > *context, const char *db_name, > else > it->after(item); /* Add 'item' to the SELECT list. */ > > + if (item->type() == Item::FIELD_ITEM && item->field_type() == > MYSQL_TYPE_BIT) > + (*hidden_bit_flds)++; > + > #ifndef NO_EMBEDDED_ACCESS_CHECKS > /* > Set privilege information for the fields of newly created views. > diff --git a/sql/sql_base.h b/sql/sql_base.h > index d6063f1b771..8d0dcb2f197 100644 > --- a/sql/sql_base.h > +++ b/sql/sql_base.h > @@ -154,11 +154,11 @@ bool fill_record_n_invoke_before_triggers(THD *thd, > TABLE *table, > enum trg_event_type event); > bool insert_fields(THD *thd, Name_resolution_context *context, > const char *db_name, const char *table_name, > - List_iterator<Item> *it, bool any_privileges); > + List_iterator<Item> *it, bool any_privileges, uint > *hidden_bit_flds); > void make_leaves_list(THD *thd, List<TABLE_LIST> &list, TABLE_LIST > *tables, > bool full_table_list, TABLE_LIST *boundary); > int setup_wild(THD *thd, TABLE_LIST *tables, List<Item> &fields, > - List<Item> *sum_func_list, uint wild_num); > + List<Item> *sum_func_list, uint wild_num, uint * > hidden_bit_flds); > bool setup_fields(THD *thd, Ref_ptr_array ref_pointer_array, > List<Item> &item, enum_mark_columns mark_used_columns, > List<Item> *sum_func_list, List<Item> *pre_fix, > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > index 7b7a7e3f804..7bae7736cf3 100644 > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -760,7 +760,8 @@ l > select_lex->leaf_tables, FALSE, > DELETE_ACL, SELECT_ACL, TRUE)) > DBUG_RETURN(TRUE); > - if ((wild_num && setup_wild(thd, table_list, field_list, NULL, > wild_num)) || > + if ((wild_num && setup_wild(thd, table_list, field_list, NULL, wild_num, > + &select_lex->hidden_bit_flds)) || > setup_fields(thd, Ref_ptr_array(), > field_list, MARK_COLUMNS_READ, NULL, NULL, 0) || > setup_conds(thd, table_list, select_lex->leaf_tables, conds) || > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 3e36cac96b9..49150c863fe 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2143,6 +2143,7 @@ void st_select_lex::init_query() > m_agg_func_used= false; > window_specs.empty(); > window_funcs.empty(); > + hidden_bit_flds= 0; > } > > void st_select_lex::init_select() > @@ -2673,6 +2674,10 @@ ulong st_select_lex::get_table_join_options() > > bool st_select_lex::setup_ref_array(THD *thd, uint order_group_num) > { > + > + if (!((options & SELECT_DISTINCT) && !group_list)) > + hidden_bit_flds= 0; > + > // find_order_in_list() may need some extra space, so multiply by two. > order_group_num*= 2; > > @@ -2687,7 +2692,8 @@ bool st_select_lex::setup_ref_array(THD *thd, uint > order_group_num) > select_n_reserved + > select_n_having_items + > select_n_where_fields + > - order_group_num) * 5; > + order_group_num + > + hidden_bit_flds) * 5; > if (!ref_pointer_array.is_null()) > { > /* > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index 6d281930d1f..8283496604e 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -956,6 +956,11 @@ class st_select_lex: public st_select_lex_node > > /* it is for correct printing SELECT options */ > thr_lock_type lock_type; > + /* > + it counts the number of bit fields in the SELECT list. These are used > when DISTINCT is > + converted to a GROUP BY involving BIT fields. > + */ > + uint hidden_bit_flds; > > void init_query(); > void init_select(); > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 4cd4754596b..83953661ce3 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -799,7 +799,9 @@ JOIN::prepare(TABLE_LIST *tables_init, > select_lex != select_lex->master_unit()->global_parameters()) > real_og_num+= select_lex->order_list.elements; > > - if (setup_wild(thd, tables_list, fields_list, &all_fields, wild_num)) > + DBUG_ASSERT(select_lex->hidden_bit_flds == 0); > + if (setup_wild(thd, tables_list, fields_list, &all_fields, wild_num, > + &select_lex->hidden_bit_flds)) > DBUG_RETURN(-1); > if (select_lex->setup_ref_array(thd, real_og_num)) > DBUG_RETURN(-1); > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index 3fe8a24f8bd..42904e341b6 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -1474,6 +1474,7 @@ bool st_select_lex::cleanup() > } > inner_refs_list.empty(); > exclude_from_table_unique_test= FALSE; > + hidden_bit_flds= 0; > DBUG_RETURN(error); > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] MDEV-6111: Optimizer trace: features and milestones
by Sergey Petrunia 11 May '18

11 May '18

Hi Zhzhzoo, Please find below some thoughts about features/milestones in the project. Items at the top of the list should be done first. The list is not necesarrily complete, any comments/additions are welcome. - Optimizer trace collection API (is it different from Json_writer that we have? Does MySQL use the same API to produce EXPLAIN FORMAT=JSON and produce optimizer_trace? Should we follow them?) - Basic optimizer trace collection (just print the query and "hello":"world" instead of trace). Clear the trace data at query start, collect the trace if tracing is switched on (and do not collect if it is off) - INFORMATION_SCHEMA.OPTIMIZER_TRACE table in I_S database. Produce optimizer_trace content for features common with MySQL: = condition rewrite = partition pruning = range analysis = ref/eq_ref analysis (update_ref_and_keys) = (TODO what else?) Produce optimizer_trace content for MariaDB-specific features: = merged semi-join conversion = non-merged semi-join conversion = join condition pushdown = (TODO what else?) - optimizer_trace_limit/optimizer_trace_offset (without this feature, just save one last trace) - Ability to produce either a pretty-printed or one-line optimizer trace - Security = INFORMATION_SCHEMA.OPTIMIZER_TRACE.INSUFFICIENT_PRIVILEGES = Who can view optimizer_trace? = Figure out and implement the necessary security restrictions - Memory limits = Honor the @@optimizer_trace_max_mem_size setting. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Please review MDEV-16134 Wrong I_S.COLUMNS.CHARACTER_XXX_LENGTH value for compressed columns
by Alexander Barkov 10 May '18

10 May '18

Hi Sergey, Please review a patch for MDEV-16134. Thanks!

1 0

[Maria-developers] MDEV-15935: Connection Redirection Mechanism in MariaDB Client/Server Protocol
by xiangyu hu 09 May '18

09 May '18

Hi developers, There is an ongoing discussion regarding to the proposal of connection redirection in JIRA. Please kindly take a look if you aren't aware of it. Link is: https://jira.mariadb.org/browse/MDEV-15935. Thank you, Xiangyu

1 0

[Maria-developers] MDEV-15450: Add Host Name Field to MariaDB Client Handshake Protocol
by xiangyu hu 09 May '18

09 May '18

Hello devs, We have initiated a proposal in JIRA where we described the necessity of adding host_name into login packet attributes. I would like to broaden the audience here to bring up more discussions. Please take a look at MDEV-15450 (https://jira.mariadb.org/browse/MDEV-15450) [MDEV-15450] Add Host Name Field to MariaDB Client ...<https://jira.mariadb.org/browse/MDEV-15450> jira.mariadb.org Why. In original, one MariaDB server instance is identified by one address (IP + port). That is not enough for a shared proxy for multi servers. Thanks, Xiangyu

1 0

Re: [Maria-developers] eb3e9cd: MDEV-15813 ASAN use-after-poison in hp_hashnr upon HANDLER READ on a versioned HEAP table.
by Sergei Golubchik 09 May '18

09 May '18

Hi, Alexey! On May 08, Alexey Botchkov wrote: > revision-id: eb3e9cdbe86b5a68c3ec8a9f26165259e4ca7eec (mariadb-10.3.6-125-geb3e9cd) > parent(s): bd09c5ca866e273b6cebbd9a15c51c82bfa3ac9b > committer: Alexey Botchkov > timestamp: 2018-05-08 14:19:55 +0400 > message: > > MDEV-15813 ASAN use-after-poison in hp_hashnr upon HANDLER READ on a versioned HEAP table. > > Returns an error if we try to use a key in a query it declared unable to > handle. > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index c55ac4f..7446a60 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7915,3 +7915,5 @@ ER_UPDATED_COLUMN_ONLY_ONCE > eng "The column %`s.%`s cannot be changed more than once in a single UPDATE statement" > ER_EMPTY_ROW_IN_TVC > eng "Row with no elements is not allowed in table value constructor in this context" > +ER_KEY_DOESNT_SUPPORT > + eng "%s index %`s does not support this operation" I think this can be safely pushed into 10.2. But not earlier, we can only add new error messages in the latest GA version, not in previos GA's. > diff --git a/sql/sql_handler.cc b/sql/sql_handler.cc > index 09883d8..12e4b5a 100644 > --- a/sql/sql_handler.cc > +++ b/sql/sql_handler.cc > @@ -661,12 +661,30 @@ mysql_ha_fix_cond_and_key(SQL_HANDLER *handler, > key_part_map keypart_map; > uint key_len; > > + if (ha_rkey_mode != HA_READ_KEY_EXACT && > + (table->file->index_flags(handler->keyno, 0, TRUE) & > + (HA_READ_NEXT | HA_READ_PREV | HA_READ_RANGE)) == 0) I suspect there can be hypothetical cases where an index supports some specific ha_rkey_mode values and has some specific flags, but not all of them, like you test here. But as the only real use case is HASH vs BTREE now, your check is fine. No need overengineer a complex test that we cannot possibly test now. > + { > + my_error(ER_KEY_DOESNT_SUPPORT, MYF(0), > + table->file->index_type(handler->keyno), keyinfo->name); > + return 1; > + } > + > if (key_expr->elements > keyinfo->user_defined_key_parts) > { > my_error(ER_TOO_MANY_KEY_PARTS, MYF(0), > keyinfo->user_defined_key_parts); > return 1; > } > + else if (key_expr->elements < keyinfo->user_defined_key_parts && > + (table->file->index_flags(handler->keyno, 0, TRUE) & > + HA_ONLY_WHOLE_INDEX)) > + { > + my_error(ER_KEY_DOESNT_SUPPORT, MYF(0), > + table->file->index_type(handler->keyno), keyinfo->name); > + return 1; Good point! > + } This is ok to push. Just two questions: 1. Why did you add it in sql_handler.cc and not in the class handler method, like ha_index_read_map or ha_index_next ? 2. Could you add tests for fulltext/rtree indexes too, please? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] GSoC 2018 - MDEV-12313 - 30th of April -> 6th of May Update
by Teodor-Vicentiu Niculescu 08 May '18

08 May '18

Greetings, First of all, I want to thank you for accepting me for this years GSoC. I plan to do my best during the following months in order to implement the SQL Statement specified in the proposal. https://docs.google.com/document/d/1dA7YlDPlRG43LhfGlHXnHJiERfjExLG9qS7BqXh… In the last week (30th of April to 6th of May) I have started the first task of the proposal, specifically extending the grammar. After receiving some initial hints from Mr. Ciorbaru, concerning where to start looking in the files ( ANALYZE_SYM in sql/sql_yacc.yy ), I quickly discovered I have no knowledge of GNU Bison. I took my time to look through the GNU Bison online documentation, I have compiled tests and written code in order to better understand parsing. Now, sql_yacc.yy is starting to make sense, which is why I plan to finish the parsing part of the project by Sunday (13th of May). Until 15th of June my schedule will be quite busy because finals are approaching. You can still contact me by e-mail. Advice and constructive criticism is always much appreciated. Thank you for this amazing opportunity. Sincerely, Teodor

1 0

Re: [Maria-developers] fa0b71b0e19: MDEV-10259 mysqld crash with certain statement length and...
by Sergei Golubchik 08 May '18

08 May '18

Hi, Sachin! On May 08, Sachin Setiya wrote: > diff --git a/unittest/sql/mf_iocache-t.cc b/unittest/sql/mf_iocache-t.cc > index 31f98562521..d5e2ffb095e 100644 > --- a/unittest/sql/mf_iocache-t.cc > +++ b/unittest/sql/mf_iocache-t.cc > @@ -187,10 +187,71 @@ void mdev9044() > close_cached_file(&info); > } > > +/* 2 Reads in cache makes second read to fail (only if first read uses temp file) */ > +void mdev10259() > +{ > + int res; > + uchar buf[CACHE_SIZE + 200]; > + memset(buf, FILL, sizeof(buf)); > + > + diag("MDEV-10259- mysqld crash with certain statement length and order with" > + " Galera and encrypt-tmp-files=1"); > + > + init_io_cache_encryption(); > + > + res= open_cached_file(&info, 0, 0, CACHE_SIZE, 0); > + ok(res == 0, "open_cached_file" INFO_TAIL); > + > + res= my_b_write(&info, buf, sizeof(buf)); > + ok(res == 0 && info.pos_in_file == CACHE_SIZE, "large write" INFO_TAIL); > + > + res= my_b_flush_io_cache(&info, 1); > + ok(res == 0, "flush" INFO_TAIL); I think this misses the point. The bug happened when the data is in the cache, and there's no file to read from, so when the data is lost - it's forever. But my_b_flush_io_cache() writes the buffer to a file, so my_b_fill() can read it even if the buffer is emptied. Meaning, I suspect that if you remove your fix from mf_iocache_encr.cc, this unit test will still pass. Please double-check that your test fails without a fix. Ok to push after that. > + res= reinit_io_cache(&info, READ_CACHE, 0, 0, 0); > + ok(res == 0, "reinit READ_CACHE" INFO_TAIL); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 7107565: MDEV-10259 mysqld crash with certain statement length and...
by Sergei Golubchik 08 May '18

08 May '18

Hi, Sachin! Great work! On Mar 27, Sachin Setiya wrote: > revision-id: 710756527ec8589bce33bc834a1d8e6604266b6d (mariadb-10.1.31-47-g7107565) > parent(s): 8152c52e1af608da5d288a4e6fc82e2d804e64cd > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2018-03-27 04:24:57 +0530 > message: > > MDEV-10259 mysqld crash with certain statement length and... > order with Galera and encrypt-tmp-files=1 > > Problem:- If trans_cache (IO_CACHE) uses encrypted tmp file > then on next DML server will crash. > > Case:- > Lets take a case , we have a table t1 , We try to do 2 inserts in t1 > 1. A really long insert so that trans_cache has to use temp_file > 2. Just a small insert > > Analysis:- Actually server crashes from inside of galera > library. > /lib64/libc.so.6(abort+0x175)[0x7fb5ba779dc5] > /usr/lib64/galera/libgalera_smm.so(_ZN6galera3FSMINS_9TrxHandle5State... > mysys/stacktrace.c:247(my_print_stacktrace)[0x7fb5a714940e] > sql/signal_handler.cc:160(handle_fatal_signal)[0x7fb5a715c1bd] > sql/wsrep_hton.cc:257(wsrep_rollback)[0x7fb5bcce923a] > sql/wsrep_hton.cc:268(wsrep_rollback)[0x7fb5bcce9368] > sql/handler.cc:1658(ha_rollback_trans(THD*, bool))[0x7fb5bcd4f41a] > sql/handler.cc:1483(ha_commit_trans(THD*, bool))[0x7fb5bcd4f804] > > but actual issue is not in galera but in mariadb, because for 2nd > insert we should never call rollback. We are calling rollback because > log_and_order fails it fails because write_cache fails , It fails > because after reinit_io_cache(trans_cache) , my_b_tell says 0 so we > look into tmp_file for data , which is obviously wrong since temp > was used for previous insert and it no longer exist. Add here something like: wsrep_write_cache_inc() reads the IO_CACHE in a loop, filling it with my_b_fill() until it returns "0 bytes read". Later MYSQL_BIN_LOG::write_cache() does the same. wsrep_write_cache_inc() assumes that reading a zero bytes past EOF leaves the old data in the cache. Let's do that. > diff --git a/sql/mf_iocache_encr.cc b/sql/mf_iocache_encr.cc > index 149e6fe..41e1333 100644 > --- a/sql/mf_iocache_encr.cc > +++ b/sql/mf_iocache_encr.cc > @@ -49,8 +49,7 @@ static int my_b_encr_read(IO_CACHE *info, uchar *Buffer, size_t Count) > > if (pos_in_file == info->end_of_file) > { here, a comment: /* reading past EOF should not empty the cache */ > - info->read_pos= info->read_end= info->buffer; > - info->pos_in_file= pos_in_file; > + info->read_pos= info->read_end; > info->error= 0; > DBUG_RETURN(MY_TEST(Count)); > } and add a test for it into unittest/sql/mf_iocache-t.cc. Something simple, like, write into a cache, reinit to READ, fill, fill, reinit to READ, verify the data is still there. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] bea915f: MDEV-15965 Invisible columns and LOAD DATA don't work well together:...
by Sergei Golubchik 07 May '18

07 May '18

Hi, sachin! On Apr 26, sachin wrote: > revision-id: bea915f21a2a2771ce7f9e19dcb9d6893f33f7bd (mariadb-10.3.6-47-gbea915f) > parent(s): 326db1a2aaa9b275a1a21a863e8cd2d9fa1b1d5f > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2018-04-26 16:49:27 +0530 > message: > > MDEV-15965 Invisible columns and LOAD DATA don't work well together:... > ER_WARN_TOO_FEW_RECORDS > > Fix mysql_load iterator to skip invisible fields. > > diff --git a/sql/sql_load.cc b/sql/sql_load.cc > index cfa92f1..fbb3926 100644 > --- a/sql/sql_load.cc > +++ b/sql/sql_load.cc > @@ -444,6 +444,8 @@ int mysql_load(THD *thd, const sql_exchange *ex, TABLE_LIST *table_list, > field_iterator.set(table_list); > for (; !field_iterator.end_of_fields(); field_iterator.next()) > { > + if (field_iterator.field()->invisible > VISIBLE) > + continue; Oh, that's all? And the syntax with SET just worked automatically? Cool. ok to push! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 326db1a: Mdev-14853 Grant does not work correctly when table contains...
by Sergei Golubchik 07 May '18

07 May '18

Hi, Sachin! On Apr 26, sachin wrote: > revision-id: 326db1a2aaa9b275a1a21a863e8cd2d9fa1b1d5f (mariadb-10.3.6-46-g326db1a) > parent(s): 9477a2a9ba17c0db362e2bb39d5048e369096f39 > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2018-04-26 12:47:25 +0530 > message: > > Mdev-14853 Grant does not work correctly when table contains... > SYSTEM_INVISIBLE or COMPLETELY_INVISIBLE > > This commit does multiple things to solve this mdev > 1st add field into the parameter of check_column_grant_in_table_ref, so that > we can find out field invisibility. > 2nd If field->invisible >= INVISIBLE_SYSTEM skip access check and simple > grant access. Looks ok. A couple of comments about the test: 1. I suppose now you can use system versioning to test system invisible fields, so there's no need to limit this test to debug-only builds. 2. please test a case when a user has *no* privileges on the table at all. To make sure he cannot select always-readable system invisible fields. and change the capitalization in the commit comment to say MDEV not Mdev - I suspect this was the reason why Jira didn't pick up your commit. Thanks! Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 5651e72: MDEV-15465 Server crash or ASAN heap-use-after-free in Item_func_match::cleanup upon using FT search with partitioning.
by Sergei Golubchik 07 May '18

07 May '18

Hi, Alexey! On May 07, Alexey Botchkov wrote: > Yes, it is an issue in 5.5. > But we don't always fix bugs in the earliest version possible. > > Ok. Do i understand it right that the patch is ok to push into 5.5? Yes. > Should i then propagate the patch upt to 10.3 muself? No, don't bother. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 5651e72: MDEV-15465 Server crash or ASAN heap-use-after-free in Item_func_match::cleanup upon using FT search with partitioning.
by Sergei Golubchik 06 May '18

06 May '18

Hi, Alexey! On May 03, Alexey Botchkov wrote: > revision-id: 5651e72b8dd20e29a4963723773ec6bcfac457ba (mariadb-10.3.6-102-g5651e72) > parent(s): 73a10cbcc5178ae5378abb821428d35d3276b4da > committer: Alexey Botchkov > timestamp: 2018-05-03 12:42:56 +0400 > message: > > MDEV-15465 Server crash or ASAN heap-use-after-free in Item_func_match::cleanup upon using FT search with partitioning. > > The Item_func_match::cleanup() uses table's internals so it's not safe > to call it after the close_thread_tables(). Let's call it in > st_select_lex::cleanup(). Oops. Agree, thanks. But why do you fix it in 10.3? Looks like a 5.5+ bug. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] cc0bc69: MDEV-15813 ASAN use-after-poison in hp_hashnr upon HANDLER READ on a
by Sergei Golubchik 06 May '18

06 May '18

Hi, Alexey! On May 06, Alexey Botchkov wrote: > revision-id: cc0bc69f2d28aace12487f930677abcdffa5ce97 (mariadb-10.3.6-102-gcc0bc69) > parent(s): 73a10cbcc5178ae5378abb821428d35d3276b4da > committer: Alexey Botchkov > timestamp: 2018-05-06 02:37:20 +0400 > message: > > MDEV-15813 ASAN use-after-poison in hp_hashnr upon HANDLER READ on a > versioned HEAP table. > > hp_rkey now returns an error if we try to use HASH key where it doesn't > work. > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index c55ac4f..fcd20d8 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7915,3 +7915,5 @@ ER_UPDATED_COLUMN_ONLY_ONCE > eng "The column %`s.%`s cannot be changed more than once in a single UPDATE statement" > ER_EMPTY_ROW_IN_TVC > eng "Row with no elements is not allowed in table value constructor in this context" > +ER_KEY_DOESNT_SUPPORT > + eng "HASH index %`s does not support this operation" Use "%s index %`s does not support this operation" please. First parameter could be taken from handler::index_type(). By the way, check what happens when the index is a fulltext or rtree. Is it also a crash or this case is handled somehow? If it does, the same check should take care of HASH indexes, no need to do a separate solution for them. > diff --git a/storage/heap/hp_rkey.c b/storage/heap/hp_rkey.c > index 692e1a0..a7b81ae 100644 > --- a/storage/heap/hp_rkey.c > +++ b/storage/heap/hp_rkey.c > @@ -60,6 +60,14 @@ int heap_rkey(HP_INFO *info, uchar *record, int inx, const uchar *key, > } > else > { > + /* > + HASH key doesn't support search other than equality, > + and doesn't support prefix search. > + */ > + if (find_flag != HA_READ_KEY_EXACT || > + (1UL << share->keydef[inx].keysegs) != keypart_map + 1) > + DBUG_RETURN(my_errno= HA_ERR_KEY_DOESNT_SUPPORT); > + No, please, not here. The parent handler object knows index types, it knows index flags and it can return an error if the corresponding index flag is missing. No need to push this check down to the engine if the server can do it in, say, ha_index_read. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-15813 ... HANDLER READ on a versioned HEAP table
by Sergei Golubchik 05 May '18

05 May '18

Hi, Alexey! On May 04, Alexey Botchkov wrote: > Hi, Sergei, guys. > > This bug iis more like a feature of the HASH indexes. > And i came upon other 'features' while experimenting. > > Firstly the 'range' queries like > HANDLER h READ `PRIMARY` < (5); > cannot be naturally handled by the HASH keys. Only equality queries > ...'PRIMARY' = (5); make sence. > And these 'range' queries just return wrong empty result. > I'd say we need to fire an error in this case. Do you agree? Yes. Like "HASH index `PRIMARY` does not support this operation" ? > Second problem - the SYSTEM VERSIONING. > It adds an extra column to the key, which makes this key virtually > useless. I think for the VERSIONING we should either replace the HASH > key with the RBTREE, or get the row_end data from the table after > getting records from the HASH. > What is your idea how to deal with this? row_end must be a part of every unique key, because uniqueness of current data should not be affected by historical, deleted, rows. In this particular use case, HANDLER is a low level command, it doesn't see high-level features like versioning. So, for HANDLER row_end is just a normal key part, should be used as HANDLER h READ `PRIMARY` = (5, '2038-01-19 03:14:07.999999'); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] : GSoC Project Optimizer Trace
by Sergey Petrunia 04 May '18

04 May '18

Hi Zhzhzoo, Please find below some thoughts about the optimizer trace project * The MDEV for the optimizer trace task is https://jira.mariadb.org/browse/MDEV-6111. Let's keep it up to date with all the changes. * I think we should target MySQL-like functionality of optimizer_trace. At least, for the first milestone. We may decide to make some things better but working in a similar way to MySQL is a nice first goal. * MariaDB's optimizer trace will contain JSON and produce similar content to MySQL. However, MariaDB's optimizer is different, so one cannot expect that the JSON document will have exactly the same fields/objects/etc. (MariaDB' EXPLAIN FORMAT=JSON output doesn't match the one in MySQL, either. And in MySQL, the output varies across its versions.) * MySQL prints conditions with heavy over-quoting. Example: "original_condition": "((`A`.`a` = `B`.`a`) and ((`A`.`a` + `B`.`a`) < 10))", MariaDB has the code to print the condition without redundant quotes (see how EXPLAIN FORMAT=JSON does it). * I assume MySQL has some class for printing well-formed JSON documents. MariaDB also has one. It is in sql/my_json_writer.h, see class Json_writer. It handles printing into JSON, and also does pretty-printing. When I looked at the code last time, it did pretty-printing better than MySQL's code for EXPLAIN FORMAT=JSON did. It would be nice to use Json_writer for collecting optimizer trace output, too. (there is a caveat: Json_writer was written specifically for producing EXPLAIN FORMAT=JSON output so it may lack some features. You may need to extend it). BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB Developers Unconference in June
by Ian Gilfillan 03 May '18

03 May '18

Hi A reminder about the next MariaDB Developers Unconference, which will be held in Tampera, Finland, from 26 June to 29 June. This will be a four-day event; the first day focusing on new contributors, the 2nd and 3rd the regular Unconference sessions and discussions, and the 4th day on getting patches reviewed and implemented, taking advantage of the chance to have in-person discussions. Details, as well as where to register and how to suggest sessions, are at https://mariadb.org/2018-2-developers-unconference-in-finland/

1 0

Re: [Maria-developers] [External] Re: Questions regarding closing partitions in MDEV-11084
by Mattias Jonsson 30 Apr '18

30 Apr '18

Hi Holyfoot, Sergei and Jacob, Thank you for looking into this. Jacob there are no Jira ticket (that I am aware of) also I do see it as two different problems with the same source. 1) Spider engine does not handle opening/closing specific partitions (as I understand it currently relies on opening/closing all partitions at once in order). Currently this is a crashing bug! 2) MDEV-11084 makes explicit partition selection work like FLUSH TABLE, due to the closing partitions part. I think it would be nice if you could create tickets for those two issues (or at least for the first one). Regards Mattias On Thu, Apr 26, 2018 at 11:26 AM, Alexey Botchkov <holyfoot(a)mariadb.com> wrote: > No, there's no JIRA task for that problem. > So i guess you can create one :) > > Best regards. > HF > > On Thu, Apr 26, 2018 at 4:50 AM, Jacob Mathew <jacob.mathew(a)mariadb.com> > wrote: > >> Hi Mattias and Holyfoot, >> >> Is there a Jira bug for this problem? If not I can create a bug for it >> and assign it to Holyfoot. >> >> Thanks, >> Jacob >> >> Jacob B. Mathew >> Spider and Server Developer >> MariaDB Corporation >> +1 408 655 8999 (mobile) >> jacob.b.mathew (Skype) >> jacob.mathew(a)MariaDB.com >> >> >> On Tue, Apr 17, 2018 at 9:11 AM, Mattias Jonsson < >> mattias.jonsson(a)booking.com> wrote: >> >>> Hi Holyfoot, >>> >>> On Mon, Apr 16, 2018 at 3:38 PM, Alexey Botchkov <holyfoot(a)askmonty.org> >>> wrote: >>> > Hi, Mattias, guys! >>> > >>> > While investigating the crash, i'd like to discuss that >>> > >>> >> it seems to close partitions whenever it >>> >> is not used in a statement (i.e. require it to be reopened in the next >>> >> statement that would use another partition >>> > >>> > >>> > Yes, it does that, handling statements with the specified 'PARTITION' >>> > option. >>> > The patch supposed to solve the problem when there are too many >>> partitions >>> > opened, >>> > so i think it must close the unused partitions sometime. >>> > No, it doesn't have to happen that often. I planned to check the >>> > table_open_cache >>> > variable before the forced close. But decided not to do that initially >>> - as >>> > it simplified testing, >>> > and i thought if someone uses the PARTITION option, he would stick to >>> using >>> > this partition >>> > anyway. And i forgot about that issue. >>> >>> The reason for not closing partitions in this case is that it turns >>> the 'SELECT col FROM t PARTITION(p)' almost into a 'FLUSH TABLES t' >>> which is kind of unexpected. >>> I did a test mixing PK selects with and without 'PARTITION (p)' clause >>> and that shows it will close all but 1 partition and then (re)open all >>> but 1 partition. >>> >>> Think of the case when a server runs in production with a heavily >>> partitioned table serving simple PK queries and then someone runs a >>> query with explicit PARTITION selection, then it will introduce a >>> short stall. First for the query itself (closing all but 1 partitions) >>> and then for the next simple PK query using the same table (opening >>> all but 1 partition). >>> >>> As I read the bug report: the reporter wants to avoid opening all >>> partitions. Not that it keeps the partitions open in the table open >>> cache (which is an issue on the architectural level of partitioning >>> not really fitting into the open table cache). >>> >>> I attached a diff with the test and results (I also added handler >>> status variables to show my point). The diff is against >>> b4a2baffa82e5c07b96a1c752228560dcac1359b. >>> >>> Here is the part of the result file that shows what I mean with extra >>> comments prepended by MJ>: >>> CREATE TABLE t1 (a int PRIMARY KEY) >>> ENGINE = InnoDB >>> PARTITION BY HASH (a) PARTITIONS 1000; >>> INSERT INTO t1 VALUES (0), (1), (2), (3); >>> FLUSH STATUS; >>> >>> SELECT a FROM t1 PARTITION(p0) WHERE a = 0; >>> a >>> 0 >>> SHOW SESSION STATUS >>> WHERE VARIABLE_NAME LIKE 'HANDLER_%' AND VALUE > 0; >>> Variable_name Value >>> MJ> Here it closes all but one partitions. >>> Handler_close 999 >>> >>> SELECT a FROM t1 WHERE a = 0; >>> a >>> 0 >>> SHOW SESSION STATUS >>> WHERE VARIABLE_NAME LIKE 'HANDLER_%' AND VALUE > 0; >>> Variable_name Value >>> Handler_close 1000 >>> Handler_external_lock 2006 >>> MJ> and here it re-opens those 'all but one' partitions. >>> Handler_open 999 >>> >>> >>> Regards >>> Mattias >>> > >>> > >>> > Best regards. >>> > HF >>> > >>> > >>> > >>> > >>> > 13.04.2018 19:07, Mattias Jonsson wrote: >>> >> >>> >> Hi MariaDB Devs, >>> >> >>> >> I tried to evaluate spider engine and found an issue where it crashes, >>> >> most likely due to MDEV-11084 (Stacktrace and reproducible test case >>> >> attached). >>> >> >>> >> That also leads me to wonder about the performance for partitioned >>> >> tables after MDEV-11084, when it seems to close partitions whenever it >>> >> is not used in a statement (i.e. require it to be reopened in the next >>> >> statement that would use another partition, effectively not using the >>> >> open table cache). >>> >> >>> >> I cannot see anything mentioned in the final commit message hinting on >>> >> that it closes partitions not used in the current query, but in the >>> >> previous patches it was mentioned without any reason. >>> >> >>> >> Why does it close the already opened partitions? >>> >> https://github.com/MariaDB/server/blob/10.3/sql/ha_partition.cc#L8365 >>> >> >>> >> I would not mind opening the partitions only when they are to be used >>> >> (even though there are engines that need to be tested more), but >>> >> closing them makes no sense to me performance wise. Also notice that >>> >> the partitions first will be put back into the open table cache and >>> >> then on the next query the non-used partitions will be closed and the >>> >> needed ones be (re)-opened. >>> >> >>> >> Regards >>> >> Mattias Jonsson >>> > >>> > >>> > >>> > >>> >> >> > -- Mattias Jonsson IT Team Lead - Development Booking.com B.V. Herengracht 597 Amsterdam 1017 CE Netherlands Direct +31207125646 [image: Booking.com] <http://www.booking.com/> The world's #1 accommodation site 43 languages, 198+ offices worldwide, 120,000+ global destinations, 1,550,000+ room nights booked every day No booking fees, best price always guaranteed Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)

3 4

Re: [Maria-developers] [Commits] 326db1a: Mdev-14853 Grant does not work correctly when table contains...
by Vicențiu Ciorbaru 26 Apr '18

26 Apr '18

Hi Sachin! Did you run this test on embedded server? Usually grant related tests require some form of not_embedded.inc include. (check roles suite for examples) Vicentiu On Thu, 26 Apr 2018 at 00:20 sachin <sachin.setiya(a)mariadb.com> wrote: > revision-id: 326db1a2aaa9b275a1a21a863e8cd2d9fa1b1d5f > (mariadb-10.3.6-46-g326db1a) > parent(s): 9477a2a9ba17c0db362e2bb39d5048e369096f39 > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2018-04-26 12:47:25 +0530 > message: > > Mdev-14853 Grant does not work correctly when table contains... > SYSTEM_INVISIBLE or COMPLETELY_INVISIBLE > > This commit does multiple things to solve this mdev > 1st add field into the parameter of check_column_grant_in_table_ref, so > that > we can find out field invisibility. > 2nd If field->invisible >= INVISIBLE_SYSTEM skip access check and simple > grant access. > > --- > mysql-test/main/invisible_field_grant.result | 111 > +++++++++++++++++++++++++++ > mysql-test/main/invisible_field_grant.test | 77 +++++++++++++++++++ > sql/sp_rcontext.cc | 9 ++- > sql/sql_acl.cc | 11 ++- > sql/sql_acl.h | 2 +- > sql/sql_base.cc | 4 +- > 6 files changed, 206 insertions(+), 8 deletions(-) > > diff --git a/mysql-test/main/invisible_field_grant.result > b/mysql-test/main/invisible_field_grant.result > new file mode 100644 > index 0000000..c3ccbb1 > --- /dev/null > +++ b/mysql-test/main/invisible_field_grant.result > @@ -0,0 +1,111 @@ > +set @old_debug= @@debug_dbug; > +create user user_1; > +show grants for user_1; > +Grants for user_1@% > +GRANT USAGE ON *.* TO 'user_1'@'%' > +# create user > +create database d; > +use d; > + > +#System_Invisible > +set debug_dbug= "+d,test_pseudo_invisible"; > +create table t1(a int); > +set debug_dbug=@old_debug; > +insert into t1 values(1); > +select a,invisible from t1; > +a invisible > +1 9 > +grant insert(a) on t1 to user_1; > +grant update(a) on t1 to user_1; > +grant select(a) on t1 to user_1; > +grant delete on t1 to user_1; > +connect con1, localhost, user_1,,test; > +connection con1; > +select user(); > +user() > +user_1@localhost > +use d; > +select * from t1; > +a > +1 > +insert into t1 values(2); > +select * from t1; > +a > +1 > +2 > +insert into t1(a) values(3); > +select * from t1; > +a > +1 > +2 > +3 > +select invisible,a from t1; > +invisible a > +9 1 > +9 2 > +9 3 > +delete from t1 where a =1; > +update t1 set a=1 where a=3; > +select * from t1; > +a > +2 > +1 > +disconnect con1; > + > +#Cleanup > +connection default; > +drop table t1; > +REVOKE ALL PRIVILEGES, GRANT OPTION FROM user_1; > + > +#Completely Invisible > +set debug_dbug= "+d,test_completely_invisible"; > +create table t1(a int); > +insert into t1 values(1); > +select a,invisible from t1; > +a invisible > +1 9 > +set debug_dbug=@old_debug; > +grant insert(a) on t1 to user_1; > +grant update(a) on t1 to user_1; > +grant select(a) on t1 to user_1; > +grant delete on t1 to user_1; > +connect con1, localhost, user_1,,test; > +connection con1; > +select user(); > +user() > +user_1@localhost > +use d; > +select * from t1; > +a > +1 > +insert into t1 values(2); > +select * from t1; > +a > +1 > +2 > +insert into t1(a) values(3); > +select * from t1; > +a > +1 > +2 > +3 > +select invisible,a from t1; > +ERROR 42S22: Unknown column 'invisible' in 'field list' > +delete from t1 where a =1; > +update t1 set a=1 where a=3; > +select * from t1; > +a > +2 > +1 > +disconnect con1; > + > +#Final Cleanup > +connection default; > +set debug_dbug= "+d,test_completely_invisible"; > +select a,invisible from t1; > +a invisible > +2 9 > +1 9 > +drop user user_1; > +drop database d; > +set @old_debug= @@debug_dbug; > diff --git a/mysql-test/main/invisible_field_grant.test > b/mysql-test/main/invisible_field_grant.test > new file mode 100644 > index 0000000..0d627e5 > --- /dev/null > +++ b/mysql-test/main/invisible_field_grant.test > @@ -0,0 +1,77 @@ > +--source include/have_debug.inc > +##TEST for invisible coloumn level 2 > +set @old_debug= @@debug_dbug; > +create user user_1; > +show grants for user_1; > +--echo # create user > +create database d; > +use d; > + > +--echo > +--echo #System_Invisible > +set debug_dbug= "+d,test_pseudo_invisible"; > +create table t1(a int); > +set debug_dbug=@old_debug; > +insert into t1 values(1); > +select a,invisible from t1; > +grant insert(a) on t1 to user_1; > +grant update(a) on t1 to user_1; > +grant select(a) on t1 to user_1; > +grant delete on t1 to user_1; > +connect (con1, localhost, user_1,,test); > +connection con1; > +select user(); > +use d; > +select * from t1; > +insert into t1 values(2); > +select * from t1; > +insert into t1(a) values(3); > +select * from t1; > +select invisible,a from t1; > +delete from t1 where a =1; > +update t1 set a=1 where a=3; > +select * from t1; > +disconnect con1; > +--source include/wait_until_disconnected.inc > + > +--echo > +--echo #Cleanup > +--connection default > +drop table t1; > +REVOKE ALL PRIVILEGES, GRANT OPTION FROM user_1; > + > +--echo > +--echo #Completely Invisible > +set debug_dbug= "+d,test_completely_invisible"; > +create table t1(a int); > +insert into t1 values(1); > +select a,invisible from t1; > +set debug_dbug=@old_debug; > +grant insert(a) on t1 to user_1; > +grant update(a) on t1 to user_1; > +grant select(a) on t1 to user_1; > +grant delete on t1 to user_1; > +connect (con1, localhost, user_1,,test); > +connection con1; > +select user(); > +use d; > +select * from t1; > +insert into t1 values(2); > +select * from t1; > +insert into t1(a) values(3); > +select * from t1; > +--error ER_BAD_FIELD_ERROR > +select invisible,a from t1; > +delete from t1 where a =1; > +update t1 set a=1 where a=3; > +select * from t1; > +disconnect con1; > +--source include/wait_until_disconnected.inc > +--echo > +--echo #Final Cleanup > +connection default; > +set debug_dbug= "+d,test_completely_invisible"; > +select a,invisible from t1; > +drop user user_1; > +drop database d; > +set @old_debug= @@debug_dbug; > diff --git a/sql/sp_rcontext.cc b/sql/sp_rcontext.cc > index 2e9ae23..dc103fa 100644 > --- a/sql/sp_rcontext.cc > +++ b/sql/sp_rcontext.cc > @@ -196,11 +196,12 @@ bool sp_rcontext::init_var_table(THD *thd, > */ > static inline bool > check_column_grant_for_type_ref(THD *thd, TABLE_LIST *table_list, > - const char *str, size_t length) > + const char *str, size_t length, > + Field *fld) > { > #ifndef NO_EMBEDDED_ACCESS_CHECKS > table_list->table->grant.want_privilege= SELECT_ACL; > - return check_column_grant_in_table_ref(thd, table_list, str, length); > + return check_column_grant_in_table_ref(thd, table_list, str, length, > fld); > #else > return false; > #endif > @@ -238,7 +239,7 @@ bool Qualified_column_ident::resolve_type_ref(THD > *thd, Column_definition *def) > { > if (!(rc= check_column_grant_for_type_ref(thd, table_list, > m_column.str, > - m_column.length))) > + m_column.length, src))) > { > *def= Column_definition(thd, src, NULL/*No defaults,no > constraints*/); > def->flags&= (uint) ~NOT_NULL_FLAG; > @@ -302,7 +303,7 @@ bool Table_ident::resolve_table_rowtype_ref(THD *thd, > LEX_CSTRING tmp= src[0]->field_name; > Spvar_definition *def; > if ((rc= check_column_grant_for_type_ref(thd, table_list, > - tmp.str, tmp.length)) || > + tmp.str, > tmp.length,src[0])) || > (rc= !(src[0]->field_name.str= thd->strmake(tmp.str, > tmp.length))) || > (rc= !(def= new (thd->mem_root) Spvar_definition(thd, *src)))) > break; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index de4e201..1c154a1 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -7775,6 +7775,8 @@ bool check_grant_column(THD *thd, GRANT_INFO *grant, > table_ref table reference where to check the field > name name of field to check > length length of name > + fld use fld object to check invisibility when it is > + not 0, not_found_field, view_ref_found > > DESCRIPTION > Check the access rights to a column depending on the type of table > @@ -7789,13 +7791,17 @@ bool check_grant_column(THD *thd, GRANT_INFO > *grant, > */ > > bool check_column_grant_in_table_ref(THD *thd, TABLE_LIST * table_ref, > - const char *name, size_t length) > + const char *name, size_t length, > + Field *fld) > { > GRANT_INFO *grant; > const char *db_name; > const char *table_name; > Security_context *sctx= table_ref->security_ctx ? > table_ref->security_ctx : thd->security_ctx; > + if (fld && fld != not_found_field && fld != view_ref_found > + && fld->invisible >= INVISIBLE_SYSTEM) > + return false; > > if (table_ref->view || table_ref->field_translation) > { > @@ -7871,6 +7877,9 @@ bool check_grant_all_columns(THD *thd, ulong > want_access_arg, > > for (; !fields->end_of_fields(); fields->next()) > { > + if (fields->field() && > + fields->field()->invisible >= INVISIBLE_SYSTEM) > + continue; > LEX_CSTRING *field_name= fields->name(); > > if (table_name != fields->get_table_name()) > diff --git a/sql/sql_acl.h b/sql/sql_acl.h > index a608ef0..6da7d4d 100644 > --- a/sql/sql_acl.h > +++ b/sql/sql_acl.h > @@ -239,7 +239,7 @@ bool check_grant_column (THD *thd, GRANT_INFO *grant, > const char *db_name, const char *table_name, > const char *name, size_t length, > Security_context *sctx); > bool check_column_grant_in_table_ref(THD *thd, TABLE_LIST * table_ref, > - const char *name, size_t length); > + const char *name, size_t length, > Field *fld); > bool check_grant_all_columns(THD *thd, ulong want_access, > Field_iterator_table_ref *fields); > bool check_grant_routine(THD *thd, ulong want_access, > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 0081365..383341c 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -5880,7 +5880,7 @@ find_field_in_table_ref(THD *thd, TABLE_LIST > *table_list, > #ifndef NO_EMBEDDED_ACCESS_CHECKS > /* Check if there are sufficient access rights to the found field. */ > if (check_privileges && > - check_column_grant_in_table_ref(thd, *actual_table, name, length)) > + check_column_grant_in_table_ref(thd, *actual_table, name, length, > fld)) > fld= WRONG_GRANT; > else > #endif > @@ -6057,7 +6057,7 @@ find_field_in_tables(THD *thd, Item_ident *item, > #ifndef NO_EMBEDDED_ACCESS_CHECKS > /* Check if there are sufficient access rights to the found field. > */ > if (found && check_privileges && > - check_column_grant_in_table_ref(thd, table_ref, name, length)) > + check_column_grant_in_table_ref(thd, table_ref, name, length, > found)) > found= WRONG_GRANT; > #endif > } > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 1

Re: [Maria-developers] Questions regarding closing partitions in MDEV-11084
by Alexey Botchkov 26 Apr '18

26 Apr '18

Hi, Mattias, guys! While investigating the crash, i'd like to discuss that > it seems to close partitions whenever it > is not used in a statement (i.e. require it to be reopened in the next > statement that would use another partition Yes, it does that, handling statements with the specified 'PARTITION' option. The patch supposed to solve the problem when there are too many partitions opened, so i think it must close the unused partitions sometime. No, it doesn't have to happen that often. I planned to check the table_open_cache variable before the forced close. But decided not to do that initially - as it simplified testing, and i thought if someone uses the PARTITION option, he would stick to using this partition anyway. And i forgot about that issue. Best regards. HF 13.04.2018 19:07, Mattias Jonsson wrote: > Hi MariaDB Devs, > > I tried to evaluate spider engine and found an issue where it crashes, > most likely due to MDEV-11084 (Stacktrace and reproducible test case > attached). > > That also leads me to wonder about the performance for partitioned > tables after MDEV-11084, when it seems to close partitions whenever it > is not used in a statement (i.e. require it to be reopened in the next > statement that would use another partition, effectively not using the > open table cache). > > I cannot see anything mentioned in the final commit message hinting on > that it closes partitions not used in the current query, but in the > previous patches it was mentioned without any reason. > > Why does it close the already opened partitions? > https://github.com/MariaDB/server/blob/10.3/sql/ha_partition.cc#L8365 > > I would not mind opening the partitions only when they are to be used > (even though there are engines that need to be tested more), but > closing them makes no sense to me performance wise. Also notice that > the partitions first will be put back into the open table cache and > then on the next query the non-used partitions will be closed and the > needed ones be (re)-opened. > > Regards > Mattias Jonsson

3 2

Re: [Maria-developers] 331034cb6e1: MDEV-11975: SQLCOM_PREPARE of EXPLAIN & ANALYZE statement do not return correct metadata info
by Sergei Golubchik 25 Apr '18

25 Apr '18

Hi, Oleksandr! On Apr 20, Oleksandr Byelkin wrote: > revision-id: 331034cb6e1552e1ab7c12b6f7341f557f76b807 (mariadb-10.3.6-16-g331034cb6e1) > parent(s): cff60be7fe159fdcb2517ce8441610ad512aa7d0 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-04-20 14:40:14 +0200 > message: > > MDEV-11975: SQLCOM_PREPARE of EXPLAIN & ANALYZE statement do not return correct metadata info > > Added metadate info after prepare EXPLAIN/ANALYZE. > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 96485ed15ec..0ec95e11829 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -2659,18 +2659,32 @@ CHANGED_TABLE_LIST* THD::changed_table_dup(const char *key, size_t key_length) > } > > > -int THD::send_explain_fields(select_result *result, uint8 explain_flags, bool is_analyze) > +void THD::prepare_explain_fields(select_result *result, > + List<Item> *field_list, > + uint8 explain_flags, > + bool is_analyze) > { > - List<Item> field_list; > + ; forgot to remove? otherwise, ok to push... Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-13695: INTERSECT precedence is not in line with Oracle even in SQL_MODE=Oracle
by Sergei Golubchik 25 Apr '18

25 Apr '18

Hi, Oleksandr! On Apr 24, Oleksandr Byelkin wrote: > revision-id: cb6aade130b75f3dc4b7c3a1dd4d33864adffd90 (mariadb-10.3.6-43-gcb6aade130b) > parent(s): f79c5a658cc33a10d7744a748a4328254e2cbaf7 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-04-24 12:04:59 +0200 > message: > > MDEV-13695: INTERSECT precedence is not in line with Oracle even in SQL_MODE=Oracle > > Switch off automatic INTERSECT priority for ORACLE MODE ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] 10.3-MDEV-11953
by Alexander Barkov 25 Apr '18

25 Apr '18

Hello Igor and Sanja, In your branch you stopped to use the global function check_simple_select(). It's not used any more. Please remove it from sql_parse.h and sql_parse.cc. Thanks.

1 0

Re: [Maria-developers] 57673bc: MDEV-14024 PCRE2
by Sergei Golubchik 24 Apr '18

24 Apr '18

Hi, Alexey! On Apr 24, Alexey Botchkov wrote: > > But for mroonga you've just changed the include path. I was > > surprised that it was enough. Perhaps mroonga doesn't use pcre at > > all? > > No, mroonga doesn't use PCRE, just mentions in comfigure scripts. > As far as i can see. At least it compiles fine with my changes. If it doesn't use PCRE, could you remove those lines with the path at all? Would be less confusing. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 57673bc: MDEV-14024 PCRE2
by Sergei Golubchik 23 Apr '18

23 Apr '18

Hi, Alexey! On Apr 23, Alexey Botchkov wrote: > > >> diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > >> index 3d11a22..d3f52ed 100644 > >... > >> #define PCRE_STATIC 1 /* Important on Windows */ > >> -#include "pcre.h" /* pcre header file */ > >> +#define PCRE2_CODE_UNIT_WIDTH 8 > > > >you sure, you want it here and not, say, in one universally-included > >headers like my_config.h? You don't define it in other places now, so > >they include different function names and macros. > > Since it's the only place where we include the 'pcre2.h', it probably > makes sence to keep all pcre2-related stuff in one place. No, that was precisely my point - you included pcre2.h in many (more than one) files, but defined PCRE2_CODE_UNIT_WIDTH only here. Otherwise I wouldn't have asked :) > Still i moved the #define to the my_config.h. Right, thanks. > >> +if(EXISTS "${MYSQL_SOURCE_DIR}/pcre2") > >> + set(MYSQL_REGEX_INCLUDE_DIR "${MYSQL_SOURCE_DIR}/pcre2/src") > > > >eh, is that enough? > >there were a lot more changes in the server, than just correcting the > >path. > > Do you mean there's more to be done? Not sure i understand right. To make the server use pcre2 you had to change something in the sources, like, API function names, types names, etc. But for mroonga you've just changed the include path. I was surprised that it was enough. Perhaps mroonga doesn't use pcre at all? Otherwise how could've it compiled without you changing function names and types? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] sql_mode=oracle : Rollback on error
by jerome brauge 23 Apr '18

23 Apr '18

Hello Alexander, I've found one another difference between Mariadb and Oracle : oracle always rollback the active transaction if an error is raised, Maria leave the transaction active. Possible workaround in stored procedure: add an exit handler which do a rollback and resignal the error. What do you think about this ? Regards

4 7

[Maria-developers] Deprecate INT(M) in SHOW?
by Alexander Barkov 23 Apr '18

23 Apr '18

Hello, We have an optional length for integer data types: TINYINT(N) SMALLINT(N) MEDIUMINT(N) INT(N) BIGINT(N) The number given in the parentheses imposes no any limits on the supported range. So for example BIGINT(1) is still a full-featured data type that can store and retrieve any 64-bit number (btw, I find this confusing). If I declare a column without N, it's later displayed in SHOW CREATE TABLE with the default length, so: CREATE TABLE t1 (a INT) becomes CREATE TABLE t1 (a INT(11)); The length is really important only in two cases: 1. In combination with ZEROFILL. 2. When the explicitly specified length is larger than the default length for the data type. For example, INT(40). In this case on cast to string, it's converted to VARCHAR(40) rather than to VARCHAR(11). In all other cases the length does not matter and has no any visible effects. Does anybody see any problems if in 10.4 we fix all metadata statements, such as: - SHOW CREATE TABLE t1 - DESCRIBE t1 - SELECT * FROM INFORMATION_SCHEMA.COLUMNS not to print the default length for non-ZEROFILL columns?

4 5

Re: [Maria-developers] Analysis of the same name of field and name check constraints
by Anel Husakovic 23 Apr '18

23 Apr '18

Hi Vincentiu, Sorry but I have seen now, after sending the mail, that example in this mail is not relevant, please take a look in a example in PR. Regards, Anel On Mon, Apr 23, 2018 at 12:45 PM, Anel Husakovic <anel(a)mariadb.org> wrote: > Hi Vicentiu, > > Regarding the check constraints with *field *and *table *constraint that > have the same name for example: > *`create table t (a int, b check(b>0), constraint b check (b>10));`* > > when new rows are inserted and an error occurs, generated error is not > pointing to exact constraint. For example: > *`insert into t values (-1);`* > ERROR 4025 (23000): CONSTRAINT `b` failed for `test`.`t` > but this means different constraint. > > In order to solve this, here are my opinions: > 1. approach is not to allow this scenario while creating the table, > 2. approach is to generate different message one message for *table *constraint > second for *field *constraint depending where error occurs- and you told > me to go with this one. > > Out of curiosity I also tried with 1. approach problem is that in > *sql/sql_table.cc **mysql_prepare_create_table 4221* > *create/alter_info.check_constraint_list *is tied to table constraints. > However there is another attribute *field_check_constraints *for > *create_info *object but it is always *0*. It is updated in > * sql/unireg.cc* *783*. Problem is that *mysql_prepare_create_table *is > called first and after that *field_check_constraints *is updated,so > there is no mechanism to allow check in early stage for *table*-*field *check > constraint. > After that there is a validation according to the each constraint in > *sql/table.cc* *verify_constraints 5232* - and this is 2. approach. > > I finished 2. approach and what I have that each test already written in > main directory failed because of different error message for a specific > case. > So here is a patch for 2. approach: > https://github.com/MariaDB/server/pull/715 > > In this approach we are handling each error independently. > Also we need to change each test regarding the ER_CONSTRAINT_FAILED where > should be raised newly created error ER_FIELD_CONSTRAINT_FAILED. > > > Also I have updated PR *IS.check_constraint *, rebased everything in one > file > https://github.com/MariaDB/server/pull/689/ > > Best regards, > Anel > > > >

1 0

Re: [Maria-developers] [Commits] a20e1ce85ec: MDEV-13232: Assertion `(&(&share->intern_lock)->m_mutex)->count > 0 && pthread_equal(pthread_self(), (&(&share->intern_lock)->m_mutex)->thread)' failed in _ma_state_info_write
by Alexander Barkov 23 Apr '18

23 Apr '18

Hi Sanja, the patch is OK to push. Thanks. On 04/23/2018 12:37 PM, Oleksandr Byelkin wrote: > revision-id: a20e1ce85ec56616fa048e9ba39c25c65b098fce (mariadb-10.3.6-36-ga20e1ce85ec) > parent(s): c6ba758d1d41c11466b8f9b61b4546efc95aa689 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-04-23 10:35:33 +0200 > message: > > MDEV-13232: Assertion `(&(&share->intern_lock)->m_mutex)->count > 0 && pthread_equal(pthread_self(), (&(&share->intern_lock)->m_mutex)->thread)' failed in _ma_state_info_write > > Limit length of result of "negative" operation to something reasonable > > --- > mysql-test/main/union.result | 14 ++++++++++++++ > mysql-test/main/union.test | 15 +++++++++++++++ > sql/item_func.cc | 3 +++ > 3 files changed, 32 insertions(+) > > diff --git a/mysql-test/main/union.result b/mysql-test/main/union.result > index 4e5f9312e03..39cd68ae851 100644 > --- a/mysql-test/main/union.result > +++ b/mysql-test/main/union.result > @@ -2500,5 +2500,19 @@ t2 CREATE TABLE `t2` ( > DROP TABLE t2; > DROP TABLE t1; > # > +# MDEV-13232: Assertion `(&(&share->intern_lock)->m_mutex)->count > 0 && > +# pthread_equal(pthread_self(), (&(&share->intern_lock)->m_mutex)-> > +# thread)' failed in _ma_state_info_write > +# > +CREATE TABLE t1 (c1 CHAR(8)); > +INSERT INTO t1 VALUES ('10'),('-10'); > +CREATE TABLE t2 (c2 CHAR); > +SET @a= CAST('10' AS CHAR); > +SELECT c1 FROM t1 UNION SELECT - @a FROM t2; > +c1 > +10 > +-10 > +drop table t1,t2; > +# > # End of 10.3 tests > # > diff --git a/mysql-test/main/union.test b/mysql-test/main/union.test > index f86cae87524..b9b38271f83 100644 > --- a/mysql-test/main/union.test > +++ b/mysql-test/main/union.test > @@ -1745,6 +1745,21 @@ SHOW CREATE TABLE t2; > DROP TABLE t2; > DROP TABLE t1; > > +--echo # > +--echo # MDEV-13232: Assertion `(&(&share->intern_lock)->m_mutex)->count > 0 && > +--echo # pthread_equal(pthread_self(), (&(&share->intern_lock)->m_mutex)-> > +--echo # thread)' failed in _ma_state_info_write > +--echo # > + > +CREATE TABLE t1 (c1 CHAR(8)); > +INSERT INTO t1 VALUES ('10'),('-10'); > + > +CREATE TABLE t2 (c2 CHAR); > +SET @a= CAST('10' AS CHAR); > + > +SELECT c1 FROM t1 UNION SELECT - @a FROM t2; > + > +drop table t1,t2; > > --echo # > --echo # End of 10.3 tests > diff --git a/sql/item_func.cc b/sql/item_func.cc > index e2740272385..ae6c61e83f4 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -1935,6 +1935,9 @@ void Item_func_neg::fix_length_and_dec_double() > set_handler(&type_handler_double); > decimals= args[0]->decimals; // Preserve NOT_FIXED_DEC > max_length= args[0]->max_length + 1; > + // Limit length with something reasonable > + uint32 mlen= type_handler()->max_display_length(this); > + set_if_smaller(max_length, mlen); > unsigned_flag= false; > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

1 0

Re: [Maria-developers] ce222a7: MDEV-15915 Add Feature_json status variable.
by Sergei Golubchik 23 Apr '18

23 Apr '18

Hi, Alexey! On Apr 23, Alexey Botchkov wrote: > > Elena commented in MDEV about "Feature_json" being possibly confusing. > I've seen Elena's comment. Can also add GEOJSON functions to her list. > Still i can't figure out anything better than Feature_json as the name > for this variable. > The self-explaining name is either too long or possible misleading. Feature_json_functions ? But I think Feature_json is fine too, one hardly thinks of "EXPLAIN JSON" when talking about "JSON support". Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] ce222a7: MDEV-15915 Add Feature_json status variable.
by Sergei Golubchik 22 Apr '18

22 Apr '18

Hi, Alexey! On Apr 22, Alexey Botchkov wrote: > revision-id: ce222a71bf75b2125cc588371a3debb3e4a22d7f (mariadb-10.3.6-34-gce222a7) > parent(s): 9f84451d87ba1924539b7d2ab69619b71d0155b9 > committer: Alexey Botchkov > timestamp: 2018-04-22 16:01:18 +0400 > message: > > MDEV-15915 Add Feature_json status variable. > > --- > mysql-test/main/status.test | 7 +++++++ > sql/item_create.cc | 26 ++++++++++++++++++++++++++ > sql/mysqld.cc | 1 + > sql/sql_class.h | 1 + > 4 files changed, 35 insertions(+) > > diff --git a/sql/item_create.cc b/sql/item_create.cc > index 901dfa0..8b8476b 100644 > --- a/sql/item_create.cc > +++ b/sql/item_create.cc > @@ -5118,6 +5118,7 @@ Create_func_json_exists Create_func_json_exists::s_singleton; > Item* > Create_func_json_exists::create_2_arg(THD *thd, Item *arg1, Item *arg2) > { > + status_var_increment(current_thd->status_var.feature_json); > return new (thd->mem_root) Item_func_json_exists(thd, arg1, arg2); I believe you've added that to all JSON functions, I didn't try to double-check that :) > } > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 6e1a599..ae14c80 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -8553,6 +8553,7 @@ SHOW_VAR status_vars[]= { > {"Feature_fulltext", (char*) offsetof(STATUS_VAR, feature_fulltext), SHOW_LONG_STATUS}, > {"Feature_gis", (char*) offsetof(STATUS_VAR, feature_gis), SHOW_LONG_STATUS}, > {"Feature_invisible_columns", (char*) offsetof(STATUS_VAR, feature_invisible_columns), SHOW_LONG_STATUS}, > + {"Feature_json", (char*) offsetof(STATUS_VAR, feature_json), SHOW_LONG_STATUS}, > {"Feature_locale", (char*) offsetof(STATUS_VAR, feature_locale), SHOW_LONG_STATUS}, > {"Feature_subquery", (char*) offsetof(STATUS_VAR, feature_subquery), SHOW_LONG_STATUS}, > {"Feature_timezone", (char*) offsetof(STATUS_VAR, feature_timezone), SHOW_LONG_STATUS}, Elena commented in MDEV about "Feature_json" being possibly confusing. Does your using "Feature_json" name mean that * you considered it and intentionally kept the old name, thinking it's fine or * you didn't see the comment, and now you might reconsider? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 67260fd: MDEV-15130 Assertion `table->s->null_bytes == 0' failed in ...
by Sergei Golubchik 20 Apr '18

20 Apr '18

Hi, Sachin! On Apr 19, sachin wrote: > diff --git a/storage/perfschema/ha_perfschema.cc b/storage/perfschema/ha_perfschema.cc > index e8e9581..d5846f2 100644 > --- a/storage/perfschema/ha_perfschema.cc > +++ b/storage/perfschema/ha_perfschema.cc > @@ -64,6 +64,16 @@ static int pfs_discover_table(handlerton *hton, THD *thd, TABLE_SHARE *share) > { > const PFS_engine_table_share *pfs_share; > > + DBUG_EXECUTE_IF("test_pseudo_invisible", { > + my_error(ER_INTERNAL_ERROR, MYF(0), "Don't query performance_schema when" > + "test_pseudo_invisible is on"); > + return 1; > + }); > + DBUG_EXECUTE_IF("test_completely_invisible", { > + my_error(ER_INTERNAL_ERROR, MYF(0), "Don't query performance_schema when" > + "test_completely_invisible is on"); > + return 1; > + }); > if ((pfs_share= find_table_share(share->db.str, share->table_name.str))) > return share->init_from_sql_statement_string(thd, false, > pfs_share->sql.str, Better put it inside TABLE_SHARE::init_from_sql_statement_string to cover Connect and FederatedX too. Or - I'd say it's an option too - just close it as not a bug. DBUG_EXECUTE_IF can cause a crash, can make a server to misbehave, etc. That's normal, it's a debug hack that intentionally makes server to do something weird. I wouldn't spend too much time on making it user friendly and error-proof. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 346c8ab9533: MDEV-11975: SQLCOM_PREPARE of EXPLAIN & ANALYZE statement do not return correct metadata info
by Sergei Golubchik 20 Apr '18

20 Apr '18

Hi, Oleksandr! On Apr 18, Oleksandr Byelkin wrote: > revision-id: 346c8ab9533a3de6a4cb348428402ffee2aa8da2 (mariadb-10.3.6-16-g346c8ab9533) > parent(s): cff60be7fe159fdcb2517ce8441610ad512aa7d0 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-04-18 19:34:12 +0200 > message: > > MDEV-11975: SQLCOM_PREPARE of EXPLAIN & ANALYZE statement do not return correct metadata info > > Added metadate info after prepare EXPLAIN/ANALYZE. > > --- > mysql-test/main/mysql_client_test.result | 122 ++++++++++++ > mysql-test/main/mysql_client_test.test | 7 +- > sql/sql_class.cc | 18 +- > sql/sql_class.h | 3 +- > sql/sql_explain.cc | 3 +- > sql/sql_parse.cc | 5 +- > sql/sql_prepare.cc | 24 ++- > sql/sql_prepare.h | 2 + > tests/mysql_client_test.c | 323 ++++++++++++++++++++++++++++++- > 9 files changed, 493 insertions(+), 14 deletions(-) > > diff --git a/mysql-test/main/mysql_client_test.result b/mysql-test/main/mysql_client_test.result > index 83ef8d442b3..20385acfa00 100644 > --- a/mysql-test/main/mysql_client_test.result > +++ b/mysql-test/main/mysql_client_test.result > @@ -122,5 +122,127 @@ EOF > mysql_stmt_next_result(): 0; field_count: 0 > # ------------------------------------ > > +# cat MYSQL_TMP_DIR/test_explain_meta.out.log > +# ------------------------------------ okay, altought I'd rather add prepare metadata output to mysqltest. Like, either print prepare metadata on --enable_metadata or introduce a new command --enable_prepare_metadata > +SELECT number of fields: 1 > +EXPALIN number of fields: 10 > + - 0: name: 'id'/''; table: ''/''; db: ''; catalog: 'def'; length: 3; max_length: 0; type: 8; decimals: 0 > + - 1: name: 'select_type'/''; table: ''/''; db: ''; catalog: 'def'; length: 57; max_length: 0; type: 253; decimals: 39 > + - 2: name: 'table'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 3: name: 'type'/''; table: ''/''; db: ''; catalog: 'def'; length: 30; max_length: 0; type: 253; decimals: 39 > + - 4: name: 'possible_keys'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 5: name: 'key'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 6: name: 'key_len'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 7: name: 'ref'/''; table: ''/''; db: ''; catalog: 'def'; length: 6144; max_length: 0; type: 253; decimals: 39 > + - 8: name: 'rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 10; max_length: 0; type: 8; decimals: 0 > + - 9: name: 'Extra'/''; table: ''/''; db: ''; catalog: 'def'; length: 765; max_length: 0; type: 253; decimals: 39 > +EXPALIN JSON number of fields: 1 > + - 0: name: 'EXPLAIN'/''; table: ''/''; db: ''; catalog: 'def'; length: 234; max_length: 0; type: 253; decimals: 39 > +ANALYZE number of fields: 13 > + - 0: name: 'id'/''; table: ''/''; db: ''; catalog: 'def'; length: 3; max_length: 0; type: 8; decimals: 0 > + - 1: name: 'select_type'/''; table: ''/''; db: ''; catalog: 'def'; length: 57; max_length: 0; type: 253; decimals: 39 > + - 2: name: 'table'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 3: name: 'type'/''; table: ''/''; db: ''; catalog: 'def'; length: 30; max_length: 0; type: 253; decimals: 39 > + - 4: name: 'possible_keys'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 5: name: 'key'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 6: name: 'key_len'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 7: name: 'ref'/''; table: ''/''; db: ''; catalog: 'def'; length: 6144; max_length: 0; type: 253; decimals: 39 > + - 8: name: 'rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 10; max_length: 0; type: 8; decimals: 0 > + - 9: name: 'r_rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 10 > + - 10: name: 'filtered'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 2 > + - 11: name: 'r_filtered'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 2 > + - 12: name: 'Extra'/''; table: ''/''; db: ''; catalog: 'def'; length: 765; max_length: 0; type: 253; decimals: 39 > +ANALYZE JSON number of fields: 1 > + - 0: name: 'ANALYZE'/''; table: ''/''; db: ''; catalog: 'def'; length: 234; max_length: 0; type: 253; decimals: 39 > +EXPALIN INSERT number of fields: 10 > + - 0: name: 'id'/''; table: ''/''; db: ''; catalog: 'def'; length: 3; max_length: 0; type: 8; decimals: 0 > + - 1: name: 'select_type'/''; table: ''/''; db: ''; catalog: 'def'; length: 57; max_length: 0; type: 253; decimals: 39 > + - 2: name: 'table'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 3: name: 'type'/''; table: ''/''; db: ''; catalog: 'def'; length: 30; max_length: 0; type: 253; decimals: 39 > + - 4: name: 'possible_keys'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 5: name: 'key'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 6: name: 'key_len'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 7: name: 'ref'/''; table: ''/''; db: ''; catalog: 'def'; length: 6144; max_length: 0; type: 253; decimals: 39 > + - 8: name: 'rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 10; max_length: 0; type: 8; decimals: 0 > + - 9: name: 'Extra'/''; table: ''/''; db: ''; catalog: 'def'; length: 765; max_length: 0; type: 253; decimals: 39 > +EXPALIN JSON INSERT number of fields: 1 > + - 0: name: 'EXPLAIN'/''; table: ''/''; db: ''; catalog: 'def'; length: 234; max_length: 0; type: 253; decimals: 39 > +ANALYZE INSERT number of fields: 13 > + - 0: name: 'id'/''; table: ''/''; db: ''; catalog: 'def'; length: 3; max_length: 0; type: 8; decimals: 0 > + - 1: name: 'select_type'/''; table: ''/''; db: ''; catalog: 'def'; length: 57; max_length: 0; type: 253; decimals: 39 > + - 2: name: 'table'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 3: name: 'type'/''; table: ''/''; db: ''; catalog: 'def'; length: 30; max_length: 0; type: 253; decimals: 39 > + - 4: name: 'possible_keys'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 5: name: 'key'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 6: name: 'key_len'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 7: name: 'ref'/''; table: ''/''; db: ''; catalog: 'def'; length: 6144; max_length: 0; type: 253; decimals: 39 > + - 8: name: 'rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 10; max_length: 0; type: 8; decimals: 0 > + - 9: name: 'r_rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 10 > + - 10: name: 'filtered'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 2 > + - 11: name: 'r_filtered'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 2 > + - 12: name: 'Extra'/''; table: ''/''; db: ''; catalog: 'def'; length: 765; max_length: 0; type: 253; decimals: 39 > +ANALYZE JSON INSERT number of fields: 1 > + - 0: name: 'ANALYZE'/''; table: ''/''; db: ''; catalog: 'def'; length: 234; max_length: 0; type: 253; decimals: 39 > +EXPALIN UPDATE number of fields: 10 > + - 0: name: 'id'/''; table: ''/''; db: ''; catalog: 'def'; length: 3; max_length: 0; type: 8; decimals: 0 > + - 1: name: 'select_type'/''; table: ''/''; db: ''; catalog: 'def'; length: 57; max_length: 0; type: 253; decimals: 39 > + - 2: name: 'table'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 3: name: 'type'/''; table: ''/''; db: ''; catalog: 'def'; length: 30; max_length: 0; type: 253; decimals: 39 > + - 4: name: 'possible_keys'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 5: name: 'key'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 6: name: 'key_len'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 7: name: 'ref'/''; table: ''/''; db: ''; catalog: 'def'; length: 6144; max_length: 0; type: 253; decimals: 39 > + - 8: name: 'rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 10; max_length: 0; type: 8; decimals: 0 > + - 9: name: 'Extra'/''; table: ''/''; db: ''; catalog: 'def'; length: 765; max_length: 0; type: 253; decimals: 39 > +EXPALIN JSON UPDATE number of fields: 1 > + - 0: name: 'EXPLAIN'/''; table: ''/''; db: ''; catalog: 'def'; length: 234; max_length: 0; type: 253; decimals: 39 > +ANALYZE UPDATE number of fields: 13 > + - 0: name: 'id'/''; table: ''/''; db: ''; catalog: 'def'; length: 3; max_length: 0; type: 8; decimals: 0 > + - 1: name: 'select_type'/''; table: ''/''; db: ''; catalog: 'def'; length: 57; max_length: 0; type: 253; decimals: 39 > + - 2: name: 'table'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 3: name: 'type'/''; table: ''/''; db: ''; catalog: 'def'; length: 30; max_length: 0; type: 253; decimals: 39 > + - 4: name: 'possible_keys'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 5: name: 'key'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 6: name: 'key_len'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 7: name: 'ref'/''; table: ''/''; db: ''; catalog: 'def'; length: 6144; max_length: 0; type: 253; decimals: 39 > + - 8: name: 'rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 10; max_length: 0; type: 8; decimals: 0 > + - 9: name: 'r_rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 10 > + - 10: name: 'filtered'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 2 > + - 11: name: 'r_filtered'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 2 > + - 12: name: 'Extra'/''; table: ''/''; db: ''; catalog: 'def'; length: 765; max_length: 0; type: 253; decimals: 39 > +ANALYZE JSON UPDATE number of fields: 1 > + - 0: name: 'ANALYZE'/''; table: ''/''; db: ''; catalog: 'def'; length: 234; max_length: 0; type: 253; decimals: 39 > +EXPALIN DELETE number of fields: 10 > + - 0: name: 'id'/''; table: ''/''; db: ''; catalog: 'def'; length: 3; max_length: 0; type: 8; decimals: 0 > + - 1: name: 'select_type'/''; table: ''/''; db: ''; catalog: 'def'; length: 57; max_length: 0; type: 253; decimals: 39 > + - 2: name: 'table'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 3: name: 'type'/''; table: ''/''; db: ''; catalog: 'def'; length: 30; max_length: 0; type: 253; decimals: 39 > + - 4: name: 'possible_keys'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 5: name: 'key'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 6: name: 'key_len'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 7: name: 'ref'/''; table: ''/''; db: ''; catalog: 'def'; length: 6144; max_length: 0; type: 253; decimals: 39 > + - 8: name: 'rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 10; max_length: 0; type: 8; decimals: 0 > + - 9: name: 'Extra'/''; table: ''/''; db: ''; catalog: 'def'; length: 765; max_length: 0; type: 253; decimals: 39 > +EXPALIN JSON DELETE number of fields: 1 > + - 0: name: 'EXPLAIN'/''; table: ''/''; db: ''; catalog: 'def'; length: 234; max_length: 0; type: 253; decimals: 39 > +ANALYZE DELETE number of fields: 13 > + - 0: name: 'id'/''; table: ''/''; db: ''; catalog: 'def'; length: 3; max_length: 0; type: 8; decimals: 0 > + - 1: name: 'select_type'/''; table: ''/''; db: ''; catalog: 'def'; length: 57; max_length: 0; type: 253; decimals: 39 > + - 2: name: 'table'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 3: name: 'type'/''; table: ''/''; db: ''; catalog: 'def'; length: 30; max_length: 0; type: 253; decimals: 39 > + - 4: name: 'possible_keys'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 5: name: 'key'/''; table: ''/''; db: ''; catalog: 'def'; length: 192; max_length: 0; type: 253; decimals: 39 > + - 6: name: 'key_len'/''; table: ''/''; db: ''; catalog: 'def'; length: 12288; max_length: 0; type: 253; decimals: 39 > + - 7: name: 'ref'/''; table: ''/''; db: ''; catalog: 'def'; length: 6144; max_length: 0; type: 253; decimals: 39 > + - 8: name: 'rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 10; max_length: 0; type: 8; decimals: 0 > + - 9: name: 'r_rows'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 10 > + - 10: name: 'filtered'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 2 > + - 11: name: 'r_filtered'/''; table: ''/''; db: ''; catalog: 'def'; length: 4; max_length: 0; type: 5; decimals: 2 > + - 12: name: 'Extra'/''; table: ''/''; db: ''; catalog: 'def'; length: 765; max_length: 0; type: 253; decimals: 39 > +ANALYZE JSON DELETE number of fields: 1 > + - 0: name: 'ANALYZE'/''; table: ''/''; db: ''; catalog: 'def'; length: 234; max_length: 0; type: 253; decimals: 39 > +# ------------------------------------ > + > + > SET @@global.general_log= @old_general_log; > SET @@global.slow_query_log= @old_slow_query_log; > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 96485ed15ec..349ec576264 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -2659,18 +2660,27 @@ CHANGED_TABLE_LIST* THD::changed_table_dup(const char *key, size_t key_length) > } > > > -int THD::send_explain_fields(select_result *result, uint8 explain_flags, bool is_analyze) > +int THD::send_explain_fields(select_result *result, > + uint8 explain_flags, > + bool is_analyze, > + Prepared_statement *stmt) > { > List<Item> field_list; > + int rc; > if (lex->explain_json) > make_explain_json_field_list(field_list, is_analyze); > else > make_explain_field_list(field_list, explain_flags, is_analyze); > > result->prepare(field_list, NULL); > - return (result->send_result_set_metadata(field_list, > - Protocol::SEND_NUM_ROWS | > - Protocol::SEND_EOF)); > + if (stmt) > + rc= send_prep_stmt(stmt, result->field_count(field_list)) || > + result->send_result_set_metadata(field_list, Protocol::SEND_EOF); > + else > + rc= result->send_result_set_metadata(field_list, > + Protocol::SEND_NUM_ROWS | > + Protocol::SEND_EOF); > + return(rc); > } dunno, I'd rather keep prepare functionality in sql_prepare.cc. Like, void THD::prepare_explain_fields(select_result *result, List<Item> *field_list, uint8 explain_flags, bool is_analyze) { if (lex->explain_json) make_explain_json_field_list(field_list, is_analyze); else make_explain_field_list(field_list, explain_flags, is_analyze); result->prepare(field_list, NULL); } int THD::send_explain_fields(select_result *result, uint8 explain_flags, bool is_analyze) { List<Item> field_list; prepare_explain_fields(result, field_list, explain_flags, is_analyze); return result->send_result_set_metadata(field_list, Protocol::SEND_NUM_ROWS | Protocol::SEND_EOF); } and in sql_prepare.cc you do List<Item> field_list; thd->prepare_explain_fields(result, field_list, explain_flags, is_analyze); return send_prep_stmt(stmt, result->field_count(field_list)) || result->send_result_set_metadata(field_list, Protocol::SEND_EOF); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] [Commits] 602865f9f24: MDEV-13727 rpl.rpl_concurrency_error failed
by Alexander Barkov 20 Apr '18

20 Apr '18

Hello Andrei, In LOGGER::slow_log_print() there is a piece of this code: query_utime= (current_utime - thd->start_utime); lock_utime= (thd->utime_after_lock - thd->start_utime); my_hrtime_t current_time= { hrtime_from_time(thd->start_time) + thd->start_time_sec_part + query_utime }; If we don't reset thd->start_utime in the beginning of sp_instr_stmt::execute(), then this code will calculate something useless. It seems that this function: bool LOGGER::slow_log_print( THD *thd, const char *query, size_t query_length, ulonglong current_utime) should not use thd->start_time, thd->start_utime, thd->start_time_sec_part. and should get this information from the argument: bool LOGGER::slow_log_print( THD *thd, const char *query, size_t query_length, const QUERY_START_TIME_INFO *start, ulonglong current_utime) And sp_instr_stmt::execute() should not do thd->set_time(). The whole logic should reside in sql_parse.cc. 1. Remember the current time 2. Execute the statement 3. Log the query if it was slow, using the time remembered at step 1. Is my understanding correct? Thanks. On 04/19/2018 03:42 PM, andrei.elkin(a)pp.inet.fi wrote: > revision-id: 602865f9f242a7ed5e3679af60a78698c03d93f5 (mariadb-10.3.6-14-g602865f9f24) > parent(s): 02e897ca57168059c6a74d1129158afea93aa875 > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-04-18 13:04:46 +0300 > message: > > MDEV-13727 rpl.rpl_concurrency_error failed > > The test actually revealed a flaw in MDEV-8305 > which inadvertently enrolled the trigger and > stored function into slow query reporting aimed exclusively > to the stored procedure. > Specifically to the test, a query on the master was logged > with a timestamp of the query's top-level statement but its (post > update) trigger computed one more (later) timestamp which got > inserted into another table. > > Master-vs-slave discrepancy became evident thanks to > different execution time of the trigger combined with the fact of the > logged master timestamp in the micro-second fraction format was > truncated on the slave. The latter means the slave execution of the query > always started earlier than on the master as far as the timestamp was concerned. > And when the fractional part was close to 1 the 2nd timestamp acquired by > MDEV-8305 piece of logics could overflow to next integer value. > That's how the master table appeared to be temporarly behind the slave's one. > > Fixed with restoring the pre-MDEV-8305 behavior not to reset > the current statement's timestamp when the query is not from a stored procedure. > That is the trigger's or stored function's queries continue to use > the top-level statement's timestamp as they are supposed to. > > --- > mysql-test/main/func_time.result | 26 ++++++++++++++++++ > mysql-test/main/func_time.test | 59 ++++++++++++++++++++++++++++++++++++++++ > sql/sp_head.cc | 4 +-- > 3 files changed, 87 insertions(+), 2 deletions(-) > > diff --git a/mysql-test/main/func_time.result b/mysql-test/main/func_time.result > index 2772f850ce9..9babfc6a48b 100644 > --- a/mysql-test/main/func_time.result > +++ b/mysql-test/main/func_time.result > @@ -3484,3 +3484,29 @@ t1 CREATE TABLE `t1` ( > `c5` varchar(100) DEFAULT NULL > ) ENGINE=MyISAM DEFAULT CHARSET=latin1 > DROP TABLE t1; > +CREATE TABLE t_ts (a timestamp(6)); > +CREATE TABLE t_trig (a timestamp(6)); > +CREATE FUNCTION fn_sleep_before_now() returns int > +BEGIN > +INSERT INTO t_ts SET a= current_timestamp(6); > +RETURN 0; > +END// > +CREATE TRIGGER trg_insert_t_ts after INSERT on t_ts for each row > +BEGIN > +INSERT into t_trig set a= current_timestamp(6); > +END// > +SET @sav_slow_query_log= @@session.slow_query_log; > +SET @@session.slow_query_log= ON; > +SELECT current_timestamp(6),fn_sleep_before_now() INTO @ts_cur, @ts_func; > +SELECT a FROM t_ts LIMIT 1 into @ts_func; > +SELECT a FROM t_trig LIMIT 1 into @ts_trig; > +DELETE FROM t_ts; > +DELETE FROM t_trig; > +SET @@session.slow_query_log= OFF; > +SELECT current_timestamp(6),fn_sleep_before_now() INTO @ts_cur, @func_ts; > +SELECT a FROM t_ts LIMIT 1 into @ts_func; > +SELECT a FROM t_trig LIMIT 1 into @ts_trig; > +SET @@session.slow_query_log= @sav_slow_query_log; > +DROP FUNCTION fn_sleep_before_now; > +DROP TRIGGER trg_insert_t_ts; > +DROP TABLE t_ts, t_trig; > diff --git a/mysql-test/main/func_time.test b/mysql-test/main/func_time.test > index 5417cb20a92..ad207218969 100644 > --- a/mysql-test/main/func_time.test > +++ b/mysql-test/main/func_time.test > @@ -2057,3 +2057,62 @@ EXECUTE IMMEDIATE > USING NULL, '10', 10, 10.0, 10e0, TIME'10:20:30'; > SHOW CREATE TABLE t1; > DROP TABLE t1; > + > + > +############ > +# MDEV-13727 > +# Current timestamp functions inside stored functions must return the > +# value of the top-level statement's timestamp (its start time). > +# This must hold regardless of @@slow_query_log option. > +# > + > +CREATE TABLE t_ts (a timestamp(6)); > +CREATE TABLE t_trig (a timestamp(6)); > +delimiter //; > +CREATE FUNCTION fn_sleep_before_now() returns int > +BEGIN > + INSERT INTO t_ts SET a= current_timestamp(6); > + RETURN 0; > +END// > +CREATE TRIGGER trg_insert_t_ts after INSERT on t_ts for each row > +BEGIN > + INSERT into t_trig set a= current_timestamp(6); > +END// > +delimiter ;// > + > +SET @sav_slow_query_log= @@session.slow_query_log; > + > +# @@slow_query_log ON check > +SET @@session.slow_query_log= ON; > +SELECT current_timestamp(6),fn_sleep_before_now() INTO @ts_cur, @ts_func; > + > +SELECT a FROM t_ts LIMIT 1 into @ts_func; > +SELECT a FROM t_trig LIMIT 1 into @ts_trig; > +if (!`SELECT @ts_cur = @ts_func and @ts_func = @ts_trig`) > +{ > + SELECT @ts_cur, @ts_func, @ts_trig; > + --die Error: timestamps must be equal but they diverge > +} > +DELETE FROM t_ts; > +DELETE FROM t_trig; > + > +# @@slow_query_log OFF check > +SET @@session.slow_query_log= OFF; > +SELECT current_timestamp(6),fn_sleep_before_now() INTO @ts_cur, @func_ts; > +SELECT a FROM t_ts LIMIT 1 into @ts_func; > +SELECT a FROM t_trig LIMIT 1 into @ts_trig; > +if (!`SELECT @ts_cur = @ts_func and @ts_func = @ts_trig`) > +{ > + SELECT @ts_cur, @ts_func, @ts_trig; > + --die Error: timestamps must be equal but they diverge > +} > + > +# Cleanup > +SET @@session.slow_query_log= @sav_slow_query_log; > +DROP FUNCTION fn_sleep_before_now; > +DROP TRIGGER trg_insert_t_ts; > +DROP TABLE t_ts, t_trig; > + > +# > +# End of MDEV-13727 > +################### > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > index f7847bae89d..90fe77e6158 100644 > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -3471,7 +3471,7 @@ sp_instr_stmt::execute(THD *thd, uint *nextp) > thd->profiling.set_query_source(m_query.str, m_query.length); > #endif > > - if ((save_enable_slow_log= thd->enable_slow_log)) > + if ((save_enable_slow_log= thd->enable_slow_log) && !thd->in_sub_stmt) > { > /* > Save start time info for the CALL statement and overwrite it with the > @@ -3544,7 +3544,7 @@ sp_instr_stmt::execute(THD *thd, uint *nextp) > } > } > /* Restore the original query start time */ > - if (thd->enable_slow_log) > + if (thd->enable_slow_log && !thd->in_sub_stmt) > thd->restore_query_start_time(&time_info); > > DBUG_RETURN(res || thd->is_error()); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

1 0

Re: [Maria-developers] Mdev-10664 Add statuses about optimistic parallel replication stalls.
by Sachin Setiya 19 Apr '18

19 Apr '18

Hi Kristian! Sorry for late reply , I was on vacations. On Tue, Jan 10, 2017 at 7:31 PM, Kristian Nielsen <knielsen(a)knielsen-hq.org> wrote: > Sachin Setiya <sachin.setiya(a)mariadb.com> writes: > > >> Right, but you still need _some_ way to synchronise multiple threads > > > I used atomic builtins my_atomic_add64_explicit. > > That seems fine (but see comment below). > > > Documentation:- > > > > Slave_DDL_Events:- > > > > Description: Number of DDL statements slave have executed in > > replication channel X. > > Scope: Per Replication Channel > > Data Type: numeric > > Introduced: MariaDB 10.1 > > > > Slave_Non_Transactional_Events:- > > > > Description: Number of Non Transactional "no idea what should be the > > next word " slave have executed in replication channel X. > > Scope: Per Replication Channel > > Data Type: numeric > > Introduced: MariaDB 10.1 > > I checked, the term "event group" is used already multiple places in the > documentation, so I think it can be used here also. An event group can be a > transaction (possibly with multiple transactional DML statements), or a DDL > or non-transactional statement. > > What you are doing is introducing status variables to help understand > optimistic (and aggressive) parallel replication modes. > > In optimistic parallel replication, a non-transactional event group is not > safe to run in parallel with other non-transactional events. So that event > is made to wait for all prior event groups to reach commit state, before > being allowed to execute. However, other transactional events are not > delayed by the non-transactional event group. > > Slave_Non_Transactional_Events counts the occurences of this. > > Then in case of a DDL statement, this can potentially be unsafe to run in > parallel even with other transactional events. So for a DDL, the DDL has to > wait for all prior event groups, _and_ subsequent event groups have to wait > for the DDL. > > Slave_DDL_Events counts these occurences. > > This is what the documentation, and commit messages, should explain, I > think. > > I think the status variables describe the values for > @@default_master_connection?, this should also be explained in the > documentation. > > Why did you decide to put this information into a status variable? > Normally, > slave status is seen in SHOW SLAVE STATUS, which supports showing status > for > a particular connection without using @@default_master_connection. > > Sorry for this , I guess It was a confusion I was looking at Slave_Running this is avaliable in show status as well as show slave status. So I thought for if I want to show some information in SHOW slave status this has to be in show status. And this is wrong. Now we can see non trans events / ddl events in show slave status. Show I remove this vars from Show Status, or it is okay ? > > diff --git a/sql/log_event.h b/sql/log_event.h > > index 90900f6..579607f 100644 > > --- a/sql/log_event.h > > +++ b/sql/log_event.h > > @@ -50,6 +50,7 @@ > > #endif > > > > #include "rpl_gtid.h" > > +#include "my_atomic.h" > > Why include my_atomic.h in the header file, when you only use it in the .cc > file? > Changed. > > > + mysql_mutex_lock(&LOCK_active_mi); > > + if (master_info_index) > > + { > > + mi= master_info_index-> > > + get_master_info(&thd->variables.default_master_connection, > > + Sql_condition::WARN_LEVEL_NOTE); > > + if (mi) > > + tmp= mi->total_ddl_events; > > > + if (mi) > > + tmp= mi->total_non_trans_events; > > Better use an atomic primitive here to read the value. Taking a mutex does > not help synchronise against my_atomic_add64_explicit(). > > Changed. > > + } > > + mysql_mutex_unlock(&LOCK_active_mi); > > + if (mi) > > + *((longlong *)buff)= tmp; > > + else > > + var->type= SHOW_UNDEF; > > + > > + return 0; > > +} > > #endif /* HAVE_REPLICATION */ > > > > static int show_open_tables(THD *thd, SHOW_VAR *var, char *buff, > > > index 9365c06..a474aa5 100644 > > --- a/sql/rpl_mi.h > > +++ b/sql/rpl_mi.h > > @@ -303,6 +303,12 @@ class Master_info : public > Slave_reporting_capability > > > > /* The parallel replication mode. */ > > enum_slave_parallel_mode parallel_mode; > > + > > + /* No of DDL statements */ > > + long total_ddl_events; > > + > > + /* No of non-transactional statements*/ > > + long total_non_trans_events; > > "long" is not the appropriate type here. Certainly not since you are using > my_atomic_add64_explicit(). Use a type that is the same size on all > platforms (and why use a signed type?). > > Changed to uint64. > - Kristian. > Attached a newer version of patch , please review it. -- Regards Sachin

3 7

[Maria-developers] CREATE SEQUENCE is coming
by Michael Widenius 18 Apr '18

18 Apr '18

Hi! For those that are trying to port applications to MariaDB from other databases or need more features than the current AUTO_INCREMENT provides, the following may be of interest: First version of sequence is pushed to bb-10.2-sequence. It should be merged to bb-10.2-compatilbity and then to 10.3 shortly (1-2 weeks). What is working for the moment: CREATE OR REPLACE [TEMPORARY] SEQUENCE [IF NOT EXISTS] name [ INCREMENT [ BY | = ] increment ] [ MINVALUE [=] minvalue | NO MINVALUE ] [ MAXVALUE [=] maxvalue | NO MAXVALUE ] [ START [ WITH | = ] start ] [ CACHE [=] cache ] [ [ NO ] CYCLE ] ENGINE=xxx COMMENT=".." SELECT NEXT VALUE FOR sequence_name; SELECT NEXTVAL(sequence_name); SELECT PREVIOUS VALUE FOR sequence_name; SELECT LASTVAL(sequence_name); SHOW CREATE SEQUENCE sequence_name; SHOW CREATE TABLE sequence_name; CREATE TABLE sequence-structure ... SEQUENCE=1 ALTER TABLE sequence RENAME TO sequence2; RENAME TABLE sequence RENAME TO sequence2; DROP [TEMPORARY] SEQUENCE sequence_name [IF EXISTS] See https://jira.mariadb.org/browse/MDEV-10139 for progress. See commit messages in bb-10.2-sequence for what is still to be done. Some documentation can be found at: https://mariadb.com/kb/en/mariadb/create-sequence/ This will be improved as the work progress. One of the goals with the SEQUENCE implementation is that all old tools, like mysqldump, should work unchanged, while still keeping normal usage of sequence standard compatibly. The make this possible, the sequence is currently implemented as a table with a few exclusive properties. The main disadvantage of having sequence as a table is that it uses the same name space as tables. In other words, you can't have a table and a sequence of the same name. The benefit is that sequences shows up in 'show tables', one can create a sequence also with 'create table' and drop it with 'drop table'. One can select from it as from any other table. This ensures that all old tools that works with tables should work with sequences. The special properties for sequence tables are: - A sequence table has always one row. - When one creates a sequence, either with CREATE TABLE or CREATE SEQUENCE, one row will be inserted. - Normal table options works for CREATE SEQUENCE. One can use ENGINE=xxx, COMMENT=xxx etc. - If one tries to insert into a sequence table, the single row will be updated. This allows mysqldump to work but also gives the additional benefit that one can change all properties of a sequence with a single insert. New applications can of course also use ALTER SEQUENCE. - Updates to the sequence table will change the single row. - Doing a select on the sequence shows the current state of the sequence, except the values that are reserved in the cache. The column 'next_value' shows the next value not reserved by the cache. - Truncate on a sequence table will give an error. - Alter table and rename works on sequences. - If one creates a sequence with INCREMENT 0, then the sequence will use auto_increment_increment and auto_increment_offset for the sequence, just like AUTO_INCREMENT. This allows sequences to work reliable in a master-master environment and with Galera. Internally sequence tables are created as a normal table without rollback (InnoDB, Aria and MySAM supports this) with is wrapped by a sequence engine. This allowed me to create sequences with almost no performance impact for normal tables. (The cost is one 'if' per insert if binary log is enabled). MariaDB 10.3 will support both the ANSI SQL and Oracle syntax for creating and accessing sequences. As ANSI SQL doesn't have an easy access to the last generated value, 103 also supports 'PREVIOUS VALUE FOR sequence_name', like IBM DB2, and LASTVAL(sequence_name) as PostgreSQL. I want to thank Jianwe Zhao from Aliyun for his work on SEQUENCE in AliSQL which gave me ideas and inspiration for this work. I also want to thank Marko Mäkelä for his help in the InnoDB part of the code. There is still a lot of work to fix edge cases, but in the current implementation most major things seams to work... Comments, suggestions or questions? Regards, Monty

2 1

Re: [Maria-developers] 5e94fe1: MDEV-15754 Server crashes in fill_record_n_invoke_before_triggers upon
by Sergei Golubchik 17 Apr '18

17 Apr '18

Hi, Sachin! Looks good, thanks! Ok to push. On Apr 16, sachin wrote: > revision-id: 5e94fe189d6325028613b2c830082e07ff303acb (mariadb-10.3.5-156-g5e94fe1) > parent(s): f0e4f94c230326a2f2e608e4119530d775c37b21 > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2018-04-16 16:27:11 +0530 > message: > > MDEV-15754 Server crashes in fill_record_n_invoke_before_triggers upon ... > insert into table with TIMESTAMP INVISIBLE > > Problem:- The segfault occurs because value is null but since timestamp field > is VISIBLE it expects a value , and it tries to call value->save_in_field(.. > Timestamp field should not be visible this is the problem. > > Solution:- While we clone field for record0_field we don't honor the field > _visibility , this patch changes that. > > diff --git a/sql/sql_trigger.cc b/sql/sql_trigger.cc > index 4c1e2a5..d523779 100644 > --- a/sql/sql_trigger.cc > +++ b/sql/sql_trigger.cc > @@ -1240,6 +1240,7 @@ bool Table_triggers_list::prepare_record_accessors(TABLE *table) > return 1; > > f->flags= (*fld)->flags; > + f->invisible= (*fld)->invisible; > f->null_ptr= null_ptr; > f->null_bit= null_bit; > if (null_bit == 128) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Questions about AS OF (possibly bugs found)
by Alexander Barkov 16 Apr '18

16 Apr '18

Hi Sergei, I'm looking at this use rule in sql_yacc.yy: history_point: temporal_literal { $$= Vers_history_point(VERS_TIMESTAMP, $1); } | function_call_keyword_timestamp { $$= Vers_history_point(VERS_TIMESTAMP, $1); } | opt_history_unit simple_expr { $$= Vers_history_point($1, $2); } ; AS OF DATE 'xxx' - returns Vers_history_point(VERS_TIMESTAMP) AS OF DATE('xxx') - returns Vers_history_point(VERS_UNDEFINED) AS OF TIMESTAMP('2001-01-01') - returns Vers_history_point(VERS_TIMESTAMP) AS OF COALESCE(TIMESTAMP('2001-01-01')) - returns Vers_history_point(VERS_UNDEFINED) Perhaps this is not expected. Also, this code looks suspicious: void Vers_history_point::resolve_unit(bool timestamps_only) { if (item && unit == VERS_UNDEFINED) { if (item->type() == Item::FIELD_ITEM || timestamps_only) unit= VERS_TIMESTAMP; else if (item->result_type() == INT_RESULT || item->result_type() == REAL_RESULT) unit= VERS_TRX_ID; else unit= VERS_TIMESTAMP; } } Why DECIMAL_RESULT is not handled? Why only Item::FIELD_ITEM is checked? Can't "history_point" be simplified to something like: history_point: simple_expr { $$= Vers_history_point(VERS_UNDEFINED, $1); } | TRANSACTION_SYM simple_expr { $$= Vers_history_point(VERS_TRX_ID, $2); } ; I'm saying "something like" because TRANSACTION_SYM will cause a conflict with simple_expr. So probably it should be cought somehow else, by catching Item_ident and checking it name. Btw, an SP variable with name "TRANSACTION" does not work well: This script works fine: DELIMITER $$ BEGIN NOT ATOMIC DECLARE TRANSACTION int DEFAULT 10; SELECT TRANSACTION; END; $$ DELIMITER ; This script: DELIMITER $$ BEGIN NOT ATOMIC DECLARE TRANSACTION int DEFAULT 10; SELECT * FROM t1 FOR SYSTEM_TIME AS OF TRANSACTION; END; $$ DELIMITER ; returns an error: ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '; Note, if I rename "TRANSACTION" to "TRANSACTION1", it works fine. Thanks!

3 8

[Maria-developers] innobase/sync0policy.h converts opaque pthread_t to ULINT
by Teodor Mircea Ionita 16 Apr '18

16 Apr '18

Hi, While trying to build on macOS 10.13.4 with clang 9.1.0 I originally came across this error: -- .../storage/innobase/include/sync0policy.h:79:4: error: cannot initialize a parameter of type 'int64' (aka 'long long') with an rvalue of type 'os_thread_id_t' (aka '_opaque_pthread_t *') my_atomic_storelint(&m_thread_id, os_thread_get_curr_id()); --- This problem was originally reported in MDEV-15778. Since pthread_t on macOS is a pointer to a struct named _opaque_pthread_t, the "obvious" solution for this was to explicitly cast to ULINT which in turn is defined as size_t, fine so far. Solution was tested in PR#691. However, upon testing my patch with buildbot, on slave kvm-fulltest2, which is 64bit qemu vm with Ubuntu precise-i386 (32bit) running and gcc compiling in -march=i686. The above "fix" now yields: -- /home/buildbot/buildbot/build/mariadb-10.3.6/storage/innobase/include/sync0policy.h:79:4: error: invalid cast from type ‘os_thread_id_t {aka long unsigned int}’ to type ‘ulint {aka unsigned int}’ --- Now reconsidering more carefully, this looks more serious then initially thought, posix pthread_t(3) is defined as: -- POSIX.1 allows an implementation wide freedom in choosing the type used to represent a thread ID; for example, representation using either an arithmetic type or a structure is permitted. Therefore, variables of type pthread_t can't portably be compared using the C equality operator (==); use pthread_equal(3)instead. --- So in this particular platform spec, where pthread_t on Linux is defined as unsigned long int integral type and if LP64 model is used, this implies pthread_t always being 64bit in either both 32 and 64 arch and trying to convert to size_t on a 32bit arch will fail. Now given that the standard particularly states pthread_t as being opaque and no assumptions should be made in regards to the actual type, not even an initializer type (which ULINT_UNDEFINED defines as -1), trying to cast m_thread_id to any other type ( (ULINT) ) either implicitly or explicit is not portable. Is this a bug that needs fixing or just impossible to handle pthread id atomically in any other fashion? How about a test case I can run to confirm/infirm erratic behavior? Since I don't have access to the particular bb slave instance to check with sizeof() maybe somebody else can help me here to confirm this? Relevant links bellow: https://jira.mariadb.org/browse/MDEV-15778 https://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/12317/… https://linux.die.net/man/3/pthread_self https://github.com/MariaDB/server/pull/691 Cheers, Teodor

4 5

Re: [Maria-developers] a91fdd1: Fixed the bug mdev-15575 different results when using CTE and big_tables=1.
by Sergei Golubchik 16 Apr '18

16 Apr '18

Hi, Igor On Apr 14, IgorBabaev wrote: > revision-id: a91fdd1288b3cbc5616f2ef9e0ce04bf7f0ef9ce (mariadb-10.2.14-35-ga91fdd1) > parent(s): 740fc2ae084f8f81990de557d696aefbc507752d > author: Igor Babaev > committer: Igor Babaev > timestamp: 2018-04-14 21:32:41 -0700 > message: > > Fixed the bug mdev-15575 different results when using CTE and big_tables=1. please, make the first line MDEV-15575 different results when using CTE and big_tables=1 most other developers use that pattern, so your commits really stand out in the history, e.g.: 4c7a1a1 MDEV-15780 : mariabackup does not handle absolute names in for system tablespaces 9124590 MDEV-12903: encryption.innodb_encryption_discard_import fails in buildbot with FOUND vs NOT FOUND dd12779 MDEV-15832 With innodb_fast_shutdown=3, skip the rollback of connected transactions 740fc2a Fixed mdev-15765 BETWEEN not working in certain cases 8334ace MDEV-14380 Reduce possibility to timing-induced error in test and the reason for that is that many tools show only first 50 characters of the commit summary (first line), so starting it with the "Fixed the bug " does not add new information, but only pushes the relevant content to the right to be truncated. > This bug happened due to a defect of the implementation of the handler > function ha_delete_all_rows() for the ARIA engine. > The function maria_delete_all_rows() truncated the table, but it didn't > touch the write cache, so the cache's write offset was not reset. > In the scenario like in the function st_select_lex_unit::exec_recursive > when first all records were deleted from the table and then several new > records were added some metadata became inconsistent with the state of > the cache. As a result the table scan function could not read records > at the end of the table. > The same defect could be found in the implementation of ha_delete_all_rows() > for the MYISAM engine mi_delete_all_rows(). This is ok to push > Additionally made late instantiation for the temporary table used to store > rows that were used for each new iteration when executing a recursive CTE. I didn't review this part. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Questions regarding closing partitions in MDEV-11084
by Mattias Jonsson 16 Apr '18

16 Apr '18

Hi MariaDB Devs, I tried to evaluate spider engine and found an issue where it crashes, most likely due to MDEV-11084 (Stacktrace and reproducible test case attached). That also leads me to wonder about the performance for partitioned tables after MDEV-11084, when it seems to close partitions whenever it is not used in a statement (i.e. require it to be reopened in the next statement that would use another partition, effectively not using the open table cache). I cannot see anything mentioned in the final commit message hinting on that it closes partitions not used in the current query, but in the previous patches it was mentioned without any reason. Why does it close the already opened partitions? https://github.com/MariaDB/server/blob/10.3/sql/ha_partition.cc#L8365 I would not mind opening the partitions only when they are to be used (even though there are engines that need to be tested more), but closing them makes no sense to me performance wise. Also notice that the partitions first will be put back into the open table cache and then on the next query the non-used partitions will be closed and the needed ones be (re)-opened. Regards Mattias Jonsson

1 0

Re: [Maria-developers] Fwd: [Commits] b32031991e4: initial oracle parser fix
by Alexander Barkov 16 Apr '18

16 Apr '18

Hello Sanja, I reviewed your recent changes in "10.3-MDEV-11953" (and the attached additional patch for sql_yacc_ora.yy) I have some proposals: 1. Can you please move huge pieces of the code from sql_yacc.yy to LEX or other relevant classes? It makes the grammar much more readable (patches are aslo much more readable). I'd move the relevant pieces of the code to LEX as a separate patch, even before fixing the grammar. 2. You're adding too many main_select_push() and pop_select(). Please move them to upper level rules (it should be possible in many cases). Add new helper rules when needed. For example, this piece of code repeats many times: + { + if (Lex->main_select_push()) + MYSQL_YYABORT; + } + expr + { + Lex->pop_select(); //main select + $$= $3; It deserved a rule, say, expr_with_select_push_pop. You can find a better name :) - Serg and I spent a lot of time working on this task: MDEV-8909 union parser cleanup (and its 13 dependency tasks, and 3 related tasks, see the "Issue links" section in MDEV). We think that it should be the parser who disallows bad grammar, instead of post-analysis with raising errors like ER_CANT_USE_OPTION_HERE. Please keep using the same approach. Thanks! On 04/06/2018 07:37 PM, Oleksandr Byelkin wrote: > > > > -------- Weitergeleitete Nachricht -------- > Betreff: [Commits] b32031991e4: initial oracle parser fix > Datum: Fri, 06 Apr 2018 17:34:31 +0200 > Von: Oleksandr Byelkin <sanja(a)mariadb.com> > Antwort an: maria-developers(a)lists.launchpad.net > An: commits(a)mariadb.org > > > > revision-id: b32031991e4ba270f5c221f7f6068732d09efd53 > (mariadb-10.3.5-110-gb32031991e4) > parent(s): 60a502eb7efbe87bd8e3af3e3ece49a9eb83f8b3 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-04-06 17:05:34 +0200 > message: > > initial oracle parser fix > > --- > sql/sql_yacc.yy | 8 +- > sql/sql_yacc_ora.yy | 2085 > +++++++++++++++++++++++++++++---------------------- > 2 files changed, 1183 insertions(+), 910 deletions(-) > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index e4cbff7114e..9372269b05d 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -904,7 +904,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t > *yystacksize); > %parse-param { THD *thd } > %lex-param { THD *thd } > /* > - Currently there are 96 shift/reduce conflicts. > + Currently there are 135 shift/reduce conflicts. > We should not introduce new conflicts any more. > */ > %expect 135 > @@ -9244,7 +9244,7 @@ table_value_constructor: > lex->push_select(sel)) > MYSQL_YYABORT; > sel->init_select(); > - sel->braces= FALSE; > + sel->braces= FALSE; // just initialisation > } > values_list > { > @@ -13763,8 +13763,10 @@ single_multi: > } > | table_wild_list > { > + /* XXX > if (Lex->main_select_push()) > MYSQL_YYABORT; > + */ > mysql_init_multi_delete(Lex); > YYPS->m_lock_type= TL_READ_DEFAULT; > YYPS->m_mdl_type= MDL_SHARED_READ; > @@ -13779,8 +13781,10 @@ single_multi: > } > | FROM table_alias_ref_list > { > + /* XXX > if (Lex->main_select_push()) > MYSQL_YYABORT; > + */ > mysql_init_multi_delete(Lex); > YYPS->m_lock_type= TL_READ_DEFAULT; > YYPS->m_mdl_type= MDL_SHARED_READ; > diff --git a/sql/sql_yacc_ora.yy b/sql/sql_yacc_ora.yy > index c1512434d72..4a3ee348f89 100644 > --- a/sql/sql_yacc_ora.yy > +++ b/sql/sql_yacc_ora.yy > @@ -186,6 +186,20 @@ void ORAerror(THD *thd, const char *s) > LEX_CSTRING name; > uint offset; > } sp_cursor_name_and_offset; > + struct > + { > + enum sub_select_type unit_type; > + bool distinct; > + } unit_operation; > + struct > + { > + SELECT_LEX *first; > + SELECT_LEX *prev_last; > + } select_list; > + SQL_I_List<ORDER> *select_order; > + Lex_select_lock select_lock; > + Lex_select_limit select_limit; > + Lex_order_limit_lock *order_limit_lock; > > /* pointers */ > Create_field *create_field; > @@ -231,6 +245,7 @@ void ORAerror(THD *thd, const char *s) > > handlerton *db_type; > st_select_lex *select_lex; > + st_select_lex_unit *select_lex_unit; > struct p_elem_val *p_elem_value; > class Window_frame *window_frame; > class Window_frame_bound *window_frame_bound; > @@ -240,7 +255,6 @@ void ORAerror(THD *thd, const char *s) > /* enums */ > enum enum_sp_suid_behaviour sp_suid; > enum enum_view_suid view_suid; > - enum sub_select_type unit_type; > enum Condition_information_item::Name cond_info_item_name; > enum enum_diag_condition_item_name diag_condition_item_name; > enum Diagnostics_information::Which_area diag_area; > @@ -277,10 +291,10 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t > *yystacksize); > %parse-param { THD *thd } > %lex-param { THD *thd } > /* > - Currently there are 104 shift/reduce conflicts. > + Currently there are 99 shift/reduce conflicts. > We should not introduce new conflicts any more. > */ > -%expect 104 > +%expect 99 > > /* > Comments for TOKENS. > @@ -599,6 +613,9 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t > *yystacksize); > %token LEAVES > %token LEAVE_SYM > %token LEFT /* SQL-2003-R */ > +%token LEFT_PAREN_ALT /* INTERNAL */ > +%token LEFT_PAREN_WITH /* INTERNAL */ > +%token LEFT_PAREN_LIKE /* INTERNAL */ > %token LESS_SYM > %token LEVEL_SYM > %token LEX_HOSTNAME > @@ -1063,7 +1080,8 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t > *yystacksize); > NCHAR_STRING > > %type <lex_str_ptr> > - opt_table_alias > + opt_table_alias_clause > + table_alias_clause > > %type <lex_string_with_pos> > ident ident_with_tok_start > @@ -1111,7 +1129,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t > *yystacksize); > opt_temporary all_or_any opt_distinct opt_glimit_clause > opt_ignore_leaves fulltext_options union_option > opt_not > - select_derived_init transaction_access_mode_types > + transaction_access_mode_types > opt_natural_language_mode opt_query_expansion > opt_ev_status opt_ev_on_completion ev_on_completion opt_ev_comment > ev_alter_on_schedule_completion opt_ev_rename_to opt_ev_sql_stmt > @@ -1231,9 +1249,8 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t > *yystacksize); > join_table_list join_table > table_factor table_ref esc_table_ref > table_primary_ident table_primary_derived > - select_derived derived_table_list > - select_derived_union > - derived_query_specification > + derived_table_list table_reference_list_parens > + nested_table_reference_list join_table_parens > %type <date_time_type> date_time_type; > %type <interval> interval > > @@ -1276,12 +1293,16 @@ bool my_yyoverflow(short **a, YYSTYPE **b, > size_t *yystacksize); > UNDERSCORE_CHARSET > > %type <select_lex> subselect > - get_select_lex get_select_lex_derived > query_specification > - query_term_union_not_ready > - query_term_union_ready > + table_value_constructor > + simple_table > + query_primary > + query_primary_parens > + > +%type <select_lex_unit> > query_expression_body > - select_paren_derived > + query_expression > + query_expression_unit > > %type <boolfunc2creator> comp_op > > @@ -1293,7 +1314,21 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t > *yystacksize); > > %type <virtual_column> opt_check_constraint check_constraint > virtual_column_func > column_default_expr > -%type <unit_type> unit_type_decl > + > +%type <unit_operation> unit_type_decl > + > +%type <select_lock> > + opt_select_lock_type > + select_lock_type > + opt_lock_wait_timeout_new > + > +%type <select_limit> opt_limit_clause limit_clause limit_options > + > +%type <order_limit_lock> > + query_expression_tail > + order_or_limit > + > +%type <select_order> opt_order_clause order_clause order_list > > %type <NONE> > analyze_stmt_command > @@ -1313,7 +1348,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t > *yystacksize); > assign_to_keycache_parts > preload_list preload_list_or_parts preload_keys preload_keys_parts > select_item_list select_item values_list no_braces > - opt_limit_clause delete_limit_clause fields opt_values values > + delete_limit_clause fields opt_values values > procedure_list procedure_list2 procedure_item > field_def handler opt_generated_always > opt_ignore opt_column opt_restrict > @@ -1333,9 +1368,8 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t > *yystacksize); > table_to_table_list table_to_table opt_table_list opt_as > handler_rkey_function handler_read_or_scan > single_multi table_wild_list table_wild_one opt_wild > - union_clause union_list > - subselect_start opt_and charset > - subselect_end select_var_list select_var_list_init help > + opt_and charset > + select_var_list select_var_list_init help > opt_extended_describe shutdown > opt_format_json > prepare prepare_src execute deallocate > @@ -1606,14 +1640,22 @@ deallocate_or_drop: > ; > > prepare: > - PREPARE_SYM ident FROM prepare_src > + PREPARE_SYM > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > + ident FROM prepare_src > { > LEX *lex= thd->lex; > if (lex->table_or_sp_used()) > my_yyabort_error((ER_SUBQUERIES_NOT_SUPPORTED, MYF(0), > "PREPARE..FROM")); > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > lex->sql_command= SQLCOM_PREPARE; > - lex->prepared_stmt_name= $2; > + lex->prepared_stmt_name= $3; > + Lex->pop_select(); //main select > } > ; > > @@ -1632,10 +1674,21 @@ execute: > LEX *lex= thd->lex; > lex->sql_command= SQLCOM_EXECUTE; > lex->prepared_stmt_name= $2; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > execute_using > - {} > - | EXECUTE_SYM IMMEDIATE_SYM prepare_src > + { > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + } > + | EXECUTE_SYM IMMEDIATE_SYM > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > + prepare_src > { > if (Lex->table_or_sp_used()) > my_yyabort_error((ER_SUBQUERIES_NOT_SUPPORTED, MYF(0), > @@ -1643,7 +1696,11 @@ execute: > Lex->sql_command= SQLCOM_EXECUTE_IMMEDIATE; > } > execute_using > - {} > + { > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + } > ; > > execute_using: > @@ -1928,13 +1985,20 @@ connection_name: > /* create a table */ > > create: > - create_or_replace opt_temporary TABLE_SYM opt_if_not_exists > table_ident > + create_or_replace opt_temporary TABLE_SYM opt_if_not_exists > { > LEX *lex= thd->lex; > lex->create_info.init(); > + if (lex->main_select_push()) > + MYSQL_YYABORT; > + lex->current_select->parsing_place= BEFORE_OPT_LIST; > if (lex->set_command_with_check(SQLCOM_CREATE_TABLE, $2, $1 > | $4)) > MYSQL_YYABORT; > - if (!lex->builtin_select.add_table_to_list(thd, $5, NULL, > + } > + table_ident > + { > + LEX *lex= thd->lex; > + if (!lex->builtin_select.add_table_to_list(thd, $6, NULL, > TL_OPTION_UPDATING, > TL_WRITE, > MDL_EXCLUSIVE)) > MYSQL_YYABORT; > @@ -1960,13 +2024,16 @@ create: > ER_WARN_USING_OTHER_HANDLER, > ER_THD(thd, > ER_WARN_USING_OTHER_HANDLER), > > hton_name(lex->create_info.db_type)->str, > - $5->table.str); > + $6->table.str); > } > create_table_set_open_action_and_adjust_tables(lex); > + Lex->pop_select(); //main select > } > | create_or_replace opt_temporary SEQUENCE_SYM opt_if_not_exists > table_ident > { > LEX *lex= thd->lex; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > lex->create_info.init(); > if (lex->set_command_with_check(SQLCOM_CREATE_SEQUENCE, $2, > $1 | $4)) > MYSQL_YYABORT; > @@ -2021,42 +2088,69 @@ create: > $5->table.str); > } > create_table_set_open_action_and_adjust_tables(lex); > + Lex->pop_select(); //main select > } > - | create_or_replace opt_unique INDEX_SYM opt_if_not_exists ident > + | create_or_replace opt_unique INDEX_SYM opt_if_not_exists > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > + ident > opt_key_algorithm_clause > ON table_ident > { > - if (Lex->add_create_index_prepare($8)) > + if (Lex->add_create_index_prepare($9)) > MYSQL_YYABORT; > - if (Lex->add_create_index($2, &$5, $6, $1 | $4)) > + if (Lex->add_create_index($2, &$6, $7, $1 | $4)) > MYSQL_YYABORT; > } > '(' key_list ')' opt_lock_wait_timeout normal_key_options > - opt_index_lock_algorithm { } > - | create_or_replace fulltext INDEX_SYM opt_if_not_exists ident > + opt_index_lock_algorithm > + { > + Lex->pop_select(); //main select > + } > + | create_or_replace fulltext INDEX_SYM > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > + opt_if_not_exists ident > ON table_ident > { > - if (Lex->add_create_index_prepare($7)) > + if (Lex->add_create_index_prepare($8)) > MYSQL_YYABORT; > - if (Lex->add_create_index($2, &$5, HA_KEY_ALG_UNDEF, $1 | $4)) > + if (Lex->add_create_index($2, &$6, HA_KEY_ALG_UNDEF, $1 | $5)) > MYSQL_YYABORT; > } > '(' key_list ')' opt_lock_wait_timeout fulltext_key_options > - opt_index_lock_algorithm { } > - | create_or_replace spatial INDEX_SYM opt_if_not_exists ident > + opt_index_lock_algorithm > + { > + Lex->pop_select(); //main select > + } > + | create_or_replace spatial INDEX_SYM > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > + opt_if_not_exists ident > ON table_ident > { > - if (Lex->add_create_index_prepare($7)) > + if (Lex->add_create_index_prepare($8)) > MYSQL_YYABORT; > - if (Lex->add_create_index($2, &$5, HA_KEY_ALG_UNDEF, $1 | $4)) > + if (Lex->add_create_index($2, &$6, HA_KEY_ALG_UNDEF, $1 | $5)) > MYSQL_YYABORT; > } > '(' key_list ')' opt_lock_wait_timeout spatial_key_options > - opt_index_lock_algorithm { } > + opt_index_lock_algorithm > + { > + Lex->pop_select(); //main select > + } > | create_or_replace DATABASE opt_if_not_exists ident > { > Lex->create_info.default_table_charset= NULL; > Lex->create_info.used_fields= 0; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > opt_create_database_options > { > @@ -2064,51 +2158,94 @@ create: > if (lex->set_command_with_check(SQLCOM_CREATE_DB, 0, $1 | $3)) > MYSQL_YYABORT; > lex->name= $4; > + Lex->pop_select(); //main select > } > | create_or_replace definer_opt opt_view_suid VIEW_SYM > opt_if_not_exists table_ident > { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > if (Lex->add_create_view(thd, $1 | $5, > DTYPE_ALGORITHM_UNDEFINED, $3, $6)) > MYSQL_YYABORT; > } > view_list_opt AS view_select > - { } > + { > + Lex->pop_select(); //main select > + } > | create_or_replace view_algorithm definer_opt opt_view_suid > VIEW_SYM > opt_if_not_exists table_ident > { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > if (Lex->add_create_view(thd, $1 | $6, $2, $4, $7)) > MYSQL_YYABORT; > } > view_list_opt AS view_select > - { } > + { > + Lex->pop_select(); //main select > + } > | create_or_replace definer_opt TRIGGER_SYM > - { Lex->create_info.set($1); } > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + Lex->create_info.set($1); > + } > trigger_tail > - { } > + { > + Lex->pop_select(); //main select > + } > | create_or_replace definer_opt PROCEDURE_SYM > - { Lex->create_info.set($1); } > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + Lex->create_info.set($1); > + } > sp_tail_standalone > - { } > + { > + Lex->pop_select(); //main select > + } > | create_or_replace definer_opt EVENT_SYM > - { Lex->create_info.set($1); } > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + Lex->create_info.set($1); > + } > event_tail > - { } > + { > + Lex->pop_select(); //main select > + } > | create_or_replace definer FUNCTION_SYM > - { Lex->create_info.set($1); } > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + Lex->create_info.set($1); > + } > sf_tail_standalone > - { } > + { > + Lex->pop_select(); //main select > + } > | create_or_replace no_definer FUNCTION_SYM > - { Lex->create_info.set($1); } > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + Lex->create_info.set($1); > + } > create_function_tail > - { } > + { > + Lex->pop_select(); //main select > + } > | create_or_replace no_definer AGGREGATE_SYM FUNCTION_SYM > { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > Lex->create_info.set($1); > Lex->udf.type= UDFTYPE_AGGREGATE; > } > udf_tail > - { } > + { > + Lex->pop_select(); //main select > + } > | create_or_replace USER_SYM opt_if_not_exists clear_privileges > grant_list > opt_require_clause opt_resource_options > { > @@ -2815,8 +2952,13 @@ call: > { > if (Lex->call_statement_start(thd, $2)) > MYSQL_YYABORT; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > + opt_sp_cparam_list > + { > + Lex->pop_select(); //main select > } > - opt_sp_cparam_list {} > ; > > /* CALL parameters */ > @@ -3346,10 +3488,16 @@ raise_stmt: > ; > > signal_stmt: > - SIGNAL_SYM signal_value opt_set_signal_information > + SIGNAL_SYM > { > - if (Lex->add_signal_statement(thd, $2)) > + if (Lex->main_select_push()) > + YYABORT; > + } > + signal_value opt_set_signal_information > + { > + if (Lex->add_signal_statement(thd, $3)) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > } > ; > > @@ -3475,9 +3623,14 @@ resignal_stmt: > ; > > get_diagnostics: > - GET_SYM which_area DIAGNOSTICS_SYM diagnostics_information > + GET_SYM which_area DIAGNOSTICS_SYM > + { > + if (Lex->main_select_push()) > + YYABORT; > + } > + diagnostics_information > { > - Diagnostics_information *info= $4; > + Diagnostics_information *info= $5; > > info->set_which_da($2); > > @@ -3486,6 +3639,7 @@ get_diagnostics: > > if (Lex->m_sql_cmd == NULL) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > } > ; > > @@ -3666,8 +3820,26 @@ sp_decl_idents: > > sp_opt_default: > /* Empty */ { $$ = NULL; } > - | DEFAULT expr { $$ = $2; } > - | SET_VAR expr { $$ = $2; } > + | DEFAULT > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > + expr > + { > + Lex->pop_select(); //main select > + $$ = $3; > + } > + | SET_VAR > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > + expr > + { > + Lex->pop_select(); //main select > + $$ = $3; > + } > ; > > sp_proc_stmt: > @@ -3784,10 +3956,15 @@ sp_proc_stmt_statement: > > sp_proc_stmt_return: > RETURN_SYM > - { Lex->sphead->reset_lex(thd); } > + { > + Lex->sphead->reset_lex(thd); > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > expr > { > LEX *lex= Lex; > + lex->pop_select(); //main select > sp_head *sp= lex->sphead; > if (sp->m_handler->add_instr_freturn(thd, sp, lex->spcont, > $3, lex) || > @@ -3902,6 +4079,8 @@ assignment_source_expr: > { > DBUG_ASSERT(thd->free_list == NULL); > Lex->sphead->reset_lex(thd, $1); > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > expr > { > @@ -3910,6 +4089,7 @@ assignment_source_expr: > $$->sp_lex_in_use= true; > $$->set_item_and_free_list($3, thd->free_list); > thd->free_list= NULL; > + Lex->pop_select(); //main select > if ($$->sphead->restore_lex(thd)) > MYSQL_YYABORT; > } > @@ -4030,9 +4210,14 @@ sp_fetch_list: > ; > > sp_if: > - { Lex->sphead->reset_lex(thd); } > + { > + Lex->sphead->reset_lex(thd); > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > expr THEN_SYM > { > + Lex->pop_select(); //main select > LEX *lex= Lex; > sp_head *sp= lex->sphead; > sp_pcontext *ctx= lex->spcont; > @@ -4144,12 +4329,18 @@ case_stmt_specification: > ; > > case_stmt_body: > - { Lex->sphead->reset_lex(thd); /* For expr $2 */ } > + { > + Lex->sphead->reset_lex(thd); /* For expr $2 */ > + > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > expr > { > if (Lex->case_stmt_action_expr($2)) > MYSQL_YYABORT; > > + Lex->pop_select(); //main select > if (Lex->sphead->restore_lex(thd)) > MYSQL_YYABORT; > } > @@ -4173,6 +4364,9 @@ simple_when_clause: > WHEN_SYM > { > Lex->sphead->reset_lex(thd); /* For expr $3 */ > + > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > expr > { > @@ -4181,6 +4375,8 @@ simple_when_clause: > LEX *lex= Lex; > if (lex->case_stmt_action_when($3, true)) > MYSQL_YYABORT; > + > + lex->pop_select(); //main select > /* For expr $3 */ > if (lex->sphead->restore_lex(thd)) > MYSQL_YYABORT; > @@ -4197,12 +4393,17 @@ searched_when_clause: > WHEN_SYM > { > Lex->sphead->reset_lex(thd); /* For expr $3 */ > + > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > expr > { > LEX *lex= Lex; > if (lex->case_stmt_action_when($3, false)) > MYSQL_YYABORT; > + > + lex->pop_select(); //main select > /* For expr $3 */ > if (lex->sphead->restore_lex(thd)) > MYSQL_YYABORT; > @@ -4442,6 +4643,7 @@ while_body: > LEX *lex= Lex; > if (lex->sp_while_loop_expression(thd, $1)) > MYSQL_YYABORT; > + Lex->pop_select(); //main select pushed before while_body use > if (lex->sphead->restore_lex(thd)) > MYSQL_YYABORT; > } > @@ -4454,7 +4656,12 @@ while_body: > > repeat_body: > sp_proc_stmts1 UNTIL_SYM > - { Lex->sphead->reset_lex(thd); } > + { > + Lex->sphead->reset_lex(thd); > + > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > expr END REPEAT_SYM > { > LEX *lex= Lex; > @@ -4465,6 +4672,8 @@ repeat_body: > if (i == NULL || > lex->sphead->add_instr(i)) > MYSQL_YYABORT; > + > + lex->pop_select(); //main select > if (lex->sphead->restore_lex(thd)) > MYSQL_YYABORT; > /* We can shortcut the cont_backpatch here */ > @@ -4493,8 +4702,12 @@ sp_labeled_control: > if (Lex->sp_push_loop_label(thd, &$1)) > MYSQL_YYABORT; > Lex->sphead->reset_lex(thd); > + > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > while_body pop_sp_loop_label > + > { } > | labels_declaration_oracle FOR_SYM > { > @@ -4546,9 +4759,13 @@ sp_unlabeled_control: > if (Lex->sp_push_loop_empty_label(thd)) > MYSQL_YYABORT; > Lex->sphead->reset_lex(thd); > + > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > while_body > { > + // while body pop main select > Lex->sp_pop_loop_empty_label(thd); > } > | FOR_SYM > @@ -4981,20 +5198,10 @@ size_number: > */ > > create_body: > - '(' create_field_list ')' > + create_field_list_parens > { Lex->create_info.option_list= NULL; } > opt_create_table_options opt_create_partitioning > opt_create_select {} > | opt_create_table_options opt_create_partitioning > opt_create_select {} > - /* > - the following rule is redundant, but there's a shift/reduce > - conflict that prevents the rule above from parsing a syntax like > - CREATE TABLE t1 (SELECT 1); > - */ > - | '(' create_select_query_specification ')' > - | '(' create_select_query_specification ')' > - { Select->set_braces(1);} union_list {} > - | '(' create_select_query_specification ')' > - { Select->set_braces(1);} union_order_or_limit {} > | create_like > { > > @@ -5010,7 +5217,7 @@ create_body: > > create_like: > LIKE table_ident { $$= $2; } > - | '(' LIKE table_ident ')' { $$= $3; } > + | LEFT_PAREN_LIKE LIKE table_ident ')' { $$= $3; } > ; > > opt_create_select: > @@ -5019,23 +5226,42 @@ opt_create_select: > ; > > create_select_query_expression: > - opt_with_clause SELECT_SYM create_select_part2 > opt_table_expression > - create_select_part4 > - { > - Select->set_braces(0); > - Select->set_with_clause($1); > + query_expression > + { > + SELECT_LEX *first_select= $1->first_select(); > + > + if (Lex->sql_command == SQLCOM_INSERT || > + Lex->sql_command == SQLCOM_REPLACE) > + { > + if (Lex->sql_command == SQLCOM_INSERT) > + Lex->sql_command= SQLCOM_INSERT_SELECT; > + else > + Lex->sql_command= SQLCOM_REPLACE_SELECT; > + } > + Lex->insert_select_hack(first_select); > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + > } > - union_clause > - | opt_with_clause SELECT_SYM create_select_part2 > - create_select_part3_union_not_ready create_select_part4 > + | LEFT_PAREN_WITH with_clause query_expression_body ')' > { > - Select->set_with_clause($1); > + SELECT_LEX *first_select= $3->first_select(); > + $3->set_with_clause($2); > + $2->attach_to(first_select); > + > + Lex->insert_select_hack(first_select); > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + > + if (Lex->sql_command == SQLCOM_INSERT || > + Lex->sql_command == SQLCOM_REPLACE) > + { > + if (Lex->sql_command == SQLCOM_INSERT) > + Lex->sql_command= SQLCOM_INSERT_SELECT; > + else > + Lex->sql_command= SQLCOM_REPLACE_SELECT; > + } > } > - | '(' create_select_query_specification ')' > - | '(' create_select_query_specification ')' > - { Select->set_braces(1);} union_list {} > - | '(' create_select_query_specification ')' > - { Select->set_braces(1);} union_order_or_limit {} > ; > > opt_create_partitioning: > @@ -5126,8 +5352,13 @@ partition_entry: > We enter here when opening the frm file to translate > partition info string into part_info data structure. > */ > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > + partition > + { > + Lex->pop_select(); //main select > } > - partition {} > ; > > partition: > @@ -5742,56 +5973,6 @@ opt_part_option: > End of partition parser part > */ > > -create_select_query_specification: > - opt_with_clause SELECT_SYM create_select_part2 > create_select_part3 > - create_select_part4 > - { > - Select->set_with_clause($1); > - } > - ; > - > -create_select_part2: > - { > - LEX *lex=Lex; > - if (lex->sql_command == SQLCOM_INSERT) > - lex->sql_command= SQLCOM_INSERT_SELECT; > - else if (lex->sql_command == SQLCOM_REPLACE) > - lex->sql_command= SQLCOM_REPLACE_SELECT; > - /* > - The following work only with the local list, the global list > - is created correctly in this case > - */ > - > lex->current_select->table_list.save_and_clear(&lex->save_list); > - mysql_init_select(lex); > - lex->current_select->parsing_place= SELECT_LIST; > - } > - select_options select_item_list > - { > - Select->parsing_place= NO_MATTER; > - } > - ; > - > -create_select_part3: > - opt_table_expression > - | create_select_part3_union_not_ready > - ; > - > -create_select_part3_union_not_ready: > - table_expression order_or_limit > - | order_or_limit > - ; > - > -create_select_part4: > - opt_select_lock_type > - { > - /* > - The following work only with the local list, the global list > - is created correctly in this case > - */ > - Lex->current_select->table_list.push_front(&Lex->save_list); > - } > - ; > - > opt_as: > /* empty */ {} > | AS {} > @@ -6195,6 +6376,13 @@ create_field_list: > } > ; > > +create_field_list_parens: > + LEFT_PAREN_ALT field_list ')' > + { > + Lex->create_last_non_select_table= Lex->last_table(); > + } > + ; > + > field_list: > field_list_item > | field_list ',' field_list_item > @@ -6462,6 +6650,8 @@ parse_vcol_expr: > Prevent the end user from invoking this command. > */ > MYSQL_YYABORT_UNLESS(Lex->parse_vcol_expr); > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > expr > { > @@ -6469,13 +6659,29 @@ parse_vcol_expr: > if (!v) > MYSQL_YYABORT; > Lex->last_field->vcol_info= v; > + Lex->pop_select(); //main select > } > ; > > parenthesized_expr: > - subselect > + remember_tok_start > + query_expression > { > - $$= new (thd->mem_root) Item_singlerow_subselect(thd, $1); > + if (!Lex->expr_allows_subselect || > + Lex->sql_command == (int)SQLCOM_PURGE) > + { > + thd->parse_error(ER_SYNTAX_ERROR, $1); > + MYSQL_YYABORT; > + } > + > + // Add the subtree of subquery to the current SELECT_LEX > + SELECT_LEX *curr_sel= Lex->select_stack_head(); > + DBUG_ASSERT(Lex->current_select == curr_sel); > + curr_sel->register_unit($2, &curr_sel->context); > + curr_sel->add_statistics($2); > + > + $$= new (thd->mem_root) > + Item_singlerow_subselect(thd, $2->first_select()); > if ($$ == NULL) > MYSQL_YYABORT; > } > @@ -7474,6 +7680,8 @@ alter: > Lex->alter_info.reset(); > Lex->no_write_to_binlog= 0; > Lex->create_info.storage_media= HA_SM_DEFAULT; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > DBUG_ASSERT(!Lex->m_sql_cmd); > } > alter_options TABLE_SYM table_ident opt_lock_wait_timeout > @@ -7494,12 +7702,15 @@ alter: > Lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_alter_table(); > if (Lex->m_sql_cmd == NULL) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > } > } > | ALTER DATABASE ident_or_empty > { > Lex->create_info.default_table_charset= NULL; > Lex->create_info.used_fields= 0; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > create_database_options > { > @@ -7508,6 +7719,7 @@ alter: > lex->name= $3; > if (lex->name.str == NULL && lex->copy_db_to(&lex->name)) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > } > | ALTER DATABASE ident UPGRADE_SYM DATA_SYM DIRECTORY_SYM NAME_SYM > { > @@ -7523,6 +7735,8 @@ alter: > > if (lex->sphead) > my_yyabort_error((ER_SP_NO_DROP_SP, MYF(0), "PROCEDURE")); > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > lex->sp_chistics.init(); > } > sp_a_chistics > @@ -7531,6 +7745,9 @@ alter: > > lex->sql_command= SQLCOM_ALTER_PROCEDURE; > lex->spname= $3; > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > } > | ALTER FUNCTION_SYM sp_name > { > @@ -7538,6 +7755,8 @@ alter: > > if (lex->sphead) > my_yyabort_error((ER_SP_NO_DROP_SP, MYF(0), "FUNCTION")); > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > lex->sp_chistics.init(); > } > sp_a_chistics > @@ -7546,14 +7765,23 @@ alter: > > lex->sql_command= SQLCOM_ALTER_FUNCTION; > lex->spname= $3; > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > } > | ALTER view_algorithm definer_opt opt_view_suid VIEW_SYM > table_ident > { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > if (Lex->add_alter_view(thd, $2, $4, $6)) > MYSQL_YYABORT; > } > view_list_opt AS view_select > - {} > + { > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + } > | ALTER definer_opt opt_view_suid VIEW_SYM table_ident > /* > We have two separate rules for ALTER VIEW rather that > @@ -7561,14 +7789,22 @@ alter: > with the ALTER EVENT below. > */ > { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > if (Lex->add_alter_view(thd, VIEW_ALGORITHM_INHERIT, $3, $5)) > MYSQL_YYABORT; > } > view_list_opt AS view_select > - {} > + { > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + } > | ALTER definer_opt remember_name EVENT_SYM sp_name > { > - /* > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + /* > It is safe to use Lex->spname because > ALTER EVENT xxx RENATE TO yyy DO ALTER EVENT RENAME TO > is not allowed. Lex->spname is used in the case of RENAME TO > @@ -7600,6 +7836,8 @@ alter: > */ > Lex->sql_command= SQLCOM_ALTER_EVENT; > Lex->stmt_definition_end= (char*)YYLIP->get_cpp_ptr(); > + > + Lex->pop_select(); //main select > } > | ALTER TABLESPACE alter_tablespace_info > { > @@ -7643,6 +7881,8 @@ alter: > lex->create_info.init(); > lex->no_write_to_binlog= 0; > DBUG_ASSERT(!lex->m_sql_cmd); > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > table_ident > { > @@ -7660,6 +7900,9 @@ alter: > Lex->m_sql_cmd= new (thd->mem_root) > Sql_cmd_alter_sequence($3); > if (Lex->m_sql_cmd == NULL) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > } > ; > > @@ -8336,9 +8579,13 @@ checksum: > lex->sql_command = SQLCOM_CHECKSUM; > /* Will be overridden during execution. */ > YYPS->m_lock_type= TL_UNLOCK; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > table_list opt_checksum_type > - {} > + { > + Lex->pop_select(); //main select > + } > ; > > opt_checksum_type: > @@ -8364,6 +8611,8 @@ repair: > lex->alter_info.reset(); > /* Will be overridden during execution. */ > YYPS->m_lock_type= TL_UNLOCK; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > repair_table_or_view > { > @@ -8372,6 +8621,7 @@ repair: > lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_repair_table(); > if (lex->m_sql_cmd == NULL) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > } > ; > > @@ -8400,6 +8650,8 @@ analyze: > ANALYZE_SYM opt_no_write_to_binlog table_or_tables > { > LEX *lex=Lex; > + if (lex->main_select_push()) > + YYABORT; > lex->sql_command = SQLCOM_ANALYZE; > lex->no_write_to_binlog= $2; > lex->check_opt.init(); > @@ -8414,6 +8666,7 @@ analyze: > lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_analyze_table(); > if (lex->m_sql_cmd == NULL) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > } > ; > > @@ -8530,6 +8783,8 @@ check: CHECK_SYM > lex->alter_info.reset(); > /* Will be overridden during execution. */ > YYPS->m_lock_type= TL_UNLOCK; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > check_view_or_table > { > @@ -8540,6 +8795,7 @@ check: CHECK_SYM > lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_check_table(); > if (lex->m_sql_cmd == NULL) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > } > ; > > @@ -8577,6 +8833,8 @@ optimize: > lex->alter_info.reset(); > /* Will be overridden during execution. */ > YYPS->m_lock_type= TL_UNLOCK; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > table_list opt_lock_wait_timeout > { > @@ -8585,6 +8843,7 @@ optimize: > lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_optimize_table(); > if (lex->m_sql_cmd == NULL) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > } > ; > > @@ -8598,9 +8857,13 @@ rename: > RENAME table_or_tables > { > Lex->sql_command= SQLCOM_RENAME_TABLE; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > table_to_table_list > - {} > + { > + Lex->pop_select(); //main select > + } > | RENAME USER_SYM clear_privileges rename_list > { > Lex->sql_command = SQLCOM_RENAME_USER; > @@ -8694,10 +8957,14 @@ preload: > LEX *lex=Lex; > lex->sql_command=SQLCOM_PRELOAD_KEYS; > lex->alter_info.reset(); > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > preload_list_or_parts > - {} > - ; > + { > + Lex->pop_select(); //main select > + } > + ; > > preload_list_or_parts: > preload_keys_parts > @@ -8763,192 +9030,353 @@ opt_ignore_leaves: > > > select: > - opt_with_clause select_init > + query_expression > { > - LEX *lex= Lex; > - lex->sql_command= SQLCOM_SELECT; > - lex->current_select->set_with_clause($1); > + Lex->selects_allow_into= TRUE; > + Lex->selects_allow_procedure= TRUE; > + Lex->set_main_unit($1); > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + > + SELECT_LEX *fake= Lex->unit.fake_select_lex; > + if (fake) > + { > + fake->no_table_names_allowed= 1; > + Lex->push_select(fake); > + } > + else > + Lex->push_select(Lex->first_select_lex()); > + Lex->sql_command= SQLCOM_SELECT; > } > ; > > -select_init: > - SELECT_SYM select_options_and_item_list select_init3 > - | '(' select_paren ')' > - | '(' select_paren ')' union_list > - | '(' select_paren ')' union_order_or_limit > - ; > > -union_list_part2: > - SELECT_SYM select_options_and_item_list > select_init3_union_query_term > - | '(' select_paren_union_query_term ')' > - | '(' select_paren_union_query_term ')' union_list > - | '(' select_paren_union_query_term ')' union_order_or_limit > +simple_table: > + query_specification { $$= $1; } > + | table_value_constructor { $$= $1; } > ; > - > -select_paren: > + > +table_value_constructor: > + VALUES > + { > + LEX *lex= Lex; > + SELECT_LEX *sel; > + //lex->field_list.empty(); > + lex->many_values.empty(); > + lex->insert_list=0; > + if (!(sel= lex->alloc_select(TRUE)) || > + lex->push_select(sel)) > + MYSQL_YYABORT; > + sel->init_select(); > + sel->braces= FALSE; // just initialisation > + } > + values_list > + { > + LEX *lex=Lex; > + $$= lex->pop_select(); // above TVC select > + if (!($$->tvc= > + new (lex->thd->mem_root) > table_value_constr(lex->many_values, > + $$, > + > $$->options))) > + MYSQL_YYABORT; > + lex->many_values.empty(); > + } > + ; > + > +query_specification: > + SELECT_SYM > { > - /* > - In order to correctly parse UNION's global ORDER BY we > need to > - set braces before parsing the clause. > - */ > - Lex->current_select->set_braces(true); > + SELECT_LEX *sel; > + LEX *lex= Lex; > + if (!(sel= lex->alloc_select(TRUE)) || > + lex->push_select(sel)) > + MYSQL_YYABORT; > + sel->init_select(); > + sel->braces= FALSE; > } > - SELECT_SYM select_options_and_item_list select_part3 > - opt_select_lock_type > + select_options > { > - DBUG_ASSERT(Lex->current_select->braces); > + Select->parsing_place= SELECT_LIST; > } > - | '(' select_paren ')' > - ; > - > -select_paren_union_query_term: > + select_item_list > { > - /* > - In order to correctly parse UNION's global ORDER BY we > need to > - set braces before parsing the clause. > - */ > - Lex->current_select->set_braces(true); > + Select->parsing_place= NO_MATTER; > } > - SELECT_SYM select_options_and_item_list > select_part3_union_query_term > - opt_select_lock_type > + opt_into > + opt_from_clause > + opt_where_clause > + opt_group_clause > + opt_having_clause > + opt_window_clause > { > - DBUG_ASSERT(Lex->current_select->braces); > + $$= Lex->pop_select(); > } > - | '(' select_paren_union_query_term ')' > ; > > -select_paren_view: > - { > - /* > - In order to correctly parse UNION's global ORDER BY we > need to > - set braces before parsing the clause. > - */ > - Lex->current_select->set_braces(true); > - } > - SELECT_SYM select_options_and_item_list select_part3_view > - opt_select_lock_type > - { > - DBUG_ASSERT(Lex->current_select->braces); > - } > - | '(' select_paren_view ')' > +opt_from_clause: > + /* Empty */ > + | from_clause > + ; > + > +query_primary: > + simple_table > + { $$= $1; } > + | query_primary_parens > + { $$= $1; } > ; > > -/* The equivalent of select_paren for nested queries. */ > -select_paren_derived: > +query_primary_parens: > + '(' query_expression_unit > { > - Lex->current_select->set_braces(true); > + SELECT_LEX *last= $2->pre_last_parse->next_select(); > + int cmp= > cmp_unit_op($2->first_select()->next_select()->linkage, > + last->linkage); > + if (cmp < 0) > + { > + if (!check_intersect_prefix($2->first_select())) > + { > + if (Lex->pop_new_select_and_wrap() == NULL) > + MYSQL_YYABORT; > + } > + } > + Lex->push_select($2->fake_select_lex); > } > - SELECT_SYM select_part2_derived > - opt_table_expression > - opt_order_clause > - opt_limit_clause > - opt_select_lock_type > + query_expression_tail ')' > { > - DBUG_ASSERT(Lex->current_select->braces); > - $$= Lex->current_select->master_unit()->first_select(); > + Lex->pop_select(); > + if ($4) > + { > + ($4)->set_to($2->fake_select_lex); > + } > + $$= $2->first_select(); > } > - | '(' select_paren_derived ')' { $$= $2; } > - ; > - > -select_init3: > - opt_table_expression > - opt_select_lock_type > + | '(' query_primary > { > - /* Parentheses carry no meaning here */ > - Lex->current_select->set_braces(false); > + Lex->push_select($2); > } > - union_clause > - | select_part3_union_not_ready > - opt_select_lock_type > + query_expression_tail ')' > { > - /* Parentheses carry no meaning here */ > - Lex->current_select->set_braces(false); > + Lex->pop_select(); > + $$= $2; > + $$->braces= TRUE; > + if ($4) > + { > + if ($2->next_select()) > + { > + SELECT_LEX_UNIT *unit= $2->master_unit(); > + if (!unit) > + unit= Lex->create_unit($2); > + if (!unit) > + YYABORT; > + if (!unit->fake_select_lex->is_set_query_expr_tail) > + $4->set_to(unit->fake_select_lex); > + else > + { > + $$= Lex->wrap_unit_into_derived(unit); > + if (!$$) > + YYABORT; > + $4->set_to($$); > + } > + } > + else if (!$2->is_set_query_expr_tail) > + { > + $4->set_to($2); > + } > + else > + { > + SELECT_LEX_UNIT *unit= Lex->create_unit($2); > + if (!unit) > + YYABORT; > + $$= Lex->wrap_unit_into_derived(unit); > + if (!$$) > + YYABORT; > + $4->set_to($$); > + } > + } > } > ; > > - > -select_init3_union_query_term: > - opt_table_expression > - opt_select_lock_type > +query_expression_unit: > + query_primary > + unit_type_decl > + query_primary > { > - /* Parentheses carry no meaning here */ > - Lex->current_select->set_braces(false); > + SELECT_LEX *sel1; > + SELECT_LEX *sel2; > + if (!$1->next_select()) > + sel1= $1; > + else > + { > + sel1= Lex->wrap_unit_into_derived($1->master_unit()); > + if (!sel1) > + YYABORT; > + } > + if (!$3->next_select()) > + sel2= $3; > + else > + { > + sel2= Lex->wrap_unit_into_derived($3->master_unit()); > + if (!sel2) > + YYABORT; > + } > + sel1->link_neighbour(sel2); > + sel2->set_linkage_and_distinct($2.unit_type, $2.distinct); > + $$= Lex->create_unit(sel1); > + $$->pre_last_parse= sel1; > + if ($$ == NULL) > + YYABORT; > } > - union_clause > - | select_part3_union_not_ready_noproc > - opt_select_lock_type > + | query_expression_unit > + unit_type_decl > + query_primary > { > - /* Parentheses carry no meaning here */ > - Lex->current_select->set_braces(false); > + SELECT_LEX *sel1; > + if (!$3->next_select()) > + sel1= $3; > + else > + { > + sel1= Lex->wrap_unit_into_derived($3->master_unit()); > + if (!sel1) > + YYABORT; > + } > + SELECT_LEX *last= $1->pre_last_parse->next_select(); > + > + int cmp= cmp_unit_op($2.unit_type, last->linkage); > + if (cmp == 0) > + { > + // do nothing, this part will be just connected > + } > + else if (cmp > 0) > + { > + // Store beginning and continue to connect parts > + if (Lex->push_new_select($1->pre_last_parse)) > + MYSQL_YYABORT; > + } > + else /* cmp < 0 */ > + { > + // wrap stored part in a select, then continue to connect > parts > + if (!check_intersect_prefix($1->first_select())) > + { > + if ((last= Lex->pop_new_select_and_wrap()) == NULL) > + MYSQL_YYABORT; > + last->set_master_unit($1); > + } > + } > + last->link_neighbour(sel1); > + sel1->set_master_unit($1); > + sel1->set_linkage_and_distinct($2.unit_type, $2.distinct); > + $$= $1; > + $$->pre_last_parse= last; > } > ; > > - > -select_init3_view: > - opt_table_expression opt_select_lock_type > +query_expression_body: > + query_primary > { > - Lex->current_select->set_braces(false); > + Lex->push_select($1); > } > - | opt_table_expression opt_select_lock_type > + query_expression_tail > { > - Lex->current_select->set_braces(false); > + Lex->pop_select(); > + SELECT_LEX *sel= $1; > + if ($3) > + { > + if ($1->next_select()) > + { > + SELECT_LEX_UNIT *unit= $1->master_unit(); > + if (!unit) > + unit= Lex->create_unit($1); > + if (!unit) > + YYABORT; > + if (!unit->fake_select_lex->is_set_query_expr_tail) > + $3->set_to(unit->fake_select_lex); > + else > + { > + SELECT_LEX *sel= Lex->wrap_unit_into_derived(unit); > + if (!sel) > + YYABORT; > + $3->set_to(sel); > + } > + } > + else if (!$1->is_set_query_expr_tail) > + $3->set_to($1); > + else > + { > + SELECT_LEX_UNIT *unit= $1->master_unit(); > + if (!unit) > + unit= Lex->create_unit($1); > + if (!unit) > + YYABORT; > + sel= Lex->wrap_unit_into_derived(unit); > + if (!sel) > + YYABORT; > + $3->set_to(sel); > + } > + } > + $$= Lex->create_unit(sel); > + if ($$ == NULL) > + YYABORT; > } > - union_list_view > - | order_or_limit opt_select_lock_type > + | query_expression_unit > { > - Lex->current_select->set_braces(false); > + SELECT_LEX *last= $1->pre_last_parse->next_select(); > + int cmp= > cmp_unit_op($1->first_select()->next_select()->linkage, > + last->linkage); > + if (cmp < 0) > + { > + if (!check_intersect_prefix($1->first_select())) > + { > + if (Lex->pop_new_select_and_wrap() == NULL) > + MYSQL_YYABORT; > + } > + } > + Lex->push_select($1->fake_select_lex); > } > - | table_expression order_or_limit opt_select_lock_type > + query_expression_tail > { > - Lex->current_select->set_braces(false); > + Lex->pop_select(); > + if ($3) > + { > + ($3)->set_to($1->fake_select_lex); > + } > + $$= $1; > } > ; > > -/* > - The SELECT parts after select_item_list that cannot be followed by > UNION. > -*/ > - > -select_part3: > - opt_table_expression > - | select_part3_union_not_ready > - ; > - > -select_part3_union_query_term: > - opt_table_expression > - | select_part3_union_not_ready_noproc > - ; > - > -select_part3_view: > - opt_table_expression > - | order_or_limit > - | table_expression order_or_limit > +query_expression: > + opt_with_clause > + query_expression_body > + { > + if ($1) > + { > + $2->set_with_clause($1); > + $1->attach_to($2->first_select()); > + } > + $$= $2; > + } > ; > > -select_part3_union_not_ready: > - select_part3_union_not_ready_noproc > - | table_expression procedure_clause > - | table_expression order_or_limit procedure_clause > - ; > +subselect: > + remember_tok_start > + query_expression > + { > + if (!Lex->expr_allows_subselect || > + Lex->sql_command == (int)SQLCOM_PURGE) > + { > + thd->parse_error(ER_SYNTAX_ERROR, $1); > + MYSQL_YYABORT; > + } > > -select_part3_union_not_ready_noproc: > - order_or_limit > - | into opt_table_expression opt_order_clause opt_limit_clause > - | table_expression into > - | table_expression order_or_limit > - | table_expression order_or_limit into > - ; > + // Add the subtree of subquery to the current SELECT_LEX > + SELECT_LEX *curr_sel= Lex->select_stack_head(); > + DBUG_ASSERT(Lex->current_select == curr_sel); > + if (curr_sel) > + { > + curr_sel->register_unit($2, &curr_sel->context); > + curr_sel->add_statistics($2); > + } > > -select_options_and_item_list: > - { > - LEX *lex= Lex; > - SELECT_LEX *sel= lex->current_select; > - if (sel->linkage != UNION_TYPE) > - mysql_init_select(lex); > - lex->current_select->parsing_place= SELECT_LIST; > - } > - select_options select_item_list > - { > - Select->parsing_place= NO_MATTER; > + $$= $2->first_select(); > } > ; > > @@ -8956,18 +9384,6 @@ select_options_and_item_list: > /** > <table expression>, as in the SQL standard. > */ > -table_expression: > - from_clause > - opt_where_clause > - opt_group_clause > - opt_having_clause > - opt_window_clause > - ; > - > -opt_table_expression: > - /* Empty */ > - | table_expression > - ; > > from_clause: > FROM table_reference_list > @@ -9005,59 +9421,63 @@ select_option: > query_expression_option > | SQL_NO_CACHE_SYM > { > - /* > - Allow this flag only on the first top-level SELECT > statement, if > - SQL_CACHE wasn't specified, and only once per query. > - */ > - if (Lex->current_select != &Lex->builtin_select) > - my_yyabort_error((ER_CANT_USE_OPTION_HERE, MYF(0), > "SQL_NO_CACHE")); > - if (Lex->builtin_select.sql_cache == SELECT_LEX::SQL_CACHE) > - my_yyabort_error((ER_WRONG_USAGE, MYF(0), "SQL_CACHE", > "SQL_NO_CACHE")); > - if (Lex->builtin_select.sql_cache == SELECT_LEX::SQL_NO_CACHE) > + /* > + Allow this flag once per query. > + */ > + if (Select->options & OPTION_NO_QUERY_CACHE) > my_yyabort_error((ER_DUP_ARGUMENT, MYF(0), "SQL_NO_CACHE")); > - > - Lex->safe_to_cache_query=0; > - Lex->builtin_select.options&= ~OPTION_TO_QUERY_CACHE; > - Lex->builtin_select.sql_cache= SELECT_LEX::SQL_NO_CACHE; > + Select->options|= OPTION_NO_QUERY_CACHE; > } > | SQL_CACHE_SYM > { > - /* > - Allow this flag only on the first top-level SELECT > statement, if > - SQL_NO_CACHE wasn't specified, and only once per query. > - */ > - if (Lex->current_select != &Lex->builtin_select) > - my_yyabort_error((ER_CANT_USE_OPTION_HERE, MYF(0), > "SQL_CACHE")); > - if (Lex->builtin_select.sql_cache == SELECT_LEX::SQL_NO_CACHE) > - my_yyabort_error((ER_WRONG_USAGE, MYF(0), "SQL_NO_CACHE", > "SQL_CACHE")); > - if (Lex->builtin_select.sql_cache == SELECT_LEX::SQL_CACHE) > + /* > + Allow this flag once per query. > + */ > + if (Select->options & OPTION_TO_QUERY_CACHE) > my_yyabort_error((ER_DUP_ARGUMENT, MYF(0), "SQL_CACHE")); > - > - Lex->safe_to_cache_query=1; > - Lex->builtin_select.options|= OPTION_TO_QUERY_CACHE; > - Lex->builtin_select.sql_cache= SELECT_LEX::SQL_CACHE; > + Select->options|= OPTION_TO_QUERY_CACHE; > } > ; > > opt_select_lock_type: > /* empty */ > - | FOR_SYM UPDATE_SYM opt_lock_wait_timeout > + { $$.empty(); } > + | select_lock_type > + { $$= $1; } > + ; > + > +select_lock_type: > + FOR_SYM UPDATE_SYM opt_lock_wait_timeout_new > { > - LEX *lex=Lex; > - lex->current_select->lock_type= TL_WRITE; > - lex->current_select->set_lock_for_tables(TL_WRITE); > - lex->safe_to_cache_query=0; > + $$= $3; > + $$.defined_lock= TRUE; > + $$.update_lock= TRUE; > } > - | LOCK_SYM IN_SYM SHARE_SYM MODE_SYM opt_lock_wait_timeout > + | LOCK_SYM IN_SYM SHARE_SYM MODE_SYM opt_lock_wait_timeout_new > { > - LEX *lex=Lex; > - lex->current_select->lock_type= TL_READ_WITH_SHARED_LOCKS; > - lex->current_select-> > - set_lock_for_tables(TL_READ_WITH_SHARED_LOCKS); > - lex->safe_to_cache_query=0; > + $$= $5; > + $$.defined_lock= TRUE; > + $$.update_lock= FALSE; > } > ; > > +opt_lock_wait_timeout_new: > + /* empty */ > + { > + $$.empty(); > + } > + | WAIT_SYM ulong_num > + { > + $$.defined_timeout= TRUE; > + $$.timeout= $2; > + } > + | NOWAIT_SYM > + { > + $$.defined_timeout= TRUE; > + $$.timeout= 0; > + } > + ; > + > select_item_list: > select_item_list ',' select_item > | select_item > @@ -11361,10 +11781,15 @@ esc_table_ref: > /* Equivalent to <table reference list> in the SQL:2003 standard. */ > /* Warning - may return NULL in case of incomplete SELECT */ > derived_table_list: > - esc_table_ref { $$=$1; } > + esc_table_ref > + { > + $$=$1; > + Select->add_joined_table($1); > + } > | derived_table_list ',' esc_table_ref > { > MYSQL_YYABORT_UNLESS($1 && ($$=$3)); > + Select->add_joined_table($3); > } > ; > > @@ -11383,11 +11808,18 @@ join_table: > left-associative joins. > */ > table_ref normal_join table_ref %prec TABLE_REF_PRIORITY > - { MYSQL_YYABORT_UNLESS($1 && ($$=$3)); $3->straight=$2; } > + { > + MYSQL_YYABORT_UNLESS($1 && ($$=$3)); > + Select->add_joined_table($1); > + Select->add_joined_table($3); > + $3->straight=$2; > + } > | table_ref normal_join table_ref > ON > { > MYSQL_YYABORT_UNLESS($1 && $3); > + Select->add_joined_table($1); > + Select->add_joined_table($3); > /* Change the current name resolution context to a local > context. */ > if (push_new_name_resolution_context(thd, $1, $3)) > MYSQL_YYABORT; > @@ -11404,6 +11836,8 @@ join_table: > USING > { > MYSQL_YYABORT_UNLESS($1 && $3); > + Select->add_joined_table($1); > + Select->add_joined_table($3); > } > '(' using_list ')' > { > @@ -11414,6 +11848,8 @@ join_table: > | table_ref NATURAL inner_join table_factor > { > MYSQL_YYABORT_UNLESS($1 && ($$=$4)); > + Select->add_joined_table($1); > + Select->add_joined_table($4); > $4->straight=$3; > add_join_natural($1,$4,NULL,Select); > } > @@ -11423,6 +11859,8 @@ join_table: > ON > { > MYSQL_YYABORT_UNLESS($1 && $5); > + Select->add_joined_table($1); > + Select->add_joined_table($5); > /* Change the current name resolution context to a local > context. */ > if (push_new_name_resolution_context(thd, $1, $5)) > MYSQL_YYABORT; > @@ -11439,6 +11877,8 @@ join_table: > | table_ref LEFT opt_outer JOIN_SYM table_factor > { > MYSQL_YYABORT_UNLESS($1 && $5); > + Select->add_joined_table($1); > + Select->add_joined_table($5); > } > USING '(' using_list ')' > { > @@ -11449,6 +11889,8 @@ join_table: > | table_ref NATURAL LEFT opt_outer JOIN_SYM table_factor > { > MYSQL_YYABORT_UNLESS($1 && $6); > + Select->add_joined_table($1); > + Select->add_joined_table($6); > add_join_natural($1,$6,NULL,Select); > $6->outer_join|=JOIN_TYPE_LEFT; > $$=$6; > @@ -11459,6 +11901,8 @@ join_table: > ON > { > MYSQL_YYABORT_UNLESS($1 && $5); > + Select->add_joined_table($1); > + Select->add_joined_table($5); > /* Change the current name resolution context to a local > context. */ > if (push_new_name_resolution_context(thd, $1, $5)) > MYSQL_YYABORT; > @@ -11476,6 +11920,8 @@ join_table: > | table_ref RIGHT opt_outer JOIN_SYM table_factor > { > MYSQL_YYABORT_UNLESS($1 && $5); > + Select->add_joined_table($1); > + Select->add_joined_table($5); > } > USING '(' using_list ')' > { > @@ -11487,6 +11933,8 @@ join_table: > | table_ref NATURAL RIGHT opt_outer JOIN_SYM table_factor > { > MYSQL_YYABORT_UNLESS($1 && $6); > + Select->add_joined_table($1); > + Select->add_joined_table($6); > add_join_natural($6,$1,NULL,Select); > LEX *lex= Lex; > if (!($$= lex->current_select->convert_right_join())) > @@ -11521,40 +11969,70 @@ use_partition: > $$= $3; > } > ; > - > -/* > - This is a flattening of the rules <table factor> and <table primary> > - in the SQL:2003 standard, since we don't have <sample clause> > > - I.e. > - <table factor> ::= <table primary> [ <sample clause> ] > -*/ > -/* Warning - may return NULL in case of incomplete SELECT */ > table_factor: > - table_primary_ident > - | table_primary_derived > + table_primary_ident { $$= $1; } > + | table_primary_derived { $$= $1; } > + | join_table_parens { $$= $1; } > + | table_reference_list_parens { $$= $1; } > ; > > +table_reference_list_parens: > + '(' table_reference_list_parens ')' { $$= $2; } > + | '(' nested_table_reference_list ')' > + { > + if (!($$= Select->end_nested_join(thd))) > + MYSQL_YYABORT; > + } > + ; > + > +nested_table_reference_list: > + table_ref ',' table_ref > + { > + if (Select->init_nested_join(thd)) > + MYSQL_YYABORT; > + Select->add_joined_table($1); > + Select->add_joined_table($3); > + $$= $1->embedding; > + } > + | nested_table_reference_list ',' table_ref > + { > + Select->add_joined_table($3); > + $$= $1; > + } > + ; > + > +join_table_parens: > + '(' join_table_parens ')' { $$= $2; } > + | '(' join_table ')' > + { > + LEX *lex= Lex; > + if (!($$= lex->current_select->nest_last_join(thd))) > + { > + thd->parse_error(); > + MYSQL_YYABORT; > + } > + } > + ; > + > + > table_primary_ident: > + table_ident opt_use_partition > + opt_table_alias_clause opt_key_definition > { > SELECT_LEX *sel= Select; > sel->table_join_options= 0; > - } > - table_ident opt_use_partition opt_table_alias opt_key_definition > - { > - if (!($$= Select->add_table_to_list(thd, $2, $4, > + if (!($$= Select->add_table_to_list(thd, $1, $3, > > Select->get_table_join_options(), > YYPS->m_lock_type, > YYPS->m_mdl_type, > Select->pop_index_hints(), > - $3))) > + $2))) > MYSQL_YYABORT; > - Select->add_joined_table($$); > } > ; > > > - > /* > Represents a flattening of the following rules from the SQL:2003 > standard. This sub-rule corresponds to the sub-rule > @@ -11572,242 +12050,56 @@ table_primary_ident: > */ > > table_primary_derived: > - '(' get_select_lex select_derived_union ')' opt_table_alias > + query_primary_parens table_alias_clause > { > - /* Use $2 instead of Lex->current_select as derived table will > - alter value of Lex->current_select. */ > - if (!($3 || $5) && $2->embedding && > - !$2->embedding->nested_join->join_list.elements) > + LEX *lex=Lex; > + lex->derived_tables|= DERIVED_SUBQUERY; > + $1->linkage= DERIVED_TABLE_TYPE; > + $1->braces= FALSE; > + // Add the subtree of subquery to the current SELECT_LEX > + SELECT_LEX *curr_sel= Lex->select_stack_head(); > + DBUG_ASSERT(Lex->current_select == curr_sel); > + SELECT_LEX_UNIT *unit= $1->master_unit(); > + if (!unit) > { > - /* we have a derived table ($3 == NULL) but no alias, > - Since we are nested in further parentheses so we > - can pass NULL to the outer level parentheses > - Permits parsing of "((((select ...))) as xyz)" */ > - $$= 0; > + unit= Lex->create_unit($1); > + if (!unit) > + YYABORT; > } > - else if (!$3) > - { > - /* Handle case of derived table, alias may be NULL if there > - are no outer parentheses, add_table_to_list() will throw > - error in this case */ > - LEX *lex=Lex; > - lex->check_automatic_up(UNSPECIFIED_TYPE); > - SELECT_LEX *sel= lex->current_select; > - SELECT_LEX_UNIT *unit= sel->master_unit(); > - lex->current_select= sel= unit->outer_select(); > - Table_ident *ti= new (thd->mem_root) Table_ident(unit); > - if (ti == NULL) > - MYSQL_YYABORT; > - if (!($$= sel->add_table_to_list(thd, > - ti, $5, 0, > - TL_READ, MDL_SHARED_READ))) > + curr_sel->register_unit(unit, &curr_sel->context); > + curr_sel->add_statistics(unit); > > - MYSQL_YYABORT; > - sel->add_joined_table($$); > - //lex->pop_context("derived"); > - lex->nest_level--; > - } > - else if ($5 != NULL) > - { > - /* > - Tables with or without joins within parentheses cannot > - have aliases, and we ruled out derived tables above. > - */ > - thd->parse_error(); > - MYSQL_YYABORT; > - } > - else > - { > - /* nested join: FROM (t1 JOIN t2 ...), > - nest_level is the same as in the outer query */ > - $$= $3; > - } > - /* > - Fields in derived table can be used in upper select in > - case of merge. We do not add HAVING fields because we do > - not merge such derived. We do not add union because > - also do not merge them > - */ > - if ($$ && $$->derived && > - !$$->derived->first_select()->next_select()) > - > $$->select_lex->add_where_field($$->derived->first_select()); > - } > - /* Represents derived table with WITH clause */ > - | '(' get_select_lex subselect_start > - with_clause query_expression_body > - subselect_end ')' opt_table_alias > - { > - LEX *lex=Lex; > - SELECT_LEX *sel= $2; > - SELECT_LEX_UNIT *unit= $5->master_unit(); > Table_ident *ti= new (thd->mem_root) Table_ident(unit); > if (ti == NULL) > MYSQL_YYABORT; > - $5->set_with_clause($4); > - lex->current_select= sel; > - if (!($$= sel->add_table_to_list(lex->thd, > - ti, $8, 0, > - TL_READ, MDL_SHARED_READ))) > + if (!($$= curr_sel->add_table_to_list(lex->thd, > + ti, $2, 0, > + TL_READ, > MDL_SHARED_READ))) > MYSQL_YYABORT; > - sel->add_joined_table($$); > - } > - ; > - > -/* > - This rule accepts just about anything. The reason is that we have > - empty-producing rules in the beginning of rules, in this case > - subselect_start. This forces bison to take a decision which rules to > - reduce by long before it has seen any tokens. This approach ties us > - to a very limited class of parseable languages, and unfortunately > - SQL is not one of them. The chosen 'solution' was this rule, which > - produces just about anything, even complete bogus statements, for > - instance ( table UNION SELECT 1 ). > - Fortunately, we know that the semantic value returned by > - select_derived is NULL if it contained a derived table, and a pointer to > - the base table's TABLE_LIST if it was a base table. So in the rule > - regarding union's, we throw a parse error manually and pretend it > - was bison that did it. > - > - Also worth noting is that this rule concerns query expressions in > - the from clause only. Top level select statements and other types of > - subqueries have their own union rules. > -*/ > -select_derived_union: > - select_derived > - | select_derived union_order_or_limit > - { > - if ($1) > - { > - thd->parse_error(); > - MYSQL_YYABORT; > - } > - } > - | select_derived union_head_non_top > - { > - if ($1) > - { > - thd->parse_error(); > - MYSQL_YYABORT; > - } > } > - union_list_derived_part2 > - | derived_query_specification opt_select_lock_type > - | derived_query_specification order_or_limit opt_select_lock_type > - | derived_query_specification opt_select_lock_type > union_list_derived > - ; > - > -union_list_derived_part2: > - query_term_union_not_ready { Lex->pop_context(); } > - | query_term_union_ready { Lex->pop_context(); } > - | query_term_union_ready { Lex->pop_context(); } > union_list_derived > - ; > - > -union_list_derived: > - union_head_non_top union_list_derived_part2 > - ; > - > - > -/* The equivalent of select_init2 for nested queries. */ > -select_init2_derived: > - select_part2_derived > - { > - Select->set_braces(0); > - } > - ; > - > -/* The equivalent of select_part2 for nested queries. */ > -select_part2_derived: > - { > - LEX *lex= Lex; > - SELECT_LEX *sel= lex->current_select; > - if (sel->linkage != UNION_TYPE) > - mysql_init_select(lex); > - lex->current_select->parsing_place= SELECT_LIST; > - } > - opt_query_expression_options select_item_list > - { > - Select->parsing_place= NO_MATTER; > - } > - ; > - > -/* handle contents of parentheses in join expression */ > -select_derived: > - get_select_lex_derived derived_table_list > + | '(' > + query_expression > + ')' table_alias_clause > { > - LEX *lex= Lex; > - /* for normal joins, $2 != NULL and end_nested_join() != NULL, > - for derived tables, both must equal NULL */ > + LEX *lex=Lex; > + lex->derived_tables|= DERIVED_SUBQUERY; > + $2->first_select()->linkage= DERIVED_TABLE_TYPE; > > - if (!($$= $1->end_nested_join(lex->thd)) && $2) > - MYSQL_YYABORT; > - if (!$2 && $$) > - { > - thd->parse_error(); > - MYSQL_YYABORT; > - } > - } > - ; > > -/* > - Similar to query_specification, but for derived tables. > - Example: the inner parenthesized SELECT in this query: > - SELECT * FROM (SELECT * FROM t1); > -*/ > -derived_query_specification: > - SELECT_SYM select_derived_init select_derived2 > - { > - if ($2) > - Select->set_braces(1); > - $$= NULL; > - } > - ; > + // Add the subtree of subquery to the current SELECT_LEX > + SELECT_LEX *curr_sel= Lex->select_stack_head(); > + DBUG_ASSERT(Lex->current_select == curr_sel); > + curr_sel->register_unit($2, &curr_sel->context); > + curr_sel->add_statistics($2); > > -select_derived2: > - { > - LEX *lex= Lex; > - lex->derived_tables|= DERIVED_SUBQUERY; > - if (!lex->expr_allows_subselect || > - lex->sql_command == (int)SQLCOM_PURGE) > - { > - thd->parse_error(); > - MYSQL_YYABORT; > - } > - if (lex->current_select->linkage == GLOBAL_OPTIONS_TYPE || > - mysql_new_select(lex, 1, NULL)) > + Table_ident *ti= new (thd->mem_root) Table_ident($2); > + if (ti == NULL) > MYSQL_YYABORT; > - mysql_init_select(lex); > - lex->current_select->set_linkage(DERIVED_TABLE_TYPE); > - lex->current_select->parsing_place= SELECT_LIST; > - } > - select_options select_item_list > - { > - Select->parsing_place= NO_MATTER; > - } > - opt_table_expression > - ; > - > -get_select_lex: > - /* Empty */ { $$= Select; } > - ; > - > -get_select_lex_derived: > - get_select_lex > - { > - LEX *lex= Lex; > - if ($1->init_nested_join(lex->thd)) > + if (!($$= curr_sel->add_table_to_list(lex->thd, > + ti, $4, 0, > + TL_READ, > MDL_SHARED_READ))) > MYSQL_YYABORT; > } > - ; > - > -select_derived_init: > - { > - LEX *lex= Lex; > - > - TABLE_LIST *embedding= lex->current_select->embedding; > - $$= embedding && > - !embedding->nested_join->join_list.elements; > - /* return true if we are deeply nested */ > - } > ; > > opt_outer: > @@ -11939,9 +12231,14 @@ table_alias: > | '=' > ; > > -opt_table_alias: > +opt_table_alias_clause: > /* empty */ { $$=0; } > - | table_alias ident > + > + | table_alias_clause { $$= $1; } > + ; > + > +table_alias_clause: > + table_alias ident > { > $$= (LEX_CSTRING*) thd->memdup(&$2,sizeof(LEX_STRING)); > if ($$ == NULL) > @@ -12108,7 +12405,7 @@ opt_window_partition_clause: > > opt_window_order_clause: > /* empty */ { } > - | ORDER_SYM BY order_list > + | ORDER_SYM BY order_list { Select->order_list= *($3); } > ; > > opt_window_frame_clause: > @@ -12232,64 +12529,35 @@ alter_order_item: > > opt_order_clause: > /* empty */ > + { $$= NULL; } > | order_clause > + { $$= $1; } > ; > > order_clause: > ORDER_SYM BY > { > - LEX *lex=Lex; > - SELECT_LEX *sel= lex->current_select; > - SELECT_LEX_UNIT *unit= sel-> master_unit(); > - if (sel->linkage != GLOBAL_OPTIONS_TYPE && > - sel->olap != UNSPECIFIED_OLAP_TYPE && > - (sel->linkage != UNION_TYPE || sel->braces)) > - { > - my_error(ER_WRONG_USAGE, MYF(0), > - "CUBE/ROLLUP", "ORDER BY"); > - MYSQL_YYABORT; > - } > - if (lex->sql_command != SQLCOM_ALTER_TABLE && > - !unit->fake_select_lex) > - { > - /* > - A query of the of the form (SELECT ...) ORDER BY > order_list is > - executed in the same way as the query > - SELECT ... ORDER BY order_list > - unless the SELECT construct contains ORDER BY or LIMIT > clauses. > - Otherwise we create a fake SELECT_LEX if it has not > been created > - yet. > - */ > - SELECT_LEX *first_sl= unit->first_select(); > - if (!unit->is_unit_op() && > - (first_sl->order_list.elements || > - first_sl->select_limit) && > - unit->add_fake_select_lex(thd)) > - MYSQL_YYABORT; > - } > - if (sel->master_unit()->is_unit_op() && !sel->braces) > - { > - /* > - At this point we don't know yet whether this is the last > - select in union or not, but we move ORDER BY to > - fake_select_lex anyway. If there would be one more select > - in union mysql_new_select will correctly throw error. > - */ > - DBUG_ASSERT(sel->master_unit()->fake_select_lex); > - lex->current_select= sel->master_unit()->fake_select_lex; > - } > + thd->where= "ORDER clause"; > } > order_list > { > - > + $$= $4; > } > ; > > order_list: > order_list ',' order_ident order_dir > - { if (add_order_to_list(thd, $3,(bool) $4)) MYSQL_YYABORT; } > + { > + $$= $1; > + if (add_to_list(thd, *$$, $3,(bool) $4)) > + MYSQL_YYABORT; > + } > | order_ident order_dir > - { if (add_order_to_list(thd, $1,(bool) $2)) MYSQL_YYABORT; } > + { > + $$= new (thd->mem_root) SQL_I_List<ORDER>(); > + if (add_to_list(thd, *$$, $1, (bool) $2)) > + MYSQL_YYABORT; > + } > ; > > order_dir: > @@ -12299,63 +12567,61 @@ order_dir: > ; > > opt_limit_clause: > - /* empty */ {} > - | limit_clause {} > + /* empty */ > + { $$.empty(); } > + | limit_clause > + { $$= $1; } > ; > > -limit_clause_init: > - LIMIT > - { > - SELECT_LEX *sel= Select; > - if (sel->master_unit()->is_unit_op() && !sel->braces) > - { > - /* Move LIMIT that belongs to UNION to fake_select_lex */ > - Lex->current_select= sel->master_unit()->fake_select_lex; > - DBUG_ASSERT(Select); > - } > - } > - ; > - > limit_clause: > - limit_clause_init limit_options > + LIMIT limit_options > { > - SELECT_LEX *sel= Select; > - if (!sel->select_limit->basic_const_item() || > - sel->select_limit->val_int() > 0) > + $$= $2; > + if (!$$.select_limit->basic_const_item() || > + $$.select_limit->val_int() > 0) > Lex->set_stmt_unsafe(LEX::BINLOG_STMT_UNSAFE_LIMIT); > } > - | limit_clause_init limit_options > + | LIMIT limit_options > ROWS_SYM EXAMINED_SYM limit_rows_option > { > + $$= $2; > Lex->set_stmt_unsafe(LEX::BINLOG_STMT_UNSAFE_LIMIT); > } > - | limit_clause_init ROWS_SYM EXAMINED_SYM limit_rows_option > + | LIMIT ROWS_SYM EXAMINED_SYM limit_rows_option > { > + $$.select_limit= 0; > + $$.offset_limit= 0; > + $$.explicit_limit= 1; > Lex->set_stmt_unsafe(LEX::BINLOG_STMT_UNSAFE_LIMIT); > } > ; > > +opt_global_limit_clause: > + opt_limit_clause > + { > + Select->explicit_limit= $1.explicit_limit; > + Select->select_limit= $1.select_limit; > + Select->offset_limit= $1.offset_limit; > + } > + > limit_options: > limit_option > { > - SELECT_LEX *sel= Select; > - sel->select_limit= $1; > - sel->offset_limit= 0; > - sel->explicit_limit= 1; > + $$.select_limit= $1; > + $$.offset_limit= 0; > + $$.explicit_limit= 1; > } > | limit_option ',' limit_option > { > - SELECT_LEX *sel= Select; > - sel->select_limit= $3; > - sel->offset_limit= $1; > - sel->explicit_limit= 1; > + $$.select_limit= $3; > + $$.offset_limit= $1; > + $$.explicit_limit= 1; > } > | limit_option OFFSET_SYM limit_option > { > - SELECT_LEX *sel= Select; > - sel->select_limit= $1; > - sel->offset_limit= $3; > - sel->explicit_limit= 1; > + $$.select_limit= $1; > + $$.offset_limit= $3; > + $$.explicit_limit= 1; > } > ; > > @@ -12424,6 +12690,66 @@ delete_limit_clause: > | LIMIT limit_option ROWS_SYM EXAMINED_SYM { thd->parse_error(); > MYSQL_YYABORT; } > ; > > +query_expression_tail: > + /* empty */ { $$= NULL; } > + | order_or_limit opt_select_lock_type > + { > + $$= $1; > + $$->lock= $2; > + } > + | order_or_limit procedure_or_into opt_select_lock_type > + { > + $$= $1; > + $$->lock= $3; > + } > + | procedure_or_into opt_select_lock_type > + { > + $$= new(thd->mem_root) Lex_order_limit_lock; > + if (!$$) > + YYABORT; > + $$->order_list= NULL; > + $$->limit.empty(); > + $$->lock= $2; > + } > + | select_lock_type > + { > + $$= new(thd->mem_root) Lex_order_limit_lock; > + if (!$$) > + YYABORT; > + $$->order_list= NULL; > + $$->limit.empty(); > + $$->lock= $1; > + } > + ; > + > +procedure_or_into: > + procedure_clause > + | into > + | procedure_clause into > + ; > + > +order_or_limit: > + order_clause opt_limit_clause > + { > + $$= new(thd->mem_root) Lex_order_limit_lock; > + if (!$$) > + YYABORT; > + $$->order_list= $1; > + $$->limit= $2; > + } > + | limit_clause > + { > + Lex_order_limit_lock *op= $$= new(thd->mem_root) > Lex_order_limit_lock; > + if (!$$) > + YYABORT; > + op->order_list= NULL; > + op->limit= $1; > + $$->order_list= NULL; > + $$->limit= $1; > + } > + ; > + > + > opt_plus: > /* empty */ > | '+' > @@ -12493,14 +12819,11 @@ bool: > | TRUE_SYM { $$= 1; } > | FALSE_SYM { $$= 0; } > > - > procedure_clause: > PROCEDURE_SYM ident /* Procedure name */ > { > LEX *lex=Lex; > > - DBUG_ASSERT(&lex->builtin_select == lex->current_select); > - > lex->proc_list.elements=0; > lex->proc_list.first=0; > lex->proc_list.next= &lex->proc_list.first; > @@ -12520,6 +12843,7 @@ procedure_clause: > parameters are reduced. > */ > Lex->expr_allows_subselect= false; > + Select->options|= OPTION_PROCEDURE_CLAUSE; > } > '(' procedure_list ')' > { > @@ -12600,8 +12924,21 @@ select_outvar: > } > ; > > +opt_into: > + /* empty */ > + | into > + ; > into: > INTO into_destination > + { > + if (!(Select->options & OPTION_INTO_CLAUSE)) > + Select->options|= OPTION_INTO_CLAUSE; > + else > + { > + my_error(ER_CANT_USE_OPTION_HERE, MYF(0), "INTO"); > + MYSQL_YYABORT; > + } > + } > ; > > into_destination: > @@ -12647,10 +12984,15 @@ do: > LEX *lex=Lex; > lex->sql_command = SQLCOM_DO; > mysql_init_select(lex); > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > expr_list > { > Lex->insert_list= $3; > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > } > ; > > @@ -12882,17 +13224,24 @@ insert: > { > LEX *lex= Lex; > lex->sql_command= SQLCOM_INSERT; > - lex->duplicates= DUP_ERROR; > - mysql_init_select(lex); > + lex->duplicates= DUP_ERROR; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + mysql_init_select(lex); > + lex->current_select->parsing_place= BEFORE_OPT_LIST; > } > insert_lock_option > opt_ignore insert2 > { > Select->set_lock_for_tables($3); > - Lex->current_select= &Lex->builtin_select; > + Lex->current_select= Lex->first_select_lex(); > } > insert_field_spec opt_insert_update > - {} > + { > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + } > ; > > replace: > @@ -12901,15 +13250,22 @@ replace: > LEX *lex=Lex; > lex->sql_command = SQLCOM_REPLACE; > lex->duplicates= DUP_REPLACE; > - mysql_init_select(lex); > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + mysql_init_select(lex); > + lex->current_select->parsing_place= BEFORE_OPT_LIST; > } > replace_lock_option insert2 > { > Select->set_lock_for_tables($3); > - Lex->current_select= &Lex->builtin_select; > + Lex->current_select= Lex->first_select_lex(); > } > insert_field_spec > - {} > + { > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + } > ; > > insert_lock_option: > @@ -12955,35 +13311,47 @@ insert_table: > table_name_with_opt_use_partition > { > LEX *lex=Lex; > - lex->field_list.empty(); > + //lex->field_list.empty(); > lex->many_values.empty(); > lex->insert_list=0; > }; > > insert_field_spec: > insert_values {} > - | '(' ')' insert_values {} > - | '(' fields ')' insert_values {} > + | insert_field_list insert_values {} > | SET > { > LEX *lex=Lex; > if (!(lex->insert_list= new (thd->mem_root) List_item) || > lex->many_values.push_back(lex->insert_list, > thd->mem_root)) > MYSQL_YYABORT; > + lex->current_select->parsing_place= NO_MATTER; > } > ident_eq_list > ; > > +insert_field_list: > + LEFT_PAREN_ALT opt_fields ')' > + { > + Lex->current_select->parsing_place= AFTER_LIST; > + } > + ; > + > +opt_fields: > + /* empty */ > + | fields > + ; > + > fields: > fields ',' insert_ident > { Lex->field_list.push_back($3, thd->mem_root); } > | insert_ident { Lex->field_list.push_back($1, thd->mem_root); } > ; > > + > + > insert_values: > - VALUES values_list {} > - | VALUE_SYM values_list {} > - | create_select_query_expression {} > + create_select_query_expression {} > ; > > values_list: > @@ -13093,6 +13461,8 @@ update: > UPDATE_SYM > { > LEX *lex= Lex; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > mysql_init_select(lex); > lex->sql_command= SQLCOM_UPDATE; > lex->duplicates= DUP_ERROR; > @@ -13118,7 +13488,14 @@ update: > */ > Select->set_lock_for_tables($3); > } > - opt_where_clause opt_order_clause delete_limit_clause {} > + opt_where_clause opt_order_clause delete_limit_clause > + { > + if ($10) > + Select->order_list= *($10); > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + } > ; > > update_list: > @@ -13164,6 +13541,8 @@ delete: > mysql_init_select(lex); > YYPS->m_lock_type= TL_WRITE_DEFAULT; > YYPS->m_mdl_type= MDL_SHARED_WRITE; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > > lex->ignore= 0; > lex->builtin_select.init_order(); > @@ -13185,7 +13564,12 @@ single_multi: > } > opt_where_clause opt_order_clause > delete_limit_clause {} > - opt_select_expressions {} > + opt_select_expressions > + { > + if ($6) > + Select->order_list= *($6); > + Lex->pop_select(); //main select > + } > | table_wild_list > { > mysql_init_multi_delete(Lex); > @@ -13196,6 +13580,9 @@ single_multi: > { > if (multi_delete_set_locks_and_link_aux_tables(Lex)) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > } > | FROM table_alias_ref_list > { > @@ -13207,6 +13594,9 @@ single_multi: > { > if (multi_delete_set_locks_and_link_aux_tables(Lex)) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > } > ; > > @@ -13284,6 +13674,7 @@ truncate: > lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_truncate_table(); > if (lex->m_sql_cmd == NULL) > MYSQL_YYABORT; > + Lex->pop_select(); //main select > } > opt_truncate_table_storage_clause { } > ; > @@ -13365,6 +13756,8 @@ show: > LEX *lex=Lex; > lex->wild=0; > lex->ident= null_clex_str; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > mysql_init_select(lex); > lex->current_select->parsing_place= SELECT_LIST; > lex->create_info.init(); > @@ -13372,6 +13765,7 @@ show: > show_param > { > Select->parsing_place= NO_MATTER; > + Lex->pop_select(); //main select > } > ; > > @@ -13387,7 +13781,7 @@ show_param: > { > LEX *lex= Lex; > lex->sql_command= SQLCOM_SHOW_TABLES; > - lex->builtin_select.db= $3; > + lex->first_select_lex()->db= $3; > if (prepare_schema_table(thd, lex, 0, SCH_TABLE_NAMES)) > MYSQL_YYABORT; > } > @@ -13395,7 +13789,7 @@ show_param: > { > LEX *lex= Lex; > lex->sql_command= SQLCOM_SHOW_TRIGGERS; > - lex->builtin_select.db= $3; > + lex->first_select_lex()->db= $3; > if (prepare_schema_table(thd, lex, 0, SCH_TRIGGERS)) > MYSQL_YYABORT; > } > @@ -13403,7 +13797,7 @@ show_param: > { > LEX *lex= Lex; > lex->sql_command= SQLCOM_SHOW_EVENTS; > - lex->builtin_select.db= $2; > + lex->first_select_lex()->db= $2; > if (prepare_schema_table(thd, lex, 0, SCH_EVENTS)) > MYSQL_YYABORT; > } > @@ -13411,7 +13805,7 @@ show_param: > { > LEX *lex= Lex; > lex->sql_command= SQLCOM_SHOW_TABLE_STATUS; > - lex->builtin_select.db= $3; > + lex->first_select_lex()->db= $3; > if (prepare_schema_table(thd, lex, 0, SCH_TABLES)) > MYSQL_YYABORT; > } > @@ -13419,7 +13813,7 @@ show_param: > { > LEX *lex= Lex; > lex->sql_command= SQLCOM_SHOW_OPEN_TABLES; > - lex->builtin_select.db= $3; > + lex->first_select_lex()->db= $3; > if (prepare_schema_table(thd, lex, 0, SCH_OPEN_TABLES)) > MYSQL_YYABORT; > } > @@ -13469,12 +13863,13 @@ show_param: > LEX *lex= Lex; > lex->sql_command= SQLCOM_SHOW_BINLOG_EVENTS; > } > - opt_limit_clause > + opt_global_limit_clause > | RELAYLOG_SYM optional_connection_name EVENTS_SYM binlog_in > binlog_from > { > LEX *lex= Lex; > lex->sql_command= SQLCOM_SHOW_RELAYLOG_EVENTS; > - } opt_limit_clause > + } > + opt_global_limit_clause > | keys_or_index from_or_in table_ident opt_db opt_where_clause > { > LEX *lex= Lex; > @@ -13516,13 +13911,13 @@ show_param: > LEX_CSTRING var= {STRING_WITH_LEN("error_count")}; > (void) create_select_for_variable(thd, &var); > } > - | WARNINGS opt_limit_clause > + | WARNINGS opt_global_limit_clause > { Lex->sql_command = SQLCOM_SHOW_WARNS;} > - | ERRORS opt_limit_clause > + | ERRORS opt_global_limit_clause > { Lex->sql_command = SQLCOM_SHOW_ERRORS;} > | PROFILES_SYM > { Lex->sql_command = SQLCOM_SHOW_PROFILES; } > - | PROFILE_SYM opt_profile_defs opt_profile_args opt_limit_clause > + | PROFILE_SYM opt_profile_defs opt_profile_args > opt_global_limit_clause > { > LEX *lex= Lex; > lex->sql_command= SQLCOM_SHOW_PROFILE; > @@ -13829,7 +14224,7 @@ describe: > explainable_command > { > LEX *lex=Lex; > - lex->builtin_select.options|= SELECT_DESCRIBE; > + lex->first_select_lex()->options|= SELECT_DESCRIBE; > } > ; > > @@ -13855,6 +14250,8 @@ analyze_stmt_command: > > opt_extended_describe: > EXTENDED_SYM { Lex->describe|= DESCRIBE_EXTENDED; } > + | EXTENDED_SYM ALL > + { Lex->describe|= DESCRIBE_EXTENDED | DESCRIBE_EXTENDED2; } > | PARTITIONS_SYM { Lex->describe|= DESCRIBE_PARTITIONS; } > | opt_format_json {} > ; > @@ -13897,8 +14294,7 @@ flush: > lex->type= 0; > lex->no_write_to_binlog= $2; > } > - flush_options > - {} > + flush_options {} > ; > > flush_options: > @@ -13915,6 +14311,7 @@ flush_options: > opt_table_list opt_flush_lock > {} > | flush_options_list > + {} > ; > > opt_flush_lock: > @@ -14070,9 +14467,13 @@ purge: > LEX *lex=Lex; > lex->type=0; > lex->sql_command = SQLCOM_PURGE; > + if (lex->main_select_push()) > + MYSQL_YYABORT; > } > purge_options > - {} > + { > + Lex->pop_select(); //main select > + } > ; > > purge_options: > @@ -14090,6 +14491,8 @@ purge_option: > lex->value_list.empty(); > lex->value_list.push_front($2, thd->mem_root); > lex->sql_command= SQLCOM_PURGE_BEFORE; > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > } > ; > > @@ -14099,6 +14502,8 @@ kill: > KILL_SYM > { > LEX *lex=Lex; > + if (lex->main_select_push()) > + YYABORT; > lex->value_list.empty(); > lex->users_list.empty(); > lex->sql_command= SQLCOM_KILL; > @@ -14107,6 +14512,7 @@ kill: > kill_type kill_option kill_expr > { > Lex->kill_signal= (killed_state) ($3 | $4); > + Lex->pop_select(); //main select > } > ; > > @@ -14167,6 +14573,8 @@ load: > $2 == FILETYPE_CSV ? "LOAD DATA" : "LOAD XML"); > MYSQL_YYABORT; > } > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > load_data_lock opt_local INFILE TEXT_STRING_filesystem > { > @@ -14193,7 +14601,11 @@ load: > opt_xml_rows_identified_by > opt_field_term opt_line_term opt_ignore_lines > opt_field_or_var_spec > opt_load_data_set_spec > - {} > + { > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + } > ; > > data_or_xml: > @@ -14597,17 +15009,21 @@ opt_with_clause: > > > with_clause: > - WITH opt_recursive > + WITH opt_recursive > { > + LEX *lex= Lex; > With_clause *with_clause= > new With_clause($2, Lex->curr_with_clause); > if (with_clause == NULL) > MYSQL_YYABORT; > - Lex->derived_tables|= DERIVED_WITH; > - Lex->curr_with_clause= with_clause; > + lex->derived_tables|= DERIVED_WITH; > + lex->curr_with_clause= with_clause; > with_clause->add_to_list(Lex->with_clauses_list_last_next); > + if (lex->current_select && > + lex->current_select->parsing_place == BEFORE_OPT_LIST) > + lex->current_select->parsing_place= NO_MATTER; > } > - with_list > + with_list > { > $$= Lex->curr_with_clause; > Lex->curr_with_clause= Lex->curr_with_clause->pop(); > @@ -14636,9 +15052,9 @@ with_list_element: > MYSQL_YYABORT; > Lex->with_column_list.empty(); > } > - AS '(' remember_name subselect remember_end ')' > + AS '(' remember_name query_expression remember_end ')' > { > - With_element *elem= new With_element($1, *$2, > $7->master_unit()); > + With_element *elem= new With_element($1, *$2, $7); > if (elem == NULL || Lex->curr_with_clause->add_with_element(elem)) > MYSQL_YYABORT; > if (elem->set_unparsed_spec(thd, $6+1, $8)) > @@ -15589,14 +16005,22 @@ set: > SET > { > LEX *lex=Lex; > + if (lex->main_select_push()) > + MYSQL_YYABORT; > lex->set_stmt_init(); > lex->var_list.empty(); > sp_create_assignment_lex(thd, yychar == YYEMPTY); > } > start_option_value_list > - {} > + { > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + } > | SET STATEMENT_SYM > { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > Lex->set_stmt_init(); > } > set_stmt_option_value_following_option_type_list > @@ -15606,6 +16030,9 @@ set: > my_yyabort_error((ER_SUBQUERIES_NOT_SUPPORTED, MYF(0), > "SET STATEMENT")); > lex->stmt_var_list= lex->var_list; > lex->var_list.empty(); > + Lex->pop_select(); //main select > + if (Lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > } > FOR_SYM verb_clause > {} > @@ -16038,9 +16465,13 @@ lock: > if (lex->sphead) > my_yyabort_error((ER_SP_BADSTATEMENT, MYF(0), "LOCK")); > lex->sql_command= SQLCOM_LOCK_TABLES; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > table_lock_list opt_lock_wait_timeout > - {} > + { > + Lex->pop_select(); //main select > + } > ; > > opt_lock_wait_timeout: > @@ -16071,7 +16502,7 @@ table_lock_list: > ; > > table_lock: > - table_ident opt_table_alias lock_option > + table_ident opt_table_alias_clause lock_option > { > thr_lock_type lock_type= (thr_lock_type) $3; > bool lock_for_write= (lock_type >= TL_WRITE_ALLOW_WRITE); > @@ -16105,9 +16536,13 @@ unlock: > if (lex->sphead) > my_yyabort_error((ER_SP_BADSTATEMENT, MYF(0), "UNLOCK")); > lex->sql_command= SQLCOM_UNLOCK_TABLES; > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > } > table_or_tables > - {} > + { > + Lex->pop_select(); //main select > + } > ; > > /* > @@ -16115,25 +16550,36 @@ unlock: > */ > > handler: > - HANDLER_SYM table_ident OPEN_SYM opt_table_alias > + HANDLER_SYM > + { > + if (Lex->main_select_push()) > + MYSQL_YYABORT; > + } > + handler_tail > + { > + Lex->pop_select(); //main select > + } > + > +handler_tail: > + table_ident OPEN_SYM opt_table_alias_clause > { > LEX *lex= Lex; > if (lex->sphead) > my_yyabort_error((ER_SP_BADSTATEMENT, MYF(0), "HANDLER")); > lex->sql_command = SQLCOM_HA_OPEN; > - if (!lex->current_select->add_table_to_list(thd, $2, $4, 0)) > + if (!lex->current_select->add_table_to_list(thd, $1, $3, 0)) > MYSQL_YYABORT; > } > - | HANDLER_SYM table_ident_nodb CLOSE_SYM > + | table_ident_nodb CLOSE_SYM > { > LEX *lex= Lex; > if (lex->sphead) > my_yyabort_error((ER_SP_BADSTATEMENT, MYF(0), "HANDLER")); > lex->sql_command = SQLCOM_HA_CLOSE; > - if (!lex->current_select->add_table_to_list(thd, $2, 0, 0)) > + if (!lex->current_select->add_table_to_list(thd, $1, 0, 0)) > MYSQL_YYABORT; > } > - | HANDLER_SYM table_ident_nodb READ_SYM > + | table_ident_nodb READ_SYM > { > LEX *lex=Lex; > if (lex->sphead) > @@ -16141,20 +16587,24 @@ handler: > lex->expr_allows_subselect= FALSE; > lex->sql_command = SQLCOM_HA_READ; > lex->ha_rkey_mode= HA_READ_KEY_EXACT; /* Avoid purify > warnings */ > - Item *one= new (thd->mem_root) Item_int(thd, (int32) 1); > - if (one == NULL) > - MYSQL_YYABORT; > - lex->current_select->select_limit= one; > - lex->current_select->offset_limit= 0; > - lex->limit_rows_examined= 0; > - if (!lex->current_select->add_table_to_list(thd, $2, 0, 0)) > + if (!lex->current_select->add_table_to_list(thd, $1, 0, 0)) > MYSQL_YYABORT; > } > - handler_read_or_scan opt_where_clause opt_limit_clause > + handler_read_or_scan opt_where_clause opt_global_limit_clause > { > - Lex->expr_allows_subselect= TRUE; > + LEX *lex=Lex; > + lex->expr_allows_subselect= TRUE; > + if (!lex->current_select->explicit_limit) > + { > + Item *one= new (thd->mem_root) Item_int(thd, (int32) 1); > + if (one == NULL) > + MYSQL_YYABORT; > + lex->current_select->select_limit= one; > + lex->current_select->offset_limit= 0; > + lex->limit_rows_examined= 0; > + } > /* Stored functions are not supported for HANDLER READ. */ > - if (Lex->uses_stored_routines()) > + if (lex->uses_stored_routines()) > { > my_error(ER_NOT_SUPPORTED_YET, MYF(0), > "stored functions in HANDLER ... READ"); > @@ -16800,83 +17250,16 @@ release: > */ > > unit_type_decl: > - UNION_SYM > - { $$= UNION_TYPE; } > + UNION_SYM union_option > + { $$.unit_type= UNION_TYPE; $$.distinct= $2; } > | INTERSECT_SYM > - { $$= INTERSECT_TYPE; } > + { $$.unit_type= INTERSECT_TYPE; $$.distinct= 1; } > | EXCEPT_SYM > - { $$= EXCEPT_TYPE; } > - > - > -union_clause: > - /* empty */ {} > - | union_list > - ; > - > -union_list: > - unit_type_decl union_option > - { > - if (Lex->add_select_to_union_list((bool)$2, $1, TRUE)) > - MYSQL_YYABORT; > - } > - union_list_part2 > - { > - /* > - Remove from the name resolution context stack the context > of the > - last select in the union. > - */ > - Lex->pop_context(); > - } > - ; > - > -union_list_view: > - unit_type_decl union_option > - { > - if (Lex->add_select_to_union_list((bool)$2, $1, TRUE)) > - MYSQL_YYABORT; > - } > - query_expression_body_view > - { > - Lex->pop_context(); > - } > - ; > - > -union_order_or_limit: > - { > - LEX *lex= thd->lex; > - DBUG_ASSERT(lex->current_select->linkage != > GLOBAL_OPTIONS_TYPE); > - SELECT_LEX *sel= lex->current_select; > - SELECT_LEX_UNIT *unit= sel->master_unit(); > - SELECT_LEX *fake= unit->fake_select_lex; > - if (fake) > - { > - fake->no_table_names_allowed= 1; > - lex->current_select= fake; > - } > - thd->where= "global ORDER clause"; > - } > - order_or_limit > - { > - thd->lex->current_select->no_table_names_allowed= 0; > - thd->where= ""; > - } > - ; > - > -order_or_limit: > - order_clause opt_limit_clause > - | limit_clause > - ; > + { $$.unit_type= EXCEPT_TYPE; $$.distinct= 1; } > > /* > Start a UNION, for non-top level query expressions. > */ > -union_head_non_top: > - unit_type_decl union_option > - { > - if (Lex->add_select_to_union_list((bool)$2, $1, FALSE)) > - MYSQL_YYABORT; > - } > - ; > > union_option: > /* empty */ { $$=1; } > @@ -16884,110 +17267,10 @@ union_option: > | ALL { $$=0; } > ; > > -/* > - Corresponds to the SQL Standard > - <query specification> ::= > - SELECT [ <set quantifier> ] <select list> <table expression> > - > - Notes: > - - We allow more options in addition to <set quantifier> > - - <table expression> is optional in MariaDB > -*/ > -query_specification: > - SELECT_SYM select_init2_derived opt_table_expression > - { > - $$= Lex->current_select->master_unit()->first_select(); > - } > - ; > - > -query_term_union_not_ready: > - query_specification order_or_limit opt_select_lock_type { $$= > $1; } > - | '(' select_paren_derived ')' union_order_or_limit { $$= > $2; } > - ; > - > -query_term_union_ready: > - query_specification opt_select_lock_type { $$= > $1; } > - | '(' select_paren_derived ')' { $$= > $2; } > - ; > - > -query_expression_body: > - query_term_union_not_ready { > $$= $1; } > - | query_term_union_ready { > $$= $1; } > - | query_term_union_ready union_list_derived { > $$= $1; } > - ; > - > -/* Corresponds to <query expression> in the SQL:2003 standard. */ > -subselect: > - subselect_start opt_with_clause query_expression_body > subselect_end > - { > - $3->set_with_clause($2); > - $$= $3; > - } > - ; > - > -subselect_start: > - { > - LEX *lex=Lex; > - if (!lex->expr_allows_subselect || > - lex->sql_command == (int)SQLCOM_PURGE) > - { > - thd->parse_error(); > - MYSQL_YYABORT; > - } > - /* > - we are making a "derived table" for the parenthesis > - as we need to have a lex level to fit the union > - after the parenthesis, e.g. > - (SELECT .. ) UNION ... becomes > - SELECT * FROM ((SELECT ...) UNION ...) > - */ > - if (mysql_new_select(Lex, 1, NULL)) > - MYSQL_YYABORT; > - } > - ; > - > -subselect_end: > - { > - LEX *lex=Lex; > - > - lex->check_automatic_up(UNSPECIFIED_TYPE); > - lex->pop_context(); > - SELECT_LEX *child= lex->current_select; > - lex->current_select = > lex->current_select->return_after_parsing(); > - lex->nest_level--; > - lex->current_select->n_child_sum_items += child->n_sum_items; > - /* > - A subselect can add fields to an outer select. Reserve > space for > - them. > - */ > - lex->current_select->select_n_where_fields+= > - child->select_n_where_fields; > - > - /* > - Aggregate functions in having clause may add fields to an > outer > - select. Count them also. > - */ > - lex->current_select->select_n_having_items+= > - child->select_n_having_items; > - } > - ; > - > -opt_query_expression_options: > - /* empty */ > - | query_expression_option_list > - ; > - > -query_expression_option_list: > - query_expression_option_list query_expression_option > - | query_expression_option > - ; > - > query_expression_option: > STRAIGHT_JOIN { Select->options|= SELECT_STRAIGHT_JOIN; } > | HIGH_PRIORITY > { > - if (check_simple_select()) > - MYSQL_YYABORT; > YYPS->m_lock_type= TL_READ_HIGH_PRIORITY; > YYPS->m_mdl_type= MDL_SHARED_READ; > Select->options|= SELECT_HIGH_PRIORITY; > @@ -16996,18 +17279,8 @@ query_expression_option: > | UNIQUE_SYM { Select->options|= SELECT_DISTINCT; } > | SQL_SMALL_RESULT { Select->options|= SELECT_SMALL_RESULT; } > | SQL_BIG_RESULT { Select->options|= SELECT_BIG_RESULT; } > - | SQL_BUFFER_RESULT > - { > - if (check_simple_select()) > - MYSQL_YYABORT; > - Select->options|= OPTION_BUFFER_RESULT; > - } > - | SQL_CALC_FOUND_ROWS > - { > - if (check_simple_select()) > - MYSQL_YYABORT; > - Select->options|= OPTION_FOUND_ROWS; > - } > + | SQL_BUFFER_RESULT { Select->options|= OPTION_BUFFER_RESULT; } > + | SQL_CALC_FOUND_ROWS { Select->options|= OPTION_FOUND_ROWS; } > | ALL { Select->options|= SELECT_ALL; } > ; > > @@ -17095,32 +17368,28 @@ view_select: > lex->parsing_options.allows_variable= FALSE; > lex->create_view->select.str= (char *) YYLIP->get_cpp_ptr(); > } > - opt_with_clause query_expression_body_view view_check_option > + query_expression > + view_check_option > { > LEX *lex= Lex; > + SQL_I_List<TABLE_LIST> *save= > &lex->first_select_lex()->table_list; > + lex->set_main_unit($2); > + if (lex->check_main_unit_semantics()) > + MYSQL_YYABORT; > + lex->first_select_lex()->table_list.push_front(save); > + lex->current_select= Lex->first_select_lex(); > size_t len= YYLIP->get_cpp_ptr() - > lex->create_view->select.str; > void *create_view_select= > thd->memdup(lex->create_view->select.str, len); > lex->create_view->select.length= len; > lex->create_view->select.str= (char *) create_view_select; > + size_t not_used; > trim_whitespace(thd->charset(), > - &lex->create_view->select); > - lex->create_view->check= $4; > + &lex->create_view->select, &not_used); > + lex->create_view->check= $3; > lex->parsing_options.allows_variable= TRUE; > - lex->current_select->set_with_clause($2); > } > ; > > -/* > - SQL Standard <query expression body> for VIEWs. > - Does not include INTO and PROCEDURE clauses. > -*/ > -query_expression_body_view: > - SELECT_SYM select_options_and_item_list select_init3_view > - | '(' select_paren_view ')' > - | '(' select_paren_view ')' union_order_or_limit > - | '(' select_paren_view ')' union_list_view > - ; > - > view_check_option: > /* empty */ { $$= VIEW_CHECK_NONE; } > | WITH CHECK_SYM OPTION { $$= VIEW_CHECK_CASCADED; } > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

2 5

[Maria-developers] MDEV-14448
by Badrul Chowdhury 16 Apr '18

16 Apr '18

Hello, Is anyone working on https://jira.mariadb.org/browse/MDEV-14448? Pointers to documentation on the procedure to adopt upstream fixes would be appreciated also. Thanks, Badrul

2 2

Re: [Maria-developers] Observation for CHECK_CONSTRAINTS MDEV 14474
by Vicențiu Ciorbaru 14 Apr '18

14 Apr '18

Hi Anel! I've reviewed your patch and generally things look good. You do some good analysis usually and I think we should highlight your work to the broader developer scene. I've thus CC'ed this email to the maria-developers mailing list. This gives us a chance to get feedback from outside, particularly as this feature is of general interest to the community. Please include the .result files in future versions as it makes reviewing a lot easier. Serg: coding style and implementation aside, I'll do the review on that bit, please have a look at Anel's patch and suggest if you think the current table header for I_S.check_constraints is good enough for our needs or if you have more input on this task. Relevant bit: +ST_FIELD_INFO check_constraints_fields_info[]= +{ + {"CONSTRAINT_CATALOG", FN_REFLEN, MYSQL_TYPE_STRING, 0, 0, 0, OPEN_FULL_TABLE}, + {"CONSTRAINT_SCHEMA", NAME_CHAR_LEN, MYSQL_TYPE_STRING, 0, 0, 0, + OPEN_FULL_TABLE}, + {"CONSTRAINT_NAME", NAME_CHAR_LEN, MYSQL_TYPE_STRING, 0, 0, 0, + OPEN_FULL_TABLE}, + {"TABLE_SCHEMA", NAME_CHAR_LEN, MYSQL_TYPE_STRING, 0, 0, 0, OPEN_FULL_TABLE}, + {"TABLE_NAME", NAME_CHAR_LEN, MYSQL_TYPE_STRING, 0, 0, 0, OPEN_FULL_TABLE}, + {"CHECK_CLAUSE", NAME_CHAR_LEN, MYSQL_TYPE_STRING, 0, 0, 0, + OPEN_FULL_TABLE}, + {0, 0, MYSQL_TYPE_STRING, 0, 0, 0, SKIP_OPEN_TABLE} +}; Vicențiu On Thu, 29 Mar 2018 at 12:01 Anel Husakovic <anel(a)mariadb.org> wrote: > Hi Vin, > > I'm pleased to announce my new patch prototype for MDEV 14474 where are > implemented field and table constraints: > > https://github.com/an3l/server/commit/f92fd77d2695be52faf280245faa014c44617… > > According to the standard there are only 4 column to represent. > CONSTRAINT_CATALOG, CONSTRAINT_SCHEMA, CONSTRAINT_NAME,CHECK_CLAUSE > But because Serg mentioned that alternative would be to change > *information_schema.table_constraints* with adding only 1 column ( > *check_clause*) I also used 2 additional columns which are represented > also in *table_constraints *and they are TABLE_SHEMA , TABLE_NAME. > > Function of interest is: get_check_constraints_record > > Also I added 2 test cases: > > One that works with only 1 table: MDEV14474-one_table_play.test > <https://github.com/an3l/server/commit/f92fd77d2695be52faf280245faa014c44617…> > Second that works with more tables: MDEV14474-I_S_check_constraint.test > <https://github.com/an3l/server/commit/f92fd77d2695be52faf280245faa014c44617…> > > Last night I had a problem with field constraints. > As a reference I used only way how currently all check constraints are > shown and its obtained via* show create table t;* > > Bellow is shown that field constraints were not visible as well as that > name used for them is *NULL*, at least this is how *show create table* > represented them ( there is no way to give a name to a field constraint* > for example: f int constraint name check(f<0)*). According to my opinion, > because this constraint belongs to the field, it should be also called as a > name of a field and not NULL. > As an argument to my opinion here is mariadb kb > https://mariadb.com/kb/en/library/constraint/ > Please take a look at *constraint **name*, which is field constraint and > error message. > > INSERT INTO t2(name, start_date, end_date) VALUES('Io', '2003-12-15', '2014-11-09');ERROR 4022 (23000): CONSTRAINT `name` failed for `test`.`t2` > > If the name of constraint = name of the field , than definitely it is not > consistent with* show create table* output, and *show create table* > output *needs to be changed*, , but this is question that I would like > to discuss with you ?! > > When you try to use show create table x > [image: image.png] > > > I'm waiting on your reply for global temporary tables as well as > check_constraints and how to proceed further > > *Not understand yet a concept*: > > In *sql/table.h* > There is a struct* st_schema_table *which has fill data and process data. > > How they work, and how to use that on other database for example PSI. > > Regards, > > Anel >

2 1

Re: [Maria-developers] [Commits] a74e3ef17e7: MDEV-14551 Can't find record in table on multi-table update with ORDER BY
by Sergey Petrunia 13 Apr '18

13 Apr '18

Hi Serg, Please find some cosmetic input below. (additionally, we've found a failing example while chatting) On Tue, Apr 10, 2018 at 02:10:17PM +0200, serg(a)mariadb.org wrote: > revision-id: a74e3ef17e7a8693d68001290a8d91b5b206804d (mariadb-10.3.5-138-ga74e3ef17e7) > parent(s): 9bd3af97dffa411657879bbdfd4dfef38c99f7cc > author: Sergei Golubchik > committer: Sergei Golubchik > timestamp: 2018-04-10 14:09:17 +0200 > message: > > MDEV-14551 Can't find record in table on multi-table update with ORDER BY ... > diff --git a/mysql-test/main/multi_update.test b/mysql-test/main/multi_update.test > index 5feebe87a5a..96e78dab82b 100644 > --- a/mysql-test/main/multi_update.test > +++ b/mysql-test/main/multi_update.test > @@ -914,3 +914,14 @@ update t1 set c1=NULL; > update t1, t2 set t1.c1=t2.c3 where t1.c3=t2.c3 order by t1.c3 desc limit 2; > select * from t1; > drop table t1, t2; > + > +# > +# MDEV-14551 Can't find record in table on multi-table update with ORDER BY > +# > +CREATE TABLE t1 (i INT) ENGINE=MEMORY; > +INSERT t1 VALUES (1),(2); > +CREATE TABLE t2 (f INT) ENGINE=MyISAM; > +INSERT t2 VALUES (1),(2); > +UPDATE t1, t2 SET f = 126 ORDER BY f LIMIT 2; > +SELECT * FROM t2; > +DROP TABLE t1, t2; > diff --git a/sql/item.h b/sql/item.h > index 9574bdc63bf..e391e7810c4 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -2023,6 +2023,7 @@ class Item: public Value_source, > { > marker &= ~EXTRACTION_MASK; > } > + virtual TABLE *rowid_table() const { return 0; } I am concerned about this approach bloating class Item's vtable. There is (and likely will ever be) only one class with a different implementation, the check for this is made only in one place - why not check Item's type() or functype(), etc? > }; > > MEM_ROOT *get_thd_memroot(THD *thd); > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index c6b0ae7fba8..9097ffca867 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -5204,3 +5204,23 @@ String *Item_func_dyncol_list::val_str(String *str) > my_free(names); > return NULL; > } > + > +Item_temptable_rowid::Item_temptable_rowid(TABLE *table_arg) > + : Item_str_func(table_arg->in_use), table(table_arg) > +{ > + max_length= table->file->ref_length; > +} > + > +void Item_temptable_rowid::fix_length_and_dec() > +{ > + used_tables_cache= table->map; > + const_item_cache= false; > +} > + > +String *Item_temptable_rowid::val_str(String *str) > +{ > + if (!((null_value= table->null_row))) > + table->file->position(table->record[0]); > + str_value.set((char*)(table->file->ref), max_length, &my_charset_bin); > + return &str_value; > +} > diff --git a/sql/item_strfunc.h b/sql/item_strfunc.h > index 18cda491efd..49de5568696 100644 > --- a/sql/item_strfunc.h > +++ b/sql/item_strfunc.h > @@ -1748,5 +1748,20 @@ class Item_func_dyncol_list: public Item_str_func > { return get_item_copy<Item_func_dyncol_list>(thd, this); } > }; > > -#endif /* ITEM_STRFUNC_INCLUDED */ Please add a note that this is not the "_rowid" that we support in the parser. > +class Item_temptable_rowid :public Item_str_func > +{ > + TABLE *table; > +public: > + Item_temptable_rowid(TABLE *table_arg); > + const Type_handler *type_handler() const { return &type_handler_string; } > + Field *create_tmp_field(bool group, TABLE *table) > + { return create_table_field_from_handler(table); } > + String *val_str(String *str); > + const char *func_name() const { return "<rowid>"; } > + void fix_length_and_dec(); > + Item *get_copy(THD *thd) > + { return get_item_copy<Item_temptable_rowid>(thd, this); } > + TABLE *rowid_table() const { return table; } > +}; > > +#endif /* ITEM_STRFUNC_INCLUDED */ ... > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 796ea569e64..22301319680 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2650,6 +2650,21 @@ bool JOIN::add_having_as_table_cond(JOIN_TAB *tab) > } > > > +bool JOIN::add_fields_for_current_rowid(JOIN_TAB *cur, List<Item> *table_fields) I think ths is actually "current rowids", in plural. As the loop shows, there are multiple current rowids (from different tables). > +{ > + for (JOIN_TAB *tab=join_tab; tab < cur; tab++) Please add a comment: this will not walk into semi-join materialization nests but this is ok because we will never need to save current rowids for those. > + { > + if (!tab->keep_current_rowid) > + continue; > + Item *item= new (thd->mem_root) Item_temptable_rowid(tab->table); > + item->fix_fields(thd, 0); > + table_fields->push_back(item, thd->mem_root); > + cur->tmp_table_param->func_count++; > + } > + return 0; > +} > + > + > /** > Set info for aggregation tables > > diff --git a/sql/sql_select.h b/sql/sql_select.h > index f8911fbba01..1da87bb9d50 100644 > --- a/sql/sql_select.h > +++ b/sql/sql_select.h > @@ -1438,6 +1438,9 @@ class JOIN :public Sql_alloc > > enum { QEP_NOT_PRESENT_YET, QEP_AVAILABLE, QEP_DELETED} have_query_plan; > > + // if keep_current_rowid=true, whether they should be saved in temporary table > + bool tmp_table_keep_current_rowid; Same input as with add_fields_for_current_rowid - use plural. > + > /* > Additional WHERE and HAVING predicates to be considered for IN=>EXISTS > subquery transformation of a JOIN object. > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index 0149c2848c2..0e29f817e56 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -507,14 +507,14 @@ void select_union_recursive::cleanup() > bool select_union_direct::change_result(select_result *new_result) > { > result= new_result; > - return (result->prepare(unit->types, unit) || result->prepare2()); > + return (result->prepare(unit->types, unit) || result->prepare2(0)); Do we really continue to use '0' when 'NULL' should be used? I prefer "NULL". > } > > > bool select_union_direct::postponed_prepare(List<Item> &types) > { > if (result != NULL) > - return (result->prepare(types, unit) || result->prepare2()); > + return (result->prepare(types, unit) || result->prepare2(0)); > else > return false; > } > diff --git a/sql/sql_update.cc b/sql/sql_update.cc > index 38638d3aa1d..4ae7e33b4e3 100644 > --- a/sql/sql_update.cc > +++ b/sql/sql_update.cc > @@ -2205,10 +2195,41 @@ multi_update::initialize_tables(JOIN *join) > DBUG_RETURN(1); > tmp_tables[cnt]->file->extra(HA_EXTRA_WRITE_CACHE); > } > + join->tmp_table_keep_current_rowid= TRUE; > DBUG_RETURN(0); > } > (Follwing explanations on Slack) Please add a comment there, saying that Here we walk through the aggegation temptable to find where rowids are stored. And then for each of the multi-update tables, we get them to copy the data from the rowid field in the aggregation temptable. > + > +int multi_update::prepare2(JOIN *join) > +{ > + if (!join->need_tmp || !join->tmp_table_keep_current_rowid) > + return 0; > + > + // there cannot be many tmp tables in multi-update > + JOIN_TAB *tmptab= join->join_tab + join->exec_join_tab_cnt(); > + > + for (Item **it= tmptab->tmp_table_param->items_to_copy; *it ; it++) > + { > + TABLE *tbl= (*it)->rowid_table(); > + if (!tbl) > + continue; > + for (uint i= 0; i < table_count; i++) > + { > + for (Item **it2= tmp_table_param[i].items_to_copy; *it2; it2++) > + { > + if ((*it2)->rowid_table() != tbl) > + continue; > + *it2= new (thd->mem_root) Item_field(thd, (*it)->get_tmp_table_field()); > + if (!*it2) > + return 1; > + } > + } > + } > + return 0; > +} > + > + > multi_update::~multi_update() > { > TABLE_LIST *table; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 5

[Maria-developers] FOR UPDATE behavior change in 10.3-MDEV-11953
by Alexander Barkov 12 Apr '18

12 Apr '18

Hi Sanja, and Igor, (resending with Igor on CC) I noticed a FOR UPDATE behavior change in 10.3-MDEV-11953 In 10.3 I run this script in a client session: CREATE OR REPLACE TABLE t1 (a INT NOT NULL PRIMARY KEY); CREATE OR REPLACE TABLE t2 (a INT NOT NULL PRIMARY KEY); INSERT INTO t1 VALUES (1); INSERT INTO t2 VALUES (2); BEGIN; SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE; Notice no COMMIT yet! It returns this result: +---+ | a | +---+ | 1 | | 2 | +---+ Now I open a new console, start a new client session and run this script: BEGIN; SELECT * FROM t2 FOR UPDATE; The second session gets locked, as expected. Now I return to the first console and run "COMMIT;". The second console gets unlocked and returns +---+ | a | +---+ | 2 | +---+ Now if I do the same in 10.3-MDEV-11953, the second session does not get locked, it returns the result immediately, even before I typed "COMMIT;" in the first session. Note, if in 10.3-MDEV-11953 I start a transaction with a simpler query (without UNION) in the first session: BEGIN; SELECT * FROM t2 FOR UPDATE; and run the same script in the second session BEGIN; SELECT * FROM t2 FOR UPDATE; then the second session gets locked as expected. Is this behavior change expected? Thanks!

2 1

Re: [Maria-developers] FOR UPDATE behavior change in 10.3-MDEV-11953
by Alexander Barkov 12 Apr '18

12 Apr '18

Hi Igor, Please do "reply all" when replying. On 04/12/2018 01:28 AM, Igor Babaev wrote: > Alexander, > > If you have the same behaviour you have to use > > SELECT * FROM t1 UNION (SELECT * FROM t2 FOR UPDATE). Thanks. This worked fine for me. > > Otherwise it's considered as > > (SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE). What does this syntax mean 10.3-MDEV-11953? Which records should be locked? From t1, from t2, or from both? It seems that it locks nothing. If so, shouldn't this be disallowed? This query (without parentheses) also locks nothing: SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE; > > MySQL manual says that 'FOR UPDATE' must follow ORDER BY, > > but it does not tell you how to interpret FOR UPDATE in unions. The old behavior was to interpret UNION parts separately. This locked only t2 in 10.3: SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE; In 10.3-MDEV-11953 it locks nothing. Upgrade looks dangerous. Can we do this way: 1. In 10.4: disallow FOR UPDATE applied to the entire UNION. 2. In 10.3: Add a warning which tells that this feature is deprecated and suggests to use parentheses around union parts when a non-parenthesized union term is used with FOR UPDATE ? > > Anyway you can't use anymore: > SELECT c FROM t1 ORDER BY a > UNION > SELECT c FROM t2 ORDER BY b > ORDER BY c. > > You have to put it so: > (SELECT c FROM t1 ORDER BY a) > UNION > (SELECT c FROM t2 ORDER BY b) > ORDER BY c. > > (see the standard). I like this change. > > It means that the ORDER BY clause followed a union relates to the union. > The same is true for 'FOR UPDATE' because it follows optional ORDER BY/ > LIMIT clauses. > > > Regards, > Igor. > > > On 04/11/2018 09:10 AM, Alexander Barkov wrote: >> Hi Sanja, and Igor, >> >> >> (resending with Igor on CC) >> >> >> I noticed a FOR UPDATE behavior change in 10.3-MDEV-11953 >> >> In 10.3 I run this script in a client session: >> >> CREATE OR REPLACE TABLE t1 (a INT NOT NULL PRIMARY KEY); >> CREATE OR REPLACE TABLE t2 (a INT NOT NULL PRIMARY KEY); >> INSERT INTO t1 VALUES (1); >> INSERT INTO t2 VALUES (2); >> BEGIN; >> SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE; >> >> Notice no COMMIT yet! It returns this result: >> +---+ >> | a | >> +---+ >> | 1 | >> | 2 | >> +---+ >> >> Now I open a new console, start a new client session and run this script: >> >> BEGIN; >> SELECT * FROM t2 FOR UPDATE; >> >> The second session gets locked, as expected. >> >> Now I return to the first console and run "COMMIT;". >> The second console gets unlocked and returns >> +---+ >> | a | >> +---+ >> | 2 | >> +---+ >> >> >> Now if I do the same in 10.3-MDEV-11953, the second session does not get >> locked, it returns the result immediately, even before I typed "COMMIT;" >> in the first session. >> >> >> Note, if in 10.3-MDEV-11953 I start a transaction with a simpler query >> (without UNION) in the first session: >> >> BEGIN; >> SELECT * FROM t2 FOR UPDATE; >> >> and run the same script in the second session >> >> BEGIN; >> SELECT * FROM t2 FOR UPDATE; >> >> then the second session gets locked as expected. >> >> >> Is this behavior change expected? >> >> >> Thanks! >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers(a)lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp >>

2 3

[Maria-developers] Issue with setting up of MariaDB and solve issue MDEV-14896
by Aman Jain 11 Apr '18

11 Apr '18

Hello developers I am setting up the codebase for MariaDB on my system to understand the codebase and solve some issues. But I am facing a problem which I am not able to solve when I run "make" command. Error log: https://pastebin.com/u5fAhMHX OS: MacOS 10.13.3 Python version: 2.7.14 Please let me know how can I set up so that I can run and test the changes. Also, I want to solve issue MDEV-14896. Please connect me to the owner of the issue so that I can discuss the changes. ( https://jira.mariadb.org/browse/MDEV-14896 ) Thanks Aman Jain

3 5

[Maria-developers] 10.3-MDEV-11953
by Alexander Barkov 11 Apr '18

11 Apr '18

Hi Sanja and Igor, In 10.3-MDEV-11953 you're adding a number of main_select_push() / pop_select() around "expr" and other rules that contain "expr". How do you decide when push/pop is needed. Why, for example, LIMIT and ORDER clause don't have their own push/pop. They do create some expressions (Items). What is the rule? Thanks.

1 0

[Maria-developers] FOR UPDATE behavior change in 10.3-MDEV-11953
by Alexander Barkov 11 Apr '18

11 Apr '18

Hi Sanja, I noticed a FOR UPDATE behavior change in 10.3-MDEV-11953 In 10.3 I run this script in a client session: CREATE OR REPLACE TABLE t1 (a INT NOT NULL PRIMARY KEY); CREATE OR REPLACE TABLE t2 (a INT NOT NULL PRIMARY KEY); INSERT INTO t1 VALUES (1); INSERT INTO t2 VALUES (2); BEGIN; SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE; Notice no COMMIT yet! It returns this result: +---+ | a | +---+ | 1 | | 2 | +---+ Now I open a new console, start a new client session and run this script: BEGIN; SELECT * FROM t2 FOR UPDATE; The second session gets locked, as expected. Now I return to the first console and run "COMMIT;". The second console gets unlocked and returns +---+ | a | +---+ | 2 | +---+ Now if I do the same in 10.3-MDEV-11953, the second session does not get locked, it returns the result immediately, even before I typed "COMMIT;" in the first session. Note, if in 10.3-MDEV-11953 I start a transaction with a simpler query (without UNION) in the first session: BEGIN; SELECT * FROM t2 FOR UPDATE; and run the same script in the second session BEGIN; SELECT * FROM t2 FOR UPDATE; then the second session gets locked as expected. Is this behavior change expected? Thanks!

1 0

Re: [Maria-developers] 57673bc: MDEV-14024 PCRE2
by Sergei Golubchik 11 Apr '18

11 Apr '18

Hi, Alexey! On Apr 03, Alexey Botchkov wrote: > revision-id: 57673bcfeecf51d1dd8f4907adc8ca11d84796d3 (mariadb-10.3.5-76-g57673bc) > parent(s): d614d4b2e61552a5b74259f91b3e14b23707743f > committer: Alexey Botchkov > timestamp: 2018-04-03 02:59:51 +0400 > message: > > MDEV-14024 PCRE2. > > server ocde switched to the PCRE2 features. ^^^ s/ocde/code/ > diff --git a/cmake/pcre.cmake b/cmake/pcre.cmake > index 4c11392..271dd06 100644 > --- a/cmake/pcre.cmake > +++ b/cmake/pcre.cmake > @@ -5,24 +5,16 @@ SET(WITH_PCRE "auto" CACHE STRING > > + IF(NOT HAVE_PCRE2 OR WITH_PCRE STREQUAL "bundled") > IF (WITH_PCRE STREQUAL "system") > - MESSAGE(FATAL_ERROR "system pcre is not found or unusable") > + MESSAGE(FATAL_ERROR "system pcre2-8 is not found or unusable") better "pcre2-8 library" or simply "pcre2" > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index 89aa307..08aa293 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -5481,15 +5481,6 @@ int Regexp_processor_pcre::default_regex_flags() > return default_regex_flags_pcre(current_thd); > } > > -void Regexp_processor_pcre::set_recursion_limit(THD *thd) > -{ > - long stack_used; > - DBUG_ASSERT(thd == current_thd); > - stack_used= available_stack_size(thd->thread_stack, &stack_used); > - m_pcre_extra.match_limit_recursion= > - (ulong)((my_thread_stack_size - STACK_MIN_SIZE - stack_used)/my_pcre_frame_size); > -} > - > > /** > Convert string to lib_charset, if needed. > @@ -5537,19 +5528,32 @@ bool Regexp_processor_pcre::compile(String *pattern, bool send_error) > if (!(pattern= convert_if_needed(pattern, &pattern_converter))) > return true; > > - m_pcre= pcre_compile(pattern->c_ptr_safe(), m_library_flags, > - &pcreErrorStr, &pcreErrorOffset, NULL); > + m_pcre= pcre2_compile((PCRE2_SPTR8) pattern->ptr(), pattern->length(), > + m_library_flags, > + &pcreErrorNumber, &pcreErrorOffset, NULL); > > if (m_pcre == NULL) > { > if (send_error) > { > char buff[MAX_FIELD_WIDTH]; > - my_snprintf(buff, sizeof(buff), "%s at offset %d", pcreErrorStr, pcreErrorOffset); > + int lmsg= pcre2_get_error_message(pcreErrorNumber, > + (PCRE2_UCHAR8 *)buff, sizeof(buff)); > + if (lmsg >= 0) > + my_snprintf(buff+lmsg, sizeof(buff)-lmsg, > + " at offset %d", pcreErrorOffset); if lmsg < 0, does it guarantee that the buffer is 0-terminated? > my_error(ER_REGEXP_ERROR, MYF(0), buff); > } > return true; > } > + > + m_pcre_match_data= pcre2_match_data_create_from_pattern(m_pcre, NULL); > + if (m_pcre_match_data == NULL) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + return true; > + } > + > return false; > } > > @@ -5570,124 +5574,43 @@ bool Regexp_processor_pcre::compile(Item *item, bool send_error) > */ > void Regexp_processor_pcre::pcre_exec_warn(int rc) const > { ... > - push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > - ER_REGEXP_ERROR, ER_THD(thd, ER_REGEXP_ERROR), errmsg); > + int errlen= pcre2_get_error_message(rc, buf, sizeof(buf)); > + if (errlen >= 0) > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_REGEXP_ERROR, ER_THD(thd, ER_REGEXP_ERROR), buf); perhaps, issue some generic message if pcre2_get_error_message fails? > } > @@ -5697,14 +5620,12 @@ bool Regexp_processor_pcre::exec(String *str, int offset, > { > if (!(str= convert_if_needed(str, &subject_converter))) > return true; > - m_pcre_exec_rc= pcre_exec_with_warn(m_pcre, &m_pcre_extra, > - str->c_ptr_safe(), str->length(), > - offset, 0, > - m_SubStrVec, array_elements(m_SubStrVec)); > + m_pcre_exec_rc= pcre_exec_with_warn(m_pcre, m_pcre_match_data, > + str->ptr(), str->length(), offset, 0); > if (m_pcre_exec_rc > 0) > { > uint i; > - for (i= 0; i < n_result_offsets_to_convert; i++) > + for (i= 0; i < n_result_offsets_to_convert && i < m_pcre_exec_rc-1; i++) hmm, what does that mean? that both n_result_offsets_to_convert < m_pcre_exec_rc-1 and m_pcre_exec_rc-1 < n_result_offsets_to_convert are possible? > { > /* > Convert byte offset into character offset. > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index 3d11a22..d3f52ed 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -25,7 +25,8 @@ > > #include "item_func.h" /* Item_int_func, Item_bool_func */ > #define PCRE_STATIC 1 /* Important on Windows */ > -#include "pcre.h" /* pcre header file */ > +#define PCRE2_CODE_UNIT_WIDTH 8 you sure, you want it here and not, say, in one universally-included headers like my_config.h? You don't define it in other places now, so they include different function names and macros. > +#include "pcre2.h" /* pcre header file */ > #include "item.h" > > extern Item_result item_cmp_type(Item_result a,Item_result b); > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index fe2b9c8..dcf0996 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -111,7 +111,7 @@ > #include "sp_rcontext.h" > #include "sp_cache.h" > #include "sql_reload.h" // reload_acl_and_cache > -#include "pcre.h" > +#include "pcre2.h" do you still need to include pcre header here? > > #ifdef HAVE_POLL_H > #include <poll.h> > diff --git a/storage/mroonga/CMakeLists.txt b/storage/mroonga/CMakeLists.txt > index 5d8e8c1..57d12a1 100644 > --- a/storage/mroonga/CMakeLists.txt > +++ b/storage/mroonga/CMakeLists.txt > @@ -189,8 +189,8 @@ else() > set(MYSQL_VARIANT "MySQL") > endif() > > -if(EXISTS "${MYSQL_SOURCE_DIR}/pcre") > - set(MYSQL_REGEX_INCLUDE_DIR "${MYSQL_SOURCE_DIR}/pcre") > +if(EXISTS "${MYSQL_SOURCE_DIR}/pcre2") > + set(MYSQL_REGEX_INCLUDE_DIR "${MYSQL_SOURCE_DIR}/pcre2/src") eh, is that enough? there were a lot more changes in the server, than just correcting the path. > else() > set(MYSQL_REGEX_INCLUDE_DIR "${MYSQL_SOURCE_DIR}/regex") > endif() Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] FYI: 10.3 master branch head builds OK Launchpad all architectures
by Otto Kekäläinen 11 Apr '18

11 Apr '18

Hello! Just for the record, I've done some small tweaks to 10.3 so that it now builds perfectly and passes build tests on all Launchpad architectures that I have access to. https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.3/+build s?build_text=&build_state=all Let's hope it stays like this for the rest of the current 10.3 cycle and will not regress by before it is announced GA.. On buildbot the Debian sid and Ubuntu Bionic builders pass now. There are however some sporadic failures left. http://buildbot.askmonty.org/buildbot/grid?branch=10.3&category=main&ca tegory=experimental On Travis everything else passes apart from one test, which has been consistently failing for some time. Everything was green in January when I wrote the protected branches blog post, so this one is a consistently failing regression. https://travis-ci.org/MariaDB/server/builds/362466962 - Otto

2 5

[Maria-developers] Temporary table bug
by Badrul Chowdhury 09 Apr '18

09 Apr '18

Hi, Is anyone actively working on https://jira.mariadb.org/browse/MDEV-12459? If not, I would like to take a look. -- Thanks, Badrul

2 2

[Maria-developers] Test failure overview for Debian/Ubuntu on 10.3 master branch
by Otto Kekäläinen 05 Apr '18

05 Apr '18

(posting this here so to ensure it is recorded in mailing list archive) On Travis the consistently failing tests are: main.type_datetime_hires main.select_pkeycache main.select main.select_jcl6 main.type_float main.func_str main.type_time_hires main.type_timestamp_hires Log: https://api.travis-ci.org/v3/job/362466968/log.txt On buildbot the consistelty failing tests for Debian/Ubuntu are: galera.galera_sst_mysqldump Log: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-x86/build s/4033/steps/mtr/logs/stdio and http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-ppc64le/b uilds/4546/steps/mtr/logs/stdio Single ones: encryption.innodb_encryption Log: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-sid-x86/builds/6 560/steps/mtr/logs/stdio rocksdb.drop_index_inplace Log: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-ppc64le/b uilds/4546/steps/mtr/logs/stdio - Otto

1 0

Re: [Maria-developers] Regarding bugs (mysqldump)
by Vicențiu Ciorbaru 01 Apr '18

01 Apr '18

Hi Rutuja! Great analysis again and lets share this with the maria-developers mailing list as well. I've tested your proposed change and the change for dump_all_tables_in_db seems to not be necessary. If you look at how dump_all_tables_in_db gets called, it's followed by dump_all_views_in_db. The actual view creation happens in dump_all_views_in_db, while dump_all_tables_in_db will create some dummy tables for views. We create dummy tables first, so as to avoid having to check dependencies within views. There's more details in the comments of why we do that in the code of get_table_structure: /* Create a table with the same name as the view and with columns of the same name in order to satisfy views that depend on this view. The table will be removed when the actual view is created. The properties of each column, are not preserved in this temporary table, because they are not necessary. This will not be necessary once we can determine dependencies between views and can simply dump them in the appropriate order. */ For dump_selected_tables, your proposed change should be sufficient and according to my tests it works. Please submit a pull request for this. Looking forward to having this fixed soon. :) Unrelated to your analysis: There are plenty of edge cases where views won't work: For example: If we have 2 databases and one view uses something from a different database, if you're not careful about the order of things, it's going to fail. To create a "truly" always working dump one would have to create a full dependency graph and then do a topological sort for all objects in the database that depend on the objects we're dumping. This is outside of the scope of this bug, however it could prove to be a "better way to dump stuff". Regards, Vicentiu On Sun, 1 Apr 2018 at 06:28 Rutuja Surve <rutuja.r.surve(a)gmail.com> wrote: > Hello, > Could you review my following approach regarding the mysqldump bug ( > https://jira.mariadb.org/browse/MDEV-15021) > > I went through the code and observed the following: > 1. dump_all_tables_in_db calls dump_table first (for the list of all > tables), then dump_table calls get_view_structure. > 2. get_view_structure dumps the view for the corresponding table to > mysqldump > 3. After this call, dump_all_tables_in_db calls dump_routines_for_db > > dump_selected_tables also behaves in the exact same way as dump_all_tables_in_db, > following steps 1,2,3, except that it does it for a list of given tables > instead of doing it for all tables. > > We want the routines to get dumped before the views for the dump to become > importable. > There are two solutions for this: > 1. Make dump_all_tables_in_db call dump_routines_for_db before it calls > dump_table. This will dump routines first and views later. > This can be done by shifting the call to dump_routines_for_db before the > loop in which dump_table is called. I.E: > Shift : > if (opt_routines && mysql_get_server_version(mysql) >= 50009) > { > DBUG_PRINT("info", ("Dumping routines for database %s", database)); > *dump_routines_for_db*(database); > } > Before this loop: > while ((table= getTableName(0))) > { > char *end= strmov(afterdot, table); > if (include_table((uchar*) hash_key, end - hash_key)) > { > *dump_table*(table,database); > my_free(order_by); > .... > } > } > > We should do this for both dump_all_tables_in_db and dump_selected_tables. > > 2. Approach 2: > As suggested in the jira thread, we can use negative logic to call > dump_table on tables we intend to skip. > This appears to be a workaround. In the place where dump_all_tables_in_db > is called, we'd rather call dump_selected_tables > with a list of tables that don't use the routines we'll be listing after > that. This would fix the issue but is not a good approach. > > If dump_table doesn't alter the values for opt_routines, the first > approach should fix the issue. > > Thanks, > Rutuja > > > > On Fri, Mar 23, 2018 at 10:36 PM, Rutuja Surve <rutuja.r.surve(a)gmail.com> > wrote: > >> Hello, >> With respect to https://jira.mariadb.org/browse/MDEV-15021 : >> From what I understood from the thread, since the routines are placed >> below the tables/views, they cannot be used in views while importing. >> I went through the problematic_dump.sql file that's attached there. What >> does the '@' in the following mean: >> SET @saved_cs_client = @@character_set_client; >> (I'm not familiar with this syntax) >> Also, what does the following piece of code do: (what does create definer >> mean and does this function simply return 1 as there are no other >> statements after begin? This is a routine defined below test_view) >> CREATE DEFINER=`root`@`localhost` FUNCTION `test_function`() RETURNS >> int(11) >> BEGIN >> >> RETURN 1; >> END ;; >> DELIMITER ; >> From my understanding, test_view uses test_function and test_view2 would >> use test_function2. >> In the mysqldump.c file, I went through the dump_routines_for_db >> function. We'd require to modify that in order to fix the bug. >> >> From the beginner bugs list, I found these 2 bugs that are also related >> to mysql dump that I could take up: >> 1. https://jira.mariadb.org/browse/MDEV-9883 >> (Flag for letting mysqldump support the 'auto' option for character set). >> Where in the code base can I find the file where the ' --default-character-set' >> flag is defined?) >> 2. https://jira.mariadb.org/browse/MDEV-8765 >> The solution for this has already been proposed in the thread. For >> mysqldump to not corrupt 4 byte UTF8 data, "change >> include/my_global.h:#define MYSQL_UNIVERSAL_CLIENT_CHARSET to utf8mb4" >> >> Thanks, >> Rutuja >> > >

1 0

Re: [Maria-developers] d975226a0d6: MDEV-15241: make SIGNAL maximum MESSAGE_TEXT length a larger value
by Sergei Golubchik 01 Apr '18

01 Apr '18

Hi, Varun! On Mar 22, Varun wrote: > revision-id: d975226a0d668775062151c9f30676db3d81c93b (mariadb-10.3.0-642-gd975226a0d6) > parent(s): 85ddd9e8ce11bef089ccc6c86745142d09639c14 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-03-22 12:45:55 +0530 > message: > > MDEV-15241: make SIGNAL maximum MESSAGE_TEXT length a larger value > > Increasing the length of MESSAGE_TEXT from 128 to MYSQL_ERRMSG_SIZE > which is the max length of the error message ok to push. Don't forget to document your change, please. > diff --git a/sql/sql_signal.cc b/sql/sql_signal.cc > index 1da8b145151..13dc3db1f10 100644 > --- a/sql/sql_signal.cc > +++ b/sql/sql_signal.cc > @@ -262,12 +262,12 @@ int Sql_cmd_common_signal::eval_signal_informations(THD *thd, Sql_condition *con > } > /* > Enforce that SET MESSAGE_TEXT = <value> evaluates the value > - as VARCHAR(128) CHARACTER SET UTF8. > + as VARCHAR(MYSQL_ERRMSG_SIZE) CHARACTER SET UTF8. > */ > bool truncated; > String utf8_text; > str= set->val_str(& str_value); > - truncated= assign_fixed_string(thd->mem_root, & my_charset_utf8_bin, 128, > + truncated= assign_fixed_string(thd->mem_root, & my_charset_utf8_bin, MYSQL_ERRMSG_SIZE, > & utf8_text, str); > if (truncated) > { Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-14929 - AddressSanitizer: memcpy-param-overlap in Field_longstr::compress
by Sergei Golubchik 31 Mar '18

31 Mar '18

Hi, Sergey, > From 6de643a580b4e706e0201927e43a033f5cb98970 Mon Sep 17 00:00:00 2001 > From: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Wed, 21 Mar 2018 13:54:15 +0400 > Subject: [PATCH] MDEV-14929 - AddressSanitizer: memcpy-param-overlap in > Field_longstr::compress > > Handle overlaping "from" and Field_blob_compressed::value for compressed > blobs similarily to regular blobs. > --- > mysql-test/r/column_compression.result | 13 +++++++++++++ > mysql-test/t/column_compression.test | 12 ++++++++++++ > sql/field.cc | 16 +++++++++++----- > 3 files changed, 36 insertions(+), 5 deletions(-) > > diff --git a/sql/field.cc b/sql/field.cc > index 9da65526e145..1b044ec740e2 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -8718,17 +8718,23 @@ int Field_blob_compressed::store(const char *from, size_t length, > { > ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED; > uint to_length= (uint)MY_MIN(max_data_length(), field_charset->mbmaxlen * length + 1); > + String tmp(from, length, cs); > int rc; > > + if (from >= value.ptr() && from <= value.ptr() + value.length() && > + tmp.copy(from, length, cs)) minor style comment: this could be if (from >= value.ptr() && from <= value.end() && tmp.copy()) Bigger question: Field_blob::store() also checks whether charset conversion is necessary. Why is it not applicable here? And, it'd be good to have a test case where from > value.ptr() (with a substring() function). > + goto oom; > + > if (value.alloc(to_length)) > - { > - set_ptr((uint32) 0, NULL); > - return -1; > - } > + goto oom; > > - rc= compress((char*) value.ptr(), &to_length, from, (uint)length, cs); > + rc= compress((char*) value.ptr(), &to_length, tmp.ptr(), (uint) length, cs); > set_ptr(to_length, (uchar*) value.ptr()); > return rc; > + > +oom: > + set_ptr((uint32) 0, NULL); > + return -1; > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] Patch for pad functions
by jerome brauge 30 Mar '18

30 Mar '18

Hello Alexander, Can you open an MDEV for rpad/lpad (return NULL instead of empty string when sql_mode=oracle) and review this patch ? Regards.

2 1

Re: [Maria-developers] [Commits] c9ae1234814: MDEV-14592: Custom Aggregates Usage Status Variable
by Vicențiu Ciorbaru 26 Mar '18

26 Mar '18

Hi Varun This generally looks good, 2 comments inline. Vicentiu diff --git a/mysql-test/t/custom_aggregate_functions.test b/mysql-test/t/custom_aggregate_functions.test index 20fcc35f39f..a93ec545f01 100644 --- a/mysql-test/t/custom_aggregate_functions.test +++ b/mysql-test/t/custom_aggregate_functions.test @@ -16,7 +16,9 @@ create table t1 (sal int(10),id int(10)); INSERT INTO t1 (sal,id) VALUES (5000,1); INSERT INTO t1 (sal,id) VALUES (2000,1); INSERT INTO t1 (sal,id) VALUES (1000,1); # Please test that this works with subqueries, views, ctes and when calling from # other stored procedures. +show global status like "Feature_custom_aggregate_functions"; select f1(sal) from t1 where id>= 1; +show global status like "Feature_custom_aggregate_functions"; select * from t2; drop table t2; drop function f1; diff --git a/sql/item_sum.cc b/sql/item_sum.cc index d4abdfc614f..7c0390e543f 100644 --- a/sql/item_sum.cc +++ b/sql/item_sum.cc @@ -313,6 +313,8 @@ bool Item_sum::check_sum_func(THD *thd, Item **ref) } } aggr_sel->set_agg_func_used(true); + if (sum_func() == SP_AGGREGATE_FUNC) + aggr_sel->set_custom_agg_func_used(true); update_used_tables(); thd->lex->in_sum_func= in_sum_func; return FALSE; diff --git a/sql/mysqld.cc b/sql/mysqld.cc index c7fbf0a594c..75ccda408b1 100644 --- a/sql/mysqld.cc +++ b/sql/mysqld.cc @@ -8491,6 +8491,7 @@ SHOW_VAR status_vars[]= { {"Executed_triggers", (char*) offsetof(STATUS_VAR, executed_triggers), SHOW_LONG_STATUS}, {"Feature_check_constraint", (char*) &feature_check_constraint, SHOW_LONG }, {"Feature_delay_key_write", (char*) &feature_files_opened_with_delayed_keys, SHOW_LONG }, Please move this up so the variable names are ordered alphabetically. + {"Feature_custom_aggregate_functions", (char*) offsetof(STATUS_VAR, feature_custom_aggregate_functions), SHOW_LONG_STATUS}, {"Feature_dynamic_columns", (char*) offsetof(STATUS_VAR, feature_dynamic_columns), SHOW_LONG_STATUS}, {"Feature_fulltext", (char*) offsetof(STATUS_VAR, feature_fulltext), SHOW_LONG_STATUS}, {"Feature_gis", (char*) offsetof(STATUS_VAR, feature_gis), SHOW_LONG_STATUS}, On Sat, 9 Dec 2017 at 08:53 Varun <varunraiko1803(a)gmail.com> wrote: > revision-id: c9ae12348143f59e875a52e2eb285c5ed7d7af56 > (mariadb-10.3.0-285-gc9ae1234814) > parent(s): c60095a818dce92838940525899a13a05633d148 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-12-09 12:22:26 +0530 > message: > > MDEV-14592: Custom Aggregates Usage Status Variable > > --- > mysql-test/r/custom_aggregate_functions.result | 6 ++++++ > mysql-test/t/custom_aggregate_functions.test | 2 ++ > sql/item_sum.cc | 2 ++ > sql/mysqld.cc | 1 + > sql/sql_lex.cc | 2 ++ > sql/sql_lex.h | 3 +++ > sql/sql_select.cc | 2 ++ > 7 files changed, 18 insertions(+) > > diff --git a/mysql-test/r/custom_aggregate_functions.result > b/mysql-test/r/custom_aggregate_functions.result > index ca8612ba652..f70675fa4db 100644 > --- a/mysql-test/r/custom_aggregate_functions.result > +++ b/mysql-test/r/custom_aggregate_functions.result > @@ -11,6 +11,9 @@ create table t1 (sal int(10),id int(10)); > INSERT INTO t1 (sal,id) VALUES (5000,1); > INSERT INTO t1 (sal,id) VALUES (2000,1); > INSERT INTO t1 (sal,id) VALUES (1000,1); > +show global status like "Feature_custom_aggregate_functions"; > +Variable_name Value > +Feature_custom_aggregate_functions 0 > select f1(sal) from t1 where id>= 1; > f1(sal) > 0 > @@ -18,6 +21,9 @@ Warnings: > Note 4093 At line 5 in test.f1 > Note 4093 At line 5 in test.f1 > Note 4093 At line 5 in test.f1 > +show global status like "Feature_custom_aggregate_functions"; > +Variable_name Value > +Feature_custom_aggregate_functions 1 > select * from t2; > sal > 5000 > diff --git a/mysql-test/t/custom_aggregate_functions.test > b/mysql-test/t/custom_aggregate_functions.test > index 20fcc35f39f..a93ec545f01 100644 > --- a/mysql-test/t/custom_aggregate_functions.test > +++ b/mysql-test/t/custom_aggregate_functions.test > @@ -16,7 +16,9 @@ create table t1 (sal int(10),id int(10)); > INSERT INTO t1 (sal,id) VALUES (5000,1); > INSERT INTO t1 (sal,id) VALUES (2000,1); > INSERT INTO t1 (sal,id) VALUES (1000,1); > +show global status like "Feature_custom_aggregate_functions"; > select f1(sal) from t1 where id>= 1; > +show global status like "Feature_custom_aggregate_functions"; > select * from t2; > drop table t2; > drop function f1; > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index d4abdfc614f..7c0390e543f 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -313,6 +313,8 @@ bool Item_sum::check_sum_func(THD *thd, Item **ref) > } > } > aggr_sel->set_agg_func_used(true); > + if (sum_func() == SP_AGGREGATE_FUNC) > + aggr_sel->set_custom_agg_func_used(true); > update_used_tables(); > thd->lex->in_sum_func= in_sum_func; > return FALSE; > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index c7fbf0a594c..75ccda408b1 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -8491,6 +8491,7 @@ SHOW_VAR status_vars[]= { > {"Executed_triggers", (char*) offsetof(STATUS_VAR, > executed_triggers), SHOW_LONG_STATUS}, > {"Feature_check_constraint", (char*) &feature_check_constraint, > SHOW_LONG }, > {"Feature_delay_key_write", (char*) > &feature_files_opened_with_delayed_keys, SHOW_LONG }, > + {"Feature_custom_aggregate_functions", (char*) offsetof(STATUS_VAR, > feature_custom_aggregate_functions), SHOW_LONG_STATUS}, > {"Feature_dynamic_columns", (char*) offsetof(STATUS_VAR, > feature_dynamic_columns), SHOW_LONG_STATUS}, > {"Feature_fulltext", (char*) offsetof(STATUS_VAR, > feature_fulltext), SHOW_LONG_STATUS}, > {"Feature_gis", (char*) offsetof(STATUS_VAR, feature_gis), > SHOW_LONG_STATUS}, > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 6cb065c3985..277469cc0aa 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2241,6 +2241,7 @@ void st_select_lex::init_query() > select_list_tables= 0; > m_non_agg_field_used= false; > m_agg_func_used= false; > + m_custom_agg_func_used= false; > window_specs.empty(); > window_funcs.empty(); > tvc= 0; > @@ -2278,6 +2279,7 @@ void st_select_lex::init_select() > merged_into= 0; > m_non_agg_field_used= false; > m_agg_func_used= false; > + m_custom_agg_func_used= false; > name_visibility_map= 0; > with_dep= 0; > join= 0; > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index 069b75628e3..9b40529d3e2 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -1213,9 +1213,11 @@ class st_select_lex: public st_select_lex_node > */ > bool non_agg_field_used() const { return m_non_agg_field_used; } > bool agg_func_used() const { return m_agg_func_used; } > + bool custom_agg_func_used() const { return m_custom_agg_func_used; } > > void set_non_agg_field_used(bool val) { m_non_agg_field_used= val; } > void set_agg_func_used(bool val) { m_agg_func_used= val; } > + void set_custom_agg_func_used(bool val) { m_custom_agg_func_used= val; } > inline void set_with_clause(With_clause *with_clause); > With_clause *get_with_clause() > { > @@ -1259,6 +1261,7 @@ class st_select_lex: public st_select_lex_node > private: > bool m_non_agg_field_used; > bool m_agg_func_used; > + bool m_custom_agg_func_used; > > /* current index hint kind. used in filling up index_hints */ > enum index_hint_type current_index_hint_type; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 96aa227b439..c0d7c720910 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2909,6 +2909,8 @@ bool JOIN::make_aggr_tables_info() > /* Count that we're using window functions. */ > status_var_increment(thd->status_var.feature_window_functions); > } > + if (select_lex->custom_agg_func_used()) > + > status_var_increment(thd->status_var.feature_custom_aggregate_functions); > > fields= curr_fields_list; > // Reset before execution > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] MDEV 10754 - Patch for trim functions
by jerome brauge 26 Mar '18

26 Mar '18

Hello Alexander, Can you review this patch for trim functions for Oracle ? Regards, Jérôme.

2 2

Re: [Maria-developers] [Commits] 2950feb5c09: MDEV-9959: A serious MariaDB server performance bug
by Sergey Petrunia 26 Mar '18

26 Mar '18

Hi Varun, Please find my feedback below. As disucssed before: please also run the MDEV's test case with this patch and post a comment with the result. On Fri, Mar 23, 2018 at 05:17:57PM +0530, Varun wrote: > revision-id: 2950feb5c09033843a6a32c7431421b45d5812dd (mariadb-10.3.0-646-g2950feb5c09) > parent(s): 9ef7ab221d70630155206910de8a6053db097164 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-03-23 17:17:09 +0530 > message: > > MDEV-9959: A serious MariaDB server performance bug > > step#1: if a derived table has SELECT DISTINCT, provide index statistics for it so that the join optimizer in the > upper select knows that ref access to the table will produce one row. > > Added handling for multiple selects in the derived table > > --- > mysql-test/r/mdev9959.result | 64 ++++++++++++++++++++++++++++++++++++++++++++ > mysql-test/t/mdev9959.test | 28 +++++++++++++++++++ > sql/sql_lex.h | 2 ++ > sql/sql_union.cc | 36 +++++++++++++++++++++++++ > sql/table.cc | 26 +++++++++++++++--- > 5 files changed, 153 insertions(+), 3 deletions(-) > > diff --git a/mysql-test/r/mdev9959.result b/mysql-test/r/mdev9959.result > new file mode 100644 > index 00000000000..5269ba9b734 > --- /dev/null > +++ b/mysql-test/r/mdev9959.result > @@ -0,0 +1,64 @@ > +create table t1(a int); > +insert into t1 values (1),(2),(3),(4),(5),(6); > +create table t2(a int, b int,c int); > +insert into t2(a,b,c) values (1,1,2),(2,2,3),(3,1,4),(4,2,2),(5,1,1),(6,2,5); > +create table t3(a int, b int); > +insert into t3(a,b) values (1,1),(2,2),(2,1),(1,2),(5,1),(9,2); > +analyze select * from t1 , ( (select distinct t2.a from t2 group by c))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 1 0.83 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using temporary; Using filesort It is not at all clear which parts of the above output matter and which don't, for example, does #rows matter for table t1? Please add a comment (into .test file with --echo # ) describing that table "<derived2>" should have type=ref and rows=1 > +select * from t1 , ( (select distinct t2.a from t2 group by c))q where t1.a=q.a; > +a a > +1 1 > +2 2 > +3 3 > +5 5 > +6 6 > +analyze select * from t1 , ( (select t2.a from t2 group by c))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 2 0.83 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using temporary; Using filesort > +select * from t1 , ( (select t2.a from t2 group by c))q where t1.a=q.a; > +a a > +1 1 > +2 2 > +3 3 > +5 5 > +6 6 > +analyze select * from t1 , ( (select distinct t2.a from t2 group by c) union all (select distinct t2.a from t2 group by c))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 2 1.67 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using temporary; Using filesort > +3 UNION t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using temporary; Using filesort What does the above test show? We've got type=ref, #rows = 2 for the derived table. The estimate is correct but where does it come from? As far as I understand this patch, it only sets the estimate rows=1 for derived tables. How do we arrive at the value of 2? (If we would have gotten rows=2 without this patch, what's the point of having the testcase as part of this patch?) > +select * from t1 , ( (select distinct t2.a from t2 group by c) union all (select distinct t2.a from t2 group by c))q where t1.a=q.a; > +a a > +1 1 > +1 1 > +2 2 > +2 2 > +3 3 > +3 3 > +5 5 > +5 5 > +6 6 > +6 6 > +analyze select * from t1 , ( (select distinct t2.a from t2 group by c) union (select distinct t2.a from t2 group by c) union (select t3.a from t3 group by b))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 2 0.83 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using temporary; Using filesort > +3 UNION t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using temporary; Using filesort > +4 UNION t3 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using temporary; Using filesort > +NULL UNION RESULT <union2,3,4> ALL NULL NULL NULL NULL NULL 5.00 NULL NULL > +select * from t1 , ( (select distinct t2.a from t2 group by c) union (select distinct t2.a from t2 group by c) union (select t3.a from t3 group by b))q where t1.a=q.a; > +a a > +1 1 > +2 2 > +3 3 > +5 5 > +6 6 > +drop table t1,t2,t3; > diff --git a/mysql-test/t/mdev9959.test b/mysql-test/t/mdev9959.test > new file mode 100644 > index 00000000000..77970f363ac > --- /dev/null > +++ b/mysql-test/t/mdev9959.test > @@ -0,0 +1,28 @@ > +create table t1(a int); > +insert into t1 values (1),(2),(3),(4),(5),(6); > +create table t2(a int, b int,c int); > +insert into t2(a,b,c) values (1,1,2),(2,2,3),(3,1,4),(4,2,2),(5,1,1),(6,2,5); > +create table t3(a int, b int); > +insert into t3(a,b) values (1,1),(2,2),(2,1),(1,2),(5,1),(9,2); > + > +# one select in derived table > + > +#with distinct > +analyze select * from t1 , ( (select distinct t2.a from t2 group by c))q where t1.a=q.a; > +select * from t1 , ( (select distinct t2.a from t2 group by c))q where t1.a=q.a; > +# without distinct > +analyze select * from t1 , ( (select t2.a from t2 group by c))q where t1.a=q.a; > +select * from t1 , ( (select t2.a from t2 group by c))q where t1.a=q.a; > + > +# multiple selects in derived table > + > +#UNION ALL > +analyze select * from t1 , ( (select distinct t2.a from t2 group by c) union all (select distinct t2.a from t2 group by c))q where t1.a=q.a; > +select * from t1 , ( (select distinct t2.a from t2 group by c) union all (select distinct t2.a from t2 group by c))q where t1.a=q.a; > + > +#UNION > +analyze select * from t1 , ( (select distinct t2.a from t2 group by c) union (select distinct t2.a from t2 group by c) union (select t3.a from t3 group by b))q where t1.a=q.a; > +select * from t1 , ( (select distinct t2.a from t2 group by c) union (select distinct t2.a from t2 group by c) union (select t3.a from t3 group by b))q where t1.a=q.a; > + > +drop table t1,t2,t3; > + > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index f0241a32acf..4128c1dd300 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -761,6 +761,7 @@ class st_select_lex_unit: public st_select_lex_node { > Procedure *last_procedure; /* Pointer to procedure, if such exists */ > > bool columns_are_renamed; > + bool union_all; /* TRUE if we have UNION ALL operation */ This variable is checked but it is never set? > > void init_query(); > st_select_lex* outer_select(); > @@ -800,6 +801,7 @@ class st_select_lex_unit: public st_select_lex_node { > bool union_needs_tmp_table(); > > void set_unique_exclude(); > + bool check_distinct_in_union(); > > friend struct LEX; > friend int subselect_union_engine::exec(); > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index 857c9a117f5..43ce0418fe5 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -1960,3 +1960,39 @@ void st_select_lex_unit::set_unique_exclude() > } > } > } > + > +/* > + Check if the selects in the derived table can give distinct rows I think "can" is the wrong word here. UNION/INTERSECT/EXCEPT output "can" be distinct if the underlying selects produce distinct sets of rows. What we are checking is whether the output will be distinct regardless of what data is there in the queried tables. > + > + for example: > + select * from > + ((select t1.a from t1) op (select t2.a from t2) op (select t3.a from t3)); > + the op here being UNION/INTERSECT/EXCEPT > + > + so this function would check if the derived table like the case above > + can give distinct rows or not. Please describe the criteria and provide a couple of examples (either here or in the comments inside the function). > + > + @retval false Distinct rows are not guarenteed > + @retval true Distinct rows are guanrenteed Typos in the above lines. > + > +*/ > + > +bool st_select_lex_unit::check_distinct_in_union() > +{ > + bool is_intersect_present=FALSE; > + st_select_lex* first= first_select(); > + for(st_select_lex *sl=first; sl ; sl=sl->next_select()) > + is_intersect_present|= (sl->linkage == INTERSECT_TYPE); > + > + if (!union_all) > + return true; > + else > + { > + if (union_select) > + { > + if (!is_intersect_present && !union_select->next_select()) > + return true; > + } > + } > + return false; > +} > diff --git a/sql/table.cc b/sql/table.cc > index f42455a8413..c474d2ed4d7 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -7113,9 +7113,29 @@ bool TABLE::add_tmp_key(uint key, uint key_parts, > if (!keyinfo->rec_per_key) > return TRUE; > bzero(keyinfo->rec_per_key, sizeof(ulong)*key_parts); > - st_select_lex* lex= pos_in_table_list ? pos_in_table_list->derived->first_select(): NULL; > - if (lex && (lex->options & SELECT_DISTINCT)) > - keyinfo->rec_per_key[key_parts-1]=1; > + > + /* > + For the case when there is a derived table that would give distinct rows, > + the index statistics are passed to the join optimizer to tell that > + a ref access to the derived table will produce only one row. > + */ > + > + st_select_lex_unit* derived= pos_in_table_list ? pos_in_table_list->derived: NULL; > + if (derived) > + { > + /* > + This handles the case when we have a single select in the derived table > + */ > + st_select_lex* first= derived->first_select(); > + if (first && !first->is_part_of_union() && > + first->options & SELECT_DISTINCT) > + keyinfo->rec_per_key[key_parts-1]=1; > + else > + { > + if (check_distinct_in_union()) > + keyinfo->rec_per_key[key_parts-1]=1; > + } I think it would be better if there was a single "if", like so: if ((this is a non-union select with SELECT_DISTINCT || check_distinct_in_union()) { keyinfo->rec_per_key[key_parts-1]=1; } > + } > keyinfo->read_stats= NULL; > keyinfo->collected_stats= NULL; > BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] b718e87caaf: MDEV-11274: Executing EXPLAIN of complex query over join limit causes server to crash
by Sergey Petrunia 26 Mar '18

26 Mar '18

Hi Varun, On Sat, Mar 24, 2018 at 03:31:59AM +0530, Varun wrote: > revision-id: b718e87caaf2d0a60c19cd042dfc04868f4bb0b4 (mariadb-5.5.56-187-gb718e87caaf) > parent(s): dfc79ae949f6d831ff65a08a5d32d14b59ba06e6 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-03-24 03:31:18 +0530 > message: > > MDEV-11274: Executing EXPLAIN of complex query over join limit causes server to crash > > For this case we have a view that is mergeable but we are not able to merge it in the > parent select because that would exceed the maximum tables allowed in the join list, so we > materialise this view > TABLE_LIST::dervied is NULL for such views, it is only set for views which have ALGORITHM=TEMPTABLE > Fixed by making sure TABLE_LIST::derived is set for views that could not be merged > > --- > mysql-test/r/view.result | 197 +++++++++++++++++++++++++++++++++++++++++++++++ > mysql-test/t/view.test | 196 ++++++++++++++++++++++++++++++++++++++++++++++ > sql/table.h | 2 + > 3 files changed, 395 insertions(+) > > diff --git a/sql/table.h b/sql/table.h > index 98f8c7ad73f..7b58f62e659 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -46,6 +46,7 @@ struct TABLE_LIST; > class ACL_internal_schema_access; > class ACL_internal_table_access; > class Field; > +struct LEX; > > /* > Used to identify NESTED_JOIN structures within a join (applicable only to What is the above needed for? Please remove this change. Ok to push after this is addressed. > @@ -2125,6 +2126,7 @@ struct TABLE_LIST > DBUG_PRINT("enter", ("Alias: '%s' Unit: %p", > (alias ? alias : "<NULL>"), > get_unit())); > + derived= get_unit(); > derived_type= ((derived_type & (derived ? DTYPE_MASK : DTYPE_VIEW)) | > DTYPE_TABLE | DTYPE_MATERIALIZE); > set_check_materialized(); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 108cdbf: MDEV-10871 Add logging capability to pam_user_map.c
by Sergei Golubchik 24 Mar '18

24 Mar '18

Hi, Alexey! On Mar 23, Alexey Botchkov wrote: > > Technically yes, there can be many arguments. > BTW i think some other arguments make sence. > We can specify the configuration file name in arguments (now it's hardcoded > as "/etc/security/user_map.conf"), > then we can specify some 'user: user_map' pairs as arguments, > to make an addition to what is in the user_map.conf or we can go by > withouth the .conf > file at all. > Is it worth adding the todo-mdev? Agree. File name — makes sense, user:map pairs probably not, I don't remember any other pam module doing that. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-15620 Crash when using "SET @@NEW.a=expr" inside a trigger
by Alexander Barkov 23 Mar '18

23 Mar '18

Hello Sanja, Please review a patch for MDEV-15620. Thanks!

1 0

Re: [Maria-developers] 108cdbf: MDEV-10871 Add logging capability to pam_user_map.c
by Sergei Golubchik 22 Mar '18

22 Mar '18

Hi, Alexey! Looks good, of course. See few minor comments below. On Mar 18, Alexey Botchkov wrote: > revision-id: 108cdbf854df6af1ed2c0fc3a63e1786f6b4b7b4 (mariadb-10.1.31-61-g108cdbf) > parent(s): a0c722d853071e605ea025168da1b01bfe21092c > committer: Alexey Botchkov > timestamp: 2018-03-18 12:03:59 +0400 > message: > > MDEV-10871 Add logging capability to pam_user_map.c. > > 'debug' option added to the pam_user_map.so. > It writes the verbose report to the syslog. don't indent the commit comment, please > > --- > plugin/auth_pam/mapper/pam_user_map.c | 74 +++++++++++++++++++++++++++++++++-- > 1 file changed, 71 insertions(+), 3 deletions(-) > > diff --git a/plugin/auth_pam/mapper/pam_user_map.c b/plugin/auth_pam/mapper/pam_user_map.c > index e62be94..1706630 100644 > --- a/plugin/auth_pam/mapper/pam_user_map.c > +++ b/plugin/auth_pam/mapper/pam_user_map.c > @@ -26,10 +26,13 @@ top: accounting > > #include <stdlib.h> > #include <stdio.h> > +#include <ctype.h> > +#include <string.h> > #include <syslog.h> > #include <grp.h> > #include <pwd.h> > > +#include <security/pam_ext.h> > #include <security/pam_modules.h> > > #define FILENAME "/etc/security/user_map.conf" > @@ -90,9 +93,40 @@ static int user_in_group(const gid_t *user_groups, int ng,const char *group) > } > > > +static void print_groups(pam_handle_t *pamh, const gid_t *user_groups, int ng) > +{ > + char buf[256]; > + char *c_buf= buf, *buf_end= buf+sizeof(buf)-1; > + struct group *gr; > + > + for (; ng > 0; user_groups++, ng--) > + { > + char *c; > + if (!(gr= getgrgid(*user_groups)) || > + !(c= gr->gr_name)) > + continue; > + while (*c) > + { > + if (c_buf == buf_end) > + break; > + *(c_buf++)= *c; > + } > + if (c_buf == buf_end) > + break; > + *(c_buf++)= ','; > + } > + *c_buf= 0; > + pam_syslog(pamh, LOG_DEBUG, "User belongs to groups [%s].\n", buf); s/pam_syslog/SYSLOG_DEBUG/ ? > +} > + > + > +static const char debug_keyword[]= "debug"; > +#define SYSLOG_DEBUG if (mode_debug) pam_syslog > + > int pam_sm_authenticate(pam_handle_t *pamh, int flags, > int argc, const char *argv[]) > { > + int mode_debug= 0; > int pam_err, line= 0; > const char *username; > char buf[256]; > @@ -101,6 +135,14 @@ int pam_sm_authenticate(pam_handle_t *pamh, int flags, > gid_t *groups= group_buffer; > int n_groups= -1; > > + for (; argc > 0; argv++) > + { > + if (strcasecmp(*argv, debug_keyword) == 0) > + mode_debug= 1; > + } can there be many arguments? > + > + SYSLOG_DEBUG(pamh, LOG_DEBUG, "Opening file '%s'.\n", FILENAME); > > + > f= fopen(FILENAME, "r"); > if (f == NULL) > { > @@ -110,12 +152,18 @@ int pam_sm_authenticate(pam_handle_t *pamh, int flags, > > pam_err = pam_get_item(pamh, PAM_USER, (const void**)&username); > if (pam_err != PAM_SUCCESS) > + { > + pam_syslog(pamh, LOG_ERR, "Cannot get username.\n"); > goto ret; > + } > + > + SYSLOG_DEBUG(pamh, LOG_DEBUG, "Incoming username '%s'.\n", username); > > while (fgets(buf, sizeof(buf), f) != NULL) > { > char *s= buf, *from, *to, *end_from, *end_to; > int check_group; > + int cmp_result; > > line++; > > @@ -124,7 +172,11 @@ int pam_sm_authenticate(pam_handle_t *pamh, int flags, > if ((check_group= *s == '@')) > { > if (n_groups < 0) > + { > n_groups= populate_user_groups(username, &groups); > + if (mode_debug) > + print_groups(pamh, groups, n_groups); > + } > s++; > } > from= s; > @@ -139,14 +191,29 @@ int pam_sm_authenticate(pam_handle_t *pamh, int flags, > if (end_to == to) goto syntax_error; > > *end_from= *end_to= 0; > - if (check_group ? > - user_in_group(groups, n_groups, from) : > - (strcmp(username, from) == 0)) > + > + cmp_result= 0; looks redundant ^^^ > + if (check_group) > + { > + cmp_result= user_in_group(groups, n_groups, from); > + SYSLOG_DEBUG(pamh, LOG_DEBUG, "Check if user is in gourp '%s': %s\n", s/gourp/group/ > + from, cmp_result ? "YES":"NO"); > + } > + else > + { > + cmp_result= (strcmp(username, from) == 0); > + SYSLOG_DEBUG(pamh, LOG_DEBUG, "Check if username '%s': %s\n", > + from, cmp_result ? "YES":"NO"); > + } > + if (cmp_result) > { > pam_err= pam_set_item(pamh, PAM_USER, to); > + SYSLOG_DEBUG(pamh, LOG_DEBUG, "User authenticated as '%s'\n", to); may be "user mapped to %s" ? > goto ret; > } > } > + > + SYSLOG_DEBUG(pamh, LOG_DEBUG, "User not found in the list.\n"); > pam_err= PAM_AUTH_ERR; > goto ret; > > @@ -162,6 +229,7 @@ int pam_sm_authenticate(pam_handle_t *pamh, int flags, > return pam_err; > } > > + > int pam_sm_setcred(pam_handle_t *pamh, int flags, > int argc, const char *argv[]) > { Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] GSoC 2018 - Histograms with equal-width bins
by Teodor Niculescu 22 Mar '18

22 Mar '18

Greetings, My name is Teodor Niculescu and I am a second year undergraduate student at the The Faculty of Automatic Control and Computers, University Politehnica of Bucharest, Romania. I am interested in contributing to MariaDB as part of this years Google Summer of Code. I am interested in the project: MDEV-12313 Histograms with equal-width bins. I would like to ask you about a good place to start reading the codebase, in order to help me get an idea about what I need to do for this project. I look forward to hearing from you. Sincerely, Teodor.

2 1

Re: [Maria-developers] MDEV-15492: Subquery crash similar to MDEV-10050
by Sergei Golubchik 21 Mar '18

21 Mar '18

Hi, Oleksandr! On Mar 21, Oleksandr Byelkin wrote: > revision-id: 49a443c53a7f01812299f20e94c7209579efd881 (mariadb-5.5.59-45-g49a443c53a7) > parent(s): 0943b33de3daa0fcbf58803be8e991941de63218 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-03-21 12:13:37 +0100 > message: > > MDEV-15492: Subquery crash similar to MDEV-10050 > > Detection of first execution of PS fixed. > More debug info. > > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index a3bf9d6c93c..369a9f3a804 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -3819,6 +3819,7 @@ bool Prepared_statement::execute(String *expanded_query, bool open_cursor) > Statement stmt_backup; > Query_arena *old_stmt_arena; > bool error= TRUE; > + bool qc_executed; > > char saved_cur_db_name_buf[SAFE_NAME_LEN+1]; > LEX_STRING saved_cur_db_name= > @@ -3931,12 +3932,14 @@ bool Prepared_statement::execute(String *expanded_query, bool open_cursor) > 1); > error= mysql_execute_command(thd); > MYSQL_QUERY_EXEC_DONE(error); > + qc_executed= FALSE; > } > else > { > thd->lex->sql_command= SQLCOM_SELECT; > status_var_increment(thd->status_var.com_stat[SQLCOM_SELECT]); > thd->update_stats(); > + qc_executed= TRUE; > } > } mostly ok, but this leaves one code path where qc_executed stays uninitialized. Better assign a default value of FALSE when declaring a variable, and set it to TRUE when really executed. ok to push after that. > > @@ -3960,7 +3963,7 @@ bool Prepared_statement::execute(String *expanded_query, bool open_cursor) > thd->set_statement(&stmt_backup); > thd->stmt_arena= old_stmt_arena; > > - if (state == Query_arena::STMT_PREPARED) > + if (state == Query_arena::STMT_PREPARED && !qc_executed) > state= Query_arena::STMT_EXECUTED; > > if (error == 0 && this->lex->sql_command == SQLCOM_CALL) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] b960f68b102: MDEV-11274: Executing EXPLAIN of complex query over join limit causes server to crash
by Sergey Petrunia 21 Mar '18

21 Mar '18

Hi Varun, On Tue, Jan 23, 2018 at 10:49:37AM +0530, Varun wrote: > revision-id: b960f68b102ec72cf75252b357760e99c5dffc18 (mariadb-5.5.56-144-gb960f68b102) > parent(s): a7a4519a40c58947796c6d9b2e4e58acc18aeef8 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-01-23 10:31:02 +0530 > message: > > MDEV-11274: Executing EXPLAIN of complex query over join limit causes server to crash > > For this case we have a view that is mergeable but we are not able to merge it in the > parent select because that would exceed the maximum tables allowed in the join list, so we > materialise this view > TABLE_LIST::dervied is NULL for such views, it is only set for views which have ALGORITHM=TEMPTABLE > Fixed by using SELECT_LEX_UNIT representing the result of the query within the view definition ... > diff --git a/sql/table.cc b/sql/table.cc > index 9cade76cb78..8f377d5bda8 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -6817,9 +6817,8 @@ int TABLE_LIST::fetch_number_of_rows() > if (jtbm_subselect) > return 0; > if (is_materialized_derived() && !fill_me) > - > { > - table->file->stats.records= ((select_union*)derived->result)->records; > + table->file->stats.records= ((select_union*)get_unit()->result)->records; > set_if_bigger(table->file->stats.records, 2); So, regular VIEWs with algorithm=TEMPTABLE have derived == view->unit This is set in mysql_make_view(): ... table->derived_type= VIEW_ALGORITHM_TMPTABLE; DBUG_PRINT("info", ("algorithm: TEMPORARY TABLE")); view_select->linkage= DERIVED_TABLE_TYPE; ... However, the view in the example has algorithm=MERGE so the execution doesn't reach that point. The decision to materialized is made in mysql_derived_merge. It calls derived->set_materialized_derived(); which calls set_check_materialized(); I have discussed this with Sanja, and the outcome is that a better solution would be to set TABLE_LIST::derived when we decide that this view is going to be handled with temp.table strategy. set_check_materialized() seems to be the right place to do it (although its name looks confusing. What does one "check"? Probably the name is a leftover from some feature backport from MySQL). Could you change the patch to set TABLE_LIST::derived and see if that works? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 0f35480f3d6: MDEV-6707: Wrong result (extra row) with group by, multi-part key
by Sergey Petrunia 20 Mar '18

20 Mar '18

Hi Varun, Nice to see feedback for the first review fixed, but there's more to do. Please find more input below. On Fri, Jan 19, 2018 at 01:07:20AM +0530, Varun wrote: > revision-id: 0f35480f3d60fac91bf1f91f8140ac5f55724139 (mariadb-5.5.56-142-g0f35480f3d6) > parent(s): fafdac3365f4943e73bcefd0e0d07d69997a9724 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-01-18 23:53:35 +0530 > message: > > MDEV-6707: Wrong result (extra row) with group by, multi-part key > > This case involves using a composite key,few parts of which are involved in GROUP BY and few in the MIN/MAX > functions in the select list. > Ranges are created in accordance with the where condition, so during the execution of such queries, > we try to find a prefix using the fields involved in GROUP BY > and we then check if this newly returned prefix lies within the range we had calculated earlier. > > For queries which use composite key, few parts of which are involved in GROUP BY and few in the MIN/MAX functtions > in the select list, we try to find the prefix of the ranges by using the fields involved in the group by clause. > We get extra rows in the output when we have same partial ranges created for the fields in the GROUP BY clause > > This issue can be fixed if we compare such partial ranges and don't lookup if we see the same prefix again. > > --- > mysql-test/r/range.result | 14 +++++++++++++ > mysql-test/r/range_mrr_icp.result | 14 +++++++++++++ > mysql-test/t/range.test | 13 ++++++++++++ > sql/opt_range.cc | 44 ++++++++++++++++++++++++++++++++++++--- > sql/opt_range.h | 5 +++-- > 5 files changed, 85 insertions(+), 5 deletions(-) > > diff --git a/mysql-test/r/range.result b/mysql-test/r/range.result > index 630a692cef6..72b31de4df4 100644 > --- a/mysql-test/r/range.result > +++ b/mysql-test/r/range.result > @@ -2144,3 +2144,17 @@ value1 1000685 12345 > value1 1003560 12345 > value1 1004807 12345 > drop table t1; > +# > +# MDEV-6707: Wrong result (extra row) with group by, multi-part key > +# > +CREATE TABLE t1 (f1 INT, f2 VARCHAR(1), KEY(f2,f1)) ENGINE=InnoDB; > +INSERT INTO t1 VALUES > +(7,'v'),(0,'s'),(9,'l'),(4,'c'); > +explain > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range f2 f2 9 NULL 2 Using where; Using index for group-by > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +MAX(f1) f2 > +4 c > +DROP TABLE t1; > diff --git a/mysql-test/r/range_mrr_icp.result b/mysql-test/r/range_mrr_icp.result > index 3f5de5b0189..1050bfcd887 100644 > --- a/mysql-test/r/range_mrr_icp.result > +++ b/mysql-test/r/range_mrr_icp.result > @@ -2146,4 +2146,18 @@ value1 1000685 12345 > value1 1003560 12345 > value1 1004807 12345 > drop table t1; > +# > +# MDEV-6707: Wrong result (extra row) with group by, multi-part key > +# > +CREATE TABLE t1 (f1 INT, f2 VARCHAR(1), KEY(f2,f1)) ENGINE=InnoDB; > +INSERT INTO t1 VALUES > +(7,'v'),(0,'s'),(9,'l'),(4,'c'); > +explain > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range f2 f2 9 NULL 2 Using where; Using index for group-by > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +MAX(f1) f2 > +4 c > +DROP TABLE t1; > set optimizer_switch=@mrr_icp_extra_tmp; > diff --git a/mysql-test/t/range.test b/mysql-test/t/range.test > index 393ca68e945..62e907b7b4a 100644 > --- a/mysql-test/t/range.test > +++ b/mysql-test/t/range.test > @@ -1718,3 +1718,16 @@ where (key1varchar='value1' AND (key2int <=1 OR key2int > 1)); > --echo # The following must show col1=12345 for all rows: > select * from t1; > drop table t1; > + > +--echo # > +--echo # MDEV-6707: Wrong result (extra row) with group by, multi-part key > +--echo # > + > +CREATE TABLE t1 (f1 INT, f2 VARCHAR(1), KEY(f2,f1)) ENGINE=InnoDB; > +INSERT INTO t1 VALUES > +(7,'v'),(0,'s'),(9,'l'),(4,'c'); > + > +explain > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +DROP TABLE t1; > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index 25a9e729a8b..be959764b16 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -11249,6 +11249,7 @@ int QUICK_RANGE_SELECT::get_next() > @param prefix_length length of cur_prefix > @param group_key_parts The number of key parts in the group prefix > @param cur_prefix prefix of a key to be searched for > + @param save_last_range INOUT Saving the last range we encountered > > Each subsequent call to the method retrieves the first record that has a > prefix with length prefix_length and which is different from cur_prefix, > @@ -11271,7 +11272,8 @@ int QUICK_RANGE_SELECT::get_next() > > int QUICK_RANGE_SELECT::get_next_prefix(uint prefix_length, > uint group_key_parts, > - uchar *cur_prefix) > + uchar *cur_prefix, > + QUICK_RANGE **save_last_range) > { > DBUG_ENTER("QUICK_RANGE_SELECT::get_next_prefix"); > const key_part_map keypart_map= make_prev_keypart_map(group_key_parts); > @@ -11301,8 +11303,43 @@ int QUICK_RANGE_SELECT::get_next_prefix(uint prefix_length, > last_range= 0; > DBUG_RETURN(HA_ERR_END_OF_FILE); > } > + > last_range= *(cur_range++); > > + /* > + While calculating these prefixes we might encounter a case where there > + woud be same partial ranges for multiple ranges. Typo. > + An example would be > + select max(f1), f2 from t1 where f2 ='c' AND f1 <> 9 group by f2; > + so the ranges would be > + (NULL,c <= f1,f2 <= c,9) > + (c,9 <= f1,f2 <= c,+infinity) This looks as if index was defined on (f1, f2) ? Then the comment doesn't make sense because loose index scan does not support queries in form: select max(key_part1) ... group by key_part2 It only can handle them when the index is on (f2,f1), but this is counterintuitive for the reader that is not familiar with the testcase for this bug. Please rephrase the comment. Please use key_part1 and key_part2 as column names. Tuple comparisons should look like this (note the braces): (a,b) < (c,d) . > + > + In this case for calculating prefixes with the group by field we take up the > + partial ranges involving field f2 those would be > + c<= f2 <=c > + c<= f2 <=c > + > + So we lookup rows with the same prefix in all such ranges and > + then we check for the other part(in this case f1) in ALL the ranges. > + So if a record lies in a range, then it would satisfy both the partial > + ranges in this case and therefore there would be multiple outputs for > + the same row. > + For such cases we should calculate the prefix only when we have the next > + partial range different from the previous one. > + */ > + > + > + if (*save_last_range) > + { > + if (!key_tuple_cmp(key_part_info, (*save_last_range)->min_key, > + last_range->min_key, prefix_length)) > + { > + last_range=NULL; > + continue; > + } > + } > + *save_last_range= last_range; > key_range start_key, end_key; > last_range->make_min_endpoint(&start_key, prefix_length, keypart_map); > last_range->make_max_endpoint(&end_key, prefix_length, keypart_map); > @@ -13315,7 +13352,7 @@ QUICK_GROUP_MIN_MAX_SELECT(TABLE *table, JOIN *join_arg, bool have_min_arg, > seen_first_key(FALSE), doing_key_read(FALSE), min_max_arg_part(min_max_arg_part_arg), > key_infix(key_infix_arg), key_infix_len(key_infix_len_arg), > min_functions_it(NULL), max_functions_it(NULL), > - is_index_scan(is_index_scan_arg) > + is_index_scan(is_index_scan_arg), save_last_range(NULL) This is not the right place for this initialization. It needs to be in QUICK_GROUP_MIN_MAX_SELECT::reset(), as the quick select may be executed multiple times. Testcase: create table ten(a int); insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); MariaDB [test]> select (SELECT MAX(f1) as MAXVAL FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2) AS SUBQ from ten; +------+ | SUBQ | +------+ | 4 | | 4 | | 4 | | 4 | | 4 | | 4 | | 4 | | 4 | | 4 | | 4 | +------+ 10 rows in set (0.00 sec) Now, let's add a HAVING clause (which always evaluates to TRUE but forces the subquery to be recomputed: MariaDB [test]> select (SELECT MAX(f1) as MAXVAL FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2 having maxval<100+ten.a) AS SUBQ from ten; +------+ | SUBQ | +------+ | 4 | | NULL | | NULL | | NULL | | NULL | | NULL | | NULL | | NULL | | NULL | | NULL | +------+ 10 rows in set (0.00 sec) To make sure this is loose index, try with IGNORE INDEX: MariaDB [test]> select (SELECT MAX(f1) as MAXVAL FROM t1 ignore index(f2) WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2 having maxval<100+ten.a) AS SUBQ from ten; +------+ | SUBQ | +------+ | 4 | | 4 | | 4 | | 4 | | 4 | | 4 | | 4 | | 4 | | 4 | | 4 | +------+ 10 rows in set (0.00 sec) Please fix this and add the above into the testcase. > { > head= table; > index= use_index; > @@ -13968,7 +14005,8 @@ int QUICK_GROUP_MIN_MAX_SELECT::next_prefix() > uchar *cur_prefix= seen_first_key ? group_prefix : NULL; > if ((result= quick_prefix_select->get_next_prefix(group_prefix_len, > group_key_parts, > - cur_prefix))) > + cur_prefix, > + &save_last_range))) > DBUG_RETURN(result); > seen_first_key= TRUE; > } > diff --git a/sql/opt_range.h b/sql/opt_range.h > index b8b46ae5ab1..206d65f6da4 100644 > --- a/sql/opt_range.h > +++ b/sql/opt_range.h > @@ -477,7 +477,7 @@ class QUICK_RANGE_SELECT : public QUICK_SELECT_I > int get_next(); > void range_end(); > int get_next_prefix(uint prefix_length, uint group_key_parts, > - uchar *cur_prefix); > + uchar *cur_prefix, QUICK_RANGE **save_last_range); > bool reverse_sorted() { return 0; } > bool unique_key_range(); > int init_ror_merged_scan(bool reuse_handler, MEM_ROOT *alloc); > @@ -916,7 +916,8 @@ class QUICK_GROUP_MIN_MAX_SELECT : public QUICK_SELECT_I > Use index scan to get the next different key instead of jumping into it > through index read > */ > - bool is_index_scan; > + bool is_index_scan; > + QUICK_RANGE *save_last_range; Please add a comment describing the new member variable. > public: > /* > The following two members are public to allow easy access from BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] GSoC 2018 Student Self Introduction & Question
by zhzhzoo zhang 20 Mar '18

20 Mar '18

Hello everybody, I am a first year graduate in Peking University major in math. I started to take part in algorithm contests at grade 7 so I have a lot of programming experience and I am very well trained in algorithms and data structures. In my childhood memory MySQL was the most popular DBMS choice for personal website (LAMP stack). However it was acquired by Oracle as I grew up. In this year's GSoC projects, `optimizer trace' interested me particularly. It would be great for MariaDB to have this functionality. This functionality is already implemented in MySQL. So my question is, can I read MySQL's source code implementing this function? Would that cause legal problems? Thanks very much! Regards, Zhang Shuai.

3 2

Re: [Maria-developers] [Commits] a99dcdf7e9e: MDEV-6731: Valgrind warnings 'Invalid read' in subselect_engine::calc_const_tables with SQ in WHERE and
by Sergey Petrunia 16 Mar '18

16 Mar '18

Hi Varun, Please fix the MDEV# in the commit comment. Ok to push after that. On Wed, Mar 14, 2018 at 12:11:20PM +0530, Varun wrote: > revision-id: a99dcdf7e9ede3b96466eb1294284141805747fc (mariadb-10.0.30-308-ga99dcdf7e9e) > parent(s): 48c11d407b409a1291c84782518c69ee9138ce72 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-03-14 12:09:49 +0530 > message: > > MDEV-6731: Valgrind warnings 'Invalid read' in subselect_engine::calc_const_tables with SQ in WHERE and > HAVING, ORDER BY, materialization+semijoin > > During cleanup a pointer to the materialised table that was freed was not set to NULL > > --- > mysql-test/r/having.result | 14 ++++++++++++++ > mysql-test/t/having.test | 18 ++++++++++++++++++ > sql/sql_select.cc | 4 +++- > 3 files changed, 35 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/having.result b/mysql-test/r/having.result > index 0b933427303..99059c25fbd 100644 > --- a/mysql-test/r/having.result > +++ b/mysql-test/r/having.result > @@ -723,4 +723,18 @@ SELECT * FROM t1 JOIN t2 ON c1 = c2 HAVING c2 > 'a' ORDER BY c2 LIMIT 1; > c1 c2 > x x > DROP TABLE t1,t2; > +# > +# MDEV-6736: Valgrind warnings 'Invalid read' in subselect_engine::calc_const_tables with SQ > +# in WHERE and HAVING, ORDER BY, materialization+semijoin > +# > +CREATE TABLE t1 (a INT) ENGINE=MyISAM; > +INSERT INTO t1 VALUES (3),(8); > +CREATE TABLE t2 (b INT) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (2),(1); > +SELECT a FROM t1 > +WHERE 9 IN ( SELECT MIN( a ) FROM t1 ) > +HAVING a <> ( SELECT COUNT(*) FROM t2 ) > +ORDER BY a; > +a > +DROP TABLE t1,t2; > End of 10.0 tests > diff --git a/mysql-test/t/having.test b/mysql-test/t/having.test > index 1682fe5b874..e624cd630dd 100644 > --- a/mysql-test/t/having.test > +++ b/mysql-test/t/having.test > @@ -759,4 +759,22 @@ SELECT * FROM t1 JOIN t2 ON c1 = c2 HAVING c2 > 'a' ORDER BY c2 LIMIT 1; > > DROP TABLE t1,t2; > > +--echo # > +--echo # MDEV-6736: Valgrind warnings 'Invalid read' in subselect_engine::calc_const_tables with SQ > +--echo # in WHERE and HAVING, ORDER BY, materialization+semijoin > +--echo # > + > +CREATE TABLE t1 (a INT) ENGINE=MyISAM; > +INSERT INTO t1 VALUES (3),(8); > + > +CREATE TABLE t2 (b INT) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (2),(1); > + > +SELECT a FROM t1 > +WHERE 9 IN ( SELECT MIN( a ) FROM t1 ) > +HAVING a <> ( SELECT COUNT(*) FROM t2 ) > +ORDER BY a; > + > +DROP TABLE t1,t2; > + > --echo End of 10.0 tests > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index fd8ff6eb016..37d68c730dd 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -11476,13 +11476,15 @@ void JOIN_TAB::cleanup() > } > else > { > + TABLE_LIST *tmp= table->pos_in_table_list; > end_read_record(&read_record); > - table->pos_in_table_list->jtbm_subselect->cleanup(); > + tmp->jtbm_subselect->cleanup(); > /* > The above call freed the materializedd temptable. Set it to NULL so > that we don't attempt to touch it if JOIN_TAB::cleanup() is invoked > multiple times (it may be) > */ > + tmp->table= NULL; > table=NULL; > } > DBUG_VOID_RETURN; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] d8e3cfacebc: MDEV-14616: WSREP has not yet prepared node for application use error
by Sergei Golubchik 15 Mar '18

15 Mar '18

Hi, Jan! On Feb 13, jan wrote: > revision-id: d8e3cfacebc47849532ef3e7bbf18f9da53bab88 (mariadb-10.1.31-20-gd8e3cfacebc) > parent(s): 474f3df092cc73342b2da03aa03ba5c66930f596 > author: Jan Lindström > committer: Jan Lindström > timestamp: 2018-02-13 10:04:48 +0200 > message: > > MDEV-14616: WSREP has not yet prepared node for application use error > > Contains following Galera fixes: > > MW-405 Make sure wsrep is ready in wait_until_connected_again.inc What is "MW-405", can I see the original bug report somewhere? > wait_until_connected_again issues 'SHOW STATUS' query repeatedly > until mysqld replies without errors. However, SHOW STATUS is > treated specially by wsrep in that it is allowed to proceed > even if wsrep is not yet in ready state. As a consequence, > after returning from wait_until_connected_again, wsrep may > not be ready yet and subsequent queries may fail with error > "1047 WSREP has not yet prepared node for application use". > To avoid those errors, the patch includes > wait_wsrep_ready.inc at the end of the wait_until_connected_again.inc I don't quite understand the issue. Old code did wait_wsrep_ready.inc after wait_until_connected_again.inc. Now wait_wsrep_ready.inc is included in wait_until_connected_again.inc. This seems to imply that the bug was "there were some cases where wait_wsrep_ready.inc was not included after wait_until_connected_again.inc". There are, of course, two ways to fix such a bug. Either include wait_wsrep_ready.inc after wait_until_connected_again.inc, or include wait_wsrep_ready.inc in wait_until_connected_again.inc. The latter fix implies that there are no possible cases when one would want to wait for a server to be connected, but before wsrep is ready. Are there? For the purpose of testing, that could be useful, could it not? > kill_galera.inc can no longer rely on wait_until_connected_again.inc. > This is because wait_until_connected_again now tries to make sure > that the server it is connected eventually transition to ready state. > Whereas some tests may need to kill galera while the server is in a > non-primary view. Which means that there are cases when a test needs the server connected, but before wsrep is ready, right? > MW-408 Fix 'WSREP error while trying to determine node state' > > mysql-test-run.pl sporadically reports 'WSREP error while > trying to determine node state' right after starting servers > for test execution. This happens because we try to execute a > SELECT statement that queries the current value of status variable > wsrep_ready. If this statement fails, the above message is reported. > The failure is due to fact that wsrep may return error > ER_LOCK_WAIT_TIMEOUT (on any SELECT statement) if it is not ready > and wsrep_sync_wait enabled for SELECTs. The fix is to disable > wsrep_sync_wait for the session that issues those SELECT statements. 1. Could you put this fix in a separate commit please? 2. Sorry, I don't understand. Where do "we try to execute a SELECT statement that queries the current value of status variable wsrep_ready" ? > --- > mysql-test/include/restart_mysqld.inc | 3 - > mysql-test/include/start_mysqld.inc | 3 - > mysql-test/include/wait_until_connected_again.inc | 11 ++- > mysql-test/mysql-test-run.pl | 104 ++++++++++++++++++++++ > mysql-test/suite/galera/include/kill_galera.inc | 1 - > 5 files changed, 111 insertions(+), 11 deletions(-) > > diff --git a/mysql-test/include/restart_mysqld.inc b/mysql-test/include/restart_mysqld.inc > index a0447280ff5..940e081c431 100644 > --- a/mysql-test/include/restart_mysqld.inc > +++ b/mysql-test/include/restart_mysqld.inc > @@ -50,9 +50,6 @@ if (!$restart_parameters) > # Call script that will poll the server waiting for it to be back online again > --source include/wait_until_connected_again.inc > > -# Wait for wsrep > ---source include/wait_wsrep_ready.inc > - > # Turn off reconnect again > --disable_reconnect > > diff --git a/mysql-test/include/start_mysqld.inc b/mysql-test/include/start_mysqld.inc > index 04dff714d49..e31f26aad8c 100644 > --- a/mysql-test/include/start_mysqld.inc > +++ b/mysql-test/include/start_mysqld.inc > @@ -16,9 +16,6 @@ if (!$restart_parameters) > # Call script that will poll the server waiting for it to be back online again > --source include/wait_until_connected_again.inc > > -# Wait for wsrep > ---source include/wait_wsrep_ready.inc > - > # Turn off reconnect again > --disable_reconnect > > diff --git a/mysql-test/include/wait_until_connected_again.inc b/mysql-test/include/wait_until_connected_again.inc > index 6f64ef45440..4958d276ebd 100644 > --- a/mysql-test/include/wait_until_connected_again.inc > +++ b/mysql-test/include/wait_until_connected_again.inc > @@ -11,10 +11,7 @@ let $counter= 5000; > let $mysql_errno= 9999; > while ($mysql_errno) > { > - # Strangely enough, the server might return "Too many connections" > - # while being shutdown, thus 1040 is an "allowed" error > - # See BUG#36228 > - --error 0,1040,1053,2002,2003,2005,2006,2013,1927 > + --error 0,ER_SERVER_SHUTDOWN,ER_CONNECTION_KILLED,2002,2003,2006,2013 > show status; > > dec $counter; > @@ -26,3 +23,9 @@ while ($mysql_errno) > } > --enable_query_log > --enable_result_log > + > +# WSREP: SHOW STATUS queries are allowed even if wsrep > +# is not ready. Make sure wsrep is ready before > +# returning from this script > + > +--source include/wait_wsrep_ready.inc > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index eaec51b82b4..324f2e98ebd 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -2883,6 +2883,49 @@ sub mysql_server_wait { > $warn_seconds); > } > > +sub have_wsrep() { > + my $wsrep_on= $mysqld_variables{'wsrep-on'}; > + return defined $wsrep_on > +} > + > + > +sub check_wsrep_support() { > + if (have_wsrep()) > + { > + mtr_report(" - binaries built with wsrep patch"); > + > + # ADD scripts to $PATH to that wsrep_sst_* can be found > + my ($path) = grep { -f "$_/wsrep_sst_rsync"; } "$::bindir/scripts", $::path_client_bindir; > + mtr_error("No SST scripts") unless $path; > + $ENV{PATH}="$path:$ENV{PATH}"; > + > + # Check whether WSREP_PROVIDER environment variable is set. > + if (defined $ENV{'WSREP_PROVIDER'}) { > + if ((mtr_file_exists($ENV{'WSREP_PROVIDER'}) eq "") && > + ($ENV{'WSREP_PROVIDER'} ne "none")) { > + mtr_error("WSREP_PROVIDER env set to an invalid path"); > + } > + # WSREP_PROVIDER is valid; set to a valid path or "none"). > + mtr_verbose("WSREP_PROVIDER env set to $ENV{'WSREP_PROVIDER'}"); > + } else { > + # WSREP_PROVIDER env not defined. Lets try to locate the wsrep provider > + # library. > + my $file_wsrep_provider= > + mtr_file_exists("/usr/lib/galera/libgalera_smm.so", > + "/usr/lib64/galera/libgalera_smm.so"); > + > + if ($file_wsrep_provider ne "") { > + # wsrep provider library found ! > + mtr_verbose("wsrep provider library found : $file_wsrep_provider"); > + $ENV{'WSREP_PROVIDER'}= $file_wsrep_provider; > + } else { > + mtr_verbose("Could not find wsrep provider library, setting it to 'none'"); > + $ENV{'WSREP_PROVIDER'}= "none"; > + } > + } > + } > +} > + > sub create_config_file_for_extern { > my %opts= > ( > @@ -3341,6 +3384,62 @@ sub run_query { > return $res > } > > +sub run_query_output { > + my ($mysqld, $query, $outfile)= @_; > + > + my $args; > + mtr_init_args(\$args); > + mtr_add_arg($args, "--defaults-file=%s", $path_config_file); > + mtr_add_arg($args, "--defaults-group-suffix=%s", $mysqld->after('mysqld')); > + > + mtr_add_arg($args, "--silent"); > + mtr_add_arg($args, "--execute=%s", $query); > + > + my $res= My::SafeProcess->run > + ( > + name => "run_query_output -> ".$mysqld->name(), > + path => $exe_mysql, > + args => \$args, > + output => $outfile, > + error => $outfile > + ); > + > + return $res > +} > + > +sub wait_wsrep_ready($$) { > + my ($tinfo, $mysqld)= @_; > + > + my $sleeptime= 100; # Milliseconds > + my $loops= ($opt_start_timeout * 1000) / $sleeptime; > + > + my $name= $mysqld->name(); > + my $outfile= "$opt_vardir/tmp/$name.wsrep_ready"; > + my $query= "SET SESSION wsrep_sync_wait = 0; > + SELECT VARIABLE_VALUE > + FROM INFORMATION_SCHEMA.GLOBAL_STATUS > + WHERE VARIABLE_NAME = 'wsrep_ready'"; > + > + for (my $loop= 1; $loop <= $loops; $loop++) > + { > + if (run_query_output($mysqld, $query, $outfile) != 0) > + { > + $tinfo->{logfile}= "WSREP error while trying to determine node state"; > + return 0; > + } > + > + if (mtr_grab_file($outfile) =~ /^ON/) > + { > + unlink($outfile); > + return 1; > + } > + > + mtr_milli_sleep($sleeptime); > + } > + > + $tinfo->{logfile}= "WSREP did not transition to state READY"; > + return 0; > +} > > sub do_before_run_mysqltest($) > { > @@ -5380,6 +5479,11 @@ sub start_servers($) { > $tinfo->{comment}= "Failed to start ".$_->name() . "\n"; > return 1; > } > + > + if (have_wsrep() && !wait_wsrep_ready($tinfo, $_)) > + { > + return 1; > + } > } > return 0; > } > diff --git a/mysql-test/suite/galera/include/kill_galera.inc b/mysql-test/suite/galera/include/kill_galera.inc > index c61bad8e19d..f95dccf2185 100644 > --- a/mysql-test/suite/galera/include/kill_galera.inc > +++ b/mysql-test/suite/galera/include/kill_galera.inc > @@ -18,4 +18,3 @@ > exit(0); > EOF > > ---source include/wait_until_disconnected.inc > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] dc3d6c9edb3: MDEV-15477: SESSION_SYSVARS_TRACKER does not track last_gtid
by Sergei Golubchik 09 Mar '18

09 Mar '18

Hi, Oleksandr! On Mar 09, Oleksandr Byelkin wrote: > revision-id: dc3d6c9edb3536e5f0c87ac1e9c04891d50fc420 (mariadb-10.2.13-32-gdc3d6c9edb3) > parent(s): 6cf25e5eb76923550173a0059036d80c80a053d6 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-03-09 14:39:40 +0100 > message: > > MDEV-15477: SESSION_SYSVARS_TRACKER does not track last_gtid > > register changes of last_gtid > > diff --git a/mysql-test/t/session_tracker_last_gtid.test b/mysql-test/t/session_tracker_last_gtid.test > --- /dev/null > +++ b/mysql-test/t/session_tracker_last_gtid.test > @@ -0,0 +1,18 @@ > + > +--source include/have_innodb.inc > +--source include/have_binlog_format_mixed_or_statement.inc Why "mixed or statement"? > diff --git a/sql/log.cc b/sql/log.cc > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -5945,7 +5945,20 @@ MYSQL_BIN_LOG::write_gtid_event(THD *thd, bool standalone, > } > if (err) > DBUG_RETURN(true); > + bool changed_gtid= (thd->last_commit_gtid.seq_no != gtid.seq_no); > thd->last_commit_gtid= gtid; > + if (changed_gtid && > + thd->session_tracker.get_tracker(SESSION_SYSVARS_TRACKER)->is_enabled()) > + { > + sys_var *svar; > + mysql_mutex_lock(&LOCK_plugin); > + if ((svar= find_sys_var_ex(thd, "last_gtid", > + sizeof("last_gtid") - 1, > + false, true))) > + thd->session_tracker.get_tracker(SESSION_SYSVARS_TRACKER)-> > + mark_as_changed(thd, (LEX_CSTRING*)svar); > + mysql_mutex_unlock(&LOCK_plugin); > + } No, please. You svar will always be Sys_last_gtid. There is no need to search for it on every write_gtid_event(), under a mutex. See in sys_vars.cc around Sys_autocommit how to do it easier. Also, perhaps it'd be safer to rename last_commit_gtid to m_last_commit_gtid, make it private and provide a setter and a getter. The setter will update the tracker. I'll make sure that last_commit_gtid is not changed anywhere else bypassing the tracker. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 6cf25e5eb76: Session tracking info support in mysqltest
by Sergei Golubchik 09 Mar '18

09 Mar '18

Hi, Oleksandr! On Mar 09, Oleksandr Byelkin wrote: > revision-id: 6cf25e5eb76923550173a0059036d80c80a053d6 (mariadb-10.2.13-31-g6cf25e5eb76) > parent(s): f2b91c878cf4ec53c292caba7f5907d1ef724de4 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-03-09 14:27:13 +0100 > message: > > Session tracking info support in mysqltest Add a test case for it. Don't forget to test whether ONCE modifier works. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] f2b91c878cf: Reset connection support in mysqltest
by Sergei Golubchik 09 Mar '18

09 Mar '18

Hi, Oleksandr! On Mar 09, Oleksandr Byelkin wrote: > revision-id: f2b91c878cf4ec53c292caba7f5907d1ef724de4 (mariadb-10.2.13-30-gf2b91c878cf) > parent(s): 8f98835bb86550a68a6c412d8417cc169dfdc206 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-03-09 14:26:10 +0100 > message: > > Reset connection support in mysqltest You add a feature - add a test case for it too. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MariaDB 5.5, Jemalloc, TokuDB, RHEL
by Michal Schorm 08 Mar '18

08 Mar '18

Hello, so far we packed MariaDB 5.5 for RHEL 7 without TokuDB and without Jemalloc. Recently, a customer asked about the availability of TokuDB, so I'm doing a research, if its possible to pack it without any issues for the users updating. I have a several questions: 1) Is Jemmaloc used elsewhere from TokuDB ? 2) If yes, what all would be affected? Could any issues arise during update from build without Jemalloc to build with it? Any expected performance changes? 3) Would be the use of the bundled Jemalloc (on of the) best option(s)? 4) Anything else that I missed? -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

2 1

[Maria-developers] Introduction as GSoC 2018 aspirant
by Arshdeep Singh Sachdeva 05 Mar '18

05 Mar '18

Dear MariaDB Developers, I am Arshdeep Singh Sachdeva, currently studying BTech ( C.S.E. ) from Punjab Engineering College, Chandigarh, India. I am interested in contibuting to MariaDB and participating in GSoC 2018 under your guidance. I am proficient in C++, python, linux, mysql. I have already used MariaDB in my projects in college. Also I am presently going through documentation and tutorials. Thanks.

1 0

[Maria-developers] Introduction as a GSoC aspirant
by Aman Jain 05 Mar '18

05 Mar '18

Hello developers and mentors I am Aman and I am interested in contributing to MariaDB as a GSoC'18 aspirant. I am familiar with C++, Web development, SQL, MongoDB and shell scripting and I have set up the MariaDB server. Please suggest me where I can find the source code so that I can get a better understanding of the project ideas and solve some issues to get started. Thanks Aman Jain

2 1

Re: [Maria-developers] [Commit] MDEV-12746 out-of-order retry
by Kristian Nielsen 17 Feb '18

17 Feb '18

andrei.elkin(a)pp.inet.fi writes: > commit 3cebb54e6387a7eace1757c82ed0efd6e11590b9 > Author: Andrei Elkin <andrei.elkin(a)mariadb.com> > Date: Fri Feb 9 15:00:23 2018 +0200 > > MDEV-12746 rpl.rpl_parallel_optimistic_nobinlog fails committing > out of order at retry > > The 2nd issue was out of order commit by transactions that > are ordered after an erroring out transaction. The latter was possible > in the test thanks to the above retry failure. > Out-of-order committing was done despite the slave was stopping. Let me see if I understand this correctly: Normally, a transaction in optimistic parallel replication cannot commit out of order because it either waits for the prior transaction to commit first, or, if the prior transaction has stopped with an error earlier, then the following transaction will not even start. But in this case, during retry, there was missing the check that the prior transaction stopped with an error. Therefore, the following transaction's wait_for_prior_commit() does nothing, and it wrongly goes ahead and runs and even commits, even though an earlier transaction failed and replication is stopping. Correct? That's a nice catch of a subtle, but potentially serious bug. I wonder how this is possible though. As I recall, when a transaction fails, this failure is marked in the wait_for_prior_commit data structures. So the retrying transaction should have gotten an error in wait_for_prior_commit() and aborted the retries. So why does this not happen in this case? > The test failures were of two sorts. One is that the number of potential > temporary errors actually exceeds the default value of the slave retry > option. > Increased @@global.slave_transaction_retries fixes that part. Hm, this shouldn't be needed, I think. The retry waits for all prior transactions to commit first, so after the first retry no further retries should be caused by other parallel replication workers. So I would expect this problem to be a side-effect of the other, real issue, and this increase should not be needed (and might hide another problem later, if left in). > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 34f9113f7fe..9f1d23a1161 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -745,7 +761,25 @@ retry_event_group(rpl_group_info *rgi, rpl_parallel_thread *rpt, > for (;;) > { > mysql_mutex_lock(&entry->LOCK_parallel_entry); > - register_wait_for_prior_event_group_commit(rgi, entry); > + if (entry->stop_on_error_sub_id == (uint64) ULONGLONG_MAX || > +#ifndef DBUG_OFF > + (DBUG_EVALUATE_IF("simulate_mdev_12746", 1, 0)) || > +#endif > + rgi->gtid_sub_id < entry->stop_on_error_sub_id) > + { > + register_wait_for_prior_event_group_commit(rgi, entry); > + } > + else > + { > + /* > + An earlier transaction by another worker may have not had not > + have the current one in subsequent committers list and thus > + the actual induced error status could not have been sumbitted. > + So it should be set by ourselves now. > + */ This comment is hard to understand - what do you mean here? > + if (!rgi->worker_error) > + rgi->worker_error= 1; > + } I would have expected an abort (goto err) here (or well just below, after unlocking the mutex). Is this not needed, and if so, why not? > @@ -716,12 +724,20 @@ retry_event_group(rpl_group_info *rgi, rpl_parallel_thread *rpt, > unregistering (and later re-registering) the wait. > */ > if(thd->wait_for_commit_ptr) > - thd->wait_for_commit_ptr->unregister_wait_for_prior_commit(); > + thd->wait_for_commit_ptr->unregister_wait_for_prior_commit(); Indentation? - Kristian.

2 5

[Maria-developers] Developers Unconference in New York
by Ian Gilfillan 16 Feb '18

16 Feb '18

A final reminder about the Developers Unconference happening in New York on Sat 24 and Sun 25 February, just before M|18. https://mariadb.org/2018-mariadb-developers-unconference/ Please sign up on the Meetup page if you are going to be attending: https://www.meetup.com/MariaDB-Developers/events/244375856/ Hope to see you there!

1 0

[Maria-developers] Google Summer of Code 2018 - MariaDB Foundation accepted
by Colin Charles 14 Feb '18

14 Feb '18

Hi, I just noticed this piece of good news — MariaDB Foundation is accepted to participate in the Google Summer of Code 2018 again — https://summerofcode.withgoogle.com/ There is an already healthy list of ideas: https://mariadb.com/kb/en/library/google-summer-of-code-2018/ Many already have mentors from MariaDB Corporation or MariaDB Foundation. Some are looking for mentors. Happy coding this summer! cheers, -colin -- Colin Charles, http://bytebot.net/blog/ twitter: @bytebot | skype: colincharles "First they ignore you, then they laugh at you, then they fight you, then you win." -- Mohandas Gandhi

1 0

Re: [Maria-developers] [Commits] f4f32ab71d5: MDEV-9744: session optimizer_use_condition_selectivity=5 causing SQL Error (1918):
by Sergey Petrunia 13 Feb '18

13 Feb '18

Hi Varun, On Sat, Feb 03, 2018 at 06:37:21PM +0530, Varun wrote: > revision-id: f4f32ab71d5c96fbec87f8b6a5c057af2bc2398b (mariadb-10.2.5-413-gf4f32ab71d5) > parent(s): 9390ff53fc39e34976cf051ce33649f9316bb8e6 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-02-03 18:24:49 +0530 > message: > > MDEV-9744: session optimizer_use_condition_selectivity=5 causing SQL Error (1918): > Encountered illegal value '' when converting to DECIMAL > > The issue was that EITS data was allocated but then not read for some reason (one being to avoid a deadlock), > then the optimizer was using these bzero'ed buffers as EITS statistics. > This should not be allowed, we should use statistcs for a table only when we have successfully loaded/read > the stats from the statistical tables. > > --- > mysql-test/r/statistics.result | 16 ++++++++++++++++ > mysql-test/t/statistics.test | 19 +++++++++++++++++++ > sql/sql_statistics.cc | 23 ++++++++++++++++------- > 3 files changed, 51 insertions(+), 7 deletions(-) > > diff --git a/mysql-test/r/statistics.result b/mysql-test/r/statistics.result > index ffaaf09acc8..1486a8ab5e6 100644 > --- a/mysql-test/r/statistics.result > +++ b/mysql-test/r/statistics.result > @@ -1716,5 +1716,21 @@ b > 0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.004,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.004,0.000,0.000,0.004,0.000,0.000,0.000,0.004,0.000,0.000,0.000,0.004,0.000,0.000,0.000,0.004,0.000,0.004,0.000,0.000,0.004,0.000,0.004,0.000,0.004,0.000,0.004,0.004,0.004,0.000,0.004,0.000,0.004,0.004,0.004,0.004,0.004,0.000,0.004,0.004,0.004,0.004,0.004,0.004,0.008,0.004,0.008,0.008,0.008,0.008,0.020,0.004,0.016,0.020,0.016,0.016,0.051,0.031,0.027,0.031,0.043,0.047,0.043,0.043,0.055,0.051,0.071,0.043,0.043,0.043,0.020,0.024,0.024,0.020,0.016,0.016,0.008,0.008,0.012,0.000 > DROP TABLE t1; > # > +# MDEB-9744: session optimizer_use_condition_selectivity=5 causing SQL Error (1918): Please change to "MDEV" > +# Encountered illegal value '' when converting to DECIMAL > +# > +set @save_optimizer_use_condition_selectivity= @@optimizer_use_condition_selectivity; > +set optimizer_use_condition_selectivity=3, use_stat_tables=preferably; > +create table a (id int(10),cost decimal(9,2)) engine=innodb; > +ANALYZE TABLE a PERSISTENT FOR ALL; > +Table Op Msg_type Msg_text > +test.a analyze status Engine-independent statistics collected > +test.a analyze status OK > +create temporary table b (id int); > +insert into b (id) select id from a where cost > 0; > +drop table a,b; Please follow the convention and use table names t1,t2,... Ok to push after this is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] MDEV-15167 Server crashes in in bitmap_bits_set upon REPAIR PARTITION ...
by Sergei Golubchik 10 Feb '18

10 Feb '18

Hi, Sachin! On Feb 09, Sachin Setiya wrote: > commit 3e49ebcacf9022931f0dfefcb44c13973128f8c4 > Author: Sachin Setiya <sachinsetia1001(a)gmail.com> > Date: Fri Feb 9 17:44:04 2018 +0530 > > MDEV-15167 Server crashes in in bitmap_bits_set upon REPAIR PARTITION > ... > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index a4d9406..3d53056 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -4583,6 +4583,11 @@ uint prep_alter_part_table(THD *thd, TABLE *table, > Alter_info *alter_info, > Alter_info::ALTER_TABLE_REORG | > Alter_info::ALTER_REBUILD_PARTITION)) > { > + /* > + You can't add column when we are doing alter related to partition > + */ > + DBUG_EXECUTE_IF("test_pseudo_invisible",{DBUG_ASSERT(0);}); > + DBUG_EXECUTE_IF("test_completely_invisible",{DBUG_ASSERT(0);}); May be return error? Like with P_S? DBUG_EXECUTE_IF("test_pseudo_invisible", {DBUG_RETURN(1)}); but test it, may be you'll need my_error too, like DBUG_EXECUTE_IF("test_pseudo_invisible", { my_error(ER_INTERNAL_ERROR, MYF(0), "Don't to it with test_pseudo_invisible"); DBUG_RETURN(1) }); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Oracle strings functions compatibility : substr / MDEV 10574
by jerome brauge 09 Feb '18

09 Feb '18

Hello Alexander, If you remember, there was still a problem with our function substr_oracle when length is less or equal to zero. Can you review this very small patch for this ? For MDEV-10574, If we overload only functions required for Oracle compatibility (ltrim/rtrim and lpad/rpad), could I make a similar patch ? (by adding functions ltrim_oracle, ... like for substr) It's for us the last blocking point. Regards, Jérôme. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : mardi 19 septembre 2017 13:16 > À : jerome brauge > Objet : Re: Oracle strings functions compatibility : substr > > Hi Jerome, > > I noticed one more difference in SUBSTR: > > > SELECT SUBSTR('aaa',1,-1) FROM DUAL; > > MariaDB returns an empty string. > Oracle returns NULL. > > Here's Oracle's documentation: > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions162.htm > > > I think we should do this in the same patch, to avoid behavior change in the > future. > > Would you like to try doing this additional change? > Or would you like me to make an incremental patch on top of yours? > > Thanks! > > On 09/19/2017 10:33 AM, jerome brauge wrote: > > Hello Alexander, > > It's done. > > > > Do you have news from Sergei for MDEV-12874, MDEV-13417 and MDEV- > 13418 ? > > I asked Sergei, awaiting for his answer. I'll let you know. > > > > > Thank you very much. > > > > > > > >> -----Message d'origine----- > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 18 > >> septembre 2017 18:12 À : jerome brauge Objet : Re: Oracle strings > >> functions compatibility : substr > >> > >> Hello Jerome, > >> > >> > >> On 09/18/2017 05:14 PM, jerome brauge wrote: > >>> Hello Alexander, > >>> I don't understand how I've missed such an obvious solution ! > >>> Here is the new patch. > >> > >> Thanks. Now it looks simpler. > >> > >> Can you please do one small thing: > >> move get_position() from "public:" to "protected:". > >> > >> Ok to make a pull request after this change. > >> > >> Thank you very much! > >> > >> > >>> > >>> Regards. > >>> Jérôme. > >>> > >>> > >>>> -----Message d'origine----- > >>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 18 > >>>> septembre 2017 11:30 À : jerome brauge Objet : Re: Oracle strings > >>>> functions compatibility : substr > >>>> > >>>> Hello Jerome, > >>>> > >>>> > >>>> On 08/17/2017 04:43 PM, jerome brauge wrote: > >>>>> Hello Alexander, > >>>>> Here is a patch for function SUBSTR in sql_mode=oracle (If > >>>>> position is 0, > >>>> then it is treated as 1). > >>>> > >>>> I'm thinking of how to get a smaller patch. > >>>> > >>>> Wouldn't it be possible to introduce: > >>>> > >>>> virtual longlong get_position(); > >>>> > >>>> > >>>> > >>>> longlong Item_func_substr::get_position() { > >>>> return args[1]->val_int(); > >>>> } > >>>> > >>>> longlong Item_func_substr_oracle::get_position() > >>>> { > >>>> longlong pos= args[1]->val_int(); > >>>> return pos == 0 ? 1 : pos; > >>>> } > >>>> > >>>> > >>>> and just replace all calls for args[1]->val_int() to get_position() ? > >>>> > >>>> Thanks! > >>>> > >>>> > >>>>> > >>>>> Regards, > >>>>> Jérôme. > >>>>>

1 1

Re: [Maria-developers] 9bc03294c89: MDEV-15033: Node fails to join mdb cluster with wsrep_sst_mariabackup due to dependency from socat
by Sergei Golubchik 06 Feb '18

06 Feb '18

Hi, Jan! No, sorry, you cannot do it. First, we cannot add new RPM dependencies in GA (we tried once and had to revert that change). Second, there was already socat dependency in RPM, added by Nirbhay in MDEV-6211 MariaDB-Galera-server uses 'socat', but 'socat' is not in the dependency list commit bd5ca5e6093046d7e69cc59f0950a7af2c0a7bf2 removed by Nirbhay in MDEV-7233: Fix issue with missing dependency socat when installing MariaDB-galera-server on RedhatEL/OracleEL/(Others?) RPM based commit 41367e4023d0823624a461597813e6902437f096 I don't see a good solution for this issue. Adding socat to mariabackup package aren't much better, as mariabackup users don't really need socat either. May be just a better error message in the script? On Feb 06, jan wrote: > revision-id: 9bc03294c89b16df5f556b7d9fa4cb1eadae451a (mariadb-10.1.30-31-g9bc03294c89) > parent(s): 1cd1db9602476ec6864f4a521de97d788fe93fac > author: Jan Lindström > committer: Jan Lindström > timestamp: 2018-02-06 10:59:27 +0200 > message: > > MDEV-15033: Node fails to join mdb cluster with wsrep_sst_mariabackup due to dependency from socat > > Add socat dependency to cpack_rpm.cmake. Similar dependency already > is on debian/control. > > diff --git a/cmake/cpack_rpm.cmake b/cmake/cpack_rpm.cmake > index e80fb199f03..5b9c6726b5e 100644 > --- a/cmake/cpack_rpm.cmake > +++ b/cmake/cpack_rpm.cmake > @@ -170,7 +170,7 @@ SETA(CPACK_RPM_server_PACKAGE_REQUIRES > IF(WITH_WSREP) > SETA(CPACK_RPM_server_PACKAGE_REQUIRES > "galera" "rsync" "lsof" "grep" "gawk" "iproute" > - "coreutils" "findutils" "tar") > + "coreutils" "findutils" "tar" "socat") > IF (RPM MATCHES "sles11") > SETA(CPACK_RPM_server_PACKAGE_REQUIRES "util-linux") > ELSE() Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] Triangular FKs - Cascade delete leaves DB in state with broken referential integrity
by Oliver Schonrock 05 Feb '18

05 Feb '18

Initially posted on "discuss" list... I have come across a very strange symptom with triangular FKs (see below) where a cascade delete of the central (member) record leaves the DB in a state with broken referential integrity. The FK structure is a little bit demanding and somewhat "triangular/circular", but has been in operation in our application for many years without problems. After recent upgrade from 10.1 to 10.2.12, we promptly got broken FKs. (I have not yet retested with 10.1). How to reproduce: (example reduced to the bare essentials with 3 tables with 1 record each). SET FOREIGN_KEY_CHECKS=0; CREATE TABLE member ( id int(11) NOT NULL AUTO_INCREMENT, default_address_id int(11) DEFAULT NULL, default_card_id int(11) DEFAULT NULL, PRIMARY KEY (id), KEY member_FI_2 (default_address_id), KEY member_FI_3 (default_card_id), CONSTRAINT member_FK_2 FOREIGN KEY (default_address_id) REFERENCES address (id) ON DELETE SET NULL ON UPDATE CASCADE, CONSTRAINT member_FK_3 FOREIGN KEY (default_card_id) REFERENCES payment_method (id) ON DELETE SET NULL ON UPDATE CASCADE ) ENGINE=InnoDB; INSERT INTO member VALUES (1,2,3); CREATE TABLE address ( id int(11) NOT NULL AUTO_INCREMENT, member_id int(11) NOT NULL, PRIMARY KEY (id), KEY address_FI_1 (member_id), CONSTRAINT address_FK_1 FOREIGN KEY (member_id) REFERENCES member (id) ON DELETE CASCADE ON UPDATE CASCADE ) ENGINE=InnoDB; INSERT INTO address VALUES (2,1); CREATE TABLE payment_method ( id int(11) NOT NULL AUTO_INCREMENT, member_id int(11) NOT NULL, cardholder_address_id int(11) DEFAULT NULL, PRIMARY KEY (id), KEY payment_method_FI_1 (member_id), KEY payment_method_FI_2 (cardholder_address_id), CONSTRAINT payment_method_FK_1 FOREIGN KEY (member_id) REFERENCES member (id) ON DELETE CASCADE ON UPDATE CASCADE, CONSTRAINT payment_method_FK_2 FOREIGN KEY (cardholder_address_id) REFERENCES address (id) ON DELETE SET NULL ON UPDATE CASCADE ) ENGINE=InnoDB; INSERT INTO payment_method VALUES (3,1,2); SET FOREIGN_KEY_CHECKS=1; SELECT * FROM member; +----+--------------------+-----------------+ | id | default_address_id | default_card_id | +----+--------------------+-----------------+ | 1 | 2 | 3 | +----+--------------------+-----------------+ SELECT * FROM address; +----+-----------+ | id | member_id | +----+-----------+ | 2 | 1 | +----+-----------+ SELECT * FROM payment_method; +----+-----------+-----------------------+ | id | member_id | cardholder_address_id | +----+-----------+-----------------------+ | 3 | 1 | 2 | +----+-----------+-----------------------+ DELETE FROM member WHERE id = 1; SELECT * FROM member; [ empty ] SELECT * FROM address; [ empty ] SELECT * FROM payment_method; +----+-----------+-----------------------+ | id | member_id | cardholder_address_id | +----+-----------+-----------------------+ | 3 | 1 | NULL | +----+-----------+-----------------------+ ------------ DB is now in an inconsistent state: payment_method.member_id=1 references a non-existent member record. Am I missing something, or should this never happen? Thanks -- Oliver Schönrock

1 6

[Maria-developers] Partial indexes?
by Jared Beck 04 Feb '18

04 Feb '18

Dear Maria, :p Thanks for all your hard work! Any plans to implement partial indexes? (https://www.postgresql.org/docs/10/static/indexes-partial.html) I hope this is the right place to suggest new features. Please redirect me if not. -- Jared Beck • (607) 216-5373 • jared(a)jaredbeck.com

3 7

[Maria-developers] MDEV-14425: Is appending never slower than writing to a preallocated file?
by Marko Mäkelä 02 Feb '18

02 Feb '18

I would like to redesign the InnoDB redo log format for better performance in MariaDB 10.4. Part of this would involve minimizing write amplification and optimizing for journaled file systems. Jun Su from Microsoft suggested to me that regular writes (as opposed to appends) could cause some write amplification inside journaled file systems. InnoDB traditionally pre-allocates both data and log files. Maybe it was a good idea in 1994 when the code was initially conceived. But we have had journaled or copy-on-write file systems and also SSDs for quite some time now. I wrote two test programs that write a 2GiB file in 2KiB blocks, either pre-allocating the file upfront, or appending to the file. On the two SSDs that I tested (with ext4fs), appending was always faster. The programs are attached to https://jira.mariadb.org/browse/MDEV-14425 I would appreciate it if someone can provide a counterexample where writing to a preallocated file would be faster than appending, on a modern file system. Also, I would like to see how HDDs would perform. With best regards, Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/

2 1

Re: [Maria-developers] [Commits] a0ea3a58224: MDEV-14857: problem with 10.2.11 server crashing when executing stored procedure
by Sergey Petrunia 31 Jan '18

31 Jan '18

Hi Sanja, Please find my review below. Lots of typos, but the two important comments are at the bottom, for the changes in sql_select.cc and sql_trigger.cc On Sat, Jan 27, 2018 at 11:39:10AM +0100, Oleksandr Byelkin wrote: > revision-id: a0ea3a5822486fa1702d6615f1be30684c3d412f (mariadb-10.1.30-26-ga0ea3a58224) > parent(s): 524221e7a34d42214e82dd868348b453f2ef1591 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-01-27 11:38:38 +0100 > message: > > MDEV-14857: problem with 10.2.11 server crashing when executing stored procedure > > Counter for select numbering made stored with the statement (before was global) > So now it does have always accurate value which does not depend on > interruption of statement prepare by errors like lack of table in > a view definition. > > --- > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 4556487bdfe..2dc2fb61e9d 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1027,6 +1027,21 @@ class Statement: public ilink, public Query_arena > > LEX_STRING name; /* name for named prepared statements */ > LEX *lex; // parse tree descriptor > + /* > + LEX whith represent curent statement (conventional, SP or PS) s/whith/which/ s/curent/current/ s/represent/represents/ > + > + For example during view parsing THD::lex will point to the views LEX and > + THD::stmt_lex will point on LEX of the statement where the view will be s/point on/point to/ > + included > + > + Now used to have alwais correct select numbering inside statemet s/Now used/Currently it is used/ s/alwais/always/ s/statemet/statement/ > + (LEX::current_select_number) without storing and restoring a global > + counter which was THD::select_number. > + > + TODO: make some unified statement representation (now SP has different) > + to store such data like LEX::current_select_number. > + */ > + LEX *stmt_lex; > /* > Points to the query associated with this statement. It's const, but > we need to declare it char * because all table handlers are written > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index b57fba08b47..a2da6f672ca 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -727,7 +727,13 @@ class st_select_lex: public st_select_lex_node > Item *prep_having;/* saved HAVING clause for prepared statement processing */ > /* Saved values of the WHERE and HAVING clauses*/ > Item::cond_result cond_value, having_value; > - /* point on lex in which it was created, used in view subquery detection */ > + /* > + Point on lex in which it was created, used in view subquery detection. "Points to the LEX...". Also please shift the whole comment 1 position to the left. > + > + TODO: make also st_select_lex::parent_stmt_lex (see THD::stmt_lex) > + and use st_select_lex::parent_lex & st_select_lex::parent_stmt_lex > + instead of global (from THD) references where it is possible. > + */ > LEX *parent_lex; > enum olap_type olap; > /* FROM clause - points to the beginning of the TABLE_LIST::next_local list. */ > @@ -2452,7 +2458,7 @@ struct LEX: public Query_tables_list > /** SELECT of CREATE VIEW statement */ > LEX_STRING create_view_select; > > - uint number_of_selects; // valid only for view > + uint current_select_number; // valid for statmet LEX (not view) s/statmet/statement/ > > /** Start of 'ON table', in trigger statements. */ > const char* raw_trg_on_table_name_begin; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 6084c59a257..8d3116cd8b3 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -6903,7 +6903,12 @@ void THD::reset_for_next_command(bool do_clear_error) > clear_error(1); > > thd->free_list= 0; > - thd->select_number= 1; > + /* > + The same thd->stmt_lex made in lex_start, but during bootstrup this "We also assign thd->stmt_lex in lex_start()" s/bootstrup/bootstrap > + code executed first code is executed > + */ > + thd->stmt_lex= &main_lex; thd->stmt_lex->current_select_number= 1; > + DBUG_PRINT("info", ("Lex %p stmt_lex: %p", thd->lex, thd->stmt_lex)); > /* > Those two lines below are theoretically unneeded as > THD::cleanup_after_query() should take care of this already. > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index b9fe8f3162a..7a6a028ee9c 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2462,6 +2462,17 @@ void JOIN::save_explain_data(Explain_query *output, bool can_overwrite, > bool need_tmp_table, bool need_order, > bool distinct) > { > + /* > + If there is SELECT in this statemet with the same number it must be the > + same SELECT > + */ > + DBUG_ASSERT(select_lex->select_number == UINT_MAX || > + select_lex->select_number == INT_MAX || > + !output || Is it really possible that output==NULL? If yes, that's probably a bug. > + !output->get_select(select_lex->select_number) || > + output->get_select(select_lex->select_number)->select_lex == > + select_lex); > + > if (select_lex->select_number != UINT_MAX && > select_lex->select_number != INT_MAX /* this is not a UNION's "fake select */ && > have_query_plan != JOIN::QEP_NOT_PRESENT_YET && > @@ -24601,6 +24612,11 @@ int JOIN::save_explain_data_intern(Explain_query *output, bool need_tmp_table, > { > explain= new (output->mem_root) Explain_select(output->mem_root, > thd->lex->analyze_stmt); > + if (!explain) > + DBUG_RETURN(1); // EoM > +#ifndef DBUG_OFF > + explain->select_lex= select_lex; > +#endif > join->select_lex->set_explain_type(true); > > explain->select_id= join->select_lex->select_number; > diff --git a/sql/sql_trigger.cc b/sql/sql_trigger.cc > index 0b4978b2862..878cd5a3b57 100644 > --- a/sql/sql_trigger.cc > +++ b/sql/sql_trigger.cc > @@ -1378,7 +1378,7 @@ bool Table_triggers_list::check_n_load(THD *thd, const char *db, > ulonglong save_sql_mode= thd->variables.sql_mode; > LEX_STRING *on_table_name; > > - thd->lex= &lex; > + thd->lex= thd->stmt_lex= &lex; > The same comment as before: we change thd->stmt_lex here. The function has code below that restores the value of thd->lex: // QQ: anything else ? lex_end(&lex); thd->lex= old_lex; But the value of thd->stmt_lex is not restored? > save_db.str= thd->db; > save_db.length= thd->db_length; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] Wording suggestion?
by Sergei Golubchik 30 Jan '18

30 Jan '18

Hi, Russell! On Jan 30, Russell J.T. Dyer wrote: > How about concurrent_assign ? I think "concurrent" already has a clear meaning, at least in MariaDB - operations are done "concurrently" if they're executed truly in parallel, usually in different threads. For example, both MyISAM and InnoDB allow concurrent SELECTs, but only InnoDB allows concurrent UPDATEs. I considered simultaneous_assign, as a slightly shorter alternative to simultaneous_assignment. But it looked somewhat unnatural to me. Still, this is the case when I happily yield to the opinion of a documentation writer :) If you think that simultaneous_assign looks just fine, I'll use it. > > It's not only for update. There is a patch to fix SELECT ... INTO, > > and I suppose we'll have "simultaneous assignment" behavior > > everywhere eventually. > > > > So, I'm leaning towards either simultaneous_assignment or > > simultaneous_set as mode names. The first is more exact, but it's > > rather long (sql_mode names range from 3 to 26 characters, and this > > one is 23). The second (with 16) fits right in the middle of the > > range, length-wise. But there's no SET keyword in SELECT ... INTO. > > On the other hand, there's no ASSIGNMENT keyword anywhere either :) > > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] Mdev-15085 Invisible Column Non-constant Default value results...
by Sergei Golubchik 30 Jan '18

30 Jan '18

Hi, Sachin! ok to push! On Jan 30, Sachin Setiya wrote: > commit a2e6d3a2fe6842bd3c35dffb1cd6f6d8a3117107 > Author: Sachin Setiya <sachin.setiya(a)mariadb.com> > Date: Tue Jan 30 14:04:05 2018 +0530 > > Mdev-15085 Invisible Column Non-constant Default value results... > > Problem:- If we create table field with dynamic default value then that > field always gets NULL value. > > Analyze:- This is because in fill_record we simple continue at Invisible > column because we though that share->default_values(default value is > always copied into table->record[0] before insert) will have a default > value for them(which is true for constant defaults , but not for dynamic > defaults). > > Solution:- We simple set all_fields_have_value to null , and this will > make call to update_default_fields (in the case of dynamic default), And > default expr will be evaluted and value will be set in field. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] Wording suggestion?
by Sergei Golubchik 30 Jan '18

30 Jan '18

Hi, Russell! On Jan 29, Russell J.T. Dyer wrote: > Is this mode related only to UPDATE and this feature? If so, maybe it > could be something like one of these: > > update_static > update_stasis > update_isolation > > These would focus on the fact that it's related just to UPDATE and that the > values held in a state temporarily, or in isolation of each other. Sorry, I should've said earlier. It's not only for update. There is a patch to fix SELECT ... INTO, and I suppose we'll have "simultaneous assignment" behavior everywhere eventually. So, I'm leaning towards either simultaneous_assignment or simultaneous_set as mode names. The first is more exact, but it's rather long (sql_mode names range from 3 to 26 characters, and this one is 23). The second (with 16) fits right in the middle of the range, length-wise. But there's no SET keyword in SELECT ... INTO. On the other hand, there's no ASSIGNMENT keyword anywhere either :) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] Wording suggestion?
by Sergei Golubchik 30 Jan '18

30 Jan '18

Hi, Ian, Russell, As you know, UPDATE in MariaDB is executed left-to-right, say, in the case of UPDATE t1 SET a=b, b=a both columns will have the same value, old value of `b`. We're now introducing a new sql_mode to have standard compatible behavior, where all assignments are executed kind of "simultaneously" (and the statement above would swap the values). What could the name for this new sql_mode be? Regards, Sergei

2 2

Re: [Maria-developers] [Commits] Mdev-15085 Rebased to latest 10.3
by Sergei Golubchik 30 Jan '18

30 Jan '18

Hi, Sachin! On Jan 29, Sachin Setiya wrote: > commit 905806689979ae91c89115f18b107ebacbc489e2 > Author: Sachin Setiya <sachin.setiya(a)maridb.com> > Date: Fri Jan 26 22:58:34 2018 +0530 > > Mdev-15085 Invisible Column Non-constant Default value results... > > Problem:- If we create table field with dynamic default value then that > field always gets NULL value. > > Analyze:- This is because in fill_record we simple continue at Invisible > column because we though that share->default_values(default value is > always copied into table->record[0] before insert) will have a default > value for them(which is true for constant defaults , but not for dynamic > defaults). > > Solution:- In case of invisible field we will assign default value to 'value' > variable, but we do this only for columns which have non dynamic default value. > We do not have to worry for Auto_increment field , storage engine will take > of it. > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 45efa63..17ff7eb 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -8382,7 +8382,12 @@ fill_record(THD *thd, TABLE *table, Field **ptr, > List<Item> &values, > > if (field->invisible) > { > - continue; > + if(field->flags & AUTO_INCREMENT_FLAG || > + !field->default_value) > + continue; > + else > + value= new (thd->mem_root) Item_default_value(thd, > + &thd->lex->select_lex.context); No-no, you cannot create items in fill_record(). It would create a new item for every inserted row. I think the only thing you need to do here (and it doesn't depend on AUTO_INCREMENT_FLAG or field->default_value), you need to set all_fields_have_values to false. Like if (field->invisible) { all_fields_have_values= false; continue; } Indeed, hidden columns did not get a value, so all_fields_have_values should be false. I'd expect the rest will work automatically :) > } > else > value=v++; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] MDEV-14849 CREATE + ALTER with user-invisible columns produce ...
by Sergei Golubchik 29 Jan '18

29 Jan '18

Hi, Sachin! On Jan 29, Sachin Setiya wrote: > commit 5bd97f22094f3b6a658d503f98022ccd9076bb19 > Author: Sachin Setiya <sachin.setiya(a)maridb.com> > Date: Mon Jan 29 12:31:07 2018 +0530 > > MDEV-14849 CREATE + ALTER with user-invisible columns produce ... > > Problem:- > create or replace table t1 (pk int auto_increment primary key invisible, i int); > alter table t1 modify pk int invisible; > This last alter makes a invisible column which is not null and does not > have default value. > > Analysis:- > This is caused because our error check for NOT_NULL_FLAG and > NO_DEFAULT_VALUE_FLAG flag misses this sql_field , but this is not the fault > of error check :).Actually this field come via mysql_prepare_alter_table > and it does not have NO_DEFAULT_VALUE_FLAG flag turned on. (If it was create > table NO_DEFAULT_VALUE_FLAG would have turned on Column_definition::check) > and this would have generated error. > > Solution:- > I have moved the error check to kind last of mysql_prepare_create_table > because upto this point we have applied NO_DEFAULT_VALUE_FLAG to required > column. ok to push! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-14785 SYSTEM_INVISIBLE behaviour not consistent ---Initial Patch
by Sergei Golubchik 29 Jan '18

29 Jan '18

Hi, Sachin! On Jan 29, Sachin Setiya wrote: > commit 5e2f5aa78a5f53b000a2b9c6989c9ed7076e2527 > Author: Sachin Setiya <sachin.setiya(a)maridb.com> Note the typo in your email ^^^^^ > Date: Mon Jan 29 17:09:32 2018 +0530 > > MDEV-14785 SYSTEM_INVISIBLE behaviour not consistent > > TODO Commit message > > diff --git a/sql/item.h b/sql/item.h > index d178d9f..0072dfd 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -2907,11 +2907,17 @@ class Item_field :public Item_ident > bool check_vcol_func_processor(void *arg) > { > context= 0; > - if (field && (field->unireg_check == Field::NEXT_NUMBER)) > + if (field) > { > - // Auto increment fields are unsupported > - return mark_unsupported_function(field_name.str, arg, VCOL_FIELD_REF | VCOL_AUTO_INC); > + if (field->unireg_check == Field::NEXT_NUMBER) > + // Auto increment fields are unsupported > + return mark_unsupported_function(field_name.str, arg, > + VCOL_FIELD_REF | VCOL_AUTO_INC); > + if (field->invisible > INVISIBLE_USER) > + return mark_unsupported_function(field_name.str, arg, > + VCOL_FIELD_REF | VCOL_SYSTEM_INVISIBLE); Why would you do that, instead of hiding system invisible fields in CREATE TABLE? From the user point of view they aren't fields, they don't exist in the table, and the error should be "no such field", not "Function or expression 'on SYSTEM_INVISIBLE column' cannot be used" > } > + > return mark_unsupported_function(field_name.str, arg, VCOL_FIELD_REF); > } > bool set_fields_as_dependent_processor(void *arg) > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 17ff7eb..89ba2ed 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -8128,6 +8128,11 @@ fill_record(THD *thd, TABLE *table_arg, List<Item> &fields, List<Item> &values, > value=v++; > DBUG_ASSERT(value); > Field *rfield= field->field; > + if (rfield->invisible == INVISIBLE_SYSTEM) > + { > + my_error(ER_BAD_FIELD_ERROR, MYF(0), rfield->field_name.str, thd->where); The error is correct. But do you need to do it in fill_record()? I'd thought you can do it when the statement is prepared, once, not in fill_record() which is called for every row, many times per statement. > + goto err; > + } > TABLE* table= rfield->table; > if (table->next_number_field && > rfield->field_index == table->next_number_field->field_index) > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index d2fd2ce..e4971d7 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -16639,6 +16639,7 @@ Field *create_tmp_field_from_field(THD *thd, Field *org_field, > new_field->next_equal_field= NULL; > new_field->option_list= NULL; > new_field->option_struct= NULL; > + new_field->invisible= org_field->invisible; What is it for? It'd mean that temporary tables can have invisible fields. What bugs does this change fix? > } > return new_field; > } > diff --git a/sql/sql_update.cc b/sql/sql_update.cc > index 7b4938f..7bdedc7 100644 > --- a/sql/sql_update.cc > +++ b/sql/sql_update.cc > @@ -1497,6 +1497,9 @@ int mysql_multi_update_prepare(THD *thd) > TABLE_LIST *table_list= lex->query_tables; > TABLE_LIST *tl; > List<Item> *fields= &lex->select_lex.item_list; > + List_iterator<Item> it(*fields); > + Item *item; > + Item_field *field; > table_map tables_for_update; > bool update_view= 0; > /* > @@ -1564,7 +1567,21 @@ int mysql_multi_update_prepare(THD *thd) > { > DBUG_RETURN(TRUE); > } > - > + //Check if we got any INVISIBLE_SYSTEM field > + while ((item= it++)) > + { > + if (Item::FIELD_ITEM == item->type()) > + { > + field= (Item_field *)item; > + if (field->field->invisible == INVISIBLE_SYSTEM) > + { > + my_error(ER_BAD_FIELD_ERROR,MYF(0), > + field->field->field_name.str, > + field->field->table->s->table_name.str); > + DBUG_RETURN(TRUE); > + } > + } > + } Dunno. What if I do UPDATE ... SET normal_field = row_start you'll have an invisible field, but it's not updated, so should be ok, right? May be something like if (field->field->invisible == INVISIBLE_SYSTEM && bitmap_is_set(field->table->write_set, field->field_index)) { ... my_error } And why here? I'd think this belongs into check_fields() function. > thd->table_map_for_update= tables_for_update= get_table_map(fields); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-13790 UNHEX() of a somewhat complicated CONCAT() returns NULL
by Alexander Barkov 29 Jan '18

29 Jan '18

Hello Sergei, Please review a patch for MDEV-13790. It removes the attempt to search the best suitable buffer and just returns the result in "str" passed to val_str(). I'm quite sure this optimization gives nothing. In can be useful only under very rare conditions, but generates so many bugs. Moreover, in 10.3 (or 10.4) we'll work on reentrant items. So all Item_xxx::tmp_buffer will be gone anyway, together with this optimization. I think an optimization proposal. Why not to introduce a new method: class Item { public: .. virtual append_to_string(String *str); .. } So for example Field_varchar could write directly to "str" passed to Item_func_concat::val_str(). This will make sure that the first allocation is done on the target "str", and "str" will always be the best buffer instead of any Item_xxx::tmp_value. Thanks!

2 3

Re: [Maria-developers] 980a5c31880: MDEV-14572: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status upon EXPLAIN UPDATE in PS
by Sergei Golubchik 28 Jan '18

28 Jan '18

Hi, Oleksandr! On Dec 22, Oleksandr Byelkin wrote: > revision-id: 980a5c318803d6aa7355e019869597976dcf19bf (mariadb-10.0.33-37-g980a5c31880) > parent(s): 042f763268c0f209e7c12e0a6a72bb5d204dfe29 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2017-12-22 14:00:10 +0100 > message: > > MDEV-14572: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status upon EXPLAIN UPDATE in PS > > Restore EXPAIN flag in SELECT_LEX before execution multi-update by flag in LEX > (the same but in other way made before INSERT/DELETE/SELECT) Please explain what was the reason for the bug, why the status was set twice, and how your change fixes it. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] ee8755e3c51: MDEV-15012: ASAN: numerous test failures in PS
by Sergei Golubchik 26 Jan '18

26 Jan '18

Hi, Oleksandr! Nice one! Ok to push! On Jan 24, Oleksandr Byelkin wrote: > revision-id: ee8755e3c51a1da8fcf108ad0257a7e62fc94347 (mariadb-5.5.59-32-gee8755e3c51) > parent(s): 76577e1e2602f3c30859a176808c433a263e1b0a > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-01-24 14:42:52 +0100 > message: > > MDEV-15012: ASAN: numerous test failures in PS > > First roll back changes, then free Items which can lead to memory freeing. > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Patch for MDEV-13417 - UPDATE produces wrong values if an updated column is later used as an update source
by jerome brauge 24 Jan '18

24 Jan '18

Hello Sergei, Here is a new version of the patch following your comments. I let you decide if SET_CONSISTENCY must also be part of TRADITIONAL and ANSI. Regards, Jérôme. > -----Message d'origine----- > De : Sergei Golubchik [mailto:serg@mariadb.org] > Envoyé : jeudi 11 janvier 2018 23:17 > À : jerome brauge > Cc : bar(a)mariadb.org > Objet : Re: Patch for MDEV-13417 - UPDATE produces wrong values if an > updated column is later used as an update source > > Hi, Jérôme! > > Happy New Year! > > Sorry for the delay. > > But I've finally got around to reviewing your MDEV-13417 > > The patch was pretty good and with good tests too! > > I had only a few minor comments, see below. > If you'd like, I can apply your patch and do these changes myself. > > See the review below. > > On Dec 28, jerome brauge wrote: > > Hello Sergei, > > Do you have time to review my patch for this mdev ? > > This is very important for us because I don't see any workaround > > (without unique key and cursor). > > > diff --git a/sql/sql_class.h b/sql/sql_class.h index f30b442..a5e2104 > > 100644 > > --- a/sql/sql_class.h > > +++ b/sql/sql_class.h > > @@ -139,6 +139,7 @@ enum enum_binlog_row_image { > > #define MODE_HIGH_NOT_PRECEDENCE (1ULL << 29) > > #define MODE_NO_ENGINE_SUBSTITUTION (1ULL << 30) > > #define MODE_PAD_CHAR_TO_FULL_LENGTH (1ULL << 31) > > +#define MODE_SET_CONSISTENCY (1ULL << 32) > > I don't particularly like "SET_CONSISTENCY" name, but cannot come up with a > good alternative. It doesn't matter, renaming the mode is a trivial change > that doesn't affect anything else. > So, unless you have a strong preference for some specific mode name - just > ignore this naming issue completely. > And I'll ask our documentation guys if they could suggest something. > > > > > /* Bits for different old style modes */ > > #define OLD_MODE_NO_DUP_KEY_WARNINGS_WITH_IGNORE (1 << 0) > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc index 60247c8..debb81a > > 100644 > > --- a/sql/sql_base.cc > > +++ b/sql/sql_base.cc > > @@ -7952,15 +7954,41 @@ fill_record(THD *thd, TABLE *table_arg, > List<Item> &fields, List<Item> &values, > > ER_THD(thd, > ER_WARNING_NON_DEFAULT_VALUE_FOR_VIRTUAL_COLUMN), > > rfield->field_name.str, table->s->table_name.str); > > } > > - if (rfield->stored_in_db() && > > - (value->save_in_field(rfield, 0)) < 0 && !ignore_errors) > > + if (rfield->stored_in_db()) > > { > > - my_message(ER_UNKNOWN_ERROR, ER_THD(thd, > ER_UNKNOWN_ERROR), MYF(0)); > > - goto err; > > + if (value->save_in_field(rfield, 0) < 0 && !ignore_errors) > > + { > > + my_message(ER_UNKNOWN_ERROR, ER_THD(thd, > ER_UNKNOWN_ERROR), MYF(0)); > > + goto err; > > + } > > + /* > > + ** In sql MODE_SET_CONSISTENCY, > > + ** move field pointer on value stored in record[1] > > + ** which contains row before update (see MDEV-13417) > > + */ > > + if (update && thd->variables.sql_mode & MODE_SET_CONSISTENCY) > > + rfield->move_field_offset((my_ptrdiff_t) (table->record[1] - > > + table->record[0])); > > Cool! A simple and nice solution :) > > > } > > rfield->set_explicit_default(value); > > } > > > > + if (update && thd->variables.sql_mode & MODE_SET_CONSISTENCY) { > > + // restore fields pointers on record[0] > > + f.rewind(); > > + while ((fld= f++)) > > + { > > + rfield= fld->field_for_view_update()->field; > > + if (rfield->stored_in_db()) > > + { > > + table= rfield->table; > > + rfield->move_field_offset((my_ptrdiff_t) (table->record[0] - > > + table->record[1])); > > + } > > + } > > + } > > + > > if (!update && table_arg->default_field && > > table_arg->update_default_fields(0, ignore_errors)) > > goto err; > > diff --git a/sql/sql_update.cc b/sql/sql_update.cc index > > 568dd40..5796b96 100644 > > --- a/sql/sql_update.cc > > +++ b/sql/sql_update.cc > > @@ -148,7 +148,25 @@ bool compare_record(const TABLE *table) > > we make temporary copy of Item_field, to avoid influence of changing > > result_field on Item_ref which refer on this field > > */ > > thd->change_item_tree(it.ref(), new (thd->mem_root) Item_field(thd, > field)); > > } > > + } > > + > > + if (thd->variables.sql_mode & MODE_SET_CONSISTENCY) { > > + // Make sure that a column is updated only once > > + List_iterator_fast<Item> it(items); > > + List<Item> ul; > > + ul.empty(); > > + while ((item= it++) && !ul.add_unique(item, &cmp_items)) ; > > Dunno, it's O(N²) and item->eq() is not trivial either. > perhaps it'd be faster, say, mark fields some way, like > > while ((item= it++)) > { > Field *f= item->field_for_view_update()->field; > if (f->has_explicit_value()) > { > my_error(ER_UPDATED_COLUMN_ONLY_ONCE, MYF(0), > f->table_name, f->field_name.str); > return TRUE; > } > f->set_has_explicit_value(); > } > > note that this assumes the change in the error message, %s -> %`s.%`s > > > + > > + if (item) > > + { > > + // same field appear more than once in a single update statement. > > + my_error(ER_UPDATED_COLUMN_ONLY_ONCE, MYF(0), item- > >full_name()); > > + return TRUE; > > + } > > + } > > return FALSE; > > } > > diff --git a/mysql-test/r/set_consistency.result > > b/mysql-test/r/set_consistency.result > > new file mode 100644 > > index 0000000..747b6b3 > > --- /dev/null > > +++ b/mysql-test/r/set_consistency.result > > @@ -0,0 +1,185 @@ > > +SET > > > +sql_mode='STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_A > UTO_CREA > > +TE_USER,NO_ENGINE_SUBSTITUTION,SET_CONSISTENCY'; > > shouldn't SET_CONSISTENCY be part of TRADITIONAL, ORACLE, and ANSI? > > > +# > > +# MDEV-13417 UPDATE produces wrong values if an UPDATEd column is > > +later used as an UPDATE source # MDEV-13418 Compatibility: The order > > +of evaluation of SELECT..INTO assignments # CREATE TABLE t1 (c1 > > +INTEGER, c2 INTEGER, c3 INTEGER) ENGINE=InnoDb; INSERT INTO > > +t1(c1,c2,c3) VALUES (1,1,1); CREATE TABLE t2 (c1 INTEGER, c2 > > +INTEGER, c3 INTEGER) ENGINE=InnoDb; INSERT INTO t2(c1,c2,c3) VALUES > > +(1,1,1); > > + > > +Check that a column is only updated once. > > + > > please, comment out these messages, just like the MDEV numbers above > > > +UPDATE t1 > > +SET c1 = 1, > > +c1 = 2; > > +ERROR HY000: The column test.t1.c1 cannot be changed more than once > > +in a single UPDATE statement > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org

2 1

Re: [Maria-developers] [Commits] 16298d9e82e: MDEV-7533: COLUMN_JSON() doesn't escape control characters in string values
by Sergei Golubchik 22 Jan '18

22 Jan '18

Hi, Oleksandr! On Jan 22, Oleksandr Byelkin wrote: > revision-id: 16298d9e82eb507c5e0ca6d097e6a7016582ccbc (mariadb-10.0.33-47-g16298d9e82e) > parent(s): 5e87f49a9927bf3fd0bd9af789bbdd329e4b4416 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-01-22 16:53:10 +0100 > message: > > MDEV-7533: COLUMN_JSON() doesn't escape control characters in string values > > escape all charecters less or equal 0x1F (control symbols) > (shorter sequence are not used to make code simple, long encoding is > always legal according to the rfc4627) ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0