- developers - lists.mariadb.org

Re: [Maria-developers] [Commits] e60cbbbda31: MDEV-12575: Server crash in AGGR_OP::put_record or in JOIN_CACHE::free or Invalid write in JOIN::make_aggr_tables_info
by Sergey Petrunia 09 Nov '18

09 Nov '18

Hi Varun, On Fri, Nov 09, 2018 at 02:26:04PM +0530, Varun wrote: > revision-id: e60cbbbda31d7a10e87c16313171b2036a4519c4 (mariadb-10.2.18-52-ge60cbbbda31) > parent(s): f8268f3cce4577c28ab62e53293556d05a74fb1a > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-11-01 14:27:24 +0530 > message: > > MDEV-12575: Server crash in AGGR_OP::put_record or in JOIN_CACHE::free or Invalid write in JOIN::make_aggr_tables_info > > During the optimize state of a query, we come know that the result set > would atmost contain one row, then for such a query we don't need > to compute GROUP BY, ORDER BY and DISTINCT. > ... > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 1fa80da85a6..a721aed6ce3 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2245,6 +2245,18 @@ JOIN::optimize_inner() > if (!tables_list || !table_count) > { > choose_tableless_subquery_plan(); > + > + /* The output has atmost one row */ > + if (group_list) > + { > + group_list= 0; > + group_optimized_away= 1; > + rollup.state= ROLLUP::STATE_NONE; > + } > + order=0; > + simple_order=1; > + select_distinct=0; > + Please fix the coding style: - "x= y" (space after the =) - use NULL for pointers - use false for booleans s/atmost/at most/ Ok to push after the above is addressed. > if (select_lex->have_window_funcs()) > { > if (!(join_tab= (JOIN_TAB*) thd->alloc(sizeof(JOIN_TAB)))) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 0187bb03884: MDEV_-17589: Stack-buffer-overflow with indexed varchar (utf8) field
by Sergey Petrunia 08 Nov '18

08 Nov '18

Hi Varun, On Mon, Nov 05, 2018 at 07:36:17PM +0530, Varun wrote: > revision-id: 0187bb03884a34ca8de96e47f4cb1f3d860e6db9 (mariadb-10.2.16-224-g0187bb03884) > parent(s): 8a346f31b913daa011085afec2b2d38450c73e00 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-11-05 19:32:26 +0530 > message: > > MDEV_-17589: Stack-buffer-overflow with indexed varchar (utf8) field > > Create a new constant MAX_DATA_LENGTH_FOR_KEY. > Replace the value of MAX_KEY_LENGTH to also include the LENGTH and NULL BYTES. > ... > diff --git a/mysql-test/r/partition_datatype.result b/mysql-test/r/partition_datatype.result > index 2e518c194f0..66d003c97cb 100644 > --- a/mysql-test/r/partition_datatype.result > +++ b/mysql-test/r/partition_datatype.result > @@ -314,7 +314,7 @@ bbbb > drop table t1; > set sql_mode=''; > create table t1 (a varchar(3070)) partition by key (a); > -ERROR HY000: The total length of the partitioning fields is too large > +drop table t1; I don't think it's a good idea that after this patch we allow the table definitions that we didn't allow before. This raises the question of datadir downgrades, etc. I think we should only remove the buffer overrun. Here's patch how to avoid the above: https://gist.github.com/spetrunia/7e7e3fcc60a81a6cb854e77c12c367d3 > create table t1 (a varchar(65532) not null) partition by key (a); > ERROR HY000: The total length of the partitioning fields is too large > create table t1 (a varchar(65533)) partition by key (a); ... > diff --git a/mysql-test/r/innodb_ext_key.result b/mysql-test/r/innodb_ext_key.result > index c55e8d138f8..c76c8613c57 100644 > --- a/mysql-test/r/innodb_ext_key.result > +++ b/mysql-test/r/innodb_ext_key.result > @@ -1168,8 +1168,8 @@ EXPLAIN > "access_type": "range", > "possible_keys": ["f2"], > "key": "f2", > - "key_length": "3070", > - "used_key_parts": ["f2", "pk1"], > + "key_length": "3074", > + "used_key_parts": ["f2", "pk1", "pk2"], Same reaction as above: This change shows that "Extended keys" feature now allows bigger set of key extensions than before. Can we avoid making this change by changing MAX_KEY_LENGTH to MAX_DATA_LENGTH_FOR_KEY somewhere? > "rows": 1, > "filtered": 100, > "index_condition": "t1.pk1 <= 5 and t1.pk2 <= 5 and t1.f2 = 'abc'", > diff --git a/sql/handler.h b/sql/handler.h > index ed2ef822c88..75f72bdfb53 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -378,6 +378,10 @@ enum enum_alter_inplace_result { > #define HA_KEY_NULL_LENGTH 1 > #define HA_KEY_BLOB_LENGTH 2 > > +#define MAX_KEY_LENGTH (MAX_DATA_LENGTH_FOR_KEY \ > + +(MAX_REF_PARTS \ > + *(HA_KEY_NULL_LENGTH + HA_KEY_BLOB_LENGTH))) > + I think this requires a comment. Please add something like /* Maximum length of any index lookup key, in bytes */ > #define HA_LEX_CREATE_TMP_TABLE 1U > #define HA_CREATE_TMP_ALTER 8U > > @@ -3421,14 +3425,14 @@ class handler :public Sql_alloc > uint max_key_parts() const > { return MY_MIN(MAX_REF_PARTS, max_supported_key_parts()); } > uint max_key_length() const > - { return MY_MIN(MAX_KEY_LENGTH, max_supported_key_length()); } > + { return MY_MIN(MAX_DATA_LENGTH_FOR_KEY, max_supported_key_length()); } > uint max_key_part_length() const > - { return MY_MIN(MAX_KEY_LENGTH, max_supported_key_part_length()); } > + { return MY_MIN(MAX_DATA_LENGTH_FOR_KEY, max_supported_key_part_length()); } > > virtual uint max_supported_record_length() const { return HA_MAX_REC_LENGTH; } > virtual uint max_supported_keys() const { return 0; } > virtual uint max_supported_key_parts() const { return MAX_REF_PARTS; } > - virtual uint max_supported_key_length() const { return MAX_KEY_LENGTH; } > + virtual uint max_supported_key_length() const { return MAX_DATA_LENGTH_FOR_KEY; } > virtual uint max_supported_key_part_length() const { return 255; } > virtual uint min_record_length(uint options) const { return 1; } > > diff --git a/sql/sql_const.h b/sql/sql_const.h > index 7395ae3c08a..0be63a2a5ad 100644 > --- a/sql/sql_const.h > +++ b/sql/sql_const.h > @@ -33,7 +33,7 @@ > #define MAX_SYS_VAR_LENGTH 32 > #define MAX_KEY MAX_INDEXES /* Max used keys */ > #define MAX_REF_PARTS 32 /* Max parts used as ref */ Please add a comment, something like /* Maximum length of the data part of an index lookup key. The "data part" is defined as the value itself, not including the NULL-indicator bytes or varchar length bytes ("the Extras"). We need this value because there was a bug where length of the Extras were not counted. You probably need MAX_KEY_LENGTH, not this constant. */ > -#define MAX_KEY_LENGTH 3072 /* max possible key */ > +#define MAX_DATA_LENGTH_FOR_KEY 3072 > #if SIZEOF_OFF_T > 4 > #define MAX_REFLENGTH 8 /* Max length for record ref */ > #else BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] b0d21b7: MDEV-14567 Include full name of object in message about incorrect value for column.
by Sergei Golubchik 07 Nov '18

07 Nov '18

Hi, Alexey! On Nov 07, Alexey Botchkov wrote: > > But that's fine to show, right? No NULL, no garbage. > > Looks like this: > Incorrect datetime value: '0000-00-00 00:00:00' for column > 'test.#sql-691c_15d.c1' at row 1 > No NULL, but that #sql-something name doesn't tell anything and is > always different so we have to add lots of --replace-result in tests. Right, but we already have a lot of replace-result in tests. #sql-691c_15d showing in error messages in a different issue (but see below). > My patch shows it as.this: > Incorrect datetime value: '0000-00-00 00:00:00' for column ' {temporary}.c1' at row 1 This isn't much better in my opinion. It only helps tests to avoid replace-result, and I always believed in solving test problems in tests, not in the server. Meaning, exactly, replace-result. For the end user #sql-691c_15d and {temporary} has about the save amount of information. > > By the way, can this error ever happen for internal temporary tables? > Yes. > In our tests it happens pretty often. > Code like this: > CREATE TABLE t3(c1 DATETIME NOT NULL); > INSERT INTO t3 VALUES (0); > > --echo > SET sql_mode = TRADITIONAL; > > --echo > --error ER_TRUNCATED_WRONG_VALUE > ALTER TABLE t3 ADD INDEX(c1); Oh, ALTER TABLE. This is different. In one of the system versioning related fixes, midenok added a TABLE_SHARE::error_table_name() method to be used in error messages in ALTER. See 912769b7dc. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] cd41d376d41: MDEV-16697: Fix difference between 32bit/windows and 64bit systems in allowed select nest level
by Sergei Golubchik 06 Nov '18

06 Nov '18

Hi, Oleksandr! Why did you do that? You wrote "Fix difference between 32bit/windows and 64bit systems", but to fix just that it would've been enough to change nesting_map to ulonglong. Using Bitmap<> you're basically preparing for any arbitrary >64 nesting depth. Is this your goal? What feature do you need it for? On Nov 06, Oleksandr Byelkin wrote: > revision-id: cd41d376d413d066f3d497a67d039515fd385a1c (versioning-1.0.5-36-gcd41d376d41) > parent(s): 1748a31ae8d69e4939336f644f884e9de3039e7f > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2018-07-05 17:49:44 +0200 > message: > > MDEV-16697: Fix difference between 32bit/windows and 64bit systems in allowed select nest level > > diff --git a/include/my_global.h b/include/my_global.h > index 11d17ad146a..26078114464 100644 > --- a/include/my_global.h > +++ b/include/my_global.h > @@ -984,7 +984,6 @@ typedef unsigned long my_off_t; > TODO Convert these to use Bitmap class. > */ > typedef ulonglong table_map; /* Used for table bits in join */ > -typedef ulong nesting_map; /* Used for flags of nesting constructs */ > > /* often used type names - opaque declarations */ > typedef const struct charset_info_st CHARSET_INFO; > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -33,6 +33,10 @@ > #include "sql_tvc.h" > #include "item.h" > > +/* Used for flags of nesting constructs */ > +#define SELECT_NESTING_MAP_SIZE 64 > +typedef Bitmap<SELECT_NESTING_MAP_SIZE> nesting_map; > + > /* YACC and LEX Definitions */ Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] fa8879bc800: MDEV-16370 row-based binlog events (updates by primary key) can not be applied multiple times to system versioned tables
by Sergei Golubchik 06 Nov '18

06 Nov '18

Hi, Aleksey! Looking at the patch and the test, I get the impression you're fixing the wrong issue. The problem in MDEV-16370 happened when replicating from non-system-versioned table on the master to a system-versioned table on the slave. You don't have a test case for that. Your fix doesn't change the behavior for this use case. So, I don't understand what you've actually fixed. On Nov 06, Aleksey Midenkov wrote: > revision-id: fa8879bc800d2babf48d082ce8c50ffe520e12cc (versioning-1.0.5-140-gfa8879bc800) > parent(s): 28ae79650d3057ffcd849d2ef28dcd80282c85d2 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2018-09-20 10:04:57 +0300 > message: > > MDEV-16370 row-based binlog events (updates by primary key) can not be applied multiple times to system versioned tables > > Rows_log_event::write_row() always overwrite historical row. > > diff --git a/mysql-test/suite/versioning/t/rpl.test b/mysql-test/suite/versioning/t/rpl.test > index a9e3af45af8..1985b17bf6d 100644 > --- a/mysql-test/suite/versioning/t/rpl.test > +++ b/mysql-test/suite/versioning/t/rpl.test > @@ -133,4 +133,24 @@ sync_slave_with_master; > connection master; > drop table t1; > > +--echo # MDEV-16370 row-based binlog events (updates by primary key) > +--echo # can not be applied multiple times to system versioned tables > +create or replace table t1 (pk int primary key, x int) with system versioning; > +insert into t1 values (1, 1); > +--sync_slave_with_master > +flush logs; > +--connection master > +update t1 set x= 2; > +--sync_slave_with_master > +select * from t1; > +--let $datadir= `select @@datadir` > +--let $f2= query_get_value(SHOW SLAVE STATUS, Relay_Log_File, 1) > +--echo # Replaying binlog... > +--exec $MYSQL_BINLOG $datadir/$f2| $MYSQL -S $SLAVE_MYSOCK test > +select * from t1; > +--connection master > + > +drop database test; > +create database test; > + > --source include/rpl_end.inc > diff --git a/sql/log_event.cc b/sql/log_event.cc > index c550adea26b..c4c04743a30 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -13139,13 +13139,26 @@ Rows_log_event::write_row(rpl_group_info *rgi, > } > > // Handle INSERT. > - if (table->versioned(VERS_TIMESTAMP)) > + if (table->versioned()) > { > - ulong sec_part; > - bitmap_set_bit(table->read_set, table->vers_start_field()->field_index); > - // Check whether a row came from unversioned table and fix vers fields. > - if (table->vers_start_field()->get_timestamp(&sec_part) == 0 && sec_part == 0) > - table->vers_update_fields(); > + if (table->versioned(VERS_TIMESTAMP)) > + { > + // Set vers fields when replicating from not system-versioned table. > + ulong sec_part; > + bitmap_set_bit(table->read_set, table->vers_start_field()->field_index); > + // Check whether a row came from unversioned table and fix vers fields. > + if (table->vers_start_field()->get_timestamp(&sec_part) == 0 && sec_part == 0) > + table->vers_update_fields(); > + else > + goto overwrite_historical_row; > + } > + else > + { > +overwrite_historical_row: > + bitmap_set_bit(table->read_set, table->vers_end_field()->field_index); > + if (!overwrite && !table->vers_end_field()->is_max()) > + overwrite= true; > + } > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 503e494ec6e: MDEV-16241 Assertion `inited==RND' failed in handler::ha_rnd_end()
by Sergei Golubchik 06 Nov '18

06 Nov '18

Hi, Aleksey! On Nov 06, Aleksey Midenkov wrote: > revision-id: 503e494ec6e103b23dc55f8fc8ae68ecec8b8316 (versioning-1.0.6-1-g503e494ec6e) > parent(s): 6120ae4acafb23badecc623e33797949e3f6fbbb > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2018-10-22 11:50:10 +0300 > message: > > MDEV-16241 Assertion `inited==RND' failed in handler::ha_rnd_end() > > Discrepancy in open indexes due to overwritten `read_partitions` upon > `ha_open()` in `ha_partition::clone()`. > > [Fixes tempesta-tech/mariadb#551] > > diff --git a/mysql-test/main/partition_innodb.result b/mysql-test/main/partition_innodb.result > index d27abc07984..c63cbc4601d 100644 > --- a/mysql-test/main/partition_innodb.result > +++ b/mysql-test/main/partition_innodb.result > @@ -925,3 +925,14 @@ test_jfg test_jfg11 > test_jfg test_jfg12#P#p1000 > test_jfg test_jfg12#P#pmax > DROP DATABASE test_jfg; > +# > +# MDEV-16241 Assertion `inited==RND' failed in handler::ha_rnd_end() > +# > +CREATE TABLE t1 (pk INT PRIMARY KEY, x INT, y INT, z INT, KEY (x), KEY (y, z)) > +WITH SYSTEM VERSIONING > +PARTITION BY SYSTEM_TIME (PARTITION p1 HISTORY, PARTITION pn CURRENT); > +INSERT INTO t1 VALUES (1, 7, 8, 9), (2, NULL, NULL, NULL), (3, NULL, NULL, NULL); > +SELECT COUNT(*) FROM t1 WHERE x IS NULL AND y IS NULL AND z IS NULL; > +COUNT(*) > +2 > +DROP TABLE t1; please, add a non-system-versioning test too. It's in one of the comments in the bug report. > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 40380ffeec5..901f812a0e6 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -3725,6 +3725,9 @@ void ha_partition::rebind_psi() > handler *ha_partition::clone(const char *name, MEM_ROOT *mem_root) > { > ha_partition *new_handler; > + MY_BITMAP read_partitions; > + MY_BITMAP lock_partitions; > + int error; > > DBUG_ENTER("ha_partition::clone"); > new_handler= new (mem_root) ha_partition(ht, table_share, m_part_info, > @@ -3748,9 +3751,21 @@ handler *ha_partition::clone(const char *name, MEM_ROOT *mem_root) > ALIGN_SIZE(m_ref_length)*2))) > goto err; > > - if (new_handler->ha_open(table, name, > + DBUG_ASSERT(m_tot_parts == m_part_info->read_partitions.n_bits); > + DBUG_ASSERT(m_tot_parts == m_part_info->lock_partitions.n_bits); > + my_bitmap_init(&read_partitions, NULL, m_tot_parts, FALSE); > + my_bitmap_init(&lock_partitions, NULL, m_tot_parts, FALSE); > + bitmap_copy(&read_partitions, &m_part_info->read_partitions); > + bitmap_copy(&lock_partitions, &m_part_info->lock_partitions); > + error= new_handler->ha_open(table, name, > table->db_stat, > - HA_OPEN_IGNORE_IF_LOCKED | HA_OPEN_NO_PSI_CALL)) > + HA_OPEN_IGNORE_IF_LOCKED | HA_OPEN_NO_PSI_CALL); > + bitmap_copy(&m_part_info->read_partitions, &read_partitions); > + bitmap_copy(&m_part_info->lock_partitions, &lock_partitions); > + my_bitmap_free(&read_partitions); > + my_bitmap_free(&lock_partitions); > + > + if (error) I don't think this is needed. ha_partition::open() is aware of clones, it has `if (m_is_clone_of)` to do some stuff only for clones and some only for fresh opens. Perhaps you just need to move the bitmap-resetting code to be done only for fresh opens? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] b0d21b7: MDEV-14567 Include full name of object in message about incorrect value for column.
by Sergei Golubchik 06 Nov '18

06 Nov '18

Hi, Alexey! On Nov 06, Alexey Botchkov wrote: > > That's why I asked what temporary tables have in table->s->db and > > in table->s->table_name. What do they have there? > > 'Persistent' temporary tables (those created by CREATE TEMPORARY > TABLE) have their database/name there. > Auxiliary temporary tables created to execute queries like SELECT have > 'normal' db value and soumething like #sql_1234 as a table name. But that's fine to show, right? No NULL, no garbage. No need to treat them specially. By the way, can this error ever happen for internal temporary tables? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] b0d21b7: MDEV-14567 Include full name of object in message about incorrect value for column.
by Sergei Golubchik 06 Nov '18

06 Nov '18

Hi, Alexey! On Nov 06, Alexey Botchkov wrote: > > >> + long_field_name(buf, sizeof(buf), table->s, field_name), > >Better change the error message to be > > Incorrect decimal value: '%s' for column %`s.%`s.%`s at row %d > > and use like > > - field_name, > > + table->s->db.str, table->s->table_name.str, field_name, > > Don't think it is better. > There are temporary tables and just Item's that launch that error. So > the 'db' and 'table' do not always defined properly. > The long_field_name() i added handles the temporary table case. Yes, I've noticed you had a specia case for temporary tables. That's why I asked what temporary tables have in table->s->db and in table->s->table_name. What do they have there? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] dc9f113111e: MDEV-17376 Server fails to set ADD_PK_INDEX, DROP_PK_INDEX
by Sergei Golubchik 05 Nov '18

05 Nov '18

Hi, Thirunarayanan! On Nov 05, Thirunarayanan Balathandayuthapani wrote: > revision-id: dc9f113111ee4daf3964db938ef3ddd82c777328 (mariadb-10.0.37-6-gdc9f113111e) > parent(s): 38b3e52c3c40b35339b1d18c09053be416a93420 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2018-11-01 21:24:03 +0530 > message: > > MDEV-17376 Server fails to set ADD_PK_INDEX, DROP_PK_INDEX > if unique index nominated as PK Subject line in git commits is always one line. Try to prefer the format as of MDEV-17376 Server fails to set ADD_PK_INDEX, DROP_PK_INDEX if unique index nominated as PK that is, MDEV-number, one space, MDEV summary verbatim, all on one line. This way it'll look good in `git log --oneline` and one can search for a bug description (and new lines won't cause the search to fail). > - Server sets the flag ADD_PK_INDEX, DROP_PK_INDEX if the unique key > nominated as Primary key. this isn't a very helpful comment, just a rephrased commit title :) Up to you, of course, it doesn't hurt. But generally, if you feel the fix it kind of obvious and you cannot come up with anything to put into commit comment body, just don't put there anything at all. > > diff --git a/mysql-test/suite/innodb/t/alter_candidate_key.test b/mysql-test/suite/innodb/t/alter_candidate_key.test > --- /dev/null > +++ b/mysql-test/suite/innodb/t/alter_candidate_key.test > @@ -0,0 +1,47 @@ > +--source include/have_innodb.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > + > +CREATE TABLE t1 (f1 INT NOT NULL, f2 INT NOT NULL, > + UNIQUE KEY uidx2(f1,f2), > + UNIQUE KEY uidx1(f2) > +) ENGINE=InnoDB; > + > +SET DEBUG_SYNC = 'innodb_inplace_alter_table_enter SIGNAL conc_dml WAIT_FOR go_ahead'; > +INSERT INTO t1 VALUES(1,1); > +--send > +alter table t1 change column f1 f11 int, algorithm=inplace; > +connect (con1,localhost,root,,); > +SET DEBUG_SYNC = 'now WAIT_FOR conc_dml'; > +DELETE FROM t1; > +SET DEBUG_SYNC = 'now SIGNAL go_ahead'; > +connection default; > +reap; > +CHECK TABLE t1; > +DROP TABLE t1; Could you explain the test case, please? What's going on here? I don't understand what you're fixing and what I'm reviewing, sorry :( Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 5403baabc7e: MDEV-16849 Extending indexed VARCHAR column should be instantaneous
by Sergei Golubchik 05 Nov '18

05 Nov '18

Hi, Thirunarayanan! On Nov 05, Thirunarayanan Balathandayuthapani wrote: > revision-id: 5403baabc7ef87e301964434e3670ded06af0802 (mariadb-10.2.18-40-g5403baabc7e) > parent(s): ab1ce2204e959bea596817494e932754ab5cbe88 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2018-10-26 16:49:08 +0530 > message: > > MDEV-16849 Extending indexed VARCHAR column should be instantaneous > > Analysis: > ======== > Increasing the length of the indexed varchar column is not an instant operation for > innodb. > > Fix: > === > - Introduce the new handler flag 'Alter_inplace_info::ALTER_COLUMN_INDEX_LENGTH' to > indicate the index length differs due to change of column length changes. > > - InnoDB makes the ALTER_COLUMN_INDEX_LENGTH flag as instant operation. > > This is a port of Mysql fix. > > diff --git a/sql/handler.cc b/sql/handler.cc > index da41daf2440..b1fa53fb145 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -4253,7 +4253,8 @@ handler::check_if_supported_inplace_alter(TABLE *altered_table, > Alter_inplace_info::CHANGE_CREATE_OPTION | > Alter_inplace_info::ALTER_PARTITIONED | > Alter_inplace_info::ALTER_VIRTUAL_GCOL_EXPR | > - Alter_inplace_info::ALTER_RENAME; > + Alter_inplace_info::ALTER_RENAME | > + Alter_inplace_info::ALTER_COLUMN_INDEX_LENGTH; You sure? It means that any engine by default can do ALTER_COLUMN_INDEX_LENGTH inplace, unless some other check in the engine disallows it. I'd rather not enable it for all engine, so better remove it. InnoDB has his own check inside ha_innobase::check_if_supported_inplace_alter() anyway. > /* Is there at least one operation that requires copy algorithm? */ > if (ha_alter_info->handler_flags & ~inplace_offline_operations) > diff --git a/sql/handler.h b/sql/handler.h > index ed2ef822c88..f2250699cfb 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1994,6 +1994,11 @@ class Alter_inplace_info > > static const HA_ALTER_FLAGS ALTER_DROP_CHECK_CONSTRAINT= 1ULL << 40; > > + /** > + Change in index length such that it doesn't require index rebuild. > + */ > + static const HA_ALTER_FLAGS ALTER_COLUMN_INDEX_LENGTH= 1ULL << 41; > + > /** > Create options (like MAX_ROWS) for the new version of table. > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 02780e7d1e0..94d35c99d0a 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6687,18 +6687,27 @@ static bool fill_alter_inplace_info(THD *thd, > key_part < end; > key_part++, new_part++) > { > + new_field= get_field_by_index(alter_info, new_part->fieldnr); > /* > + If there is a change in index length due to column expansion > + like varchar(X) changed to varchar(X + N) and has a compatible > + packed data representation, we mark it for fast/INPLACE change > + in index definition. InnoDB supports INPLACE for this cases > + > Key definition has changed if we are using a different field or > - if the used key part length is different. It makes sense to > - check lengths first as in case when fields differ it is likely > - that lengths differ too and checking fields is more expensive > - in general case. > + if the user key part length is different. > */ > - if (key_part->length != new_part->length) > + if (key_part->length != new_part->length && > + (ha_alter_info->alter_info->flags & Alter_info::ALTER_CHANGE_COLUMN) && why is that? Can it happen that ALTER_CHANGE_COLUMN flag is not set? > + (key_part->field->is_equal((Create_field*) new_field) > + == IS_EQUAL_PACK_LENGTH)) > + { > + ha_alter_info->handler_flags |= > + Alter_inplace_info::ALTER_COLUMN_INDEX_LENGTH; > + } > + else if (key_part->length != new_part->length) > goto index_changed; > > - new_field= get_field_by_index(alter_info, new_part->fieldnr); > - > /* > For prefix keys KEY_PART_INFO::field points to cloned Field > object with adjusted length. So below we have to check field Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] b0d21b7: MDEV-14567 Include full name of object in message about incorrect value for column.
by Sergei Golubchik 05 Nov '18

05 Nov '18

Hi, Alexey! On Nov 04, Alexey Botchkov wrote: > revision-id: b0d21b7832cea92522d0f8d280ceb8b13b335cbf (mariadb-10.2.18-56-gb0d21b7) > parent(s): 1c6b982e02eeaa75bb6c2f2a3c2b64491dd6d3c8 > committer: Alexey Botchkov > timestamp: 2018-11-04 09:14:40 +0400 > message: > > MDEV-14567 Include full name of object in message about incorrect value for column. Wrong MDEV number again. It should be MDEV-14576 > > Put the full field name in the ER_TRUNCATED_WRONG_VALUE_FOR_FIELD > error message. > > +static char *long_field_name(char *buf, size_t buf_len, > + const TABLE_SHARE *s, const char *field_name) > +{ > + bool is_temp= !s|| > + (tmp_file_prefix_length < s->table_name.length && > + memcmp(s->table_name.str, > + tmp_file_prefix, tmp_file_prefix_length) == 0); > + > + if (is_temp) > + strxnmov(buf, buf_len, "{temporary}.", field_name, NullS); > + else > + strxnmov(buf, buf_len, > + s->db.str, ".", s->table_name.str, ".", field_name, NullS); > + > + return buf; > +} > + > + > int Field_geom::store(const char *from, uint length, CHARSET_INFO *cs) > { > if (!length) > @@ -8620,10 +8638,11 @@ int Field_geom::store(const char *from, uint length, CHARSET_INFO *cs) > geom_type != Field::GEOM_GEOMETRYCOLLECTION && > (uint32) geom_type != wkb_type) > { > + char buf[193]; > my_error(ER_TRUNCATED_WRONG_VALUE_FOR_FIELD, MYF(0), > Geometry::ci_collection[geom_type]->m_name.str, > Geometry::ci_collection[wkb_type]->m_name.str, > - field_name, > + long_field_name(buf, sizeof(buf), table->s, field_name), > (ulong) table->in_use->get_stmt_da()-> > current_row_for_warning()); > goto err_exit; Better change the error message to be Incorrect decimal value: '%s' for column %`s.%`s.%`s at row %d and use like my_error(ER_TRUNCATED_WRONG_VALUE_FOR_FIELD, MYF(0), Geometry::ci_collection[geom_type]->m_name.str, Geometry::ci_collection[wkb_type]->m_name.str, - field_name, + table->s->db.str, table->s->table_name.str, field_name, (ulong) table->in_use->get_stmt_da()-> current_row_for_warning()); What would that do for temporary tables? Anyway, the correct quoting is still `db`.`table`.`column`, not 'db.table.column'. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-15740 (InnoDB lost Durability due to incorrect fix of MDEV-11937)
by Marko Mäkelä 02 Nov '18

02 Nov '18

Hi Kristian, Could you please take a look at this InnoDB regression in MariaDB 10.2 and later: https://jira.mariadb.org/browse/MDEV-15740 Because of this bug, we can no longer write tests that would ensure that certain changes are present in the redo log, before killing and restarting the server. This already hurt me a bit when testing MDEV-15562 (instant DROP COLUMN) in 10.4. I had to add extra sleeps to test cases in order to actually test redo log roll-forward followed by rollback. -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation

2 3

Re: [Maria-developers] [Commits] 737de39: Issue #885: Can't locate lib/mtr_process.pl in @INC on Ubuntu 18.04.1
by Kristian Nielsen 02 Nov '18

02 Nov '18

Hi Sergey, psergey(a)askmonty.org (Sergei Petrunia) writes: > committer: Sergei Petrunia > branch nick: mysql-5.6-rocksdb-spetrunia > Issue #885: Can't locate lib/mtr_process.pl in @INC on Ubuntu 18.04.1 > > Add "." to @INC array like upstream does. Just want to make sure you are aware of this: https://github.com/MariaDB/server/commit/d185f1d68bb1f37bea10d8ac6188e5a04f… I see the branch nick is mysql-5.6-<something>, so maybe this is like this for legacy reasons, which is fine. But this is basically the wrong fix. Clearly sometimes in ancient history, someone did not understand Perl `require` semantics, and ended up with a horrible mess. For example, if someone ever finds they need `require mtr_misc.pl` in a third file, they will find it will not work. Anyway, just a pointer in case you were not aware of a small land-mine hidden under your feet ;-) - Kristian.

2 1

Re: [Maria-developers] f95616dc477: MDEV-17401: LOAD DATA from very big file into MyISAM table results in EOF error and corrupt index
by Sergei Golubchik 31 Oct '18

31 Oct '18

Hi, Oleksandr! ok to push! On Oct 30, Oleksandr Byelkin wrote: > revision-id: f95616dc4774328381044b53263d6a23ae19bfb4 (mariadb-5.5.61-39-gf95616dc477) > parent(s): 893ebb739e24637985892da1555f2e6f1a8ac5b6 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-10-30 19:36:17 +0100 > message: > > MDEV-17401: LOAD DATA from very big file into MyISAM table results in EOF error and corrupt index > > my_read fixed as in higher versions. > my_pread made as my_read aware of partial read of huge chunks of files > MY_FULL_IO enabled for file operations > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] c29706f: MDEV-17286 Decimal field multiplication bug.
by Sergei Golubchik 30 Oct '18

30 Oct '18

Hi, Alexey! On Oct 30, Alexey Botchkov wrote: > > What if both arguments will be zeros, but have intg=DIG_PER_DEC1-1. > > Won't the result have more that DIG_PER_DEC1 zeros? > > Leading zero 'digits' are truncated, so we never get more than DIG_PER_DEC > there. Actually we don't need to do that check at all (intg <= > DIG_PER_DEC1), > i removed it already. Put it in an assert please. I find asserts quite useful as a documentation tool :) Then ok to push! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] c29706f: MDEV-17286 Decimal field multiplication bug.
by Sergei Golubchik 30 Oct '18

30 Oct '18

Hi, Alexey! On Oct 24, Alexey Botchkov wrote: > revision-id: c29706f7560148488791acca3c92dde1b188334e (mariadb-5.5.61-32-gc29706f) > parent(s): e31e697f17f79ffa6913499e7e2d29866f24b475 > committer: Alexey Botchkov > timestamp: 2018-10-24 01:26:49 +0400 > message: > > MDEV-17286 Decimal field multiplication bug. > > We should clear trailing zeroes in frac part. Otherwise > that tail is growing quickly and forces unnecessary truncating of > arguments. This looks good, thanks. Please add a test for create select, as discussed. And fix the bug number in the commit comment. See one question below. > @@ -2109,6 +2104,14 @@ int decimal_mul(const decimal_t *from1, const decimal_t *from2, decimal_t *to) > for (; d_to_move--; cur_d++, buf1++) > *cur_d= *buf1; > } > + > + /* Now we have to check for -0.000 case */ > + if (to->sign && to->intg <= DIG_PER_DEC1 && to->frac == 0 && > + to->buf[0] == 0) What if both arguments will be zeros, but have intg=DIG_PER_DEC1-1. Won't the result have more that DIG_PER_DEC1 zeros? > + { > + /* We got decimal zero */ > + decimal_make_zero(to); > + } > return error; > } > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] c29706f: MDEV-17286 Decimal field multiplication bug.
by Sergei Golubchik 29 Oct '18

29 Oct '18

Hi, Alexey! On Oct 24, Alexey Botchkov wrote: > revision-id: c29706f7560148488791acca3c92dde1b188334e (mariadb-5.5.61-32-gc29706f) > parent(s): e31e697f17f79ffa6913499e7e2d29866f24b475 > committer: Alexey Botchkov > timestamp: 2018-10-24 01:26:49 +0400 > message: > > MDEV-17286 Decimal field multiplication bug. it's MDEV-17256 :) > > We should clear trailing zeroes in frac part. Otherwise > that tail is growing quickly and forces unnecessary truncating of > arguments. I don't understand this, sorry. Could you elaborate, please? I understand the "tail is growing quickly" part. How does it "force unnecessary truncating of arguments"? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-12321 authentication plugin: SET PASSWORD support
by Sergei Golubchik 27 Oct '18

27 Oct '18

Hi, Please review commits for MDEV-12321 "authentication plugin: SET PASSWORD support" They're now in the bb-10.4-serg branch, the last seven commits: ========================= commit 62e8340f513 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Wed Oct 17 12:48:13 2018 +0200 MDEV-12321 authentication plugin: SET PASSWORD support Support SET PASSWORD for authentication plugins. Authentication plugin API is extended with two optional methods: * hash_password() is used to compute a password hash (or digest) from the plain-text password. This digest will be stored in mysql.user table * preprocess_hash() is used to convert this digest into some memory representation that can be later used to authenticate a user. Build-in plugins convert the hash from hexadecimal or base64 to binary, to avoid doing it on every authentication attempt. Note a change in behavior: when loading privileges (on startup or on FLUSH PRIVILEGES) an account with an unknown plugin was loaded with a warning (e.g. "Plugin 'foo' is not loaded"). But such an account could not be used for authentication until the plugin is installed. Now an account like that will not be loaded at all (with a warning, still). Indeed, without plugin's preprocess_hash() method the server cannot know how to load an account. Thus, if a new authentication plugin is installed run-time, one might need FLUSH PRIVILEGES to activate all existing accounts that were using this new plugin. commit 8266ca26f27 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Mon Oct 15 01:39:03 2018 +0200 misc cleanups * remove dead code (from .yy) * remove redundant commands from the test * extract common code into a reusable function (get_auth_plugin, push_new_user) * rename update_user_table->update_user_table_password * simplify acl_update_role * don't strdup a string that's already in a memroot (in ACL_ROLE::ACL_ROLE(ACL_USER*)) * create parent_grantee and role_grants dynamic arrays with size 0. to avoid any memory allocations when roles aren't used. commit fd0bcb5e791 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Sun Oct 14 13:52:52 2018 +0200 Use mysql.user.authentication_string for password Don't distinguish between a "password hash" and "authentication string" anymore. Now both are stored in mysql.user.authentication_string, both are handled identically internally. A "password hash" is just how some particular plugins interpret authentication string. Set mysql.user.plugin even if there is no password. The server will use mysql_native_password plugin in these cases, let's make it expicit. Remove LEX_USER::pwhash. commit 15f30100e0d Author: Sergei Golubchik <serg(a)mariadb.org> Date: Sat Oct 13 18:32:05 2018 +0200 cleanup: sql_acl.cc remove fix_plugin_ptr() it was doing two my_strcasecmp() unconditionally, to optimize away one conditional my_strcasecmp() later. commit df9d95d2a85 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Sat Oct 13 11:30:39 2018 +0200 cleanup: sql_acl.cc remove username=NULL Some parts of sql_acl.cc historically assumed that empty username is represented by username=NULL, other parts used username="" for that. And most of the code wasn't sure and checked both (like in `if (!user || !user[0])`). Change it to use an empty string everywhere. commit 8b3cbc1e469 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Fri Oct 12 18:48:15 2018 +0200 cleanup: sql_acl.cc password->LEX_CSTRING commit 2fc7a20da6a Author: Sergei Golubchik <serg(a)mariadb.org> Date: Fri Oct 12 19:24:28 2018 +0200 cleanup: safe_lexcstrdup_root() ========================= I've split somewhat independent changes into separate commits, but, of course, if you'd like to review one big patch you can `git diff` them all together. Now, the syntax problem. Old MySQL syntax (for the last ~20 years) was (1) GRANT ... TO user IDENTIFIED BY 'plain-text password' (2) GRANT ... TO user IDENTIFIED BY PASSWORD 'password hash' (3) SET PASSWORD = 'password hash' (4) SET PASSWORD = PASSWORD('plain-text password') (5) SET PASSWORD = OLD_PASSWORD('plain-text password') here, syntax (1) and (4) were forcing mysql_native_password authentication, (5) was forcing mysql_old_password, and (2) and (3) were auto-detecting, based on the hash length. Later MariaDB and MySQL added support for pluggable authentication with the syntax (6) GRANT ... TO user IDENTIFIED VIA plugin AS 'password hash' MySQL 5.7 added support for specifying plain-text password for plugins using the syntax (7) GRANT ... TO user IDENTIFIED WITH plugin BY 'plain-text password' I don't quite like it because there's no logical reason why "BY" means a plain-text password, while "AS" means a hash. Also we support "USING" instead of "AS", which also means a hash. One can easily get lost in all these USING/AS/BY and what special semantics each of them has. The syntax I've implemented is based on SET PASSWORD: (8) GRANT ... TO user IDENTIFIED VIA plugin AS PASSWORD('plain-text password') This is quite intuitive and pretends that there's a sql function PASSWORD() which returns a hash and it's used as an expression where a hash is anyway expected. Same works in SET PASSWORD too, obviously. A PASSWORD() function actually exists and returns the password hash. The problem here is that PASSWORD() function becomes quite magical. It returns a different password, depending on what plugin the user is using. One can still do SELECT PASSWORD("foo"), OLD_PASSWORD("foo"), but they'll return values for mysql_native_password and mysql_old_password as before. In the context of SET PASSWORD or GRANT (or CREATE/ALTER USER) it becomes context dependent, it's a bit difficult to swallow. Another approach would be not to pretend it's a function. Say (9) GRANT ... TO user IDENTIFIED VIA plugin AS PASSWORD 'plain-text password' SET PASSWORD = PASSWORD 'plain-text password' but, unfortunately, it is exactly backwards from the historical behaviour of (1) and (2). All in all I'm leaning towards (8), but I'm not quite happy with it :( One way to solve it could be to extend PASSWORD() function to allow a second argument, plugin name, like in SELECT PASSWORD("foo", "ed25519") Yet another way could be to remove SQL-level functions PASSWORD() and OLD_PASSWORD(). That would be my favorite, they always were nothing but trouble. But I wouldn't risk doing it now :) Regards, Sergei

2 3

Re: [Maria-developers] cc5c19ae523: MDEV-17401: LOAD DATA from very big file into MyISAM table results in EOF error and corrupt index
by Sergei Golubchik 24 Oct '18

24 Oct '18

Hi, Oleksandr! On Oct 12, Oleksandr Byelkin wrote: > revision-id: cc5c19ae5233ba90de086de76043774ae6c78cd7 (mariadb-5.5.61-30-gcc5c19ae523) > parent(s): acf8fc1ff8a7b2d49e25279670b04b8eb096ce0c > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-10-12 09:07:05 +0200 > message: > > MDEV-17401: LOAD DATA from very big file into MyISAM table results in EOF error and corrupt index > > my_read fixed as in higher versions. > my_pread made as my_read aware of partial read of huge chunks of files > MY_FULL_IO enabled for file operations > > --- > mysys/mf_iocache.c | 4 ++++ > mysys/my_pread.c | 25 ++++++++++++++++++------- > mysys/my_read.c | 52 ++++++++++++++++++++++++++++------------------------ > 3 files changed, 50 insertions(+), 31 deletions(-) > > diff --git a/mysys/mf_iocache.c b/mysys/mf_iocache.c > index bd71e2ae527..9a051cf6a9e 100644 > --- a/mysys/mf_iocache.c > +++ b/mysys/mf_iocache.c > @@ -282,6 +282,10 @@ int init_io_cache(IO_CACHE *info, File file, size_t cachesize, > } > info->inited=info->aio_result.pending=0; > #endif > + if (type == READ_CACHE || type == WRITE_CACHE || type == SEQ_READ_APPEND) > + info->myflags|= MY_FULL_IO; > + else > + info->myflags&= ~MY_FULL_IO; 1. that's a bit redundant, it'd be enough to do it in init_io_cache. but ok. 2. why you didn't remove MY_FULL_IO setting from reinit_io_cache? > DBUG_RETURN(0); > } /* init_io_cache */ > > diff --git a/mysys/my_pread.c b/mysys/my_pread.c > index 51b4c5f5599..5cbd6b4316b 100644 > --- a/mysys/my_pread.c > +++ b/mysys/my_pread.c > @@ -47,8 +47,7 @@ > size_t my_pread(File Filedes, uchar *Buffer, size_t Count, my_off_t offset, > myf MyFlags) > { > - size_t readbytes; > - int error= 0; > + size_t readbytes, save_count= 0; > > DBUG_ENTER("my_pread"); > > @@ -66,11 +65,10 @@ size_t my_pread(File Filedes, uchar *Buffer, size_t Count, my_off_t offset, > #else > readbytes= pread(Filedes, Buffer, Count, offset); > #endif > - error = (readbytes != Count); > > - if (error) > + if (readbytes != Count) > { > - my_errno= errno ? errno : -1; > + my_errno= errno; > if (errno == 0 || (readbytes != (size_t) -1 && > (MyFlags & (MY_NABP | MY_FNABP)))) > my_errno= HA_ERR_FILE_TOO_SHORT; > @@ -82,6 +80,17 @@ size_t my_pread(File Filedes, uchar *Buffer, size_t Count, my_off_t offset, > (int) readbytes)); > continue; /* Interrupted */ > } > + > + /* Do a read retry if we didn't get enough data on first read */ > + if (readbytes != (size_t) -1 && readbytes != 0 && > + (MyFlags & MY_FULL_IO)) > + { > + Buffer+= readbytes; > + Count-= readbytes; > + save_count+= readbytes; > + continue; eh, did you test your patch? I don't see how it could work without offset+= readbytes; > + } > + > if (MyFlags & (MY_WME | MY_FAE | MY_FNABP)) > { > if (readbytes == (size_t) -1) > @@ -97,8 +106,10 @@ size_t my_pread(File Filedes, uchar *Buffer, size_t Count, my_off_t offset, > DBUG_RETURN(MY_FILE_ERROR); /* Return with error */ > } > if (MyFlags & (MY_NABP | MY_FNABP)) > - DBUG_RETURN(0); /* Read went ok; Return 0 */ > - DBUG_RETURN(readbytes); /* purecov: inspected */ > + readbytes= 0; /* Read went ok; Return 0 */ > + else > + readbytes+= save_count; > + DBUG_RETURN(readbytes); > } > } /* my_pread */ > > diff --git a/mysys/my_read.c b/mysys/my_read.c > index 883a1c5fdc7..47f2d725f65 100644 > --- a/mysys/my_read.c > +++ b/mysys/my_read.c > @@ -35,17 +35,16 @@ this is copied from 10.1 verbatim, I presume Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] c850799967c: MDEV-10963 Fragmented BINLOG query
by Sergei Golubchik 23 Oct '18

23 Oct '18

Hi, Andrei! First, when I run it I got: Warning: 112 bytes lost at 0x7f212f26a670, allocated by T@0 at sql/sql_binlog.cc:68, sql/sql_binlog.cc:181, sql/sql_parse.cc:5625, sql/sql_parse.cc:7465, sql/sql_parse.cc:1497, sql/sql_parse.cc:1124, sql/sql_connect.cc:1330, sql/sql_connect.cc:1243 so you have a memory leak with your thd->lex->comment.str Second, BINGLOG CONCAT() is totally possible, inventing a new keyword defeats the whole point of using a familiar and intuitive syntax. I've attached a simple patch that implements it. On the other hand, I just thought that CONCAT() syntax might be not so intuitive after all, as - strictly speaking - a result of CONCAT cannot be larger than max_allowed_packet. So it might be confusing to use CONCAT here. May be just use your idea of BINLOG @a, @b; with no CONCAT or anything else at all? Also, I'm thinking whether BINLOG (in this last form) could automatically clear its variables @a and @b. Without an explicit SET @a=NULL. It's a bit weird to have such a side effect. But it'll help to save memory - the event size it over 3GB, right? So you'll have it split in two variables. Then you create a concatenated copy (it's over 3GB more), then you decode it (already more than 9GB in memory), and then it's executed where copies of huge blob values are created again, may be more than once. Freeing @a and @b and the concatenated copy as soon as possible will save over 6GB in RAM usage. Note, we cannot auto-reset variables in the syntax with CONCAT(), DEFRAGMENT() or whatever, because functions cannot do it to their arguments. On Oct 18, Andrei Elkin wrote: > revision-id: c850799967c561d08242987269d94af6ae4c7c5e (mariadb-10.1.35-59-gc850799967c) > parent(s): d3a8b5aa9cee24a6397662e30df2e915f45460e0 > author: Andrei Elkin <andrei.elkin(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2018-09-21 18:48:06 +0300 > message: > > MDEV-10963 Fragmented BINLOG query > > The problem was originally stated in > http://bugs.mysql.com/bug.php?id=82212 > The size of an base64-encoded Rows_log_event exceeds its > vanilla byte representation in 4/3 times. > When a binlogged event size is about 1GB mysqlbinlog generates > a BINLOG query that can't be send out due to its size. > > It is fixed with fragmenting the BINLOG argument C-string into > (approximate) halves when the base64 encoded event is over 1GB size. > The mysqlbinlog in such case puts out > > SET @binlog_fragment_0='base64-encoded-fragment_0'; > SET @binlog_fragment_1='base64-encoded-fragment_1'; > BINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_1); > > to represent a big BINLOG 'base64-encoded-"total"'. > Two more statements are composed to promptly release memory > SET @binlog_fragment_0=NULL; > SET @binlog_fragment_1=NULL; > > The 2 fragments are enough, though the client and server still may > need to tweak their @@max_allowed_packet to satisfy to the fragment > size (which they would have to do anyway with greater number of > fragments, should that be desired). > > On the lower level the following changes are made: > > Log_event::print_base64() > remains to call encoder and store the encoded data into a cache but > now *without* doing any formatting. The latter is left for time > when the cache is copied to an output file (e.g mysqlbinlog output). > No formatting behavior is also reflected by the change in the meaning > of the last argument which specifies whether to cache the encoded data. > > my_b_copy_to_file() > is turned into my_b_copy_to_file_frag() > which accepts format specifier arguments to build a proper syntax BINLOG > query in both the fragmented (n_frag > 1) and non-fragmented (n_frag == 1) > cases. > > Rows_log_event::print_helper() > Takes decision whether to fragment, prepares respective format specifiers > and invokes the cache-to-file copying function, which is now > > copy_cache_frag_to_file_and_reinit() > replaces original copy_event_cache_to_file_and_reinit() to pass > extra arguments to my_b_copy_to_file() successor of > > my_b_copy_to_file_frag() > replaces the former pure copier not to also conduct wrapping the encoded > data per format specifiers. > With its 'n_frag' argument as 1 > and the rest or args NULL works as the original function. > > diff --git a/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row_frag.test b/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row_frag.test > new file mode 100644 > index 00000000000..5318f8acec6 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row_frag.test > @@ -0,0 +1,43 @@ > +--source include/have_debug.inc > +--source include/have_log_bin.inc > +--source include/have_binlog_format_row.inc > + > +--let $MYSQLD_DATADIR= `select @@datadir` > +--let $max_size=1024 > + > +CREATE TABLE t (a TEXT); > +# events of interest are guaranteed to stay in 000001 log > +RESET MASTER; > +--eval INSERT INTO t SET a=repeat('a', $max_size) > +SELECT a from t into @a; > +FLUSH LOGS; > +DELETE FROM t; > + > +--exec $MYSQL_BINLOG --debug-binlog-row-event-max-encoded-size=256 $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/mysqlbinlog.sql > + > +--exec $MYSQL test < $MYSQLTEST_VARDIR/tmp/mysqlbinlog.sql how do you know the event was split, if you aren't looking into mysqlbinlog.sql file? may be your code doesn't work at all, or it was removed by some incorrect merge? the test will pass anyway... > + > +SELECT a LIKE @a as 'true' FROM t; > +SELECT @binlog_fragment_0, @binlog_fragment_1 as 'NULL'; > + > +# improper syntax error > +--echo BINLOG number-of-fragments must be exactly two > +--error ER_PARSE_ERROR > +BINLOG DEFRAGMENT(@binlog_fragment); > +--error ER_PARSE_ERROR > +BINLOG DEFRAGMENT(@binlog_fragment, @binlog_fragment, @binlog_fragment); > + > +# corrupted fragments error check (to the expected error code notice, > +# the same error code occurs in a similar unfragmented case) > +SET @binlog_fragment_0='012345'; > +SET @binlog_fragment_1='012345'; > +--error ER_SYNTAX_ERROR > +BINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_1); > + > +# Not existing fragment is not allowed > +SET @binlog_fragment_0='012345'; > +--error ER_BASE64_DECODE_ERROR > +BINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_not_exist); > + > +--echo # Cleanup > +DROP TABLE t; > diff --git a/mysys/mf_iocache2.c b/mysys/mf_iocache2.c > index 2499094037d..6b7ff8a7568 100644 > --- a/mysys/mf_iocache2.c > +++ b/mysys/mf_iocache2.c > @@ -22,13 +22,23 @@ > #include <stdarg.h> > #include <m_ctype.h> > > -/* > +/** > Copy contents of an IO_CACHE to a file. > > SYNOPSIS > - my_b_copy_to_file() > - cache IO_CACHE to copy from > - file File to copy to > + my_b_copy_to_file_frag > + > + cache IO_CACHE to copy from > + file File to copy to > + n_frag # of fragments > + > + Other arguments represent format strings to enable wrapping > + of the fragments and their total, including > + > + before_frag before a fragment > + after_frag after a fragment > + after_last_frag after all the fragments > + final_per_frag in the end per each fragment if you edit a comment, change it to Doxygen syntax. Especially, as you start it with Doxygen's /**, it is simply invalid to have SYNOPSIS and DESCRIPTION inside. > > DESCRIPTION > Copy the contents of the cache to the file. The cache will be > @@ -38,33 +48,120 @@ > If a failure to write fully occurs, the cache is only copied > partially. > > - TODO > - Make this function solid by handling partial reads from the cache > - in a correct manner: it should be atomic. > + The copying is made in so many steps as the number of fragments as > + specified by the parameter 'n_frag'. Each step is wrapped with > + writing to the file 'before_frag' and 'after_frag' formated > + strings, unless the parameters are NULL. In the end, optionally, > + first 'after_last_frag' string is appended to 'file' followed by > + 'final_per_frag' per each fragment. > + final item. > > RETURN VALUE > 0 All OK > 1 An error occurred > + > + TODO > + Make this function solid by handling partial reads from the cache > + in a correct manner: it should be atomic. > */ > int > -my_b_copy_to_file(IO_CACHE *cache, FILE *file) > +my_b_copy_to_file_frag(IO_CACHE *cache, FILE *file, > + uint n_frag, > + const char* before_frag, > + const char* after_frag, > + const char* after_last, > + const char* final_per_frag, > + char* buf) > { > - size_t bytes_in_cache; > - DBUG_ENTER("my_b_copy_to_file"); > + size_t bytes_in_cache; // block, may have short size in the last one > + size_t written_off_last_block; // consumed part of the block by last fragment > + size_t total_size= my_b_tell(cache); > + size_t frag_size= total_size / n_frag + 1; > + size_t total_written= 0; > + size_t frag_written; // bytes collected in the current fragment > + uint i; > + > + DBUG_ENTER("my_b_copy_to_file_frag"); > + > + DBUG_ASSERT(cache->type == WRITE_CACHE); > > /* Reinit the cache to read from the beginning of the cache */ > if (reinit_io_cache(cache, READ_CACHE, 0L, FALSE, FALSE)) > DBUG_RETURN(1); > - bytes_in_cache= my_b_bytes_in_cache(cache); > - do > - { > - if (my_fwrite(file, cache->read_pos, bytes_in_cache, > - MYF(MY_WME | MY_NABP)) == (size_t) -1) > - DBUG_RETURN(1); > - } while ((bytes_in_cache= my_b_fill(cache))); > - if(cache->error == -1) > - DBUG_RETURN(1); > - DBUG_RETURN(0); > + > + for (i= 0, written_off_last_block= 0, bytes_in_cache= my_b_bytes_in_cache(cache); > + i < n_frag; > + i++, total_written += frag_written) > + { > + frag_written= 0; > + if (before_frag) > + { > + sprintf(buf, before_frag, i); > + my_fwrite(file, (uchar*) buf, strlen(buf), MYF(MY_WME | MY_NABP)); > + } > + do > + { > + /* > + Either the current block is the last (L) in making the > + current fragment (and possibly has some extra not to fit (LG) into > + the fragment), or (I) the current (whole then) block is > + intermediate. > + */ > + size_t block_to_write= (frag_written + bytes_in_cache >= frag_size) ? > + frag_size - frag_written : bytes_in_cache; > + > + DBUG_ASSERT(n_frag != 1 || > + (block_to_write == bytes_in_cache && > + written_off_last_block == 0)); > + > + if (my_fwrite(file, cache->read_pos + written_off_last_block, > + block_to_write, > + MYF(MY_WME | MY_NABP)) == (size_t) -1) > + /* no cache->error is set here */ > + DBUG_RETURN(1); > + > + frag_written += block_to_write; > + if (frag_written == frag_size) // (L) > + { > + DBUG_ASSERT(block_to_write <= bytes_in_cache); > + written_off_last_block= block_to_write; > + bytes_in_cache -= written_off_last_block; // (LG) when bytes>0 > + /* > + Nothing should be left in cache at the end of the > + last fragment construction. > + */ > + DBUG_ASSERT(i != n_frag - 1 || bytes_in_cache == 0); > + > + break; > + } > + else > + { > + written_off_last_block= 0; // (I) > + } > + } while ((bytes_in_cache= my_b_fill(cache))); > + > + if (after_frag) > + { > + sprintf(buf, after_frag, NULL); > + my_fwrite(file, (uchar*) buf, strlen(buf), MYF(MY_WME | MY_NABP)); > + } > + } > + > + DBUG_ASSERT(total_written == total_size); // output == input > + > + if (after_last) > + { > + sprintf(buf, after_last, n_frag); > + my_fwrite(file, (uchar*) buf, strlen(buf), MYF(MY_WME | MY_NABP)); > + } > + > + for (i= 0; final_per_frag && i < n_frag ; i++) > + { > + sprintf(buf, final_per_frag, i); > + my_fwrite(file, (uchar*) buf, strlen(buf), MYF(MY_WME | MY_NABP)); > + } > + > + DBUG_RETURN(cache->error == -1); this is a big and very specialized function with non-standard indentation. Please, move it from mysys to mysqlbinlog.cc and fix the indentation. > } > > > diff --git a/sql/log_event.cc b/sql/log_event.cc > index e1912ad4620..0c39200148f 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -10474,12 +10482,108 @@ void Rows_log_event::pack_info(Protocol *protocol) > #endif > > #ifdef MYSQL_CLIENT > +void copy_cache_to_file_wrapped(FILE *file, > + PRINT_EVENT_INFO *print_event_info, > + IO_CACHE *body, > + bool do_print_encoded) > +{ > + uint n_frag= 1; > + const char* before_frag= NULL; > + char* after_frag= NULL; > + char* after_last= NULL; > + char* final_per_frag= NULL; > + /* > + 2 fragments can always represent near 1GB row-based > + base64-encoded event as two strings each of size less than > + max(max_allowed_packet). Greater number of fragments does not > + save from potential need to tweak (increase) @@max_allowed_packet > + before to process the fragments. So 2 is safe and enough. > + */ > + const char fmt_last_frag2[]= > + "\nBINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_1)%s\n"; > + const char fmt_last_per_frag[]= "\nSET @binlog_fragment_%%d = NULL%s\n"; this is completely unnecessary "generalization", print simply "SET @binlog_fragment_0=NULL,binlog_fragment_1=NULL%s\n" in particular, as your "BINLOG DEFRAGMENT" statement is not generalized. (or to the cleanup inside BINLOG, as discussed above). > + const char fmt_before_frag[]= "\nSET @binlog_fragment_%d ='\n"; > + /* > + Buffer to pass to copy_cache_frag_to_file_and_reinit to > + compute formatted strings according to specifiers. > + The sizes may depend on an actual fragment number size in terms of decimal > + signs so its maximum is estimated (not precisely yet safely) below. > + */ > + char buf[(sizeof(fmt_last_frag2) + sizeof(fmt_last_per_frag)) > + + ((sizeof(n_frag) * 8)/3 + 1) // decimal index > + + sizeof(print_event_info->delimiter) + 3]; // delim, \n and 0. > + > + if (do_print_encoded) > + { > + after_frag= (char*) my_malloc(sizeof(buf), MYF(MY_WME)); > + sprintf(after_frag, "'%s\n", print_event_info->delimiter); > + if (my_b_tell(body) > opt_binlog_rows_event_max_encoded_size) > + n_frag= 2; > + if (n_frag > 1) > + { > + before_frag= fmt_before_frag; > + after_last= (char*) my_malloc(sizeof(buf), MYF(MY_WME)); > + sprintf(after_last, fmt_last_frag2, (char*) print_event_info->delimiter); > + final_per_frag= (char*) my_malloc(sizeof(buf), MYF(MY_WME)); > + sprintf(final_per_frag, fmt_last_per_frag, > + (char*) print_event_info->delimiter); > + } > + else > + { > + before_frag= "\nBINLOG '\n"; > + } > + } > + if (copy_cache_frag_to_file_and_reinit(body, file, n_frag, > + before_frag, after_frag, > + after_last, final_per_frag, buf)) 1. get rid of this copy_cache_frag_to_file_and_reinit function, it's not helping. 2. instead of pushing all this complex logic of printing fragments with before and after and in-between and start and final, it'd be *much* simpler to do it here and just extend my_b_copy_to_file() to print at most N bytes. > + { > + body->error= -1; > + goto err; > + } > + > +err: > + my_free(after_frag); > + my_free(after_last); > + my_free(final_per_frag); > +} > + > +/* > + The function invokes base64 encoder to run on the current > + event string and store the result into two caches. > + When the event ends the current statement the caches are is copied into > + the argument file. > + Copying is also concerned how to wrap the event, specifically to produce > + a valid SQL syntax. > + When the encoded data size is within max(MAX_ALLOWED_PACKET) > + a regular BINLOG query is composed. Otherwise it is build as fragmented > + > + SET @binlog_fragment_0='...'; > + SET @binlog_fragment_1='...'; > + BINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_1); > + > + where fragments are represented by a sequence of "indexed" user > + variables. > + Two more statements are composed as well > + > + SET @binlog_fragment_0=NULL; > + SET @binlog_fragment_1=NULL; > + > + to promptly release memory. > + > + NOTE. > + If any changes made don't forget to duplicate them to > + Old_rows_log_event as long as it's supported. > +*/ > void Rows_log_event::print_helper(FILE *file, > PRINT_EVENT_INFO *print_event_info, > char const *const name) > { > IO_CACHE *const head= &print_event_info->head_cache; > IO_CACHE *const body= &print_event_info->body_cache; > + bool do_print_encoded= > + print_event_info->base64_output_mode != BASE64_OUTPUT_DECODE_ROWS && > + !print_event_info->short_form; > + > if (!print_event_info->short_form) > { > bool const last_stmt_event= get_flags(STMT_END_F); > diff --git a/sql/sql_binlog.cc b/sql/sql_binlog.cc > index 91cf038907e..b15d72e036a 100644 > --- a/sql/sql_binlog.cc > +++ b/sql/sql_binlog.cc > @@ -28,6 +28,73 @@ > // START_EVENT_V3, > // Log_event_type, > // Log_event > + > +/* > + Copy fragments into the standard placeholder thd->lex->comment.str. > + > + Compute the size of the (still) encoded total, > + allocate and then copy fragments one after another. > + The size can exceed max(max_allowed_packet) which is not a > + problem as no String instance is created off this char array. > + > + Return 0 at success, -1 otherwise. > +*/ > +int binlog_defragment(THD *thd) > +{ > + LEX_STRING *curr_frag_name; > + > + thd->lex->comment.length= 0; > + thd->lex->comment.str= NULL; > + /* compute the total size */ > + for (uint i= 0; i < thd->lex->fragmented_binlog_event.n_frag; i++) > + { > + user_var_entry *entry; > + > + curr_frag_name= &thd->lex->fragmented_binlog_event.frag_name[i]; > + entry= > + (user_var_entry*) my_hash_search(&thd->user_vars, > + (uchar*) curr_frag_name->str, > + curr_frag_name->length); > + if (!entry || entry->type != STRING_RESULT) > + { > + my_printf_error(ER_BASE64_DECODE_ERROR, > + "%s: BINLOG fragment user " > + "variable '%s' has unexpectedly no value", MYF(0), > + ER_THD(thd, ER_BASE64_DECODE_ERROR), curr_frag_name->str); > + return -1; > + } > + thd->lex->comment.length += entry->length; > + } > + thd->lex->comment.str= // to be freed by the caller > + (char *) my_malloc(thd->lex->comment.length, MYF(MY_WME)); > + if (!thd->lex->comment.str) > + { > + my_error(ER_OUTOFMEMORY, MYF(ME_FATALERROR), 1); > + return -1; > + } > + > + /* fragments are merged into allocated buf */ > + size_t gathered_length= 0; > + for (uint i= 0; i < thd->lex->fragmented_binlog_event.n_frag; i++) > + { > + user_var_entry *entry; > + > + curr_frag_name= &thd->lex->fragmented_binlog_event.frag_name[i]; > + entry= > + (user_var_entry*) my_hash_search(&thd->user_vars, > + (uchar*) curr_frag_name->str, > + curr_frag_name->length); > + memcpy(thd->lex->comment.str + gathered_length, > + entry->value, entry->length); > + > + gathered_length += entry->length; > + } > + DBUG_ASSERT(gathered_length == thd->lex->comment.length); ok. 1. BINLOG uses LEX::comment to avoid increasing sizeof(LEX) for a rarely used statement. So don't increase it for a much-more-rarely used event overlow case. That is, remove fragmented_binlog_event. You can use a pair of LEX_STRING's in LEX (e.g. comment/ident) or (less hackish) List<LEX_STRING>, which are plenty in LEX. 2. Don't look up variables twice, remember them in the first loop. You can unroll it too, there're only two variables anyway. E.g. the second loop can be replaced by just memcpy(thd->lex->comment.str, var[0]->value, var[0]->length); memcpy(thd->lex->comment.str + var[0]->length, var[1]->value, var[1]->length); or even shorter, if you start your function with LEX_STRING *out= &thd->lex->comment; 3. Not "unexpectedly no value". ER_WRONG_TYPE_FOR_VAR looks more appropriate there. > + > + return 0; > +} > + > + > /** > Execute a BINLOG statement. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 4

[Maria-developers] MDEV-13912
by Michal Schorm 22 Oct '18

22 Oct '18

Hello, can I get some status update on MDEV-13912 please? It is still an issue for the customer on my side and I'd love to see fixed it in 5.5.62 release. Michal -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

2 2

Re: [Maria-developers] cf9bafc: MDEV-14815 - Server crash or AddressSanitizer errors or valgrind warnings
by Sergei Golubchik 19 Oct '18

19 Oct '18

Hi, Sergey! On Oct 19, Sergey Vojtovich wrote: > revision-id: cf9bafc8cb0943c914d118525888140ea15e6b55 (mariadb-10.0.36-50-gcf9bafc) > parent(s): 8e716138cef2d9be603cdf71701d82bcb72dfd69 > author: Sergey Vojtovich > committer: Sergey Vojtovich > timestamp: 2018-10-18 23:50:50 +0400 > message: > > MDEV-14815 - Server crash or AddressSanitizer errors or valgrind warnings > in thr_lock / has_old_lock upon FLUSH TABLES > > Explicit partition access of partitioned MEMORY table under LOCK TABLES > may cause subsequent statements to crash the server, deadlock, trigger > valgrind warnings or ASAN errors. Freed memory was being used due to > incorrect cleanup. > > At least MyISAM and InnoDB don't seem to be affected, since their > THR_LOCK structures don't survive FLUSH TABLES. MEMORY keeps table shared > data (including THR_LOCK) even if there're no open instances. > > There's partition_info::lock_partitions bitmap, which holds bits of > partitions allowed to be accessed after pruning. This bitmap is > updated for each individual statement. > > This bitmap was abused in ha_partition::store_lock() such that when we > need to unlock a table, locked by LOCK TABLES, only locks for partitions > that were accessed by previous statement were released. > > Eventually FLUSH TABLES frees THR_LOCK_DATA objects, which are still > linked into THR_LOCK lists. When such THR_LOCK gets reused we end up with > freed memory access. > > Fixed by using ha_partition::m_locked_partitions bitmap similarly to > ha_partition::external_lock(). Thanks, good comment. > Some unused code removed. Generally, it's better to do that in a separate commit. It's very easy to do with git citool, and doesn't requite any advance planning, you just edit whatever you like and then run 'git citool' twice, selecting what lines you want to commit each time. > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 0488ebf..02de92a 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -3907,9 +3907,14 @@ THR_LOCK_DATA **ha_partition::store_lock(THD *thd, > } > else > { > - for (i= bitmap_get_first_set(&(m_part_info->lock_partitions)); > + MY_BITMAP *used_partitions= lock_type == TL_UNLOCK || > + lock_type == TL_IGNORE ? why TL_IGNORE too? external_lock only checks for TL_UNLOCK. > + &m_locked_partitions : > + &m_part_info->lock_partitions; > + > + for (i= bitmap_get_first_set(used_partitions); > i < m_tot_parts; > - i= bitmap_get_next_set(&m_part_info->lock_partitions, i)) > + i= bitmap_get_next_set(used_partitions, i)) > { > DBUG_PRINT("info", ("store lock %d iteration", i)); > to= m_file[i]->store_lock(thd, to, lock_type); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] 7ce7184cf0e: MDEV-17133 dump thread reads from a past position
by Sergei Golubchik 19 Oct '18

19 Oct '18

Hi, Andrei! ok to push. But please add a meaningful commit comment. Thanks! On Oct 19, Andrei Elkin wrote: > revision-id: 7ce7184cf0e354a1c9114bb31df1fb63b1fbbcc7 (mariadb-10.1.35-59-g7ce7184cf0e) > parent(s): d3a8b5aa9cee24a6397662e30df2e915f45460e0 > author: Andrei Elkin <andrei.elkin(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2018-09-18 23:21:18 +0300 > message: > > MDEV-17133 dump thread reads from a past position > > --- > mysys/mf_iocache.c | 8 +++-- > unittest/sql/mf_iocache-t.cc | 74 +++++++++++++++++++++++++++++++++++++++++++- > 2 files changed, 79 insertions(+), 3 deletions(-) > > diff --git a/mysys/mf_iocache.c b/mysys/mf_iocache.c > index 56b1ae3fc6e..8f829a8c9c2 100644 > --- a/mysys/mf_iocache.c > +++ b/mysys/mf_iocache.c > @@ -563,7 +563,7 @@ int _my_b_write(IO_CACHE *info, const uchar *Buffer, size_t Count) > > int _my_b_cache_read(IO_CACHE *info, uchar *Buffer, size_t Count) > { > - size_t length, diff_length, left_length= 0, max_length; > + size_t length= 0, diff_length, left_length= 0, max_length; > my_off_t pos_in_file; > DBUG_ENTER("_my_b_cache_read"); > > @@ -668,7 +668,11 @@ int _my_b_cache_read(IO_CACHE *info, uchar *Buffer, size_t Count) > else > { > info->error= 0; > - DBUG_RETURN(0); /* EOF */ > + if (length == 0) > + { > + DBUG_RETURN(0); // EOF > + } > + length= 0; > } > } > else if ((length= mysql_file_read(info->file,info->buffer, max_length, Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Correct parsing of BINLOG 'XXX' with comment inside
by Sachin Setia 19 Oct '18

19 Oct '18

Hi Bar, Currently if we have generated the sql file using mysqlbinlog -v and if we have big statement binlog , then mysqlbinlog add comment in between of Binlog 'XXX' statement , but unfortunately base64_decode does not understands it, and it through error since # is not a base64 char This patches solves this. There are 2 patches one for 5.5 and 2nd for 10.0 and above Please review it.

4 7

Re: [Maria-developers] 9b453c1ebf4: MDEV-10963 Fragmented BINLOG query
by Sergei Golubchik 18 Oct '18

18 Oct '18

Hi, Andrei! On Sep 07, andrei.elkin(a)pp.inet.fi wrote: > revision-id: 9b453c1ebf47a20ea1436de3284f8810d0e64c4c (mariadb-10.1.34-8-g9b453c1ebf4) > parent(s): c09a8b5b36edb494e2bcc93074c06e26cd9f2b92 > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-09-07 20:36:16 +0300 > message: > > MDEV-10963 Fragmented BINLOG query > > The problem was originally stated in > http://bugs.mysql.com/bug.php?id=82212 > The size of an base64-encoded Rows_log_event exceeds its > vanilla byte representation in 4/3 times. > When a binlogged event size is about 1GB mysqlbinlog generates > a BINLOG query that can't be send out due to its size. > > It is fixed with fragmenting the BINLOG argument C-string into > (approximate) halves when the base64 encoded event is over 1GB size. > The mysqlbinlog in such case puts out > > SET @binlog_fragment_0='base64-encoded-fragment_0'; > SET @binlog_fragment_1='base64-encoded-fragment_1'; > BINLOG 2, 'binlog_fragment'; No, that's totally ridiculous. A statement that internally walks the list of user variables and collects values from variables named according to a specific pattern, seriously? Please, make it BINLOG CONCAT(@binlog_fragment_0, @binlog_fragment_1) that'll work with no questions asked, everybody understands what it means. The parser doesn't need to accept an arbitrary expression here, it'd be simpler and safer to hard-code the syntax as above. A couple of minor generic comments below, but I've stopped reviewing, because the code will change anyway. > to represent a big BINLOG 'base64-encoded-"total"'. > Two more statements are composed to promptly release memory > SET @binlog_fragment_0=NULL; > SET @binlog_fragment_1=NULL; > > diff --git a/client/mysqlbinlog.cc b/client/mysqlbinlog.cc > index 9753125dd67..f0689631cfb 100644 > --- a/client/mysqlbinlog.cc > +++ b/client/mysqlbinlog.cc > @@ -56,7 +56,13 @@ Rpl_filter *binlog_filter= 0; > > #define BIN_LOG_HEADER_SIZE 4 > #define PROBE_HEADER_LEN (EVENT_LEN_OFFSET+4) > - > +/* > + 2 fragments can always represent near 1GB row-based base64-encoded event as > + two strings each of size less than max(max_allowed_packet). > + Bigger number of fragments does not safe from potential need to tune (increase) > + @@max_allowed_packet before to process the fragments. So 2 is safe and enough. > +*/ > +#define BINLOG_ROWS_EVENT_ENCODED_FRAGMENTS 2 > > #define CLIENT_CAPABILITIES (CLIENT_LONG_PASSWORD | CLIENT_LONG_FLAG | CLIENT_LOCAL_FILES) > > @@ -71,6 +77,9 @@ ulong bytes_sent = 0L, bytes_received = 0L; > ulong mysqld_net_retry_count = 10L; > ulong open_files_limit; > ulong opt_binlog_rows_event_max_size; > +#ifndef DBUG_OFF > +ulong opt_binlog_rows_event_max_encoded_size; > +#endif Try to avoid unnecessary ifdefs. Here you could've defined the variable unconditionally and suppressed the warning with __attribute__((unused)), of you could've used it unconditionally as a splitting threshold below and only hidden the command line option. > uint test_flags = 0; > static uint opt_protocol= 0; > static FILE *result_file; > @@ -813,7 +822,14 @@ write_event_header_and_base64(Log_event *ev, FILE *result_file, > > /* Write header and base64 output to cache */ > ev->print_header(head, print_event_info, FALSE); > - ev->print_base64(body, print_event_info, FALSE); > + > + /* the assert states the only current use case for the function */ > + DBUG_ASSERT(print_event_info->base64_output_mode == > + BASE64_OUTPUT_ALWAYS); 1. don't document the obvious 2. don't wrap the line if it fits in 80-char line > + > + ev->print_base64(body, print_event_info, > + print_event_info->base64_output_mode != > + BASE64_OUTPUT_DECODE_ROWS); > > /* Read data from cache and write to result file */ > if (copy_event_cache_to_file_and_reinit(head, result_file) || > @@ -1472,6 +1490,15 @@ that may lead to an endless loop.", > "This value must be a multiple of 256.", > &opt_binlog_rows_event_max_size, &opt_binlog_rows_event_max_size, 0, > GET_ULONG, REQUIRED_ARG, UINT_MAX, 256, ULONG_MAX, 0, 256, 0}, > +#ifndef DBUG_OFF > + {"binlog-row-event-max-encoded-size", 0, all debug command-line options must start with "debug-", that is, either rename it to "debug-binlog-row-event-max-encoded-size" or make it non-debug. > + "The maximum size of base64-encoded rows-event in one BINLOG pseudo-query " > + "instance. When the computed actual size exceeds the limit " > + "the BINLOG's argument string is fragmented in two.", > + &opt_binlog_rows_event_max_encoded_size, > + &opt_binlog_rows_event_max_encoded_size, 0, > + GET_ULONG, REQUIRED_ARG, UINT_MAX/4, 256, ULONG_MAX, 0, 256, 0}, > +#endif > {"verify-binlog-checksum", 'c', "Verify checksum binlog events.", > (uchar**) &opt_verify_binlog_checksum, (uchar**) &opt_verify_binlog_checksum, > 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] [Commits] 2f4a0c5be2c: Fix accumulation of old rows in mysql.gtid_slave_pos
by Kristian Nielsen 13 Oct '18

13 Oct '18

Hi Andrei! Can you review this patch? This fixes a rather serious bug in parallel replication that was reported on maria-discuss@ here: https://lists.launchpad.net/maria-discuss/msg05253.html The user ended up with millions of rows in mysql.gtid_slave_pos, which is quite bad. I think this should go in 10.1. The bug is probably also present in 10.0, but it will be much harder to trigger since 10.0 does not have optimistic parallel replication. So leaving it unfixed in 10.0 seems ok. I plan to manually merge it to 10.2 and 10.3 as well, since the merge to 10.3 is not entirely trivial, due to interaction with @@gtid_pos_auto_engines. Github branch here: https://github.com/knielsen/server/commits/gtid_table_garbage_rows - Kristian. Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > revision-id: 2f4a0c5be2c5d5153c4253a49ba8820ab333a9a0 (mariadb-10.1.35-71-g2f4a0c5be2c) > parent(s): 1fc5a6f30c3a9c047dcf9a36b00026d98f286f6b > author: Kristian Nielsen > committer: Kristian Nielsen > timestamp: 2018-10-07 18:59:52 +0200 > message: > > Fix accumulation of old rows in mysql.gtid_slave_pos > > This would happen especially in optimistic parallel replication, where there > is a good chance that a transaction will be rolled back (due to conflicts) > after it has executed record_gtid(). If the transaction did any deletions of > old rows as part of record_gtid(), those deletions will be undone as well. > And the code did not properly ensure that the deletions would be re-tried. > > This patch makes record_gtid() remember the list of deletions done as part > of a transaction. Then in rpl_slave_state::update() when the changes have > been committed, we discard the list. However, in case of error and rollback, > in cleanup_context() we will instead put the list back into > rpl_global_gtid_slave_state so that the deletions will be re-tried later. > > Probably fixes part of the cause of MDEV-12147 as well. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > > --- > .../suite/rpl/r/rpl_parallel_optimistic.result | 6 ++ > .../suite/rpl/t/rpl_parallel_optimistic.test | 17 +++++ > sql/log_event.cc | 6 +- > sql/rpl_gtid.cc | 64 ++++++++++++------ > sql/rpl_gtid.h | 2 +- > sql/rpl_rli.cc | 79 ++++++++++++++++++++++ > sql/rpl_rli.h | 11 +++ > 7 files changed, 161 insertions(+), 24 deletions(-) > > diff --git a/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result b/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > index 3cd4f8231bf..99bd8562ffe 100644 > --- a/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > +++ b/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > @@ -571,4 +571,10 @@ SET GLOBAL slave_parallel_mode=@old_parallel_mode; > SET GLOBAL slave_parallel_threads=@old_parallel_threads; > include/start_slave.inc > DROP TABLE t1, t2, t3; > +include/save_master_gtid.inc > +include/sync_with_master_gtid.inc > +Check that no more than the expected last two GTIDs are in mysql.gtid_slave_pos > +select count(*) from mysql.gtid_slave_pos order by domain_id, sub_id; > +count(*) > +2 > include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test b/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > index 9f6669279db..3867a3fdf3a 100644 > --- a/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > +++ b/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > @@ -549,5 +549,22 @@ SET GLOBAL slave_parallel_threads=@old_parallel_threads; > > --connection server_1 > DROP TABLE t1, t2, t3; > +--source include/save_master_gtid.inc > + > +--connection server_2 > +--source include/sync_with_master_gtid.inc > +# Check for left-over rows in table mysql.gtid_slave_pos (MDEV-12147). > +# > +# There was a bug when a transaction got a conflict and was rolled back. It > +# might have also handled deletion of some old rows, and these deletions would > +# then also be rolled back. And since the deletes were never re-tried, old no > +# longer needed rows would accumulate in the table without limit. > +# > +# The earlier part of this test file have plenty of transactions being rolled > +# back. But the last DROP TABLE statement runs on its own and should never > +# conflict, thus at this point the mysql.gtid_slave_pos table should be clean. > +--echo Check that no more than the expected last two GTIDs are in mysql.gtid_slave_pos > +select count(*) from mysql.gtid_slave_pos order by domain_id, sub_id; > > +--connection server_1 > --source include/rpl_end.inc > diff --git a/sql/log_event.cc b/sql/log_event.cc > index e1912ad4620..e07b7002398 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -4429,7 +4429,7 @@ int Query_log_event::do_apply_event(rpl_group_info *rgi, > > gtid= rgi->current_gtid; > if (rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, > - true, false)) > + rgi, false)) > { > int errcode= thd->get_stmt_da()->sql_errno(); > if (!is_parallel_retry_error(rgi, errcode)) > @@ -7132,7 +7132,7 @@ Gtid_list_log_event::do_apply_event(rpl_group_info *rgi) > { > if ((ret= rpl_global_gtid_slave_state->record_gtid(thd, &list[i], > sub_id_list[i], > - false, false))) > + NULL, false))) > return ret; > rpl_global_gtid_slave_state->update_state_hash(sub_id_list[i], &list[i], > NULL); > @@ -7639,7 +7639,7 @@ int Xid_log_event::do_apply_event(rpl_group_info *rgi) > rgi->gtid_pending= false; > > gtid= rgi->current_gtid; > - err= rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, true, > + err= rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, rgi, > false); > if (err) > { > diff --git a/sql/rpl_gtid.cc b/sql/rpl_gtid.cc > index 7b1acf17ef5..94944b5b3e5 100644 > --- a/sql/rpl_gtid.cc > +++ b/sql/rpl_gtid.cc > @@ -77,7 +77,7 @@ rpl_slave_state::record_and_update_gtid(THD *thd, rpl_group_info *rgi) > rgi->gtid_pending= false; > if (rgi->gtid_ignore_duplicate_state!=rpl_group_info::GTID_DUPLICATE_IGNORE) > { > - if (record_gtid(thd, &rgi->current_gtid, sub_id, false, false)) > + if (record_gtid(thd, &rgi->current_gtid, sub_id, NULL, false)) > DBUG_RETURN(1); > update_state_hash(sub_id, &rgi->current_gtid, rgi); > } > @@ -328,6 +328,8 @@ rpl_slave_state::update(uint32 domain_id, uint32 server_id, uint64 sub_id, > } > } > rgi->gtid_ignore_duplicate_state= rpl_group_info::GTID_DUPLICATE_NULL; > + > + rgi->pending_gtid_deletes_clear(); > } > > if (!(list_elem= (list_element *)my_malloc(sizeof(*list_elem), MYF(MY_WME)))) > @@ -377,15 +379,24 @@ int > rpl_slave_state::put_back_list(uint32 domain_id, list_element *list) > { > element *e; > + int err= 0; > + > + mysql_mutex_lock(&LOCK_slave_state); > if (!(e= (element *)my_hash_search(&hash, (const uchar *)&domain_id, 0))) > - return 1; > + { > + err= 1; > + goto end; > + } > while (list) > { > list_element *next= list->next; > e->add(list); > list= next; > } > - return 0; > + > +end: > + mysql_mutex_unlock(&LOCK_slave_state); > + return err; > } > > > @@ -468,12 +479,12 @@ gtid_check_rpl_slave_state_table(TABLE *table) > /* > Write a gtid to the replication slave state table. > > - Do it as part of the transaction, to get slave crash safety, or as a separate > - transaction if !in_transaction (eg. MyISAM or DDL). > - > gtid The global transaction id for this event group. > sub_id Value allocated within the sub_id when the event group was > read (sub_id must be consistent with commit order in master binlog). > + rgi rpl_group_info context, if we are recording the gtid transactionally > + as part of replicating a transactional event. NULL if called from > + outside of a replicated transaction. > > Note that caller must later ensure that the new gtid and sub_id is inserted > into the appropriate HASH element with rpl_slave_state.add(), so that it can > @@ -481,13 +492,13 @@ gtid_check_rpl_slave_state_table(TABLE *table) > */ > int > rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > - bool in_transaction, bool in_statement) > + rpl_group_info *rgi, bool in_statement) > { > TABLE_LIST tlist; > int err= 0; > bool table_opened= false; > TABLE *table; > - list_element *elist= 0, *next; > + list_element *elist= 0, *cur, *next; > element *elem; > ulonglong thd_saved_option= thd->variables.option_bits; > Query_tables_list lex_backup; > @@ -558,7 +569,7 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > thd->wsrep_ignore_table= true; > #endif > > - if (!in_transaction) > + if (!rgi) > { > DBUG_PRINT("info", ("resetting OPTION_BEGIN")); > thd->variables.option_bits&= > @@ -601,9 +612,9 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > if ((elist= elem->grab_list()) != NULL) > { > /* Delete any old stuff, but keep around the most recent one. */ > - list_element *cur= elist; > - uint64 best_sub_id= cur->sub_id; > + uint64 best_sub_id= elist->sub_id; > list_element **best_ptr_ptr= &elist; > + cur= elist; > while ((next= cur->next)) > { > if (next->sub_id > best_sub_id) > @@ -636,7 +647,8 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > table->file->print_error(err, MYF(0)); > goto end; > } > - while (elist) > + cur = elist; > + while (cur) > { > uchar key_buffer[4+8]; > > @@ -646,9 +658,9 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > /* `break' does not work inside DBUG_EXECUTE_IF */ > goto dbug_break; }); > > - next= elist->next; > + next= cur->next; > > - table->field[1]->store(elist->sub_id, true); > + table->field[1]->store(cur->sub_id, true); > /* domain_id is already set in table->record[0] from write_row() above. */ > key_copy(key_buffer, table->record[0], &table->key_info[0], 0, false); > if (table->file->ha_index_read_map(table->record[1], key_buffer, > @@ -662,8 +674,7 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > not want to endlessly error on the same element in case of table > corruption or such. > */ > - my_free(elist); > - elist= next; > + cur= next; > if (err) > break; > } > @@ -686,18 +697,31 @@ IF_DBUG(dbug_break:, ) > */ > if (elist) > { > - mysql_mutex_lock(&LOCK_slave_state); > put_back_list(gtid->domain_id, elist); > - mysql_mutex_unlock(&LOCK_slave_state); > + elist = 0; > } > > ha_rollback_trans(thd, FALSE); > } > close_thread_tables(thd); > - if (in_transaction) > + if (rgi) > + { > thd->mdl_context.release_statement_locks(); > + /* > + Save the list of old gtid entries we deleted. If this transaction > + fails later for some reason and is rolled back, the deletion of those > + entries will be rolled back as well, and we will need to put them back > + on the to-be-deleted list so we can re-do the deletion. Otherwise > + redundant rows in mysql.gtid_slave_pos may accumulate if transactions > + are rolled back and retried after record_gtid(). > + */ > + rgi->pending_gtid_deletes_save(gtid->domain_id, elist); > + } > else > + { > thd->mdl_context.release_transactional_locks(); > + rpl_group_info::pending_gtid_deletes_free(elist); > + } > } > thd->lex->restore_backup_query_tables_list(&lex_backup); > thd->variables.option_bits= thd_saved_option; > @@ -1080,7 +1104,7 @@ rpl_slave_state::load(THD *thd, char *state_from_master, size_t len, > > if (gtid_parser_helper(&state_from_master, end, &gtid) || > !(sub_id= next_sub_id(gtid.domain_id)) || > - record_gtid(thd, &gtid, sub_id, false, in_statement) || > + record_gtid(thd, &gtid, sub_id, NULL, in_statement) || > update(gtid.domain_id, gtid.server_id, sub_id, gtid.seq_no, NULL)) > return 1; > if (state_from_master == end) > diff --git a/sql/rpl_gtid.h b/sql/rpl_gtid.h > index 79d566bddbf..7bd639b768f 100644 > --- a/sql/rpl_gtid.h > +++ b/sql/rpl_gtid.h > @@ -182,7 +182,7 @@ struct rpl_slave_state > uint64 seq_no, rpl_group_info *rgi); > int truncate_state_table(THD *thd); > int record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > - bool in_transaction, bool in_statement); > + rpl_group_info *rgi, bool in_statement); > uint64 next_sub_id(uint32 domain_id); > int iterate(int (*cb)(rpl_gtid *, void *), void *data, > rpl_gtid *extra_gtids, uint32 num_extra, > diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc > index 64a1b535307..b35130c1505 100644 > --- a/sql/rpl_rli.cc > +++ b/sql/rpl_rli.cc > @@ -1680,6 +1680,7 @@ rpl_group_info::reinit(Relay_log_info *rli) > long_find_row_note_printed= false; > did_mark_start_commit= false; > gtid_ev_flags2= 0; > + pending_gtid_delete_list= NULL; > last_master_timestamp = 0; > gtid_ignore_duplicate_state= GTID_DUPLICATE_NULL; > speculation= SPECULATE_NO; > @@ -1804,6 +1805,12 @@ void rpl_group_info::cleanup_context(THD *thd, bool error) > erroneously update the GTID position. > */ > gtid_pending= false; > + > + /* > + Rollback will have undone any deletions of old rows we might have made > + in mysql.gtid_slave_pos. Put those rows back on the list to be deleted. > + */ > + pending_gtid_deletes_put_back(); > } > m_table_map.clear_tables(); > slave_close_thread_tables(thd); > @@ -2027,6 +2034,78 @@ rpl_group_info::unmark_start_commit() > } > > > +/* > + When record_gtid() has deleted any old rows from the table > + mysql.gtid_slave_pos as part of a replicated transaction, save the list of > + rows deleted here. > + > + If later the transaction fails (eg. optimistic parallel replication), the > + deletes will be undone when the transaction is rolled back. Then we can > + put back the list of rows into the rpl_global_gtid_slave_state, so that > + we can re-do the deletes and avoid accumulating old rows in the table. > +*/ > +void > +rpl_group_info::pending_gtid_deletes_save(uint32 domain_id, > + rpl_slave_state::list_element *list) > +{ > + /* > + We should never get to a state where we try to save a new pending list of > + gtid deletes while we still have an old one. But make sure we handle it > + anyway just in case, so we avoid leaving stray entries in the > + mysql.gtid_slave_pos table. > + */ > + DBUG_ASSERT(!pending_gtid_delete_list); > + if (unlikely(pending_gtid_delete_list)) > + pending_gtid_deletes_put_back(); > + > + pending_gtid_delete_list= list; > + pending_gtid_delete_list_domain= domain_id; > +} > + > + > +/* > + Take the list recorded by pending_gtid_deletes_save() and put it back into > + rpl_global_gtid_slave_state. This is needed if deletion of the rows was > + rolled back due to transaction failure. > +*/ > +void > +rpl_group_info::pending_gtid_deletes_put_back() > +{ > + if (pending_gtid_delete_list) > + { > + rpl_global_gtid_slave_state->put_back_list(pending_gtid_delete_list_domain, > + pending_gtid_delete_list); > + pending_gtid_delete_list= NULL; > + } > +} > + > + > +/* > + Free the list recorded by pending_gtid_deletes_save(). Done when the deletes > + in the list have been permanently committed. > +*/ > +void > +rpl_group_info::pending_gtid_deletes_clear() > +{ > + pending_gtid_deletes_free(pending_gtid_delete_list); > + pending_gtid_delete_list= NULL; > +} > + > + > +void > +rpl_group_info::pending_gtid_deletes_free(rpl_slave_state::list_element *list) > +{ > + rpl_slave_state::list_element *next; > + > + while (list) > + { > + next= list->next; > + my_free(list); > + list= next; > + } > +} > + > + > rpl_sql_thread_info::rpl_sql_thread_info(Rpl_filter *filter) > : rpl_filter(filter) > { > diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h > index 74d5b6fe416..b40a34a54e6 100644 > --- a/sql/rpl_rli.h > +++ b/sql/rpl_rli.h > @@ -676,6 +676,11 @@ struct rpl_group_info > /* Needs room for "Gtid D-S-N\x00". */ > char gtid_info_buf[5+10+1+10+1+20+1]; > > + /* List of not yet committed deletions in mysql.gtid_slave_pos. */ > + rpl_slave_state::list_element *pending_gtid_delete_list; > + /* Domain associated with pending_gtid_delete_list. */ > + uint32 pending_gtid_delete_list_domain; > + > /* > The timestamp, from the master, of the commit event. > Used to do delayed update of rli->last_master_timestamp, for getting > @@ -817,6 +822,12 @@ struct rpl_group_info > char *gtid_info(); > void unmark_start_commit(); > > + static void pending_gtid_deletes_free(rpl_slave_state::list_element *list); > + void pending_gtid_deletes_save(uint32 domain_id, > + rpl_slave_state::list_element *list); > + void pending_gtid_deletes_put_back(); > + void pending_gtid_deletes_clear(); > + > time_t get_row_stmt_start_timestamp() > { > return row_stmt_start_timestamp; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 5

[Maria-developers] SP memory usage
by jerome brauge 12 Oct '18

12 Oct '18

Hello Alexander, As stored procedures seems to be very memory aggressive, I found two possible optimizations to avoid having three strings containing the stored procedure code in memory. Probably, same optimizations may be applied to packages. I haven't add any tests because I think everything is covered by the existing tests. What do you think about ? In a second time, I think working on SP cache. Currently it' s a simple hash table and all the hash table is flushed as soon as the threshold (stored_program_cache) is reached. It's not very good. We could : - Use a more intelligent cache (LFU ?) - Add some performance counters Regards. Ce message et les pi?ces jointes sont confidentiels et ?tablis ? l'attention exclusive de ses destinataires. Toute utilisation ou diffusion, m?me partielle, non autoris?e est interdite. Tout message ?lectronique est susceptible d'alt?ration; CEGID d?cline donc toute responsabilit? au titre de ce message. Si vous n'?tes pas le destinataire de ce message, merci de le d?truire et d'avertir l'exp?diteur. This message and any attachments are confidential and intended solely for the addressees. Any unauthorized use or disclosure, either whole or partial is prohibited. E-mails are susceptible to alteration; CEGID shall therefore not be liable for the content of this message. If you are not the intended recipient of this message, please delete it and notify the sender.

1 0

[Maria-developers] [beginner] how can I assign an existing ticket to myself?
by Takashi Sasaki 12 Oct '18

12 Oct '18

Hello, I wanted to contribute something to MariaDB, so I signed up to Jira, but how can I assign an existing ticket to myself? [programming experience]: I have been programming for over 10 years but recently using Java, Python, Node.js at work and C ++ is a little played a few years ago. [knowledge of the MariaDB source] I just started reading the code recently and I am almost an amateur. [how much you know about using MySQL/MariaDB] I use MySQL for WordPress backend as a user and I use Oracle for work, so I have knowledge of general RDBMS, but I do not have deep knowledge specific to the products. Thanks, Takashi Sasaki

3 2

Re: [Maria-developers] [Commits] 1edec241473: MDEV-17137: Syntax errors with VIEW using MEDIAN
by Sergey Petrunia 11 Oct '18

11 Oct '18

Hi Varun, On Wed, Sep 12, 2018 at 06:13:23PM +0530, Varun wrote: > revision-id: 1edec241473a122553fa9a27be995f3aef52654e (mariadb-10.3.7-171-g1edec241473) > parent(s): 5a1868b58d26b286b6ad433096e7184895953311 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-09-12 18:10:16 +0530 > message: > > MDEV-17137: Syntax errors with VIEW using MEDIAN > > The syntax error happened because we had not implemented a different print for > percentile functions. The syntax is a bit different when we use percentile functions > as window functions in comparision to normal window functions. > Implemented a seperate print function for percentile functions > ... > diff --git a/sql/item_windowfunc.h b/sql/item_windowfunc.h > index b3e23748246..268182a2788 100644 > --- a/sql/item_windowfunc.h > +++ b/sql/item_windowfunc.h > @@ -1311,6 +1311,7 @@ class Item_window_func : public Item_func_or_sum > bool resolve_window_name(THD *thd); > > void print(String *str, enum_query_type query_type); > + void print_percentile_functions(String *str, enum_query_type query_type); > * Please make this function private * I think "print_percentile_functions" is a bad name as it implies multiple functions are printed. In fact, only one function is printed. How about "print_for_percentile_functions"? Ok to push after the above is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] bb2db687a05: 26 aug (MDEV-371 Unique indexes for blobs)
by Sergei Golubchik 10 Oct '18

10 Oct '18

Hi, Sachin! Could you please explain what you store in the frm file and how you modify keyinfo/keypart structures in memory? Needs to be tested with partitioned tables. Particularly with MDEV-14005. Tests were great and thourough, good job. It'd be useful to have at least some tests for innodb too, though. See other comments below. On Oct 08, Sachin Setiya wrote: > diff --git a/mysql-test/main/long_unique.result b/mysql-test/main/long_unique.result > new file mode 100644 > index 00000000000..1a06c230c72 > --- /dev/null > +++ b/mysql-test/main/long_unique.result > @@ -0,0 +1,1391 @@ > +#Structure of tests > +#First we will check all option for > +#table containing single unique column > +#table containing keys like unique(a,b,c,d) etc > +#then table containing 2 blob unique etc > +set @allowed_packet= @@max_allowed_packet; > +#table with single long blob column; > +create table t1(a blob unique); > +insert into t1 values(1),(2),(3),(56),('sachin'),('maria'),(123456789034567891),(null),(null),(123456789034567890); > +#table structure; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table t1 > +Non_unique 0 > +Key_name a > +Seq_in_index 1 > +Column_name a > +Collation A > +Cardinality NULL > +Sub_part NULL > +Packed NULL > +Null YES > +Index_type HASH_INDEX I'm not sure about it. One cannot write UNIQUE (a) USING HASH_INDEX in CREATE TABLE, right? May be just report HASH here? (and fix the parser to behave accordingly) > +Comment > +Index_comment > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 10 Deleted blocks: 0 > +Recordlength: 20 > + > +table description: > +Key Start Len Index Type > +1 12 8 multip. ulonglong NULL good idea > +select * from information_schema.columns where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +def test t1 a 1 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob UNI select,insert,update,references NEVER NULL > +select * from information_schema.statistics where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME NON_UNIQUE INDEX_SCHEMA INDEX_NAME SEQ_IN_INDEX COLUMN_NAME COLLATION CARDINALITY SUB_PART PACKED NULLABLE INDEX_TYPE COMMENT INDEX_COMMENT > +def test t1 0 test a 1 a A NULL NULL NULL YES HASH_INDEX > +select * from information_schema.key_column_usage where table_schema= 'test' and table_name= 't1'; > +CONSTRAINT_CATALOG def > +CONSTRAINT_SCHEMA test > +CONSTRAINT_NAME a > +TABLE_CATALOG def > +TABLE_SCHEMA test > +TABLE_NAME t1 > +COLUMN_NAME a > +ORDINAL_POSITION 1 > +POSITION_IN_UNIQUE_CONSTRAINT NULL > +REFERENCED_TABLE_SCHEMA NULL > +REFERENCED_TABLE_NAME NULL > +REFERENCED_COLUMN_NAME NULL > +# table select we should not be able to see db_row_hash_column; > +select * from t1; > +a > +1 > +2 > +3 > +56 > +sachin > +maria > +123456789034567891 > +NULL > +NULL > +123456789034567890 > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +#duplicate entry test; > +insert into t1 values(2); > +ERROR 23000: Duplicate entry '2' for key 'a' > +insert into t1 values('sachin'); > +ERROR 23000: Duplicate entry 'sachin' for key 'a' > +insert into t1 values(123456789034567891); > +ERROR 23000: Duplicate entry '123456789034567891' for key 'a' looks great :) > +select * from t1; > +a > +1 > +2 > +3 > +56 > +sachin > +maria > +123456789034567891 > +NULL > +NULL > +123456789034567890 > +insert into t1 values(11),(22),(33); > +insert into t1 values(12),(22); > +ERROR 23000: Duplicate entry '22' for key 'a' > +select * from t1; > +a > +1 > +2 > +3 > +56 > +sachin > +maria > +123456789034567891 > +NULL > +NULL > +123456789034567890 > +11 > +22 > +33 > +12 > +insert into t1 values(repeat('s',4000*10)),(repeat('s',4001*10)); > +insert into t1 values(repeat('m',4000*10)),(repeat('m',4000*10)); > +ERROR 23000: Duplicate entry 'mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm' for key 'a' > +insert into t1 values(repeat('m',4001)),(repeat('m',4002)); > +truncate table t1; > +insert into t1 values(1),(2),(3),(4),(5),(8),(7); > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 7 Deleted blocks: 0 > +Recordlength: 20 > + > +table description: > +Key Start Len Index Type > +1 12 8 multip. ulonglong NULL > +#now some alter commands; > +alter table t1 add column b int; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b int(11) YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)) I think there shouldn't be (65535) here > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(1,2); > +ERROR 23000: Duplicate entry '1' for key 'a' > +insert into t1 values(2,2); > +ERROR 23000: Duplicate entry '2' for key 'a' > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +#now try to change db_row_hash_1 column; > +alter table t1 drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 add column d int , add column e int , drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 modify column db_row_hash_1 int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column a int , add column b int, modify column db_row_hash_1 int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 change column db_row_hash_1 dsds int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column asd int, change column db_row_hash_1 dsds int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 drop column b , add column c int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +#now add some column with name db_row_hash; > +alter table t1 add column db_row_hash_1 int unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(45,1,55),(46,1,55); > +ERROR 23000: Duplicate entry '55' for key 'db_row_hash_1' > +alter table t1 add column db_row_hash_2 int, add column db_row_hash_3 int; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +c int(11) YES NULL > +db_row_hash_1 int(11) YES UNI NULL > +db_row_hash_2 int(11) YES NULL > +db_row_hash_3 int(11) YES NULL > +#this should also drop the unique index ; > +alter table t1 drop column a; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +#add column with unique index on blob ; > +alter table t1 add column a blob unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `a` blob DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +# try to change the blob unique column name; > +#this will change index to b tree; > +alter table t1 modify column a int ; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +alter table t1 add column clm blob unique; > +#try changing the name ; > +alter table t1 change column clm clm_changed blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + `clm_changed` blob DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `clm` (`clm_changed`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +t1 0 clm 1 clm_changed A NULL NULL NULL YES HASH_INDEX > +#now drop the unique key; > +alter table t1 drop key clm; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + `clm_changed` blob DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +drop table t1; > +create table t1 (a TEXT CHARSET latin1 COLLATE latin1_german2_ci unique); > +desc t1; > +Field Type Null Key Default Extra > +a text YES UNI NULL > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH_INDEX > +insert into t1 values ('ae'); > +insert into t1 values ('AE'); > +ERROR 23000: Duplicate entry 'AE' for key 'a' > +insert into t1 values ('Ä'); this should've failed, shouldn't it? > +drop table t1; > +create table t1 (a int primary key, b blob unique); > +desc t1; > +Field Type Null Key Default Extra > +a int(11) NO PRI NULL > +b blob YES UNI NULL > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 PRIMARY 1 a A 0 NULL NULL BTREE > +t1 0 b 1 b A NULL NULL NULL YES HASH_INDEX > +insert into t1 values(1,1),(2,2),(3,3); > +insert into t1 values(1,1); > +ERROR 23000: Duplicate entry '1' for key 'b' > +insert into t1 values(7,1); > +ERROR 23000: Duplicate entry '1' for key 'b' > +drop table t1; > +#table with multiple long blob column and varchar text column ; > +create table t1(a blob unique, b int , c blob unique , d text unique , e varchar(3000) unique); > +insert into t1 values(1,2,3,4,5),(2,11,22,33,44),(3111,222,333,444,555),(5611,2222,3333,4444,5555), > +('sachin',341,'fdf','gfgfgfg','hghgr'),('maria',345,'frter','dasd','utyuty'), > +(123456789034567891,353534,53453453453456,64565464564564,45435345345345), > +(123456789034567890,43545,657567567567,78967657567567,657567567567567676); > +#table structure; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b int(11) YES NULL > +c blob YES UNI NULL > +d text YES UNI NULL > +e varchar(3000) YES UNI NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `c` (`c`), > + UNIQUE KEY `d` (`d`), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH_INDEX > +t1 0 c 1 c A NULL NULL NULL YES HASH_INDEX > +t1 0 d 1 d A NULL NULL NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 8 Deleted blocks: 0 > +Recordlength: 3072 > + > +table description: > +Key Start Len Index Type > +1 3063 8 multip. ulonglong NULL > +2 3055 8 multip. ulonglong NULL > +3 3047 8 multip. ulonglong NULL > +4 3039 8 multip. ulonglong prefix NULL why "prefix" here? how is it different from others? > +select * from information_schema.columns where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +def test t1 a 1 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob UNI select,insert,update,references NEVER NULL > +def test t1 b 2 NULL YES int NULL NULL 10 0 NULL NULL NULL int(11) select,insert,update,references NEVER NULL > +def test t1 c 3 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob UNI select,insert,update,references NEVER NULL > +def test t1 d 4 NULL YES text 65535 65535 NULL NULL NULL latin1 latin1_swedish_ci text UNI select,insert,update,references NEVER NULL > +def test t1 e 5 NULL YES varchar 3000 3000 NULL NULL NULL latin1 latin1_swedish_ci varchar(3000) UNI select,insert,update,references NEVER NULL > +select * from information_schema.statistics where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME NON_UNIQUE INDEX_SCHEMA INDEX_NAME SEQ_IN_INDEX COLUMN_NAME COLLATION CARDINALITY SUB_PART PACKED NULLABLE INDEX_TYPE COMMENT INDEX_COMMENT > +def test t1 0 test a 1 a A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test c 1 c A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test d 1 d A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test e 1 e A NULL NULL NULL YES HASH_INDEX > +select * from information_schema.key_column_usage where table_schema= 'test' and table_name= 't1'; > +CONSTRAINT_CATALOG CONSTRAINT_SCHEMA CONSTRAINT_NAME TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION POSITION_IN_UNIQUE_CONSTRAINT REFERENCED_TABLE_SCHEMA REFERENCED_TABLE_NAME REFERENCED_COLUMN_NAME > +def test a def test t1 a 1 NULL NULL NULL NULL > +def test c def test t1 c 1 NULL NULL NULL NULL > +def test d def test t1 d 1 NULL NULL NULL NULL > +def test e def test t1 e 1 NULL NULL NULL NULL > +#table select we should not be able to see db_row_hash_1 column; > +select * from t1; > +a b c d e > +1 2 3 4 5 > +2 11 22 33 44 > +3111 222 333 444 555 > +5611 2222 3333 4444 5555 > +sachin 341 fdf gfgfgfg hghgr > +maria 345 frter dasd utyuty > +123456789034567891 353534 53453453453456 64565464564564 45435345345345 > +123456789034567890 43545 657567567567 78967657567567 657567567567567676 > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +select db_row_hash_2 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_2' in 'field list' > +select db_row_hash_3 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_3' in 'field list' > +#duplicate entry test; > +insert into t1 values(21,2,3,42,51); > +ERROR 23000: Duplicate entry '3' for key 'c' > +insert into t1 values('sachin',null,null,null,null); > +ERROR 23000: Duplicate entry 'sachin' for key 'a' > +insert into t1 values(1234567890345671890,4353451,6575675675617,789676575675617,657567567567567676); > +ERROR 23000: Duplicate entry '657567567567567676' for key 'e' > +select * from t1; > +a b c d e > +1 2 3 4 5 > +2 11 22 33 44 > +3111 222 333 444 555 > +5611 2222 3333 4444 5555 > +sachin 341 fdf gfgfgfg hghgr > +maria 345 frter dasd utyuty > +123456789034567891 353534 53453453453456 64565464564564 45435345345345 > +123456789034567890 43545 657567567567 78967657567567 657567567567567676 > +insert into t1 values(repeat('s',4000*10),100,repeat('s',4000*10),repeat('s',4000*10), > +repeat('s',400)),(repeat('s',4001*10),1000,repeat('s',4001*10),repeat('s',4001*10), > +repeat('s',2995)); > +insert into t1 values(repeat('m',4000*11),10,repeat('s',4000*11),repeat('s',4000*11),repeat('s',2995)); > +ERROR 23000: Duplicate entry 'ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss' for key 'e' > +truncate table t1; > +insert into t1 values(1,2,3,4,5),(2,11,22,33,44),(3111,222,333,444,555),(5611,2222,3333,4444,5555); > +#now some alter commands; > +alter table t1 add column f int; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b int(11) YES NULL > +c blob YES UNI NULL > +d text YES UNI NULL > +e varchar(3000) YES UNI NULL > +f int(11) YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `c` (`c`(65535)), > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +#unique key should not break; > +insert into t1 values(1,2,3,4,5,6); > +ERROR 23000: Duplicate entry '1' for key 'a' > +select db_row_hash_1 , db_row_hash_2, db_row_hash_3 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +#now try to change db_row_hash_1 column; > +alter table t1 drop column db_row_hash_1, drop column db_row_hash_2, drop column db_row_hash_3; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 add column dg int , add column ef int , drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 modify column db_row_hash_1 int , modify column db_row_hash_2 int, modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column ar int , add column rb int, modify column db_row_hash_1 int , modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 change column db_row_hash_1 dsds int , change column db_row_hash_2 dfdf int , change column db_row_hash_3 gdfg int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column asd int, drop column a, change column db_row_hash_1 dsds int, change db_row_hash_3 fdfdfd int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 drop column b , add column g int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `c` (`c`(65535)), > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +#now add some column with name db_row_hash; > +alter table t1 add column db_row_hash_1 int unique; > +alter table t1 add column db_row_hash_2 int unique; > +alter table t1 add column db_row_hash_3 int unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `c` (`c`(65535)), > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `db_row_hash_3` (`db_row_hash_3`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +alter table t1 add column db_row_hash_7 int, add column db_row_hash_5 int , add column db_row_hash_4 int ; > +alter table t1 drop column db_row_hash_7,drop column db_row_hash_3, drop column db_row_hash_4; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +c blob YES UNI NULL > +d text YES UNI NULL > +e varchar(3000) YES UNI NULL > +f int(11) YES NULL > +g int(11) YES NULL > +db_row_hash_1 int(11) YES UNI NULL > +db_row_hash_2 int(11) YES UNI NULL > +db_row_hash_5 int(11) YES NULL > +#this show now break anything; "this should not" ? > +insert into t1 values(1,2,3,4,5,6,23,5,6); > +ERROR 23000: Duplicate entry '1' for key 'a' > +#this should also drop the unique index; > +alter table t1 drop column a, drop column c; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +#add column with unique index on blob; > +alter table t1 add column a blob unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` blob DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES HASH_INDEX > +#try to change the blob unique column name; > +#this will change index to b tree; > +alter table t1 modify column a int , modify column e int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +alter table t1 add column clm1 blob unique,add column clm2 blob unique; > +#try changing the name; > +alter table t1 change column clm1 clm_changed1 blob, change column clm2 clm_changed2 blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + `clm_changed1` blob DEFAULT NULL, > + `clm_changed2` blob DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `clm1` (`clm_changed1`), > + UNIQUE KEY `clm2` (`clm_changed2`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +t1 0 clm1 1 clm_changed1 A NULL NULL NULL YES HASH_INDEX > +t1 0 clm2 1 clm_changed2 A NULL NULL NULL YES HASH_INDEX > +#now drop the unique key; > +alter table t1 drop key clm1, drop key clm2; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + `clm_changed1` blob DEFAULT NULL, > + `clm_changed2` blob DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +drop table t1; > +#now the table with key on multiple columns; the ultimate test; > +create table t1(a blob, b int , c varchar(2000) , d text , e varchar(3000) , f longblob , g int , h text , > +unique(a,b,c), unique(c,d,e),unique(e,f,g,h), unique(b,d,g,h)); > +insert into t1 values(1,1,1,1,1,1,1,1),(2,2,2,2,2,2,2,2),(3,3,3,3,3,3,3,3),(4,4,4,4,4,4,4,4),(5,5,5,5,5,5,5,5), > +('maria',6,'maria','maria','maria','maria',6,'maria'),('mariadb',7,'mariadb','mariadb','mariadb','mariadb',8,'mariadb') > +,(null,null,null,null,null,null,null,null),(null,null,null,null,null,null,null,null); > +#table structure; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES MUL NULL > +b int(11) YES MUL NULL > +c varchar(2000) YES MUL NULL > +d text YES NULL > +e varchar(3000) YES MUL NULL > +f longblob YES NULL > +g int(11) YES NULL > +h text YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`,`e`), > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`), > + UNIQUE KEY `b` (`b`,`d`,`g`,`h`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH_INDEX > +t1 0 a 2 b A NULL NULL NULL YES HASH_INDEX > +t1 0 a 3 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 1 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 NULL NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 1 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 2 f A 0 NULL NULL YES HASH_INDEX > +t1 0 e 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 e 4 h A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 NULL NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 NULL NULL YES HASH_INDEX > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 9 Deleted blocks: 0 > +Recordlength: 5092 > + > +table description: > +Key Start Len Index Type > +1 5081 8 multip. ulonglong prefix NULL > +2 5073 8 multip. ulonglong prefix NULL > +3 5065 8 multip. ulonglong prefix NULL > +4 5057 8 multip. ulonglong NULL > +select * from information_schema.columns where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +def test t1 a 1 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob MUL select,insert,update,references NEVER NULL > +def test t1 b 2 NULL YES int NULL NULL 10 0 NULL NULL NULL int(11) MUL select,insert,update,references NEVER NULL > +def test t1 c 3 NULL YES varchar 2000 2000 NULL NULL NULL latin1 latin1_swedish_ci varchar(2000) MUL select,insert,update,references NEVER NULL > +def test t1 d 4 NULL YES text 65535 65535 NULL NULL NULL latin1 latin1_swedish_ci text select,insert,update,references NEVER NULL > +def test t1 e 5 NULL YES varchar 3000 3000 NULL NULL NULL latin1 latin1_swedish_ci varchar(3000) MUL select,insert,update,references NEVER NULL > +def test t1 f 6 NULL YES longblob 4294967295 4294967295 NULL NULL NULL NULL NULL longblob select,insert,update,references NEVER NULL > +def test t1 g 7 NULL YES int NULL NULL 10 0 NULL NULL NULL int(11) select,insert,update,references NEVER NULL > +def test t1 h 8 NULL YES text 65535 65535 NULL NULL NULL latin1 latin1_swedish_ci text select,insert,update,references NEVER NULL > +select * from information_schema.statistics where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME NON_UNIQUE INDEX_SCHEMA INDEX_NAME SEQ_IN_INDEX COLUMN_NAME COLLATION CARDINALITY SUB_PART PACKED NULLABLE INDEX_TYPE COMMENT INDEX_COMMENT > +def test t1 0 test a 1 a A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test a 2 b A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test a 3 c A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test c 1 c A NULL NULL NULL YES HASH_INDEX > +def test t1 0 test c 2 d A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test c 3 e A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test e 1 e A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test e 2 f A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test e 3 g A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test e 4 h A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test b 1 b A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test b 2 d A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test b 3 g A 0 NULL NULL YES HASH_INDEX > +def test t1 0 test b 4 h A 0 NULL NULL YES HASH_INDEX > +select * from information_schema.key_column_usage where table_schema= 'test' and table_name= 't1'; > +CONSTRAINT_CATALOG CONSTRAINT_SCHEMA CONSTRAINT_NAME TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION POSITION_IN_UNIQUE_CONSTRAINT REFERENCED_TABLE_SCHEMA REFERENCED_TABLE_NAME REFERENCED_COLUMN_NAME > +def test a def test t1 a 1 NULL NULL NULL NULL > +def test a def test t1 b 2 NULL NULL NULL NULL > +def test a def test t1 c 3 NULL NULL NULL NULL > +def test c def test t1 c 1 NULL NULL NULL NULL > +def test c def test t1 d 2 NULL NULL NULL NULL > +def test c def test t1 e 3 NULL NULL NULL NULL > +def test e def test t1 e 1 NULL NULL NULL NULL > +def test e def test t1 f 2 NULL NULL NULL NULL > +def test e def test t1 g 3 NULL NULL NULL NULL > +def test e def test t1 h 4 NULL NULL NULL NULL > +def test b def test t1 b 1 NULL NULL NULL NULL > +def test b def test t1 d 2 NULL NULL NULL NULL > +def test b def test t1 g 3 NULL NULL NULL NULL > +def test b def test t1 h 4 NULL NULL NULL NULL > +# table select we should not be able to see db_row_hash_1 column; > +select * from t1; > +a b c d e f g h > +1 1 1 1 1 1 1 1 > +2 2 2 2 2 2 2 2 > +3 3 3 3 3 3 3 3 > +4 4 4 4 4 4 4 4 > +5 5 5 5 5 5 5 5 > +maria 6 maria maria maria maria 6 maria > +mariadb 7 mariadb mariadb mariadb mariadb 8 mariadb > +NULL NULL NULL NULL NULL NULL NULL NULL > +NULL NULL NULL NULL NULL NULL NULL NULL > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +select db_row_hash_2 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_2' in 'field list' > +select db_row_hash_3 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_3' in 'field list' > +#duplicate entry test; > +#duplicate keys entry; > +insert into t1 values(1,1,1,0,0,0,0,0); > +ERROR 23000: Duplicate entry '1-1-1' for key 'a' > +insert into t1 values(0,0,1,1,1,0,0,0); > +ERROR 23000: Duplicate entry '1-1-1' for key 'c' > +insert into t1 values(0,0,0,0,1,1,1,1); > +ERROR 23000: Duplicate entry '1-1-1-1' for key 'e' > +insert into t1 values(1,1,1,1,1,0,0,0); > +ERROR 23000: Duplicate entry '1-1-1' for key 'a' > +insert into t1 values(0,0,0,0,1,1,1,1); > +ERROR 23000: Duplicate entry '1-1-1-1' for key 'e' > +insert into t1 values(1,1,1,1,1,1,1,1); > +ERROR 23000: Duplicate entry '1-1-1' for key 'a' > +select db_row_hash_1,db_row_hash_2,db_row_hash_3,db_row_hash_4,db_row_hash_5 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +alter table t1 drop column db_row_hash_1, drop column db_row_hash_2, drop column db_row_hash_3; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 add column dg int , add column ef int , drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 modify column db_row_hash_1 int , modify column db_row_hash_2 int, modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column ar int , add column rb int, modify column db_row_hash_1 int , modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 change column db_row_hash_1 dsds int , change column db_row_hash_2 dfdf int , change column db_row_hash_3 gdfg int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column asd int, drop column a, change column db_row_hash_1 dsds int, change db_row_hash_3 fdfdfd int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`,`e`), > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`), > + UNIQUE KEY `b` (`b`,`d`,`g`,`h`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +# add column named db_row_hash_*; > +alter table t1 add column db_row_hash_7 int , add column db_row_hash_5 int, > +add column db_row_hash_1 int, add column db_row_hash_2 int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + `db_row_hash_7` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 2 b A NULL NULL NULL YES HASH_INDEX > +t1 0 a 3 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 1 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 1 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 2 f A 0 NULL NULL YES HASH_INDEX > +t1 0 e 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 e 4 h A 0 65535 NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +alter table t1 drop column db_row_hash_7 , drop column db_row_hash_5 , > +drop column db_row_hash_1, drop column db_row_hash_2 ; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 2 b A NULL NULL NULL YES HASH_INDEX > +t1 0 a 3 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 1 c A NULL NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 1 e A 0 NULL NULL YES HASH_INDEX > +t1 0 e 2 f A 0 NULL NULL YES HASH_INDEX > +t1 0 e 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 e 4 h A 0 65535 NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +#try to change column names; > +alter table t1 change column a aa blob , change column b bb blob , change column d dd blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `aa` blob DEFAULT NULL, > + `bb` blob DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `dd` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `a` (`aa`(65535),`bb`,`c`), > + UNIQUE KEY `c` (`c`,`dd`(65535),`e`), > + UNIQUE KEY `b` (`bb`,`dd`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 e 2 f A NULL NULL NULL YES HASH_INDEX > +t1 0 e 3 g A NULL NULL NULL YES HASH_INDEX > +t1 0 e 4 h A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 1 aa A 0 65535 NULL YES HASH_INDEX > +t1 0 a 2 bb A 0 NULL NULL YES HASH_INDEX > +t1 0 a 3 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 1 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 2 dd A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 bb A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 dd A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +alter table t1 change column aa a blob , change column bb b blob , change column dd d blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `a` (`a`(65535),`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 e 2 f A NULL NULL NULL YES HASH_INDEX > +t1 0 e 3 g A NULL NULL NULL YES HASH_INDEX > +t1 0 e 4 h A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 1 a A 0 65535 NULL YES HASH_INDEX > +t1 0 a 2 b A 0 NULL NULL YES HASH_INDEX > +t1 0 a 3 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 1 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +#now we will change the data type to int and varchar limit so that we no longer require hash_index; > +#on key a_b_c; > +alter table t1 modify column a varchar(20) , modify column b varchar(20) , modify column c varchar(20); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` varchar(20) DEFAULT NULL, > + `b` varchar(20) DEFAULT NULL, > + `c` varchar(20) DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 e 2 f A NULL NULL NULL YES HASH_INDEX > +t1 0 e 3 g A NULL NULL NULL YES HASH_INDEX > +t1 0 e 4 h A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +t1 0 a 2 b A NULL NULL NULL YES BTREE > +t1 0 a 3 c A 0 NULL NULL YES BTREE > +t1 0 c 1 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +#change it back; > +alter table t1 modify column a blob , modify column b blob , modify column c blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 e 2 f A NULL NULL NULL YES HASH_INDEX > +t1 0 e 3 g A NULL NULL NULL YES HASH_INDEX > +t1 0 e 4 h A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 1 a A 0 NULL NULL YES HASH_INDEX > +t1 0 a 2 b A 0 NULL NULL YES HASH_INDEX > +t1 0 a 3 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 1 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +#try to delete blob column in unique; > +truncate table t1; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`(65535)), > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 e 1 e A NULL NULL NULL YES HASH_INDEX > +t1 0 e 2 f A NULL NULL NULL YES HASH_INDEX > +t1 0 e 3 g A NULL NULL NULL YES HASH_INDEX > +t1 0 e 4 h A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 1 a A 0 NULL NULL YES HASH_INDEX > +t1 0 a 2 b A 0 NULL NULL YES HASH_INDEX > +t1 0 a 3 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 1 c A 0 NULL NULL YES HASH_INDEX > +t1 0 c 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 c 3 e A 0 NULL NULL YES HASH_INDEX > +t1 0 b 1 b A 0 NULL NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +#now try to delete keys; > +alter table t1 drop key c, drop key e; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`b`(65535),`c`(65535)), > + UNIQUE KEY `b` (`b`(65535),`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 2 b A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 3 c A 0 65535 NULL YES HASH_INDEX > +t1 0 b 1 b A 0 65535 NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 b 3 g A 0 NULL NULL YES HASH_INDEX > +t1 0 b 4 h A 0 65535 NULL YES HASH_INDEX > +drop table t1; > +#now alter table containing some data basically some tests with ignore; > +create table t1 (a blob); > +insert into t1 values(1),(2),(3); > +#normal alter table; > +alter table t1 add unique key(a); > +alter table t1 drop key a; > +truncate table t1; > +insert into t1 values(1),(1),(2),(2),(3); > +alter table t1 add unique key(a); > +ERROR 23000: Duplicate entry '1' for key 'a' > +alter ignore table t1 add unique key(a); > +select * from t1; > +a > +1 > +2 > +3 > +insert into t1 values(1); > +ERROR 23000: Duplicate entry '1' for key 'a' > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH_INDEX > +drop table t1; > +#Now with multiple keys; > +create table t1(a blob , b blob, c blob , d blob , e int); > +insert into t1 values (1,1,1,1,1); > +insert into t1 values (1,1,1,1,1); > +insert into t1 values (2,1,1,1,1); > +insert into t1 values (2,2,2,2,2); > +insert into t1 values (3,3,4,4,4); > +insert into t1 values (4,4,4,4,4); > +alter table t1 add unique key(a,c), add unique key(b,d), add unique key(e); > +ERROR 23000: Duplicate entry '1-1' for key 'a' > +alter ignore table t1 add unique key(a,c), add unique key(b,d), add unique key(e); > +select * from t1; > +a b c d e > +1 1 1 1 1 > +2 2 2 2 2 > +3 3 4 4 4 > +insert into t1 values (1,12,1,13,14); > +ERROR 23000: Duplicate entry '1-1' for key 'a' > +insert into t1 values (12,1,14,1,14); > +ERROR 23000: Duplicate entry '1-1' for key 'b' > +insert into t1 values (13,12,13,14,4); > +ERROR 23000: Duplicate entry '4' for key 'e' > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`c`(65535)), > + UNIQUE KEY `b` (`b`(65535),`d`(65535)), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH_INDEX > +t1 0 a 2 c A NULL 65535 NULL YES HASH_INDEX > +t1 0 b 1 b A NULL 65535 NULL YES HASH_INDEX > +t1 0 b 2 d A 0 65535 NULL YES HASH_INDEX > +t1 0 e 1 e A 0 NULL NULL YES BTREE > +drop table t1; > +#visibility of db_row_hash > +create table t1 (a blob unique , b blob unique); > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b blob YES UNI NULL > +insert into t1 values(1,19); > +insert into t1 values(2,29); > +insert into t1 values(3,39); > +insert into t1 values(4,49); > +create table t2 (DB_ROW_HASH_1 int, DB_ROW_HASH_2 int); > +insert into t2 values(11,1); > +insert into t2 values(22,2); > +insert into t2 values(33,3); > +insert into t2 values(44,4); > +select * from t1; > +a b > +1 19 > +2 29 > +3 39 > +4 49 > +select * from t2; > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +22 2 > +33 3 > +44 4 > +select DB_ROW_HASH_1, DB_ROW_HASH_2 from t1; > +ERROR 42S22: Unknown column 'DB_ROW_HASH_1' in 'field list' > +#bug > +select DB_ROW_HASH_1, DB_ROW_HASH_2 from t1,t2; > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +11 1 > +11 1 > +11 1 > +22 2 > +22 2 > +22 2 > +22 2 > +33 3 > +33 3 > +33 3 > +33 3 > +44 4 > +44 4 > +44 4 > +44 4 > +select * from t1 where DB_ROW_HASH_1 in (select DB_ROW_HASH_1 from t2); > +ERROR 42S22: Unknown column 'DB_ROW_HASH_1' in 'IN/ALL/ANY subquery' > +select DB_ROW_HASH_1, DB_ROW_HASH_2 from t1,t2 where DB_ROW_HASH_1 in (select DB_ROW_HASH_1 from t2); > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +22 2 > +33 3 > +44 4 > +11 1 > +22 2 > +33 3 > +44 4 > +11 1 > +22 2 > +33 3 > +44 4 > +11 1 > +22 2 > +33 3 > +44 4 > +select * from t2 where DB_ROW_HASH_1 in (select DB_ROW_HASH_1 from t1); > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +22 2 > +33 3 > +44 4 > +select DB_ROW_HASH_1 from t1,t2 where t1.DB_ROW_HASH_1 = t2.DB_ROW_HASH_2; > +ERROR 42S22: Unknown column 't1.DB_ROW_HASH_1' in 'where clause' > +select DB_ROW_HASH_1 from t1 inner join t2 on t1.a = t2.DB_ROW_HASH_2; > +DB_ROW_HASH_1 > +11 > +22 > +33 > +44 > +drop table t1,t2; > +#very long blob entry; > +SET @@GLOBAL.max_allowed_packet=67108864; > +connect 'newcon', localhost, root,,; > +connection newcon; > +show variables like 'max_allowed_packet'; > +Variable_name Value > +max_allowed_packet 67108864 > +create table t1(a longblob unique, b longblob , c longblob , unique(b,c)); > +desc t1; > +Field Type Null Key Default Extra > +a longblob YES UNI NULL > +b longblob YES MUL NULL > +c longblob YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` longblob DEFAULT NULL, > + `b` longblob DEFAULT NULL, > + `c` longblob DEFAULT NULL, > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `b` (`b`,`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH_INDEX > +t1 0 b 1 b A NULL NULL NULL YES HASH_INDEX > +t1 0 b 2 c A 0 NULL NULL YES HASH_INDEX > +insert into t1 values(concat(repeat('sachin',10000000),'1'),concat(repeat('sachin',10000000),'1'), > +concat(repeat('sachin',10000000),'1')); > +insert into t1 values(concat(repeat('sachin',10000000),'2'),concat(repeat('sachin',10000000),'2'), > +concat(repeat('sachin',10000000),'1')); > +insert into t1 values(concat(repeat('sachin',10000000),'2'),concat(repeat('sachin',10000000),'2'), > +concat(repeat('sachin',10000000),'4')); > +ERROR 23000: Duplicate entry 'sachinsachinsachinsachinsachinsachinsachinsachinsachinsachinsach' for key 'a' > +insert into t1 values(concat(repeat('sachin',10000000),'3'),concat(repeat('sachin',10000000),'1'), > +concat(repeat('sachin',10000000),'1')); > +ERROR 23000: Duplicate entry 'sachinsachinsachinsachinsachinsachinsachinsachinsachinsachinsach' for key 'b' > +drop table t1; > +#long key unique with different key length > +create table t1(a blob, unique(a(3000))); > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 3000 NULL YES HASH_INDEX > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + UNIQUE KEY `a` (`a`(3000)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 value(concat(repeat('s',3000),'1')); > +insert into t1 value(concat(repeat('s',3000),'2')); > +ERROR 23000: Duplicate entry 'ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss' for key 'a' > +insert into t1 value(concat(repeat('a',3000),'2')); > +drop table t1; > +create table t1(a varchar(4000), b longblob , c varchar(5000), d longblob, > +unique(a(3500), b(1000000)), unique(c(4500), d(10000000))); > +desc t1; > +Field Type Null Key Default Extra > +a varchar(4000) YES MUL NULL > +b longblob YES NULL > +c varchar(5000) YES MUL NULL > +d longblob YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` varchar(4000) DEFAULT NULL, > + `b` longblob DEFAULT NULL, > + `c` varchar(5000) DEFAULT NULL, > + `d` longblob DEFAULT NULL, > + UNIQUE KEY `a` (`a`(3500),`b`), > + UNIQUE KEY `c` (`c`(4500),`d`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 3500 NULL YES HASH_INDEX > +t1 0 a 2 b A NULL NULL NULL YES HASH_INDEX > +t1 0 c 1 c A 0 4500 NULL YES HASH_INDEX > +t1 0 c 2 d A 0 NULL NULL YES HASH_INDEX > +drop table t1; > +disconnect newcon; > +connection default; > +SET @@GLOBAL.max_allowed_packet=4194304; > +#ext bug > +create table t1(a int primary key, b blob unique, c int, d blob , index(c)); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) NOT NULL, > + `b` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + `d` blob DEFAULT NULL, > + PRIMARY KEY (`a`), > + UNIQUE KEY `b` (`b`), > + KEY `c` (`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(1,23,1,33); > +insert into t1 values(2,23,1,33); > +ERROR 23000: Duplicate entry '23' for key 'b' > +drop table t1; > +create table t2 (a blob unique , c int , index(c)); > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `a` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`), > + KEY `c` (`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t2 values(1,1); > +insert into t2 values(2,1); > +drop table t2; > +#not null test //todo solve warnings > +create table t1(a blob unique not null); > +desc t1; > +Field Type Null Key Default Extra > +a blob NO PRI NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob NOT NULL, > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(1); > +insert into t1 values(3); > +insert into t1 values(1); > +ERROR 23000: Duplicate entry '1' for key 'a' > +drop table t1; > +create table t1(a int primary key, b blob unique , c blob unique not null); > +insert into t1 values(1,1,1); > +insert into t1 values(2,1,2); > +ERROR 23000: Duplicate entry '1' for key 'b' > +insert into t1 values(3,3,1); > +ERROR 23000: Duplicate entry '1' for key 'c' > +drop table t1; > +create table t1 (a blob unique not null, b blob not null, c blob not null, unique(b,c)); > +desc t1; > +Field Type Null Key Default Extra > +a blob NO PRI NULL > +b blob NO MUL NULL > +c blob NO NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob NOT NULL, > + `b` blob NOT NULL, > + `c` blob NOT NULL, > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `b` (`b`,`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values (1, 2, 3); > +insert into t1 values (2, 1, 3); > +insert into t1 values (2, 1, 3); > +ERROR 23000: Duplicate entry '2' for key 'a' > +drop table t1; > +set @@GLOBAL.max_allowed_packet= @allowed_packet; > diff --git a/sql/field.h b/sql/field.h > index b6f28808e2e..13eea28fe8a 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -539,6 +539,7 @@ class Virtual_column_info: public Sql_alloc > Item *expr; > LEX_CSTRING name; /* Name of constraint */ > uint flags; > + LEX_CSTRING hash_expr; It is not a good idea to put your hash-unique-constraint specific data into a common Virtual_column_info. Luckily, you don't need it. Just like "current_timestamp()" expression is generated on the fly as needed in parse_vcol_defs(), you can generate the hash expression there too, no need to do it in TABLE_SHARE::init_from_binary_frm_image(). So, just remove hash_expr. > > Virtual_column_info() > : vcol_type((enum_vcol_info_type)VCOL_TYPE_NONE), > @@ -681,7 +688,7 @@ class Field: public Value_source > GEOM_MULTIPOINT = 4, GEOM_MULTILINESTRING = 5, GEOM_MULTIPOLYGON = 6, > GEOM_GEOMETRYCOLLECTION = 7 > }; > - enum imagetype { itRAW, itMBR}; > + enum imagetype { itRAW, itMBR, itHASH}; itHASH isn't used anywhere in your patch :) > > utype unireg_check; > uint32 field_length; // Length of field > diff --git a/sql/field.cc b/sql/field.cc > index dc854826ed6..f75f52141e3 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -9822,7 +9822,7 @@ int Field_bit::key_cmp(const uchar *str, uint length) > } > > > -int Field_bit::cmp_offset(uint row_offset) > +int Field_bit::cmp_offset(long row_offset) may be my_ptrdiff_t ? > { > if (bit_len) > { > diff --git a/sql/item_func.h b/sql/item_func.h > index 3b6cb4ceeac..0391de0c5ad 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -803,6 +803,19 @@ class Item_long_func: public Item_int_func > }; > > > +class Item_func_hash: public Item_int_func > +{ > +public: > + Item_func_hash(THD *thd, List<Item> &item): Item_int_func(thd, item) > + {} > + longlong val_int(); > + void fix_length_and_dec(); > + const Type_handler *type_handler() const { return &type_handler_long; } > + Item *get_copy(THD *thd) > + { return get_item_copy<Item_func_hash>(thd, this); } > + const char *func_name() const { return "HASH"; } is it a user visible function? Can one write SELECT HASH(a) FROM t1 ? looks like yes, so 1. please add tests for it 2. we need to document what hash function, exactly, is used. 3. May be better to call it MARIADB_HASH? to highlight its internal-use nature and a home-baked hash function? And to reduce a chance for a collision with user defined functions. 4. An even better approach would be not to create any user visible function at all. Currently unpack_vcol_info_from_frm() calls the parser and then initializes vcol_info and calls fix_fields. If you split it in two you can do `new Item_func_hash` manually and skip the first part of unpack_vcol_info_from_frm(). And you won't need HASH to be user-visible. As a minor optimization, the same can be done for "current_timestamp()" > +}; > + > class Item_longlong_func: public Item_int_func > { > public: > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 4923c628bf9..c620ea23165 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -2300,7 +2300,7 @@ int show_create_table(THD *thd, TABLE_LIST *table_list, String *packet, > */ > packet->append(STRING_WITH_LEN("PRIMARY KEY")); > } > - else if (key_info->flags & HA_NOSAME) > + else if (key_info->flags & HA_NOSAME || key_info->flags & HA_LONG_UNIQUE_HASH) Is it needed? As far as I can see, you always set HA_NOSAME whenever you set HA_LONG_UNIQUE_HASH. > packet->append(STRING_WITH_LEN("UNIQUE KEY ")); > else if (key_info->flags & HA_FULLTEXT) > packet->append(STRING_WITH_LEN("FULLTEXT KEY ")); > diff --git a/include/my_base.h b/include/my_base.h > index c36072c0bfa..ba17f8d80dc 100644 > --- a/include/my_base.h > +++ b/include/my_base.h > @@ -290,6 +291,11 @@ enum ha_base_keytype { > #define HA_KEY_HAS_PART_KEY_SEG 65536 > /* Internal Flag Can be calcaluted */ > #define HA_INVISIBLE_KEY 2<<18 > +/* > + Some more flags for keys these are not stored in > + frm it is calculated on the fly in init_from_binary_frm_image > +*/ 1. I don't think you need to make the same comment more verbose every time you repeat it :) Just say, like #define HA_INVISIBLE_KEY 2<<18 /* this is calculated too */ #define HA_LONG_UNIQUE_HASH 2<<19 /* this is calculated too */ 2. Do you need a separate flag for that, just HA_INVISIBLE_KEY isn't enough? > +#define HA_LONG_UNIQUE_HASH 2<<19 > /* Automatic bits in key-flag */ > > #define HA_SPACE_PACK_USED 4 /* Test for if SPACE_PACK used */ > @@ -317,6 +323,13 @@ enum ha_base_keytype { > #define HA_BIT_PART 1024 > #define HA_CAN_MEMCMP 2048 /* internal, never stored in frm */ > > +/* > + Used for key parts whole length is greater then > file->max_key_part_length > + Only used for HA_LONG_UNIQUE_HASH keys > +*/ //TODO a better name ?? > +#define HA_HASH_KEY_PART_FLAG 4096 > +/* Field need to be frees externally */ > +#define HA_FIELD_EX_FREED 8192 Okay, this is a weird name, agree. But it is not used anywere, so there's an easy solution :) > /* optionbits for database */ > #define HA_OPTION_PACK_RECORD 1U > #define HA_OPTION_PACK_KEYS 2U > diff --git a/sql/table.h b/sql/table.h > index 785fd9f3427..9ed7fcd7beb 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -346,9 +346,39 @@ enum field_visibility_t { > INVISIBLE_FULL > }; > > -#define INVISIBLE_MAX_BITS 3 > +#define INVISIBLE_MAX_BITS 3 > +/* We will store the info into 3rd bit if field is hash field */ > +#define HASH_FIELD_MASK 15 > +#define HASH_FIELD_MASK_SHIFT 4 > +#define HA_HASH_FIELD_LENGTH 8 > +#define HA_HASH_KEY_LENGTH_WITHOUT_NULL 8 > +#define HA_HASH_KEY_LENGTH_WITH_NULL 9 > +//TODO is this correct ? how about size of char ptr on 32/16 bit machine? I don't understand that, sorry > +#define HA_HASH_KEY_PART_LENGTH 4 + 8 // 4 for length , 8 for portable size of char ptr you almost always should use parentheses in macro expressions > > +const LEX_CSTRING ha_hash_str {STRING_WITH_LEN("HASH")}; ??? you create a copy of ha_hash_str symbol in every .o file that included table.h? > > + > +int find_field_pos_in_hash(Item *hash_item, const char * field_name); > + > +int fields_in_hash_str(Item *hash_item); > + > +Field * field_ptr_in_hash_str(Item *hash_item, int index); > + > +int get_key_part_length(KEY *keyinfo, int index); > + > +void calc_hash_for_unique(ulong &nr1, ulong &nr2, String *str); > + > +void create_update_handler(THD *thd, TABLE *table); > + > +void delete_update_handler(THD *thd, TABLE *table); > + > +void setup_table_hash(TABLE *table); > + > +void re_setup_table(TABLE *table); > + > +int get_hash_key(THD *thd, TABLE *table, handler *h, uint key_index, uchar *rec_buf, > + uchar *key_buff); > /** > Category of table found in the table share. > */ > @@ -1094,6 +1124,17 @@ struct TABLE > THD *in_use; /* Which thread uses this */ > > uchar *record[3]; /* Pointer to records */ > + /* record buf to resolve hash collisions for long UNIQUE constraints */ > + uchar *check_unique_buf; > + handler *update_handler; /* Handler used in case of update */ > + /* > + In the case of write row for long unique we are unable of find > + Whick key is voilated. Because we in case of duplicate hash we never reach > + handler write_row function. So print_error will always print that > + key 0 is voilated. We store which key is voilated in this variable > + by default this should be initialized to -1 typos :) "unable to find", "which", "violated" > + */ > + int dupp_hash_key; > uchar *write_row_record; /* Used as optimisation in > THD::write_row */ > uchar *insert_values; /* used by INSERT ... UPDATE */ > @@ -2898,6 +2939,7 @@ void append_unescaped(String *res, const char *pos, size_t length); > void prepare_frm_header(THD *thd, uint reclength, uchar *fileinfo, > HA_CREATE_INFO *create_info, uint keys, KEY *key_info); > const char *fn_frm_ext(const char *name); > +void calc_hash_for_unique(ulong &nr1, ulong &nr2, String *str); you have this prototype twice in table.h > > /* Check that the integer is in the internal */ > static inline int set_zone(int nr,int min_zone,int max_zone) > diff --git a/sql/table.cc b/sql/table.cc > index 73b1a6bd9b2..8cd4db2844d 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -747,7 +748,13 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > if (i == 0) > { > ext_key_parts+= (share->use_ext_keys ? first_keyinfo->user_defined_key_parts*(keys-1) : 0); > + /* > + Some keys can be HA_LONG_UNIQUE_HASH , but we do not know at this point , > + how many ?, but will always be less than or equal to total num of > + keys. Each HA_LONG_UNIQUE_HASH key require one extra key_part in which > + it stored hash. On safe side we will allocate memory for each key. > + */ > - n_length=keys * sizeof(KEY) + ext_key_parts * sizeof(KEY_PART_INFO); > + n_length=keys * sizeof(KEY) + (ext_key_parts +keys) * sizeof(KEY_PART_INFO); Hmm, why wouldn't you store the number of HA_KEY_ALG_LONG_HASH keys in EXTRA2_LONG_UNIQUES ? Then you'll know it here. > if (!(keyinfo= (KEY*) alloc_root(&share->mem_root, > n_length + len))) > return 1; > @@ -798,6 +805,14 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > } > key_part->store_length=key_part->length; > } > + if (keyinfo->algorithm == HA_KEY_ALG_LONG_HASH) > + { > + keyinfo->flags|= HA_LONG_UNIQUE_HASH | HA_NOSAME; > + keyinfo->key_length= 0; > + share->ext_key_parts++; > + // This empty key_part for storing Hash > + key_part++; > + } so, you write keyinfo's for HA_LONG_UNIQUE_HASH keys into the frm? why don't you write keysegs for them? I mean, it is not very logical. I'd thought you won't write either keyinfos or keysegs. Or, okay, you could write both. But only keyinfos and no keysegs? That's strange > > /* > Add primary key to end of extended keys for non unique keys for > @@ -1143,13 +1159,21 @@ bool parse_vcol_defs(THD *thd, MEM_ROOT *mem_root, TABLE *table, > pos+= expr_length; > } > > - /* Now, initialize CURRENT_TIMESTAMP fields */ > + /* Now, initialize CURRENT_TIMESTAMP and UNIQUE_INDEX_HASH_FIELD fields */ > for (field_ptr= table->field; *field_ptr; field_ptr++) > { > Field *field= *field_ptr; > - if (field->has_default_now_unireg_check()) > + if (field->vcol_info && (length = field->vcol_info->hash_expr.length)) > { > expr_str.length(parse_vcol_keyword.length); > + expr_str.append((char*)field->vcol_info->hash_expr.str, length); > + vcol= unpack_vcol_info_from_frm(thd, mem_root, table, &expr_str, > + &(field->vcol_info), error_reported); see above about not going through the parser for hash keys. > + *(vfield_ptr++)= *field_ptr; > + > + } > + if (field->has_default_now_unireg_check()) > + { > expr_str.append(STRING_WITH_LEN("current_timestamp(")); > expr_str.append_ulonglong(field->decimals()); > expr_str.append(')'); > @@ -2106,7 +2132,8 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > uchar flags= *extra2_field_flags++; > if (flags & VERS_OPTIMIZED_UPDATE) > reg_field->flags|= VERS_UPDATE_UNVERSIONED_FLAG; > - > + if (flags & EXTRA2_LONG_UNIQUE_HASH_FIELD) > + reg_field->flags|= LONG_UNIQUE_HASH_FIELD; so, you write LONG_UNIQUE_HASH_FIELD fields to frm too. Why? > reg_field->invisible= f_visibility(flags); > } > if (reg_field->invisible == INVISIBLE_USER) > @@ -2350,6 +2438,8 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > key_part= keyinfo->key_part; > uint key_parts= share->use_ext_keys ? keyinfo->ext_key_parts : > keyinfo->user_defined_key_parts; > + if (keyinfo->flags & HA_LONG_UNIQUE_HASH) > + key_parts++; key_parts++ ? Doesn't your HA_LONG_UNIQUE_HASH key have only one part, always? > for (i=0; i < key_parts; key_part++, i++) > { > Field *field; > @@ -2363,7 +2453,16 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > > field= key_part->field= share->field[key_part->fieldnr-1]; > key_part->type= field->key_type(); > + if (keyinfo->flags & HA_LONG_UNIQUE_HASH > + &&(key_part->length > handler_file->max_key_part_length() > + || key_part->length == 0)) 1. fix the spacing and the indentation here, please 2. what should happen if HA_LONG_UNIQUE_HASH flag is set, but the key_part->length is small ? > + { > + key_part->key_part_flag= HA_HASH_KEY_PART_FLAG; > + key_part->store_length= HA_HASH_KEY_PART_LENGTH; > + } > + /* Invisible Full is currently used by long uniques */ > - if (field->invisible > INVISIBLE_USER && !field->vers_sys_field()) > + if ((field->invisible == INVISIBLE_USER || > + field->invisible == INVISIBLE_SYSTEM )&& !field->vers_sys_field()) why is this change? > keyinfo->flags |= HA_INVISIBLE_KEY; > if (field->null_ptr) > { > @@ -2428,7 +2527,8 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > field->part_of_sortkey= share->keys_in_use; > } > } > - if (field->key_length() != key_part->length) > + if (field->key_length() != key_part->length && > + !(keyinfo->flags & HA_LONG_UNIQUE_HASH)) why is that? > { > #ifndef TO_BE_DELETED_ON_PRODUCTION > if (field->type() == MYSQL_TYPE_NEWDECIMAL) > @@ -2470,7 +2570,8 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > if (!(key_part->key_part_flag & (HA_BLOB_PART | HA_VAR_LENGTH_PART | > HA_BIT_PART)) && > key_part->type != HA_KEYTYPE_FLOAT && > - key_part->type == HA_KEYTYPE_DOUBLE) > + key_part->type == HA_KEYTYPE_DOUBLE && > + !(keyinfo->flags & HA_LONG_UNIQUE_HASH)) why is that? > key_part->key_part_flag|= HA_CAN_MEMCMP; > } > keyinfo->usable_key_parts= usable_parts; // Filesort > @@ -8346,6 +8454,377 @@ double KEY::actual_rec_per_key(uint i) > } > > > +/* > + find out the field positoin in hash_str() > + position starts from 0 > + else return -1; > +*/ > +int find_field_pos_in_hash(Item *hash_item, const char * field_name) > +{ > + Item_func_or_sum * temp= static_cast<Item_func_or_sum *>(hash_item); > + Item_args * t_item= static_cast<Item_args *>(temp); > + uint arg_count= t_item->argument_count(); > + Item ** arguments= t_item->arguments(); > + Field * t_field; > + > + for (uint j=0; j < arg_count; j++) > + { > + DBUG_ASSERT(arguments[j]->type() == Item::FIELD_ITEM || > + arguments[j]->type() == Item::FUNC_ITEM); > + if (arguments[j]->type() == Item::FIELD_ITEM) > + { > + t_field= static_cast<Item_field *>(arguments[j])->field; > + } > + else > + { > + Item_func_left *fnc= static_cast<Item_func_left *>(arguments[j]); > + t_field= static_cast<Item_field *>(fnc->arguments()[0])->field; > + } > + if (!my_strcasecmp(system_charset_info, t_field->field_name.str, field_name)) > + return j; > + } > + return -1; > +} > + > +/* > + find total number of field in hash_str > +*/ > +int fields_in_hash_str(Item * hash_item) can be static > +{ > + Item_func_or_sum * temp= static_cast<Item_func_or_sum *>(hash_item); > + Item_args * t_item= static_cast<Item_args *>(temp); > + return t_item->argument_count(); > +} > + > +/* > + Returns fields ptr given by hash_str index > + Index starts from 0 > +*/ > +Field * field_ptr_in_hash_str(Item *hash_item, int index) > +{ > + Item_func_or_sum * temp= static_cast<Item_func_or_sum *>(hash_item); > + Item_args * t_item= static_cast<Item_args *>(temp); > + return static_cast<Item_field *>(t_item->arguments()[index])->field; > +} > + > +//NO longer Needed remove it, then :) > +int get_key_part_length(KEY *keyinfo, int index) > +{ > + DBUG_ASSERT(keyinfo->flags & HA_LONG_UNIQUE_HASH); > + TABLE *tbl= keyinfo->table; > + Item *h_item= keyinfo->key_part->field->vcol_info->expr; > + Field *fld= field_ptr_in_hash_str(h_item, index); > + if (!index) > + return fld->pack_length()+HA_HASH_KEY_LENGTH_WITH_NULL+1; > + return fld->pack_length()+1; > +} > +/** > + @brief clone of current handler. > + Creates a clone of handler used in update for > + unique hash key. > + @param thd Thread Object > + @param table Table Object > + @return handler object > +*/ > +void create_update_handler(THD *thd, TABLE *table) better, clone_handler_for_update > +{ > + handler *update_handler= NULL; > + for (uint i= 0; i < table->s->keys; i++) indentation > + { > + if (table->key_info[i].flags & HA_LONG_UNIQUE_HASH) > + { > + update_handler= table->file->clone(table->s->normalized_path.str, > + thd->mem_root); > + update_handler->ha_external_lock(thd, F_RDLCK); > + table->update_handler= update_handler; why do you store it in TABLE? you can just keep it in a local variable in mysql_update, you don't need it outside of mysql_update anyway. (and multi-update) also, you shouldn't need to scan all table keys here. use some flag or property in TABLE_SCHEMA to check quickly whether the table has these lock unique keys. > + return; > + } > + } > + return; > +} > + > +/** > + @brief Deletes update handler object > + @param thd Thread Object > + @param table Table Object > +*/ > +void delete_update_handler(THD *thd, TABLE *table) > +{ > + if (table->update_handler) > + { > + table->update_handler->ha_external_lock(thd, F_UNLCK); > + table->update_handler->ha_close(); > + delete table->update_handler; > + table->update_handler= NULL; > + } > +} > +/** > + @brief This function makes table object with > + long unique keys ready for storage engine. > + It makes key_part of HA_LONG_UNIQUE_HASH point to > + hash key_part. > + @param table Table object > + */ > +void setup_table_hash(TABLE *table) this and other functions could be methods in TABLE > +{ > + /* > + Extra parts of long unique key which are used only at server level > + for example in key unique(a, b, c) //a b c are blob > + extra_key_part_hash is 3 > + */ > + uint extra_key_part_hash= 0; > + uint hash_parts= 0; > + KEY *s_keyinfo= table->s->key_info; > + KEY *keyinfo= table->key_info; > + /* > + Sometime s_keyinfo or key_info can be null. So > + two different loop for keyinfo and s_keyinfo > + reference test case:- main.subselect_sj2 how they could be null here? > + */ > + > + if (keyinfo) > + { > + for (uint i= 0; i < table->s->keys; i++, keyinfo++) > + { > + if (keyinfo->flags & HA_LONG_UNIQUE_HASH) > + { > + DBUG_ASSERT(keyinfo->user_defined_key_parts == > + keyinfo->ext_key_parts); > + keyinfo->flags&= ~(HA_NOSAME | HA_LONG_UNIQUE_HASH); > + keyinfo->algorithm= HA_KEY_ALG_UNDEF; > + extra_key_part_hash+= keyinfo->ext_key_parts; > + hash_parts++; > + keyinfo->key_part= keyinfo->key_part+ keyinfo->ext_key_parts; > + keyinfo->user_defined_key_parts= keyinfo->usable_key_parts= > + keyinfo->ext_key_parts= 1; > + keyinfo->key_length= keyinfo->key_part->store_length; > + } > + } > + table->s->key_parts-= extra_key_part_hash; > + table->s->key_parts+= hash_parts; > + table->s->ext_key_parts-= extra_key_part_hash; I don't understand what you're doing here. Could you add a comment, explaning the resulting structure of keys and keysegs and what's where? > + } > + if (s_keyinfo) > + { > + for (uint i= 0; i < table->s->keys; i++, s_keyinfo++) > + { > + if (s_keyinfo->flags & HA_LONG_UNIQUE_HASH) > + { > + DBUG_ASSERT(s_keyinfo->user_defined_key_parts == > + s_keyinfo->ext_key_parts); > + s_keyinfo->flags&= ~(HA_NOSAME | HA_LONG_UNIQUE_HASH); > + s_keyinfo->algorithm= HA_KEY_ALG_BTREE; > + extra_key_part_hash+= s_keyinfo->ext_key_parts; > + s_keyinfo->key_part= s_keyinfo->key_part+ s_keyinfo->ext_key_parts; > + s_keyinfo->user_defined_key_parts= s_keyinfo->usable_key_parts= > + s_keyinfo->ext_key_parts= 1; > + s_keyinfo->key_length= s_keyinfo->key_part->store_length; > + } > + } > + if (!keyinfo) > + { > + table->s->key_parts-= extra_key_part_hash; > + table->s->key_parts+= hash_parts; > + table->s->ext_key_parts-= extra_key_part_hash; > + } > + } > +} > + > +/** > + @brief Revert the effect of setup_table_hash > + @param table Table Object > + */ > +void re_setup_table(TABLE *table) > +{ > + //extra key parts excluding hash , which needs to be added in keyparts > + uint extra_key_parts_ex_hash= 0; > + uint extra_hash_parts= 0; // this var for share->extra_hash_parts > + KEY *s_keyinfo= table->s->key_info; > + KEY *keyinfo= table->key_info; > + /* > + Sometime s_keyinfo can be null so > + two different loop for keyinfo and s_keyinfo > + ref test case:- main.subselect_sj2 > + */ > + if (keyinfo) > + { > + for (uint i= 0; i < table->s->keys; i++, keyinfo++) > + { > + if (keyinfo->user_defined_key_parts == 1 && > + keyinfo->key_part->field->flags & LONG_UNIQUE_HASH_FIELD) > + { > + keyinfo->flags|= (HA_NOSAME | HA_LONG_UNIQUE_HASH); > + keyinfo->algorithm= HA_KEY_ALG_LONG_HASH; > + /* Sometimes it can happen, that we does not parsed hash_str. > + Like when this function is called in ha_create. So we will > + Use field from table->field rather then share->field*/ > + Item *h_item= table->field[keyinfo->key_part->fieldnr - 1]-> > + vcol_info->expr; > + uint hash_parts= fields_in_hash_str(h_item); > + keyinfo->key_part= keyinfo->key_part- hash_parts; > + keyinfo->user_defined_key_parts= keyinfo->usable_key_parts= > + keyinfo->ext_key_parts= hash_parts; > + extra_key_parts_ex_hash+= hash_parts; > + extra_hash_parts++; > + keyinfo->key_length= -1; > + } > + } > + table->s->key_parts-= extra_hash_parts; > + table->s->key_parts+= extra_key_parts_ex_hash; > + table->s->ext_key_parts+= extra_key_parts_ex_hash + extra_hash_parts; > + } > + if (s_keyinfo) > + { > + for (uint i= 0; i < table->s->keys; i++, s_keyinfo++) > + { > + if (s_keyinfo->user_defined_key_parts == 1 && > + s_keyinfo->key_part->field->flags & LONG_UNIQUE_HASH_FIELD) > + { > + s_keyinfo->flags|= (HA_NOSAME | HA_LONG_UNIQUE_HASH); > + s_keyinfo->algorithm= HA_KEY_ALG_LONG_HASH; > + extra_hash_parts++; > + /* Sometimes it can happen, that we does not parsed hash_str. > + Like when this function is called in ha_create. So we will > + Use field from table->field rather then share->field*/ > + Item *h_item= table->field[s_keyinfo->key_part->fieldnr - 1]-> > + vcol_info->expr; > + uint hash_parts= fields_in_hash_str(h_item); > + s_keyinfo->key_part= s_keyinfo->key_part- hash_parts; > + s_keyinfo->user_defined_key_parts= s_keyinfo->usable_key_parts= > + s_keyinfo->ext_key_parts= hash_parts; > + extra_key_parts_ex_hash+= hash_parts; > + s_keyinfo->key_length= -1; > + } > + } > + if (!keyinfo) > + { > + table->s->key_parts-= extra_hash_parts; > + table->s->key_parts+= extra_key_parts_ex_hash; > + table->s->ext_key_parts+= extra_key_parts_ex_hash + extra_hash_parts; > + } > + } > +} > + > +/** > + @brief set_hash > + @param table > + @param key_index > + @param key_buff > + > +int get_hash_key(THD *thd,TABLE *table, handler *h, uint key_index, > + uchar * rec_buff, uchar *key_buff) unused? > +{ > + KEY *keyinfo= &table->key_info[key_index]; > + DBUG_ASSERT(keyinfo->flags & HA_LONG_UNIQUE_HASH); > + KEY_PART_INFO *temp_key_part= table->key_info[key_index].key_part; > + Field *fld[keyinfo->user_defined_key_parts]; > + Field *t_field; > + uchar hash_buff[9]; > + ulong nr1= 1, nr2= 4; > + String *str1, temp1; > + String *str2, temp2; > + bool is_null= false; > + bool is_same= true; > + bool is_index_inited= h->inited; > + /* difference between field->ptr and start of rec_buff * > + long diff = rec_buff- table->record[0]; > + int result= 0; > + for (uint i=0; i < keyinfo->user_defined_key_parts; i++, temp_key_part++) > + { > + uint maybe_null= MY_TEST(temp_key_part->null_bit); > + fld[i]= temp_key_part->field->new_key_field(thd->mem_root, table, > + key_buff + maybe_null, 12, > + maybe_null?key_buff:0, 1, > + temp_key_part->key_part_flag > + & HA_HASH_KEY_PART_FLAG); > + if (fld[i]->is_real_null()) > + { > + is_null= true; > + break; > + } > + str1= fld[i]->val_str(&temp1); > + calc_hash_for_unique(nr1, nr2, str1); > + key_buff+= temp_key_part->store_length; > + } > + if (is_null && !(keyinfo->flags & HA_NULL_PART_KEY)) > + return HA_ERR_KEY_NOT_FOUND; > + if (keyinfo->flags & HA_NULL_PART_KEY) > + { > + hash_buff[0]= is_null; > + int8store(hash_buff + 1, nr1); > + } > + else > + int8store(hash_buff, nr1); > + if (!is_index_inited) > + result= h->ha_index_init(key_index, 0); > + if (result) > + return result; > + > + setup_table_hash(table); > + result= h->ha_index_read_map(rec_buff, hash_buff, HA_WHOLE_KEY, > + HA_READ_KEY_EXACT); > + re_setup_table(table); > + if (!result) > + { > + for (uint i=0; i < keyinfo->user_defined_key_parts; i++) > + { > + t_field= keyinfo->key_part[i].field; > + t_field->move_field(t_field->ptr+diff, > + t_field->null_ptr+diff, t_field->null_bit); > + } > + do > + { > + re_setup_table(table); > + is_same= true; > + for (uint i=0; i < keyinfo->user_defined_key_parts; i++) > + { > + t_field= keyinfo->key_part[i].field; > + if (fld[i]->is_real_null() && t_field->is_real_null()) > + continue; > + if (!fld[i]->is_real_null() && !t_field->is_real_null()) > + { > + str1= t_field->val_str(&temp1); > + str2= fld[i]->val_str(&temp2); > + if (my_strcasecmp(str1->charset(), str1->c_ptr_safe(), > + str2->c_ptr_safe())) > + { > + is_same= false; > + break; > + } > + } > + else > + { > + is_same= false; > + break; > + } > + } > + setup_table_hash(table); > + } > + while (!is_same && !(result= h->ha_index_next_same(rec_buff, hash_buff, > + keyinfo->key_length))); > + for (uint i=0; i < keyinfo->user_defined_key_parts; i++) > + { > + t_field= keyinfo->key_part[i].field; > + t_field->move_field(t_field->ptr-diff, > + t_field->null_ptr-diff, t_field->null_bit); > + } > + } > + if (!is_index_inited) > + h->ha_index_end(); > + re_setup_table(table); > + for (uint i=0; i < keyinfo->user_defined_key_parts; i++) > + { > + if (keyinfo->key_part[i].key_part_flag & HA_FIELD_EX_FREED) > + { > + Field_blob *blb= static_cast<Field_blob *>(fld[i]); > + uchar * addr; > + blb->get_ptr(&addr); > + my_free(addr); > + } > + } > + return result; > +} > +*/ > LEX_CSTRING *fk_option_name(enum_fk_option opt) > { > static LEX_CSTRING names[]= > @@ -8814,3 +9293,15 @@ bool TABLE::export_structure(THD *thd, Row_definition_list *defs) > } > return false; > } > + > +void calc_hash_for_unique(ulong &nr1, ulong &nr2, String *str) > +{ > + CHARSET_INFO *cs; > + uchar l[4]; > + int4store(l, str->length()); > + cs= &my_charset_bin; > + cs->coll->hash_sort(cs, l, sizeof(l), &nr1, &nr2); > + cs= str->charset(); > + cs->coll->hash_sort(cs, (uchar *)str->ptr(), str->length(), &nr1, &nr2); > + sql_print_information("setiya %lu, %s", nr1, str->ptr()); really? :) > +} > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 4ac8b102d79..f42687c0377 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -3343,6 +3343,87 @@ mysql_add_invisible_index(THD *thd, List<Key> *key_list, > key_list->push_back(key, thd->mem_root); > return key; > } > +/** > + Add hidden level 3 hash field to table in case of long s/hidden level 3/fully invisible/ > + unique column > + @param thd Thread Context. > + @param create_list List of table fields. > + @param cs Field Charset > + @param key_info Whole Keys buffer > + @param key_index Index of current key 1. instead of key_info and key_index, you can just take a key_info of the current key here (that is key_info + key_index). 2. what is the "current key"? Is it the unique key over blobs that you create a hash field for? please clarify the comment. > +*/ > + > +static void add_hash_field(THD * thd, List<Create_field> *create_list, > + CHARSET_INFO *cs, KEY *key_info, int key_index) > +{ > + List_iterator<Create_field> it(*create_list); > + Create_field *dup_field, *cf= new (thd->mem_root) Create_field(); > + cf->flags|= UNSIGNED_FLAG | LONG_UNIQUE_HASH_FIELD; > + cf->charset= cs; > + cf->decimals= 0; > + cf->length= cf->char_length= cf->pack_length= HA_HASH_FIELD_LENGTH; > + cf->invisible= INVISIBLE_FULL; > + cf->pack_flag|= FIELDFLAG_MAYBE_NULL; > + uint num= 1; > + char *temp_name= (char *)thd->alloc(30); > + my_snprintf(temp_name, 30, "DB_ROW_HASH_%u", num); please, give names for 30 and for the whole my_snprintf - macros, functions, whatever. And better use a LEX_STRING for it, for example: #define LONG_HASH_FIELD_NAME_LENGTH 30 static inline make_long_hash_field_name(LEX_STRING buf, uint num) { buf->length= my_snprintf(buf->str, LONG_HASH_FIELD_NAME_LENGTH, "DB_ROW_HASH_%u", num); } > + /* > + Check for collusions collisions :) > + */ > + while ((dup_field= it++)) > + { > + if (!my_strcasecmp(system_charset_info, temp_name, dup_field->field_name.str)) > + { > + num++; > + my_snprintf(temp_name, 30, "DB_ROW_HASH_%u", num); > + it.rewind(); > + } > + } > + it.rewind(); > + cf->field_name.str= temp_name; > + cf->field_name.length= strlen(temp_name); you won't need strlen here, if you use LEX_STRING above > + cf->set_handler(&type_handler_longlong); > + /* > + We have added db_row_hash field in starting of > + fields array , So we have to change key_part > + field index > + for (int i= 0; i <= key_index; i++, key_info++) > + { > + KEY_PART_INFO *info= key_info->key_part; > + for (uint j= 0; j < key_info->user_defined_key_parts; j++, info++) > + { > + info->fieldnr+= 1; > + info->offset+= HA_HASH_FIELD_LENGTH; > + } > + }*/ Forgot to remove it? > + key_info[key_index].flags|= HA_NOSAME; > + key_info[key_index].algorithm= HA_KEY_ALG_LONG_HASH; > + it.rewind(); > + uint record_offset= 0; > + while ((dup_field= it++)) > + { > + dup_field->offset= record_offset; > + if (dup_field->stored_in_db()) > + record_offset+= dup_field->pack_length; Why do you change all field offsets? You only need to put your field last, that's all. Like while ((dup_field= it++)) set_if_bigger(record_offset, dup_field->offset + dup_field->pack_length) > + } > + cf->offset= record_offset; > + /* > + it.rewind(); > + while ((sql_field= it++)) > + { > + if (!sql_field->stored_in_db()) > + { > + sql_field->offset= record_offset; > + record_offset+= sql_field->pack_length; > + } > + } why is that? > + */ > + /* hash column should be fully hidden */ > + //prepare_create_field(cf, NULL, 0); Forgot to remove it? > + create_list->push_back(cf,thd->mem_root); > +} > + > + > /* > Preparation for table creation > > @@ -3868,6 +3951,8 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > } > > cols2.rewind(); > + key_part_info->fieldnr= field; > + key_part_info->offset= (uint16) sql_field->offset; why did you need to move that? > if (key->type == Key::FULLTEXT) > { > if ((sql_field->real_field_type() != MYSQL_TYPE_STRING && > @@ -3922,8 +4007,19 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > column->length= MAX_LEN_GEOM_POINT_FIELD; > if (!column->length) > { > - my_error(ER_BLOB_KEY_WITHOUT_LENGTH, MYF(0), column->field_name.str); > - DBUG_RETURN(TRUE); > + if (key->type == Key::PRIMARY) > + { > + my_error(ER_BLOB_KEY_WITHOUT_LENGTH, MYF(0), column->field_name.str); > + DBUG_RETURN(TRUE); > + } > + else if (!is_hash_field_added) > + { > + add_hash_field(thd, &alter_info->create_list, > + create_info->default_table_charset, > + *key_info_buffer, key_number); > + column->length= 0; why column->length= 0 ? > + is_hash_field_added= true; > + } > } > } > #ifdef HAVE_SPATIAL > @@ -4062,11 +4159,29 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > } > else > { > - my_error(ER_TOO_LONG_KEY, MYF(0), key_part_length); > - DBUG_RETURN(TRUE); > - } > + if(key->type == Key::UNIQUE) > + { > + if (!is_hash_field_added) > + { > + add_hash_field(thd, &alter_info->create_list, > + create_info->default_table_charset, > + *key_info_buffer, key_number); > + is_hash_field_added= true; instead of is_hash_field_added and many add_hash_field() here and there, I'd rather rename the variable to hash_field_needed. And only do hash_field_needed= true. And at the end if (hash_field_needed) add_hash_field(...) > + } > + } > + else > + { > + my_error(ER_TOO_LONG_KEY, MYF(0), key_part_length); any test cases for this error? > + DBUG_RETURN(TRUE); > + } > + } > } > - key_part_info->length= (uint16) key_part_length; > + /* We can not store key_part_length more then 2^16 - 1 in frm > + So we will simply make it zero */ Not really. If someone explicitly asks for a long prefix, it should be an error. Like CREATE TABLE (a blob, UNIQUE (a(65537)); -- this is an error > + if (is_hash_field_added && key_part_length > (2<<16) - 1) > + key_part_info->length= 0; > + else > + key_part_info->length= (uint16) key_part_length; > /* Use packed keys for long strings on the first column */ > if (!((*db_options) & HA_OPTION_NO_PACK_KEYS) && > !((create_info->table_options & HA_OPTION_NO_PACK_KEYS)) && > @@ -4122,12 +4237,37 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > if (key->type == Key::UNIQUE && !(key_info->flags & HA_NULL_PART_KEY)) > unique_key=1; > key_info->key_length=(uint16) key_length; > - if (key_length > max_key_length && key->type != Key::FULLTEXT) > + if (key_length > max_key_length && key->type != Key::FULLTEXT && > + !is_hash_field_added) > { > my_error(ER_TOO_LONG_KEY,MYF(0),max_key_length); > DBUG_RETURN(TRUE); > } > > + if (is_hash_field_added) > + { > + if (key_info->flags & HA_NULL_PART_KEY) > + null_fields++; > + else > + { > + uint elements= alter_info->create_list.elements; > + Create_field *hash_fld= static_cast<Create_field *>(alter_info-> > + create_list.elem(elements -1 )); > + hash_fld->flags|= NOT_NULL_FLAG; > + hash_fld->pack_flag&= ~FIELDFLAG_MAYBE_NULL; > + /* > + Althought we do not need default value anywhere in code , but if we create > + table with non null long columns , then at the time of insert we get warning. > + So default value is used so solve this warning. > + Virtual_column_info *default_value= new (thd->mem_root) Virtual_column_info(); > + char * def_str= (char *)alloc_root(thd->mem_root, 2); > + strncpy(def_str, "0", 1); > + default_value->expr_str.str= def_str; > + default_value->expr_str.length= 1; > + default_value->expr_item= new (thd->mem_root) Item_int(thd,0); > + hash_fld->default_value= default_value; */ > + } Forgot to remove it? > + } > if (validate_comment_length(thd, &key->key_create_info.comment, > INDEX_COMMENT_MAXLEN, > ER_TOO_LONG_INDEX_COMMENT, > @@ -8328,6 +8468,11 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > enum Key::Keytype key_type; > LEX_CSTRING tmp_name; > bzero((char*) &key_create_info, sizeof(key_create_info)); > + if (key_info->flags & HA_LONG_UNIQUE_HASH) > + { > + key_info->flags&= ~(HA_LONG_UNIQUE_HASH); > + key_info->algorithm= HA_KEY_ALG_UNDEF; > + } Why? > key_create_info.algorithm= key_info->algorithm; > /* > We copy block size directly as some engines, like Area, sets this > diff --git a/sql/handler.cc b/sql/handler.cc > index b77b2a3fa2c..cda53dfdc87 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -6179,6 +6185,185 @@ int handler::ha_reset() > DBUG_RETURN(reset()); > } > > +static int check_duplicate_long_entry_key(TABLE *table, handler *h, uchar *new_rec, > + uint key_no) > +{ > + Field *hash_field; > + int result, error= 0; > + if (!(table->key_info[key_no].user_defined_key_parts == 1 > + && table->key_info[key_no].key_part->field->flags & LONG_UNIQUE_HASH_FIELD )) > + return 0; What if LONG_UNIQUE_HASH_FIELD is set but user_defined_key_parts != 1? > + hash_field= table->key_info[key_no].key_part->field; > + DBUG_ASSERT((table->key_info[key_no].flags & HA_NULL_PART_KEY && > + table->key_info[key_no].key_length == HA_HASH_KEY_LENGTH_WITH_NULL) > + || table->key_info[key_no].key_length == HA_HASH_KEY_LENGTH_WITHOUT_NULL); > + uchar ptr[HA_HASH_KEY_LENGTH_WITH_NULL]; > + > + if (hash_field->is_real_null()) > + return 0; > + > + key_copy(ptr, new_rec, &table->key_info[key_no], > + table->key_info[key_no].key_length, false); good, use existing function, no need to reinvent the wheel > + > + if (!table->check_unique_buf) > + table->check_unique_buf= (uchar *)alloc_root(&table->mem_root, > + table->s->reclength*sizeof(uchar)); > + > + result= h->ha_index_init(key_no, 0); > + if (result) > + return result; > + result= h->ha_index_read_map(table->check_unique_buf, > + ptr, HA_WHOLE_KEY, HA_READ_KEY_EXACT); > + if (!result) > + { > + bool is_same; > + do > + { > + Item_func_or_sum * temp= static_cast<Item_func_or_sum *>(hash_field-> > + vcol_info->expr); > + Item_args * t_item= static_cast<Item_args *>(temp); > + uint arg_count= t_item->argument_count(); > + Item ** arguments= t_item->arguments(); > + long diff= table->check_unique_buf - new_rec; > + Field * t_field; > + is_same= true; > + for (uint j=0; j < arg_count; j++) > + { > + DBUG_ASSERT(arguments[j]->type() == Item::FIELD_ITEM || > + // this one for left(fld_name,length) > + arguments[j]->type() == Item::FUNC_ITEM); > + if (arguments[j]->type() == Item::FIELD_ITEM) > + { > + t_field= static_cast<Item_field *>(arguments[j])->field; > + if (t_field->cmp_offset(diff)) > + is_same= false; > + } > + else > + { > + Item_func_left *fnc= static_cast<Item_func_left *>(arguments[j]); > + DBUG_ASSERT(!my_strcasecmp(system_charset_info, "left", fnc->func_name())); > + //item_data= fnc->val_str(&tmp1); > + DBUG_ASSERT(fnc->arguments()[0]->type() == Item::FIELD_ITEM); > + t_field= static_cast<Item_field *>(fnc->arguments()[0])->field; > + // field_data= t_field->val_str(&tmp2); > + // if (my_strnncoll(t_field->charset(),(const uchar *)item_data->ptr(), > + // item_data->length(), > + // (const uchar *)field_data.ptr(), > + // item_data->length())) > + // return 0; > + uint length= fnc->arguments()[1]->val_int(); > + if (t_field->cmp_max(t_field->ptr, t_field->ptr + diff, length)) > + is_same= false; > + } > + } you don't need all that code, use key_cmp() or key_cmp_if_same() from key.cc > + } > + while (!is_same && !(result= table->file->ha_index_next_same(table->check_unique_buf, > + ptr, table->key_info[key_no].key_length))); > + if (is_same) > + { > + table->dupp_hash_key= key_no; > + error= HA_ERR_FOUND_DUPP_KEY; > + goto exit; > + } > + else > + goto exit; > + } > + if (result == HA_ERR_LOCK_WAIT_TIMEOUT) > + { > + table->dupp_hash_key= key_no; > + //TODO check if this is the only case > + error= HA_ERR_FOUND_DUPP_KEY; Why? > + } > + exit: > + h->ha_index_end(); > + return error; > +} > +/** @brief > + check whether inserted/updated records breaks the > + unique constraint on long columns. > + In the case of update we just need to check the specic key > + reason for that is consider case > + create table t1(a blob , b blob , x blob , y blob ,unique(a,b) > + ,unique(x,y)) > + and update statement like this > + update t1 set a=23+a; in this case if we try to scan for > + whole keys in table then index scan on x_y will return 0 > + because data is same so in the case of update we take > + key as a parameter in normal insert key should be -1 > + @returns 0 if no duplicate else returns error > + */ > +static int check_duplicate_long_entries(TABLE *table, handler *h, uchar *new_rec) > +{ > + table->dupp_hash_key= -1; > + int result; > + for (uint i= 0; i < table->s->keys; i++) > + { > + if ((result= check_duplicate_long_entry_key(table, h, new_rec, i))) > + return result; > + } > + return 0; > +} > + > +/** @brief > + check whether updated records breaks the > + unique constraint on long columns. > + @returns 0 if no duplicate else returns error > + */ > +static int check_duplicate_long_entries_update(TABLE *table, handler *h, uchar *new_rec) > +{ > + Field **f, *field; > + Item *h_item; > + int error= 0; > + bool is_update_handler_null= false; > + /* > + Here we are comparing whether new record and old record are same > + with respect to fields in hash_str > + */ > + long reclength= table->record[1]-table->record[0]; > + for (uint i= 0; i < table->s->keys; i++) > + { > + if (table->key_info[i].user_defined_key_parts == 1 && > + table->key_info[i].key_part->field->flags & LONG_UNIQUE_HASH_FIELD) > + { > + /* > + Currently mysql_update is pacthed so that it will automatically set the > + Update handler and then free it but ha_update_row is used in many function ( > + like in case of reinsert) Instead of patching them all here we check is > + update_handler is null then set it And then set it null again > + */ > + if (!table->update_handler) > + { > + create_update_handler(current_thd, table); > + is_update_handler_null= true; > + } > + h_item= table->key_info[i].key_part->field->vcol_info->expr; > + for (f= table->field; f && (field= *f); f++) > + { > + if ( find_field_pos_in_hash(h_item, field->field_name.str) != -1) > + { > + /* Compare fields if they are different then check for duplicates*/ > + if(field->cmp_binary_offset(reclength)) > + { > + if((error= check_duplicate_long_entry_key(table, table->update_handler, > + new_rec, i))) > + goto exit; > + /* > + break beacuse check_duplicate_long_entrie_key will > + take care of remaning fields > + */ > + break; > + } > + } > + } > + } > + } > + exit: > + if (is_update_handler_null) > + { > + delete_update_handler(current_thd, table); > + } > + return error; > +} > > int handler::ha_write_row(uchar *buf) > { > @@ -6189,14 +6374,21 @@ int handler::ha_write_row(uchar *buf) > DBUG_ENTER("handler::ha_write_row"); > DEBUG_SYNC_C("ha_write_row_start"); > > + setup_table_hash(table); No-no. You cannot modify table structure back and forth for *every inserted row*. It's ok to do it once, when a table is opened. But not for every row, > MYSQL_INSERT_ROW_START(table_share->db.str, table_share->table_name.str); > mark_trx_read_write(); > increment_statistics(&SSV::ha_write_count); > > + if ((error= check_duplicate_long_entries(table, table->file, buf))) > + { > + re_setup_table(table); > + DBUG_RETURN(error); > + } > TABLE_IO_WAIT(tracker, m_psi, PSI_TABLE_WRITE_ROW, MAX_KEY, 0, > { error= write_row(buf); }) > > MYSQL_INSERT_ROW_DONE(error); > + re_setup_table(table); > if (likely(!error) && !row_already_logged) > { > rows_changed++; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-17359
by jerome brauge 09 Oct '18

09 Oct '18

Hello Alexander, Can you review this patch for MDEV-17359 ? Best regards. Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de ses destinataires. Toute utilisation ou diffusion, même partielle, non autorisée est interdite. Tout message électronique est susceptible d'altération; CEGID décline donc toute responsabilité au titre de ce message. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'avertir l'expéditeur. This message and any attachments are confidential and intended solely for the addressees. Any unauthorized use or disclosure, either whole or partial is prohibited. E-mails are susceptible to alteration; CEGID shall therefore not be liable for the content of this message. If you are not the intended recipient of this message, please delete it and notify the sender.

2 9

Re: [Maria-developers] d1ccc60: MDEV-14005 Remove need for Partition Key to be part of Primary Key.
by Sergei Golubchik 05 Oct '18

05 Oct '18

Hi, Alexey! Some minor comments, questions, see below... On Aug 21, Alexey Botchkov wrote: > revision-id: d1ccc60360c79cd456abbebafc1c80c8043b7dce (mariadb-10.3.6-106-gd1ccc60) > parent(s): ead9a34a3e934f607c2ea7a6c68f7f6d9d29b5bd > committer: Alexey Botchkov > timestamp: 2018-08-21 14:47:45 +0400 > message: > > MDEV-14005 Remove need for Partition Key to be part of Primary Key. > > The limitation was removed, so now we check all the partition > for unique key duplicates in these cases. > > diff --git a/mysql-test/main/partition_unique.result b/mysql-test/main/partition_unique.result > new file mode 100644 > index 0000000..17a9550 > --- /dev/null > +++ b/mysql-test/main/partition_unique.result > @@ -0,0 +1,51 @@ > +CREATE TABLE t1 ( > +id int(10) NOT NULL, > +status varchar(1) DEFAULT NULL, > +PRIMARY KEY (`id`) > +) > +PARTITION BY LIST COLUMNS(`status`) > +( > +PARTITION `a` VALUES IN ('A'), > +PARTITION `b` VALUES IN ('B'), > +PARTITION `c` VALUES IN ('C'), > +PARTITION `d` DEFAULT); > +INSERT INTO t1 VALUES (4, 'A'); > +INSERT INTO t1 VALUES (6, 'A'); > +INSERT INTO t1 VALUES (4, 'C'); > +ERROR 23000: Duplicate entry '4' for key 'PRIMARY' > +INSERT INTO t1 VALUES (5, 'C'); > +UPDATE t1 SET id=4 WHERE id=5; > +ERROR 23000: Duplicate entry '4' for key 'PRIMARY' > +UPDATE t1 SET id=4 WHERE id=5 AND status='C'; > +ERROR 23000: Duplicate entry '4' for key 'PRIMARY' > +UPDATE t1 SET id=6 WHERE id=4 AND status='A'; > +ERROR 23000: Duplicate entry '6' for key 'PRIMARY' > +select * from t1; > +id status > +4 A > +6 A > +5 C > +connect con1,localhost,root,,test; > +connect con2,localhost,root,,test; > +connection con1; > +SET DEBUG_SYNC= 'berfore_part_unique_check SIGNAL bpu_hit WAIT_FOR bpu_flushed'; may be s/berfore/before/ ? :) > +INSERT INTO t1 VALUES(7, 'A'); > +connection con2; > +SET DEBUG_SYNC= 'now WAIT_FOR bpu_hit'; > +INSERT INTO t1 VALUES(7, 'C'); > +connection default; > +SET DEBUG_SYNC= 'now SIGNAL bpu_flushed'; > +connection con1; > +connection con2; > +ERROR 23000: Duplicate entry '7' for key 'PRIMARY' > +disconnect con1; > +disconnect con2; > +connection default; > +select * from t1; > +id status > +4 A > +6 A > +7 A > +5 C > +DROP TABLE t1; > +set debug_sync= "RESET"; > diff --git a/mysql-test/main/partition_unique.test b/mysql-test/main/partition_unique.test > new file mode 100644 > index 0000000..f292359 > --- /dev/null > +++ b/mysql-test/main/partition_unique.test > @@ -0,0 +1,58 @@ > +--source include/have_innodb.inc > +--source include/have_partition.inc > + > +CREATE TABLE t1 ( > + id int(10) NOT NULL, > + status varchar(1) DEFAULT NULL, > + PRIMARY KEY (`id`) > + ) > + PARTITION BY LIST COLUMNS(`status`) > + ( > + PARTITION `a` VALUES IN ('A'), > + PARTITION `b` VALUES IN ('B'), > + PARTITION `c` VALUES IN ('C'), > + PARTITION `d` DEFAULT); you don't seem to test innodb, this looks like a partitioned myisam table > + > +INSERT INTO t1 VALUES (4, 'A'); > +INSERT INTO t1 VALUES (6, 'A'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES (4, 'C'); > +INSERT INTO t1 VALUES (5, 'C'); > +--error ER_DUP_ENTRY > +UPDATE t1 SET id=4 WHERE id=5; > +--error ER_DUP_ENTRY > +UPDATE t1 SET id=4 WHERE id=5 AND status='C'; > +--error ER_DUP_ENTRY > +UPDATE t1 SET id=6 WHERE id=4 AND status='A'; > +select * from t1; > + > +connect (con1,localhost,root,,test); > +connect (con2,localhost,root,,test); > +connection con1; > +SET DEBUG_SYNC= 'berfore_part_unique_check SIGNAL bpu_hit WAIT_FOR bpu_flushed'; > +send INSERT INTO t1 VALUES(7, 'A'); > + > +connection con2; > +SET DEBUG_SYNC= 'now WAIT_FOR bpu_hit'; > +send INSERT INTO t1 VALUES(7, 'C'); add also a test that you can insert some other value in 'A' and 'C' partitions. That is, partitions aren't completely locked. > + > +connection default; > +SET DEBUG_SYNC= 'now SIGNAL bpu_flushed'; > + > +connection con1; > +--reap > +connection con2; > +--error ER_DUP_ENTRY > +--reap > + > +disconnect con1; > +disconnect con2; > + > +connection default; > + > +select * from t1; > + > +DROP TABLE t1; > + > +set debug_sync= "RESET"; > + > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 262e791..da07075 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -368,6 +368,7 @@ void ha_partition::init_handler_variables() > part_share= NULL; > m_new_partitions_share_refs.empty(); > m_part_ids_sorted_by_num_of_records= NULL; > + m_cu_errkey= (uint) -1; rather confusing name. Could you de-confuse it? Why not simply m_errkey? or m_err_dupkey, or whatever and why did you make it unsigned? you need to cast in every second occurence of it :) > m_partitions_to_open= NULL; > > m_range_info= NULL; > @@ -4198,6 +4199,95 @@ void ha_partition::try_semi_consistent_read(bool yes) > } > > > +int ha_partition::check_files_for_key(uchar *key, int n_key, > + int part_begin, int part_end, > + int part_to_skip, > + int *res) > +{ > + DBUG_ASSERT(inited == NONE || > + (inited == RND && !m_scan_value)); > + > + for (int n=part_begin; n < part_end; n++) > + { > + handler *f= m_file[n]; > + init_stat sav_inited; > + > + if ((int) n == part_to_skip) > + continue; > + > + if ((sav_inited= f->inited) == RND) > + f->ha_rnd_end(); When can this happen? > + > + *res= f->index_read_idx_map(m_part_info->table->record[0], > + n_key, key, HA_WHOLE_KEY, HA_READ_KEY_EXACT); > + > + f->ha_end_keyread(); > + > + if (sav_inited == RND) > + { > + int ires= f->ha_rnd_init(FALSE); > + if (ires) > + *res= ires; > + } > + > + if (*res == HA_ERR_KEY_NOT_FOUND) > + continue; > + > + if (*res == 0) > + *res= HA_ERR_FOUND_DUPP_KEY; > + > + m_last_part= n; > + m_cu_errkey= n_key; > + return 1; > + } > + > + *res= 0; > + return 0; > +} > + > + > +int ha_partition::check_uniques_insert(uchar *buf, > + int part_begin, int part_end, > + KEY **dup_key, > + int *res) > +{ > + uint n_key; > + > + for (n_key=0; n_key < m_part_info->n_uniques_to_check; n_key++) > + { > + uchar *cbuf= m_part_info->unique_key_buf[0]; > + KEY *ckey= m_part_info->uniques_to_check[n_key]; > + uint n; > + > + for (n=0; n < ckey->user_defined_key_parts; n++) > + { can you use key_copy() here? > + const KEY_PART_INFO *cpart= ckey->key_part + n; > + uint maybe_null= MY_TEST(cpart->null_bit); > + Field *f= cpart->field; > + my_ptrdiff_t ofs= buf-table->record[0]; > + > + f->move_field_offset(ofs); > + if (maybe_null) > + cbuf[0]= f->is_null(); > + f->get_key_image(cbuf+maybe_null, cpart->length, Field::itRAW); > + cbuf+= cpart->store_length; > + f->move_field_offset(-ofs); > + } > + > + if (check_files_for_key(m_part_info->unique_key_buf[0], > + table->key_info - ckey, > + part_begin, part_end, -1, res)) > + { > + *dup_key= ckey; > + return 1; > + } > + } > + > + *res= 0; > + return 0; > +} > + > + > /**************************************************************************** > MODULE change record > ****************************************************************************/ > @@ -4307,7 +4398,41 @@ int ha_partition::write_row(uchar * buf) > start_part_bulk_insert(thd, part_id); > > tmp_disable_binlog(thd); /* Do not replicate the low-level changes. */ > + > + /* > + Check unique keys (if there is any) for duplications in > + partitions 0 .. inserted partition, then > + do the write_row then check the unique in > + partitions inserted partition +1 .. m_tot_parts. > + We do so to keep the order of locking always same to > + avoid deadlocks. > + */ > + if (table->part_info->n_uniques_to_check && > + check_uniques_insert(buf, 0, part_id, &dup_key, &error)) > + { > + goto exit; > + } > + > error= m_file[part_id]->ha_write_row(buf); > + > + DEBUG_SYNC(thd, "berfore_part_unique_check"); > + > + if (!error && table->part_info->n_uniques_to_check && > + check_uniques_insert(buf, part_id+1, m_tot_parts, &dup_key, &error)) > + { > + /* > + Errors here are ignored, as the error already found. > + and i don't have a good idea what to do if things go > + wrong here. > + */ > + if (!(m_file[part_id]->ha_index_read_idx_map(table->record[1], > + dup_key - table->key_info, table->part_info->unique_key_buf[0], > + HA_WHOLE_KEY, HA_READ_KEY_EXACT))) 1. why do you need ha_index_read_idx_map here? you've just did ha_write_row, isn't it enough? 2. if the table is transational and no IGNORE was used, you don't need to delete anything, the statement will be rolled back anyway. an easy optimization. > + { > + (void) m_file[part_id]->ha_delete_row(buf); > + } > + } > + > if (have_auto_increment && !table->s->next_number_keypart) > set_auto_increment_if_higher(table->next_number_field); > reenable_binlog(thd); > @@ -4317,6 +4442,59 @@ int ha_partition::write_row(uchar * buf) > table->auto_increment_field_not_null= saved_auto_inc_field_not_null; > DBUG_RETURN(error); > } > + > + > +int ha_partition::check_uniques_update(const uchar *old_data, > + const uchar *new_data, > + int new_part_id, int *res) > +{ > + uint n_key; > + > + for (n_key=0; n_key < m_part_info->n_uniques_to_check; n_key++) > + { > + uchar *buf0= m_part_info->unique_key_buf[0]; > + uchar *buf1= m_part_info->unique_key_buf[1]; > + KEY *ckey= m_part_info->uniques_to_check[n_key]; > + uint n; > + > + for (n=0; n < ckey->user_defined_key_parts; n++) > + { > + const KEY_PART_INFO *cpart= ckey->key_part + n; > + uint maybe_null= MY_TEST(cpart->null_bit); > + Field *f= cpart->field; > + my_ptrdiff_t dif; > + > + dif= old_data - table->record[0]; > + f->move_field_offset(dif); > + if (maybe_null) > + buf0[0]= f->is_null(); > + f->get_key_image(buf0+maybe_null, cpart->length, Field::itRAW); > + buf0+= cpart->store_length; > + f->move_field_offset(-dif); > + > + dif= new_data - table->record[0]; > + f->move_field_offset(dif); > + if (maybe_null) > + buf1[0]= f->is_null(); > + f->get_key_image(buf1+maybe_null, cpart->length, Field::itRAW); > + buf1+= cpart->store_length; > + f->move_field_offset(-dif); > + } > + > + if (memcmp(m_part_info->unique_key_buf[0], m_part_info->unique_key_buf[1], > + buf0 - m_part_info->unique_key_buf[0]) == 0) > + { > + /* Key did not change. */ > + continue; > + } > + > + if (check_files_for_key(m_part_info->unique_key_buf[1], > + table->key_info - ckey, 0, m_tot_parts, new_part_id, res)) How does that work? I'd expect you to do like you did for write_row: - check from 0 to new_part_id, skipping old_part_id, - insert the row - check from new_part_id+1 to m_last_part, skipping old_part_id. > + return 1; > + } > + > + return 0; > +} > > > /* > @@ -5754,11 +5935,14 @@ int ha_partition::index_read_idx_map(uchar *buf, uint index, > get_partition_set(table, buf, index, &m_start_key, &m_part_spec); > > /* > - We have either found exactly 1 partition > + If there is no 'unbound' unique keys where not all keyparts > + are partition definition fields, > + we have either found exactly 1 partition I don't understand. It's HA_READ_KEY_EXACT, so it can never find more than one row in a unique key, and one row can never be in more than one partition. Or do you mean unique key over nullable column that can have many null values? Then mention it in the comment, please. > (in which case start_part == end_part) > - or no matching partitions (start_part > end_part) > + or no matching partitions (start_part > end_part), > */ > - DBUG_ASSERT(m_part_spec.start_part >= m_part_spec.end_part); > + DBUG_ASSERT(m_part_spec.start_part >= m_part_spec.end_part || > + m_part_info->n_uniques_to_check); > /* The start part is must be marked as used. */ > DBUG_ASSERT(m_part_spec.start_part > m_part_spec.end_part || > bitmap_is_set(&(m_part_info->read_partitions), > @@ -8315,15 +8499,20 @@ int ha_partition::info(uint flag) > { > handler *file= m_file[m_last_part]; > DBUG_PRINT("info", ("info: HA_STATUS_ERRKEY")); > - /* > - This flag is used to get index number of the unique index that > - reported duplicate key > - We will report the errkey on the last handler used and ignore the rest > - Note: all engines does not support HA_STATUS_ERRKEY, so set errkey. > - */ > - file->errkey= errkey; > - file->info(HA_STATUS_ERRKEY | no_lock_flag); > - errkey= file->errkey; > + if ((int) m_cu_errkey >= 0) > + errkey= m_cu_errkey; > + else > + { > + /* > + This flag is used to get index number of the unique index that > + reported duplicate key > + We will report the errkey on the last handler used and ignore the rest > + Note: all engines does not support HA_STATUS_ERRKEY, so set errkey. I don't understand what this means (I know it's an old comment, but still) Is it "not all engines support HA_STATUS_ERRKEY"? If yes, please change it to say that. > + */ > + file->errkey= errkey; > + file->info(HA_STATUS_ERRKEY | no_lock_flag); > + errkey= file->errkey; > + } > } > if (flag & HA_STATUS_TIME) > { > diff --git a/sql/partition_info.h b/sql/partition_info.h > index e00a2c4..eeac0d9 100644 > --- a/sql/partition_info.h > +++ b/sql/partition_info.h > @@ -283,6 +283,14 @@ class partition_info : public Sql_alloc > bool is_auto_partitioned; > bool has_null_value; > bool column_list; // COLUMNS PARTITIONING, 5.5+ > + /* > + Unique keys that don't have all the partitioning fields in them > + need to be checked when INSERT/UPDATE. > + So they are collected here. > + */ > + KEY **uniques_to_check; > + uint n_uniques_to_check; > + uchar *unique_key_buf[2]; Is it safe? Isn't partition_info shared between many handlers so that every handler can only use it read-only? it's not quite clear to me. it seems that every TABLE has its own partition_info. But cloned handlers all share the same partition_info. > > partition_info() > : get_partition_id(NULL), get_part_partition_id(NULL), > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 3133b94..6cd46dd 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -988,15 +945,15 @@ static bool check_unique_keys(TABLE *table) > check_fields_in_PF(table->part_info->full_part_field_array, > &all_fields, &some_fields); > clear_indicator_in_key_fields(table->key_info+i); > - if (unlikely(!all_fields)) > + if (!all_fields) > { > - my_error(ER_UNIQUE_KEY_NEED_ALL_FIELDS_IN_PF,MYF(0),"UNIQUE INDEX"); you can rename ER_UNIQUE_KEY_NEED_ALL_FIELDS_IN_PF to ER_UNUSED with the next free number > - result= TRUE; > - break; > + ++keys_found; > + if (table->part_info->n_uniques_to_check >= keys_found) > + table->part_info->uniques_to_check[keys_found-1]= table->key_info+i; > } > } > } > - DBUG_RETURN(result); > + DBUG_RETURN(keys_found); > } > > > @@ -2798,6 +2765,47 @@ bool partition_key_modified(TABLE *table, const MY_BITMAP *fields) > } > > > + /* > + Check if there are unique keys that not entirely 'inside' the partition > + key and if any fields of such keys are modified. > + If it's so, it usually means we have to use tamporary storage for records "temporary" > + handling the UPDATE command. > + > + SYNOPSIS please, use doxygen format for new functions > + partition_unique_modified > + table TABLE object for which partition fields are set-up > + fields Bitmap representing fields to be modified > + > + RETURN VALUES > + TRUE Need special handling of UPDATE > + FALSE Normal UPDATE handling is ok > +*/ > + > +bool partition_unique_modified(TABLE *table, const MY_BITMAP *fields) > +{ > + partition_info *part_info= table->part_info; > + DBUG_ENTER("partition_unique_modified"); > + > + if (!part_info) > + DBUG_RETURN(FALSE); > + > + if (part_info->uniques_to_check == 0) > + DBUG_RETURN(FALSE); > + > + for (uint n_key=0; n_key < part_info->n_uniques_to_check; n_key++) > + { > + KEY *ckey= part_info->uniques_to_check[n_key]; > + for (uint n_part=0; n_part < ckey->user_defined_key_parts; n_part++) > + { > + if (bitmap_is_set(fields, ckey->key_part[n_part].field->field_index)) > + DBUG_RETURN(TRUE); > + } > + } > + > + DBUG_RETURN(FALSE); > +} > + > + > /* > A function to handle correct handling of NULL values in partition > functions. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 0f97f6b8398: MDEV-17346 parallel slave start and stop races to workers disappeared
by Kristian Nielsen 05 Oct '18

05 Oct '18

Hi Andrei! andrei.elkin(a)pp.inet.fi writes: > revision-id: 0f97f6b8398054ccb0507fbacc76c9deeddd47a4 (mariadb-10.1.35-71-g0f97f6b8398) > author: Andrei Elkin > timestamp: 2018-10-03 15:42:12 +0300 > message: > > MDEV-17346 parallel slave start and stop races to workers disappeared Ooh, that's a nice catch. The code around slave start/stop really is terribly tricky, glad that you were able to sort this one out. Patch looks good to push. What about test case? I assume this was caught from occasional random failures in buildbot? That should be ok as test case for this bug, I think. It can be a _lot_ of work to write reliable tests for stuff like this, and the start/stop slave is stressed quite a bit by the existing replication tests. > The bug appears as a slave SQL thread hanging in > rpl_parallel_thread_pool::get_thread() while there are no slave worker > threads to awake it. > > The hang could occur at parallel slave worker pool activation by a > "new" started SQL thread when the pool was concurrently deactivated by > being terminated "old" SQL thread. At reading the current pool size > the SQL thread did not employ necessary protection designed by > MDEV-9573. The pool can't be deactivated when there is an active slave > but the "new" slave might set its active status too late while seeing > the pool still non-empty. The end product of four computaional events > > Old_slave:any_slave_sql_running() => 0 > New_slave:slave_running= "active" > New_slave:global_rpl_thread_pool.size > 0 => true > Old_slave:global_rpl_thread_pool.size := 0 > > could led to the observed hang. The new SQL thread proceeds to scheduling > having all workers gone. > > Fixed with making the SQL thread at the pool activation first > to grab the same lock as potential deactivator also does prior > to destroy the pool. At first I did not see how this fixes the problem. But it is because of the extra check inside rpl_parallel_change_thread_count(): if (!new_count && !force) { if (any_slave_sql_running()) { pool_mark_not_busy(pool); return 0; // Ok to not resize pool } } So one possibility is that New_slave has set running=active when Old_slave hits that test. Then Old_slave will leave the pool intact thanks to this test. The other possibility is that New_slave has not yet set running=active at this point, so Old_slave can empty the pool. But then Old_slave already marked the pool busy. So the patch will ensure that New_slave will see that the pool has been emptied, and will know to re-populate it before starting. Seems solid. Thanks for fixing! - Kristian. > --- > sql/rpl_parallel.cc | 25 ++++++++++++++++++++++--- > 1 file changed, 22 insertions(+), 3 deletions(-) > > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 35cddee6d4d..8fef2d66635 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1617,13 +1617,32 @@ int rpl_parallel_resize_pool_if_no_slaves(void) > } > > > +/** > + Pool activation is preceeded by taking a "lock" of pool_mark_busy > + which guarantees the number of running slaves drops to zero atomicly > + with the number of pool workers. > + This resolves race between the function caller thread and one > + that may be attempting to deactivate the pool. > +*/ > int > rpl_parallel_activate_pool(rpl_parallel_thread_pool *pool) > { > + int rc= 0; > + > + if ((rc= pool_mark_busy(pool, current_thd))) > + return rc; // killed > + > if (!pool->count) > - return rpl_parallel_change_thread_count(pool, opt_slave_parallel_threads, > - 0); > - return 0; > + { > + pool_mark_not_busy(pool); > + rc= rpl_parallel_change_thread_count(pool, opt_slave_parallel_threads, > + 0); > + } > + else > + { > + pool_mark_not_busy(pool); > + } > + return rc; > }

1 0

Re: [Maria-developers] c45b4a7: MDEV-17200 - pthread_detach called for already detached threads
by Sergei Golubchik 05 Oct '18

05 Oct '18

Hi, Sergey! ok to push! On Sep 21, Sergey Vojtovich wrote: > revision-id: c45b4a774b6d1404a080a1c1759f780fa78f223b (mariadb-10.1.35-65-gc45b4a7) > parent(s): 82524239c48a2603f115a6117e061751f6427d08 > author: Sergey Vojtovich > committer: Sergey Vojtovich > timestamp: 2018-09-21 16:24:23 +0400 > message: > > MDEV-17200 - pthread_detach called for already detached threads > > pthread_detach_this_thread() was intended to be defined to something > meaningful only on some ancient unixes, which don't have > pthread_attr_setdetachstate() defined. Otherwise, on normal unixes, > threads are created detached in the first place. > > This was broken in 0f01bf267680244ec488adaf65a42838756ed48e so that > we started calling pthread_detach() for already detached threads. > Intention was to detach aria checkpoint thread. > > However in 87007dc2f71634cc460271eb277ad851ec69c04b aria service threads > were made joinable with appropriate handling, which makes breaking > revision unneccessary. > > Revert remnants of 0f01bf267680244ec488adaf65a42838756ed48e, so that > pthread_detach_this_thread() is meaningful only on some ancient unixes > again. > > --- > include/my_pthread.h | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/include/my_pthread.h b/include/my_pthread.h > index 6b830ca..ae2f912 100644 > --- a/include/my_pthread.h > +++ b/include/my_pthread.h > @@ -184,7 +184,7 @@ int pthread_cancel(pthread_t thread); > #define pthread_key(T,V) pthread_key_t V > #define my_pthread_getspecific_ptr(T,V) my_pthread_getspecific(T,(V)) > #define my_pthread_setspecific_ptr(T,V) pthread_setspecific(T,(void*) (V)) > -#define pthread_detach_this_thread() { pthread_t tmp=pthread_self() ; pthread_detach(tmp); } > +#define pthread_detach_this_thread() > #define pthread_handler_t EXTERNC void * > typedef void *(* pthread_handler)(void *); > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 0f97f6b8398: MDEV-17346 parallel slave start and stop races to workers disappeared
by andrei.elkin＠pp.inet.fi 03 Oct '18

03 Oct '18

Kristian, hello. Perhaps you could review this worker pool patch though it actually follows up changes made by MDEV-9573, by Monty. Cheers, Andrei > revision-id: 0f97f6b8398054ccb0507fbacc76c9deeddd47a4 (mariadb-10.1.35-71-g0f97f6b8398) > parent(s): 1fc5a6f30c3a9c047dcf9a36b00026d98f286f6b > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-10-03 15:42:12 +0300 > message: > > MDEV-17346 parallel slave start and stop races to workers disappeared > > The bug appears as a slave SQL thread hanging in > rpl_parallel_thread_pool::get_thread() while there are no slave worker > threads to awake it. > > The hang could occur at parallel slave worker pool activation by a > "new" started SQL thread when the pool was concurrently deactivated by > being terminated "old" SQL thread. At reading the current pool size > the SQL thread did not employ necessary protection designed by > MDEV-9573. The pool can't be deactivated when there is an active slave > but the "new" slave might set its active status too late while seeing > the pool still non-empty. The end product of four computaional events > > Old_slave:any_slave_sql_running() => 0 > New_slave:slave_running= "active" > New_slave:global_rpl_thread_pool.size > 0 => true > Old_slave:global_rpl_thread_pool.size := 0 > > could led to the observed hang. The new SQL thread proceeds to scheduling > having all workers gone. > > Fixed with making the SQL thread at the pool activation first > to grab the same lock as potential deactivator also does prior > to destroy the pool. > > --- > sql/rpl_parallel.cc | 25 ++++++++++++++++++++++--- > 1 file changed, 22 insertions(+), 3 deletions(-) > > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 35cddee6d4d..8fef2d66635 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1617,13 +1617,32 @@ int rpl_parallel_resize_pool_if_no_slaves(void) > } > > > +/** > + Pool activation is preceeded by taking a "lock" of pool_mark_busy > + which guarantees the number of running slaves drops to zero atomicly > + with the number of pool workers. > + This resolves race between the function caller thread and one > + that may be attempting to deactivate the pool. > +*/ > int > rpl_parallel_activate_pool(rpl_parallel_thread_pool *pool) > { > + int rc= 0; > + > + if ((rc= pool_mark_busy(pool, current_thd))) > + return rc; // killed > + > if (!pool->count) > - return rpl_parallel_change_thread_count(pool, opt_slave_parallel_threads, > - 0); > - return 0; > + { > + pool_mark_not_busy(pool); > + rc= rpl_parallel_change_thread_count(pool, opt_slave_parallel_threads, > + 0); > + } > + else > + { > + pool_mark_not_busy(pool); > + } > + return rc; > } > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Track of memory consumption.
by jerome brauge 02 Oct '18

02 Oct '18

Hi, I'd like to know what is the best way to track memory consumption /leak in mariadb. Our application use intensively stored procedure and with only 35 connection we have to restart mariadb each 1.5 hours because it use 15Go of memory very fast. Mysqladmin doesn't show excessive memory consumption : | Max_used_connections | 42 | | Memory_used | 2422894288 | | Memory_used_initial | 291786624 | But [root@vmdevmdb ~]# ps -up 20039 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND mysql 20039 10.5 73.8 14574168 12000852 ? Ssl 14:30 11:19 /usr/sbin/mysqld --> 12Go of resident memory in only 11 minutes of cpu. Best regards Jérôme. Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de ses destinataires. Toute utilisation ou diffusion, même partielle, non autorisée est interdite. Tout message électronique est susceptible d'altération; CEGID décline donc toute responsabilité au titre de ce message. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'avertir l'expéditeur. This message and any attachments are confidential and intended solely for the addressees. Any unauthorized use or disclosure, either whole or partial is prohibited. E-mails are susceptible to alteration; CEGID shall therefore not be liable for the content of this message. If you are not the intended recipient of this message, please delete it and notify the sender.

1 0

[Maria-developers] CONC-312; caching_sha2_plugin
by Michal Schorm 02 Oct '18

02 Oct '18

Hello, Can I get some explanation, why was https://jira.mariadb.org/browse/CONC-312 closed as WON'T FIX ? -- MySQL 8.0 implemented "caching_sha2_password" as its default authentification plugin / method. MariaDB doesn't have support for it so mariadb-connector-c cannot be used for MySQL 8. That makes problems for nearly every Linux OS, beacuse they chose MariaDB over MySQL as the default implemetation. I'd like to know the MariaDB stance to it. Do you see it as an issue as well? Are there any plans to deal with it? How? Michal Schorm -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

2 2

Re: [Maria-developers] e11e0e6c8b9: MDEV-17167 - InnoDB: Failing assertion: table->get_ref_count() == 0 upon
by Sergey Vojtovich 24 Sep '18

24 Sep '18

Hi Sergei, On Mon, Sep 24, 2018 at 03:02:40PM +0200, Sergei Golubchik wrote: > Hi, Sergey! > > On Sep 12, Sergey Vojtovich wrote: > > revision-id: e11e0e6c8b9a85ca7f16c6b9b28fb505d464a96e (mariadb-10.3.7-186-ge11e0e6c8b9) > > parent(s): f2f661b848c02ebd7c444ba645b26416d3e4d2ad > > author: Sergey Vojtovich > > committer: Sergey Vojtovich > > timestamp: 2018-09-12 16:36:45 +0400 > > message: > > > > MDEV-17167 - InnoDB: Failing assertion: table->get_ref_count() == 0 upon > > truncating a temporary table > > > > TRUNCATE expects only one TABLE instance (which is used by TRUNCATE > > itself) to be open. However this requirement wasn't enforced after > > "MDEV-5535: Cannot reopen temporary table". > > > > Fixed by closing unused table instances before performing TRUNCATE. > > > diff --git a/sql/sql_class.h b/sql/sql_class.h > > index 6256522bf5c..e7dcd997e05 100644 > > --- a/sql/sql_class.h > > +++ b/sql/sql_class.h > > @@ -4612,6 +4612,7 @@ class THD :public Statement, > > > > TMP_TABLE_SHARE* save_tmp_table_share(TABLE *table); > > void restore_tmp_table_share(TMP_TABLE_SHARE *share); > > + void close_unused_temporary_table_instances(const TABLE_LIST *tl); > > > > private: > > /* Whether a lock has been acquired? */ > > diff --git a/sql/sql_truncate.cc b/sql/sql_truncate.cc > > index 201825d4593..695fcb538f9 100644 > > --- a/sql/sql_truncate.cc > > +++ b/sql/sql_truncate.cc > > @@ -401,6 +401,8 @@ bool Sql_cmd_truncate_table::truncate_table(THD *thd, TABLE_LIST *table_ref) > > /* In RBR, the statement is not binlogged if the table is temporary. */ > > binlog_stmt= !thd->is_current_stmt_binlog_format_row(); > > > > + thd->close_unused_temporary_table_instances(table_ref); > > Is it only TRUNCATE that needs it? What about ALTER? REPAIR? Other > similar commands? Right, do you think it is worth to exapnd scope of this bug and look for other possible issues? > > +*/ > > + > > +void THD::close_unused_temporary_table_instances(const TABLE_LIST *tl) > > +{ > > + TMP_TABLE_SHARE *share= find_tmp_table_share(tl); > > + > > + if (share) > > + { > > + Share_free_tables::List purge_tables; > > + All_share_tables_list::Iterator tables_it(share->all_tmp_tables); > > + > > + while (TABLE *table= tables_it++) > > + { > > + if (table->query_id == 0) > > + purge_tables.push_front(table); > > + } > > + > > + while (TABLE *table= purge_tables.pop_front()) > > + { > > + share->all_tmp_tables.remove(table); > > + free_temporary_table(table); > > + } > > Why are you doing it in two loops? Because free_temporary_table() > invalidates the iterator? Not free_temporary_table(), but rather share->all_tmp_tables.remove(table). OTOH it seems to be safe to remove current element with it++, but not with ++it. I'll try to remove extra loop if it works. Thanks, Sergey

2 1

[Maria-developers] unsubscribe
by Valentin Rakush 17 Sep '18

17 Sep '18

-- Best Regards, Valentin Rakush.

1 0

[Maria-developers] Compiling default MariaDB with OpenSSL in Windows
by Braiam Peguero 14 Sep '18

14 Sep '18

Hi, Following Georg Richter recommendations for compiling MariaDB on Windows 64bit, with OpenSSL using the following cmake commands: cmake .. -DWITH_SSL=system -G "Visual Studio 15 2017 Win64" cmake --build . --config relwithdebinfo --target package However, several recipes fail to build, for the following errors: "builddir\mysys_ssl\mysys_ssl.vcxproj" (default target) (8) -> (ClCompile target) -> c:\users\braiam\src\mariadb-10.3.9\mysys_ssl\my_crypt.cc(42): error C3861: 'bzero': identifier not found [C:\Users\br aiam\src\mariadb-10.3.9\bld\mysys_ssl\mysys_ssl.vcxproj] and "builddir\package.vcxproj" (default target) (1) -> "builddir\ALL_BUILD.vcxproj" (default target) (3) -> "builddir\libmariadb\sha256_password.vcxproj" (default target) (205) -> (Link target) -> sha256_pw.obj : error LNK2019: unresolved external symbol BIO_free referenced in function auth_sha256_client [C:\User s\braiam\src\mariadb-10.3.9\bld\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol BIO_new_mem_buf referenced in function auth_sha256_client [ builddir\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol RSA_size referenced in function auth_sha256_client [C:\User s\braiam\src\mariadb-10.3.9\bld\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol RSA_public_encrypt referenced in function auth_sha256_clien t [builddir\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol RSA_free referenced in function auth_sha256_client [C:\User s\braiam\src\mariadb-10.3.9\bld\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol PEM_read_bio_RSA_PUBKEY referenced in function auth_sha256_ client [builddir\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol ERR_clear_error referenced in function auth_sha256_client [ builddir\libmariadb\sha256_password.vcxproj] builddir\libmariadb\RelWithDebInfo\sha256_password.dll : fatal error LNK1120: 7 unresol ved externals [builddir\libmariadb\sha256_password.vcxproj] Using the OpenSSL 1.1.0i 64bit binaries from Shinning Light Productions. Using the 1.0.2p version produce even more linking errors. I also used FireDaemon but failed. Is there a recommended Windows binary for compiling? Or do I need to compile my own openssl? -- Braiam Peguero

2 3

Re: [Maria-developers] [Commits] 3c7e2540f63: MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion
by Sergey Petrunia 12 Sep '18

12 Sep '18

Hi Varun, Looking at the testcase name, I have found Serg's patch for MDEV-4285. His patch skips sorting if the keys have zero length. He does it at a late stage, in Filesort_buffer::sort_buffer(). His patch doesn't cover the example of MDEV-17020, because the testcase in MDEV-17020 uses LIMIT and so uses a Priority Queue (and not Filesort_buffer::sort_buffer which uses qsort). However, your patch doesn't cover all cases either. There are other ways to produce zero-length keys. I have found this one: SELECT * FROM t1 ORDER BY 'foo', substring(pk, 0) LIMIT 2; it crashes in the same way as the test for MDEV-17020 used to do. Possible options: 1. Find all Items that might produce zero-length keys. It's difficult not to miss anything. 2. Instead of #1, add a universal logic "if sort key length is 0, don't sort". Is this doable? 3. Extend the Serg's approach. We cannot use his code directly ("dont sort"), because Priority Queue API doesn't easily allow it, but perhaps we could add a comparison function that will allow to compare zero-length keys? Or make them 1-byte long with a dummy value (do we need the sorting to be stable??) On Fri, Aug 31, 2018 at 12:18:08AM +0530, Varun wrote: > revision-id: 3c7e2540f63c6801a0d0a27fbea7c934bb61221f (mariadb-10.0.36-13-g3c7e2540f63) > parent(s): b3c320bb0b93e516cda4db277cfa3efeef48c988 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-08-31 00:14:20 +0530 > message: > > MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion > > We have a case here where the the entire sort key is empty and so sorting is not needed. > The crash happens because filesort expects the sort key to have length > 0. > The case here is like ORDER BY CONVERT(a,char(0)), which if evaluated is a constant so > can be removed from the ORDER BY clause > > --- > mysql-test/r/order_by_zerolength-4285.result | 28 +++++++++++++++++----------- > mysql-test/t/order_by_zerolength-4285.test | 13 +++++++++++++ > sql/item_timefunc.h | 6 ++++++ > 3 files changed, 36 insertions(+), 11 deletions(-) > > diff --git a/mysql-test/r/order_by_zerolength-4285.result b/mysql-test/r/order_by_zerolength-4285.result > index f60ce7d90c7..f77b284df3c 100644 > --- a/mysql-test/r/order_by_zerolength-4285.result > +++ b/mysql-test/r/order_by_zerolength-4285.result > @@ -12,15 +12,21 @@ pk > 8 > 9 > 10 > -Warnings: > -Warning 1292 Truncated incorrect CHAR(0) value: '1' > -Warning 1292 Truncated incorrect CHAR(0) value: '2' > -Warning 1292 Truncated incorrect CHAR(0) value: '3' > -Warning 1292 Truncated incorrect CHAR(0) value: '4' > -Warning 1292 Truncated incorrect CHAR(0) value: '5' > -Warning 1292 Truncated incorrect CHAR(0) value: '6' > -Warning 1292 Truncated incorrect CHAR(0) value: '7' > -Warning 1292 Truncated incorrect CHAR(0) value: '8' > -Warning 1292 Truncated incorrect CHAR(0) value: '9' > -Warning 1292 Truncated incorrect CHAR(0) value: '10' > +drop table t1; > +# > +# MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion > +# > +set @save_sql_mode= @@sql_mode; > +SET @@sql_mode= ''; > +CREATE TABLE t1 (pk INT PRIMARY KEY); > +INSERT INTO t1 VALUES (1),(2); > +explain > +SELECT * FROM t1 ORDER BY 'foo', CONVERT(pk, CHAR(0)) LIMIT 2; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index NULL PRIMARY 4 NULL 2 Using index > +SELECT * FROM t1 ORDER BY 'foo', Cast(pk as CHAR(0)) LIMIT 2; > +pk > +1 > +2 > +set @@sql_mode= @save_sql_mode; > drop table t1; > diff --git a/mysql-test/t/order_by_zerolength-4285.test b/mysql-test/t/order_by_zerolength-4285.test > index 2fb58edd36d..f03d528320c 100644 > --- a/mysql-test/t/order_by_zerolength-4285.test > +++ b/mysql-test/t/order_by_zerolength-4285.test > @@ -6,3 +6,16 @@ insert into t1 values (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); > select * from t1 order by now(), cast(pk as char(0)); > drop table t1; > > +--echo # > +--echo # MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion > +--echo # > + > +set @save_sql_mode= @@sql_mode; > +SET @@sql_mode= ''; > +CREATE TABLE t1 (pk INT PRIMARY KEY); > +INSERT INTO t1 VALUES (1),(2); > +explain > +SELECT * FROM t1 ORDER BY 'foo', CONVERT(pk, CHAR(0)) LIMIT 2; > +SELECT * FROM t1 ORDER BY 'foo', Cast(pk as CHAR(0)) LIMIT 2; > +set @@sql_mode= @save_sql_mode; > +drop table t1; > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index 80b98758fb7..461c3feed8c 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -903,6 +903,12 @@ class Item_char_typecast :public Item_str_func > String *val_str(String *a); > void fix_length_and_dec(); > void print(String *str, enum_query_type query_type); > + table_map used_tables() const > + { > + if (!cast_length) > + return (table_map) 0L; > + return Item_str_func::used_tables(); > + } > }; > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 17540ad: MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>.
by Sergei Golubchik 04 Sep '18

04 Sep '18

Hi, Alexey! On Sep 03, Alexey Botchkov wrote: > revision-id: 17540adc625ff4ac0f00aa10796cb6e763838d18 (mariadb-10.1.35-25-g17540ad) > parent(s): 34e026f35a7fbeeaec412547ee207c880944ae68 > committer: Alexey Botchkov > timestamp: 2018-09-03 16:49:46 +0400 > message: > > MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>. > > LOCK view WRITE shouldn't block FLUSH view. > So we set the view's mdl_request type to it's tables. > > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 6bd6b6a..d7a2a6b 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1534,8 +1534,7 @@ bool mysql_make_view(THD *thd, TABLE_SHARE *share, TABLE_LIST *table, > for (tbl= view_main_select_tables; tbl; tbl= tbl->next_local) > { > tbl->lock_type= table->lock_type; > - tbl->mdl_request.set_type((tbl->lock_type >= TL_WRITE_ALLOW_WRITE) ? > - MDL_SHARED_WRITE : MDL_SHARED_READ); > + tbl->mdl_request.set_type(table->mdl_request.type); > } I agree it's looks like a proper fix. But I'm a bit cautious about possible side effects. A more conservative fix would be only change the lock level for TL_NO_READ_WRITE, like tbl->lock_type == TL_NO_READ_WRITE ? TL_NO_READ_WRITE : tbl->lock_type >= TL_WRITE_ALLOW_WRITE ? MDL_SHARED_WRITE : MDL_SHARED_READ (or with an if(), for readability). But I don't see whether it'll make any difference as compared to your fix. Please, run all tests locally, and I mean *all*, big, ps, embedded. And see if you can think of any cases where this change would cause a change in behavior. After that - ok to push! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] 2 pull requests
by Ming Lin 04 Sep '18

04 Sep '18

Hi Sergey, Could you take a look at these 2 PRs? MDEV-16768: https://github.com/MariaDB/server/pull/837 You approved this one, but not merged yet. MDEV-15816: https://github.com/MariaDB/server/pull/849 This one was already merged by facebook. Thanks, Ming

1 1

[Maria-developers] Binlog handling of faulty DDL query
by Sachin Setiya 03 Sep '18

03 Sep '18

Hi Everyone! Suppose this case CREATE USER user1@localhost IDENTIFIED BY 'BsG9#9.cem#!85', user2@localhost IDENTIFIED BY 'x'; user2 has too short passowrd which will give error (if we use security plugin) IN the case of *mariadb* we will create user1 and wont create user2 In the case of *mysql (8.0)* we will return error and we will not create any user Our approach create a lots of problem for binary log and replication(MDEV-14716, MDEV-16728) Because in Query_log_event::write() we write the whole query not the parts of it. So we have 2 options to solve this either write modified query into binlog or do what mysql does. So what you think , which approach we should use ? Or there is another better way ? -- Regards sachin

5 6

Re: [Maria-developers] 63ad6a9: MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>.
by Sergei Golubchik 02 Sep '18

02 Sep '18

Hi, Alexey! On Sep 02, Alexey Botchkov wrote: > revision-id: 63ad6a9e1a33ddd5547767b2894e09ae66196f69 (mariadb-10.1.35-23-g63ad6a9) > parent(s): 288212f489c8cd88c4cc98f8aecc3366c85a90be > committer: Alexey Botchkov > timestamp: 2018-09-02 09:24:33 +0400 > message: > > MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>. > > Check if the argument of the FLUSH TABLE is a VIEW and handle it > accordingly. > > diff --git a/mysql-test/r/flush.result b/mysql-test/r/flush.result > index b643510..2d7b81b 100644 > --- a/mysql-test/r/flush.result > +++ b/mysql-test/r/flush.result > @@ -496,3 +496,27 @@ flush relay logs,relay logs; > ERROR HY000: Incorrect usage of FLUSH and RELAY LOGS > flush slave,slave; > ERROR HY000: Incorrect usage of FLUSH and SLAVE > +# > +# MDEV-15890 Strange error message if you try to > +# FLUSH TABLES <view> after LOCK TABLES <view>. > +# > +CREATE TABLE t1 (qty INT, price INT); > +CREATE VIEW v1 AS SELECT qty, price, qty*price AS value FROM t1; > +LOCK TABLES v1 READ; > +FLUSH TABLES v1; > +ERROR HY000: Table 't1' was locked with a READ lock and can't be updated > +UNLOCK TABLES; > +LOCK TABLES v1 WRITE; > +FLUSH TABLES v1; > +ERROR HY000: Table 't1' was locked with a READ lock and can't be updated Why is that? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MariaDB v10 on AIX
by REIX, Tony 31 Aug '18

31 Aug '18

Hi, We are porting MariaDB v10(.3.7/9) on AIX now. First, I'd like to know if there are people porting (or who has ported) MariaDB on AIX in this mailing list. Second, we have found some issue dealing with: AIX, -O0, and inline, which leads to a symbol missing at link time: my_ulonglong2double . It deals with the code below. Since we are beginners with MariaDB, I'd like to have some comments and/or recommendations about how it should be fixed. By using "static __inline" instead of "inline" below, and adding it too to the ma_global.h definition of my_ulonglong2double() , we are able to fix it. However, probably that using: #define my_ulonglong2double(nr) ((double) nr) would be enough. Or adding my_port.c to the MYSYS_SOURCES variable in mysys/CMakeLists.txt would help: IF (AIX) SET (MYSYS_SOURCES ${MYSYS_SOURCES} my_port.c) ENDIF() however, that does not work, and I do not see my_port.c being added to the list of files to be added to mysys. Any idea why "IF (AIX)" does not work? (I'm also a CMake beginner). Thanks/Regards, Tony Reix mysys/my_port.c: #include "mysys_priv.h" #ifdef _AIX double my_ulonglong2double(unsigned long long nr) { return (double) nr; } mysys/mysys_priv.h: #include <my_global.h> mysys/CMakeLists.txt : SET(MYSYS_SOURCES ........ No my_port.c !! include/my_global.h: #ifdef _AIX /* By soren(a)t.dk */ #define ulonglong2double(A) my_ulonglong2double(A) C_MODE_START inline double my_ulonglong2double(unsigned long long A) { return (double)A; } C_MODE_END libmariadb/include/ma_global.h: #ifdef _AIX /* By soren(a)t.dk */ #define ulonglong2double(A) my_ulonglong2double(A) #ifdef __cplusplus extern "C" { #endif double my_ulonglong2double(unsigned long long A); déclaration externe #ifdef __cplusplus Cordialement, Tony Reix tony.reix(a)atos.net ATOS / Bull SAS ATOS Expert IBM Coop Architect & Technical Leader Office : +33 (0) 4 76 29 72 67 1 rue de Provence - 38432 Échirolles - France www.atos.net<https://mail.ad.bull.net/owa/redir.aspx?C=PvphmPvCZkGrAgHVnWGsdMcDKgzl_dEIs…>

2 1

Re: [Maria-developers] 44692f17a62: MDEV-15436: If log_bin and log_bin_index is different SST with rsync fails.
by Sergei Golubchik 31 Aug '18

31 Aug '18

Hi, Jan! On Aug 31, serg(a)mariadb.org wrote: > revision-id: 44692f17a621d7f0261d84b97a7334352463038c (mariadb-10.2.16-6-g44692f17a62) > parent(s): 7d0d934ca642e485b2c008727dc20c83e26cce10 > author: Jan Lindström > committer: Jan Lindström > timestamp: 2018-06-26 12:56:19 +0300 > message: > > MDEV-15436: If log_bin and log_bin_index is different SST with rsync fails. > > Problem was that in SST log_bin_index name and directory was not > handled and passed to rsync SST script. > > mysql-test/suite/galera/r/galera_sst_rsync2.result | 396 +++++++++++++++++++++ > mysql-test/suite/galera/t/galera_sst_rsync2.cnf | 15 + > mysql-test/suite/galera/t/galera_sst_rsync2.test | 12 + I didn't review the test, but I presume you've verified that it fails without the bug fix. > diff --git a/scripts/wsrep_sst_rsync.sh b/scripts/wsrep_sst_rsync.sh > index 05c99880d75..6886d2ce1e1 100644 > --- a/scripts/wsrep_sst_rsync.sh > +++ b/scripts/wsrep_sst_rsync.sh > @@ -132,6 +132,12 @@ then > BINLOG_FILENAME=$(basename $WSREP_SST_OPT_BINLOG) > fi > > +if ! [ -z $WSREP_SST_OPT_BINLOG_INDEX ] > +then > + BINLOG_INDEX_DIRNAME=$(dirname $WSREP_SST_OPT_BINLOG_INDEX) > + BINLOG_INDEX_FILENAME=$(basename $WSREP_SST_OPT_BINLOG_INDEX) > +fi > + > WSREP_LOG_DIR=${WSREP_LOG_DIR:-""} > # if WSREP_LOG_DIR env. variable is not set, try to get it from my.cnf > if [ -z "$WSREP_LOG_DIR" ]; then > @@ -204,12 +210,20 @@ then > OLD_PWD="$(pwd)" > cd $BINLOG_DIRNAME > > - binlog_files_full=$(tail -n $BINLOG_N_FILES ${BINLOG_FILENAME}.index) > + if ! [ -z $WSREP_SST_OPT_BINLOG_INDEX ] first, I think it'd be simpler to do, like if [ -z $WSREP_SST_OPT_BINLOG_INDEX ] then WSREP_SST_OPT_BINLOG_INDEX=$WSREP_SST_OPT_BINLOG.index fi it's enough to do it once before setting BINLOG_INDEX_DIRNAME and BINLOG_INDEX_FILENAME and you woudn't need if's here and below. second, you apparently are trying to take into account that binlog index may be a different directory. Please use this configuration in your test case, put binlog index in a different directory. otherwise ok. please, push after adjusting the test case to put binlog index in a different directory. > + binlog_files_full=$(tail -n $BINLOG_N_FILES ${BINLOG_FILENAME}.index) > + then > + cd $BINLOG_INDEX_DIRNAME > + binlog_files_full=$(tail -n $BINLOG_N_FILES ${BINLOG_INDEX_FILENAME}.index) > + fi > + > + cd $BINLOG_DIRNAME > binlog_files="" > for ii in $binlog_files_full > do > binlog_files="$binlog_files $(basename $ii)" > done > + > if ! [ -z "$binlog_files" ] > then > wsrep_log_info "Preparing binlog files for transfer:" > @@ -391,7 +405,11 @@ EOF > tar -xvf $BINLOG_TAR_FILE >&2 > for ii in $(ls -1 ${BINLOG_FILENAME}.*) > do > - echo ${BINLOG_DIRNAME}/${ii} >> ${BINLOG_FILENAME}.index > + if ! [ -z $WSREP_SST_OPT_BINLOG_INDEX ] > + echo ${BINLOG_DIRNAME}/${ii} >> ${BINLOG_FILENAME}.index > + then > + echo ${BINLOG_DIRNAME}/${ii} >> ${BINLOG_INDEX_DIRNAME}/${BINLOG_INDEX_FILENAME}.index > + fi > done > fi > cd "$OLD_PWD" Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 6be8bd6: MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>.
by Sergei Golubchik 31 Aug '18

31 Aug '18

Hi, Alexey! On Jun 24, Alexey Botchkov wrote: > revision-id: 6be8bd6ee05c6f18b6a6a0375f9bc2b68ef6c43e (mariadb-10.3.7-59-g6be8bd6) > parent(s): 1abd877e2df9e83bc1c2f5195796f427a35bd3f1 > committer: Alexey Botchkov > timestamp: 2018-06-24 23:12:20 +0400 > message: > > MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>. > > Check if the argument of the FLUSH TABLE is a VIEW and handle it > accordingly. Looks great! But please add more tests. A test with a write lock, and tests with locking the view and flushing the table or vice versa. After that ok to push! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] 10.2 and merges (e.g. TokuDB)
by Phil Sweeney 28 Aug '18

28 Aug '18

Hi there, It seems 10.2.6 is still shipping with TokuDB from PS 5.6.35-80 (Feb 8th 2017). I can't see a jira issue for a 10.2 merge since 10.2.4. Perhaps intentional to help stabilise 10.2 for release? 10.0.30 also shipped with the same Feb version (in March). 10.1.23 however was released in May and still has this version from Feb, which seems unfortunate, but I know it's because it has to flow through 10.0 first. I see 10.0.31 is due to get TokuDB from PS 5.6.36-82 And that should flow into 10.1.24 soon too. However, 10.2.7 isn't due until July, so by then TokuDB version will be 5 months old in 10.2 I'm waiting on a fairly important row count fix that should appear in the next Percona Server release. When it does, if the current merging strategy continues, do I have to first wait for 10.0.32 to get it in late July, then 10.1.26 in August, then 10.2 after that? Or with 10.0, 10.1 and 10.2 all being supported and needing merges, is there a new strategy on the cards to shorten this lag? Thanks! Phil

3 7

[Maria-developers] MDEV-371 Long unique Index
by Sachin Setiya 28 Aug '18

28 Aug '18

Hi Serg! Please review the patch for Unique Index For Long (MDEV-371) (It is in bb-10.4-blob-index branch) Architecture:- We will create a virtual column DB_ROW_HASH_X (X is any number which makes this field name unique) for each long unique column which will store hash. At the time of ha_write_row we will check for hash collusions if we find same hash then we will retrieve the blob and then compare byte by byte. key_info for long unique have one extra key_part which will point to hash field. For example unique(a,b) key will have 3 key_parts :- a ,b DB_ROW_HASH_X But on server layer we can just see a,b parts on storage layer we can just see DB_ROW_HASH_X key_part. I have created 2 function setup_table_hash and resetup_table which helps in this transition. If user specify length of blob then we use LEFT function to trim the value. -- Regards Sachin Setiya Software Engineer at MariaDB

1 0

Re: [Maria-developers] [Commits] f522fb9: MDEV-16934 Query with very large IN clause lists runs slowly
by Sergey Petrunia 17 Aug '18

17 Aug '18

Hello Igor, I am debugging with the patch and I see that PARAM::range_count is not reset after the eq_ranges_exceeds_limit() call. This causes different kinds of undesired effects. For example, here create table ten(a int); insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table t1 (pk int, kp1 int, kp2 int, kp3 int, col1 int, key(kp1,kp2,kp3), primary key(pk)); insert into t1 select A.a + 10*B.a+100* C.a, A.a, B.a , C.a, 1234 from ten A, ten B, ten C; alter table t1 add key3 int, add key2 int; update t1 set key3=kp1; update t1 set key2=kp1; alter table t1 add key(key2), add key(key3); set eq_range_index_dive_limit=10; explain format=json select 1 from t1 force index (key2, key3) where key3=1 or key2=1; index_merge will use Sort-Union instead of Union. I assume we are targeting only IN (...) cases? Both our and MySQL's fixes will still do many records_in_range calls for queries like keypart1 IN (1,2,3,...) AND keypart2 IN (1,2,3,...) AND keypart3 >= 10 (because here the ranges will not be equality ranges). I assume this is outside of the scope of this fix? If so, then the patch is ok to push after PARAM::range_count issue is resolved. On Wed, Aug 15, 2018 at 07:48:30PM -0700, IgorBabaev wrote: > revision-id: f522fb9f63f88195bccbbc0326a4a318cc2d3db0 (mariadb-10.2.16-50-gf522fb9) > parent(s): 3c141e319ab29afdfe843553da385fe5d981906e > author: Igor Babaev > committer: Igor Babaev > timestamp: 2018-08-15 19:48:30 -0700 > message: > > MDEV-16934 Query with very large IN clause lists runs slowly > > This patch introduces support for the system variable eq_range_index_dive_limit > that existed in MySQL starting from 5.6. The variable sets a limit for > index dives into equality ranges. Index dives are performed by optimizer > to estimate the number of rows in range scans. Index dives usually provide > good estimate but they are pretty expensive. To estimate the number of rows > in equality ranges statistical data on indexes can be employed. Its usage gives > not so good estimates but it's cheap. So if the number of equality dives > required by an index scan exceeds the set limit no dives for equality > ranges are performed by the optimizer for this index. > > As the new system variable is introduced in a stable version the default > value for it is set to a special value meaning there is no limit for the number > of index dives performed by the optimizer. > > The patch partially uses the MySQL code for WL 5957 > 'Statistics-based Range optimization for many ranges'. > > --- > mysql-test/r/mysqld--help.result | 6 +++ > mysql-test/r/range.result | 41 +++++++++++++++++ > mysql-test/r/range_mrr_icp.result | 41 +++++++++++++++++ > .../sys_vars/r/sysvars_server_embedded.result | 14 ++++++ > .../sys_vars/r/sysvars_server_notembedded.result | 14 ++++++ > mysql-test/t/range.test | 33 ++++++++++++++ > sql/multi_range_read.cc | 11 +++++ > sql/opt_range.cc | 27 +++++++++++ > sql/opt_range.h | 4 +- > sql/opt_range_mrr.cc | 53 +++++++++++++++------- > sql/sql_class.h | 1 + > sql/sql_statistics.h | 2 +- > sql/sys_vars.cc | 10 ++++ > 13 files changed, 238 insertions(+), 19 deletions(-) > > diff --git a/mysql-test/r/mysqld--help.result b/mysql-test/r/mysqld--help.result > index 3b7d5e2..51ece33 100644 > --- a/mysql-test/r/mysqld--help.result > +++ b/mysql-test/r/mysqld--help.result > @@ -197,6 +197,11 @@ The following specify which files/extra groups are read (specified before remain > cache, etc) > --enforce-storage-engine=name > Force the use of a storage engine for new tables > + --eq-range-index-dive-limit=# > + The optimizer will use existing index statistics instead > + of doing index dives for equality ranges if the number of > + equality ranges for the index is larger than or equal to > + this number. If set to 0, index dives are always used. > --event-scheduler[=name] > Enable the event scheduler. Possible values are ON, OFF, > and DISABLED (keep the event scheduler completely > @@ -1259,6 +1264,7 @@ encrypt-binlog FALSE > encrypt-tmp-disk-tables FALSE > encrypt-tmp-files FALSE > enforce-storage-engine (No default value) > +eq-range-index-dive-limit 0 > event-scheduler OFF > expensive-subquery-limit 100 > expire-logs-days 0 > diff --git a/mysql-test/r/range.result b/mysql-test/r/range.result > index 3a71d08..0c6be5e 100644 > --- a/mysql-test/r/range.result > +++ b/mysql-test/r/range.result > @@ -3002,5 +3002,46 @@ deallocate prepare stmt; > set optimizer_switch=@save_optimizer_switch; > drop table t1,t2,t3; > # > +# MDEV-16934: using system variable eq_range_index_dive_limit > +# to reduce the number of index dives > +# > +create table t1 (a int, b varchar(31), index idx(a)); > +insert into t1 values > +(7,'xxxx'), (1,'yy'), (3,'aaa'), (1,'bbb'), (2,'zz'), > +(4,'vvvvv'), (7,'ddd'), (9,'zzzzz'), (1,'cc'), (5,'ffff'); > +insert into t1 select a+10, concat(b,'zz') from t1; > +insert into t1 select a+15, concat(b,'yy') from t1; > +insert into t1 select a+100, concat(b,'xx') from t1; > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status OK > +select cast(count(a)/count(distinct a) as unsigned) as rec_per_key from t1; > +rec_per_key > +2 > +set eq_range_index_dive_limit=0; > +explain select * from t1 where a in (8, 15, 31, 1, 9); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range idx idx 5 NULL 7 Using index condition > +select * from t1 where a in (8, 15, 31, 1, 9); > +a b > +1 yy > +1 bbb > +1 cc > +9 zzzzz > +15 ffffzz > +set eq_range_index_dive_limit=2; > +explain select * from t1 where a in (8, 15, 31, 1, 9); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range idx idx 5 NULL 10 Using index condition > +select * from t1 where a in (8, 15, 31, 1, 9); > +a b > +1 yy > +1 bbb > +1 cc > +9 zzzzz > +15 ffffzz > +set eq_range_index_dive_limit=default; > +drop table t1; > +# > # End of 10.2 tests > # > diff --git a/mysql-test/r/range_mrr_icp.result b/mysql-test/r/range_mrr_icp.result > index 799a299..93f414f 100644 > --- a/mysql-test/r/range_mrr_icp.result > +++ b/mysql-test/r/range_mrr_icp.result > @@ -3014,6 +3014,47 @@ deallocate prepare stmt; > set optimizer_switch=@save_optimizer_switch; > drop table t1,t2,t3; > # > +# MDEV-16934: using system variable eq_range_index_dive_limit > +# to reduce the number of index dives > +# > +create table t1 (a int, b varchar(31), index idx(a)); > +insert into t1 values > +(7,'xxxx'), (1,'yy'), (3,'aaa'), (1,'bbb'), (2,'zz'), > +(4,'vvvvv'), (7,'ddd'), (9,'zzzzz'), (1,'cc'), (5,'ffff'); > +insert into t1 select a+10, concat(b,'zz') from t1; > +insert into t1 select a+15, concat(b,'yy') from t1; > +insert into t1 select a+100, concat(b,'xx') from t1; > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status OK > +select cast(count(a)/count(distinct a) as unsigned) as rec_per_key from t1; > +rec_per_key > +2 > +set eq_range_index_dive_limit=0; > +explain select * from t1 where a in (8, 15, 31, 1, 9); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range idx idx 5 NULL 7 Using index condition; Rowid-ordered scan > +select * from t1 where a in (8, 15, 31, 1, 9); > +a b > +1 yy > +1 bbb > +9 zzzzz > +1 cc > +15 ffffzz > +set eq_range_index_dive_limit=2; > +explain select * from t1 where a in (8, 15, 31, 1, 9); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range idx idx 5 NULL 10 Using index condition; Rowid-ordered scan > +select * from t1 where a in (8, 15, 31, 1, 9); > +a b > +1 yy > +1 bbb > +9 zzzzz > +1 cc > +15 ffffzz > +set eq_range_index_dive_limit=default; > +drop table t1; > +# > # End of 10.2 tests > # > set optimizer_switch=@mrr_icp_extra_tmp; > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > index f7a5cf5..9d7fa10 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > @@ -765,6 +765,20 @@ NUMERIC_BLOCK_SIZE NULL > ENUM_VALUE_LIST NULL > READ_ONLY NO > COMMAND_LINE_ARGUMENT NULL > +VARIABLE_NAME EQ_RANGE_INDEX_DIVE_LIMIT > +SESSION_VALUE 0 > +GLOBAL_VALUE 0 > +GLOBAL_VALUE_ORIGIN COMPILE-TIME > +DEFAULT_VALUE 0 > +VARIABLE_SCOPE SESSION > +VARIABLE_TYPE INT UNSIGNED > +VARIABLE_COMMENT The optimizer will use existing index statistics instead of doing index dives for equality ranges if the number of equality ranges for the index is larger than or equal to this number. If set to 0, index dives are always used. > +NUMERIC_MIN_VALUE 0 > +NUMERIC_MAX_VALUE 4294967295 > +NUMERIC_BLOCK_SIZE 1 > +ENUM_VALUE_LIST NULL > +READ_ONLY NO > +COMMAND_LINE_ARGUMENT REQUIRED > VARIABLE_NAME ERROR_COUNT > SESSION_VALUE 0 > GLOBAL_VALUE NULL > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > index 034195e..cada139 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > @@ -779,6 +779,20 @@ NUMERIC_BLOCK_SIZE NULL > ENUM_VALUE_LIST NULL > READ_ONLY NO > COMMAND_LINE_ARGUMENT NULL > +VARIABLE_NAME EQ_RANGE_INDEX_DIVE_LIMIT > +SESSION_VALUE 0 > +GLOBAL_VALUE 0 > +GLOBAL_VALUE_ORIGIN COMPILE-TIME > +DEFAULT_VALUE 0 > +VARIABLE_SCOPE SESSION > +VARIABLE_TYPE INT UNSIGNED > +VARIABLE_COMMENT The optimizer will use existing index statistics instead of doing index dives for equality ranges if the number of equality ranges for the index is larger than or equal to this number. If set to 0, index dives are always used. > +NUMERIC_MIN_VALUE 0 > +NUMERIC_MAX_VALUE 4294967295 > +NUMERIC_BLOCK_SIZE 1 > +ENUM_VALUE_LIST NULL > +READ_ONLY NO > +COMMAND_LINE_ARGUMENT REQUIRED > VARIABLE_NAME ERROR_COUNT > SESSION_VALUE 0 > GLOBAL_VALUE NULL > diff --git a/mysql-test/t/range.test b/mysql-test/t/range.test > index ab95180..ed68cfb 100644 > --- a/mysql-test/t/range.test > +++ b/mysql-test/t/range.test > @@ -2046,6 +2046,39 @@ set optimizer_switch=@save_optimizer_switch; > drop table t1,t2,t3; > > --echo # > +--echo # MDEV-16934: using system variable eq_range_index_dive_limit > +--echo # to reduce the number of index dives > +--echo # > + > +create table t1 (a int, b varchar(31), index idx(a)); > + > +insert into t1 values > + (7,'xxxx'), (1,'yy'), (3,'aaa'), (1,'bbb'), (2,'zz'), > + (4,'vvvvv'), (7,'ddd'), (9,'zzzzz'), (1,'cc'), (5,'ffff'); > +insert into t1 select a+10, concat(b,'zz') from t1; > +insert into t1 select a+15, concat(b,'yy') from t1; > +insert into t1 select a+100, concat(b,'xx') from t1; > + > +analyze table t1; > + > +select cast(count(a)/count(distinct a) as unsigned) as rec_per_key from t1; > + > +let $q= > +select * from t1 where a in (8, 15, 31, 1, 9); > + > +set eq_range_index_dive_limit=0; > +eval explain $q; > +eval $q; > + > +set eq_range_index_dive_limit=2; > +eval explain $q; > +eval $q; > + > +set eq_range_index_dive_limit=default; > + > +drop table t1; > + > +--echo # > --echo # End of 10.2 tests > --echo # > > diff --git a/sql/multi_range_read.cc b/sql/multi_range_read.cc > index 50918d8..94a96c2 100644 > --- a/sql/multi_range_read.cc > +++ b/sql/multi_range_read.cc > @@ -17,6 +17,7 @@ > #include <my_bit.h> > #include "sql_select.h" > #include "key.h" > +#include "sql_statistics.h" > > /**************************************************************************** > * Default MRR implementation (MRR to non-MRR converter) > @@ -67,6 +68,9 @@ handler::multi_range_read_info_const(uint keyno, RANGE_SEQ_IF *seq, > /* Default MRR implementation doesn't need buffer */ > *bufsz= 0; > > + bool use_statistics_for_eq_range= eq_ranges_exceeds_limit(seq, > + seq_init_param); > + > seq_it= seq->init(seq_init_param, n_ranges, *flags); > while (!seq->next(seq_it, &range)) > { > @@ -87,8 +91,15 @@ handler::multi_range_read_info_const(uint keyno, RANGE_SEQ_IF *seq, > min_endp= range.start_key.length? &range.start_key : NULL; > max_endp= range.end_key.length? &range.end_key : NULL; > } > + int keyparts_used= my_count_bits(range.start_key.keypart_map); > if ((range.range_flag & UNIQUE_RANGE) && !(range.range_flag & NULL_RANGE)) > rows= 1; /* there can be at most one row */ > + else if (use_statistics_for_eq_range && > + !(range.range_flag & NULL_RANGE) && > + (range.range_flag & EQ_RANGE) && > + table->key_info[keyno].actual_rec_per_key(keyparts_used - 1) > 0.5) > + rows= > + (ha_rows) table->key_info[keyno].actual_rec_per_key(keyparts_used - 1); > else > { > if (HA_POS_ERROR == (rows= this->records_in_range(keyno, min_endp, > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index b011c01..c9c37a2 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -14616,6 +14616,33 @@ void QUICK_GROUP_MIN_MAX_SELECT::add_keys_and_lengths(String *key_names, > } > > > +/* Check whether the number for equality ranges exceeds the set threshold */ > + > +bool eq_ranges_exceeds_limit(RANGE_SEQ_IF *seq, void *seq_init_param) > +{ > + KEY_MULTI_RANGE range; > + range_seq_t seq_it; > + uint count = 0; > + PARAM *param= ((SEL_ARG_RANGE_SEQ*) seq_init_param)->param; > + uint limit= param->thd->variables.eq_range_index_dive_limit; > + > + if (limit == 0) > + { > + /* 'Statistics instead of index dives' feature is turned off */ > + return false; > + } > + seq_it= seq->init(seq_init_param, 0, 0); > + while (!seq->next(seq_it, &range)) > + { > + if ((range.range_flag & EQ_RANGE) && !(range.range_flag & NULL_RANGE)) > + { > + if (++count >= limit) > + return true; > + } > + } > + return false; > +} > + > #ifndef DBUG_OFF > > static void print_sel_tree(PARAM *param, SEL_TREE *tree, key_map *tree_map, > diff --git a/sql/opt_range.h b/sql/opt_range.h > index c1f7079..6698c98 100644 > --- a/sql/opt_range.h > +++ b/sql/opt_range.h > @@ -242,7 +242,7 @@ class SEL_ARG :public Sql_alloc > Number of children of this element in the RB-tree, plus 1 for this > element itself. > */ > - uint16 elements; > + uint32 elements; > /* > Valid only for elements which are RB-tree roots: Number of times this > RB-tree is referred to (it is referred by SEL_ARG::next_key_part or by > @@ -1664,6 +1664,8 @@ SQL_SELECT *make_select(TABLE *head, table_map const_tables, > > bool calculate_cond_selectivity_for_table(THD *thd, TABLE *table, Item **cond); > > +bool eq_ranges_exceeds_limit(RANGE_SEQ_IF *seq, void *seq_init_param); > + > #ifdef WITH_PARTITION_STORAGE_ENGINE > bool prune_partitions(THD *thd, TABLE *table, Item *pprune_cond); > #endif > diff --git a/sql/opt_range_mrr.cc b/sql/opt_range_mrr.cc > index ace6208..7f80a3e 100644 > --- a/sql/opt_range_mrr.cc > +++ b/sql/opt_range_mrr.cc > @@ -272,25 +272,44 @@ bool sel_arg_range_seq_next(range_seq_t rseq, KEY_MULTI_RANGE *range) > key_info= NULL; > else > key_info= &seq->param->table->key_info[seq->real_keyno]; > - > + > /* > - Conditions below: > - (1) - range analysis is used for estimating condition selectivity > - (2) - This is a unique key, and we have conditions for all its > - user-defined key parts. > - (3) - The table uses extended keys, this key covers all components, > - and we have conditions for all key parts. > + This is an equality range (keypart_0=X and ... and keypart_n=Z) if > + (1) - There are no flags indicating open range (e.g., > + "keypart_x > y") or GIS. > + (2) - The lower bound and the upper bound of the range has the > + same value (min_key == max_key). > */ > - if (!(cur->min_key_flag & ~NULL_RANGE) && !cur->max_key_flag && > - (!key_info || // (1) > - ((uint)key_tree->part+1 == key_info->user_defined_key_parts && // (2) > - key_info->flags & HA_NOSAME) || // (2) > - ((key_info->flags & HA_EXT_NOSAME) && // (3) > - (uint)key_tree->part+1 == key_info->ext_key_parts) // (3) > - ) && > - range->start_key.length == range->end_key.length && > - !memcmp(seq->param->min_key,seq->param->max_key,range->start_key.length)) > - range->range_flag= UNIQUE_RANGE | (cur->min_key_flag & NULL_RANGE); > + const uint is_open_range = > + (NO_MIN_RANGE | NO_MAX_RANGE | NEAR_MIN | NEAR_MAX | GEOM_FLAG); > + const bool is_eq_range_pred = > + !(cur->min_key_flag & is_open_range) && // (1) > + !(cur->max_key_flag & is_open_range) && // (1) > + range->start_key.length == range->end_key.length && // (2) > + !memcmp(seq->param->min_key, seq->param->max_key, // (2) > + range->start_key.length); > + > + if (is_eq_range_pred) > + { > + range->range_flag = EQ_RANGE; > + > + /* > + Conditions below: > + (1) - Range analysis is used for estimating condition selectivity > + (2) - This is a unique key, and we have conditions for all its > + user-defined key parts. > + (3) - The table uses extended keys, this key covers all components, > + and we have conditions for all key parts. > + */ > + if ( > + !key_info || // (1) > + ((uint)key_tree->part+1 == key_info->user_defined_key_parts && // (2) > + key_info->flags & HA_NOSAME) || // (2) > + ((key_info->flags & HA_EXT_NOSAME) && // (3) > + (uint)key_tree->part+1 == key_info->ext_key_parts) // (3) > + ) > + range->range_flag |= UNIQUE_RANGE | (cur->min_key_flag & NULL_RANGE); > + } > > if (seq->param->is_ror_scan) > { > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 6663d9e..d2f00b2d 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -569,6 +569,7 @@ typedef struct system_variables > ha_rows max_join_size; > ha_rows expensive_subquery_limit; > ulong auto_increment_increment, auto_increment_offset; > + uint eq_range_index_dive_limit; > ulong lock_wait_timeout; > ulong join_cache_level; > ulong max_allowed_packet; > diff --git a/sql/sql_statistics.h b/sql/sql_statistics.h > index f465838..4ff85cd 100644 > --- a/sql/sql_statistics.h > +++ b/sql/sql_statistics.h > @@ -21,7 +21,7 @@ enum enum_use_stat_tables_mode > { > NEVER, > COMPLEMENTARY, > - PEFERABLY, > + PREFERABLY, > } Use_stat_tables_mode; > > typedef > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index e6aec47..650b17a 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -2578,6 +2578,16 @@ static Sys_var_ulong Sys_div_precincrement( > SESSION_VAR(div_precincrement), CMD_LINE(REQUIRED_ARG), > VALID_RANGE(0, DECIMAL_MAX_SCALE), DEFAULT(4), BLOCK_SIZE(1)); > > +static Sys_var_uint Sys_eq_range_index_dive_limit( > + "eq_range_index_dive_limit", > + "The optimizer will use existing index statistics instead of " > + "doing index dives for equality ranges if the number of equality " > + "ranges for the index is larger than or equal to this number. " > + "If set to 0, index dives are always used.", > + SESSION_VAR(eq_range_index_dive_limit), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, UINT_MAX32), DEFAULT(0), > + BLOCK_SIZE(1)); > + > static Sys_var_ulong Sys_range_alloc_block_size( > "range_alloc_block_size", > "Allocation block size for storing ranges during optimization", > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] adaafcf5184: MDEV-16930: Crash when VALUES in derived table contains expressions
by Sergei Golubchik 16 Aug '18

16 Aug '18

Hi, Oleksandr! On Aug 16, Oleksandr Byelkin wrote: > revision-id: adaafcf5184375369c0c4b3ccb1e82e846024ef8 (mariadb-10.3.7-137-gadaafcf5184) > parent(s): 8716bb3b72b7f5fed69b6dde413c2138f6d175b2 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-08-16 13:55:36 +0200 > message: > > MDEV-16930: Crash when VALUES in derived table contains expressions > > Give names to the value constructor columns as in SELECT-list. > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index f915895a260..20cea4959eb 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -13306,15 +13306,20 @@ opt_values: > ; > > values: > - values ',' expr_or_default > + values ',' remember_name expr_or_default remember_end > { > - if (unlikely(Lex->insert_list->push_back($3, thd->mem_root))) > + if (unlikely(Lex->insert_list->push_back($4, thd->mem_root))) > MYSQL_YYABORT; > + // give some name in case of using in table value constuctor (TVC) > + $4->set_name(thd, $3, (uint) ($5 - $3), thd->charset()); > + > } > - | expr_or_default > + | remember_name expr_or_default remember_end > { > - if (unlikely(Lex->insert_list->push_back($1, thd->mem_root))) > + if (unlikely(Lex->insert_list->push_back($2, thd->mem_root))) > MYSQL_YYABORT; > + // give some name in case of using in table value constuctor (TVC) > + $2->set_name(thd, $1, (uint) ($3 - $1), thd->charset()); > } > ; Could you please create a special rule for that? Like named_value: remember_name expr_or_default remember_end { if (unlikely(Lex->insert_list->push_back($2, thd->mem_root))) MYSQL_YYABORT; // give some name in case of using in table value constuctor (TVC) $2->set_name(thd, $1, (uint) ($3 - $1), thd->charset()); } values: values ',' named_value | named_value ; if it'll work, that is Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 7fc57734575: MDEV-6439: Server crashes in Explain_union::print_explain with explain in slow log, tis620 charset
by Oleksandr Byelkin 15 Aug '18

15 Aug '18

Hi! See small comment inline. (fix and OK to push) Am 15.08.2018 um 13:36 schrieb Sergey Petrunia: > MDEV-6439: Server crashes in Explain_union::print_explain with explain in slow log, tis620 charset > > Item_subselect::is_expensive() used to return FALSE (Inexpensive) whenever > it saw that one of SELECTs in the Subquery's UNION is degenerate. It > ignored the fact that other parts of the UNION might not be inexpensive, > including the case where pther parts of the UNION have no query plan yet. > > For a subquery in form col >= ANY (SELECT 'foo' UNION SELECT 'bar') > this would cause the query to be considered inexpensive when there is > no query plan for the second part of the UNION, which in turn would > cause the SELECT 'foo' to compute and free itself while still inside > JOIN::optimize for that SELECT (See MDEV comment for full description). > > diff --git a/mysql-test/r/subselect_extra_no_semijoin.result b/mysql-test/r/subselect_extra_no_semijoin.result > index 79bca38..200682b 100644 > --- a/mysql-test/r/subselect_extra_no_semijoin.result > +++ b/mysql-test/r/subselect_extra_no_semijoin.result > @@ -482,3 +482,22 @@ DROP TABLE t1,t2; > set optimizer_switch= @tmp_subselect_extra_derived; > set optimizer_switch= @subselect_extra_no_sj_tmp; > set @optimizer_switch_for_subselect_extra_test=null; > +# > +# MDEV-6439: Server crashes in Explain_union::print_explain with explain in slow log, tis620 charset > +# > +SET NAMES tis620; > +set @tmp= @@global.slow_query_log; > +SET GLOBAL slow_query_log = 1; > +SET long_query_time = 0.000001; > +SET log_slow_verbosity = 'explain'; > +CREATE TABLE t1 (a VARCHAR(3)) ENGINE=MyISAM; > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo'); > +a > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo' UNION SELECT 'bar' ); > +ERROR HY000: Illegal mix of collations (tis620_thai_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation '<=' > +create table t2 (b int); > +insert into t2 values (1),(2),(3); > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo' FROM t2); > +ERROR HY000: Illegal mix of collations (tis620_thai_ci,COERCIBLE) and (latin1_swedish_ci,IMPLICIT) for operation '<=' > +drop table t1,t2; > +SET GLOBAL slow_query_log=@tmp; > diff --git a/mysql-test/t/subselect_extra_no_semijoin.test b/mysql-test/t/subselect_extra_no_semijoin.test > index 8aba3dd..41593fb 100644 > --- a/mysql-test/t/subselect_extra_no_semijoin.test > +++ b/mysql-test/t/subselect_extra_no_semijoin.test > @@ -6,4 +6,34 @@ set @optimizer_switch_for_subselect_extra_test='semijoin=off,firstmatch=off,loo > > set optimizer_switch= @subselect_extra_no_sj_tmp; > > -set @optimizer_switch_for_subselect_extra_test=null; > \ No newline at end of file > +set @optimizer_switch_for_subselect_extra_test=null; > + > +--echo # > +--echo # MDEV-6439: Server crashes in Explain_union::print_explain with explain in slow log, tis620 charset > +--echo # > + > +## Using a separate client connection is easier than restoring state > +connect(con1,localhost,root,,); > + > +SET NAMES tis620; > +set @tmp= @@global.slow_query_log; > +SET GLOBAL slow_query_log = 1; > +SET long_query_time = 0.000001; > +SET log_slow_verbosity = 'explain'; > + please remove above space at the end of line and push. > +CREATE TABLE t1 (a VARCHAR(3)) ENGINE=MyISAM; > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo'); > +--error ER_CANT_AGGREGATE_2COLLATIONS > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo' UNION SELECT 'bar' ); > + > +create table t2 (b int); > +insert into t2 values (1),(2),(3); > + > +--error ER_CANT_AGGREGATE_2COLLATIONS > +SELECT * FROM t1 WHERE a >= ANY ( SELECT 'foo' FROM t2); > + > +drop table t1,t2; > +SET GLOBAL slow_query_log=@tmp; > +disconnect con1; > +connection default; > + > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index a8dfdff..14b2ccd 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -575,7 +575,7 @@ bool Item_subselect::is_expensive() > */ > if (cur_join->optimized && > (cur_join->zero_result_cause || !cur_join->tables_list)) > - return false; > + continue; > > /* > If a subquery is not optimized we cannot estimate its cost. A subquery is

1 0

[Maria-developers] MDEV-16768 and MDEV-16703 patch review
by Ming Lin 10 Aug '18

10 Aug '18

Hi Sergey, Could you help to review below 2 patches please? MDEV-16768: https://github.com/MariaDB/server/pull/837 MDEV-16703: https://github.com/MariaDB/server/pull/839 Thanks, Ming

1 0

Re: [Maria-developers] 6df4e4a855f: MDEV-16217: Assertion `!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))' failed in Field_num::get_date
by Sergei Golubchik 09 Aug '18

09 Aug '18

Hi, Oleksandr! See questions below. There are few changes that I'm not quite sure about. On Aug 08, Oleksandr Byelkin wrote: > revision-id: 6df4e4a855fe223988c12681f8caec6c49b2f629 (mariadb-10.2.16-66-g6df4e4a855f) > parent(s): 4ddcb4eb46c62cf459936554d43351db740ba14d > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-08-08 12:28:15 +0200 > message: > > MDEV-16217: Assertion `!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))' failed in Field_num::get_date > > - clean up DEFAULT() to work only with default value and correctly print > itself. > - fix of DBUG_ASSERT about fields read/write > - fix of firld marking for write based really on the thd->mark_used_columns flag > > --- > mysql-test/r/func_default.result | 2 +- > mysql-test/r/func_time.result | 28 ++++++++++++++++++++++++++++ > mysql-test/t/func_time.test | 21 +++++++++++++++++++++ > sql/field.cc | 21 +++++++++++++++++++-- > sql/field.h | 1 + > sql/item.cc | 25 ++++++++++++++++++++++++- > sql/item.h | 5 +++++ > sql/sql_base.cc | 2 +- > 8 files changed, 100 insertions(+), 5 deletions(-) > > diff --git a/mysql-test/r/func_default.result b/mysql-test/r/func_default.result > index 535be10da86..1f331af287c 100644 > --- a/mysql-test/r/func_default.result > +++ b/mysql-test/r/func_default.result > @@ -8,7 +8,7 @@ explain extended select default(str), default(strnull), default(intg), default(r > id select_type table type possible_keys key key_len ref rows filtered Extra > 1 SIMPLE t1 system NULL NULL NULL NULL 1 100.00 > Warnings: > -Note 1003 select default('') AS `default(str)`,default('') AS `default(strnull)`,default(0) AS `default(intg)`,default(0) AS `default(rel)` from dual > +Note 1003 select default(`str`) AS `default(str)`,default(`strnull`) AS `default(strnull)`,default(`intg`) AS `default(intg)`,default(`rel`) AS `default(rel)` from dual This looks rather fishy. There can be no columns str and strnull in dual. Please add a test case with a view, to make sure you only get this effect (column names and dual) in explain extended (where it doesn't matter), but not in a view (where it does). > select * from t1 where str <> default(str); > str strnull intg rel > 0 0 > diff --git a/mysql-test/r/func_time.result b/mysql-test/r/func_time.result > index 55c2c93eb56..51f51820efc 100644 > --- a/mysql-test/r/func_time.result > +++ b/mysql-test/r/func_time.result > @@ -3260,3 +3260,31 @@ DROP TABLE t1,t2; > # > # End of 10.1 tests > # > +# > +# MDEV-16217: Assertion `!table || (!table->read_set || > +# bitmap_is_set(table->read_set, field_index))' > +# failed in Field_num::get_date > +# > +CREATE TABLE t1 (pk int default 0, a1 date); > +INSERT INTO t1 VALUES (1,'1900-01-01'),(2,NULL),(3,NULL),(4,NULL); > +CREATE VIEW v1 AS > +SELECT t1.pk AS pk, t1.a1 AS a1 FROM t1; > +SELECT a1 BETWEEN (('2018-08-24')) AND (DEFAULT(pk)) FROM v1; > +a1 BETWEEN (('2018-08-24')) AND (DEFAULT(pk)) > +0 > +NULL > +NULL > +NULL > +SELECT a1 BETWEEN (('2018-08-24')) AND (~ DEFAULT(pk)) FROM v1; > +a1 BETWEEN (('2018-08-24')) AND (~ DEFAULT(pk)) > +0 > +NULL > +NULL > +NULL > +Warnings: > +Warning 1292 Incorrect datetime value: '18446744073709551615' > +DROP VIEW v1; > +DROP TABLE t1; > +# > +# End of 10.2 tests > +# > diff --git a/mysql-test/t/func_time.test b/mysql-test/t/func_time.test > index 3503b6d5bc6..c8859feee93 100644 > --- a/mysql-test/t/func_time.test > +++ b/mysql-test/t/func_time.test > @@ -1857,3 +1857,24 @@ DROP TABLE t1,t2; > --echo # > --echo # End of 10.1 tests > --echo # > + > +--echo # > +--echo # MDEV-16217: Assertion `!table || (!table->read_set || > +--echo # bitmap_is_set(table->read_set, field_index))' > +--echo # failed in Field_num::get_date > +--echo # > +CREATE TABLE t1 (pk int default 0, a1 date); > +INSERT INTO t1 VALUES (1,'1900-01-01'),(2,NULL),(3,NULL),(4,NULL); > + > +CREATE VIEW v1 AS > +SELECT t1.pk AS pk, t1.a1 AS a1 FROM t1; > + > +SELECT a1 BETWEEN (('2018-08-24')) AND (DEFAULT(pk)) FROM v1; > +SELECT a1 BETWEEN (('2018-08-24')) AND (~ DEFAULT(pk)) FROM v1; > + > +DROP VIEW v1; > +DROP TABLE t1; > + > +--echo # > +--echo # End of 10.2 tests > +--echo # > diff --git a/sql/field.cc b/sql/field.cc > index 9ca9663f066..e12adbf47b3 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -70,8 +70,25 @@ const char field_separator=','; > #define BLOB_PACK_LENGTH_TO_MAX_LENGH(arg) \ > ((ulong) ((1LL << MY_MIN(arg, 4) * 8) - 1)) > > -#define ASSERT_COLUMN_MARKED_FOR_READ DBUG_ASSERT(!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))) > -#define ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED DBUG_ASSERT(is_stat_field || !table || (!table->write_set || bitmap_is_set(table->write_set, field_index) || (table->vcol_set && bitmap_is_set(table->vcol_set, field_index)))) > +// Column marked for read or the field set to read out or record[0] or [1] > +#define ASSERT_COLUMN_MARKED_FOR_READ \ > + DBUG_ASSERT(!table || \ > + (!table->read_set || \ > + bitmap_is_set(table->read_set, field_index) || \ > + (!(ptr >= table->record[0] && \ > + ptr < table->record[0] + table->s->reclength) && \ > + !(ptr >= table->record[1] && \ > + ptr < table->record[1] + table->s->reclength)))) why did you add record[1] to the assert? I think, having just record[0] would've been enough > + > +#define ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED \ > + DBUG_ASSERT(is_stat_field || !table || \ > + (!table->write_set || \ > + bitmap_is_set(table->write_set, field_index) || \ > + (!(ptr >= table->record[0] && \ > + ptr < table->record[0] + table->s->reclength) && \ > + !(ptr >= table->record[1] && \ > + ptr < table->record[1] + table->s->reclength))) || \ > + (table->vcol_set && bitmap_is_set(table->vcol_set, field_index))) > > #define FLAGSTR(S,F) ((S) & (F) ? #F " " : "") > > diff --git a/sql/field.h b/sql/field.h > index 22c276478b6..55c3ed4c4bd 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -630,6 +630,7 @@ class Virtual_column_info: public Sql_alloc > bool utf8; /* Already in utf8 */ > Item *expr; > LEX_STRING name; /* Name of constraint */ > + /* see VCOL_* (VCOL_FIELD_REF, ...) */ > uint flags; > > Virtual_column_info() > diff --git a/sql/item.cc b/sql/item.cc > index 58d2b7dbfc0..91fe7ff03aa 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -8917,8 +8917,19 @@ bool Item_default_value::fix_fields(THD *thd, Item **items) > fixed= 1; > return FALSE; > } > + > + /* > + DEFAULT() do not need table field so should not ask handler to bring > + field value (mark column for read) > + */ > + enum_mark_columns save_mark_used_columns= thd->mark_used_columns; > + thd->mark_used_columns= MARK_COLUMNS_NONE; > if (!arg->fixed && arg->fix_fields(thd, &arg)) > + { > + thd->mark_used_columns= save_mark_used_columns; > goto error; > + } > + thd->mark_used_columns= save_mark_used_columns; Hmm, I thought that the field wasn't marked in read_set, and that failed the assert. Now you're saying that it was? > real_arg= arg->real_item(); > @@ -8938,12 +8949,16 @@ bool Item_default_value::fix_fields(THD *thd, Item **items) > goto error; > memcpy((void *)def_field, (void *)field_arg->field, > field_arg->field->size_of()); > - IF_DBUG(def_field->is_stat_field=1,); // a hack to fool ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED ah, good. thanks! > + // If non-constant default value expression > if (def_field->default_value && def_field->default_value->flags) > { > uchar *newptr= (uchar*) thd->alloc(1+def_field->pack_length()); > if (!newptr) > goto error; > + /* > + Even if DEFAULT() do not read tables fields, the default value > + expression can do it. > + */ > fix_session_vcol_expr_for_read(thd, def_field, def_field->default_value); > if (thd->mark_used_columns != MARK_COLUMNS_NONE) > def_field->default_value->expr->walk(&Item::register_field_in_read_map, 1, 0); better call update_used_tables() from here. > @@ -8970,6 +8985,14 @@ void Item_default_value::print(String *str, enum_query_type query_type) > return; > } > str->append(STRING_WITH_LEN("default(")); > + /* > + We take DEFAULT from a field so do not need it value in case of const > + tables but its name so we set QT_NO_DATA_EXPANSION (as we print for > + table definition, also we do not need table and database name) > + */ > + query_type= (enum_query_type) (query_type | QT_NO_DATA_EXPANSION | > + QT_ITEM_IDENT_SKIP_DB_NAMES | > + QT_ITEM_IDENT_SKIP_TABLE_NAMES); in a view definition you might need table and database names. For example, a view that joins two tables from different databases, and they both have a column named `a`. And the query uses DEFAULT(db1.t1.a). > arg->print(str, query_type); > str->append(')'); > } > diff --git a/sql/item.h b/sql/item.h > index 8fad8dadf22..d252a256f76 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -5316,6 +5316,11 @@ class Item_default_value : public Item_field > return false; > } > table_map used_tables() const; > + virtual void update_used_tables() > + { > + if (field && field->default_value) > + field->default_value->expr->walk(&Item::register_field_in_read_map, 1, 0); > + } > Field *get_tmp_table_field() { return 0; } > Item *get_tmp_table_item(THD *thd) { return this; } > Item_field *field_for_view_update() { return 0; } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] Spelling errors in MariaDB sources
by Otto Kekäläinen 31 Jul '18

31 Jul '18

Hello! I recall I've made several big patches on MariaDB that fixes spelling errors. Somehow the same spelling errors seem to creep back in every now and then. I notice them while uploading stuff to Debian and using the Debian QA tool "Lintian". I haven't done git blame and looked deeply in it. I suspect it is related to how we merge old branches to new branches and occasionally loose stuff. Would somebody like to look into this? It could be an relatively easy task for some new contributor? $ grep spelling-error lintian-22e620c534.log I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_chk noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_chk thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_chk theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_dump_log noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_dump_log thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_dump_log theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_ftdump noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_ftdump thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_ftdump theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_pack noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_pack thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_pack theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_read_log noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_read_log thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/aria_read_log theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisam_ftdump noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisam_ftdump thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisam_ftdump theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamchk noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamchk thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamchk theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamlog noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamlog thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisamlog theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisampack noone no one I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisampack thats that's I: mariadb-server-10.1: spelling-error-in-binary usr/bin/myisampack theres there's I: mariadb-server-10.1: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_federated.so Allows to Allows one to I: mariadb-server-10.1: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_federatedx.so Allows to Allows one to I: mariadb-server-10.1: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_innodb.so allow to allow one to I: libmariadbd18: spelling-error-in-binary usr/lib/x86_64-linux-gnu/libmysqld.so.18 extentions extensions I: libmariadbd18: spelling-error-in-binary usr/lib/x86_64-linux-gnu/libmysqld.so.18 noone no one I: libmariadbd18: spelling-error-in-binary usr/lib/x86_64-linux-gnu/libmysqld.so.18 thats that's I: libmariadbd18: spelling-error-in-binary usr/lib/x86_64-linux-gnu/libmysqld.so.18 theres there's I: libmariadbd18: spelling-error-in-binary usr/lib/x86_64-linux-gnu/libmysqld.so.18 allow to allow one to I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup yuR your I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup occured occurred I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup accessable accessible I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup extentions extensions I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup reseting resetting I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup succeded succeeded I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup settting setting I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup postion position I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup noone no one I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup thats that's I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup theres there's I: mariadb-client-10.1: spelling-error-in-binary usr/bin/mariabackup allow to allow one to I: mariadb-client-10.1: spelling-error-in-manpage usr/share/man/man1/innotop.1.gz Additonal Additional I: mariadb-plugin-tokudb: spelling-error-in-binary usr/bin/tokuftdump dont don't I: mariadb-plugin-tokudb: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_tokudb.so dont don't I: mariadb-client-core-10.1: spelling-error-in-binary usr/bin/mysql_embedded extentions extensions I: mariadb-client-core-10.1: spelling-error-in-binary usr/bin/mysql_embedded noone no one I: mariadb-client-core-10.1: spelling-error-in-binary usr/bin/mysql_embedded thats that's I: mariadb-client-core-10.1: spelling-error-in-binary usr/bin/mysql_embedded theres there's I: mariadb-client-core-10.1: spelling-error-in-binary usr/bin/mysql_embedded allow to allow one to I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld extentions extensions I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld reseting resetting I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld succeded succeeded I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld settting setting I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld postion position I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld noone no one I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld thats that's I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld theres there's I: mariadb-server-core-10.1: spelling-error-in-binary usr/sbin/mysqld allow to allow one to I: mariadb-plugin-connect: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_connect.so Mutiple Multiple I: mariadb-plugin-connect: spelling-error-in-binary usr/lib/x86_64-linux-gnu/mariadb18/plugin/ha_connect.so Openning Opening

1 0

[Maria-developers] Missing man pages
by Otto Kekäläinen 31 Jul '18

31 Jul '18

Hello Ian and others! While working on uploading latest MariaDB 10.1 into Debian I noticed it complains about quite a few missing man pages: $ grep manpage lintian-22e620c534.log W: mariadb-server-10.1: binary-without-manpage usr/bin/wsrep_sst_mariabackup W: mariadb-server-10.1: binary-without-manpage usr/bin/wsrep_sst_rsync_wan I: mariadb-client-10.1: spelling-error-in-manpage usr/share/man/man1/innotop.1.gz Additonal Additional W: mariadb-client-10.1: binary-without-manpage usr/bin/mariabackup W: mariadb-client-10.1: binary-without-manpage usr/bin/mbstream W: mariadb-client-10.1: binary-without-manpage usr/bin/mytop W: mariadb-plugin-tokudb: binary-without-manpage usr/bin/tokuft_logprint W: mariadb-client-core-10.1: binary-without-manpage usr/bin/mysql_embedded Could you Ian check if we have these on some branch already and make sure they and up an all the same branches and releases as the binaries themselves? And those who make new binaries, please at least write a stub for your new binary. It makes downstream work much faster when the man page in the source explains directly what the new binaries are and hints how they should be packaged. - Otto

1 0

[Maria-developers] Please review MDEV-16584 SP with a cursor inside a loop wastes THD memory aggressively
by Alexander Barkov 30 Jul '18

30 Jul '18

Hello Sanja, Please review a patch for MDEV-16584. Thanks!

2 1

Re: [Maria-developers] [Commits] 108af298032: MDEV-15454: Nested SELECT IN returns wrong results
by Sergey Petrunia 25 Jul '18

25 Jul '18

On Fri, May 11, 2018 at 03:08:29AM +0530, Varun wrote: > revision-id: 108af298032cb4cea316a08c304c306268a2badf (mariadb-5.5.56-213-g108af298032) > parent(s): 318097bb8f6e12c546b5dcd287416158209dbb39 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-05-11 02:56:33 +0530 > message: > > MDEV-15454: Nested SELECT IN returns wrong results > > In this case we are setting the field Item_func_eq::in_eqaulity_no for the semi-join equalities. > This helps us to remove these equalites as the inner tables are not available during parent select execution > while the outer tables are not available during materialization phase. > We only have it set for the equalites for the fields involved with the IN subquery > and reset it for the equalities which do not belong to the IN subquery. > > For example in case of nested IN subqueries: > > SELECT t1.a FROM t1 WHERE t1.a IN > (SELECT t2.a FROM t2 where t2.b IN > (select t3.b from t3 where t3.c=27 )) > > there are two equalites involving the fields of the IN subquery > > 1) t2.b = t3.b : the field Item_func_eq::in_eqaulity_no is set when we merge the grandchild select into the child select > 2) t1.a = t2.a : the field Item_func_eq::in_eqaulity_no is set when we merge the child select into the parent select > > But when we perform case 2) we should ensure that we reset the equalities in the child's WHERE clause. > > --- > mysql-test/r/subselect_sj_mat.result | 46 +++++++++++++++++++++++++ > mysql-test/t/subselect_sj_mat.test | 49 ++++++++++++++++++++++++++ > sql/opt_subselect.cc | 66 ++++++++++++++++++++++++++++++++++++ > 3 files changed, 161 insertions(+) > ... > diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc > index a7edd64e68b..31213a5cf1d 100644 > --- a/sql/opt_subselect.cc > +++ b/sql/opt_subselect.cc > @@ -1481,6 +1543,7 @@ static int subq_sj_candidate_cmp(Item_in_subselect* el1, Item_in_subselect* el2, > TRUE Out of memory error > */ > > + > static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect *subq_pred) > { > SELECT_LEX *parent_lex= parent_join->select_lex; > @@ -1713,6 +1776,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect *subq_pred) > > if (subq_pred->left_expr->cols() == 1) > { > + reset_equality_number_for_subq_conds(sj_nest->sj_on_expr); > /* add left = select_list_element */ > nested_join->sj_outer_expr_list.push_back(&subq_pred->left_expr); > /* > @@ -1735,6 +1799,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect *subq_pred) > } > else if (subq_pred->left_expr->type() == Item::ROW_ITEM) > { > + reset_equality_number_for_subq_conds(sj_nest->sj_on_expr); > /* > disassemple left expression and add > left1 = select_list_element1 and left2 = select_list_element2 ... > @@ -1759,6 +1824,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect *subq_pred) > } > else > { > + reset_equality_number_for_subq_conds(sj_nest->sj_on_expr); > /* > add row operation > left = (select_list_element1, select_list_element2, ...) So, the code here is if (...) { variant1; } else if (...) { variant2; } else { variant3; } and there is the same call added to all three: > + reset_equality_number_for_subq_conds(sj_nest->sj_on_expr); Can you move it out of the "if" and make it one call instead of three? Otherwise, the patch is ok, ok to push after the above is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] d277c4682e4: MDEV-16751: Server crashes in st_join_table::cleanup or TABLE_LIST::is_with_table_recursive_reference
by Sergey Petrunia 24 Jul '18

24 Jul '18

On Sun, Jul 22, 2018 at 03:56:37PM +0530, Varun wrote: > revision-id: d277c4682e4c6bb0e708296e6ae520c149dd78f4 (mariadb-5.5.60-47-gd277c4682e4) > parent(s): 9cea4ccf12cb6e8746b9b440d9c62408a9ef04af > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-07-22 15:56:25 +0530 > message: > > MDEV-16751: Server crashes in st_join_table::cleanup or TABLE_LIST::is_with_table_recursive_reference > with join_cache_level>2 > > During muliple equality propagation for a query in which we have an IN subquery, the items in the select list of the > subquery may not be part of the multiple equality because there might be another occurence of the same field in the > where clause of the subquery. > So we keyuse_is_valid_for_access_in_chosen_plan function which expects the items in the select list of the subquery to > be same to the ones in the multiple equality (through these multiple equalities we create keyuse array). > The solution would be that we expect the same field not the same Item because when we have SEMI JOIN MATERIALIZATION SCAN, > we use copy back technique to copies back the materialised table fields to the original fields of the base tables. > > --- > mysql-test/r/subselect_mat.result | 35 +++++++++++++++++++++++++++++++++++ > mysql-test/r/subselect_sj_mat.result | 35 +++++++++++++++++++++++++++++++++++ > mysql-test/t/subselect_sj_mat.test | 25 +++++++++++++++++++++++++ > sql/sql_select.cc | 4 +++- > 4 files changed, 98 insertions(+), 1 deletion(-) ... > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 6a64a0e9952..af6f63a55f1 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -7438,7 +7438,9 @@ bool JOIN_TAB::keyuse_is_valid_for_access_in_chosen_plan(JOIN *join, > st_select_lex *sjm_sel= emb_sj_nest->sj_subq_pred->unit->first_select(); > for (uint i= 0; i < sjm_sel->item_list.elements; i++) > { > - if (sjm_sel->ref_pointer_array[i] == keyuse->val) > + DBUG_ASSERT(keyuse->val->type() == Item::FIELD_ITEM); ^^(1) > + Field *field = ((Item_field*)sjm_sel->ref_pointer_array[i])->field; ^^(2) > + if (field->eq(((Item_field*)keyuse->val)->field)) > return true; > } > return false; As for typecast (2): As far as I understand, it is valid because of the following: - We get to this point in this function only when Semi-Join-Materialization-Scan is used. - SJM-Scan is only used when the subquery's select list has Item_field objects (This is checked in subquery_types_allow_materialization()). * Please add a comment in subquery_types_allow_materialization() about this function relying on that logic. * Please add here an assert that sjm_sel->ref_pointer_array[i] is an Item_field. The logic is complex and can be potentially broken. As for typecast (1): What if keyuse->val is not an Item_field? If we get to this point in this function, it means it's an expression that only depends on the SJ-Inner table. This cannot just happen due to limitations in subquery_types_allow_materialization(), and I wasn't able to construct an example where it would happen (tried playing with multiple-equalities, but substitute_for_best_equal_item is called after get_best_combination(), tried playing with constant tables but then SJ-Materialization is not used, etc. ) I think we will be safer by changing the assert to be this check: if (keyuse->val->type() == Item::FIELD_ITEM) { ... everything else } The idea is that a KEYUSE that satisfies the requirement will still be available, and we just want to filter out any extra KEYUSE that might pop up due to some optimization. what do you think? Ok to push after the above is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Updated Gtid_slave_pos of Untracked domain creates skipped events
by Sachin Setiya 23 Jul '18

23 Jul '18

Hi Andrei, Kristian! Problem:- This issue is regarding Mdev-9107 , where we have 3 master master with do_domain_ids of replication channel master domain-id . , When one replication channel is stopped and that master is updated , slave server updates the gtid slave_pos from second replication channel. And then when replication channel is brought up again then since we have updated the gtid_slave_pos we skip that event. I have created a abstract test case for this probem. (3) ^ ^ / \ (2)----->(1) 3 and 1 is configured with log slave updates. 3 has 2 replication channel m2_s3(do_domain_id=2 ), m1_s3(do_domain_id=1) 1 has one replication channel m2_s1(do_domain_id=2) I have tried with master_use_gtid == slave_pos and current_pos both give same result. When I stop replication channel m2_s3 and then write one-one event in (2) and (1) server. Server 3 will update its gtid_slave_pos and it will have latest gtid event seq no from domain id 2 and 1. When replication channel m2_s3 is bought up again since gtid_slave_pos of (3) is equal to (2) so we actually skipped the event generated on server 2. (I believe network lag between 2-->3 will have similar effect) (I have attached the test cases) Possible Solution. May be we should have one more option in master_use_gtid = binlog_state ? which will compare its binlog_state to arriving event gtid no ? (It is different from current_pos , current_pos will select the maximum seq_no between binlog_state and slave_pos) -- Regards Sachin Setiya Software Engineer at MariaDB

2 5

[Maria-developers] Please review a patch for MDEV-15758 Split Item_bool_func::get_mm_leaf() into virtual methods in Field and Type_handler
by Alexander Barkov 20 Jul '18

20 Jul '18

Hello Vicentiu, Please review my patch for MDEV-15758. It also fixes this bug: MDEV-15759 Expect "Impossible WHERE" for indexed_int_column=out_of_range_int_constant Thanks!

3 4

[Maria-developers] To MDEV-6608 Slave loses master binlog filename
by andrei.elkin＠pp.inet.fi 13 Jul '18

13 Jul '18

Hello, Kristian. I read through your analysis of the case to agree with the plan and its motivation: MDEV-6608> Generally, it is a very bad idea to destroy data during crash recovery ... try an approach that leaves the crashed binlog file intact, and instead ensures that the crashed binlog can be properly handled by the rest of the code. So in case of a partially written event group (transaction) the short (xid-less therefore) transaction is to be rolled back on the master. So why won't we properly enclose the group in the binlog at the end of recovery which means to append ROLLBACK Query-log-event, naturally before that the last event of the group needs sanity check? Was that that you had in mind actually? Cheers, Andrei

2 1

[Maria-developers] MDEV-13118 Wrong results with LOWER and UPPER and subquery
by Alexander Barkov 12 Jul '18

12 Jul '18

Hello Sergei, Please review a patch for MDEV-13118. Thanks!

2 4

[Maria-developers] Selecting Blob by parts (suppose select 10 to 100 char from blob )
by Sachin Setiya 12 Jul '18

12 Jul '18

Hello everyone! On optimizer call (11-7-2018) it was discussed , instead of selecting the whole blob , sometime we just need to select the subpart of it. And we should extend handler/engine api to facilitate this. Although this looks like a interesting idea , Serg (and I ) did not understand why we need this , what is use case of it. Some use case which I think can be like optimizing substring SELECT SUBSTRING(blob_col, 4,4) FROM t1; And we can push down this SUBSTRING condition directly to storage engine. But apart from this what use case this have ? -- Regards Sachin Setiya Software Engineer at MariaDB

1 0

[Maria-developers] What can MariaDB developers learn from Debian and Ubuntu Quality Assurance systems?
by Otto Kekäläinen 11 Jul '18

11 Jul '18

Hello! I had the presentation with the title in the subject line at the developers unconference some weeks ago. The slides are available here: https://mariadb.org/wp-content/uploads/2018/07/2018-06-27-mariadb-dev-learn… I recommend all developers skim though them. Debian has some really nice QA systems. All presentations from the event can be found at: https://mariadb.org/tampere-mariadb-developers-unconference-reportback-pres…

1 0

Re: [Maria-developers] 1a2748c: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 10 Jul '18

10 Jul '18

Hi, Alexey! On Jul 10, Alexey Botchkov wrote: > > > Did you test that it works? > Of course i tested the mysql_install_db script a lot. > Didn't test the .rpm and .deb though. As far as i can see they plainly run > mysql_install_db --user=.. under root. could you please test with rpm and deb too? should only take 10-20 minutes in a buildbot VM. ok to push after that (if all goes well :)) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 1a2748c: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 09 Jul '18

09 Jul '18

Hi, Alexey! On Jul 09, Alexey Botchkov wrote: > revision-id: 1a2748ceb0e30770eebc97d82d7baa885b332c49 (mariadb-10.3.6-45-g1a2748c) > parent(s): aa01f51bdef9cc38d8e0a75ea9e2788651e41d16 > committer: Alexey Botchkov > timestamp: 2018-07-09 13:20:42 +0400 > message: > > MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server. > > Scripts added to set safe permissions for the auth_pam_tool and it's > directory. > > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > index ad7c028..fd26228 100644 > --- a/scripts/mysql_install_db.sh > +++ b/scripts/mysql_install_db.sh > @@ -308,6 +308,7 @@ then > srcpkgdatadir="$srcdir/scripts" > buildpkgdatadir="$builddir/scripts" > plugindir="$builddir/plugin/auth_socket" > + pamtooldir="$builddir/plugin/auth_pam" > elif test -n "$basedir" > then > bindir="$basedir/bin" # only used in the help text > @@ -337,6 +338,7 @@ then > exit 1 > fi > plugindir=`find_in_dirs --dir auth_socket.so $basedir/lib*/plugin $basedir/lib*/mysql/plugin` > + pamtooldir=`find_in_dirs --dir auth_pam.so $basedir/lib*/plugin $basedir/lib*/mysql/plugin` Why not just pamtooldir=$plugindir ? > else > basedir="@prefix@" > bindir="@bindir@" > @@ -445,6 +448,13 @@ done > > if test -n "$user" > then > + chown $user "$pamtooldir/auth_pam_tool_dir" > + if test $? -ne 0 > + then > + echo "Cannot change ownership of the '$pamtooldir\auth_pam_tool_dir' directory" > + echo " to the '$user' user. Check that you have the necessary permissions and try again." > + exit 1 > + fi Looks ok. Did you test that it works? I mean, a test woud be to install from a tarball and run mysql_install_db or install from rpm or deb (they run the install script automatically) and then check that all permissions are correct. Also, I suspect that at additional chown root "$pamtooldir/auth_pam_tool_dir/auth_pam_tool could also be useful here. But not when the script is running in the builddir. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 765ba2a: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 06 Jul '18

06 Jul '18

Hi, Alexey! On Jul 05, Alexey Botchkov wrote: > revision-id: 765ba2ac76dab984183bf829dc3407713a4d5d9b (mariadb-10.3.6-40-g765ba2a) > parent(s): 7e704a2308e25953b5f8fb154eb325df3e25c2ec > committer: Alexey Botchkov > timestamp: 2018-07-05 17:00:47 +0400 > message: > > MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server. > > Proper access permissions for the auth_pam_tool_dir and auth_pam_tool. > > diff --git a/plugin/auth_pam/CMakeLists.txt b/plugin/auth_pam/CMakeLists.txt > index 4943d57..b9313de 100644 > --- a/plugin/auth_pam/CMakeLists.txt > +++ b/plugin/auth_pam/CMakeLists.txt > @@ -11,7 +11,13 @@ IF(HAVE_PAM_APPL_H) > ADD_DEFINITIONS(-D_GNU_SOURCE) > MYSQL_ADD_PLUGIN(auth_pam_v1 auth_pam_v1.c LINK_LIBRARIES pam MODULE_ONLY) > MYSQL_ADD_PLUGIN(auth_pam auth_pam.c LINK_LIBRARIES pam dl MODULE_ONLY) > - MYSQL_ADD_EXECUTABLE(auth_pam_tool auth_pam_tool.c DESTINATION ${INSTALL_PLUGINDIR}/auth_pam_tool_dir COMPONENT Server) > + MYSQL_ADD_EXECUTABLE(auth_pam_tool auth_pam_tool.c DESTINATION ${INSTALL_PLUGINDIR}/auth_pam_tool_dir COMPONENT Server) > TARGET_LINK_LIBRARIES(auth_pam_tool pam) > + INSTALL(CODE "EXECUTE_PROCESS( > + COMMAND chmod u=rwx,g=,o= auth_pam_tool_dir > + COMMAND chmod u=rwx,g=rx,o=rx auth_pam_tool_dir/auth_pam_tool > + COMMAND chmod +s auth_pam_tool_dir/auth_pam_tool > + WORKING_DIRECTORY \$ENV{DESTDIR}\${CMAKE_INSTALL_PREFIX}/${INSTALL_PLUGINDIR}/)" > + COMPONENT Server) I think it's generally ok. Two comments: minor: you can combine two chmods on auth_pam_tool in one, like u=rwxs major: you still need to make auth_pam_tool_dir to be owned by mysql user. I'm afraid the only way to do it is from a post-install scriptlet or from mysql_install_db. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] af15686: MDEV-16242: MyRocks: parallel slave on a table without PK can stop with ER_KEY_NOT_FOUND
by Kristian Nielsen 05 Jul '18

05 Jul '18

psergey(a)askmonty.org (Sergei Petrunia) writes: > MDEV-16242: MyRocks: parallel slave on a table without PK can stop with ER_KEY_NOT_FOUND > > DRAFT: If RBR event applier uses a secondary key or a full table scan > to locate a row, force waiting for prior commit to complete. I think if you wait_for_prior_commit() also for secondary key case, you should definitely restrict it to be only for conservative parallel replication, not for optimistic/agressive. Using a secondary key seems generally fine (as opposed to full table scan), and this problem should affect only conservative mode, no need to penalise optimistic/aggressive. (But maybe that is what is meant with "DRAFT"). - Kristian.

2 2

Re: [Maria-developers] b88e98df9da: C++11 experiment: check if all builders support it
by Sergei Golubchik 04 Jul '18

04 Jul '18

Hi, Sergey! > Current status is: all staging builders pass, except for Ubuntu Precise. But we no longer build on precise, do we? Any followup MDEVs after this one is pushed? Like migrate atomics ops to C++11 or something? > revision-id: b88e98df9daa586bd6ed1f5908ededaa65a6e464 (mariadb-10.3.6-16-gb88e98df9da) > parent(s): 9988a423d229730b252942b7fa066d1591b7eca9 > author: Sergey Vojtovich > committer: Sergey Vojtovich > timestamp: 2018-06-04 12:41:18 +0400 > message: > > C++11 experiment: check if all builders support it > > diff --git a/CMakeLists.txt b/CMakeLists.txt > index 76b0817f8c7..efeae3d1ca5 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -89,6 +89,14 @@ ELSE() > ENDIF() > PROJECT(${MYSQL_PROJECT_NAME}) > > +IF(CMAKE_VERSION VERSION_LESS "3.1") > + IF(CMAKE_CXX_COMPILER_ID STREQUAL "GNU") > + SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11") Why gnu++11 and not c++11 ? And why not for clang? > + ENDIF() > +ELSE() > + SET(CMAKE_CXX_STANDARD 11) I'd expect this to mean c++11, not gnu++11. > +ENDIF() > + > SET(CPACK_PACKAGE_NAME "MariaDB") > SET(CPACK_PACKAGE_DESCRIPTION_SUMMARY "MariaDB: a very fast and robust SQL database server") > SET(CPACK_PACKAGE_URL "http://mariadb.org") > diff --git a/storage/tokudb/PerconaFT/portability/toku_crash.cc b/storage/tokudb/PerconaFT/portability/toku_crash.cc > index 0af85342a99..297cc29d9ca 100644 > --- a/storage/tokudb/PerconaFT/portability/toku_crash.cc > +++ b/storage/tokudb/PerconaFT/portability/toku_crash.cc > @@ -70,7 +70,7 @@ run_gdb(pid_t parent_pid, const char *gdb_path) { > "-ex", "thread apply all bt", > "-ex", "thread apply all bt full", > exe_buf, pid_buf, > - NULL); > + (char*) NULL); Why's that? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

3 3

[Maria-developers] MDEV-13995 MAX(timestamp) returns a wrong result near DST change
by Alexander Barkov 03 Jul '18

03 Jul '18

Hi Sergei, Please review a patch for MDEV-13995. Thanks!

1 0

[Maria-developers] FOR loop documentation
by Alexander Barkov 03 Jul '18

03 Jul '18

Hi Ian, can you please document the FOR loop? I've created an MDEV for it: MDEV-16674 Document FOR loop with links to all relevant tasks. Thanks!

1 0

Re: [Maria-developers] ed2dcf0: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 01 Jul '18

01 Jul '18

Hi, Alexey! Looks good, few comments/suggestions below. That and my reply to your email (about tests and plugin name). Just push when you're ready, no need to put it In-Review again. Thanks! On Jul 01, Alexey Botchkov wrote: > revision-id: ed2dcf00fd26e363f34638a841ff17a111b7e324 (mariadb-10.3.6-33-ged2dcf0) > parent(s): 4b0cedf82d8d8ba582648dcb4a2620c146862a43 > committer: Alexey Botchkov > timestamp: 2018-07-01 12:23:54 +0400 > message: > > MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server. > > The new 'safe' version of the PAM plugin. > It works using the auth_pam_tool executable. That isolates possible > crashes in pam modules and, since we can set the +s bit for the tool, > we can normally run modules that read protected information. > > diff --git a/plugin/auth_pam/CMakeLists.txt b/plugin/auth_pam/CMakeLists.txt > index 5131752..9c0859c 100644 > --- a/plugin/auth_pam/CMakeLists.txt > +++ b/plugin/auth_pam/CMakeLists.txt > @@ -8,6 +8,10 @@ IF(HAVE_PAM_APPL_H) > IF(HAVE_STRNDUP) > ADD_DEFINITIONS(-DHAVE_STRNDUP) > ENDIF(HAVE_STRNDUP) > + ADD_DEFINITIONS(-D_GNU_SOURCE) > MYSQL_ADD_PLUGIN(auth_pam auth_pam.c LINK_LIBRARIES pam MODULE_ONLY) > + MYSQL_ADD_PLUGIN(auth_pam_safe auth_pam_safe.c LINK_LIBRARIES pam dl MODULE_ONLY) perhaps, rename to auth_pam_new.c (and auth_pam_new.so)? > + MYSQL_ADD_EXECUTABLE(auth_pam_tool auth_pam_tool.c DESTINATION ${INSTALL_PLUGINDIR}/auth_pam_tool_dir COMPONENT Server) > + TARGET_LINK_LIBRARIES(auth_pam_tool pam) > ENDIF(HAVE_PAM_APPL_H) > > diff --git a/plugin/auth_pam/auth_pam.c b/plugin/auth_pam/auth_pam.c > index ffc3d6f..c785ba4 100644 > --- a/plugin/auth_pam/auth_pam.c > +++ b/plugin/auth_pam/auth_pam.c > @@ -1,5 +1,5 @@ > /* > - Copyright (c) 2011, 2012, Monty Program Ab > + Copyright (c) 2018 MariaDB Corporation should be "2011, 2018" not just "2018" > This program is free software; you can redistribute it and/or modify > it under the terms of the GNU General Public License as published by > static struct st_mysql_auth info = > diff --git a/plugin/auth_pam/auth_pam_base.c b/plugin/auth_pam/auth_pam_base.c > new file mode 100644 > index 0000000..30b4e01 > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_base.c > @@ -0,0 +1,171 @@ > +/* > + Copyright (c) 2018 MariaDB Corporation > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111-1301 USA */ > + > +// struct param { > + // unsigned char buf[10240], *ptr; > + // MYSQL_PLUGIN_VIO *vio; > +// }; > +//static int write_packet(struct param *param, const unsigned char *buf, > + //int buf_len) > +//static int read_packet(struct param *param, unsigned char **pkt) forgot to remove these comments? > + > +#include <stdio.h> > +#include <string.h> > +#include <security/pam_appl.h> > +#include <security/pam_modules.h> > + > diff --git a/plugin/auth_pam/auth_pam_safe.c b/plugin/auth_pam/auth_pam_safe.c > new file mode 100644 > index 0000000..9dac90d > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_safe.c > @@ -0,0 +1,235 @@ ... > + PAM_DEBUG((stderr, "PAM: Child process prepares pipes.\n")); > + > + if (close(p_to_c[1]) < 0 || > + close(c_to_p[0]) < 0 || > + dup2(p_to_c[0], 0) < 0 || /* Parent's pipe to STDIN. */ > + dup2(c_to_p[1], 1) < 0) /* Sandbox's pipe to STDOUT. */ > + { > + exit(-1); > + } > + > + PAM_DEBUG((stderr, "PAM: check tool directory: %s, %s.\n", > + opt_plugin_dir, tool_name)); > + plugin_dir_len= strlen(opt_plugin_dir); > + if (plugin_dir_len + tool_name_len + 2 > sizeof(toolpath)) > + { > + /* Tool path too long. */ > + exit(-1); > + } > + > + memcpy(toolpath, opt_plugin_dir, plugin_dir_len); > + if (plugin_dir_len && toolpath[plugin_dir_len-1] != FN_LIBCHAR) > + toolpath[plugin_dir_len++]= FN_LIBCHAR; > + memcpy(toolpath+plugin_dir_len, tool_name, tool_name_len+1); check the length before copying? like, toolpath+plugin_dir_len+tool_name_len+1 < sizeof(toolpath) ? or simply strnxfrm(toolpath, sizeof(toolpath), opt_plugin_dir, "/", tool_name, NULL); > + > + PAM_DEBUG((stderr, "PAM: execute pam sandbox [%s].\n", toolpath)); > + (void) execl(toolpath, toolpath, NULL); > + PAM_DEBUG((stderr, "PAM: exec() failed.\n")); > + exit(-1); > + } ... > +static char use_cleartext_plugin; > +static MYSQL_SYSVAR_BOOL(use_cleartext_plugin, use_cleartext_plugin, > + PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY, > + "Use mysql_cleartext_plugin on the client side instead of the dialog " > + "plugin. This may be needed for compatibility reasons, but it only " > + "supports simple PAM policies that don't require anything besides " > + "a password", NULL, NULL, 0); > + > +#ifndef DBUG_OFF > +static MYSQL_SYSVAR_BOOL(debug, pam_debug, PLUGIN_VAR_OPCMDARG, > + "Log all PAM activity", NULL, NULL, 0); > +#endif still a lot of similar code in auth_pam.c and auth_pam_safe.c. could be moved to auth_pam_common.c? > + > + > diff --git a/plugin/auth_pam/auth_pam_tool.c b/plugin/auth_pam/auth_pam_tool.c > new file mode 100644 > index 0000000..5f19d29 > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_tool.c > @@ -0,0 +1,132 @@ > + > +int main(int argc, char **argv) > +{ > + struct param param; > + MYSQL_SERVER_AUTH_INFO info; > + unsigned char field; > + int res; > + char a_buf[MYSQL_USERNAME_LENGTH + 1 + 1024]; > + > + if (read(0, &field, 1) < 1) > + return -1; > +#ifndef DBUG_OFF > + pam_debug= field; > +#endif > + > + PAM_DEBUG((stderr, "PAM: sandbox started [%s].\n", argv[0])); > + > + info.user_name= a_buf; > + if ((res= read_string(0, info.user_name, sizeof(a_buf))) < 0) > + return -1; > + PAM_DEBUG((stderr, "PAM: sandbox username [%s].\n", info.user_name)); > + > +#ifndef DBUG_OFF > + /* Produce the crash for testing purposes. */ > + if ((res == 14) && > + memcmp(info.user_name, "crash_pam_tool", 14) == 0) > + { > + info.user_name= 0; > + memcpy(info.user_name, a_buf, sizeof(a_buf)); > + return -1; > + } now I think it's better to put it into pam_mariadb_mtr.c, sorry :( then it doesn't need to be in debug-only builds > +#endif > + > + info.auth_string= info.user_name + res + 1; > + if (read_string(0, info.auth_string, sizeof(a_buf) - 1 - res) < 0) > + return -1; > + > + PAM_DEBUG((stderr, "PAM: sandbox auth string [%s].\n", info.auth_string)); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 84c9ec5: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 01 Jul '18

01 Jul '18

Hi, Alexey! On Jun 30, Alexey Botchkov wrote: > > not sure about calling it "safe". I think it's more of a side > > effect, the main feature it that it works, while old pam plugin > > simply doesn't :) unless mysqld is run as root. > > > > and not totally sure about calling it also pam. it means one won't > > be able to load it and the old pam plugin at the same time. I > > suspect it'll still ok and benefits overweight it. > > Well i'm open to any ideas here :) > We can swap 'safe' with somethin else. 'box' for instance. > Or we can instead rename the original version like 'fast' or 'old'. > Finally we can build only one version of the plugin. The 'safe' > whenever possible. I'd rather rename the old version to (old) or (requires root) or something. > I also have a question about the testing. > Now pam*.test-s rely on some mariadb_mtr setup for PAM. How can i see what > is in the expected pam.d configuration file, and what pam modules are used? See plugin/auth_pam/testing/pam_mariadb_mtr.c Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] FYI: 10.3 master branch head builds OK Travis-CI
by Otto Kekäläinen 30 Jun '18

30 Jun '18

Hello! I am happy to announce that of this morning, Travis-CI for MariaDB 10.3 shows all green again! [1] Developers: please rebase your 10.3 branches on top of the latest 10.3 to get the Travis-CI benefits, and please don't break Travis. Last time it was all green in January it took only a few weeks until it was broken again. We don't have protected branches enabled yet, so we rely on all developers to be diligent and not push changes that break the all-green-status on Travis-CI for 10.3. Otto [1] https://travis-ci.org/MariaDB/server/builds [2} https://mariadb.org/protected-branches-ensuring-code-quality-git/

1 0

[Maria-developers] MariaDB 10.3.7 and GSSAPI : Issue
by REIX, Tony 28 Jun '18

28 Jun '18

Hi, We are porting MariaDB (10.3.7 now) on AIX. On some machines we have the following error message when running: + cmake . --trace --debug-trycompile --check-system-vars -GUnix Makefiles -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DINSTALL_LAYOUT=RPM -DLOG_LOCATION=/opt/freeware/mariadb/mariadb.log -DPID_FILE_DIR=%{_rundir}/mariadb -DNICE_PROJECT_NAME=MariaDB -DCMAKE_INSTALL_PREFIX=/opt/freeware -DCMAKE_AR=/usr/bin/ar -DCMAKE_SYSTEM_NAME=AIX -DINSTALL_SYSCONFDIR=/opt/freeware/etc -DINSTALL_SYSCONF2DIR=/opt/freeware/etc/my.cnf.d -DINSTALL_INCLUDEDIR=include/mysql -DINSTALL_INFODIR=share/info -DINSTALL_LIBDIR=/opt/freeware/lib64 -DINSTALL_MANDIR=share/man -DINSTALL_MYSQLSHAREDIR=share/mariadb -DINSTALL_PLUGINDIR=/opt/freeware/lib64/mariadb/plugin -DINSTALL_MYSQLTESTDIR=share/mysql-test -DINSTALL_SBINDIR=libexec -DINSTALL_SCRIPTDIR=bin -DINSTALL_SQLBENCHDIR=share -DINSTALL_SUPPORTFILESDIR=share/mariadb -DMYSQL_DATADIR=/opt/freeware/var/lib/mysql -DMYSQL_UNIX_ADDR=/var/lib64/mysql/mysql.sock -DTMPDIR=/var/tmp -DENABLED_LOCAL_INFILE=ON -DSECURITY_HARDENED=ON -DENABLE_DTRACE=ON -DWITH_EMBEDDED_SERVER=ON -DWITH_MARIABACKUP=ON -DWITH_UNIT_TESTS=ON -DCONC_WITH_SSL=ON -DWITH_SSL=system -DWITH_ZLIB=system -DWITH_JEMALLOC=no -DPLUGIN_MROONGA=DYNAMIC -DPLUGIN_OQGRAPH=DYNAMIC -DPLUGIN_CRACKLIB_PASSWORD_CHECK=DYNAMIC -DPLUGIN_ROCKSDB=DYNAMIC -DPLUGIN_SPHINX=DYNAMIC -DPLUGIN_TOKUDB=DYNAMIC -DPLUGIN_CONNECT=DYNAMIC -DCONNECT_WITH_MONGO=OFF -DCONNECT_WITH_JDBC=OFF -DCMAKE_CROSSCOMPILING=FALSE -DWITH_VALGRIND=OFF -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN=OFF -DWITH_INNODB_EXTRA_DEBUG=ON ...... /opt/freeware/share/cmake/Modules/CheckSymbolExists.cmake(87): try_compile(HAVE_KRB5_FREE_UNPARSED_NAME ${CMAKE_BINARY_DIR} /opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/CMakeFiles/CMakeTmp/CheckSymbolExists.cxx COMPILE_DEFINITIONS ${CMAKE_REQUIRED_DEFINITIONS} ${CHECK_SYMBOL_EXISTS_LIBS} CMAKE_FLAGS -DCOMPILE_DEFINITIONS:STRING=${MACRO_CHECK_SYMBOL_EXISTS_FLAGS} ${CMAKE_SYMBOL_EXISTS_INCLUDES} OUTPUT_VARIABLE OUTPUT ) CMake Error at /opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/CMakeFiles/CMakeTmp/CMakeLists.txt:14 (add_executable): Target "cmTC_3efe0" links to target "-L/opt/freeware/lib -blibpath:/opt/freeware/lib::/usr/lib:/lib -I/usr/include -I/opt/freeware/include -L/opt/freeware/lib -blibpath:/opt/freeware/lib:/usr/lib:/lib -bmaxdata:0x80000000 -brtl -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lpthreads" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing? CMake Error at /opt/freeware/share/cmake/Modules/CheckSymbolExists.cmake:87 (try_compile): Failed to generate test project build system. Call Stack (most recent call first): /opt/freeware/share/cmake/Modules/CheckCXXSymbolExists.cmake:39 (__CHECK_SYMBOL_EXISTS_IMPL) plugin/auth_gssapi/CMakeLists.txt:24 (CHECK_CXX_SYMBOL_EXISTS) In file: mariadb-10.3.7/64bit/CMakeFiles/CMakeTmp/CMakeLists.txt , there is: cmake_minimum_required(VERSION 3.11.4.0) set(CMAKE_MODULE_PATH "/opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/cmake;/opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/plugin/auth_gssapi/cmake") project(CMAKE_TRY_COMPILE CXX) set(CMAKE_VERBOSE_MAKEFILE 1) set(CMAKE_CXX_FLAGS " -pie -fPIC -fPIC -fno-rtti") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${COMPILE_DEFINITIONS}") set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${EXE_LINKER_FLAGS}") include_directories(${INCLUDE_DIRECTORIES}) set(CMAKE_SUPPRESS_REGENERATION 1) link_directories(${LINK_DIRECTORIES}) add_definitions(-DPACKAGE=test -D_LARGEFILE_SOURCE -D_LARGE_FILES -D_FILE_OFFSET_BITS=64 -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS) cmake_policy(SET CMP0065 OLD) set(CMAKE_RUNTIME_OUTPUT_DIRECTORY "/opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/CMakeFiles/CMakeTmp") add_executable(cmTC_3efe0 "/opt/freeware/src/packages/BUILD/mariadb-10.3.7/64bit/CMakeFiles/CMakeTmp/CheckSymbolExists.cxx") target_link_libraries(cmTC_3efe0 "-L/opt/freeware/lib -blibpath:/opt/freeware/lib::/usr/lib:/lib -I/usr/include -I/opt/freeware/include -L/opt/freeware/lib -blibpath:/opt/freeware/lib:/usr/lib:/lib -bmaxdata:0x80000000 -brtl -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lpthreads" ) Talking with CMake guys, they said: > I suspect mariadb's source is setting CMAKE_REQUIRED_LIBRARIES > to a space-separated value instead of a ;-separated value. > That's broken here: https://github.com/MariaDB/server/blob/ed0b84a0270bd99b001dd00654875d26e29b…<https://github.com/MariaDB/server/blob/ed0b84a0270bd99b001dd00654875d26e29b…> Do you have an idea about what is broken and what should be changed? either on AIX or for all target? Thanks/Regards, Cordialement, Tony Reix ATOS / Bull SAS ATOS Expert IBM Coop Architect & Technical Leader Office : +33 (0) 4 76 29 72 67 1 rue de Provence - 38432 Échirolles - France www.atos.net<https://mail.ad.bull.net/owa/redir.aspx?C=PvphmPvCZkGrAgHVnWGsdMcDKgzl_dEIs…>

2 3

[Maria-developers] Dead code in class sp_head
by Alexander Barkov 26 Jun '18

26 Jun '18

Hi Aleksey, I found some dead code in sp_head that seem to have appeared in your patch for "IB: 0.2 part IV". If you don't mind, I'll remove it. Please have a look into the attached patch. Thanks.

2 1

Re: [Maria-developers] [Commits] MDEV-16166 RBR breaks with HA_ERR_KEY_NOT_FOUND upon DELETE
by Alexey Botchkov 25 Jun '18

25 Jun '18

Ok to push. 05.06.2018 13:45, Sachin Setiya wrote: > commit bde95bc70d0ac011f55f1b3b55fee1c9276859da > Author: sachin <sachin.setiya(a)maridb.com > <mailto:sachin.setiya@maridb.com>> > Date: Tue Jun 5 15:14:19 2018 +0530 > > MDEV-16166 RBR breaks with HA_ERR_KEY_NOT_FOUND upon DELETE from > table... > with spatial index > So the issue is since it is spatial index , at the time of > searching index > for key (Rows_log_event::find_row) we use wrong field image we use > Field::itRAW while we should be using Field::itMBR > > diff --git a/mysql-test/suite/rpl/r/rpl_row_spatial.result > b/mysql-test/suite/rpl/r/rpl_row_spatial.result > new file mode 100644 > index 00000000000..8f546fc479e > --- /dev/null > +++ b/mysql-test/suite/rpl/r/rpl_row_spatial.result > @@ -0,0 +1,14 @@ > +include/master-slave.inc > +[connection master] > +CREATE TABLE t1 (g POINT NOT NULL, SPATIAL INDEX(g)); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(1 1)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(2 1)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(1 2)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(2 2)')); > +DELETE FROM t1 where MBREqual(g, ST_GEOMFROMTEXT('Point(1 2)')); > +select count(*) from t1; > +count(*) > +3 > +DELETE FROM t1; > +drop table t1; > +include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_row_spatial.test > b/mysql-test/suite/rpl/t/rpl_row_spatial.test > new file mode 100644 > index 00000000000..00c3dd7c54d > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_row_spatial.test > @@ -0,0 +1,17 @@ > +--source include/have_binlog_format_row.inc > +--source include/master-slave.inc > + > +CREATE TABLE t1 (g POINT NOT NULL, SPATIAL INDEX(g)); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(1 1)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(2 1)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(1 2)')); > +INSERT INTO t1 VALUES (ST_GEOMFROMTEXT('Point(2 2)')); > +DELETE FROM t1 where MBREqual(g, ST_GEOMFROMTEXT('Point(1 2)')); > + > +--sync_slave_with_master > +select count(*) from t1; > + > +--connection master > +DELETE FROM t1; > +drop table t1; > +--source include/rpl_end.inc > diff --git a/sql/key.cc b/sql/key.cc > index 700bf6a05a6..7e5a3309b10 100644 > --- a/sql/key.cc > +++ b/sql/key.cc > @@ -145,7 +145,8 @@ void key_copy(uchar *to_key, uchar *from_record, > KEY *key_info, > { > key_length-= HA_KEY_BLOB_LENGTH; > length= min(key_length, key_part->length); > - uint bytes= key_part->field->get_key_image(to_key, length, > Field::itRAW); > + uint bytes= key_part->field->get_key_image(to_key, length, > +key_info->flags & HA_SPATIAL ? Field::itMBR : Field::itRAW); > if (with_zerofill && bytes < length) > bzero((char*) to_key + bytes, length - bytes); > to_key+= HA_KEY_BLOB_LENGTH; > > > -- > Regards > Sachin Setiya > Software Engineer at MariaDB > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] MDEV-12574 MAX(old_decimal) produces a column of the old DECIMAL type
by Alexander Barkov 24 Jun '18

24 Jun '18

Hi Sergei, Please review a patch for MDEV-12574. I suggest we push it into 10.2 (not into 10.3). Thanks!

2 1

Re: [Maria-developers] 37b432c1d59: MDEV-15242 Poor RBR update performance with partitioned tables
by Sergei Golubchik 22 Jun '18

22 Jun '18

Hi, andrei.elkin! On Jun 19, andrei.elkin(a)pp.inet.fi wrote: > revision-id: 37b432c1d59e66b742f97a280cae7784d85e62ec (mariadb-10.0.35-41-g37b432c1d59) > parent(s): d2e1ed8b936a78ceeec0e64231997d7ed18a4daf > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-06-19 18:14:47 +0300 > message: > > MDEV-15242 Poor RBR update performance with partitioned tables > > Observed and described > partitioned engine execution time difference > between master and slave was caused by excessive invocation > of base_engine::rnd_init which was done also for partitions > uninvolved into Rows-event operation. > The bug's slave slowdown therefore scales with the number of partitions. Ok to push > Fixed with applying an upstream patch. > > References: > ---------- > https://bugs.mysql.com/bug.php?id=73648 > Bug#25687813 REPLICATION REGRESSION WITH RBR AND PARTITIONED TABLES > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MariaDB Developers Unconference next week
by Otto Kekäläinen 22 Jun '18

22 Jun '18

Hello everybody! If you plan to attend the MariaDB Developers Unconference and related activities next week,* make sure you have signed up <https://www.meetup.com/MariaDB-Developers/events/249024232/> to the meetup* . We will soon be posting important information to attendees and only people who have signed up will receive it. Info: - www.meetup.com/MariaDB-Developers/events/249024232/ <https://www.meetup.com/MariaDB-Developers/events/249024232/> - mariadb.org/2018-2-developers-unconference-in-finland/ - mariadb.org/travel-tips-to-tampere-finland/

1 0

Re: [Maria-developers] 84c9ec5: MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server.
by Sergei Golubchik 21 Jun '18

21 Jun '18

Hi, Alexey! Just a couple of comments (args[] vs stdio and how to prevent brute-forcing), see below. Otherwise looks pretty good. On Jun 06, Alexey Botchkov wrote: > revision-id: 84c9ec5552bf14bda826912fb045ae36f34a4490 (mariadb-10.3.6-9-g84c9ec5) > parent(s): 637af7838316a49267e55cde7cf96e23f63e5c9d > committer: Alexey Botchkov > timestamp: 2018-06-06 14:13:07 +0400 > message: > > MDEV-15473 Isolate/sandbox PAM modules, so that they can't crash the server. > > Crash-safe version of the PAM plugin added. It runs the auth_pam_tool > application that actually calls the PAM modules. > > diff --git a/plugin/auth_pam/auth_pam_base.c b/plugin/auth_pam/auth_pam_base.c > new file mode 100644 > index 0000000..885e6f0 > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_base.c > @@ -0,0 +1,173 @@ > +/* > + Copyright (c) 2011, 2012, Monty Program Ab could be updated... > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111-1301 USA */ > + > +// struct param { > + // unsigned char buf[10240], *ptr; > + // MYSQL_PLUGIN_VIO *vio; > +// }; > +//static int write_packet(struct param *param, const unsigned char *buf, > + //int buf_len) > +//static int read_packet(struct param *param, unsigned char **pkt) > + > +#define _GNU_SOURCE 1 /* for strndup */ > + > +#include <stdio.h> > +#include <string.h> > +#include <security/pam_appl.h> > +#include <security/pam_modules.h> > + > +/* It least solaris doesn't have strndup */ > + > +#ifndef HAVE_STRNDUP > +char *strndup(const char *from, size_t length) > +{ > + char *ptr; > + size_t max_length= strlen(from); > + if (length > max_length) > + length= max_length; > + if ((ptr= (char*) malloc(length+1)) != 0) > + { > + memcpy((char*) ptr, (char*) from, length); > + ptr[length]=0; > + } > + return ptr; > +} > +#endif > + > +#ifndef DBUG_OFF > +static char pam_debug = 0; > +#define PAM_DEBUG(X) do { if (pam_debug) { fprintf X; } } while(0) > +#else > +#define PAM_DEBUG(X) /* no-op */ > +#endif > + > +static int conv(int n, const struct pam_message **msg, > + struct pam_response **resp, void *data) > +{ > + struct param *param = (struct param *)data; > + unsigned char *end = param->buf + sizeof(param->buf) - 1; > + int i; > + > + *resp = 0; > + > + for (i = 0; i < n; i++) > + { > + /* if there's a message - append it to the buffer */ > + if (msg[i]->msg) > + { > + int len = strlen(msg[i]->msg); > + if (len > end - param->ptr) > + len = end - param->ptr; > + if (len > 0) > + { > + memcpy(param->ptr, msg[i]->msg, len); > + param->ptr+= len; > + *(param->ptr)++ = '\n'; > + } > + } > + /* if the message style is *_PROMPT_*, meaning PAM asks a question, > + send the accumulated text to the client, read the reply */ > + if (msg[i]->msg_style == PAM_PROMPT_ECHO_OFF || > + msg[i]->msg_style == PAM_PROMPT_ECHO_ON) > + { > + int pkt_len; > + unsigned char *pkt; > + > + /* allocate the response array. > + freeing it is the responsibility of the caller */ > + if (*resp == 0) > + { > + *resp = calloc(sizeof(struct pam_response), n); > + if (*resp == 0) > + return PAM_BUF_ERR; > + } > + > + /* dialog plugin interprets the first byte of the packet > + as the magic number. > + 2 means "read the input with the echo enabled" > + 4 means "password-like input, echo disabled" > + C'est la vie. */ > + param->buf[0] = msg[i]->msg_style == PAM_PROMPT_ECHO_ON ? 2 : 4; > + PAM_DEBUG((stderr, "PAM: conv: send(%.*s)\n", > + (int)(param->ptr - param->buf - 1), param->buf)); > + if (write_packet(param, param->buf, param->ptr - param->buf - 1)) > + return PAM_CONV_ERR; > + > + pkt_len = read_packet(param, &pkt); > + if (pkt_len < 0) > + { > + PAM_DEBUG((stderr, "PAM: conv: recv() ERROR\n")); > + return PAM_CONV_ERR; > + } > + PAM_DEBUG((stderr, "PAM: conv: recv(%.*s)\n", pkt_len, pkt)); > + /* allocate and copy the reply to the response array */ > + if (!((*resp)[i].resp= strndup((char*) pkt, pkt_len))) > + return PAM_CONV_ERR; > + param->ptr = param->buf + 1; > + } > + } > + return PAM_SUCCESS; > +} > + > +#define DO(X) if ((status = (X)) != PAM_SUCCESS) goto end > + > +#if defined(SOLARIS) || defined(__sun) > +typedef void** pam_get_item_3_arg; > +#else > +typedef const void** pam_get_item_3_arg; > +#endif > + > +static int pam_auth_base(struct param *param, MYSQL_SERVER_AUTH_INFO *info) > +{ > + pam_handle_t *pamh = NULL; > + int status; > + const char *new_username= NULL; > + /* The following is written in such a way to make also solaris happy */ > + struct pam_conv pam_start_arg = { &conv, (char*) param }; > + > + /* > + get the service name, as specified in > + > + CREATE USER ... IDENTIFIED WITH pam AS "service" > + */ > + const char *service = info->auth_string && info->auth_string[0] > + ? info->auth_string : "mysql"; > + > + param->ptr = param->buf + 1; > + > + PAM_DEBUG((stderr, "PAM: pam_start(%s, %s)\n", service, info->user_name)); > + DO( pam_start(service, info->user_name, &pam_start_arg, &pamh) ); > + > + PAM_DEBUG((stderr, "PAM: pam_authenticate(0)\n")); > + DO( pam_authenticate (pamh, 0) ); > + > + PAM_DEBUG((stderr, "PAM: pam_acct_mgmt(0)\n")); > + DO( pam_acct_mgmt(pamh, 0) ); > + > + PAM_DEBUG((stderr, "PAM: pam_get_item(PAM_USER)\n")); > + DO( pam_get_item(pamh, PAM_USER, (pam_get_item_3_arg) &new_username) ); > + > + if (new_username && strcmp(new_username, info->user_name)) > + strncpy(info->authenticated_as, new_username, > + sizeof(info->authenticated_as)); > + info->authenticated_as[sizeof(info->authenticated_as)-1]= 0; > + > +end: > + pam_end(pamh, status); > + PAM_DEBUG((stderr, "PAM: status = %d user = %s\n", status, info->authenticated_as)); > + return status == PAM_SUCCESS ? CR_OK : CR_ERROR; > +} > + > diff --git a/plugin/auth_pam/auth_pam_safe.c b/plugin/auth_pam/auth_pam_safe.c > new file mode 100644 > index 0000000..28c10b8 > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_safe.c > @@ -0,0 +1,235 @@ > +/* > + Copyright (c) 2011, 2012, Monty Program Ab could be updated... > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111-1301 USA */ > + > +#define _GNU_SOURCE 1 /* for strndup */ > + > +#include <unistd.h> > +#include <string.h> > +#include <mysql/plugin_auth.h> > +#include "auth_pam_tool.h" > +#include <my_global.h> > + > +#ifndef DBUG_OFF > +static char pam_debug = 0; > +#define PAM_DEBUG(X) do { if (pam_debug) { fprintf X; } } while(0) > +#else > +#define PAM_DEBUG(X) /* no-op */ > +#endif > + > +static char *opt_plugin_dir; /* To be dynamically linked. */ > +static const char *tool_name= "auth_pam_tool"; > +static const int tool_name_len= 13; > + > +static int pam_auth(MYSQL_PLUGIN_VIO *vio, MYSQL_SERVER_AUTH_INFO *info) > +{ > + int p_to_c[2], c_to_p[2]; /* Parent-to-child and child-to-parent pipes. */ > + pid_t proc_id; > + int result= CR_ERROR;; double semicolon :) > + > + PAM_DEBUG((stderr, "PAM: opening pipes.\n")); > + if (pipe(p_to_c) < 0 || pipe(c_to_p) < 0) > + { > + /* Error creating pipes. */ > + return CR_ERROR; > + } > + PAM_DEBUG((stderr, "PAM: forking.\n")); > + if ((proc_id= fork()) < 0) > + { > + /* Error forking. */ > + close(p_to_c[0]); > + close(c_to_p[1]); > + goto error_ret; > + } > + > + if (proc_id == 0) > + { > + /* The 'sandbox' process started. */ > + const char *arg[5]; > + char toolpath[FN_REFLEN]; > + size_t plugin_dir_len; > + > + PAM_DEBUG((stderr, "PAM: Child process prepares pipes.\n")); > + > + if (close(p_to_c[1]) < 0 || > + close(c_to_p[0]) < 0 || > + dup2(p_to_c[0], 0) < 0 || /* Parent's pipe to STDIN. */ > + dup2(c_to_p[1], 1) < 0) /* Sandbox's pipe to STDOUT. */ > + { > + exit(-1); > + } > + > + PAM_DEBUG((stderr, "PAM: check tool directory: %s, %s.\n", > + opt_plugin_dir, tool_name)); > + plugin_dir_len= strlen(opt_plugin_dir); > + if (plugin_dir_len + tool_name_len + 2 > sizeof(toolpath)) > + { > + /* Tool path too long. */ > + exit(-1); > + } > + > + memcpy(toolpath, opt_plugin_dir, plugin_dir_len); > + if (plugin_dir_len && toolpath[plugin_dir_len-1] != FN_LIBCHAR) > + toolpath[plugin_dir_len++]= FN_LIBCHAR; > + memcpy(toolpath+plugin_dir_len, tool_name, tool_name_len+1); > + > + arg[0]= toolpath; > + arg[1]= info->user_name; > + arg[2]= info->auth_string; > + if (pam_debug) > + { > + arg[3]= "DEBUG"; > + arg[4]= 0; > + } > + else > + arg[3]= 0; I'd rather passed that via the pipe, like, in the first three lines. so that it wouldn't be seen in `ps` output. May be it's harmless, but better to avoid this argument altogether. Just do, like fprintf(p_to_c[0], "%s\n%s\n%d\n", info->user_name, info->auth_string, pam_debug); and in the tool, fscanf("%s\n%s\n%d\n"); Hmm. Taking into account that the tool is installed SUID, you need to take extra care to avoid buffer overflows, assuming someone can invoke the tool directly. So, a little bit more complex than fscanf("%s\n%s\n%d\n"), sorry > + > + PAM_DEBUG((stderr, "PAM: execute pam sandbox:[%s] [%s] [%s] [%s].\n", > + arg[0], arg[1], arg[2], > + arg[3] ? arg[3] : "No debug")); > + (void) execv(arg[0], (char **) arg); > + PAM_DEBUG((stderr, "PAM: exec() failed.\n")); > + exit(-1); > + } > + > + /* Parent process continues. */ > + > + PAM_DEBUG((stderr, "PAM: parent continues.\n")); > + if (close(p_to_c[0]) < 0 || > + close(c_to_p[1]) < 0) > + goto error_ret; > + > + for (;;) > + { > + unsigned char field; > + > + PAM_DEBUG((stderr, "PAM: listening to the sandbox.\n")); > + if (read(c_to_p[0], &field, 1) < 1) > + { > + PAM_DEBUG((stderr, "PAM: read failed.\n")); > + goto error_ret; > + } > + > + if (field == AP_EOF) > + { > + PAM_DEBUG((stderr, "PAM: auth OK returned.\n")); > + break; > + } > + > + switch (field) > + { > + case AP_AUTHENTICATED_AS: > + PAM_DEBUG((stderr, "PAM: reading authenticated_as string.\n")); > + if (read_string(c_to_p[0], info->authenticated_as, > + sizeof(info->authenticated_as)) < 0) > + goto error_ret; > + break; > + > + case AP_CONV: > + { > + unsigned char buf[10240]; > + int buf_len; > + unsigned char *pkt; > + > + PAM_DEBUG((stderr, "PAM: getting CONV string.\n")); > + if ((buf_len= read_string(c_to_p[0], (char *) buf, sizeof(buf))) < 0) > + goto error_ret; > + > + PAM_DEBUG((stderr, "PAM: sending CONV string.\n")); > + if (vio->write_packet(vio, buf, buf_len)) > + goto error_ret; > + > + PAM_DEBUG((stderr, "PAM: reading CONV answer.\n")); > + if ((buf_len= vio->read_packet(vio, &pkt)) < 0) > + goto error_ret; > + > + PAM_DEBUG((stderr, "PAM: answering CONV.\n")); > + if (write_string(p_to_c[1], pkt, buf_len)) > + goto error_ret; > + } > + break; > + > + default: > + PAM_DEBUG((stderr, "PAM: unknown sandbox field.\n")); > + goto error_ret; > + } > + } > + result= CR_OK; > + > +error_ret: > + close(p_to_c[1]); > + close(c_to_p[0]); > + > + PAM_DEBUG((stderr, "PAM: auth result %d.\n", result)); > + return result; > +} > + > +static struct st_mysql_auth info = > +{ > + MYSQL_AUTHENTICATION_INTERFACE_VERSION, > + "dialog", > + pam_auth > +}; > + > +static char use_cleartext_plugin; > +static MYSQL_SYSVAR_BOOL(use_cleartext_plugin, use_cleartext_plugin, > + PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY, > + "Use mysql_cleartext_plugin on the client side instead of the dialog " > + "plugin. This may be needed for compatibility reasons, but it only " > + "supports simple PAM policies that don't require anything besides " > + "a password", NULL, NULL, 0); > + > +#ifndef DBUG_OFF > +static MYSQL_SYSVAR_BOOL(debug, pam_debug, PLUGIN_VAR_OPCMDARG, > + "Log all PAM activity", NULL, NULL, 0); > +#endif > + > + > +static struct st_mysql_sys_var* vars[] = { > + MYSQL_SYSVAR(use_cleartext_plugin), > +#ifndef DBUG_OFF > + MYSQL_SYSVAR(debug), > +#endif > + NULL > +}; > + > + > +static int init(void *p __attribute__((unused))) > +{ > + if (use_cleartext_plugin) > + info.client_auth_plugin= "mysql_clear_password"; > + if (!(opt_plugin_dir= dlsym(RTLD_DEFAULT, "opt_plugin_dir"))) > + return 1; > + > + return 0; > +} > + > +maria_declare_plugin(pam) > +{ > + MYSQL_AUTHENTICATION_PLUGIN, > + &info, > + "pam", > + "MariaDB Corp", > + "PAM based authentication (safe)", not sure about calling it "safe". I think it's more of a side effect, the main feature it that it works, while old pam plugin simply doesn't :) unless mysqld is run as root. and not totally sure about calling it also pam. it means one won't be able to load it and the old pam plugin at the same time. I suspect it'll still ok and benefits overweight it. > + PLUGIN_LICENSE_GPL, > + init, > + NULL, > + 0x0100, > + NULL, > + vars, > + "1.0", > + MariaDB_PLUGIN_MATURITY_STABLE Let's start from beta and change it to stable later :) > +} > +maria_declare_plugin_end; > diff --git a/plugin/auth_pam/auth_pam_tool.c b/plugin/auth_pam/auth_pam_tool.c > new file mode 100644 > index 0000000..e4addd3 > --- /dev/null > +++ b/plugin/auth_pam/auth_pam_tool.c > @@ -0,0 +1,115 @@ > +/* > + Copyright (c) 2011, 2012, Monty Program Ab could be updated... > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111-1301 USA */ > + > +#define _GNU_SOURCE 1 /* for strndup */ you could just set _GNU_SOURCE once in CMakeLists.txt instead of doing it in every file > + > +#include <stdlib.h> > +#include <unistd.h> > +#include <mysql/plugin_auth_common.h> > + > +struct param { > + unsigned char buf[10240], *ptr; > +}; > + > + > +#include "auth_pam_tool.h" > + > + > +static int write_packet(struct param *param __attribute__((unused)), > + const unsigned char *buf, int buf_len) > +{ > + unsigned char b= AP_CONV; > + return write(1, &b, 1) < 1 || > + write_string(1, buf, buf_len); > +} > + > + > +static int read_packet(struct param *param, unsigned char **pkt) > +{ > + *pkt= (unsigned char *) param->buf; > + return read_string(0, (char *) param->buf, (int) sizeof(param->buf)) - 1; > +} > + > + > +typedef struct st_mysql_server_auth_info > +{ > + /** > + User name as sent by the client and shown in USER(). > + NULL if the client packet with the user name was not received yet. > + */ > + char *user_name; > + > + /** > + A corresponding column value from the mysql.user table for the > + matching account name > + */ > + char *auth_string; > + > + /** > + Matching account name as found in the mysql.user table. > + A plugin can override it with another name that will be > + used by MySQL for authorization, and shown in CURRENT_USER() > + */ > + char authenticated_as[MYSQL_USERNAME_LENGTH+1]; > +} MYSQL_SERVER_AUTH_INFO; > + > + > +#include "auth_pam_base.c" > + > + > +int main(int argc, char **argv) > +{ > + struct param param; > + MYSQL_SERVER_AUTH_INFO info; > + unsigned char field; > + int res; > + > + if (argc < 3) > + return -1; > + > + info.user_name= argv[1]; > + info.auth_string= argv[2]; > + > + if (argc >= 4) > + pam_debug= 1; As we discussed, there should be some ways to prevent it being used to brute-force passwords. It'll be suid root, so it cannot be made executable only by "mysql" user. I see two approaches here: 1. getuid() and check that the user name is "mysql" 2. getppid() and check that it matches the value in a pid_file second is heavier (one might need to read my.cnf to find the location of a pid_file). first hard-codes the user name. oh, a third approach. create a new directory under plugin-dir, say auth_tool with permissions r-x------ and owned by "mysql". Put the actual executable there, so nobody besides "mysql" user will be able to get to it. Yes, this seems the easiest, fastest, and most flexible too :) > + > + PAM_DEBUG((stderr, "PAM: sandbox started [%s] [%s] [%s] [%s].\n", argv[0], > + info.user_name, info.auth_string, > + pam_debug ? argv[3] : "No debug")); Hmm, where will it go? messages written by the tool to its stderr > + > + if ((res= pam_auth_base(&param, &info)) != CR_OK) > + { > + PAM_DEBUG((stderr, "PAM: auth failed, sandbox closed.\n")); > + return -1; > + } > + > + if (info.authenticated_as[0]) > + { > + PAM_DEBUG((stderr, "PAM: send authenticated_as field.\n")); > + field= AP_AUTHENTICATED_AS; > + if (write(1, &field, 1) < 1 || > + write_string(1, (unsigned char *) info.authenticated_as, > + strlen(info.authenticated_as))) > + return -1; > + } > + > + PAM_DEBUG((stderr, "PAM: send OK result.\n")); > + field= AP_EOF; > + write(1, &field, 1); > + > + PAM_DEBUG((stderr, "PAM: sandbox closed.\n")); > + return 0; > +} Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 980a5c31880: MDEV-14572: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status upon EXPLAIN UPDATE in PS
by Sergei Golubchik 19 Jun '18

19 Jun '18

Hi, Oleksandr! On Jan 28, Oleksandr Byelkin wrote: > Hi, Sergei! > > Flag in SELECT_LEX options dropped at the end of execution (in our case > prepare). All commands where EXPLAIN is possible restore the flag in main > SELECT_LEX (in mysql_explain_union()) except UPDATE (because it do > not use mysql_explain_union() > for explain). Without the flag it first try to send OK (as part of update) > then EOF as part of EXPLAIN. Ok. 1. Why does it not use mysql_explain_union() ? 2. Why EXPLAIN flag is need to be removed and restored anyway? 3. Please append the following to the commit comment: " Without it, mysql_update() didn't know that there will be EXPLAIN result set and was sending OK at the end of the update, which conflicted with the EOF sent later by EXPLAIN. " > On Sun, Jan 28, 2018 at 2:45 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > > > Hi, Oleksandr! > > > > On Dec 22, Oleksandr Byelkin wrote: > > > revision-id: 980a5c318803d6aa7355e019869597976dcf19bf (mariadb-10.0.33-37-g980a5c31880) > > > parent(s): 042f763268c0f209e7c12e0a6a72bb5d204dfe29 > > > author: Oleksandr Byelkin > > > committer: Oleksandr Byelkin > > > timestamp: 2017-12-22 14:00:10 +0100 > > > message: > > > > > > MDEV-14572: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status upon EXPLAIN UPDATE in PS > > > > > > Restore EXPAIN flag in SELECT_LEX before execution multi-update by flag in LEX > > > (the same but in other way made before INSERT/DELETE/SELECT) > > > > Please explain what was the reason for the bug, why the status was set > > twice, and how your change fixes it. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] 398ea6a: MDEV-11917 enum/set command-line options aren't respecting max-* settings.
by Sergei Golubchik 18 Jun '18

18 Jun '18

Hi, Alexey! On Mar 13, Alexey Botchkov wrote: > revision-id: 398ea6ab012938d08126c8d0e0291000d06f6cd6 (mariadb-10.2.13-43-g398ea6a) > parent(s): 9d95b8665a3d87ab857e77e220effc64454eb881 > committer: Alexey Botchkov > timestamp: 2018-03-13 17:13:27 +0400 > message: > > MDEV-11917 enum/set command-line options aren't respecting max-* settings. > > Code added to make maximum- working for Sys_var_enum > and Sys_var_set. > > diff --git a/sql/set_var.cc b/sql/set_var.cc > index e96e636..32e52a9 100644 > --- a/sql/set_var.cc > +++ b/sql/set_var.cc > @@ -452,6 +452,22 @@ void sys_var::do_deprecated_warning(THD *thd) > > @retval true on error, false otherwise (warning or ok) > */ > + > + > +bool throw_bounds_warning(THD *thd, const char *name,const char *v) > +{ > + if (thd->variables.sql_mode & MODE_STRICT_ALL_TABLES) > + { > + my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), name, v); This is rather illogical, setting a variable doesn't touch any tables, so MODE_STRICT_ALL_TABLES should not affect it. Also, the error message could be improved. But it's how it was, and it's not subject of this bugfix, I agree. > + return true; > + } > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_TRUNCATED_WRONG_VALUE, > + ER_THD(thd, ER_TRUNCATED_WRONG_VALUE), name, v); > + return false; > +} > + > + > bool throw_bounds_warning(THD *thd, const char *name, > bool fixed, bool is_unsigned, longlong v) > { > @@ -464,14 +480,7 @@ bool throw_bounds_warning(THD *thd, const char *name, > else > llstr(v, buf); > > - if (thd->variables.sql_mode & MODE_STRICT_ALL_TABLES) > - { > - my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), name, buf); > - return true; > - } > - push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > - ER_TRUNCATED_WRONG_VALUE, > - ER_THD(thd, ER_TRUNCATED_WRONG_VALUE), name, buf); ah, good. only one place to change when we fix the above. thanks. > + return throw_bounds_warning(thd, name, buf); > } > return false; > } > diff --git a/sql/sys_vars.ic b/sql/sys_vars.ic > index bc913b1..adcda5d 100644 > --- a/sql/sys_vars.ic > +++ b/sql/sys_vars.ic all ok, looks good. Just one comment. Please make max_var_ptr() logic to be in one place only. If C++ cannot do it, then, in the worst case, make it a macro, like #define define_max_var_ptr(TYPE) \ TYPE *max_var_ptr() { return scope() == SESSION ? (TYPE*)(((uchar*)&max_system_variables) + offset) : 0; } after that ok to push, thanks! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] f3dda8e: MDEV-15357 Wrong result with join_cache_level=4, BNLH join
by Sergei Golubchik 15 Jun '18

15 Jun '18

Hi, Igor! On Apr 30, Igor Babaev wrote: > revision-id: f3dda8e2002dc2ad630c07bda0d8c00be7269907 (mariadb-10.2.14-72-gf3dda8e) > parent(s): b4c5e4a717e3ce2c2d434106cc74417fe9a1d3dc > author: Igor Babaev > committer: Igor Babaev > timestamp: 2018-04-30 21:26:38 -0700 > message: > > MDEV-15357 Wrong result with join_cache_level=4, BNLH join > > This bug was introduced by the architectural changes of the patch > for MDEV-11640. The patch moved initialization of join caches after > the call of make_join_readinfo(). As any failure to initialize > a join cache caused denial of its usage the execution code for > the query had to be revised. This revision required rolling back many > actions taken by make_join_readinfo(). It was not done in the patch. > As a result if a denial of join cache happened some join conditions > could be lost. This was exactly the cause of wrong results in the > bug's reported test case. > > Thus the introduced architectural change is not valid and it would be > better to roll it back. At the same time two new methods > adjust_read_set_for_vcol_keyread() and get_covering_index_for_scan() > were added to the class JOIN_TAB to resolve the problems of MDEV-11640. > > diff --git a/sql/sql_join_cache.cc b/sql/sql_join_cache.cc > index 3612cb6..242cb91 100644 > --- a/sql/sql_join_cache.cc > +++ b/sql/sql_join_cache.cc > @@ -226,6 +226,16 @@ void JOIN_CACHE::calc_record_fields() > flag_fields+= MY_TEST(tab->table->maybe_null); > fields+= tab->used_fields; > blobs+= tab->used_blobs; > + if (tab->type == JT_ALL || tab->type == JT_HASH) > + { > + uint idx= tab->get_covering_index_for_scan(); as you moved JOIN_CACHE initialization back, I suspect that this calc_record_fields() now happens too early for get_covering_index_for_scan() to be useful. And why only JT_ALL and JT_HASH? > + if (idx != MAX_KEY) > + { > + fields-= bitmap_bits_set(tab->table->read_set); > + tab->adjust_read_set_for_vcol_keyread(idx); > + fields+= bitmap_bits_set(tab->table->read_set); > + } > + } > } > if ((with_match_flag= join_tab->use_match_flag())) > flag_fields++; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 30ef26f..e5b75ec 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -11404,6 +11375,61 @@ void JOIN_TAB::remove_redundant_bnl_scan_conds() > } > > > +uint JOIN_TAB::get_covering_index_for_scan() > +{ > + int idx= MAX_KEY; > + if (table->no_keyread) > + return idx; > + if (select && select->quick && > + select->quick->index != MAX_KEY && //not index_merge > + table->covering_keys.is_set(select->quick->index)) > + idx= select->quick->index; > + else if (!table->covering_keys.is_clear_all() && > + !(select && select->quick)) > + { // Only read index tree > + if (loosescan_match_tab) > + idx= loosescan_key; > + else > + { > +#ifdef BAD_OPTIMIZATION > + /* > + It has turned out that the below change, while speeding things > + up for disk-bound loads, slows them down for cases when the data > + is in disk cache (see BUG#35850): > + See bug #26447: "Using the clustered index for a table scan > + is always faster than using a secondary index". > + */ > + if (table->s->primary_key != MAX_KEY && > + table->file->primary_key_is_clustered()) > + idx= table->s->primary_key; > + else > +#endif > + idx= find_shortest_key(table, & table->covering_keys); > + } > + } > + return idx; > +} > + > + > +void JOIN_TAB::adjust_read_set_for_vcol_keyread(uint keyread_idx) > +{ > + if (!table->vcol_set || bitmap_is_clear_all(table->vcol_set)) > + return; > + > + MY_BITMAP *keyread_set= &table->cond_set; // is free for use at call normally one uses tmp_set for this, not hijacks cond_set > + bitmap_clear_all(keyread_set); > + table->mark_columns_used_by_index(keyread_idx, keyread_set); > + bitmap_intersect(keyread_set, table->read_set); > + bitmap_intersect(keyread_set, table->vcol_set); this looks wrong, you probably should do keyread_set INTERSECT (read_set UNION vcol_set) > + if (!bitmap_is_clear_all(keyread_set)) > + { > + bitmap_clear_all(keyread_set); > + table->mark_columns_used_by_index(keyread_idx, keyread_set); > + bitmap_intersect(table->read_set, keyread_set); > + } I don't understand that. 1. What if bitmap_is_clear_all(keyread_set) - what does it mean? 2. Why do you recalculate keyread_set, why not simply copy keyread_set into read_set ? > +} > + > + > /* > Plan refinement stage: do various setup things for the executor > > @@ -11526,7 +11552,10 @@ make_join_readinfo(JOIN *join, ulonglong options, uint no_jbuf_after) > tab->read_first_record= tab->type == JT_SYSTEM ? join_read_system > : join_read_const; > if (table->covering_keys.is_set(tab->ref.key) && !table->no_keyread) > + { > + tab->adjust_read_set_for_vcol_keyread(tab->ref.key); > table->file->ha_start_keyread(tab->ref.key); better combine both into inline void JOIN_TAB::start_keyread() { adjust_read_set_for_vcol_keyread(ref.key); tab->table->file->ha_start_keyread(ref.key); } or even JOIN_TAB::start_keyread_if_needed() putting the if() inside too > + } > else if ((!jcl || jcl > 4) && !tab->ref.is_access_triggered()) > push_index_cond(tab, tab->ref.key); > break; > @@ -11606,35 +11641,19 @@ make_join_readinfo(JOIN *join, ulonglong options, uint no_jbuf_after) > } > } > } > - if (!table->no_keyread) > + uint idx= tab->get_covering_index_for_scan(); > + if (idx != MAX_KEY) > { > if (tab->select && tab->select->quick && > tab->select->quick->index != MAX_KEY && //not index_merge > table->covering_keys.is_set(tab->select->quick->index)) > - table->file->ha_start_keyread(tab->select->quick->index); > + table->file->ha_start_keyread(idx); why do you need an if() here - you already checked that in get_covering_index_for_scan(). I generally don't like this code at all. In JOIN_CACHE you simply use tab->get_covering_index_for_scan(), but here you use it *plus* a bunch of if()s afterwards. It surely creates an impression that JOIN_CACHE and make_join_readinfo may have a different idea about whether keyread is used. Ideally here should be just if (idx != MAX_KEY) tab->start_keyread(idx); and even then it would be kinda questionable (because preconditions might change between two get_covering_index_for_scan() calls). > else if (!table->covering_keys.is_clear_all() && > !(tab->select && tab->select->quick)) > - { // Only read index tree > - if (tab->loosescan_match_tab) > - tab->index= tab->loosescan_key; > - else > - { > -#ifdef BAD_OPTIMIZATION > - /* > - It has turned out that the below change, while speeding things > - up for disk-bound loads, slows them down for cases when the data > - is in disk cache (see BUG#35850): > - See bug #26447: "Using the clustered index for a table scan > - is always faster than using a secondary index". > - */ > - if (table->s->primary_key != MAX_KEY && > - table->file->primary_key_is_clustered()) > - tab->index= table->s->primary_key; > - else > -#endif > - tab->index=find_shortest_key(table, & table->covering_keys); > - } > + { // Only read index tree > + tab->index= idx; > tab->read_first_record= join_read_first; > + table->file->ha_start_keyread(tab->index); > /* Read with index_first / index_next */ > tab->type= tab->type == JT_ALL ? JT_NEXT : JT_HASH_NEXT; > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-6111: Optimizer trace: let's get rid of "optimizer_trace_enabled"
by Sergey Petrunia 15 Jun '18

15 Jun '18

Hi Zhzhzoo, I see that the current patch uses @@optimizer_trace_enabled variable to control it. Meanwhile, in MySQL the name of the variable is "optimizer_trace". It is a flag set: mysql> select @@optimizer_trace; +-------------------------+ | @@optimizer_trace | +-------------------------+ | enabled=on,one_line=off | +-------------------------+ 1 row in set (0.00 sec) but one can set it to 1 or 0 to just set enabled on/off: mysql> set @@optimizer_trace=1; Query OK, 0 rows affected (0.00 sec) mysql> set @@optimizer_trace=0; Query OK, 0 rows affected (0.00 sec) This obviously carries some legacy but I think we should have it the same way, in order to be compatible with MySQL where possible. Please use the same name/values as MySQL does. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-6111: Optimizer trace: on the first output
by Sergey Petrunia 15 Jun '18

15 Jun '18

On Tue, Jun 12, 2018 at 07:40:08AM +0800, zhzhzoo zhang wrote: > I've done > condition processing and some range analysis. > and for the test input ( > https://dev.mysql.com/doc/internals/en/tracing-example.html) It now > outputs something like > (not very readable from the email so I've made two pastes, from the current tree and from MySQL 8): https://gist.github.com/spetrunia/cb4a1a382a9e81a60491d524dbfc08f6 https://gist.github.com/spetrunia/62347cc3bf26e003909a32b378b6684f > ```{ > "join_optimization": { > "select #": 1, > "steps": { > "inner": { > "select #": 1, Two 'select #' :1 seem to be redundant. There are also two nested "steps" elements which seem redundant as well. > "steps": [ > { > "condition_processing": { > "condition": "WHERE", > "original_condition": "`test`.`alias1`.`pk` <> 0 and > `test`.`alias2`.`pk` = `test`.`alias1`.`col_int_key`", > "steps": [ > { > "transformation": "equality_propagation", > "subselect_evaluation": [], > "resulting_condition": "`test`.`alias1`.`pk` <> 0 and > multiple equal(`test`.`alias2`.`pk`, `test`.`alias1`.`col_int_key`)???" What are the "???" at the end? Please get them removed. also, if you check EXPLAIN FORMAT=JSON output, it prints conditions without overquoting, like so: "attached_condition": "alias1.pk <> 0 and alias1.col_int_key is not null" This is done by using QT_EXPLAIN parameter: item->print(&str, QT_EXPLAIN); Let's print them this way here as well. > }, > { > "transformation": "constant_propagation", > "subselect_evaluation": [], > "resulting_condition": "`test`.`alias1`.`pk` <> 0 and > multiple equal(`test`.`alias2`.`pk`, `test`.`alias1`.`col_int_key`)???" > }, BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 46e0c8a9219: MDEV-11071: Assertion `thd->transaction.stmt.is_empty()' failed in Locked_tables_list::unlock_locked_table
by Sergei Golubchik 14 Jun '18

14 Jun '18

Hi, Oleksandr! Looks good, but please see a few comments below. On May 08, Oleksandr Byelkin wrote: > revision-id: 46e0c8a921978e7b95adf3f17f3c85b18d9f9ef6 (mariadb-10.2.14-78-g46e0c8a9219) > parent(s): 9bcd0f5fea8ca26742b10d37b95a966c69909ff1 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-05-08 15:26:26 +0200 > message: > > MDEV-11071: Assertion `thd->transaction.stmt.is_empty()' failed in Locked_tables_list::unlock_locked_table > > fix_length_and_dec now return result (error/OK) > diff --git a/sql/item.h b/sql/item.h > index 8921ee76f6a..e9713730a1c 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -4210,7 +4210,7 @@ class Item_func_or_sum: public Item_result_field, > also to make printing of items inherited from Item_sum uniform. > */ > virtual const char *func_name() const= 0; > - virtual void fix_length_and_dec()= 0; > + virtual bool fix_length_and_dec()= 0; 1. I wonder, if you change fix_length_and_dec to void in one of the derived classes, will you get a compiler warning for that? 2. may be also __attribute__ ((warn_unused_result)) here? I don't know how it'll work on virtual methods, may be it won't. both questions are for the use case "what if we merge some fix_length_and_dec related changes from 10.1, will the compiler tell us to update them?" > bool const_item() const { return const_item_cache; } > table_map used_tables() const { return used_tables_cache; } > Item* build_clone(THD *thd, MEM_ROOT *mem_root); > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index 44cc4f3cae9..46cf412e229 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -535,8 +535,9 @@ void Item_bool_rowready_func2::fix_length_and_dec() > we have to check for out of memory conditions here > */ > if (!args[0] || !args[1]) > - return; > - setup_args_and_comparator(current_thd, &cmp); > + return FALSE; > + bool res= setup_args_and_comparator(current_thd, &cmp); > + return res; not that it matters much (no need to change), but why didn't you return setup_args_and_comparator(current_thd, &cmp); directly? > } > > > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index de1b27cff1a..5efb98bc81d 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -911,10 +911,12 @@ class Item_func_strcmp :public Item_int_func > longlong val_int(); > uint decimal_precision() const { return 1; } > const char *func_name() const { return "strcmp"; } > - void fix_length_and_dec() > + bool fix_length_and_dec() > { > - agg_arg_charsets_for_comparison(cmp_collation, args, 2); > + if (agg_arg_charsets_for_comparison(cmp_collation, args, 2)) > + return TRUE; > fix_char_length(2); // returns "1" or "0" or "-1" this comment is kinda weird. remove? > + return FALSE; > } > Item *get_copy(THD *thd, MEM_ROOT *mem_root) > { return get_item_copy<Item_func_strcmp>(thd, mem_root, this); } > @@ -1987,10 +1997,10 @@ class Item_func_like :public Item_bool_func2 > const char *func_name() const { return "like"; } > enum precedence precedence() const { return CMP_PRECEDENCE; } > bool fix_fields(THD *thd, Item **ref); > - void fix_length_and_dec() > + bool fix_length_and_dec() > { > max_length= 1; > - agg_arg_charsets_for_comparison(cmp_collation, args, 2); should this one also be __attribute__ ((warn_unused_result)) ? > + return agg_arg_charsets_for_comparison(cmp_collation, args, 2); > } > void cleanup(); > > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index 4214980c36c..ca49e06a9e7 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -904,9 +905,10 @@ Item::Type Item_subselect::type() const > } > > > -void Item_subselect::fix_length_and_dec() > +bool Item_subselect::fix_length_and_dec() > { > engine->fix_length_and_dec(0); what about subselect_engine::fix_length_and_dec() ? should return bool too, I guess, right? > + return FALSE; > } > > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 9e7973b745c..f3cb85f01d3 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -4894,7 +4894,13 @@ int create_table_impl(THD *thd, > file= mysql_create_frm_image(thd, orig_db, orig_table_name, create_info, > alter_info, create_table_mode, key_info, > key_count, frm); > - if (!file) > + /* > + We have to check thd->is_error() here because it can be set by > + Item::val* for example, and before it will be cought accidentally by > + Item_func::fix_fields() of the next call. Now we removed the check > + from Item_func::fix_fields() > + */ this is a very confusing comment, don't write in comments that "it used to be this way, but now we've changed it and it is that way" and I still don't understand why do you need to check for thd->is_error() here > + if (!file || thd->is_error()) > goto err; > if (rea_create_table(thd, frm, path, db, table_name, create_info, > file, frm_only)) > @@ -7377,7 +7383,8 @@ static bool mysql_inplace_alter_table(THD *thd, > */ > if (mysql_rename_table(db_type, alter_ctx->new_db, alter_ctx->tmp_name, > alter_ctx->db, alter_ctx->alias, > - FN_FROM_IS_TMP | NO_HA_TABLE)) > + FN_FROM_IS_TMP | NO_HA_TABLE) || > + thd->is_error()) and here > { > // Since changes were done in-place, we can't revert them. > (void) quick_rm_table(thd, db_type, Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 6

Re: [Maria-developers] 0918ae42a83: MDEV-14014 Multi-Slave Replication Fail: bogus data in log event
by Sergei Golubchik 13 Jun '18

13 Jun '18

Hi, Andrei! On Jun 13, andrei.elkin(a)pp.inet.fi wrote: > revision-id: 0918ae42a8394b6faf28f67b70550894851be3bb (mariadb-10.1.33-36-g0918ae42a83) > parent(s): 26be5072429082e634b8fc102609370975443439 > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-06-13 16:27:00 +0300 > message: > > MDEV-14014 Multi-Slave Replication Fail: bogus data in log event > > MDEV-7257 made a dump thread to read from binlog concurrently with > writers as long as the read bytes are below a water-mark > (MYSQL_BIN_LOG::binlog_end_pos). However it appeared to be possible a > dump thread reader reach out for bytes past the water mark through a > feature of IO_CACHE that fills in the internal buffer and while doing > so it could read what the reader is not supposed to see (the bytes > above MYSQL_BIN_LOG::binlog_end_pos). > > The issue is fixed with constraining the IO_CACHE buffer fill to respect > the watermark. > An added test simulates potentially unconstrained buffer fill and an > assert guards this is not the case anymore. Looks good, thanks! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] unbundle RocksDB
by Michal Schorm 13 Jun '18

13 Jun '18

Hello, Since we have a rocksdb package ready in Fedora, I tried to unbundle it, using the system package instead. I wasn't successful though. I haven't found any CMake argument which seemed related and from the build log it seems, the code tries to link against the static libraries in any case. Before I'll start to dig further, I'm stopping by on this list, making sure it is possible. - is it currently supported? - have I overlooked any related CMake argument? Links: MariaDB in Fedora: https://src.fedoraproject.org/rpms/mariadb/tree/master -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

3 3

Re: [Maria-developers] d9913834ceb: MDEV-14014 Multi-Slave Replication Fail: bogus data in log event
by Sergei Golubchik 13 Jun '18

13 Jun '18

Hi, andrei.elkin! On May 30, andrei.elkin(a)pp.inet.fi wrote: > revision-id: d9913834cebe4ed8494b3d76762cd882788a8fd5 (mariadb-10.1.33-28-gd9913834ceb) > parent(s): c1698e8dc50f0c342c1a6886426ba1d43396cd1e > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-05-30 20:06:44 +0300 > message: > > MDEV-14014 Multi-Slave Replication Fail: bogus data in log event > > MDEV-7257 made a dump thread to read from binlog concurrently with > writers as long as the read bytes are below a water-mark > (MYSQL_BIN_LOG::binlog_end_pos). However it appeared to be possible a > dump thread reader reach out for bytes past the water mark through a > feature of IO_CACHE that fills in the internal buffer and while doing > so it could read what the reader is not supposed to see (the bytes > above MYSQL_BIN_LOG::binlog_end_pos). > > The issue is fixed with constraining the IO_CACHE buffer fill to respect > the watermark. > An added test simulates potentially unconstrained buffer fill and an > assert guards this is not the case anymore. What about a much simpler one-liner fix: --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -2549,6 +2549,7 @@ static int send_events(binlog_send_info *info, IO_CACHE* return 1; info->last_pos= linfo->pos; + log->end_of_file= end_pos; error= Log_event::read_log_event(log, packet, info->fdev, opt_master_verify_checksum ? info->current_checksum_alg : BINLOG_CHECKSUM_ALG_OFF); It doesn't change IO_CACHE structure and doesn't add any overhead to the IO_CACHE. Did you already test that your commit actually fixes the issue? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] 20f3479: MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>.
by Sergei Golubchik 12 Jun '18

12 Jun '18

Hi, Alexey! On Jun 11, Alexey Botchkov wrote: > revision-id: 20f347974f6a9b4e2c65a5ca79aefb8e118ba6cd (mariadb-10.1.33-33-g20f3479) > parent(s): 3627dd7f6a1cf5bd7151ff23290d64cb6dffea90 > committer: Alexey Botchkov > timestamp: 2018-06-11 14:58:46 +0400 > message: > > MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>. > > The reload_acl_and_cache() now looks into the VIEW definition > to check the involved tables if the locked_tables_mode. That's a pretty neat solution :) But it seems that - contrary to what I expected - that MariaDB actually distinguishes between locking a view a locking all its tables. Consider this test case: ---- create table t1 (a int); create view v1 as select * from t1; lock table v1 write; insert v1 values (1); select * from t1; unlock tables; drop view v1; drop table t1; ---- this works and inserts 1 just fine. If you change the lock table statement to lock t1, it'll with Table 'v1' was not locked with LOCK TABLES It'd be great if FLUSH TABLES will exhibit the same behavior, it needs to know if the view is locked, not only if all underlying tables are locked. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Maria-discuss] MariaDB crashes because of "long semaphore wait" after migrating from 10.1 to 10.3
by Vicențiu Ciorbaru 11 Jun '18

11 Jun '18

Hi! I have cc'ed the maria-developers mailing list as this is potentially a dev issue and also the devs responsible for Windows and InnoDB development. There might be some people on vacation right now, but this should get looked into in a week or two at most. Let's find a solution for your problem. :) Vicentiu On Sun, 10 Jun 2018, 19:11 TD, <tdcf(a)gmx.net> wrote: > Hi, > > I have a Windows server that has been running MariaDB 10.1 successfully > for over a year. The server remains mostly idle for long times with some > read access, but occasionally there are transactions that add data (about > 500k rows per commit). There can be up to 10 such transactions (one per > database) at the same time and during those times the server is under quite > some load (the code processing the data resides on the same server as the > database). We're using an 8 core AMD Ryzen, 32 GB RAM and a Samsung SSD. > > Like I said, this worked fine with MariaDB 10.1 all the time. > > We're currently in the process of setting up a new server (specs similar > to the ones above) and because MariaDB 10.3 has been released recently, I > was curious and installed it. The database now constantly shuts down > because of "[FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We > intentionally crash the server because it appears to be hung." whenever > there is one of the load situations described above. If I install MariaDB > 10.1 on the new server, it runs as fine as the old one. The crashes also > happen when I start with a clean database (no data migrated from 10.1 > databases). > > I'm not a huge MariaDB config pro, as it's only a tool to store the data, > it just needs to run with reasonable performance and has always worked > until now. As far as I know and can judge from comments in the file, the > my.ini in use is (mostly?) based on my-innodb-heavy-4G.cnf, but coworkers > might have made modifications. I've attached the contents below. > > I don't know how I can debug this issue further. Windows event viewer > contains several log warnings with the text "InnoDB: A long semaphore > wait:" but no further information why this wait happens. I couldn't find > any other error logs. I guess it is some configuration error (I read about > MariaDB switching from XtraDB to InnoDB, so I guess that the issue might be > related). > > Migrating to 10.3 is not an absolute must, but it would be nice to be able > to fix the problem. > > Any help would be greatly appreciated and if you need more information, > I'll gladly try to provide it. > > Thanks, > Tom. > > my.ini: > [mysqld] > max_connections = 100 > max_allowed_packet = 16M > binlog_cache_size = 1M > max_heap_table_size = 64M > read_buffer_size = 2M > read_rnd_buffer_size = 16M > sort_buffer_size = 8M > join_buffer_size = 8M > thread_cache_size = 10 > query_cache_size = 64M > query_cache_limit = 2M > thread_stack = 240K > tmp_table_size = 64M > slow_query_log > expire_logs_days = 14 > log-bin = mysql-bin > character-set-server=utf8 > collation-server=utf8_general_ci > > #*** MyISAM Specific options > key_buffer_size = 32M > bulk_insert_buffer_size = 64M > myisam_sort_buffer_size = 128M > myisam_max_sort_file_size = 10G > myisam_repair_threads = 1 > myisam_recover > > # *** INNODB Specific options *** > #innodb_fatal_semaphore_wait_threshold = 6000 #tried that, didn't help > #innodb_adaptive_hash_index = 0 #tried that (suggested in some stack > overflow post, didn't help) > innodb_buffer_pool_size = 2G > innodb_data_file_path = ibdata1:10M:autoextend > innodb_write_io_threads = 8 > innodb_read_io_threads = 8 > innodb_thread_concurrency = 16 > innodb_log_buffer_size = 8M > innodb_log_file_size = 256M > innodb_log_files_in_group = 3 > innodb_max_dirty_pages_pct = 10 > innodb_io_capacity = 2000 > innodb_lock_wait_timeout = 120 > > [mysql] > no-auto-rehash > > [myisamchk] > key_buffer_size = 512M > sort_buffer_size = 512M > read_buffer = 8M > write_buffer = 8M > > [mysqld_safe] > open-files-limit = 8192 > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp >

3 2

[Maria-developers] MariaDB Developers Unconference in a few weeks!
by Otto Kekäläinen 08 Jun '18

08 Jun '18

Hello everybody! The next developers unconference will be in a few weeks: https://mariadb.org/2018-2-developers-unconference-in-finland/ There are still open slots so we encourage you to sign up now. https://www.meetup.com/MariaDB-Developers/events/249024232/ We have plenty of break-out rooms this time and there is space, so we also encourage you to suggest sessions/discussions you would like to have to ian(a)mariadb.org. Around mid-summer is a great time to visit Finland. The nights that don't get dark are worth experiencing! Getting to the venue in Tampere., Finlnd is also very easy. https://mariadb.org/travel-tips-to-tampere-finland/#why-visit Hope to see you there! - Otto

1 0

Re: [Maria-developers] [Commits] 56d339111a4: MDEV-16374: Filtered shows 0 for materilization scan for a semi join, which makes optimizer always picks
by Sergey Petrunia 05 Jun '18

05 Jun '18

Hi Varun, On Mon, Jun 04, 2018 at 12:34:49PM +0530, Varun wrote: > revision-id: 56d339111a4e5c868ffa3b1f2538c5f101f67767 (mariadb-10.0.30-366-g56d339111a4) > parent(s): a61724a3ca187f6eb44fc67948acb76a94efa783 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-06-04 12:26:56 +0530 > message: > > MDEV-16374: Filtered shows 0 for materilization scan for a semi join, which makes optimizer always picks > materialization scan over materialization lookup > > For non-mergeable semi-joins we don't store the estimates of the IN subquery in table->file->stats.records. > In the function TABLE_LIST::fetch_number_of_rows, we store the number of rows in the tables > (estimates in case of derived table/views). > Currently we don't store the estimates for non-mergeable semi-joins, which leads to a problem of selecting > materialization scan over materialization lookup. > Fixed this by storing these estimated appropriately > * This is a change in the optimizer. Should we really do it in 10.0? In my opinion, this should go into 10.3 or 10.4. Let's discuss this, and I think Igor may have something to say, too. * The patch updates selectivity.test but not selectivity_innodb.result (which includes selectivity.test). * Please see my last comments at https://jira.mariadb.org/browse/MDEV-16374 . Here is an alternative patch (it passes the testsuite, and produces the same result for selectivity* tes): diff --git a/sql/sql_select.cc b/sql/sql_select.cc index 1285835..884997c 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -6602,7 +6602,7 @@ double matching_candidates_in_table(JOIN_TAB *s, bool with_found_constraint, { TABLE *table= s->table; double sel= table->cond_selectivity; - double table_records= (double)table->stat_records(); + double table_records= s->records; // psergey-10 dbl_records= table_records * sel; return dbl_records; } What do you think about it? I would like to get Igor's opinion also because I'm not certain which member variable should be changed here. (Why does derived table modify table->stats.records while JTBM semi-joins modify JOIN_TAB::records? These seem like two solutions for the same problem?) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Proposal for Negative Grants Project
by Rutuja Surve 04 Jun '18

04 Jun '18

Hello, Please find the proposal for the Negative Grants project attached along with this e-mail. It would be great to hear your opinions and suggestions on the approach proposed here, before delving into its implementation. Some context about the project: *The current privilege system allows access in this manner:GRANT SELECT ON some_database.* TO a_user@%;Any revoke would revoke access from the entire database. We want to support a revoke that would disable select on a specific table, like:REVOKE SELECT ON some_database.secret_table FROM a_user@%;Reference:[1] https://jira.mariadb.org/browse/MDEV-14443 <https://jira.mariadb.org/browse/MDEV-14443>[2] https://mariadb.com/kb/en/library/google-summer-of-code-2018/ <https://mariadb.com/kb/en/library/google-summer-of-code-2018/>Thanks,Rutuja*

3 6

[Maria-developers] Conservative parallel slave is "too optimistic" for certain engines
by Sergey Petrunia 01 Jun '18

01 Jun '18

Hello, This comes from MDEV-16242. == Symptoms == When one runs a parallel slave (mode=conservative) and replicates DML for MyRocks table without a Primary Key, replication may stop with a ER_KEY_NOT_FOUND error. This may happen even if the queries were run on the master sequentially. == A detail about conservative replication == Suppose the master runs these two statements, sequentially: INSERT INTO t1 VALUES (5),(6),(7); DELETE FROM t1; Parallel slave may schedule the INSERT to Thread1, and the DELETE to Thread2. In conservative parallel replication, the DELETE "will wait" to be executed after the INSERT. One may expect that "will wait" here means that DELETE execution does not start until the INSERT has committed. But it's more subtle than that: actually, DELETE execution will start as soon as the INSERT is ready to do the prepare/commit steps. I assume this was done to increase the parallelism: initial phases of DELETE can be ran in parallel with commit/prepare steps of the INSERT. This is safe to do: 1. The INSERT has acquired locks for all rows it is about to touch. DELETE will not be able to prevent INSERT from committing. 2. The DELETE starts its execution on a database that doesn't include the results of the INSERT (the INSERT has not committed yet!). But this is fine, because the DELETE locks the rows it is about to modify. If it attempts to access a row that the INSERT is about to insert, it will block on a lock until the INSERT finishes. == Applying this to MyRocks == The critical part is #2. It works if the DELETE command will acquire row locks for rows it about to DELETE. This is normally true: If the storage engine supports gap locks, attempting to read a locked gap will cause DELETE to wait for INSERT If the storage engine doesn't support gap locks, it will use Row-Based-Replication. RBR's Delete_rows_event includes Primary Key value of the row that should be deleted. Doing a point lookup on PK will cause the DELETE to block on the row lock. The remaining case is - storage engine that doesn't support gap locks - Row-based-replication of table without primary key In this case, finding the row to delete is done by scanning the table until we find the row where all columns match. The scan does not see the rows that the INSERT is about to INSERT, and DELETE stops with an error, "we could not find the row to delete". == Possible solutions == 1. Implement gap locking in MyRocks. This will take some time. 2. Change parallel slave to wait *for commit*. This should only be done if tables that are updated do not support Gap Locking. This is hard, it will require making risky changes on the SQL layer. 3. Disallow parallel slave execution for tables without Gap Locking and without primary key. (Looks most promising but I am not sure if it is sufficient). BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 2

[Maria-developers] Test failure overview for Debian/Ubuntu on 10.3 master branch (June 1st 2018 status report)
by Otto Kekäläinen 01 Jun '18

01 Jun '18

Hello! I reported here about the test failure status in early April and here is a follow-up. My old emails can be found here: https://lists.launchpad.net/maria-developers/msg11171.html https://lists.launchpad.net/maria-developers/msg11170.html If one looks at the picture from April and compares to the status today, the situation is a bit sad as we have even less green than before: https://lists.launchpad.net/maria-developers/pngcpxTB4wjTW.png Current issues we have in addition to the usual mtr test failures are: - On multiple bionic builds the upgrade fails to run with no apparent reason. The builder script does not seem to find any previous installation of anything, so that probably is missing from the images: + dpkg -l + grep -iE 'maria|mysql|galera' + true Thus the build fails. Example log: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-bionic-amd64/bui lds/196/steps/upgrade2/logs/stdio - On bionic and xenial aarch64 mysqld fails to start. In syslog we can see May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 0 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGN ORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION',' NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_O PTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANS I','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES' ,'STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES',' ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_N OT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH','EMPT Y_STRING_IS_NULL','SIMULTANEOUS_ASSIGNMENT'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGN ORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION',' NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_O PTIONS','NO_TABLE_OPTIONS','NO May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: _FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALU May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 0 [ERROR] mysqld: Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler. May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 6 [ERROR] InnoDB: Column last_update in table `mysql`.`innodb_table_stats` is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch). May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 6 [ERROR] InnoDB: Fetch of persistent statistics requested for table `mysql`.`gtid_slave_pos` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. http://buildbot.askmonty.org/buildbot/builders/kvm-deb-xenial-aarch64/b uilds/277/steps/upgrade1/logs/syslog - stretch amd64 fails to compile due to librocksdblib.a(compaction_job.cc.o): In function `rocksdb::ZSTD_TrainDictionary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long)': ./builddir/storage/rocksdb/./storage/rocksdb/rocksdb/util/compression.h :812: undefined reference to `ZDICT_trainFromBuffer' ./builddir/storage/rocksdb/./storage/rocksdb/rocksdb/util/compression.h :813: undefined reference to `ZDICT_isError' collect2: error: ld returned 1 exit status http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/bu ilds/3986/steps/compile/logs/stdio - jessie amd64 builds to compile due to /usr/bin/ld.bfd.real: /tmp/ccOIwjFo.ltrans0.ltrans.o: relocation R_X86_64_PC32 against symbol `toku_product_name_strings' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld.bfd.real: final link failed: Bad value collect2: error: ld returned 1 exit status http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/bui lds/4298/steps/compile/logs/stdio This -fPIC issue might be related to a change in hardening flags I "forward-ported" from Debian and I will look into it. All the other issues are things I cannot do anything about.. :( On the bright side I can report that there are at least no build regressions on Launchpad with current 10.3: https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.3/+build s?build_text=&build_state=all - Otto

1 0

[Maria-developers] With_sum_func_cache
by Alexander Barkov 31 May '18

31 May '18

Hello Sanja, I recently worked on MDEV-16309 and had hard time to find which Item classes in the hierarchy: - have always with_sum_func==true - have always with_sum_func==false - have variable with_sum_func To make it sure, before actually working on MDEV-16309, I had to create a separate working tree and did with Item::with_sum_func the same change that we previously did for Item::with_subselect in: MDEV-14517 Cleanup for Item::with_subselect and Item::has_subquery() (which you reviewed) - I find the code easier to read this way (instead of searching for all possible assignments, one can search who overrides the method) - Also, some Item can become smaller in size. It's pity to throw the patch away. So perhaps we could just apply it. Can you please have a look? Thanks. I the meanwhile I'll create a new MDEV for it (with a similar description to MDEV-14517)

2 2