- developers - lists.mariadb.org

[Maria-developers] Buildbot image update request
by Otto Kekäläinen 01 Apr '19

01 Apr '19

Hello! Can you please update the image used by test kvm-deb-jessie-ppc64le so that the 'apt-get update' is fresh and package lists are recent? It would fix the error in http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-ppc64le/build… E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/m/mpfr4/libmpfr4_3.1.2-2_ppc64el.… 404 Not Found [IP: 64.50.233.100 80] E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/g/gcc-4.9/libatomic1_4.9.2-10_ppc… 404 Not Found [IP: 64.50.233.100 80] E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/r/rsync/rsync_3.1.1-3_ppc64el.deb 404 Not Found [IP: 64.50.233.100 80] E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/s/socat/socat_1.7.2.4-2_ppc64el.d… 404 Not Found [IP: 64.50.233.100 80] E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/libd/libdbd-mysql-perl/libdbd-mys… 404 Not Found [IP: 64.50.233.100 80] Also, can you please run 'apt-get install libzstd-dev' in the image used in arm64-xenial-deb? It would fix http://buildbot.askmonty.org/buildbot/builders/arm64-xenial-deb/builds/2973… dpkg-checkbuilddeps: error: Unmet build dependencies: libzstd-dev dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting Thanks!

2 1

Re: [Maria-developers] required mysql_upgrade; MDEV-14637
by Michal Schorm 01 Apr '19

01 Apr '19

On Thu, Mar 28, 2019 at 11:16 AM Michal Schorm <mschorm(a)redhat.com> wrote: > > Hello, > I have a questions regarding the warnings showing up at 10.2 and 10.3 releases: > > When upgrading from MariaDB 10.2.16 or earlier to MariaDB 10.2.17 or higher, > running mysql_upgrade is required due to changes introduced in MDEV-14637. > > When upgrading from MariaDB 10.3.8 or earlier to MariaDB 10.3.9 or higher, > running mysql_upgrade is required due to changes introduced in MDEV-14637. > > -- > > 1) The same applies for upgrading <=10.2.16 to >= 10.3.9, right? > > 2) Does it mean, it is impossible to use "dump and restore" technique, > or it only means I need to run the mysql_upgrade afterwards? > > 3) The problem is not triggered when upgrading from 5.5, 10.0 and 10.1? > 4) The problem is not triggered when upgrading to 10.4 ? I found a reply in the mailing list archive which haven't reached my inbox: From: Reindl Harald Date: Thu, 28 Mar 2019 11:59:50 +0100 > just run "mysql_upgrade" after each update and you are done > our production datadirs date back to 2002, originally on windows then > moved to MacOS and in 2008 to Fedora and i did not dump/restore a single > time in my whole life, this is not postgresql My question is focused on the clarification in our downstream documentation. So while usually I'd be fine with personal experience, in this case, I'd prefer more in-depth answer covering. I'm forwarding this mail to devel mailing list. -- Michal Schorm Software Engineer Core Services - Databases Team Red Hat --

1 0

Re: [Maria-developers] 9976699033c: Test fixes (versioning.replace, versioning.foreign)
by Sergei Golubchik 31 Mar '19

31 Mar '19

Hi, Aleksey! On Mar 27, Aleksey Midenkov wrote: > revision-id: 9976699033c (mariadb-10.3.7-30-g9976699033c) > parent(s): c0f97710582 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2018-05-30 13:12:47 +0300 > message: > > Test fixes (versioning.replace, versioning.foreign) > > * replace.test: unwrapped create_table procedure; > * foreign.test: use key_type combinations. > > diff --git a/mysql-test/suite/versioning/key_type.combinations b/mysql-test/suite/versioning/key_type.combinations > index 1929aee9a84..fb3888d3d0f 100644 > --- a/mysql-test/suite/versioning/key_type.combinations > +++ b/mysql-test/suite/versioning/key_type.combinations > @@ -1,2 +1,3 @@ > [unique] > -[pk] > +[primary] > +[secondary] generally prefer shorter combination names, otherwise mtr output becomes garbled and difficult to read. > diff --git a/mysql-test/suite/versioning/primary_or_unique.inc b/mysql-test/suite/versioning/primary_or_unique.inc > new file mode 100644 > index 00000000000..199d8bab942 > --- /dev/null > +++ b/mysql-test/suite/versioning/primary_or_unique.inc > @@ -0,0 +1,5 @@ > +--source suite/versioning/key_type.inc > +--require suite/versioning/primary_or_unique.require don't do .require, please, it's a technique from 1998, before mysqltest had the "skip" command. Use if (...) { skip; } > +disable_query_log; > +eval select "$KEY_TYPE" != "key" as pk_or_unique; > +enable_query_log; > diff --git a/mysql-test/suite/versioning/t/engines.combinations b/mysql-test/suite/versioning/t/engines.combinations > deleted file mode 100644 > index 561c5656929..00000000000 > --- a/mysql-test/suite/versioning/t/engines.combinations > +++ /dev/null > @@ -1,8 +0,0 @@ > -[timestamp] > -default-storage-engine=innodb > - > -[trx_id] > -default-storage-engine=innodb > - > -[myisam] > -default-storage-engine=myisam Is that intentional? I thought many tests use it to run in different versioning modes. > diff --git a/mysql-test/suite/versioning/t/foreign.test b/mysql-test/suite/versioning/t/foreign.test > index 566d481c2a8..72e5b047cbb 100644 > --- a/mysql-test/suite/versioning/t/foreign.test > +++ b/mysql-test/suite/versioning/t/foreign.test > @@ -1,11 +1,14 @@ > +--source suite/versioning/primary_only.inc > --source suite/versioning/common.inc what's the point of this? if you're only allowing primary key here, you can just as well write manually "primary key" in all tests, without $KEY_TYPE and replace_result. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] EITS member variables in both TABLE and TABLE_SHARE?
by Sergey Petrunia 29 Mar '19

29 Mar '19

Hello Igor, TABLE_SHARE has TABLE_STATISTICS_CB, which has these members: struct TABLE_STATISTICS_CB { ... bool stats_is_read; /* Statistical data for table has been read from statistical tables */ bool histograms_are_read; } At the same time, TABLE has: struct TABLE { ... bool stats_is_read; /* Persistent statistics is read for the table */ bool histograms_are_read; I would like to understand why both structures need these variables? Is there any difference in meaning between them? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 74b2eba1ca6: MDEV-15458 Segfault in heap_scan() upon UPDATE after ADD SYSTEM VERSIONING
by Sergei Golubchik 29 Mar '19

29 Mar '19

Hi, Aleksey! On Mar 26, Aleksey Midenkov wrote: > revision-id: 74b2eba1ca6 (mariadb-10.3.12-115-g74b2eba1ca6) > parent(s): 2c0901d808b > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2019-03-26 17:43:38 +0100 > message: > > MDEV-15458 Segfault in heap_scan() upon UPDATE after ADD SYSTEM VERSIONING > > * HA_EXTRA_REMEMBER_POS, HA_EXTRA_RESTORE_POS for HEAP storage engine > * Versioning tests support > > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index abe4254db3e..1d51947cc86 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -1643,7 +1643,12 @@ int vers_insert_history_row(TABLE *table) > if (row_start->cmp(row_start->ptr, row_end->ptr) >= 0) > return 0; > > - return table->file->ha_write_row(table->record[0]); > + int res; > + if ((res= table->file->extra(HA_EXTRA_REMEMBER_POS))) > + return res; > + if ((res= table->file->ha_write_row(table->record[0]))) > + return res; > + return table->file->extra(HA_EXTRA_RESTORE_POS); > } Frankly, I don't like it. I mean, this particular fix is fine, but here's the problem: We now have (at least) three features where the server might need to do something (insert/update/search) during the index/rnd scan. And three different solutions: you use HA_EXTRA_REMEMBER_POS in system versioning, Nikita simply sets delete_while_scanning=false in application time period tables, and Sachin uses handler::clone() to create a second handler that can be used to check for hash collisions without disrupting the scan on the primary handler. I think it's getting somewhat out of hands. Can we please reduce the number of approaches to fix the same issue? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

4 4

Re: [Maria-developers] ab6ef429a84: MDEV-16370 row-based binlog events (updates by primary key) can not be applied multiple times to system versioned tables
by Sergei Golubchik 29 Mar '19

29 Mar '19

Hi, Aleksey! On Jan 31, Aleksey Midenkov wrote: > revision-id: ab6ef429a84 (versioning-1.0.7-2-gab6ef429a84) > parent(s): 14c2f90ad08 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2018-11-26 12:46:37 +0300 > message: > > MDEV-16370 row-based binlog events (updates by primary key) can not be applied multiple times to system versioned tables > > Case 2: > > * Idempotent BINLOG updates: duplicate key on history row is skipped; > * Fix row start and warn when applying BINLOG from the past. As far as I can see now you cannot just arbitrarily adjust row_start. Think of it in term or application time period tables, a primary key is (kind of) defined WITHOUT OVERLAPS. There is no explicit constraint, it happens automatically when the history is valid. Now, here's your test case: timestamp=100; insert (1,1) -> the table has (1,1,100,MAX); timestamp=300; update to (1,3) -> (1,1,100,300),(1,3,300,MAX); timestamp=200; update to (1,2) -> here the hisory row should've been (1,3,300,200) but you adjust it to get 1,1,100,300 1,3,199,200 1,2,200,MAX and if you select AS OF 199, you'll get two rows for the pk=1 So, let's * ignore HA_ERR_FOUND_DUPP_KEY (please check if it's needed for insert or delete too) * add TIMESTAMP=xxx to mysqldumps. can vers_insert_history_row see row_start>row_end in a normal use case? no replication or any other way of adjusting the timestamp. equality can surely happen, but greater than ? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] THD::set_time()
by Aleksey Midenkov 28 Mar '19

28 Mar '19

What if system_time.sec_part == TIME_MAX_SECOND_PART here: else { start_time= t; start_time_sec_part= ++system_time.sec_part; } ? -- All the best, Aleksey Midenkov @midenok

1 0

[Maria-developers] (no subject)
by Samyak Jain 28 Mar '19

28 Mar '19

I want to contribute for full outer join gsoc project of mariaDB. Please provide the technical requirements for this and how I start contributing on this

1 0

Re: [Maria-developers] c1bf5766aed: MDEV-16490: It's possible to make a system versioned table without any versioning field
by Sergei Golubchik 27 Mar '19

27 Mar '19

Hi, Nikita! On Mar 27, Nikita Malyavin wrote: > revision-id: c1bf5766aed (versioning-1.0.5-41-gc1bf5766aed) > parent(s): 141a5b24843 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2018-07-20 23:37:26 +1000 > message: > > MDEV-16490: It's possible to make a system versioned table without any versioning field > > * do not allow versioned table to be without versioned (non-system) fields > * prohibit changing field versioning, when removing table versioning > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index dbf400b53d7..ee4cf457f45 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -8473,11 +8473,21 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > } > } > > - if (table->versioned() && !(alter_info->flags & ALTER_DROP_SYSTEM_VERSIONING) && > - new_create_list.elements == VERSIONING_FIELDS) > + if (table->versioned()) > { > - my_error(ER_VERS_TABLE_MUST_HAVE_COLUMNS, MYF(0), table->s->table_name.str); > - goto err; > + uint versioned_fields= 0; > + field_it.rewind(); > + while (Create_field *f= field_it++) > + { > + if (!(f->flags & VERS_UPDATE_UNVERSIONED_FLAG)) > + versioned_fields++; > + } > + if (!(alter_info->flags & ALTER_DROP_SYSTEM_VERSIONING) && > + versioned_fields == VERSIONING_FIELDS) > + { > + my_error(ER_VERS_TABLE_MUST_HAVE_COLUMNS, MYF(0), table->s->table_name.str); > + goto err; > + } wouldn't that fit naturally into Vers_parse_info::fix_alter_info() ? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] e0fa1993692: MDEV-18899: Server crashes in Field::set_warning_truncated_wrong_value
by Sergey Petrunia 27 Mar '19

27 Mar '19

Hi Varun, First, Some input on the test: On Wed, Mar 27, 2019 at 01:55:16PM +0530, Varun wrote: > revision-id: e0fa1993692a3552feddae56cbee3078d4da2121 (mariadb-10.2.22-91-ge0fa1993692) > parent(s): 50a8fc52988d13a5164a1a542b9d7a85e3ecc1c1 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2019-03-27 13:48:44 +0530 > message: > > MDEV-18899: Server crashes in Field::set_warning_truncated_wrong_value > > To fix the crash there we need to make sure that the > server while storing the statistical values in statistical tables should do it > in a multi-byte safe way. > Also there is no need to throw warnings if there is truncation while storing > values from statistical fields. > ... > --- a/mysql-test/r/stat_tables_innodb.result > +++ b/mysql-test/r/stat_tables_innodb.result > @@ -651,6 +651,63 @@ SELECT MAX(pk) FROM t1; > MAX(pk) > NULL > DROP TABLE t1; > +# > +# MDEV-18899: Server crashes in Field::set_warning_truncated_wrong_value > +# > +set names utf8; > +set @save_optimizer_use_condition_selectivity=@@optimizer_use_condition_selectivity; > +set optimizer_use_condition_selectivity=4; > +set use_stat_tables=preferably; > +set @save_histogram_size= @@histogram_size; > +set histogram_size=255; > +create table t1 ( a varchar(255) character set utf8); > +insert into t1 values (REPEAT('ӥ',255)), (REPEAT('ç',255)); > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status OK > +select HEX(RIGHT(min_value, 1)), length(min_value) from mysql.column_stats; > +HEX(RIGHT(min_value, 1)) length(min_value) > +A7 254 > +select HEX(RIGHT(max_value, 1)), length(max_value) from mysql.column_stats; > +HEX(RIGHT(max_value, 1)) length(max_value) > +A5 254 > +analyze select * from t1 where a >= 'ӥ'; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 2.00 50.00 50.00 Using where > +set @save_sql_mode= @@sql_mode; > +set sql_mode='ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'; > +update mysql.column_stats set min_value= REPEAT('ӥ',255); This looks scary, what if there were some other data in these tables? Can you change add a WHERE db_name=database() and table_name='t1' ? > +Warnings: > +Warning 1265 Data truncated for column 'min_value' at row 1 > +select HEX(RIGHT(min_value, 1)), length(min_value) from mysql.column_stats; > +HEX(RIGHT(min_value, 1)) length(min_value) > +D3 255 > +analyze select * from t1 where a >= 'ӥ'; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 2.00 50.00 50.00 Using where > +set names latin1; > +drop table t1; > +CREATE TABLE t1 (col1 date); > +INSERT INTO t1 VALUES('2004-01-01'),('2004-02-29'); > +INSERT INTO t1 VALUES('0000-10-31'); > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status OK > +update mysql.column_stats set min_value='2004-0-31123'; The same as above. > +select min_value from mysql.column_stats; > +min_value > +2004-0-31123 This case also makes one wonder whether not producing a warning was a good idea after all. A varchar string that is 255-byte long, and after 255 bytes it gets truncated in the middle of a character is a rare case. An incorrect datetime value... not as rare? > +select * from t1; Is the above supposed to read the EITS data? It doesn't as the query doesn't use the WHERE clause. > +col1 > +2004-01-01 > +2004-02-29 > +0000-10-31 > +drop table t1; > +set @@sql_mode= @save_sql_mode; > set use_stat_tables=@save_use_stat_tables; > +set @@histogram_size= @save_histogram_size; > +set @@optimizer_use_condition_selectivity=@save_optimizer_use_condition_selectivity; > set optimizer_switch=@save_optimizer_switch_for_stat_tables_test; > SET SESSION STORAGE_ENGINE=DEFAULT; > diff --git a/mysql-test/t/stat_tables.test b/mysql-test/t/stat_tables.test > index b89ab2bbd2d..114c5c97a7e 100644 > --- a/mysql-test/t/stat_tables.test > +++ b/mysql-test/t/stat_tables.test > @@ -401,4 +401,44 @@ SELECT MAX(pk) FROM t1; > > DROP TABLE t1; > > +--echo # > +--echo # MDEV-18899: Server crashes in Field::set_warning_truncated_wrong_value > +--echo # > + > +set names utf8; > +set @save_optimizer_use_condition_selectivity=@@optimizer_use_condition_selectivity; > +set optimizer_use_condition_selectivity=4; > +set use_stat_tables=preferably; > +set @save_histogram_size= @@histogram_size; > +set histogram_size=255; > + > +create table t1 ( a varchar(255) character set utf8); > +insert into t1 values (REPEAT('ӥ',255)), (REPEAT('ç',255)); > + > +analyze table t1; > +select HEX(RIGHT(min_value, 1)), length(min_value) from mysql.column_stats; > +select HEX(RIGHT(max_value, 1)), length(max_value) from mysql.column_stats; > +analyze select * from t1 where a >= 'ӥ'; > + > +set @save_sql_mode= @@sql_mode; > +set sql_mode='ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'; > +update mysql.column_stats set min_value= REPEAT('ӥ',255); > +select HEX(RIGHT(min_value, 1)), length(min_value) from mysql.column_stats; > +analyze select * from t1 where a >= 'ӥ'; > + > +set names latin1; > +drop table t1; > + > +CREATE TABLE t1 (col1 date); > +INSERT INTO t1 VALUES('2004-01-01'),('2004-02-29'); > +INSERT INTO t1 VALUES('0000-10-31'); > +analyze table t1; > +update mysql.column_stats set min_value='2004-0-31123'; > +select min_value from mysql.column_stats; > +select * from t1; > +drop table t1; > + > +set @@sql_mode= @save_sql_mode; > set use_stat_tables=@save_use_stat_tables; > +set @@histogram_size= @save_histogram_size; > +set @@optimizer_use_condition_selectivity=@save_optimizer_use_condition_selectivity; > diff --git a/sql/field.cc b/sql/field.cc > index 080cf34c76d..c26195a3650 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -2219,7 +2219,6 @@ bool Field_str::can_be_substituted_to_equal_item(const Context &ctx, > return false; > } > > - > void Field_num::make_field(Send_field *field) > { > Field::make_field(field); > @@ -7023,12 +7022,16 @@ Field_longstr::check_string_copy_error(const String_copier *copier, > { > const char *pos; > char tmp[32]; > + THD *thd= get_thd(); > > if (!(pos= copier->most_important_error_pos())) > return FALSE; > > - convert_to_printable(tmp, sizeof(tmp), pos, (end - pos), cs, 6); > - set_warning_truncated_wrong_value("string", tmp); > + if (thd->count_cuted_fields) > + { > + convert_to_printable(tmp, sizeof(tmp), pos, (end - pos), cs, 6); > + set_warning_truncated_wrong_value("string", tmp); > + } > return TRUE; > } > > diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc > index b5811c683e8..0960a9a1ec5 100644 > --- a/sql/sql_statistics.cc > +++ b/sql/sql_statistics.cc > @@ -1044,6 +1044,7 @@ class Column_stat: public Stat_table > { > char buff[MAX_FIELD_WIDTH]; > String val(buff, sizeof(buff), &my_charset_bin); > + uint32 length= 0; > > for (uint i= COLUMN_STAT_MIN_VALUE; i <= COLUMN_STAT_HISTOGRAM; i++) > { > @@ -1060,7 +1061,9 @@ class Column_stat: public Stat_table > else > { > table_field->collected_stats->min_value->val_str(&val); > - stat_field->store(val.ptr(), val.length(), &my_charset_bin); > + length= Well_formed_prefix(val.charset(), val.ptr(), > + MY_MIN(val.length(), stat_field->field_length)).length(); > + stat_field->store(val.ptr(), length, &my_charset_bin); > } > break; > case COLUMN_STAT_MAX_VALUE: > @@ -1069,7 +1072,9 @@ class Column_stat: public Stat_table > else > { > table_field->collected_stats->max_value->val_str(&val); > - stat_field->store(val.ptr(), val.length(), &my_charset_bin); > + length= Well_formed_prefix(val.charset(), val.ptr(), > + MY_MIN(val.length(), stat_field->field_length)).length(); > + stat_field->store(val.ptr(), length, &my_charset_bin); > } > break; > case COLUMN_STAT_NULLS_RATIO: > @@ -2934,7 +2939,6 @@ int update_statistics_for_table(THD *thd, TABLE *table) > DBUG_RETURN(rc); > } > > - > /** > @brief > Read statistics for a table from the persistent statistical tables > @@ -2980,6 +2984,7 @@ int read_statistics_for_table(THD *thd, TABLE *table, TABLE_LIST *stat_tables) > Table_statistics *read_stats= table_share->stats_cb.table_stats; > > DBUG_ENTER("read_statistics_for_table"); > + DBUG_ASSERT(thd->count_cuted_fields == CHECK_FIELD_IGNORE); > > /* Read statistics from the statistical table table_stats */ > stat_table= stat_tables[TABLE_STAT].table; > @@ -3059,7 +3064,7 @@ int read_statistics_for_table(THD *thd, TABLE *table, TABLE_LIST *stat_tables) > } > } > } > - > + > table->stats_is_read= TRUE; > > DBUG_RETURN(0); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0