- developers - lists.mariadb.org

[Maria-developers] custom storage backend rnd_next() implementation and caching questions
by Andras Szabo 03 Feb '15

03 Feb '15

Hi Everyone, I am developing a custom mariadb storage backend which is intended to give filesystem related information via mariadb tables. For example one could tell: CREATE TEMPORARY TABLE IF NOT EXISTS `/var/log` ENGINE=fsview; which would create a table (using the assisted discovery method) with the following schema: MariaDB [test]> describe `/var`; +-----------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-----------+--------------+------+-----+---------+-------+ | file_name | varchar(255) | YES | | NULL | | | contents | longblob | YES | | NULL | | +-----------+--------------+------+-----+---------+———+ However i have some implementation issues: 1. I am having trouble implementing rnd_next() in the backend: int ha_fsview::rnd_next(uchar *buf) { DBUG_ENTER(__PRETTY_FUNCTION__); if (!dirp) { DBUG_RETURN(HA_ERR_INTERNAL_ERROR); } if (struct dirent* entry = readdir(dirp)) { uchar *pos = buf; *pos = 0; ++pos; // file name (1bytes size + str) *pos = strlen(entry->d_name); ++pos; size_t bytes_to_copy = strlen(entry->d_name); memcpy(pos, entry->d_name, bytes_to_copy); pos+=bytes_to_copy; DBUG_PRINT("info", ("fsview filename: %s", entry->d_name)); // content (4 bytes size + content ptr) static const std::string fake_content = "hello world"; int s = fake_content.size(); memcpy(pos, &s, sizeof(s)); pos+=sizeof(s); const char* c = fake_content.c_str(); memcpy(pos, &c, sizeof(c)); pos+=sizeof(c); DBUG_RETURN(0); } else { DBUG_RETURN(HA_ERR_END_OF_FILE); } } My problem is that filenames (first field) is ok, but i cannot really see the expected content “hello world” in the second field. I am assuming that i am doing something wrong, but i cannot really find the reason. Could somebody help me, please? 2. If i am issuing: SELECT * FROM `/var`; twice, then the server is not initiating a table scan again on my backend (most likely results from the first run are stored in some cache). So if filesystem contents are changed between the two statements, then it won’t be reflected in the resultset. Of course if i do a: SELECT SQL_NO_CACHE * FROM `/var`; then everything is fine, however i would not like to put the burden on the user to specify the option to not cache the results. My question is whether it’s possible to specify in the storage backend that the results from the tables should not be cached. Thank for the help in advance -- Andras Szabo Sent with Airmail

3 7

Re: [Maria-developers] [Commits] Rev 4577: MDEV-5719: Wrong result with GROUP BY and LEFT OUTER JOIN in lp:~maria-captains/maria/10.0
by Sergey Petrunia 29 Jan '15

29 Jan '15

Hi Sergei, == TL;DR == I have debugged that LEFT JOIN problem. The fix is not pretty, but correct. I will now push it into the 10.0 tree. == Details == So, the commit changes - return (table->null_row ? TRUE : - null_ptr ? MY_TEST(null_ptr[row_offset] & null_bit) : 0); + return real_maybe_null() ? + MY_TEST(null_ptr[row_offset] & null_bit) : table->null_row; The patch changes the behavior for NULLable fields. Suppose * real_maybe_null()= TRUE * table->null_row==TRUE // the table has a NULL-complemented row * (null_ptr[row_offset] & null_bit) == FALSE // however the value // is not null. the old code will return TRUE the new code will return FALSE The new code may seem wrong at first glance (it's a NULL-complemented row, why we return FALSE from is_null()?) However, if we look where table->null_row is set to true, we will find that it is done here: inline void mark_as_null_row(TABLE *table) { table->null_row=1; table->status|=STATUS_NULL_ROW; bfill(table->null_flags,table->s->null_bytes,255); } The function also sets all NULL-bits, so normally one can assume that (table->null_row=TRUE) => ((null_ptr[..] & null_bit) == TRUE) However, there is a case when the above does not hold. It happens when we run GROUP BY query, and GROUP BY is satisfied by scanning an index which produces rows in group order. In that case, the following can occur: consider the query from the testcase: SELECT ... FROM table_one a LEFT JOIN table_two b ON a.oid=b.oid GROUP BY a.oid * we read from a row (1,3) * we make index lookup in table b and find row (3). * row combination a(1,3)-b(3) passes the WHERE clause, GROUP BY execution code records it as current row. * there are no other matches in table b, so we get back to reading table a. * we read row a(1,4) * we look for matches in table b. there are no matches, so we produce a NULL-complemented row b(NULLs) * row combination a(1,4)-b(NULLs) is passed to the GROUP BY runtime. The runtime sees that we are now in a different group, and figures that it should pass the row combination representing the previous group into query output. * It attempts to pass the row combination a(1,3)-b(3) (*) into query output. However, the current row for table b is already the NULL-complemented row. the row combination (*) is stored in the "result_field" objects (and not in "regular field" objects which refer to table->record[0]). It seems wrong that we look at result_field (which points to a row that's different from table->record[0]), and at the same time we are looking at table->null_row, which refers to the record in table->record[0]. However, "result_field" storage has no place for table->null_row bit. The fix basically means: "for NULL-able fields, field's NULL-bit should override table->null_row". Considering the above, it is correct (albeit ugly). Now, what about GROUP BY and NOT NULL fields? I discovered the following: if a table has table->maybe_null, then Field objects that refer to such table and are pointed by Item_field::result_field, are always NULLable. (code-wise, check out setup_copy_fields(), these calls: item->result_field=field->new_field(thd->mem_root,field->table, 1); item->result_field->move_field(copy->to_ptr,copy->to_null_ptr,1); result_field object of NOT NULL column is null-able. On Fri, Jan 23, 2015 at 08:25:59AM -0800, Nirbhay Choubey wrote: > At lp:~maria-captains/maria/10.0 > > ------------------------------------------------------------ > revno: 4577 > revision-id: nirbhay(a)mariadb.com-20150123162551-h88uxn780ivj5k2n > parent: sergii(a)pisem.net-20150119153032-fqrmxjr13z0p1ly0 > committer: Nirbhay Choubey <nirbhay(a)mariadb.com> > branch nick: b5719-10.0 > timestamp: Fri 2015-01-23 11:25:51 -0500 > message: > MDEV-5719: Wrong result with GROUP BY and LEFT OUTER JOIN > > Merged revision 5224 from mysql-5.6 and added a test case. > .. > revno: 5224 > committer: Sergey Glukhov <sergey.glukhov(a)oracle.com> > branch nick: mysql-5.6 > timestamp: Wed 2013-06-19 14:24:08 +0400 > message: > Bug#16620047 INCORRECT QUERY RESULT (AFTER SERVER UPGRADE) > .. > === modified file 'mysql-test/r/group_by_innodb.result' > --- a/mysql-test/r/group_by_innodb.result 2014-01-26 20:49:11 +0000 > +++ b/mysql-test/r/group_by_innodb.result 2015-01-23 16:25:51 +0000 > @@ -57,3 +57,26 @@ > NULL 11.1,22.2 > DROP TABLE t1; > End of 5.5 tests > +# > +# MDEV-5719: Wrong result with GROUP BY and LEFT OUTER JOIN > +# > +CREATE TABLE table_one (oidGroup INT, oid INT PRIMARY KEY)ENGINE=INNODB; > +INSERT INTO table_one VALUES (1,1),(1,2),(1,3),(1,4); > +CREATE TABLE table_two (oid INT PRIMARY KEY)ENGINE=INNODB; > +INSERT INTO table_two VALUES (3); > +SELECT a.oidGroup, a.oid, b.oid FROM table_one a LEFT JOIN table_two b ON > +a.oid=b.oid WHERE a.oidGroup=1; > +oidGroup oid oid > +1 1 NULL > +1 2 NULL > +1 3 3 > +1 4 NULL > +SELECT a.oidGroup, a.oid, b.oid FROM table_one a LEFT JOIN table_two b ON > +a.oid=b.oid WHERE a.oidGroup=1 GROUP BY a.oid; > +oidGroup oid oid > +1 1 NULL > +1 2 NULL > +1 3 3 > +1 4 NULL > +DROP TABLE table_one, table_two; > +# End of tests > > === modified file 'mysql-test/t/group_by_innodb.test' > --- a/mysql-test/t/group_by_innodb.test 2014-01-26 20:49:11 +0000 > +++ b/mysql-test/t/group_by_innodb.test 2015-01-23 16:25:51 +0000 > @@ -67,3 +67,22 @@ > > --echo End of 5.5 tests > > +--echo # > +--echo # MDEV-5719: Wrong result with GROUP BY and LEFT OUTER JOIN > +--echo # > +CREATE TABLE table_one (oidGroup INT, oid INT PRIMARY KEY)ENGINE=INNODB; > +INSERT INTO table_one VALUES (1,1),(1,2),(1,3),(1,4); > + > +CREATE TABLE table_two (oid INT PRIMARY KEY)ENGINE=INNODB; > +INSERT INTO table_two VALUES (3); > + > +# Returns a value > +SELECT a.oidGroup, a.oid, b.oid FROM table_one a LEFT JOIN table_two b ON > +a.oid=b.oid WHERE a.oidGroup=1; > + > +SELECT a.oidGroup, a.oid, b.oid FROM table_one a LEFT JOIN table_two b ON > +a.oid=b.oid WHERE a.oidGroup=1 GROUP BY a.oid; > + > +DROP TABLE table_one, table_two; > + > +--echo # End of tests > > === modified file 'sql/field.h' > --- a/sql/field.h 2014-10-20 12:42:00 +0000 > +++ b/sql/field.h 2015-01-23 16:25:51 +0000 > @@ -656,21 +656,28 @@ > inline bool is_null(my_ptrdiff_t row_offset= 0) const > { > /* > + If the field is NULLable, it returns NULLity based > + on null_ptr[row_offset] value. Otherwise it returns > + NULL flag depending on TABLE::null_row value. > + > The table may have been marked as containing only NULL values > for all fields if it is a NULL-complemented row of an OUTER JOIN > or if the query is an implicitly grouped query (has aggregate > functions but no GROUP BY clause) with no qualifying rows. If > - this is the case (in which TABLE::null_row is true), the field > - is considered to be NULL. > + this is the case (in which TABLE::null_row is true) and the > + field is not nullable, the field is considered to be NULL. > + > + Do not change the order of testing. Fields may be associated > + with a TABLE object without being part of the current row. > + For NULL value check to work for these fields, they must > + have a valid null_ptr, and this pointer must be checked before > + TABLE::null_row. > + > Note that if a table->null_row is set then also all null_bits are > set for the row. > - > - Otherwise, if the field is NULLable, it has a valid null_ptr > - pointer, and its NULLity is recorded in the "null_bit" bit of > - null_ptr[row_offset]. > */ > - return (table->null_row ? TRUE : > - null_ptr ? MY_TEST(null_ptr[row_offset] & null_bit) : 0); > + return real_maybe_null() ? > + MY_TEST(null_ptr[row_offset] & null_bit) : table->null_row; > } > inline bool is_real_null(my_ptrdiff_t row_offset= 0) const > { return null_ptr ? (null_ptr[row_offset] & null_bit ? 1 : 0) : 0; } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 2

[Maria-developers] Dynamic columns support in libmaria client library?
by Peter Laursen 28 Jan '15

28 Jan '15

This page https://mariadb.com/kb/en/mariadb/documentation/nosql/dynamic-columns-api/ does not mention anything about it. it just says *The API is a part of libmysql C client library. In order to use it, one needs to include this header file* *#include <mysql/ma_dyncol.h>* *and link against libmysql.* Now can ma_dyncol.h be linked against libmaria as well? There are 2 questions actually 1) technical: will it work with libmaria? 2) legal: can a client program using libmaria under LGPL license include ma_dyncol.h? -- Peter -- Webyog

3 5

Re: [Maria-developers] [Commits] 51fa027: Improve performance for calculating memory allocation
by Sergei Golubchik 25 Jan '15

25 Jan '15

Hi, Monty! On Jan 23, Michael Widenius wrote: > revision-id: 51fa0272febd782674b3d44828718e805c44c1a8 > parent(s): ab440b0fb7302d707ba0ba41382bf911404db1cb > committer: Michael Widenius > branch nick: maria-10.1 > timestamp: 2015-01-23 14:32:53 +0200 > message: > > Improve performance for calculating memory allocation See my comments below: > diff --git a/include/mysql/plugin.h b/include/mysql/plugin.h > index 16b0c6e..9ae0fe8 100644 > --- a/include/mysql/plugin.h > +++ b/include/mysql/plugin.h > @@ -190,6 +196,7 @@ struct st_mysql_show_var { > > #define SHOW_VAR_FUNC_BUFF_SIZE (256 * sizeof(void*)) > typedef int (*mysql_show_var_func)(MYSQL_THD, struct st_mysql_show_var*, char *); > +typedef int (*mysql_show_var_extended_func)(MYSQL_THD, struct st_mysql_show_var*, enum enum_var_type, char *); I'd put scope the last making mysql_show_var_extended_func backward compatible with mysql_show_var_func. this makes some code simpler, see below. > > /* > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 044b60d..8eed7b9 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -3115,9 +3115,16 @@ static bool show_status_array(THD *thd, const char *wild, > if var->type is SHOW_FUNC or SHOW_SIMPLE_FUNC, call the function. > Repeat as necessary, if new var is again one of the above > */ > - for (var=variables; var->type == SHOW_FUNC || > - var->type == SHOW_SIMPLE_FUNC; var= &tmp) > - ((mysql_show_var_func)(var->value))(thd, &tmp, buff); > + for (var=variables; > + var->type == SHOW_FUNC || var->type == SHOW_SIMPLE_FUNC || > + var->type == SHOW_EXTENDED_FUNC; > + var= &tmp) > + { > + if (var->type == SHOW_EXTENDED_FUNC) > + ((mysql_show_var_extended_func)(var->value))(thd, &tmp, scope, buff); > + else > + ((mysql_show_var_func)(var->value))(thd, &tmp, buff); You won't need this if() when scope is the last argument. In fact I'm still thinking whether it would be a good idea to add scope to mysql_show_var_func and SHOW_FUNC, without introducing SHOW_EXTENDED_FUNC at all. This wouldn't break any existing plugin, as it's perfectly binary compatible extension. Also, see below. > + } > > SHOW_TYPE show_type=var->type; > if (show_type == SHOW_ARRAY) > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 0d49407..d200af6 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1747,6 +1749,16 @@ void add_to_status(STATUS_VAR *to_var, STATUS_VAR *from_var) > to_var->binlog_bytes_written+= from_var->binlog_bytes_written; > to_var->cpu_time+= from_var->cpu_time; > to_var->busy_time+= from_var->busy_time; > + to_var->local_memory_used+= from_var->local_memory_used; why? it's not additive, right? and global_status_var.local_memory_used is never shown anyway. > + > + /* > + Update global_memory_used. We have to do this with atomic_add as the > + global value can change outside of LOCK_status. > + */ > + // workaround for gcc 4.2.4-1ubuntu4 -fPIE (from DEB_BUILD_HARDENING=1) > + int64 volatile * volatile ptr= &to_var->global_memory_used; I'd rather remove this workaround. 4.2.4 is ancient, and quite probably this bug doesn't happen with MY_MEMORY_ORDER_RELAXED at all > + my_atomic_add64_explicit(ptr, from_var->global_memory_used, > + MY_MEMORY_ORDER_RELAXED); > } > > /* > @@ -1784,6 +1796,11 @@ void add_diff_to_status(STATUS_VAR *to_var, STATUS_VAR *from_var, > dec_var->binlog_bytes_written; > to_var->cpu_time+= from_var->cpu_time - dec_var->cpu_time; > to_var->busy_time+= from_var->busy_time - dec_var->busy_time; > + > + /* > + We don't need to accumulate memory_used as these are not after > + this function is called. Sorry, I cannot parse that :( > + */ > } > > #define SECONDS_TO_WAIT_FOR_KILL 2 > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index ea53e47..c4e1dc3 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -3972,19 +3972,24 @@ static void my_malloc_size_cb_func(long long size, my_bool is_thread_specific) > if (thd) > { > DBUG_PRINT("info", ("memory_used: %lld size: %lld", > - (longlong) thd->status_var.memory_used, size)); > + (longlong) thd->status_var.local_memory_used, > + size)); > - thd->status_var.memory_used+= size; > + thd->status_var.local_memory_used+= size; > - DBUG_ASSERT((longlong) thd->status_var.memory_used >= 0); > + DBUG_ASSERT((longlong) thd->status_var.local_memory_used >= 0); > } > } > } > + else if (likely(thd)) > + thd->status_var.global_memory_used+= size; Hmm, ok. So, you think that current_thd is cheaper than updating a global variable? > + else > + { > // workaround for gcc 4.2.4-1ubuntu4 -fPIE (from DEB_BUILD_HARDENING=1) > - int64 volatile * volatile ptr=&global_status_var.memory_used; > + int64 volatile * volatile ptr=&global_status_var.global_memory_used; I would've removed this workaround too > my_atomic_add64_explicit(ptr, size, MY_MEMORY_ORDER_RELAXED); > + } > } > } > > - > static int init_common_variables() > { > umask(((~my_umask) & 0666)); > @@ -8028,6 +8033,21 @@ static int show_default_keycache(THD *thd, SHOW_VAR *var, char *buff) > return 0; > } > > + > +static int show_memory_used(THD *thd, SHOW_VAR *var, enum enum_var_type scope, > + char *buff) > +{ > + var->type= SHOW_LONGLONG; > + var->value= buff; > + if (scope == OPT_GLOBAL) > + *(longlong*) buff= (global_status_var.local_memory_used + > + global_status_var.global_memory_used); > + else > + *(longlong*) buff= thd->status_var.local_memory_used; > + return 0; I'm surprised you haven't added SHOW_EXTENDED_SIMPLE_FUNC :) That would've make a very confusing name though. May be we should indeed add scope as the last argument to mysql_show_var_func and reuse both SHOW_SIMPLE_FUNC and SHOW_FUNC? I'm starting to think it's a good thing to do. Makes the API simpler and the number of choices to understand - smaller. If you do that - please don't forget to increment MARIA_PLUGIN_INTERFACE_VERSION. > +} > + > + > #ifndef DBUG_OFF > static int debug_status_func(THD *thd, SHOW_VAR *var, char *buff) > { Regards, Sergei

1 0

[Maria-developers] Additions to GIS in 10.1.
by Alexey Botchkov 23 Jan '15

23 Jan '15

Hi all. What's new about GIS in 10.1. Related tasks in JIRA are: MDEV-4045 Missing OGC Spatial functions. MDEV-60 Support for Spatial Reference systems for the GIS data. MDEV-12 OpenGIS: create required tables: GeometryColumns, related views. Speaking shortly, the MariaDB GIS part is now OpenGIS compliant, and passes all the OpenGIS conformance tests To get that the next things were done: Missing functions were added: IsRing(ln) The return type is Integer, with a return value of 1 for TRUE, 0 for FALSE, and –1 for UNKNOWN corresponding to a function invocation on NULL arguments; return TRUE if c is a ring, i.e., if c is closed and simple. A simple Curve does not pass through the same Point more than once. PointOnSurface(s) The return type is POINT, returns a Point guaranteed to lie on the Surface. arguments: POLYGON or MYLTIPOLYGON. NumInteriorRing(p) The return type is Integer, returns the number of interiorRings arguments: POLYGON. Relate(g1 Geometry, g2 Geometry, patternMatrix String) The return type is Integer, with a return value of 1 for TRUE, 0 for FALSE, and –1 for UNKNOWN corresponding to a function invocation on NULL arguments; returns TRUE if the spatial relationship specified by the pattern Matrix holds. ConvexHull(g1 Geometry) The return is Geometry, returna a geometric object that is the convex hull of g1 arguments: GEOMETRY. The information about the GEOMETRY column's reference system (SRID) is now stored in the field's metadata. Syntax of the CREATE and ALTER was modified to specify the SRID for the geometry columns. With this information some views required by OpenGIS, were added: GEOMETRY_COLUMNS, SPATIAL_REF_SYS: These are links to the INFORMATION_SCHEMA.GEOMETRY_COLUMNS and INFORMATION_SCHEMA.SPATIAL_REF_SYS respectively. INFORMATION_SCHEMA.GEOMETRY_COLUMNS: describes the available feature tables and their Geometry properties. Fields: F_TABLE_CATALOG CHARACTER VARYING(256) NOT NULL, F_TABLE_SCHEMA CHARACTER VARYING(256) NOT NULL, F_TABLE_NAME CHARACTER VARYING(256) NOT NULL, the above 3 fields are the fully qualified name of the feature table containing the geometry column; F_GEOMETRY_COLUMN CHARACTER VARYING(256) NOT NULL, the name of the column in the feature table that is the Geometry Column. G_TABLE_CATALOG CHARACTER VARYING(256) NOT NULL, G_TABLE_SCHEMA CHARACTER VARYING(256) NOT NULL, G_TABLE_NAME CHARACTER VARYING(256) NOT NULL, the above 3 fields are the name of the geometry table and its schema and catalog. The geometry table implements the geometry column; STORAGE_TYPE INTEGER, always 1 in MariaDB (means binary geometry implementation) GEOMETRY_TYPE INTEGER, the type of geometry values stored in this column. 0 = GEOMETRY 1 = POINT 3 = LINESTRING 5 = POLYGON 7 = MULTIPOINT 9 = MULTILINESTRING 11 = MULTIPOLYGON COORD_DIMENSION INTEGER, the number of dimensions in the spatial reference system. Always 2 in MariaDB. MAX_PPR INTEGER, Always 0 in MariaDB. SRID INTEGER the ID of the Spatial Reference System used for the coordinate geometry in this table. It is a foreign key reference to the SPATIAL_REF_SYS table. INFORMATION_SCHEMA.SPATIAL_REF_SYS: stores information on each spatial reference system used in the database. Fields: SRID INTEGER NOT NULL PRIMARY KEY, an integer value that uniquely identifies each Spatial Reference System within a database. AUTH_NAME CHARACTER VARYING(256), the name of the standard or standards body that is being cited for this reference system. In most cases the value is "EPSG". AUTH_SRID INTEGER TEXT the Well-known Text Representation of the Spatial Reference System. Required stored procedures added: AddGeometryColumn(FEATURE_TABLE_CATALOG, FEATURE_TABLE_SCHEMA, FEATURE_TABLE_NAME, GEOMETRY_COLUMN_NAME, SRID) adds new column of spatial type to the specified table DropGeometryColumn(FEATURE_TABLE_CATALOG, FEATURE_TABLE_SCHEMA, FEATURE_TABLE_NAME, GEOMETRY_COLUMN_NAME) removes the spatial column from the table. That's it for now. Best regards. HF

1 0

[Maria-developers] Renaming Virtual Columns to Computed Columns
by Ian Gilfillan 19 Jan '15

19 Jan '15

There was a suggestion on the docs mailing list that we rename Virtual Columns to Computed Columns. I think it makes sense, but before we implement I want to make sure everyone is happy with the change, so am repeating it here as not everyone follows docs. Currently, we use the term Virtual Columns (see https://mariadb.com/kb/en/mariadb/documentation/sql-structure-and-commands/…) Oracle uses this term, while MS SQL and Firebird use the term Computed Columns to describe similar functionality. MariaDB's virtual column implementation has both persistent and virtual attributes. Talking of "virtual virtual" and "persistent virtual" can be confusing. Oracle's implementation, to my knowledge, only has virtual virtual, while MS SQL's Computed Columns offers both persistent ("persisted" in their case) and virtual. I think "Computed" is a slightly more descriptive term, and avoids the ambiguity of "persistent virtual" and "virtual virtual". The MariaDB error messages already use the term computed, for example "Constant expression in computed column function is not allowed". Unless anyone has a good reason to prefer the term virtual, we're going to rather use the term "computed column" in the docs. The article will still mention virtual column as an alternative etc, but computed will be the preferred term. ian

3 3

Re: [Maria-developers] [Commits] Rev 4562: MDEV-7402 'reset master' hangs, waits for signalled COND_xid_list in lp:~maria-captains/maria/10.0
by Kristian Nielsen 19 Jan '15

19 Jan '15

Sergei Golubchik <serg(a)mariadb.org> writes: > ------------------------------------------------------------ > revno: 4562 > revision-id: sergii(a)pisem.net-20150118130747-rs67yzzk8a7ebwdd > parent: sergii(a)pisem.net-20150118002113-snztyzwvzc500c2m > fixes bug: https://mariadb.atlassian.net/browse/MDEV-7402 > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 10.0 > timestamp: Sun 2015-01-18 14:07:47 +0100 > message: > MDEV-7402 'reset master' hangs, waits for signalled COND_xid_list > > Using a boolean flag for 'there is a RESET MASTER in progress' doesn't > work very well for multiple concurrent RESET MASTER statements. Ok to push with a testcase like the below, as we discussed. - Kristian. ----------------------------------------------------------------------- --echo *** MDEV-7402: 'reset master' hangs, waits for signalled COND_xid_list *** --source include/wait_for_binlog_checkpoint.inc connect(con3,localhost,root,,); # Make the binlog background thread wait before clearing the pending checkpoint. # The bug was that one RESET MASTER would clear the reset_master_pending # flag set by another RESET MASTER; this could cause the wakeup from the # binlog background thread not to be sent, and thus the second RESET MASTER # to wait infinitely. SET debug_sync="reset_logs_after_set_reset_master_pending SIGNAL reset_master_ready WAIT_FOR reset_master_cont"; send RESET MASTER; --connection default SET @old_dbug= @@global.DEBUG_DBUG; SET GLOBAL debug_dbug="+d,inject_binlog_background_thread_before_mark_xid_done"; SET debug_sync="now WAIT_FOR reset_master_ready"; RESET MASTER; SET debug_sync="now WAIT_FOR injected_binlog_background_thread"; SET GLOBAL debug_dbug=@old_dbug; SET debug_sync="now SIGNAL reset_master_cont"; --connection con3 REAP; --connection default SET debug_sync = 'reset';

1 0

Re: [Maria-developers] [MDEV-7362] ANALYZE TABLES crash with table-independent-statistics gathering
by Sergei Golubchik 17 Jan '15

17 Jan '15

Hi, Vicențiu! On Jan 17, Vicențiu Ciorbaru wrote: > > Here's the new diff, let me know if you have any more comments. Shall I > push this as is? Yes, just one small detail (file name and the comment), see below. Then please push, no need to ask for another review :) But... 1. Please don't paste patches into the email anymore. At least not with your mail client that wraps long lines around and sends everything as html. See https://mariadb.com/kb/en/using-git/#commit-emails 2. Why git? This is the patch for 10.0 bug, at the moment 10.0 is still in bzr. 3. This looks like a reverse diff, how did you manage to do that? :) > diff --git a/mysql-test/t/statistics_index_crash.test b/mysql-test/t/statistics_index_crash.test > deleted file mode 100644 > index dbbdca3..0000000 > --- a/mysql-test/t/statistics_index_crash.test if you create a new test file for one bug only (as opposite to adding to an existing test file), mention the bug number in the file name. Like statistics_index_crash-7362.test > +++ /dev/null > @@ -1,26 +0,0 @@ > ---source include/have_stat_tables.inc > ---source include/have_innodb.inc Start a test case for a bug with a comment that says the bug number, the synopsys, and optionally more information, if needed. The bare minimum looks like # # MDEV-7362 ANALYZE TABLES crash with table-independent-statistics gathering # MDEV-7380 engine-independent stats SEGV on ANALYZE TABLE (#2) # > - > -CREATE TABLE t1 (a longtext, FULLTEXT KEY (`a`)) ENGINE=InnoDB; > -INSERT INTO t1 VALUES (unhex('3E0D0A4141414142334E7A6143317963324541414141424977414141674541726D')); > -ANALYZE TABLE t1 PERSISTENT FOR ALL; > ---sorted_result > -SELECT * FROM mysql.index_stats WHERE index_name='a' AND table_name='t1'; > -DROP TABLE t1; > - > -CREATE TABLE t1 (a longtext, FULLTEXT KEY (`a`)) ENGINE=MyISAM; > -INSERT INTO t1 VALUES (unhex('3E0D0A4141414142334E7A6143317963324541414141424977414141674541726D')); > -ANALYZE TABLE t1 PERSISTENT FOR ALL; > ---sorted_result > -SELECT * FROM mysql.index_stats WHERE index_name='a' AND table_name='t1'; > -DROP TABLE t1; > - > -CREATE TABLE geom (g GEOMETRY NOT NULL, SPATIAL INDEX(g)) ENGINE=MyISAM; > -INSERT INTO geom VALUES > - (MultiPolygonFromText('MULTIPOLYGON(((28 26,28 0,84 0,84 42,28 26),(52 18,66 23,73 9,48 6,52 18)),((59 18,67 18,67 13,59 13,59 18)))')), > - (MPolyFromText('MULTIPOLYGON(((28 26,28 0,84 0,84 42,28 26),(52 18,66 23,73 9,48 6,52 18)),((59 18,67 18,67 13,59 13,59 18)))')), > - (MPolyFromWKB(AsWKB(MultiPolygon(Polygon(LineString(Point(0, 3), Point(3, 3), Point(3, 0), Point(0, 3))))))); > -ANALYZE TABLE geom PERSISTENT FOR ALL; > ---sorted_result > -SELECT * FROM mysql.index_stats WHERE index_name='g' AND table_name='geom'; > -DROP TABLE geom; Regards, Sergei

1 0

[Maria-developers] [MDEV-7362] ANALYZE TABLES crash with table-independent-statistics gathering
by Vicențiu Ciorbaru 17 Jan '15

17 Jan '15

Hi Sergei! I've implemented a fix for the bug described in MDEV-7362, that also fixes MDEV-7380. The test case also covers MDEV-7380. With the following fix,the I've spent a bit of time trying to figure out the architecture of things there. I am not confident my fix is complete, or if this is the way we want to go about fixing it. The fix does what we discussed over IRC, that we are not interested in computing statistics for a FULLTEXT index. What I think would be the root cause of the problem is calling ha_index_init() on an index that does not support this. By calling table->file->ha_index_init(), the table->field array gets corrupted and that ultimately leads to a crash in the function using it afterwards. Then again, this module is completely new to me, so any comments are appreciated. Here is the diff of the first changes. diff --git a/mysql-test/t/mdev-7362.test b/mysql-test/t/mdev-7362.test new file mode 100644 index 0000000..6551893 --- /dev/null +++ b/mysql-test/t/mdev-7362.test @@ -0,0 +1,14 @@ +CREATE TABLE t1 (a longtext, FULLTEXT KEY (`a`)) ENGINE=MyISAM; +insert into t1 values (unhex('3E0D0A4141414142334E7A6143317963324541414141424977414141674541726D')); +analyze table t1 persistent for all; +drop table t1; + +CREATE TABLE t1 (a longtext, FULLTEXT KEY (`a`)) ENGINE=MyISAM; +insert into t1 values (unhex('3E0D0A4141414142334E7A6143317963324541414141424977414141674541')); +analyze table t1 persistent for all; +drop table t1; + +CREATE TABLE t1 (f longtext NOT NULL, FULLTEXT KEY f (f)) ENGINE=MyISAM; +INSERT INTO t1 VALUES (REPEAT('a',27)),('foo'); +ANALYZE TABLE t1 PERSISTENT FOR ALL; +drop table t1; diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc index d368145..2c1cccf 100644 --- a/sql/sql_statistics.cc +++ b/sql/sql_statistics.cc @@ -2367,6 +2367,9 @@ int collect_statistics_for_index(THD *thd, TABLE *table, uint index) DBUG_RETURN(rc); } + if (key_info->flags & HA_FULLTEXT) // skip if FULLTEXT index + DBUG_RETURN(rc); + table->key_read= 1; table->file->extra(HA_EXTRA_KEYREAD); Regards, Vicențiu

3 5

[Maria-developers] MDEV-7440 Build fails in libgroonga on Solaris
by Alexander Barkov 16 Jan '15

16 Jan '15

Hello Kentoku, Can you please have a look into this problem? https://mariadb.atlassian.net/browse/MDEV-7440 Can you fix please? Thanks.

1 0