March 2015 - developers - lists.mariadb.org

[Maria-developers] GSoC 15 : MaxScale Project under Markus Makela
by Lokesh Walase 04 Mar '15

04 Mar '15

Hello guys, As per the official list <https://mariadb.com/kb/en/mariadb/google-summer-of-code-2015/> of projects by MariaDB, I'm interested in the project - MaxScale filter to capture incoming operations for consumption in external sources ( Mentor : Markus Makela ) *I've 3 requests/doubts : * 0.I've downloaded the MaxScale code from github <https://github.com/mariadb-corporation/MaxScale> and have read the documentation. It says the filter configuration file is : MaxScale.ini I didn't find this file in the codebase. Nor was I able to figure out the code-location related to the "filter" module. 1.Kindly suggest for the further things I should do/try. Shall I install MariaDB & MaxScale? What part of code-base should I look into & play around? 2.What is the UTC time when most of you guys are online on IRC-channel (#maria)? Since I'm from India, I need to know the time when most of you guys are online to get immediate help on IRC. Thanks in advance. -- Lokesh Walase

1 0

[Maria-developers] MDEV-6838: Using too big key for internal temp tables
by Vicențiu Ciorbaru 03 Mar '15

03 Mar '15

Hi Sergei and Sergey! I've implemented the fix for MDEV-6838. I'd like a review from both of you regarding the patch. There are a couple of changes made. The main issues are outlined in the commit message. I've changed the way create_key_part_by_field gets called in order to decouple the KEY_PART_INFO creation process from any KEYINFO modifications and also documented what the method does. My other concern is that create_key_part_by_field does not seem to make any sense within the table class as it's basically a constructor for KEY_PART_INFO and makes no use of table fields. I suggest we move the code as either a constructor for KEY_PART_INFO, or an init() function. I think that would be better left as a separate patch, but the method itself is very easy to factor out in case you agree with the change. I'm looking forward for your input. Regards, Vicențiu ---------- Forwarded message ---------- From: <vicentiu(a)mariadb.org> Date: 28 February 2015 at 23:58 Subject: 45b6edb: MDEV-6838: Using too big key for internal temp tables To: commits(a)mariadb.org revision-id: 45b6edb158f8101d641f550179ee15df363f686f parent(s): fa87fc733d7855e0e5f9b959ca0bddf772ca57e5 committer: Vicențiu Ciorbaru branch nick: server timestamp: 2015-02-28 23:58:05 +0200 message: MDEV-6838: Using too big key for internal temp tables This bug manifests due to wrong computation and evaluation of keyinfo->key_length. The issues were: * Using table->file->max_key_length() as an absolute value that must not be reached for a key, while it represents the maximum number of bytes possible for a table key. * Incorrectly computing the keyinfo->key_length size during KEY_PART_INFO creation. The metadata information regarding the key such the field length (for strings) was added twice. --- mysql-test/r/table_keyinfo-6838.result | 12 ++++++++++++ mysql-test/t/table_keyinfo-6838.test | 18 ++++++++++++++++++ sql/sql_select.cc | 7 ++++--- sql/structs.h | 1 + sql/table.cc | 34 ++++++++++++++++++++++++++-------- sql/table.h | 2 +- 6 files changed, 62 insertions(+), 12 deletions(-) diff --git a/mysql-test/r/table_keyinfo-6838.result b/mysql-test/r/table_keyinfo-6838.result new file mode 100644 index 0000000..55b0350 --- /dev/null +++ b/mysql-test/r/table_keyinfo-6838.result @@ -0,0 +1,12 @@ +CREATE TABLE t1 (i INT, state VARCHAR(997)) ENGINE=MyISAM; +INSERT INTO t1 VALUES (2,'Louisiana'),(9,'Maine'); +CREATE TABLE t2 (state VARCHAR(997), j INT) ENGINE=MyISAM; +INSERT INTO t2 VALUES ('Louisiana',9),('Alaska',5); +INSERT INTO t2 SELECT t2.* FROM t2 JOIN t2 AS t3 JOIN t2 AS t4 JOIN t2 AS t5 JOIN t2 AS t6; +SET @@max_heap_table_size= 16384; +set @@optimizer_switch='derived_merge=OFF'; +set @@optimizer_switch='extended_keys=ON'; +SELECT * FROM t1 AS t1_1 LEFT JOIN ( t1 AS t1_2 INNER JOIN (SELECT * FROM t2) v2 ON t1_2.i = j ) ON t1_1.state = v2.state LIMIT 1; +i state i state state j +2 Louisiana 9 Maine Louisiana 9 +DROP TABLE t1, t2; diff --git a/mysql-test/t/table_keyinfo-6838.test b/mysql-test/t/table_keyinfo-6838.test new file mode 100644 index 0000000..5f4c9f4 --- /dev/null +++ b/mysql-test/t/table_keyinfo-6838.test @@ -0,0 +1,18 @@ +# Test case for MDEV-6838. +# Due to incorrect key_length computation, this usecase failed with the error +# Using too big key for internal temp table. + +CREATE TABLE t1 (i INT, state VARCHAR(997)) ENGINE=MyISAM; +INSERT INTO t1 VALUES (2,'Louisiana'),(9,'Maine'); + +CREATE TABLE t2 (state VARCHAR(997), j INT) ENGINE=MyISAM; +INSERT INTO t2 VALUES ('Louisiana',9),('Alaska',5); +INSERT INTO t2 SELECT t2.* FROM t2 JOIN t2 AS t3 JOIN t2 AS t4 JOIN t2 AS t5 JOIN t2 AS t6; + +SET @@max_heap_table_size= 16384; +set @@optimizer_switch='derived_merge=OFF'; +set @@optimizer_switch='extended_keys=ON'; + +SELECT * FROM t1 AS t1_1 LEFT JOIN ( t1 AS t1_2 INNER JOIN (SELECT * FROM t2) v2 ON t1_2.i = j ) ON t1_1.state = v2.state LIMIT 1; + +DROP TABLE t1, t2; diff --git a/sql/sql_select.cc b/sql/sql_select.cc index 387fdb3..dabe46e 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -7957,7 +7957,8 @@ static bool create_hj_key_for_table(JOIN *join, JOIN_TAB *join_tab, { Field *field= table->field[keyuse->keypart]; uint fieldnr= keyuse->keypart+1; - table->create_key_part_by_field(keyinfo, key_part_info, field, fieldnr); + table->create_key_part_by_field(key_part_info, field, fieldnr); + keyinfo->key_length += key_part_info->store_length; key_part_info++; } } @@ -15921,7 +15922,7 @@ bool create_internal_tmp_table(TABLE *table, KEY *keyinfo, goto err; bzero(seg, sizeof(*seg) * keyinfo->key_parts); - if (keyinfo->key_length >= table->file->max_key_length() || + if (keyinfo->key_length > table->file->max_key_length() || keyinfo->key_parts > table->file->max_key_parts() || share->uniques) { @@ -16107,7 +16108,7 @@ bool create_internal_tmp_table(TABLE *table, KEY *keyinfo, goto err; bzero(seg, sizeof(*seg) * keyinfo->key_parts); - if (keyinfo->key_length >= table->file->max_key_length() || + if (keyinfo->key_length > table->file->max_key_length() || keyinfo->key_parts > table->file->max_key_parts() || share->uniques) { diff --git a/sql/structs.h b/sql/structs.h index ae71819..9840cec 100644 --- a/sql/structs.h +++ b/sql/structs.h @@ -76,6 +76,7 @@ typedef struct st_key_part_info { /* Info about a key part */ */ uint16 store_length; uint16 key_type; + /* Fieldnr begins counting from 1 */ uint16 fieldnr; /* Fieldnum in UNIREG */ uint16 key_part_flag; /* 0 or HA_REVERSE_SORT */ uint8 type; diff --git a/sql/table.cc b/sql/table.cc index cfa0b6c..7df2ae6 100644 --- a/sql/table.cc +++ b/sql/table.cc @@ -5968,18 +5968,32 @@ bool TABLE::alloc_keys(uint key_count) } -void TABLE::create_key_part_by_field(KEY *keyinfo, - KEY_PART_INFO *key_part_info, +/** + @brief + Populate a KEY_PART_INFO structure with the data related to a field entry. + + @param key_part_info The structure to fill. + @param field The field entry that represents the key part. + @param fleldnr The number of the field, count starting from 1. + + TODO: This method does not make use of any table specific fields. It + could be refactored to act as a constructor for KEY_PART_INFO instead. +*/ + +void TABLE::create_key_part_by_field(KEY_PART_INFO *key_part_info, Field *field, uint fieldnr) -{ +{ key_part_info->null_bit= field->null_bit; key_part_info->null_offset= (uint) (field->null_ptr - (uchar*) record[0]); key_part_info->field= field; key_part_info->fieldnr= fieldnr; key_part_info->offset= field->offset(record[0]); - key_part_info->length= (uint16) field->pack_length(); - keyinfo->key_length+= key_part_info->length; + /* + field->key_length() accounts for the raw length of the field, excluding + any metadata such as length of field or the NULL flag. + */ + key_part_info->length= (uint16) field->key_length(); key_part_info->key_part_flag= 0; /* TODO: The below method of computing the key format length of the @@ -5991,17 +6005,20 @@ void TABLE::create_key_part_by_field(KEY *keyinfo, */ key_part_info->store_length= key_part_info->length; + /* + The total store length of the key part is the raw length of the field + + any metadata information, such as its length for strings and/or the null + flag. + */ if (field->real_maybe_null()) { key_part_info->store_length+= HA_KEY_NULL_LENGTH; - keyinfo->key_length+= HA_KEY_NULL_LENGTH; } if (field->type() == MYSQL_TYPE_BLOB || field->type() == MYSQL_TYPE_GEOMETRY || field->real_type() == MYSQL_TYPE_VARCHAR) { key_part_info->store_length+= HA_KEY_BLOB_LENGTH; - keyinfo->key_length+= HA_KEY_BLOB_LENGTH; // ??? key_part_info->key_part_flag|= field->type() == MYSQL_TYPE_BLOB ? HA_BLOB_PART: HA_VAR_LENGTH_PART; } @@ -6124,7 +6141,8 @@ bool TABLE::add_tmp_key(uint key, uint key_parts, if (key_start) (*reg_field)->key_start.set_bit(key); (*reg_field)->part_of_key.set_bit(key); - create_key_part_by_field(keyinfo, key_part_info, *reg_field, fld_idx+1); + create_key_part_by_field(key_part_info, *reg_field, fld_idx+1); + keyinfo->key_length += key_part_info->store_length; (*reg_field)->flags|= PART_KEY_FLAG; key_start= FALSE; key_part_info++; diff --git a/sql/table.h b/sql/table.h index 7a1e380..8324454 100644 --- a/sql/table.h +++ b/sql/table.h @@ -1269,7 +1269,7 @@ struct TABLE bool add_tmp_key(uint key, uint key_parts, uint (*next_field_no) (uchar *), uchar *arg, bool unique); - void create_key_part_by_field(KEY *keyinfo, KEY_PART_INFO *key_part_info, + void create_key_part_by_field(KEY_PART_INFO *key_part_info, Field *field, uint fieldnr); void use_index(int key_to_save); void set_table_map(table_map map_arg, uint tablenr_arg)

2 1

[Maria-developers] MDEV-6089 review
by Sergei Golubchik 03 Mar '15

03 Mar '15

Hi. Here're I'll be asking questions about the patches I'm looking at. 1. Is this mostly a merge of WL#7305 or mostly your own implementation? That is, most of the changes and design decisions are yours or "because MySQL did it"? 2. Do you work like I do (commit, push, fix failing tests, combine commits with "git rebase -i" to get a clean history) or you've just magically got 32bit rdiffs correctly in every commit? :) 3. Why "MDL objects cache won't be needed"? The comment said "On some systems (e.g. Windows XP) constructing/destructing MDL_lock objects can be fairly expensive. We use this cache to avoid these costs..." Why is this no longer an issue? 4. MDL_lock_strategy - this object is in 5.7 too, but you've done it very differently. Why? And I wonder why didn't you move the common part of two strategies to the base class. You wanted a pure interface class? 5. Added initializer callback to lf-hash. May be you should remove this hack with decreasing LF_HASH::element_size etc and move this functionality to a custom initializer? It was added in 6ba12f. In particular, see changes to lf* files and whether they can use your initializer instead. 6. Replaced hash with lock-free hash: test result changes. Is the order of rows stable? perhaps you should use --sort_results ? in MDL_lock constructor, I'd set m_strategy to 0 (ori to 1 or, may be, keep it uninitialized) as a safety measure, to ensure it's not used before MDL_lock::lf_hash_initializer(). Otherwise looks ok. Mostly a straightforward change. Regards, Sergei

2 3

[Maria-developers] passwordless mariadb root login with auht_socket in Debian
by Otto Kekäläinen 03 Mar '15

03 Mar '15

Hello Daniel! It seems there are some issues with the passwordless login patch you sent me some weeks ago (https://github.com/ottok/mariadb-10.0/pull/3) Can you help me debug the failing postinst script? I added 'set -x' to the mariadb-server-10.0.postinst script and when it is run during the configure phase it will exit with code 1 and make dpkg abort the installation. Last lines of execution: SET sql_log_bin=0; CREATE TABLE IF NOT EXISTS plugin (name char(64) COLLATE utf8_bin NOT NULL DEFAULT '\'''\'', dl char(128) COLLATE utf8_bin NOT NULL DEFAULT '\'''\'', PRIMARY KEY (name)) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='\''MySQL plugins'\'';' + logger -p daemon.err -t mysqld_safe -i + /usr/sbin/mysqld --bootstrap --user=mysql --skip-grant-tables --default-storage-engine=myisam --plugin-load-add=auth_socket + set -e + echo 'SET sql_log_bin=0; USE mysql; DELIMITER //; CREATE PROCEDURE debian_plugin_install(IN plugin_name CHAR(50), IN soname CHAR(50)) BEGIN DECLARE CONTINUE HANDLER FOR NOT FOUND EXECUTE inst_plug; set @plugin_name=plugin_name; set @soname=soname ;set @install_plugin=CONCAT("INSTALL PLUGIN ",@plugin_name," SONAME '\''", @soname, "'\''");PREPARE inst_plug FROM @install_plugin ; select PLUGIN_NAME INTO @a from information_schema.plugins where PLUGIN_NAME=@plugin_name AND PLUGIN_STATUS='\''ACTIVE'\'' AND PLUGIN_TYPE='\''AUTHENTICATION'\'' AND PLUGIN_LIBRARY LIKE concat(@soname,'\''%'\'' ); DEALLOCATE PREPARE inst_plug; END// CALL debian_plugin_install('\''unix_socket'\'', '\''auth_socket'\'') // DROP PROCEDURE debian_plugin_install//' + logger -p daemon.err -t mysqld_safe -i + /usr/sbin/mysqld --bootstrap --user=mysql --skip-grant-tables --default-storage-engine=myisam --plugin-load-add=auth_socket timo@debunstable:~$ echo $? 1 timo@debunstable:~$ echo $? 0 This implies that the exit code from the line '/usr/sbin/mysqld --bootstrap --user=mysql --skip-grant-tables --default-storage-engine=myisam --plugin-load-add=auth_socket' is 1. Any ideas why? If you want to test it yourself, add to you apt sources the test repo deb http://labs.seravo.fi/~otto/mariadb-repo/ mariabd-10.0-sid-amd64/

4 15

[Maria-developers] 10.0 is now on github
by Sergei Golubchik 02 Mar '15

02 Mar '15

Hi, Olivier, Kentoku. Since when we've moved 10.1 development to git (about a year ago), the plan always was to move other trees to git too. So, last month we've moved all our development to git, 5.5 and 10.0 trees are now on github, in the https://github.com/MariaDB/server repository. I still keep launchpad trees, lp:maria/10.0, in particular, to be able to merge 10.0-connect, 10.0-spider, and 10.0-mroonga. When needed, I merge these trees to lp:maria/10.0, and then merge from lp:maria/10.0 to a 10.0 branch in the git repository. But it would be great if you could also move the development to git and github. The easiest and safest approach would be as follows: 1. you tell me when to do it. 2. I do the last merge of your lp tree to 10.0 and then to git and push it to github. 3. you fork https://github.com/MariaDB/server under your github account and then you can create your branch and do anything you want there. 4. tell me the url of your forked tree and I'll configure buildbot to build it. this way you don't need to export your changes from bzr to git (which takes few hours) - I'll do it (it'll be a couple of minutes for me, because I've already run git-remote-bzr many times, and it has all the data cached). And the history will be safer, as we can be sure that all already exported commits won't be re-exported the second time. So, just tell me when you're ready :) By the way, if you haven't pushed anything since 10.0.17 - that is, there's nothing to merge - you skip the first two steps and fork https://github.com/MariaDB/server right away. Good luck! Regards, Sergei

1 0