- developers - lists.mariadb.org

[Maria-developers] [GSoC] MDEV-7502 Automatic provisioning of slave
by Martin Kaluznik 18 Mar '15

18 Mar '15

Hello, I am student interested in GSoC project Automatic provisioning of slave ( https://mariadb.atlassian.net/browse/MDEV-7502 ). I have read current code and put together my idea of how it could be implemented. Please correct me if there is better solution for some steps or I just misunderstood something. Slave starts regular sql thread and something really similar to current io thread (maybe best to begin with will be copy of handle_slave_io and related functions, it may still end with some duplicate code, which at the end can be moved to new common functions). Slave will receive gtids of current master as response to copy-modified version of COM_BINLOG_DUMP, they will mark position from which binlogs will be sent to slave. Master proceeds with building of list containing what needs to be sent to slave. Here exact implementation will depend on whether the master itself will block any DDL queries during provisioning process or not. If he will (probably won’t – it’s not necessary and it will just slow him down), we will have guaranteed that internal containers with databases/tables won’t change and they can be just iterated through safely (not true for procedure/trigger containers?). In other case, it will be probably required to prefetch list of databases/tables to copy to slave (it can be done lazily with tables/triggers/procedures/…, holding only list of tables/… for database which is currently being processed). In the second case, slave will have to detect DDL event and interrupt connection. The range scans on primary key mentioned in description, I assume it would look like SELECT * FROM tbl WHERE pk > @lastChunkKey ORDER BY pk LIMIT @chunkSize; just done with internal functions more efficiently, and the result will be converted to Rows_log_event which will be sent to slave. To restrictions I would add at least one more, provisioning cannot be continued if master is restarted during process (or can we actually store information about provisioning process in database?). Another possible problem could be slave reconnect. I found kill_zombie_dump_threads function, in this case, new thread could take information about provisioning progress from old thread and with help of fake gtids bundled with chunks somehow figure out which chunk was last successfully received by slave before disconnect. It would require additional memory to store few (or more?) most recently sent chunks and wouldn't guarantee success. Great way to solve this would be, to be able to map gtid sequence number to exact chunk of data, but I can’t see how it can be done without assuming something about maximum amount of rows in table / number of tables / triggers / … Thanks for feedback, Martin Kaluznik ( martin.kaluznik [at] gmail.com ). I can be also found on IRC under nick Farnham

2 1

Re: [Maria-developers] [Commits] 2bdbfd3: Fix assertion failure seen on Buildbot win32-debug
by Sergei Golubchik 18 Mar '15

18 Mar '15

Hi, Jan! On Mar 18, Jan Lindström wrote: > revision-id: 2bdbfd334bba93f75b2952e8e76f20ed29fb2bd4 > parent(s): c14d9c21f0a6974d9909e00a23522114caad3bfc > committer: Jan Lindström > branch nick: 10.0-git > timestamp: 2015-03-18 12:18:39 +0200 > message: > > Fix assertion failure seen on Buildbot win32-debug > > There is a bug in Visual Studio 2010 > Visual Studio has a feature "Checked Iterators". In a debug build, every > iterator operation is checked at runtime for errors, e g, out of range. > Disable this "Checked Iterators" for Windows and Debug if defined. This comment, > +/* There is a bug in Visual Studio 2010 > +Visual Studio has a feature "Checked Iterators". In a debug build, every > +iterator operation is checked at runtime for errors, e g, out of range. > +Disable this "Checked Iterators" for Windows and Debug if defined. > +*/ and this comment don't explain what bug is it. They create an impression that Visual Studio has run-time checks for an iterator running out of range. And that our code fails this check (because the iterator runs out of range). And that your "fix" is to disable the check instead of fixing the buggy out-of-range iterator. I'm not saying that's what happens, I'm saying that your comment suggests that this happens, because you didn't describe what was the actual bug. A better comment should say that our iterator is correct, but a buggy check erroneously marks it as out-of-range. It should explain what iterator is it, exactly, why it's correct, etc. Could you please fix the comment please? (or the iterator, whatever should be fixed) > +#ifdef UNIV_DEBUG > +#ifdef __WIN__ > +#ifdef _ITERATOR_DEBUG_LEVEL > +#define _ITERATOR_DEBUG_LEVEL 0 this will produce a warning "macro _ITERATOR_DEBUG_LEVEL is redefined" > +#endif /* _ITERATOR_DEBUG_LEVEL */ > +#endif /* __WIN__*/ > +#endif /* UNIV_DEBUG */ Regards, Sergei

1 0

Re: [Maria-developers] [Commits] 97037da: Replace static usage of AES_CTR with current encryption algorithm.
by Jeremy Cole 18 Mar '15

18 Mar '15

Howdy Jan, > But the whole encryption_algorithm stuff seems not well thought out in any >> case. >> > > This was introduced because we can't assume that all installations will > install openSSL especially on Windows, thus encryption should work > with bundled yaSSL also. Do you have some better suggestion ? > Yes, I assumed so, but the current implementation is lazy and user-hostile. And, to me, having a single global variable controlling "the" encryption algorithm (for what, exactly?) is weird. Presumably if it makes sense to have it changeable, it also makes sense to set different algorithms for different objects. At a minimum, IMHO: store the encryption_algorithm selection with each encrypted object so that data can be migrated in the future. Then, implement sufficient support in the various necessary places so that it can be safely changed. Currently as far as I can tell it is globally set once, and impossible to change in the future. This could be done via stealing some bits from either the stored encryption scheme (1 byte) or key version (4 bytes) or adding a new encryption algorithm field which is stored alongside those everywhere (this is probably the best strategy IMHO, but takes a bit of thought). Alternatively as an absolute (and even lower) minimum: ensure that changing it once the database is initialized produces a loud and very clear error message to the user. Regards, Jeremy

1 0

[Maria-developers] Fwd: Port InnoDB memcached interface to MariaDB (GSOC 2015)
by Aalekh Nigam 18 Mar '15

18 Mar '15

Hello, I am Aalekh Nigam, 3rd Year Undergraduate student at JIIT Noida, India. I would like to propose my participation for Port InnoDB Memcache Interface to MariaDB (https://mariadb.atlassian.net/browse/MDEV-4674) as GSOC 2015 Student. Although I have been continuously involved with discussion regarding the project with the assigned mentor (Colin Charles), but would also like to propose my raw architecture for community participation , thus I would like to *request* you all to please go through the given architecture and follow up with your comments and remarks.Since the feature is critical to MariaDB, thus I will work hard on your comments and remarks to make this project a success. Architecture: According to my understanding this is the working that should be done for the project : Following are the main advantages which will come after integrating memcache to XtraDB (MariaDB Engine): - This memcache interfacing with MaraDB would help in faster execution of query in MaraDB Server. - Data can be unstructured or structured, depending on the type of application. You can make an all-new table for the data, or map the NoSQL-style processing to one or more existing tables. - Data that is requested through the memcached protocol is transparently queried from an XtraDB table, without going through the MariaDB SQL layer. - Because memcached consumes relatively little CPU, and its memory footprint is easy to control, it can run comfortably alongside a MariaDB instance on the same system. - Because memcached consumes relatively little CPU, and its memory footprint will be easy to control, it would run comfortably alongside a MariaDB instance on the same system. Detail Architecture: - Will have multi column key value support. - Will control memcache parameters through parameters like: read/write batch size, commit interval. - Direct access to XtraDb tables, bypassing the MariaDB SQL parser, the optimizer, and even the Handler API layer Thank You *Aalekh*

1 0

Re: [Maria-developers] [Commits] 97037da: Replace static usage of AES_CTR with current encryption algorithm.
by Jeremy Cole 17 Mar '15

17 Mar '15

> > I don't get it. Why is example_key_management_plugin special here? If >> you're going to support "none", why not here as well? >> > > This is exactly similar to file_key_management_plugin i.e. do not > initialize if NONE. > Heh, okay. > + >>> + my_aes_init_dynamic_encrypt(current_aes_dynamic_method); >>> >> >> Isn't this redundant? It should already be initialized by the call in >> init_server_components... >> >> >> > Maybe, but what if you load this plugin after server has started ? > Why would it matter? Unless I miss something, my_aes_encrypt_dynamic/my_aes_decrypt_dynamic are global. But the whole encryption_algorithm stuff seems not well thought out in any case. Regards, Jeremy

2 1

Re: [Maria-developers] [Commits] 97037da: Replace static usage of AES_CTR with current encryption algorithm.
by Jeremy Cole 17 Mar '15

17 Mar '15

Jan, > - my_aes_init_dynamic_encrypt(MY_AES_ALGORITHM_CTR); > + if (current_aes_dynamic_method == MY_AES_ALGORITHM_NONE) > + { > + sql_print_error("No encryption method choosen with > --encryption-algorithm. " > + "example_key_management_plugin disabled"); > + return 1; > + } > I don't get it. Why is example_key_management_plugin special here? If you're going to support "none", why not here as well? > + > + my_aes_init_dynamic_encrypt(current_aes_dynamic_method); > Isn't this redundant? It should already be initialized by the call in init_server_components... Regards, Jeremy

1 0

[Maria-developers] MDEV-6714: A couple of questions
by Vicențiu Ciorbaru 16 Mar '15

16 Mar '15

Hi Sergei, I've achieved the desired performance speedup related to MDEV-6714. You can check the preliminary patch on the commits mailing-list. I'll clean up the code and then submit a final version for review. I do however have two concerns: * By using SHOW TABLES LIKE %s, we performed a case insensitive search to find the table's correct name case. This command takes a long time with a lot of tables. Using an equality operator in I_D speeds it up, but it no longer matches if we have a different case. Possible solution: Try to search for the table with the provided string via INFORMATION_SCHEMA using the equals operator. If we can not find the table, fall back to a case insensitive search that takes longer. * The I_S.triggers and I_S.tables are available since 5.5 and 5.1 respectively. I suppose it is safe to assume we will have these tables available when running mysqldump. I can code all the changes to fall back to the previous query if the one that uses I_S tables fails. Regards, Vicentiu

3 3

Re: [Maria-developers] 53b73a7: MDEV-6714 mysqldump slow with tables in big databases
by Sergei Golubchik 16 Mar '15

16 Mar '15

Hi, Vicentiu! On Mar 16, vicentiu(a)mariadb.org wrote: > revision-id: 53b73a7940039809a02184a69502142d767b0741 > parent(s): 197afb413fcc9f06b5e5e6ef41ce980d108b354f > committer: Vicențiu Ciorbaru > branch nick: server > timestamp: 2015-03-16 21:59:26 +0200 > message: > > MDEV-6714 mysqldump slow with tables in big databases > > First draft commit. Performance speedup achieved. > > --- > client/mysqldump.c | 15 ++++++++++----- > 1 file changed, 10 insertions(+), 5 deletions(-) > > diff --git a/client/mysqldump.c b/client/mysqldump.c > index 2da4ce6..6138e94 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -3341,7 +3341,8 @@ static int dump_triggers_for_table(char *table_name, char *db_name) > /* Get list of triggers. */ > > my_snprintf(query_buff, sizeof(query_buff), > - "SHOW TRIGGERS LIKE %s", > + "SELECT TRIGGER_NAME from INFORMATION_SCHEMA.TRIGGERS " > + "where TRIGGER_SCHEMA = DATABASE() and EVENT_OBJECT_TABLE = %s", This is wrong, should be EVENT_OBJECT_SCHEMA, not TRIGGER_SCHEMA. Why? That's how I_S optimizations work - for file-based objects it might either use readdir to get all files and then select those that it needs. Or it can open the requested file directly, if it knows the database (= directory name) and the object name (= file name). It kind of works like an index over (db_name, object_name). And what fields are "indexed" is specified in the ST_SCHEMA_TABLE structure for the I_S table in question. For I_S.TRIGGERS it's {"TRIGGERS", triggers_fields_info, 0, get_all_tables, make_old_format, get_schema_triggers_record, 5, 6, 0, OPEN_TRIGGER_ONLY|OPTIMIZE_I_S_TABLE}, where triggers_fields_info is the list of fields and 5,6 are "indexed" fields. Looking at triggers_fields_info you'll see that 5,6 fields are EVENT_OBJECT_SCHEMA and EVENT_OBJECT_TABLE. > quote_for_like(table_name, name_buff)); > > if (mysql_query_with_error_report(mysql, &show_triggers_rs, query_buff)) > @@ -4716,8 +4717,12 @@ static char *get_actual_table_name(const char *old_table_name, MEM_ROOT *root) > > /* Check memory for quote_for_like() */ > DBUG_ASSERT(2*sizeof(old_table_name) < sizeof(show_name_buff)); > - my_snprintf(query, sizeof(query), "SHOW TABLES LIKE %s", > + my_snprintf(query, sizeof(query), > + "select table_name from information_schema.tables " > + "where table_schema = DATABASE() " > + "and table_name = %s", > quote_for_like(old_table_name, show_name_buff)); This is wrong, here and everywhere. quote_for_like() quotes, exactly, for LIKE. It is a wrong kind of quoting for equality operator. > + verbose_msg(query); > Regards, Sergei

1 0

Re: [Maria-developers] Gsoc 2015 - Enhancing mysqlbinlog
by Kristian Nielsen 16 Mar '15

16 Mar '15

Christos Daskalakis <daskalakis.chris(a)gmail.com> writes: > i found this very interesting project on gsoc 2015 and with another > colleague (Stelios Agapiou steagap06(a)gmail.com) we would be very > interested and happy to work on Enhancing mysqlbinlog , as it is referred > to in mariadb.com . I assume you are refering to MDEV-4989: https://mariadb.atlassian.net/browse/MDEV-4989 Rohit Kashyap also showed some interest in this project, and it was discussed a bit on the maria-developers@ mailing list (Cc:ed). So let's coordinate the efforts here. I updated the MDEV-4989 with what was discussed so far, it now better reflects what the project entails. It also now has some pointers to suggested background reading. Thank you for your interest in MariaDB, and MariaDB GTID! - Kristian.

2 1

Re: [Maria-developers] 7c21ea9: MDEV-7772: SIGSEGV on my_aes_encrypt_cbc when -DWITH_SSL=bundled
by Sergei Golubchik 14 Mar '15

14 Mar '15

Hi, Jan! On Mar 14, Jan Lindström wrote: > > and thanks for the review, some comments/questions below: > > > 1. Does file_key_management_plugin work with CTR? > > > Yes it does, but only with openSSL. Then I'd suggest to * add aes_ctr combination to have_file_key_management_plugin.combinations * use only one combinations file, not two identical ones (it can be called, say, encryption_algorithms.inc) and included into both have_*_key_management_plugin.inc files * disable aes_ctr combination from suite.pm, now it has $skip{'include/have_openssl_ctr.inc'} = 'no or too old openssl' unless $::mysqld_variables{'version-ssl-library'} =~ /OpenSSL (\S+)/ and $1 ge "1.0.1"; which disables the complete have_openssl_ctr.inc, it can be changed to disable just one combination (like for have_innodb.inc): $skip{'include/encryption_algorithms.combinations'} = [ 'aes_ctr' ] unless $::mysqld_variables{'version-ssl-library'} =~ /OpenSSL (\S+)/ and $1 ge "1.0.1"; > > This doesn't make a lot of sense now, because > > example_key_management_plugin forces CTR: > > > > my_aes_init_dynamic_encrypt(MY_AES_ALGORITHM_CTR); > > > Is there some reason why it does this here? I will change that on separate > fix to current_aes_dynamic_method if != MY_AES_ALGORITHM_NONE I suspect it was in the original google patch and nobody fixed it yet. It would be good to change it. Regards, Sergei

1 0