- developers - lists.mariadb.org

[Maria-developers] Rollback causes assertion with Galera 10.0
by Jan Lindström 10 Sep '13

10 Sep '13

Hi, Background: I merged 10.0 to lp:~maria-captains/maria/maria-10.0-galera/, merge result is pushed to the tree. In testing I hit repeatable to following assertion #0 0x00007f1ff987f037 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007f1ff9882698 in __GI_abort () at abort.c:90 #2 0x00007f1ff9877e03 in __assert_fail_base (fmt=0x7f1ff99ce578 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x109b4ff "empty()", file=file@entry=0x109b4d8 "/home/jan/mysql/galera-10.0/sql/log.cc", line=line@entry=294, function=function@entry=0x10a01e0 <binlog_cache_data::reset()::__PRETTY_FUNCTION__> "void binlog_cache_data::reset()") at assert.c:92 #3 0x00007f1ff9877eb2 in __GI___assert_fail (assertion=0x109b4ff "empty()", file=0x109b4d8 "/home/jan/mysql/galera-10.0/sql/log.cc", line=294, function=0x10a01e0 <binlog_cache_data::reset()::__PRETTY_FUNCTION__> "void binlog_cache_data::reset()") at assert.c:101 #4 0x00000000009497d4 in binlog_cache_data::reset (this=0x7f1f7403ad60) at /home/jan/mysql/galera-10.0/sql/log.cc:294 #5 0x0000000000949b6a in binlog_cache_mngr::reset (this=0x7f1f7403ab90, do_stmt=false, do_trx=true) at /home/jan/mysql/galera-10.0/sql/log.cc:462 #6 0x0000000000935a4c in binlog_truncate_trx_cache (thd=0x45b7d40, cache_mngr=0x7f1f7403ab90, all=true) at /home/jan/mysql/galera-10.0/sql/log.cc:1991 #7 0x0000000000936123 in binlog_rollback (hton=0x3891fb0, thd=0x45b7d40, all=true) at /home/jan/mysql/galera-10.0/sql/log.cc:2180 #8 0x000000000086b750 in ha_rollback_trans (thd=0x45b7d40, all=true) at /home/jan/mysql/galera-10.0/sql/handler.cc:1616 #9 0x00000000007a0c5f in trans_rollback (thd=0x45b7d40) at /home/jan/mysql/galera-10.0/sql/transaction.cc:321 #10 0x00000000006718b2 in mysql_execute_command (thd=0x45b7d40) at /home/jan/mysql/galera-10.0/sql/sql_parse.cc:4634 #11 0x0000000000677d52 in mysql_parse (thd=0x45b7d40, rawbuf=0x7f1f74005288 "ROLLBACK", length=8, parser_state=0x7f1fe83bb610) at /home/jan/mysql/galera-10.0/sql/sql_parse.cc:6816 #12 0x00000000006774d3 in wsrep_mysql_parse (thd=0x45b7d40, rawbuf=0x7f1f74005288 "ROLLBACK", length=8, parser_state=0x7f1fe83bb610) at /home/jan/mysql/galera-10.0/sql/sql_parse.cc:6647 #13 0x00000000006688cd in dispatch_command (command=COM_QUERY, thd=0x45b7d40, packet=0x44807c1 "ROLLBACK ", packet_length=9) at /home/jan/mysql/galera-10.0/sql/sql_parse.cc:1451 #14 0x00000000006678a3 in do_command (thd=0x45b7d40) at /home/jan/mysql/galera-10.0/sql/sql_parse.cc:1077 #15 0x000000000078ba4d in do_handle_one_connection (thd_arg=0x45b7d40) at /home/jan/mysql/galera-10.0/sql/sql_connect.cc:1404 #16 0x000000000078b78d in handle_one_connection (arg=0x45b7d40) at /home/jan/mysql/galera-10.0/sql/sql_connect.cc:1311 #17 0x0000000000b0c20a in pfs_spawn_thread (arg=0x45713a0) at /home/jan/mysql/galera-10.0/storage/perfschema/pfs.cc:1853 #18 0x00007f1ffa423f8e in start_thread (arg=0x7f1fe83bc700) at pthread_create.c:311 #19 0x00007f1ff9941e1d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 (gdb) We go from log.cc::binlog_rollback : else if (ending_trans(thd, all) || (!(thd->variables.option_bits & OPTION_KEEP_LOG) && (!stmt_has_updated_non_trans_table(thd) || WSREP_FORMAT(thd->variables.binlog_format) != BINLOG_FORMAT_STMT) && (!cache_mngr->trx_cache.changes_to_non_trans_temp_table() || WSREP_FORMAT(thd->variables.binlog_format) != BINLOG_FORMAT_MIXED))) error= binlog_truncate_trx_cache(thd, cache_mngr, all); end there to cache_mngr->reset(false, true); and there is DBUG_ASSERT(empty()); bool empty() const { return pending() == NULL && my_b_tell(&cache_log) == 0; } At empty() this is what does not hold : my_b_tell(&cache_log) == 0; // not sure what this means At test output there is at the end: # 2013-09-09T10:03:21 [6620] Query: UPDATE LOW_PRIORITY IGNORE `table100_int_autoinc` AS X SET `col_char_12` = 220 WHERE X . `col_char_12_key` IN ( 2427781120 , 1 , 'i' , 'm' , 'o' ) ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 8 failed: 1213 Deadlock found when trying to get lock; try restarting transaction. Further errors of this kind will be suppressed. # 2013-09-09T10:04:14 [6620] Query: DELETE LOW_PRIORITY IGNORE FROM `table10_key_pk_parts_2_int_autoinc` WHERE `col_char_12_key` IN ( 4200660992 , 1646198784 , 3888709632 , 32 , 'c' ) ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 1 failed: 1205 Lock wait timeout exceeded; try restarting transaction. Further errors of this kind will be suppressed. Unfortunately, general log contains so many statements from different connections that they are not really helpful for getting repeatable test case. R: Jan -- Jan Lindström Principal Engineer MariaDB | MaxScale | skype: jan_p_lindstrom www.skysql.com <http://www.skysql.com/> Twitter <http://twitter.com/skysql> Blog <http://www.skysql.com/blog/> Facebook <http://www.facebook.com/skysql> LinkedIn <http://www.linkedin.com/company/1214250> Google+ <https://plus.google.com/117544963211695643458/posts>

2 4

[Maria-developers] AIX "make fails" - related to: MDEV-4941
by Michael Felt 09 Sep '13

09 Sep '13

AIX make fails with syntax errors. One has been fixed, per https://mariadb.atlassian.net/browse/MDEV-4941. Rather than submit error by error I thought it would be more productive to provide the output from make -i. After trying, but not seeing it 'attach', I am going to try attaching the output of make -i here. regards, Michael

2 2

[Maria-developers] Some clarifications requested for C client
by Peter Laursen 09 Sep '13

09 Sep '13

This is probably for Georg Richter (and/or @Wlad). I have a few qeustion! 1) https://bazaar.launchpad.net/~maria-captains/mariadb-native-client/trunk/re… "Fix for CONC-44: LOAD DATA INFILE can't open utf16le encoded filenames" But we we can stil not specify UTF16LE as the CHARSET, I think? Say I have a file *named* "ÄÜÑБГД.csv". (both Western special characters and cyrillic characters in the file name). This file I will be able to read now (using revision 80), right? But if file *content* is this single line (one row, two columns) ÄÜÑ,БГД and if it is encoded as 'Windows Unicode'/utf16le I still will need to recode the file to utf8 before executing it with LOAD DATA (as UTF16LE still cannot be specified for CHARSET in LOAD DATA), right? Please confirm that I understand this! 2) https://bazaar.launchpad.net/~maria-captains/mariadb-native-client/trunk/re… "Added support for embedded (sqlite)". This happened by beginning of this year but I just noticed now. Now what can we actually do with SQLite and the MariaDB C-API and what we the purpose of this revision? Is it to make it easier to write client applications using SQLite for internal storage? Or to make the C client code 'a generic API' for connecting to both MySQL (of all flavors) and SQLite? Is the the code/files that that were added different from offical SQLite downloads from http://www.sqlite.org/? I am not able to find any documentation anywhere. Also not a blog or whatever either. There must be some reason to add sqlite3 code?! 3) This seems to be the latest revision: https://bazaar.launchpad.net/~maria-captains/mariadb-native-client/trunk/re… "Added dyamic column api; Added character set conversion" ('dyamic' is misspelled BTW!) Won't we need some documentation here? The claim that MariaDB's connector "has exactly the same API as the MySQL Connector/C for MySQL 5.5" ( https://kb.askmonty.org/en/mariadb-client-library-for-c/) cannot be true now as it adds funtionalities for managing MariaDB ' dymamic column waht oracle's API has not? The API from Oracle's library is now a *subset* or MariaDB API (or MariaDB's is a *superset* or Oracle). So much I can figure out, but should I need to code with it I would need description of functions' syntax, expected results and error listings. Also I am not able to figure out what the "character set conversion" referred to is doing and why it was added. 4) Why does the changelog here https://kb.askmonty.org/en/mariadb-client-library-for-c-100-changelog/ stop at revision 77? -- Peter Laursen -- Webyog

2 3

Re: [Maria-developers] Rev 3883: MDEV-4472 Audit-plugin. Server-related part of the task
by Sergei Golubchik 08 Sep '13

08 Sep '13

Hi, Holyfoot! Just one question, see below. Otherwise ok. On Sep 08, holyfoot(a)askmonty.org wrote: > revno: 3883 > revision-id: holyfoot(a)askmonty.org-20130908092209-xar471jadl7lmwzo > parent: sergii(a)pisem.net-20130907203634-htu89dqmwe5gqwy5 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: 55-4472 > timestamp: Sun 2013-09-08 14:22:09 +0500 > message: > MDEV-4472 Audit-plugin. Server-related part of the task. > file_logger became the service. > Data like query_id now are sent to the audit plugin. > Fix for MDEV-4770 ported from 10.0. > Fix added for the read_maria_plugin_info(). > Log rotation can be disabled with 'set rotations=0'. > === renamed file 'plugin/sql_errlog/sql_logger.cc' => 'mysys/file_logger.c' > --- a/plugin/sql_errlog/sql_logger.cc 2013-06-22 12:02:03 +0000 > +++ b/mysys/file_logger.c 2013-09-08 09:22:09 +0000 > @@ -16,7 +16,7 @@ > > #include "my_global.h" > #include <my_sys.h> > -#include "service_logger.h" > +#include <mysql/service_logger.h> > #include <my_pthread.h> > > extern MYSQL_PLUGIN_IMPORT char *mysql_data_home; I believe we agreed to keep file_logger.c completely clean from dynamic-plugin-related code. Did you forget to remove the line above? Regards, Sergei

1 0

Re: [Maria-developers] MDEV-4472 Audit-plugin. Server-related part of the task
by Sergei Golubchik 08 Sep '13

08 Sep '13

Hi, Holyfoot! On Sep 05, holyfoot(a)askmonty.org wrote: > At file:///home/hf/wmar/55-4472/ > > === renamed file 'plugin/sql_errlog/sql_logger.cc' => 'mysys/file_logger.c' > --- a/plugin/sql_errlog/sql_logger.cc 2013-06-22 12:02:03 +0000 > +++ b/mysys/file_logger.c 2013-09-05 12:47:00 +0000 > @@ -16,10 +16,12 @@ > > #include "my_global.h" > #include <my_sys.h> > -#include "service_logger.h" > +#include <mysql/service_logger.h> > #include <my_pthread.h> > > +#ifndef _WIN32 > extern MYSQL_PLUGIN_IMPORT char *mysql_data_home; > +#endif /*!_WIN32*/ > > #ifdef HAVE_PSI_INTERFACE > /* These belong to the service initialization */ > @@ -51,6 +53,12 @@ LOGGER_HANDLE *logger_open(const char *p > unsigned int rotations) > { > LOGGER_HANDLE new_log, *l_perm; > + //char *local_mysql_data_home; > +//#ifdef _WIN32 > + //local_mysql_data_home= (char *)GetProcAddress(0, "mysql_data_home"); > +//#else > + //local_mysql_data_home= mysql_data_home; > +//#endif /*_WIN32*/ I think you can completely remove this and the #ifndef above. For your MySQL plugin you can simply #include this in the plugin body like extern char *mysql_data_home; #include "file_logger.c" and later in your plugin_init function: #ifdef _WIN32 mysql_data_home= (char *)GetProcAddress(0, "mysql_data_home"); #endif This way you will keep file_logger.c completely free of the dynamic plugin hacks. > /* > I don't think we ever need more rotations, > @@ -168,6 +176,29 @@ int logger_vprintf(LOGGER_HANDLE *log, c > int logger_rotate(LOGGER_HANDLE *log) > { > int result; I believe you've added recently a test to disable rotation if number of rotations is 0? Regards, Sergei

2 1

[Maria-developers] first contribution
by AugustQ 06 Sep '13

06 Sep '13

Hi, after some struggling I got this message from Bazaar: Created new stacked branch referring to /+branch-id/633549. I hope everything worked fine. If not let me know so that I can correct it. I created a branch to 10.0.4 of MariaDB and named it: contrib_AQ_01. I've uploaded this some minutes before. For some time I played with the source of MySQL 5.5.8. I've taken some of my modifications to MariaDB 10.0.4 and want to give them to you for further discussions. To say it first: this is not production-ready-code, it's a proof of a concept. My modifications are made for the MyIsam-engine in the handling of a full table-scan with a table of dynamic sized records. The reason for uploading this is: during my tests I executed statements like SELECT * FROM table INTO OUTFILE 'x.x'; With my test-data this took about 20 seconds with the original code and 12 seconds with my modifications. I've tested my code with a single table which contains 10 Mio records, a total size of 1GB. As i've written before: it's a proof of concept. I've not implemented all the possible cases. I wanted to know if a full implementation is a complete waste of time or if it is a bit useful (in this case I will do it). Some more details: in mi_dynrec.c there is a funtion _mi_get_block_info() and i've added a funtion AQ_mi_get_block_info(), these 2 functions do the same thing but in a different way. In my version of this function I've implemented only those case that I needed for my tests. Because I started with examining the source of MySQL I modified the code in the MyIsam-engine and applied these to the MyIsam-engine in MariaDB. For staying compatible with MySQL I think it's better to use these modifications in the Maria-engine of MariaDB, but only if you agree with my approach. This is what I modified: in myisamdef.h I added: #define USE_CODE_OF_AQ converting this line in a comment allowed me to use the old code, a define with this identifier activates my code after a recompile. This allowed me to do some comparisons. You will find my modifications in myisamdef.h, mi_dynrec.c and mi_extra.c if you look for this identifier. Please look at my approach. If you agree I will continue with this and check all the cases which i didn't touch for now. thank you AugustQ

1 0

Re: [Maria-developers] MDEV-4978 - Server cursor is broken with blobs in the select list
by Sergei Golubchik 06 Sep '13

06 Sep '13

Hi, Sergey! On Sep 06, Sergey Vojtovich wrote: > revno: 3878 > revision-id: svoj(a)mariadb.org-20130906111656-oy2c3pwy2o1q0zqn > parent: wlad(a)montyprogram.com-20130903204512-hgwu77prle0lw7a3 > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5-mdev4978 > timestamp: Fri 2013-09-06 15:16:56 +0400 > message: > MDEV-4978 - Server cursor is broken with blobs in the select list, > ORDER BY does not work > > Use "dynamic" row format (instead of "block") for MARIA internal > temporary tables created for cursors. > With "block" row format MARIA may shuffle rows, with "dynamic" row > format records are inserted sequentially (there are no gaps in data > file while we fill temporary tables). > This is needed to preserve row order when scanning materialized cursors. > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2013-09-03 14:41:07 +0000 > +++ b/sql/sql_select.cc 2013-09-06 11:16:56 +0000 > @@ -15373,7 +15374,8 @@ create_tmp_table(THD *thd, TMP_TABLE_PAR > if (thd->is_fatal_error) // If end of memory > goto err; /* purecov: inspected */ > share->db_record_offset= 1; > - table->used_for_duplicate_elimination= (param->sum_func_count == 0 && > + table->used_for_duplicate_elimination= keep_row_order || > + (param->sum_func_count == 0 && > (table->group || table->distinct)); That's very confusing. Why keep_row_order means that the table is "used for duplicate elimination"? I had to check the code to see that used_for_duplicate_elimination affects the row format in Aria. This is not obvious too. And it forces the row format to be DYNAMIC for a very different reason, not for preserving row order. Perhaps it'd be better to rename used_for_duplicate_elimination to "force_dynamic_Aria_row_format" ? A prettier solution would be to introduce another flag: table->keep_row_order. Regards, Sergei

2 2

[Maria-developers] Bazaar-problem
by AugustQ 06 Sep '13

06 Sep '13

Hi, I'm struggling with Bazaar. Maybe you can help me. I've made some modifications to the source-code and committed these to the local repository. Next I tried to upload them via push. This is what I receive: Cannot lock LockDir(chroot-70341520:///% 2Bbranch/maria/.bzr/branch/lock): Transport operation not possible: readonly transport As I'm using Eclipse I found this within Eclipse on the console: Error while executing push Cannot lock LockDir(chroot-70341520:///% 2Bbranch/maria/.bzr/branch/lock): Transport operation not possible: readonly transport ------ XmlRpcClient Executing: push "bzr+ssh://bazaar.launchpad.net/+branch/maria/" Can you please give me a tip. Thanks AugustQ

5 9

[Maria-developers] Incorrect format description event skipping in queue_event()
by Pavel Ivanov 04 Sep '13

04 Sep '13

Kristian, You have the following comment in the queue_event() in sql/slave.cc: /* Do not queue any format description event that we receive after a reconnect where we are skipping over a partial event group received before the reconnect. (If we queued such an event, and it was the first format_description event after master restart, the slave SQL thread would think that the partial event group before it in the relay log was from a previous master crash and should be rolled back). */ I don't understand which failure scenario you are talking about here and I claim that this bypassing of queuing into relay log is incorrect. My reasoning is: relay log can always be rotated in the middle of event group. During rotation when new relay log file is created format description event is always written at the beginning of the file. So in this case SQL thread will see format description event in the middle of event group and according to my testing it doesn't rollback the active transaction because of that. So when do you think it will rollback? Now, you may say that there's no problem in skipping the format description event because everything works. But it's only for now. When IO thread is reconnecting it rotates relay log and as I said it writes format description event at the beginning of the new file. But it writes an event that it created itself, i.e. not the one that master have sent. And as format description event from master is not written into relay log SQL thread from this point on starts to use format description generated by slave which may be different from the one generated by master. It may lead to a broken replication and SQL thread may not recognize events in the relay log if difference between master's and slave's format descriptions is significant. Another somewhat related question: Gtid_log_event::peek() (as well as Gtid_log_event constructor from const char* buf) is implemented with assumption that Format_description_log_event::common_header_len is always equal to LOG_EVENT_HEADER_LEN. While currently it's true I believe common_header_len can be increased in future MariaDB versions. And when it happens old slave won't be able to replicate from a new master. I'd think this is a bug. What do you think? Pavel

2 10

Re: [Maria-developers] Rev 3876: MDEV-4472 Audit-plugin. Server-related part of the task
by Sergei Golubchik 04 Sep '13

04 Sep '13

Hi, Holyfoot! Mostly ok. There are few comments, see below: On Sep 03, holyfoot(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3876 > revision-id: holyfoot(a)askmonty.org-20130903082521-xpia2wond6oggpsk > parent: igor(a)askmonty.org-20130831163309-vqtls4oubhvz9zcz > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: 55-4472 > timestamp: Tue 2013-09-03 13:25:21 +0500 > message: > MDEV-4472 Audit-plugin. Server-related part of the task. > file_logger became the service. > Data like query_id now are sent to the audit plugin. > Fix for MDEV-4770 ported from 10.0. > Fix added for the read_maria_plugin_info(). > === modified file 'include/mysql/plugin_audit.h' > --- a/include/mysql/plugin_audit.h 2013-04-19 10:50:16 +0000 > +++ b/include/mysql/plugin_audit.h 2013-09-03 08:25:21 +0000 > @@ -25,7 +25,7 @@ > > #define MYSQL_AUDIT_CLASS_MASK_SIZE 1 > > -#define MYSQL_AUDIT_INTERFACE_VERSION 0x0301 > +#define MYSQL_AUDIT_INTERFACE_VERSION 0x0302 > > > /************************************************************************* > @@ -59,6 +59,10 @@ struct mysql_event_general > struct charset_info_st *general_charset; > unsigned long long general_time; > unsigned long long general_rows; > + /* Added for 0x302 */ A bit confusing. Better "Added in version 0x0302" > + long long query_id; should it rather be unsigned? > + const char *database; > + unsigned int database_length; > }; > > > @@ -140,6 +144,8 @@ struct mysql_event_table > unsigned int new_database_length; > const char *new_table; > unsigned int new_table_length; > + /* Added for 0x302 */ > + long long query_id; > }; > > /************************************************************************* > > === renamed file 'plugin/sql_errlog/sql_logger.cc' => 'mysys/file_logger.c' > --- a/plugin/sql_errlog/sql_logger.cc 2013-06-22 12:02:03 +0000 > +++ b/mysys/file_logger.c 2013-09-03 08:25:21 +0000 > @@ -51,6 +53,12 @@ LOGGER_HANDLE *logger_open(const char *p > unsigned int rotations) > { > LOGGER_HANDLE new_log, *l_perm; > + char *local_mysql_data_home; > +#ifdef _WIN32 > + local_mysql_data_home= (char *)GetProcAddress(0, "mysql_data_home"); > +#else > + local_mysql_data_home= mysql_data_home; > +#endif /*_WIN32*/ You shouldn't need it, file_logger.c is part of the server, not a dynamically loaded plugin. > > /* > I don't think we ever need more rotations, > === modified file 'sql/sql_audit.h' > --- a/sql/sql_audit.h 2013-04-19 10:50:16 +0000 > +++ b/sql/sql_audit.h 2013-09-03 08:25:21 +0000 > @@ -93,7 +93,8 @@ void mysql_audit_general_log(THD *thd, t > > mysql_audit_notify(thd, MYSQL_AUDIT_GENERAL_CLASS, MYSQL_AUDIT_GENERAL_LOG, > 0, time, user, userlen, cmd, cmdlen, > - query, querylen, clientcs, 0); > + query, querylen, clientcs, (ha_rows) 0, > + thd->db, thd->db_length); no query_id? > } > } > > @@ -139,7 +140,8 @@ void mysql_audit_general(THD *thd, uint > > mysql_audit_notify(thd, MYSQL_AUDIT_GENERAL_CLASS, event_subtype, > error_code, time, user, userlen, msg, msglen, > - query.str(), query.length(), query.charset(), rows); > + query.str(), query.length(), query.charset(), rows, > + thd->db, thd->db_length); no query_id? > } > } > > === modified file 'sql/sql_plugin.cc' > --- a/sql/sql_plugin.cc 2013-06-13 18:19:11 +0000 > +++ b/sql/sql_plugin.cc 2013-09-03 08:25:21 +0000 > @@ -2645,13 +2647,16 @@ static void update_func_longlong(THD *th > static void update_func_str(THD *thd, struct st_mysql_sys_var *var, > void *tgt, const void *save) > { > - char *old= *(char **) tgt; > - *(char **)tgt= *(char **) save; > + char *value= *(char**) save; > if (var->flags & PLUGIN_VAR_MEMALLOC) > { > - *(char **)tgt= my_strdup(*(char **) save, MYF(0)); > + char *old= *(char**) tgt; > + if (value) > + *(char**) tgt= my_strdup(value, MYF(0)); add else *(char**) tgt= 0; > my_free(old); > } > + else > + *(char**) tgt= value; > } > Regards, Sergei

2 3