- developers - lists.mariadb.org

[Maria-developers] Global transaction ID
by Kristian Nielsen 05 Oct '12

05 Oct '12

Hi Kristian, Monty asked me to connect with you regarding global transaction IDs in MariaDB. I reviewed some time ago the MySQL 5.6 global transaction ID design and implementation. Unfortunately I really did not like what I found, to the point that I decided this was not something we would want to have or support in MariaDB (see https://lists.launchpad.net/maria-developers/msg04837.html for an ealier mail on this). Both you and Giuseppe also had critisisms about it. Instead I came up with a (in my opinion) better design. I have now written this up in (great) detail here: https://mariadb.atlassian.net/browse/MDEV-26 It would be great if you (/your team) could comment on this and how it relates to what you need or any comments/suggestions etc. I know it is a long text, and a complex problem, so also feel free to just ask me for any clarifications or explanations you would like, even without digging through the whole text. The basic idea is to keep the existing concept of a stream of replication events and a slave's current position within this stream, but in a way that is uniform across the whole multi-level replication topology (global transaction ID), and that facilitates simple scripting of automated promotion of a new master. I have thought for a long time over this problem and how to best approach it and make it work well both with existing replication usage and with new features like multi-source and parallel replication. It is hard to get right, but I am fairly pleased with what I have ended up with so far. I am hopeful that this will be a good way forward, but I very much welcome any comments or alternate suggestions. Regards, - (the other) Kristian.

2 2

[Maria-developers] MariaDB Optimization for MySQL Users
by Daniel Bartholomew 03 Oct '12

03 Oct '12

MariaDB contains many new options and optimizations which, for compatibility or other reasons, are not enabled in the default install. Enabling them helps you gain extra performance from the same hardware when upgrading from MySQL to MariaDB. Monty thought this would be a good article for the Knowledgebase and I agree. I've created a stub article for this using a few short items Monty provided (just to get the ball rolling), and I am now soliciting suggestions from any and all of you so I can expand it. https://kb.askmonty.org/en/mariadb-optimization-for-mysql-users/ Please send me your MariaDB-only tips and recommended settings and other things people should enable or tweak after upgrading to MariaDB and I'll add them to the page (or feel free to do it yourself). Thanks! -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com

1 0

Re: [Maria-developers] multi-source replication, first (untested) version
by Kristian Nielsen 02 Oct '12

02 Oct '12

Michael Widenius <monty(a)askmonty.org> writes: > I have now got the first version of multi source patch ported to > MariadB. As requested, I did a quick read through of the patch. I took the patch from your 10.0-mdev253 tree, this revision: revid:monty@askmonty.org-20120930233044-e0m6u6t9q3jeqx03 Below are my comments. Overall, it was nice to see the relatively modest changes needed to get this working. - Kristian. ----------------------------------------------------------------------- > === modified file 'sql/sys_vars.cc' > --- sql/sys_vars.cc 2012-09-22 12:30:24 +0000 > +++ sql/sys_vars.cc 2012-10-01 09:55:23 +0000 > +static Sys_var_multi_source_ulong > +Sys_slave_skip_counter("sql_slave_skip_counter", > + "Skip the next N events from the master log", > + SESSION_VAR(slave_skip_counter), > + NO_CMD_LINE, > + offsetof(Master_info, rli.slave_skip_counter), This use of offsetof() on a C++ class (non-POD) is incorrect. Even if it works for you with your current version of GCC and current compiler options, it is likely to break at some point on some future compiler. And it will be hell to track down. Look, I understand that you do not like some of the ways that the C++ language is defined, like here that offsetof() is not allowed on non-POD data structures. But we have to live with it. Surely, it is better to have a standard language that compiler writers and compiler users agree on how works - even if that language is less convenient than what we would like. So let us please change this to be correct C++. I can change it for you if you want. > === modified file 'sql/slave.cc' > --- sql/slave.cc 2012-09-13 12:31:29 +0000 > +++ sql/slave.cc 2012-10-01 09:55:23 +0000 > +bool show_all_master_info(THD* thd) > +{ > + uint i, elements; > + Master_info **tmp; > + DBUG_ENTER("show_master_info"); > + > + if (send_show_master_info_header(thd, active_mi, 1)) > + DBUG_RETURN(TRUE); > + > + if (!(elements= master_info_index->master_info_hash.records)) > + goto end; > + > + /* > + Sort lines to get them into a predicted order > + (needed for test cases and to not confuse users) > + */ > + if (!(tmp= (Master_info**) thd->alloc(sizeof(Master_info*) * elements))) > + DBUG_RETURN(TRUE); > + > + for (i= 0; i < elements; i++) > + { > + tmp[i]= (Master_info *) my_hash_element(&master_info_index-> > + master_info_hash, i); > + } Isn't there a race condition here? What happens if the number of elements changes between looking up number of elements and the loop? If there is a lock protecting this, please add a safe_mutex_assert_owner() and/or a comment. The call of show_all_master_info() is like this: mysql_mutex_lock(&LOCK_active_mi); if (lex->verbose) res= show_all_master_info(thd); Does LOCK_active_mi protect all changes to multi-master state? If so, I think it should be renamed - "active_mi" is rather confusing in this case. > === modified file 'sql/sql_class.h' > --- sql/sql_class.h 2012-09-22 14:11:40 +0000 > +++ sql/sql_class.h 2012-10-01 09:55:23 +0000 > + /** > + Place holders to store Multi-source variables in sys_var.cc during > + update and show of variables. > + */ > + ulong slave_skip_counter; > + ulong max_relay_log_size; > + /* Names. These will be allocated in buffers in thd */ > + LEX_STRING default_master_connection; > + > + LEX_STRING connection_name; /* If slave */ > + char default_master_connection_buff[MAX_CONNECTION_NAME+1]; Is there not some other way to do this? THD seems to have become some kind of garbage dump for all kinds of special-purpose data. This is not good, it pollutes the CPU caches for all threads, and generally makes everything a mess :-/ I would like to see us move towards having less things in THD that are not relevant to all threads, not more. Could we add a single void * somewhere in THD for special-purpose threads? And then replication threads could use that for their special-purpose stuff. Or maybe sub-class THD. Then later we could start moving other stuff out of THD that is not relevant to all threads... > === modified file 'sql/rpl_rli.h' > --- sql/rpl_rli.h 2012-08-27 16:13:17 +0000 > +++ sql/rpl_rli.h 2012-10-01 09:55:23 +0000 > + volatile ulong slave_skip_counter; /* Must be ulong */ Such comments are extremely unhelpful. Instead explain why it must be ulong. > === modified file 'sql/item_create.cc' > --- sql/item_create.cc 2012-01-13 14:50:02 +0000 > +++ sql/item_create.cc 2012-10-01 09:55:23 +0000 > @@ -4393,27 +4393,36 @@ Create_func_master_pos_wait::create_nati > + thd->lex->safe_to_cache_query= 0; > case 3: > thd->lex->safe_to_cache_query= 0; > + case 4: > + thd->lex->safe_to_cache_query= 0; Isn't it redundant to set safe_to_cache_query twice here? > === modified file 'sql/rpl_mi.cc' > --- sql/rpl_mi.cc 2012-03-27 23:04:46 +0000 > +++ sql/rpl_mi.cc 2012-10-01 09:55:23 +0000 > + /* Store connection name and lower case connection name */ > + connection_name.length= cmp_connection_name.length= > + connection_name_arg->length; > + if ((connection_name.str= (char*) my_malloc(connection_name_arg->length*2+2, > + MYF(MY_WME)))) > + { > + cmp_connection_name.str= (connection_name.str + > + connection_name_arg->length+1); > + strmake(connection_name.str, connection_name_arg->str, > + connection_name.length); > + memcpy(cmp_connection_name.str, connection_name_arg->str, > + connection_name.length+1); > + my_casedn_str(system_charset_info, cmp_connection_name.str); > + } Is it really correct to ignore out-of-memory error here, and just do nothing? Ah, I think this is handled by checking Master_info::error() in other places? If so, then please add a comment here explaining this. > +/** > + Create a log file with a signed suffix. I do not know what is meant with the term "signed suffix". This is not something that people will understand. Does it mean "Create a log file name by appending a dash "-" followed by a suffix" ? Also rename the file create_signed_file_name() - again, the "signed" is confusing. For example "create_augmented_file_name", or "create_name_with_dash_and_suffix" or something. > + It's valid if it's a valid system name, is less than > + MAX_CONNECTION_NAME. "It is valid if it is a valid system name of length less than MAX_CONNECTION_NAME". > === modified file 'client/mysqltest.cc' > --- client/mysqltest.cc 2012-08-31 21:54:54 +0000 > +++ client/mysqltest.cc 2012-10-01 09:55:23 +0000 > + if (buff) > + my_free(start); Better use my_free(buff) here. > === modified file 'sql/rpl_mi.h' > --- sql/rpl_mi.h 2012-03-27 23:04:46 +0000 > +++ sql/rpl_mi.h 2012-10-01 09:55:23 +0000 > + char master_log_name[FN_REFLEN+6]; /* Place for multi-*/ "place for" -> "room for". > === modified file 'sql/mysqld.cc' > --- sql/mysqld.cc 2012-09-22 14:11:40 +0000 > +++ sql/mysqld.cc 2012-10-01 09:55:23 +0000 > + /* > + We must have LOCK_open before LOCK_global_system_variables because > + LOCK_open is hold while sql_plugin.c::intern_sys_var_ptr() is called. "is hold" -> "is held" > === modified file 'sql/item_create.cc' > --- sql/item_create.cc 2012-01-13 14:50:02 +0000 > +++ sql/item_create.cc 2012-10-01 09:55:23 +0000 > @@ -4393,27 +4393,36 @@ Create_func_master_pos_wait::create_nati > if (item_list != NULL) > arg_count= item_list->elements; > > + if (arg_count < 2 || arg_count >4) Missing space before "4"

2 1

[Maria-developers] Json Explain
by Peter Laursen 30 Sep '12

30 Sep '12

Is Explain output in Json format planned in MariaDB (refer http://glebshchepa.blogspot.in/2012/04/optimizer-new-explain-formatjson.html) ? If so when? and will the implementation details and output format be fully identical to Oracle/MySQL 5.6? (I did not find it mentioned here: http://monty-says.blogspot.dk/2012/09/now-is-good-time-to-be-part-of-future… http://blog.mariadb.org/mariadb-directions/ .. sorry if I overlooked it somewhere) -- Peter -- Webyog

4 8

Re: [Maria-developers] [Commits] Rev 3578: MDEV-495 backport --ignore-db-dir
by Sergei Golubchik 25 Sep '12

25 Sep '12

Hi, Holyfoot! On Sep 25, holyfoot(a)askmonty.org wrote: > revno: 3578 > revision-id: holyfoot(a)askmonty.org-20120925094024-yv0wpbl6k7b6bgg8 > parent: sanja(a)montyprogram.com-20120920094859-lcrxe8lr1mtg8h10 > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: mdev495 > timestamp: Tue 2012-09-25 14:40:24 +0500 > message: > MDEV-495 backport --ignore-db-dir. > The feature was backported from MySQL 5.6. > Some code was added to make commands as > SELECT * FROM ignored_db.t1; > CALL ignored_db.proc(); > USE ignored_db; > to take that option into account. Just a few comments: 1. add a test case for CREATE DATABASE, CREATE DATABASE ignored_db should fail. 2. I'd prefer to get an error "db not found" instead of "incorrect db name", is it difficult to do? 3. add a test for the following: lost+found is ignored, but CREATE DATABASE `lost+found` should succeed! (indeed, it creates a directory lost@002bfound, which is not ignored), and later SELECT, CALL, USE, SHOW DATABASES, should show that new db. > === added file 'mysql-test/r/ignore_db_dirs_basic.result' > --- a/mysql-test/r/ignore_db_dirs_basic.result 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/r/ignore_db_dirs_basic.result 2012-09-25 09:40:24 +0000 > @@ -0,0 +1,28 @@ > +select @@ignore_db_dirs; > +@@ignore_db_dirs > +e,lost+found,.mysqlgui,ignored_db > +# Check that SHOW DATABASES ignores all directories from > +# @@ignore_db_dirs and all directories with names starting > +# with '.' > +SHOW DATABASES; > +Database > +information_schema > +#mysql50#.otherdir > +mtr > +mysql > +test > +USE ignored_db; > +ERROR 42000: Incorrect database name 'ignored_db' > +SELECT * FROM ignored_db.t1; > +ERROR 42000: Incorrect database name 'ignored_db' > +CALL ignored_db.p1(); > +ERROR 42000: Incorrect database name 'ignored_db' > +SELECT COUNT(*) FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME='ignored_db'; > +COUNT(*) > +1 > +SET @@global.ignore_db_dirs = 'aha'; > +ERROR HY000: Variable 'ignore_db_dirs' is a read only variable > +SET @@local.ignore_db_dirs = 'aha'; > +ERROR HY000: Variable 'ignore_db_dirs' is a read only variable > +SET @@ignore_db_dirs = 'aha'; > +ERROR HY000: Variable 'ignore_db_dirs' is a read only variable When merging to 5.5 you'll need to move this test to sys_vars suite (otherwise sys_vars.all_vars will fail) > === modified file 'sql/sql_show.cc' > --- a/sql/sql_show.cc 2012-08-24 13:39:34 +0000 > +++ b/sql/sql_show.cc 2012-09-25 09:40:24 +0000 > @@ -379,6 +379,239 @@ bool mysqld_show_privileges(THD *thd) > } > > > +/** Hash of LEX_STRINGs used to search for ignored db directories. */ > +static HASH ignore_db_dirs_hash; > + > +/** > + An array of LEX_STRING pointers to collect the options at > + option parsing time. > +*/ > +static DYNAMIC_ARRAY ignore_db_dirs_array; I don't think that sql_show.cc is where all this code really belongs, but ok, as you like. Regards, Sergei

1 0

[Maria-developers] FYI: More ha_cassandra benchmark data
by Sergei Petrunia 23 Sep '12

23 Sep '12

Hi! For the record, posting some ha_cassandra benchmarks: <contents> Run #1: re-run previous benchmarks Run #2: evaluate max. network performance Run #3: Load two dbt3 databases at once Run #4: load two datasets into different nodes Results Conclusions </contents> === Run #1: re-run previous benchmarks === My results with - m1.large mysqld node (mysqld uses optimized binary) - Cassandra on 2*m1.large nodes - Loading data with the standard LOAD DATA INFILE command. - %CPU mysqld =14 sar -n DEV 10: 03:04:41 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:04:51 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 03:04:51 PM eth0 1575.20 156.60 81.47 4458.13 0.00 0.00 0.00 On cassandra node: - jsvc %CPU 70-170... 02:51:00 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 02:51:05 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:51:05 PM eth0 8325.20 7333.80 3376.25 1696.74 0.00 0.00 0.00 Total load time: time: 19 min. === Run #2: evaluate max. network performance === for comparison, sending /dev/zero on one host to /dev/null on another with nc: 03:39:24 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:39:34 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 03:39:34 PM eth0 16756.70 3688.50 851.91 87477.61 0.00 0.00 0.00 sending in both directions: SQL: 03:45:36 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:45:46 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 03:45:46 PM eth0 63308.20 4473.50 73002.80 81484.77 0.00 0.00 0.00 Cassandra: 03:47:32 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 03:47:42 PM lo 0.20 0.20 0.01 0.01 0.00 0.00 0.00 03:47:42 PM eth0 60741.50 15241.70 84110.07 70712.32 0.00 0.00 0.00 === Run #3: Load two dbt3 databases at once === - Loading two DBT-3 databases at once. - The databases have their own key spaces, and mysql tables - Still, both loaders connect to the same cassandra host. Cassandra host: 04:27:57 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 04:28:07 PM eth0 9979.10 8984.00 6006.77 2621.08 0.00 0.00 0.00 04:29:07 PM eth0 9181.80 7315.00 6416.25 2686.83 0.00 0.00 0.00 04:29:17 PM eth0 10277.00 9629.60 6047.00 2672.02 0.00 0.00 0.00 04:30:23 PM CPU %user %nice %system %iowait %steal %idle 04:30:33 PM all 34.56 24.80 24.71 0.18 0.05 15.69 04:30:53 PM all 32.08 11.20 24.25 1.31 0.08 31.07 04:31:03 PM all 35.59 22.55 22.28 0.27 0.00 19.31 it has periodical slow-downs like this: 04:32:38 PM all 53.25 1.30 19.05 3.03 0.00 23.38 04:32:39 PM all 65.84 10.40 21.78 0.00 0.00 1.98 04:32:40 PM all 37.14 33.81 18.10 0.95 0.00 10.00 04:32:41 PM all 42.65 25.98 26.96 0.00 0.00 4.41 04:32:42 PM all 33.17 39.42 19.23 0.00 0.00 8.17 04:32:43 PM all 36.89 36.89 19.90 0.00 0.00 6.31 04:32:44 PM all 39.81 35.55 13.74 0.00 0.00 10.90 04:32:45 PM all 34.45 33.97 22.01 0.00 0.00 9.57 04:32:46 PM all 35.58 36.06 21.15 0.00 0.00 7.21 04:32:47 PM all 31.05 35.16 15.07 0.00 0.00 18.72 04:32:48 PM all 41.31 26.29 20.66 0.00 0.00 11.74 04:32:49 PM all 39.02 32.20 25.37 0.00 0.00 3.41 04:32:50 PM all 26.64 34.50 13.97 0.00 0.00 24.89 04:32:51 PM all 56.93 27.72 12.87 0.00 0.00 2.48 04:32:52 PM all 71.14 15.42 11.44 0.00 0.00 1.99 04:32:53 PM all 64.22 8.82 23.04 0.00 0.00 3.92 04:32:54 PM all 76.73 1.98 19.31 0.00 0.00 1.98 04:32:55 PM all 61.24 0.00 30.14 0.96 0.00 7.66 04:32:56 PM all 29.20 0.00 32.40 0.00 0.00 38.40 04:32:57 PM all 30.29 0.00 29.88 0.00 0.00 39.83 04:32:58 PM all 23.19 0.00 20.29 0.00 0.36 56.16 04:32:59 PM all 23.34 0.00 16.38 0.00 0.35 59.93 04:33:00 PM all 20.07 0.00 21.90 0.00 0.00 58.03 04:33:01 PM all 24.20 0.00 18.86 0.00 0.00 56.94 04:33:02 PM all 20.07 0.00 22.18 0.00 0.70 57.04 Is this a compaction? mysqld host: 04:29:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 04:29:01 PM eth0 1416.20 247.90 75.43 6578.11 0.00 0.00 0.00 04:29:11 PM eth0 1175.00 199.60 62.49 5292.67 0.00 0.00 0.00 04:29:21 PM eth0 1074.60 186.90 57.21 4952.19 0.00 0.00 0.00 04:30:30 PM CPU %user %nice %system %iowait %steal %idle 04:30:40 PM all 7.77 0.00 0.56 0.00 0.05 91.62 04:30:50 PM all 8.20 0.00 0.60 0.00 0.30 90.89 04:31:00 PM all 8.01 0.00 0.66 0.00 0.05 91.28 Conclusions: - mysqld is still idle most of the time. - Cassandra is busier. Sometimes, it is 1014/Sep/12 12:50 PM0% busy. - network use betwen mysqld and cassandra is 2x more, which is expected Run time: 24 minutes for both (compare to 19 minutes) === Run #4: load two datasets into different nodes === ==== Network ==== SQL node: 05:44:21 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 05:44:41 PM eth0 1178.10 210.10 62.81 5510.03 0.00 0.00 0.00 05:44:51 PM eth0 1062.10 200.90 57.53 5152.63 0.00 0.00 0.00 05:45:01 PM eth0 965.80 194.00 52.53 5012.62 0.00 0.00 0.00 Cassandra1: 05:44:19 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 05:44:49 PM eth0 10159.00 8659.20 4443.26 1798.75 0.00 0.00 0.00 05:44:59 PM eth0 9018.20 7750.10 3982.21 1578.30 0.00 0.00 0.00 05:45:09 PM eth0 8741.30 7521.70 3921.69 1541.55 0.00 0.00 0.00 Cassandra2: 05:44:56 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 05:45:06 PM eth0 9305.60 7906.00 4121.10 1622.92 0.00 0.00 0.00 05:45:16 PM eth0 9186.50 7828.80 4258.12 1681.78 0.00 0.00 0.00 05:45:26 PM eth0 9675.70 8283.10 4306.89 1698.37 0.00 0.00 0.00 ==== CPU ==== SQL node: 05:46:45 PM CPU %user %nice %system %iowait %steal %idle 05:46:55 PM all 8.39 0.00 0.70 0.00 0.40 90.51 05:47:05 PM all 7.65 0.00 0.74 0.00 0.70 90.91 05:47:15 PM all 8.76 0.00 0.64 0.00 0.89 89.70 cassandra 1: 05:47:01 PM CPU %user %nice %system %iowait %steal %idle 05:47:11 PM all 27.28 28.24 27.55 0.00 0.05 16.89 05:47:21 PM all 31.45 24.44 27.56 1.38 0.05 15.13 05:47:31 PM all 29.46 19.60 23.19 2.35 0.04 25.36 05:47:41 PM all 38.47 0.04 14.21 0.15 0.07 47.06 05:47:51 PM all 31.04 9.92 17.97 0.39 0.08 40.61 05:48:01 PM all 24.73 28.07 21.77 0.26 0.04 25.12 05:48:11 PM all 19.79 11.67 19.23 1.83 0.07 47.41 05:48:21 PM all 20.29 0.00 11.07 0.06 0.13 68.44 05:48:31 PM all 24.08 0.00 11.96 0.13 0.13 63.69 Cassandra 2: 05:46:56 PM CPU %user %nice %system %iowait %steal %idle 05:47:06 PM all 31.97 27.14 26.91 0.09 0.05 13.85 05:47:16 PM all 32.79 20.43 23.39 2.34 0.04 21.01 05:47:26 PM all 26.43 27.07 27.29 0.14 0.09 18.98 05:47:36 PM all 17.72 8.59 12.00 2.97 0.21 58.52 05:47:46 PM all 13.63 0.00 6.77 0.06 0.30 79.23 05:47:56 PM all 27.19 3.92 8.52 0.11 0.25 60.01 05:48:06 PM all 19.94 27.83 18.47 0.13 0.08 33.54 05:48:16 PM all 13.69 16.01 14.24 2.69 0.15 53.22 05:48:26 PM all 18.13 0.00 9.57 0.25 0.38 71.66 === Results === The first client failed with ERROR 1928 (HY000) at line 145: Internal error: 'TimedOutException: Default TException.' real 4m3.463s user 0m0.008s sys 0m0.000s The second completed after: real 19m57.875s (which matches load time for loading one dataset). === Conclusions === - need to fix the Timeout error (filed as MDEV-535) - CPU time spent by mysqld is negligible now (I don't know whether this comes from using faster CPU for SQL node, or using a release build of mariadb, or both) - A confirmation of previous findings: if we use only one connection, the speed at which we can insert the data is very limited (see: with two connections, we have inserted 2x data in 20% more time. This is where we used two connections to connect to the *same* node) - I am not sure whether I should just start using multiple connections, or also automatically route the data (both writes and reads) to the right node? - need to check out Hector client. It seems to be able to connect to multiple nodes of Cassandra cluster. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Updates to mytop (from MariaDB 5.5 distribution)
by Michael Widenius 21 Sep '12

21 Sep '12

Hi! I have done some extensions to mytop as part of porting, fixing and enhancing it for MariaDB. I have now merged my changes with 1.91 and thought it was time to share the changes with you. Sorry for not doing it sooner! I also have added Jean Weisbuch patches for --full-queries and reading of .my.cnf Here is a diff of my changes. Hope you find some of them useful! If you have any questions, feel free to ask! Best regards, Monty Creator of MySQL and MariaDB

1 0

[Maria-developers] Cassandra SE: problems including it into release
by Sergei Petrunia 21 Sep '12

21 Sep '12

Hello, I've figured there may be an additional challenge in getting Cassandra storage engine into a *MariaDB release*. The problem is packaging. <contents> Cassandra SE packaging problems Need Thrift library Need recent gcc How did Sphinx SE manage to avoid a similar mess? Possible way out </contents> == Cassandra SE packaging problems == === Need recent gcc === C++ code generated by Thrift uses boost, and so one needs a recent gcc to compile it with MySQL. We have the same issue with oqgraph. See MDEV-501 for details. I am not aware of crashing problems that ograph+boost combination has. Still, recent-gcc requirement may limit the range of platforms we can build for. For example, I never managed to compile on Ubuntu 9.04. === Need Thrift library === Cassandra SE uses Thrift protocol to talk to server. In order to do this, it needs to link againist libthrift.so. There are no Thrift packages in either Fedora or Ubuntu (actually I haven't seen Thrift packages for any OS). Thrift homepage only offers source tarballs. The source tarball uses automake and compiles into 1. thrift compiler binary (we dont actually need it. Source code generated by thrift needs to be modified anyway.) 2. thrift libraries for a dozen of languages, including libthrift.so, which we do need. == How did Sphinx SE manage to avoid a similar mess? == I've asked myself a question how do we manage to include Sphinx SE without having to mess with Sphinx packages. The answer is: Sphinx SE doesn't need any sphinx libraries, either at compile time, or at runtime. It implements its client protocol fully, except for INSERT commands, for which it uses the available mysql client library to talk to Sphinx server using mysql client-server protocol. == An unlikely but possible way out == Cassandra 1.2 (the next version that is promised to be released "in Q4 2012" will introduce native network protocol. A week ago they've pushed first documentation draft. The native network protocol should allow to do everything that Thrift API allows, and more. There are no C++ clients yet, though. If I am faced with a choice betwen 1. implementing a relatively simple network protocol 2. Making Thrift packages for various platforms (and potentially convincing downstream maintainers to accept them) then I will definitely pick #1 (I have implemented network protocols in the past). There is a big risk, though: native protocol support is very new in Cassandra, they pushed it about a week ago. It was pushed into their "trunk" which is under development. I don't know if the code in their trunk is sufficiently stable for us to develop againist. BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

3 3

Re: [Maria-developers] New to MariaDB
by Giacomo Picchiarelli 21 Sep '12

21 Sep '12

Thanks! At beginning Il'll try to do some benchmarks! On 13/09/2012 22:11, Michael Widenius wrote: > Hi! > >>>>>> "Sergei" == Sergei Golubchik <serg(a)askmonty.org> writes: > Sergei> Hi, Giacomo! > Sergei> On Aug 11, Giacomo Picchiarelli wrote: >>> Hi, >>> >>> I'm new to MariaDB and I'm reading the knowledge base. >>> I'm interested in studying Aria Engine (and contribute, if I can :) ), >>> are there tasks for newbie developers? > Sergei> I'm afraid, there are no tasks in the Aria engine right now :( > > Sergei> If you're interested to look at other tasks - here is the list of task, > Sergei> feel free to pick anything from it: > > Sergei> https://mariadb.atlassian.net/secure/IssueNavigator.jspa?requestId=11300 > > Actually, there are some open bugs on Aria: > > https://mariadb.atlassian.net/browse/MDEV-333 > > You can go to: > https://mariadb.atlassian.net/secure/QuickSearch.jspa > And search for "Aria" > > Same goes for lanchpad: > > https://bugs.launchpad.net/maria?field.searchtext=aria&search=Search&field.… > > Note that many of the bugs in launchpad for Aria are already fixed. > However it would be quite valuable if you could help us go trough > them and mark which still are open and try to fix them yourself. > > Another thing that can be done on Aria is to run some benchmarks on it > and try to find ways to improve the performance on it. > > Regards, > Michael Widenius > -- Giacomo Picchiarelli Skype: g.picchiarelli

2 1

Re: [Maria-developers] [Commits] Rev 3577: MDEV-521 fix. in file:///home/bell/maria/bzr/work-maria-5.3-MDEV-521/
by Sergei Petrunia 20 Sep '12

20 Sep '12

Hi Sanja, Ok to push. On Thu, Sep 20, 2012 at 12:49:03PM +0300, sanja(a)montyprogram.com wrote: > At file:///home/bell/maria/bzr/work-maria-5.3-MDEV-521/ > > ------------------------------------------------------------ > revno: 3577 > revision-id: sanja(a)montyprogram.com-20120920094859-lcrxe8lr1mtg8h10 > parent: timour(a)askmonty.org-20120917081346-nqd08d0v27w8zp5c > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.3-MDEV-521 > timestamp: Thu 2012-09-20 12:48:59 +0300 > message: > MDEV-521 fix. > > After pullout item during single row subselect transformation it should be fixed properly. > === modified file 'mysql-test/r/subselect.result' > --- a/mysql-test/r/subselect.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect.result 2012-09-20 09:48:59 +0000 > @@ -6163,5 +6163,20 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > > === modified file 'mysql-test/r/subselect_no_mat.result' > --- a/mysql-test/r/subselect_no_mat.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_mat.result 2012-09-20 09:48:59 +0000 > @@ -6162,6 +6162,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set optimizer_switch=default; > > === modified file 'mysql-test/r/subselect_no_opts.result' > --- a/mysql-test/r/subselect_no_opts.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_opts.result 2012-09-20 09:48:59 +0000 > @@ -6158,6 +6158,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set @optimizer_switch_for_subselect_test=null; > > === modified file 'mysql-test/r/subselect_no_scache.result' > --- a/mysql-test/r/subselect_no_scache.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_scache.result 2012-09-20 09:48:59 +0000 > @@ -6169,6 +6169,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set optimizer_switch=default; > > === modified file 'mysql-test/r/subselect_no_semijoin.result' > --- a/mysql-test/r/subselect_no_semijoin.result 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/r/subselect_no_semijoin.result 2012-09-20 09:48:59 +0000 > @@ -6158,6 +6158,21 @@ a ( 3, 3 ) NOT IN ( SELECT NULL, NULL ) > 3 NULL > set optimizer_switch=@mdev367_optimizer_switch; > DROP TABLE t1; > +# > +# MDEV-521 single value subselect transformation problem > +# > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +f1 > +u1 > +u2 > +DROP TABLE t1; > # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > set @optimizer_switch_for_subselect_test=null; > > === modified file 'mysql-test/t/subselect.test' > --- a/mysql-test/t/subselect.test 2012-08-25 06:15:57 +0000 > +++ b/mysql-test/t/subselect.test 2012-09-20 09:48:59 +0000 > @@ -5229,5 +5229,18 @@ set optimizer_switch=@mdev367_optimizer_ > > DROP TABLE t1; > > +--echo # > +--echo # MDEV-521 single value subselect transformation problem > +--echo # > +CREATE TABLE t1 (f1 char(2), PRIMARY KEY (f1)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('u1'),('u2'); > + > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > +FLUSH TABLES; > +SELECT a.* FROM t1 a WHERE ( SELECT EXISTS ( SELECT 1 FROM t1 b WHERE b.f1 = a.f1 ) ); > + > +# Cleanup > +DROP TABLE t1; > + > --echo # return optimizer switch changed in the beginning of this test > set optimizer_switch=@subselect_tmp; > > === modified file 'sql/item_subselect.cc' > --- a/sql/item_subselect.cc 2012-09-17 08:13:46 +0000 > +++ b/sql/item_subselect.cc 2012-09-20 09:48:59 +0000 > @@ -1007,11 +1007,9 @@ Item_singlerow_subselect::select_transfo > } > substitution= select_lex->item_list.head(); > /* > - as far as we moved content to upper level, field which depend of > - 'upper' select is not really dependent => we remove this dependence > + as far as we moved content to upper level we have to fix dependences & Co > */ > - substitution->walk(&Item::remove_dependence_processor, 0, > - (uchar *) select_lex->outer_select()); > + substitution->fix_after_pullout(select_lex->outer_select(), &substitution); > } > DBUG_RETURN(false); > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0