- developers - lists.mariadb.org

[Maria-developers] Removing very old options for --old, --new, --safe-mode
by Michael Widenius 23 Sep '10

23 Sep '10

Hi! I did notice that we haven't cleaned up the usage of the --old, --new, --safe-mode and --skip-new options in a long time. Here is a cleanup of this for MariaDB 5.2 (soon to be made gamma). After this suggested change, here is how the options would work: --new Use new functionality that will exist in next version of MariaDB. This function exist to make it easier to prepare for an upgrade. For version 5.1 this functions enables the LIST and RANGE partitions functions for ndbcluster. --old Use compatible behavior with previous main version for some functionality. For MariaDB 5.1 this means that we are using the old, MySQL 5.1 compatible, way to calculate checksums for records. If you are using --old, CHECKSUM TABLE will always do a full table scan. --safe_mode Disable some potential unsafe optimization. For 5.2 these are: INSERT DELAYED is disabled, myisam_recover_options is set to DEFAULT (automatically recover crashed MyISAM files). For Aria table, disable bulk insert optimization to enable one to use maria_read_log to recover tables even if tables are deleted (good for testing recovery). --skip-new Skip some new potentially unsafe functions. For 5.2 these are: INSERT DELAYED is disabled, Query cache size is reset. Here is the patch against MariaDB 5.2: === modified file 'sql/field.cc' --- sql/field.cc 2010-08-05 19:56:11 +0000 +++ sql/field.cc 2010-09-14 15:45:01 +0000 @@ -6703,8 +6703,7 @@ void Field_string::sql_type(String &res) length= cs->cset->snprintf(cs,(char*) res.ptr(), res.alloced_length(), "%s(%d)", - ((type() == MYSQL_TYPE_VAR_STRING && - !thd->variables.new_mode) ? + (type() == MYSQL_TYPE_VAR_STRING ? (has_charset() ? "varchar" : "varbinary") : (has_charset() ? "char" : "binary")), (int) field_length / charset()->mbmaxlen); === modified file 'sql/item_cmpfunc.cc' --- sql/item_cmpfunc.cc 2010-08-05 19:56:11 +0000 +++ sql/item_cmpfunc.cc 2010-09-14 16:41:01 +0000 @@ -4683,8 +4683,7 @@ bool Item_func_like::fix_fields(THD *thd We could also do boyer-more for non-const items, but as we would have to recompute the tables for each row it's not worth it. */ - if (args[1]->const_item() && !use_strnxfrm(collation.collation) && - !(specialflag & SPECIAL_NO_NEW_FUNC)) + if (args[1]->const_item() && !use_strnxfrm(collation.collation)) { String* res2 = args[1]->val_str(&cmp.value2); if (!res2) === modified file 'sql/mysqld.cc' --- sql/mysqld.cc 2010-08-24 22:44:50 +0000 +++ sql/mysqld.cc 2010-09-14 16:40:09 +0000 @@ -8725,11 +8725,7 @@ mysqld_get_one_option(int optid, case (int) OPT_SKIP_NEW: opt_specialflag|= SPECIAL_NO_NEW_FUNC; delay_key_write_options= (uint) DELAY_KEY_WRITE_NONE; - myisam_concurrent_insert=0; - myisam_recover_options= HA_RECOVER_NONE; - sp_automatic_privileges=0; - my_use_symdir=0; - ha_open_options&= ~(HA_OPEN_ABORT_IF_CRASHED | HA_OPEN_DELAY_KEY_WRITE); + ha_open_options&= ~(HA_OPEN_DELAY_KEY_WRITE); #ifdef HAVE_QUERY_CACHE query_cache_size=0; #endif === modified file 'sql/records.cc' --- sql/records.cc 2010-07-17 19:58:08 +0000 +++ sql/records.cc 2010-09-14 16:40:09 +0000 @@ -216,7 +216,6 @@ bool init_read_record(READ_RECORD *info, */ if (!disable_rr_cache && !table->sort.addon_field && - ! (specialflag & SPECIAL_SAFE_MODE) && thd->variables.read_rnd_buff_size && !(table->file->ha_table_flags() & HA_FAST_KEY_READ) && (table->db_stat & HA_READ_ONLY || === modified file 'sql/sql_delete.cc' --- sql/sql_delete.cc 2010-08-05 19:56:11 +0000 +++ sql/sql_delete.cc 2010-09-14 16:40:10 +0000 @@ -130,7 +130,6 @@ bool mysql_delete(THD *thd, TABLE_LIST * - there should be no delete triggers associated with the table. */ if (!using_limit && const_cond_result && - !(specialflag & (SPECIAL_NO_NEW_FUNC | SPECIAL_SAFE_MODE)) && (thd->lex->sql_command == SQLCOM_TRUNCATE || (!thd->current_stmt_binlog_row_based && !(table->triggers && table->triggers->has_delete_triggers())))) === modified file 'sql/sql_parse.cc' --- sql/sql_parse.cc 2010-08-05 19:56:11 +0000 +++ sql/sql_parse.cc 2010-09-14 16:40:09 +0000 @@ -3086,9 +3086,7 @@ end_with_restore_list: goto error; /* purecov: inspected */ thd->enable_slow_log= opt_log_slow_admin_statements; thd->query_plan_flags|= QPLAN_ADMIN; - res= (specialflag & (SPECIAL_SAFE_MODE | SPECIAL_NO_NEW_FUNC)) ? - mysql_recreate_table(thd, first_table) : - mysql_optimize_table(thd, first_table, &lex->check_opt); + res= mysql_optimize_table(thd, first_table, &lex->check_opt); /* ! we write after unlocking the table */ if (!res && !lex->no_write_to_binlog) { === modified file 'sql/sql_select.cc' --- sql/sql_select.cc 2010-08-24 22:44:50 +0000 +++ sql/sql_select.cc 2010-09-14 16:39:57 +0000 @@ -7187,8 +7187,6 @@ eq_ref_table(JOIN *join, ORDER *start_or static bool only_eq_ref_tables(JOIN *join,ORDER *order,table_map tables) { - if (specialflag & SPECIAL_SAFE_MODE) - return 0; // skip this optimize /* purecov: inspected */ tables&= ~PSEUDO_TABLE_BITS; for (JOIN_TAB **tab=join->map2table ; tables ; tab++, tables>>=1) { ------------- Any comments ? Regards, Monty

2 2

Re: [Maria-developers] [Commits] Rev 2939: Fixed bug #53161. in file:///home/igor/maria/maria-5.1-bug53161/
by Oleksandr Byelkin 23 Sep '10

23 Sep '10

Hi! On 21.09.2010 07:22, igor(a)askmonty.org wrote: > At file:///home/igor/maria/maria-5.1-bug53161/ > > ------------------------------------------------------------ > revno: 2939 > revision-id: igor(a)askmonty.org-20100921042200-o52vwhnkz6kuwrcn > parent: monty(a)askmonty.org-20100915130103-x6i3x2mjc5gxsu77 > committer: Igor Babaev<igor(a)askmonty.org> > branch nick: maria-5.1-bug53161 > timestamp: Mon 2010-09-20 21:22:00 -0700 > message: > Fixed bug #53161. > The implementation of the virtual method not_null_tables for the class > Item_outer_ref must always return 0. > Patch looks OK, thanks.

1 0

[Maria-developers] DS-MRR improvements patch r2 ready for review
by Sergey Petrunya 21 Sep '10

21 Sep '10

Hello Igor, Please find attached the combined patch that addresses all of the review feedback. The tree is in launchpad and buildbot also: https://code.launchpad.net/~maria-captains/maria/5.3-dsmrr-cpk BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] WL#149 New (by Dreas): Add inet6_ntop() and inet6_pton()
by worklog-noreply＠askmonty.org 20 Sep '10

20 Sep '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add inet6_ntop() and inet6_pton() CREATION DATE..: Mon, 20 Sep 2010, 13:38 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 149 (http://askmonty.org/worklog/?tid=149) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: It would be nice if this is supported by MariaDB 5.1. UDFs are available at http://labs.watchmouse.com/2009/10/extending-mysql-5-with-ipv6-functions/ ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] online alter table in MariaDB?
by Zardosht Kasheff 19 Sep '10

19 Sep '10

Hello all, I saw the following on planet mysql today, http://www.facebook.com/note.php?note_id=430801045932, and it got me thinking. I remember at the 2010 storage engine summit, we discussed having more types of alter table be done online. Here are the meeting notes on that topic: Smarter "ALTER TABLE" Comment: Need to fix ALTER TABLE to fully support online operations. Jonas: This is already implemented, but currently only in the -telco cluster trees. Antony is willing to look into backporting new ALTER TABLE to MariaDB Does anyone know if any investigation has been done? I am curious to learn about people's findings. I have done a little poking at the mysql cluster code that has this feature, but I do not understand it well enough to know integration risk and complexity. Does anyone have any thoughts? I am curious to learn more. Thanks -Zardosht

2 1

[Maria-developers] Phone home
by Sergei Golubchik 19 Sep '10

19 Sep '10

Hi. So, "Phone Home" or "MySQL feedback daemon" or "better name wanted" feature. It is something that can be installed together with MariaDB, it will gather different statistic about how MariaDB is used and will send this information anonymously to mariadb.org. Not unlike the Uptimes Project or Debian Popularity Contest. The complete specs will be here: http://askmonty.org/worklog/Server-Sprint/?tid=12 There are basically four questions I'm thinking on. 1. Should that be a MariaDB plugin or a separate executable ? I tend to prefer a separate executable. There is no need to keep it in memory constantly - cron job can do. Being separate its bugs won't affect the server. Being separate one instance can monitor many MariaDB servers. It can be upgraded separately - and it's not tied to the server release schedule. The drawback - it won't be able to grab MariaDB internals easily, which means it may not report some data that are worth reporting. But to solve this we can add an I_S table that provides this information. This way there's no "hidden" data to report, everything is available from the SQL. Which is good :) 2. How to send the data. We'll use HTTP. Seems to be the most universally working transport. That's what other projects are using too - Uptimes Project uses UDP or HTTP, Debian Popularity Contest - SMTP or HTTP. We *may* want to add SMTP later, if needed. 3. Auditing. How can we prove to paranoid users that we only send what we are saying we send, and none of potentially private information. Possible solution: http sending should support a proxy (to work behind firewalls), so one can install a logging proxy and record all the data sent. On the other hand, we'd like to use SSL too. We can support, besides direct http, a "wget mode" where the data are sent by invoking wget (which supports proxies, SSL and --post-file) and one could easily replace wget with a simple script that logs all the data. 4. What to report. That's the most interesting part :) note that not everything from below is collected in MariaDB now, but I describe the ideal case, what would be useful to know to steer MariaDB development in the right direction. The principle I used was not "let's grab as much as we can" but "on a need-to-know basis". For example, we may need to decide whether to optimize huge IN (...) lists or GIS first. Knowing what is used more often would help to make a correct decision. hardware: CPU, RAM OS (linux distribution, kernel) mariadb version, memory usage parts of config (e.g. buffer sizes) list of installed plugins number of databases, max/avg number of tables in a database, max/avg db/table size uptime something that indicates the load, e.g. average qps how much a particular feature is used: Com_ counters from SHOW STATUS plugin usage counters per feature, like GIS, replication, etc. per query parts, like ORDER BY, subquery in the FROM, IN subquery ... how useful is query cache (hit ratio?) What else ? Regards, Sergei

2 5

Re: [Maria-developers] Seeking command syntax feedback (Still Re: Per-db binlogging)
by Weldon Whipple 16 Sep '10

16 Sep '10

On Wed, Sep 15, 2010 at 2:00 PM, Sergei Golubchik <serg(a)askmonty.org> wrote: > Hi, Weldon! Hi, Sergei! > > On Sep 15, Weldon Whipple wrote: <snip/> >> 2. FLUSH TABLES WITH READ LOCK FOR DATABASE dbname; >> >> (I just added an optional 'FOR DATABASE dbname' to the already >> existing command--which still works, BTW. Is this a misuse of the word >> "FOR"?) >> This is the command that locks the tables before someone (some >> program) dumps/copies the tables that will be moved to another server. >> It flushes and locks only one database. > > This would be difficult to implement. > > FLUSH TABLES WITH READ LOCK is not implemented in a way that can be > generalized to support per-database locks :( I had guessed as much. (The method in my DB_BINLOG_MGR method that implements that functionality consists of a mutex lock, a comment about some sort of miracle happening, then a mutex unlock.) I wonder if there might be a way of inhibiting some user database (temporarily) from changing ... Alternately, I suppose we could do a global FLUSH TABLE WITH READ LOCK/ UNLOCK TABLES. Or ... (???) > Besides, I personally wouldn't worry much about the syntax. It's > something you can easily change anytime - e.g. replace BINLOG with > BINLOGGING or FOR with something else. Yeah, every time I write a new prototype I tweak the syntax a bit. At the moment I'm just trying to settle on an implementation that would work for our company (and still be of some sort of general use). > > Regards, > Sergei > Thanks, Weldon

2 1

Re: [Maria-developers] [Maria-discuss] Per-db binlogging
by Henrik Ingo 15 Sep '10

15 Sep '10

On Tue, Sep 14, 2010 at 3:02 AM, MARK CALLAGHAN <mdcallag(a)gmail.com> wrote: > +maria-discuss as they are working in replication too To be exact, maria-developers@ is the counterpart to internals@mysql, although everyone should be reading maria-discuss@ too. I'm now leaving both in CC. > On Mon, Sep 13, 2010 at 1:52 PM, Weldon Whipple <weldon(a)whipple.org> wrote: >> Greetings! >> >> My employer has asked me to implement per-DB binlogging for >> MySQL. I've been working on prototypes for 3-4 weeks now, and decided >> it's time to post to this list to ask for comments, suggestions, >> condolences, etc. (I've been a "lurker" for quite awhile. I hope this >> post isn't too far off the mark) The assignment is *only* for master-side >> binlogging. (It doesn't require a slave/replication-side implementation.) >> >> >> Why I Want to Do It. >> ------------------- >> >> I work for an ISP that hosts (last I heard--I HOPE I'm not lying) over >> 2 million domains. Customers are allowed to have (almost) unlimited >> MySQL databases. Migrating an account from one physical server to >> another is a daunting task. (A typical server has easily 2000+ >> [sometimes far more] databases.) Global mysqld binlogging isn't >> feasible when we want to migrate (for example) a single account to >> another server. >> >> Our proposed scenario goes something like this: <cut> Do I understand correctly, that in the normal case you do not run with binlog on at all? You would only turn on this kind of per-database binlog when needed? If that would be the case, then I don't see why --binlog-do-db wouldn't already do exactly what you want, except that you'd require a restart to change it. So all you need to implement would be the ability to set it as a system variable and not just in the conf file. (http://dev.mysql.com/doc/refman/5.1/en/replication-options-binary-log.html#…) Alternatively, if you are also running a "global" binlog, then as Mark suggested, don't write additional per database binlogs, rather get your information from the "global" binlog and filter out what should be replicated instead by using --replicate-do-db. Again, it is not currently possible to set this on/off without restart. (http://dev.mysql.com/doc/refman/5.1/en/replication-options-slave.html#optio…) Finally, for anyone wishing to implement their own binlogging and replication, I should advertise our work in MariaDB on a generic "replication plugin API". Using this you could implement your own replication plugin that would do precisely what you want, but you could still leave the original replication code untouched. (You could of course "fork" it to make your own plugin if you wanted.) Kristian Nielsen in CC is working on that, and you can read about it here: http://askmonty.org/wiki/ReplicationProject http://askmonty.org/worklog/?tid=107 http://askmonty.org/worklog/?tid=120 henrik PS: It is not exactly the same use case, but also see mine and Igor's presentation from last years MySQL user conference. http://assets.en.oreilly.com/1/event/36/Valuable%20MariaDB%20Features%20Tha… ...from slide 10 onwards. Summary: You can use mysqlbinlog instead of replication and this allows all kinds of scriptable hacks to occur between master and slave. henrik -- henrik.ingo(a)avoinelama.fi +358-40-5697354 www.openlife.cc

2 1

[Maria-developers] Why delete stuff from MWL#39???
by Henrik Ingo 15 Sep '10

15 Sep '10

Regarding http://askmonty.org/worklog/Client-RawIdeaBin/?tid=39 ...user "Guest" in June has deleted all the dependencies on a group of mysqlbinlog tasks. Why? MWL#39 existed to act as an entry point to some interrelated mysqlbinlog tasks. Now there is no simple way for me to look the up all at once. *grumble* henrik -- henrik.ingo(a)avoinelama.fi +358-40-5697354 www.openlife.cc

1 0

[Maria-developers] ./configure report
by Sergei Golubchik 15 Sep '10

15 Sep '10

Hi. For a while, I had a small personal gripe that it is not easy to see what plugins are configured to be built and how. That is, I know where to look (configure prints all that when working, but not alphabetically, so a particular plugin name isn't easy to spot) but I had to explain this to other often enough to realize that it's not at all obvious. The configure script prints at the end the small configuration report. I've extended it from --< cut >------------------------------------ config.status: executing my_config.h commands You can find information about MariaDB at http://askmonty.org/wiki/index.php/MariaDB Remember to check the platform specific part of the reference manual for hints about installing MariaDB on your platform. Also have a look at the files in the Docs directory. --- Configuration summary for MariaDB Server version 5.1.50-MariaDB * Installation prefix: /usr/local * System type: unknown-linux-gnu * Host CPU: x86_64 * C Compiler: gcc (Gentoo 4.4.3-r2 p1.2) 4.4.3 * C++ Compiler: g++ (Gentoo 4.4.3-r2 p1.2) 4.4.3 * Debug enabled: no * Community Features: yes --- Thank you for choosing MariaDB! --< cut >------------------------------------ to include all plugins in alphabetical order (and moved informational text down to improve its visibility, as the plugin list is long): --< cut >------------------------------------ config.status: executing my_config.h commands --- Configuration summary for MariaDB Server version 5.2.2-MariaDB-gamma * Archive Storage Engine: plugin * Aria Storage Engine: yes * Blackhole Storage Engine: plugin * CSV Storage Engine: yes * Cluster Storage Engine: no * Collection of Authentication Plugins: plugin * Daemon Example Plugin: plugin * Example Storage Engine: plugin * Federated Storage Engine: plugin * FederatedX Storage Engine: plugin * Graph Storage Engine: plugin * IBM DB2 for i Storage Engine: no * InnoDB Storage Engine: plugin * Memory Storage Engine: yes * MyISAM MERGE Engine: yes * MyISAM Storage Engine: yes * PBXT Storage Engine: no * Partition Support: no * Simple Parser: plugin * Sphinx Storage Engine: plugin * XtraDB Storage Engine: plugin * Installation prefix: /usr/local * System type: unknown-linux-gnu * Host CPU: x86_64 * C Compiler: gcc (Gentoo 4.4.3-r2 p1.2) 4.4.3 * C++ Compiler: g++ (Gentoo 4.4.3-r2 p1.2) 4.4.3 * Debug enabled: no * Community Features: yes --- You can find information about MariaDB at http://askmonty.org/wiki/index.php/MariaDB Remember to check the platform specific part of the reference manual for hints about installing MariaDB on your platform. Also have a look at the files in the Docs directory. Thank you for choosing MariaDB! --< cut >------------------------------------ So, what do you think? Too long to be useful? Or shall I push ? Need to reorder sections? (yes, it's a bikeshed color question, I know) Regards, Sergei

2 1