- developers - lists.mariadb.org

[Maria-developers] SphinxSE pushed into 5.2
by Kristian Nielsen 09 Jul '10

09 Jul '10

Hi, I have pushed SphinxSE into MariaDB 5.2. It should be available in the upcoming MariaDB 5.2.2, which is currently planned within a few weeks. For now, only the code for the storage engine is pushed, not for the mysql-test-run.pl stuff. Sergei Golubchik wanted to modify mysql-test-run.pl first so that engine-specific code can be added in a more modular fashion (he will do it when he returns from vacation). For now, the engine is by default built as a loadable .so plugin. This is the general policy for MariaDB for new plugins. I think the idea is that engines that prove to be stable and well supported by their maintainer can get promoted to build statically by default (we are still flexing out the new policy). Note that the engine will still be available by default, the user will just need to do a one-time INSTALL PLUGIN command. Note also that SphinxSE can still be built statically with ./configure --with-plugin-sphinx, only the default is dynamic. Daniel, you should probably look into documenting the SphinxSE for MariaDB 5.2.2. Andrew kindly gave us permission to use what we need from the Sphinx documentation: http://sphinxsearch.com/docs/current.html#sphinxse Andrew suggested including also a link to the Sphinx webpage in the MariaDB documentation for SphinxSE, which is of course a very good idea. Also you should document how to do the INSTALL PLUGIN command (it's similar to for OQGraph). Note that the sphinx plugin will be included in eg. deb packages and so on and installed by default, just the INSTALL PLUGIN command is needed to enable it. Andrew, thanks for all your work with this! It is very good to have SphinxSE included, there have been several requests for this. We (mainly Sergei) made some changes to the code during review. I attach a patch for all the changes since your original import, in case you are interested. Note that I re-added the code when I merged to 5.2 to get a clean history, so future work on SphinxSE in MariaDB should be based on lp:maria/5.2, not our old sphinx bzr trees. - Kristian.

1 0

[Maria-developers] Get rid of most 32 bit Visual Studio compiler warnings
by Bo Thorsen 08 Jul '10

08 Jul '10

The attached patch reduces the warnings in a 32 bit build with Visual Studio 2008 to a set in the flex/bison generated code. I'll handle those later. I'll add comments on some of the changes here. The libmysqld.def file: The description line isn't supported in the more recent compilers. In fsp0fsp.c, there is an explicit cast to ullint. In this case, the code does what's intended, and the compiler warning is one of the "are you sure this is right" warnings. C4334 gives a warning if you make a 1u << 10 and store that in a 64 bit variable, because you could have meant 1i64 << 10. The change in i_s.cc is the one I'm most worried about. It used to store the unsigned long long in a double. The change I did can be wrong. But even if it's not, I'm worried what happens when this runs on an existing set of tables. We had a very long discussion about the cast in row0sel.c. The conclusion was that * auto_increment on doubles or floats is a very odd case * there is a compiler bug in Visual Studio for very large values (above around 2^53 it converts doubles wrong to uint64 I'd like to separate this discussion from this patch, though, and submit a bug report on the innodb code on mysql. I assume there will be something here I should change, but if parts of the patch are ready for pushing, let me know that. Bo Thorsen. Monty Program AB. -- MariaDB: MySQL replacement Community developed. Feature enhanced. Backward compatible.

1 1

[Maria-developers] How MySQL/MariaDB handles aborted DDL
by Arjen Lentz 07 Jul '10

07 Jul '10

Hi Philip, all Philip's bug on recovery from a kill -9 during DML reminded me... from my experience, aborting a DDL query (ALTER TABLE - things like adding an index usually) can be tricky and more often than not causes hassles. By abort in this case I mean killing the connection/thread that does the DDL. It would be great to somehow have a few test cases in the suite and it needs to be tested with all storage engines. Provided the table it operates on is big enough, it should be possible to just zap it by timing. Thoughts? Cheers, Arjen.

2 2

[Maria-developers] Sphinx and other storage engines documentation
by Timour Katchaounov 07 Jul '10

07 Jul '10

Danny, this the log from a short discussion about how to provide the documentation for Sphinx to our users. Can you think of some easy way to automate at least the notification process that the docs changed? <timour> knielsen, shodan IMHO it wuold be best if Sphinx could notify us automatically about changes in the docs, rather than having a possibly outdated documentation, with a link to the up-to-date docs. <timour> If we take this manual approach to copy/paste docs for a dozen storage engines, we will end up in a big mess. <shodan> timour: would http://code.google.com/feeds/p/sphinxsearch/svnchanges/basic do for notifications? <timour> shodan, isn't that a feed with all changes? <shodan> timour: yes, and so doc/ changes are there too <timour> knielsen, perhaps we could pull the docs automatically? <shodan> timour: maybe we could setup a special hook on commits to just doc/ but def not to a specific section <timour> shodan, I think we have to ask our docs guy what is the best way to at least be automatically notified of doc changes (for the same release). <timour> To me there are 2 kinds of doc changes: <timour> - improvements of the docs for the same version of the code, <timour> - changes due to a new release. <timour> The latter will be straightforward, because we will know when we release a new version of Sphinx, but not the former. What do you think about the problem with outdated docs of storage engines? Timour

2 1

[Maria-developers] Compile for 64 bit
by Bo Thorsen 07 Jul '10

07 Jul '10

Hi all, The attached patch adds a -64 keyword to the script make_mariadb_win_dist. With this patch, the script produces both 32 bit zip files (without any arguments) and 64 bit zip files (with -64). I also expanded it so it's possible to run it with "-nobuild -64" and a help text. Kristian, when this patch is pushed, you can add a new buildbot slave based on the one that builds the 32 bit zip file and installer. Adding the -64 argument on the call to this script is the only thing necessary for buildbot to produce 64 bit binaries. The result files are named -win64 instead of -win32. Ok to push this to 5.1? Bo Thorsen. Monty Program AB. -- MariaDB: MySQL replacement Community developed. Feature enhanced. Backward compatible.

2 4

[Maria-developers] Ideas for improving MariaDB/MySQL replication
by Kristian Nielsen 05 Jul '10

05 Jul '10

The three companies Continuent, Codership, and Monty Program are planning to start working on some enhancements to the replication system in MariaDB, together with anyone interested in joining in. At this stage, there are no fixed directions for the project, and to do this in as open a way possible with the maximum community involvement and interest, we agreed to start with an email discussion on the maria-developers@ mailing list. So consider it started! The plan so far is: 1) The parties taking this initiative, MP, Continuent, and Codership, present their own ideas in this thread on maria-developers@ (and everyone else who wants to chime in at this stage). 2) Once we have some concrete suggestions as a starting point, we use this to reach out in a broader way with some blog posts on planetmysql / planetmariadb to encourage further input and discussions for possible directions of the project. Eventually we want to end up with a list of the most important goals and a possible roadmap for replication enhancements. (It is best to have something concrete as a basis of a broad community discussion/process). To start of, here are some points of interest that I collected. Everyone please chime in with your own additional points, as well as comments and further details on these one. Three areas in particular seem to be of high interest in the community currently (not excluding any other areas): - High Availability * Most seems to focus on building HA solutions on top of MySQL replication, eg. MMM and Tungsten. * For this project, seems mainly to be to implement improvements to replication that help facilitate improving these on-top HA solutions. * Tools to automate (or help automate) failover from a master. * Better facilities to do initial setup of new slave without downtime, or re-sync of an old master or slave that has been outside of the replication topology for some period of time. - Performance, especially scalability * Multi-threaded slave SQL thread. * Store the binlog inside a transactional engine (eg. InnoDB) to reduce I/O, solve problems like group commit, and simplify crash recovery. - More pluggable replication * Make the replication code and APIs be more suitable for people to build extra functionality into or on top of the stock MySQL replication. * Better documentation of C++ APIs and binlog format. * Adding extra information to binlog that can be useful for non-standard replication stuff. For example column names (for RBR), checksums. * Refactoring the server code to be more modular with APIs more suitable for external usage. * Add support for replication plugins, types to be determined. For example binlog filtering plugins? It is also very important to consider the work that the replication team at MySQL is doing (and has done). I found a good deal of interesting information about this here: http://forge.mysql.com/wiki/MySQL_Replication:_Walk-through_of_the_new_5.1_…) This describes a number of 6.0/5.4 and preview features that we could merge and/or contribute to. Here are the highlights that I found: - Features included in 6.0/5.4 (which are cancelled I think, but presumably this will go in a milestone release): * CHANGE MASTER ... IGNORE_SERVER_IDS for better support of circular replication. * Replication heartbeat. * sync_relay_log_info, sync_master_info, sync_relay_log, relay_log_recovery for crash recovery on slave. * Binlog Performance Optimization (lock contention improvement). * Semi-synchronous Replication, with Pluggable Replication Architecture. http://forge.mysql.com/wiki/ReplicationFeatures/SemiSyncReplication - Feature previews: * Parallel slave application: WL#4648 http://forge.mysql.com/wiki/ReplicationFeatures/ParallelSlave * Time-delayed replication: WL#344 http://forge.mysql.com/wiki/ReplicationFeatures/DelayedReplication * Scriptable Replication: WL#4008 http://forge.mysql.com/wiki/ReplicationFeatures/ScriptableReplication * Synchronous Replication. Drizzle is also doing work on a new replication system. I read through the series of blog posts that Jay Pipes wrote on this subject. They mostly deal with how this is designed in terms of the Drizzle server code, and is low on detail about how the replication will actually work (the only thing I could really extract was that it is a form of row-based replication). If someone has links to information about this that I missed, it could be interesting. Let the discussion begin! - Kristian.

13 67

[Maria-developers] MWL#123, sql layer part (was: Re: mwl#121: follow up)
by Sergey Petrunya 05 Jul '10

05 Jul '10

Hello Igor, On Thu, Jul 01, 2010 at 11:55:58PM -0700, Igor Babaev wrote: > According to our agreement I introduced a new flag for MRR. > I called it HA_MRR_MATERIALIZED_KEYS. This flag passed to any MMR > interface function that takes mrr_mode as a parameter says: > key values used in ranges are materialized in some buffers external to MRR. > This patch only gives DS-MRR information that the keys are materialized. However, DS-MRR needs to work on (key, range_id) pairs. The patch doesn't allow to assoicate key with range_id (or vice versa), so DS-MRR will have to keep (key_pointer, range_id) tuples. If we consider 64-bit environment, then sizeof(key_value_pointer)== sizeof(key_value), and this patch won't bring any benefit at all. I was expecting that the patch will provide some means to associate key_value_pointer with range_id, so that DS-MRR won't need to store both. Can we discuss this on scrum meeting or Monday evening? > === modified file 'sql/handler.h' > --- sql/handler.h 2010-03-20 12:01:47 +0000 > +++ sql/handler.h 2010-07-01 20:00:35 +0000 > @@ -1212,6 +1212,12 @@ void get_sweep_read_cost(TABLE *table, h > */ > #define HA_MRR_NO_NULL_ENDPOINTS 128 > > +/* > + The MRR user has materialized range keys somewhere in the user's buffer. > + This can be used for optimization of the procedure that sorts these keys > + since in this case key values don't have to be copied into the MRR buffer. > +*/ > +#define HA_MRR_MATERIALIZED_KEYS 256 > > > /* > > === modified file 'sql/sql_join_cache.cc' > --- sql/sql_join_cache.cc 2010-03-07 15:41:45 +0000 > +++ sql/sql_join_cache.cc 2010-07-01 19:59:55 +0000 > @@ -651,6 +651,9 @@ int JOIN_CACHE_BKA::init() > > use_emb_key= check_emb_key_usage(); > > + if (use_emb_key) > + mrr_mode|= HA_MRR_MATERIALIZED_KEYS; > + > create_remaining_fields(FALSE); > > set_constants(); > @@ -2617,6 +2620,8 @@ int JOIN_CACHE_BKA_UNIQUE::init() > data_fields_offset+= copy->length; > } > > + mrr_mode|= HA_MRR_MATERIALIZED_KEYS; > + > DBUG_RETURN(rc); > } > > BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] [5.3 merge] Item_in_subselect::init_left_expr_cache() question.
by Sergey Petrunya 03 Jul '10

03 Jul '10

Hi Timour, I'm having difficulties with finishing 5.2->5.3 merge. Could you please take a look at https://bugs.launchpad.net/maria/+bug/598972 ? The questions are in the bug entry. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 1

[Maria-developers] DS-MRR: extra work filed as new WL entries, questions
by Sergey Petrunya 03 Jul '10

03 Jul '10

Hello Igor, Based on our discussions, I've filed * http://askmonty.org/worklog/Server-RawIdeaBin/index.pl?tid=123 "DS-MRR for clustered PKs: more efficient buffer use" * http://askmonty.org/worklog/Server-RawIdeaBin/index.pl?tid=124 "DS-MRR for clustered PKs: cost function" * http://askmonty.org/worklog/Client-BackLog/index.pl?tid=125 "Make DS-MRR sort the ranges before scanning the index" Could you please check if that correctly describes the conclusions we've arrived at? Also, WL texts contain several unresolved questions, marked with "TODO". BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] WL#122 New (by Sergei): fix locking in XA tc_log
by worklog-noreply＠askmonty.org 30 Jun '10

30 Jun '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: fix locking in XA tc_log CREATION DATE..: Wed, 30 Jun 2010, 17:26 SUPERVISOR.....: Sergei IMPLEMENTOR....: Sergei COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 122 (http://askmonty.org/worklog/?tid=122) VERSION........: Server-5.1 STATUS.........: Assigned PRIORITY.......: 90 WORKED HOURS...: 0 ESTIMATE.......: 8 (hours remain) ORIG. ESTIMATE.: 8 PROGRESS NOTES: DESCRIPTION: We need to analyze and fix the mutex lock order in TC_LOG code of the XA. It's needed to fix the https://bugs.launchpad.net/maria/+bug/578117 ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0