July 2013 - developers - lists.mariadb.org

[Maria-developers] Extendet FederatedX storage engine to support updating Sphinx RT indexes
by Markus Lidel 11 Jun '14

11 Jun '14

Hello, the Sphinx Search engine has RT indexes, which could be updated with a subset of SQL commands, refered to as SphinxQL: http://sphinxsearch.com/docs/2.1.1/sphinxql-reference.html Because the FederatedX storage engine uses some commands, which Sphinx does not know, here is a patch, wich extends the FederatedX storage engine to support this subset of commands. Now its possible to update Sphinx RT indexes directly from MariaDB. To create a connection to the RT index in the Sphinx default RT-index: index rt { type = rt path = @CONFDIR@/data/rt rt_field = title rt_field = content rt_attr_string = title rt_attr_string = content rt_attr_uint = gid } you just have to call: CREATE TABLE `rt` ( `id` BIGINT UNSIGNED NOT NULL, `title` TEXT, `content` TEXT, `gid` INT UNSIGNED ) ENGINE=FEDERATED CONNECTION='sphinxql://root@localhost:9306/rt'; It would be great, if the attached patch could be integrated into MariaDB. If you have any suggestions, please feel free to contact me. Best regards, Markus Lidel

4 8

[Maria-developers] Flashback Feature patch review
by Lixun Peng 22 Aug '13

22 Aug '13

Hi Monty, I patched my flashback feature to 10.0-base, and this is the diff file, please help me to review it. How to use this feature? I added several new arguments for mysqlbinlog: --flashback (-B), it will let mysqlbinlog working on FLASHBACK mode. --table (-T), it likes -d, but this argument is for table. --review, it will le mysqlbinlog print the SQL for reviewing. Reviewing feature will create a new "review" table to record the data that will be changed by FLASHBACK feature. If you aren't special the --review-dbname/tablename, "review" table will created on current DB, and the table name will be "__"+original_table_name. And for getting the original table struct, mysqlbinlog need to connect mysql, so --user/--host/--password is needed if necessary. --review-dbname, the DB that you want to store the review table. --review-tablename the TABLE that you want to store the changed data. Only if you set -T, this argument is useful. For example: I have a table "t" in database "test", CREATE TABLE `t` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `info` varchar(32) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=9043880 DEFAULT CHARSET=utf8; And I done this operation, root@localhost : (none) 14:52:50> reset master; root@localhost : (none) 14:53:02> use test root@localhost : test 14:53:08> delete from t limit 10; root@localhost : test 14:53:25> insert into t (info) select "你好"; root@localhost : test 14:54:01> update t set info ='hello你好' limit 5; We can compare the output with --flashback or not. #client/mysqlbinlog /data/mysqldata_10.0/binlog/mysql-bin.000001 -vv -d test -T t --start-datetime="2013-03-27 14:54:00" > /tmp/1.sql #client/mysqlbinlog /data/mysqldata_10.0/binlog/mysql-bin.000001 -vv -uroot -d test -T t --start-datetime="2013-03-27 14:54:00" -B --review > /tmp/2.sql How to implement Flashback: 1. mysqlbinlog tool can parse the binlog files and print them, so we can use the most of code in mysqlbinlog. 2. Changing the event type (INSERT -> DELETE, DELETE -> INSERT) 3. Exchanging the SET/WHERE part in UPDATE (UPDATE table SET x=1 WHERE y=2 -> UPDATE table SET y=2 WHERE x=1) 4. Invert the event output, from end to beginning (binlog: e1,e2,e3... -> flashback_log: en...e3,e2,e1) Then, importing the flashback output can flashback database/table to the special time. But binlog_format should be ROW. Thanks, Lixun -- Senior MySQL Developer @ Taobao.com Mobile Phone: +86 18658156856 (Hangzhou) Gtalk: penglixun(at)gmail.com Twitter: http://www.twitter.com/plinux Blog: http://www.penglixun.com

2 3

[Maria-developers] ALTER ONLINE TABLE syntax
by Sergei Golubchik 18 Aug '13

18 Aug '13

Hi, You've added this ALTER ONLINE TABLE syntax, where ONLINE de facto means "without copying the data". Now, after 5.6 merge we also have ALGORITHM=COPY|INPLACE and LOCK=NONE|SHARED|EXCLUSIVE. I've mapped ONLINE to ALGORITHM=INPLACE, because that's what it means. But it's confusing, one can have an "inplace" operation under an exclusive lock, if the engine wants it that way. And it won't be "online" by any stretch of imagination. One can also technically have a "copy" operation with a very permissive write-allow-write lock, and it'll be as online as one can get it. I mean, logically ONLINE should mean LOCK=NONE, not ALGORITHM=INPLACE. But that would change the existing behavior and the test results too. For example, *any* ALTER on the temporary table would always be "online", because, well, it wouldn't take any lock on that table. Alternatively, we can keep ONLINE as ALGORITHM=INPLACE, and deprecate it, issue a warning, suggesting to use the new 5.6 syntax. Opinions? Regards, Sergei

5 7

[Maria-developers] 10.x compatibility and separate namespace
by Honza Horak 13 Aug '13

13 Aug '13

Hi Guys, since 10.x won't be compatible with mysql in every aspect any more, I'm wondering if there happen to be any plans to move all compatibility left-overs (client binary names, client library name, daemon binary name, etc.). Are there any such ideas? I understand that we would lose the compatibility for any later (keeping original names was mentioned in [1] as a MUST for compatibility), but when keeping the original names, it could become tricky to say which tools are compatible and which not any more in mariadb 10+. How this information will be provided actually? How to get users informed properly, which tools are not compatible any more? And a bit another POV -- is the incompatibility in 10.x strictly limited to the server only (so client side will stay compatible for anytime later)? [1] https://lists.launchpad.net/maria-discuss/msg00658.html Regards, Honza

3 2

[Maria-developers] RFC: Compile-time checking for unmatched DBUG_ENTER/LEAVE via unused variable
by Jeremy Cole 11 Aug '13

11 Aug '13

Maria developers/captains, Prior to posing it as a final patch contribution or opening a JIRA ticket, I wanted to propose this idea. The attached patch adds what amounts to compile-time checking for unmatched DBUG_ENTER/DBUG_LEAVE (DBUG_RETURN, DBUG_VOID_RETURN) by introducing a variable in DBUG_ENTER which is only used in DBUG_LEAVE. This allows any compiler which can robustly detect unused variables to detect the mismatch at compile-time. There is already a run-time check for this case, but it is somewhat limited as it requires _db_return_ to be called in order to detect the mismatch, and this is in practice not always the case. Particularly three cases allow this to escape detection: 1. Some instrumented functions are called only from non-instrumented functions. 2. Some instrumented functions are called only from non-returning functions, such as abort or die functions. 3. Some instrumented functions are called from functions who call my_end or my_thread_end (making cs unavailable) before returning. In any case please see the attached DRAFT patch for a full implementation of the idea. Note one bit of ugliness is that non-returning functions, after this patch, must be marked explicitly as DBUG_ENTER_NO_RETURN -- this actually is not really a bad thing, though in and of itself. Regards, Jeremy

4 4

[Maria-developers] Is it a bug?
by Alexey Botchkov 09 Aug '13

09 Aug '13

Hi, SerG. The code in sql/sql_plugin.cc function read_maria_plugin_info() sym= dlsym(plugin_dl->handle, maria_plugin_declarations_sym); ... if (plugin_dl->mariaversion != MARIA_PLUGIN_INTERFACE_VERSION) { ... if ((sym= dlsym(plugin_dl->handle, maria_sizeof_st_plugin_sym))) sizeof_st_plugin= *(int *)sym; ... if (sizeof_st_plugin != sizeof(st_mysql_plugin)) { ... sym= cur; } } plugin_dl->plugins= (struct st_maria_plugin *)sym; So that when we enter the first 'if' but not the second (sizeof_st_plugin == sizeof(st_mysql_plugin) the sym still points at the maria_sizeof_st_plugin_sym which makes a bad st_maria_plugin pointer. I think it's a bug, isn't it? Best regards. HF

2 1

Re: [Maria-developers] [Commits] Rev 3772: MDEV-4645: Incorrect reads of frozen binlog events; FDE corrupted in relay log in file:///home/bell/maria/bzr/work-maria-10.0-MDEV-4548/
by Jeremy Cole 08 Aug '13

08 Aug '13

Sanja, Re: https://mariadb.atlassian.net/browse/MDEV-4645 Re: http://lists.askmonty.org/pipermail/commits/2013-July/005093.html I saw your commit on the commits list (sorry, wasn't on the list before the mail went out, so I can't reply directly to that message). A couple of points related to the commit: - You seem to have an error in the get_header_len, as it's implemented only in the base class, and you've left functions with the correct signature for it but the wrong names in the two frozen classes (they were left named has_fixed_minimal_header). This would mean the change is completely broken. :) - The change to get_header_len I think is in general fine, although it does alter the point of the whole thing -- there is again no way to exactly identify which events have frozen headers, programmatically (i.e. you can't differentiate them from events who just "happen" to have header_len = LOG_EVENT_MINIMAL_HEADER_LEN). I think that doesn't matter for this patch, but does defeat the point of what I was trying to add. - I don't really understand why you've made the hex printing code so complex and completely unreadable (IMHO). This is not performance-sensitive code and in my opinion should favor readability over performance. All you seem to have saved in this change is to avoid printing some static formatting characters ("#", "|", spaces, etc.) inside the loop. However, I can hardly even figure out what it's supposed to be doing now. - Additionally, you've changed the output slightly -- normally the ASCII character section prints its ending "|" immediately after the last character; it seems now to be fixed position (leaving extra spaces before "|". This differs from the output of hexdump, and it somewhat ambiguous/confusing. - You've mixed tabs and spaces. :( Regards, Jeremy

2 8

Re: [Maria-developers] [Commits] Rev 3215: MDEV-4823: Server crashes in Item_func_not::fix_fields on creating a table with a virtual column using NOT in file:///home/bell/maria/bzr/work-maria-5.2-MDEV-4823/
by Sergei Petrunia 31 Jul '13

31 Jul '13

Hi Sanja, On Tue, Jul 30, 2013 at 11:00:59AM +0300, sanja(a)montyprogram.com wrote: > At file:///home/bell/maria/bzr/work-maria-5.2-MDEV-4823/ > > ------------------------------------------------------------ > revno: 3215 > revision-id: sanja(a)montyprogram.com-20130730080057-ylgl6h11l2b3yflc > parent: wlad(a)montyprogram.com-20130709202457-6n9syq59jdd2sy3b > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.2-MDEV-4823 > timestamp: Tue 2013-07-30 11:00:57 +0300 > message: > MDEV-4823: Server crashes in Item_func_not::fix_fields on creating a table with a virtual column using NOT > > fix_field() call protocol was brocken (zero pointer passed as link to item which is possible only if you are sure that there can not be Items which transforms). ... > === modified file 'sql/table.cc' > --- a/sql/table.cc 2013-01-16 00:46:27 +0000 > +++ b/sql/table.cc 2013-07-30 08:00:57 +0000 > @@ -1812,7 +1812,7 @@ bool fix_vcol_expr(THD *thd, > bool result= TRUE; > TABLE_LIST tables; > TABLE_LIST *save_table_list, *save_first_table, *save_last_table; > - int error; > + int error= 0; > Name_resolution_context *context; > const char *save_where; > char* db_name; > @@ -1860,7 +1860,13 @@ bool fix_vcol_expr(THD *thd, > save_use_only_table_context= thd->lex->use_only_table_context; > thd->lex->use_only_table_context= TRUE; > /* Fix fields referenced to by the virtual column function */ > - error= func_expr->fix_fields(thd, (Item**)0); > + if (!func_expr->fixed) > + error= func_expr->fix_fields(thd, (Item**)&vcol_info->expr_item); There is no need to use (Item**) cast, please remove it. Ok to push after that. > + > + /* fix_fields could change the expression */ > + func_expr= vcol_info->expr_item; > + /* Number of columns will be checked later */ > + > /* Restore the original context*/ > thd->lex->use_only_table_context= save_use_only_table_context; > context->table_list= save_table_list; > BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Plugin security
by Leif Walsh 30 Jul '13

30 Jul '13

Hi, I'm working on adding a plugin API (not exactly a storage engine API, rather an extension one) to TokuMX[1] and I'm looking for advice on security. The basic idea is fairly simple: a plugin is a shared library (we're only targeting Linux right now, by the way) that defines a symbol to bootstrap the loading of additional functionality. We have an admin-only command that calls dlopen and dlsym to load the plugin. I haven't read any of the MariaDB plugin code yet, so pointers to where to start reading would be appreciated. Apart from that, I'm just looking for any advice from you all about what to watch out for. My concerns break down into two main areas: 1. When loading a single plugin, what should I watch out for? The command itself is admin-only so calling it is protected, but what about an attacker that drops a malicious plugin in a directory earlier on the load path? What should I check about the permissions/owner of the library file and the directory in which it lives? Is erroring out on world-writable libraries enough? If my process is setuid, should I be more careful? What does Maria do here? 2. About plugin persistence/autoloading: I understand MariaDB has a system table that tracks which plugins are installed and automatically loads those on startup. I believe I can do the same thing in TokuMX and I can restrict access to that collection to the admin user. What does MariaDB store in that table and how does it use it? I could imagine storing the checksum of the installed plugin in the system table and verifying it before loading the plugin, but this seems rather draconian and it might make plugin upgrades too difficult. Does MariaDB verify the permissions of the data files that store this system table? I could imagine an attack where you would replace this system table with one that includes instructions to load your malicious plugin, while the server is offline. Again, how does MariaDB solve these problems? Another option is to autoload all libraries in some directory, and to just be very careful about the permissions of that directory, but this seems very hard to get totally right so I'm leaning away from it. [1]: http://www.tokutek.com/products/tokumx-for-mongodb -- Cheers, Leif

3 5

[Maria-developers] Master slave
by Richard Couture 30 Jul '13

30 Jul '13

Is it possible to set up replication in such a manner as to permit writing records to either a master or a slave replica? We want to set up a fail safe replica that can easily take over when our connection to the main replica fails, which, unfortunately, is fairly often. I read a document, some time ago which mentioned several types of replication, i.e.: by record, by table ... and the pitfalls, like IDs not necessarily matching... which confused me more than anything else... Can someone point me to some good documentation to enable me to pick the correct type of replication that I need to accomplish my task?? Thanks Richard -- LinuxCabal Asociación Civil Ing. Richard Couture Novell CNE, ECNE, MCNE HP/Compaq ASE Tel.: (+52) (333) 145-2638 Cel.: (+52) (044) 333 377-7505 Cel.: (+52) (044) 333 377-7506 Web: http://www.LinuxCabal.org E-Mail: rrc(a)linuxcabal.org Hosted en la nube Cloud Sigma - www.CloudSigma.com AVISO DE CONFIDENCIALIDAD: Este correo electrónico, incluyendo en su caso, los archivos adjuntos al mismo, pueden contener información de carácter confidencial y/o privilegiada, y se envían a la atención única y exclusivamente de la persona y/o entidad a quien va dirigido. La copia, revisión, uso, revelación y/o distribución de dicha información confidencial sin la autorización por escrito de LinuxCabal está prohibida. Si usted no es el destinatario a quien se dirige el presente correo, favor de contactar al remitente respondiendo al presente correo y eliminar el correo original incluyendo sus archivos, así como cualesquiera copia del mismo. Mediante la recepción del presente correo usted reconoce y acepta que en caso de incumplimiento de su parte y/o de sus representantes a los términos antes mencionados, LinuxCabal tendrá derecho a los daños y perjuicios que esto le cause.

1 0