August 2014 - developers - lists.mariadb.org

[Maria-developers] Cross-compiling MariaDB galera cluster
by Sylvain Raybaud 30 Oct '14

30 Oct '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear MariaDB community, Recently I've been trying to cross-compile MariaDB and I've been running in some problems. I'd like to have your opinion on them. 1) Determining stack direction 2) Building Percona XtraDB 3) Building jemalloc Let me first tell that I'm trying to build MariaDB galera version 10.0.12. I'm compiling for Arm but may be compiling for something else in the unpredictable future. 1) Determining stack direction I started by passing the following options to cmake (via the buildroot system): - -DCMAKE_CROSSCOMPILING=1 -DWITH_WSREP=1 -DWITH_INNODB_DISALLOW_WRITES=1 The first problem I ran into was cmake complaining: STACK_DIRECTION is not defined. Please specify -DSTACK_DIRECTION=1 or -DSTACK_DIRECTION=-1 when calling cmake. It seems that STACK_DIRECTION is not automatically determined when cross-compiling. I may determine the correct value for a given target. But if I need a portable solution (and I need one in order to create a buildroot package so that I can build for many different target without too much hassle) I'm not sure what my options are. It seems that the toolchain is aware of this value since stack direction (or more precisely STACK_GROWS_DOWNWARD) is set in ./output/host/opt/ext-toolchain/lib/gcc/arm-none-linux-gnueabi/4.8.1/plugin/include/config/arm/arm.h Maybe there is a way to use this somehow? What do you think? 2) Building Percona XtraDB For the sake of trying I explicitly set -DSTACK_DIRECTION=1 just to see what happens. Building goes on until it reaches the xtradb module. Then it fails with: CMake Error at storage/xtradb/CMakeLists.txt:434 (MESSAGE): Percona XtraDB is not supported on this platform (Here I'll just copy-paste something I wrote on maria-discuss list a few days ago) This error is triggered in storage/xtradb/CMakeLists.txt when XTRADB_OK is not set; it seems to me that XTRADB_OK is set in either one of the following cases: a) CMAKE_CROSSCOMPILING is NOT set and some tests are successful (thus setting HAVE_IB_GCC_ATOMIC_BUILTINS) or b) We're compiling for MSVC or CMAKE_SYSTEM_NAME is "SunOS" and some other conditions are met. Since I'm not compiling for MSVC or SunOS and I'm crosscompiling indeed, XTRADB_OK is not set and the error is triggered. Here I'm stuck. I've got the feeling that as such one cannot cross-compile xtradb because XTRADB_OK will never be set; or maybe with proper options passed to cmake the tests would be made elsewhere and HAVE_IB_GCC_ATOMIC_BUILTINS would be set beforehand? Or maybe I should patch this CMakeLists.txt? Or maybe I should just compile without xtradb altogether, if that's possible? Finally I just passed -DXTRADB_OK just to see what would happen, again (obviously there is little chance that it would result in anything usable). And reached problem #3: 3) Building jemalloc This one fails at configure time with: checking whether we are cross compiling... configure: error: in `/home/sylvain/DEPOTS/buildroot/output/build/mariadb-galera-10.0.12/extra/jemalloc/build': configure: error: cannot run C compiled programs. If you meant to cross compile, use `--host'. Somehow I was expecting that having passed -DCMAKE_CROSSCOMPILING=1 and the host being set by buildroot it would propagate down to the modules... Am I missing something? At this point I'm out of dirty hacks, even to "see what happens" ;) Cheers, - -- Sylvain Raybaud www.green-communications.fr -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJT4ke/AAoJEEkkwl4JtJ9yJW0QALBz3wj0VsfMtUBUabP13q74 ++y1AiCLZ0js+UGCqxB9PVz4yvhhnMAz8QnPpH37XJtDm+VaJBQTEZL65CHF3nlr dHe46/q/wswlbdhNsQu6JmMe3JoOe2TFbjDenQMiUvW9mOilt86ioveHVLGYbpnq 5LYMrPlYXUBpDcxEYXXbTpIHsAoHUrOE1Gj5AsxwbUvQuHDDr9M/Ws01iIXLbzT8 0pImhgBPYUwqaj9tJnPkls+9LzoOb2m269n9TCbmWk0ixakpHjeNMn5lnRDhOwNh AKkHrGdMUomSBca4G0kQoeNxwC2rCjO8+bRxjh7v6GTdME/sgsgn3yOv2EznldUr tFaO/JfraHSUgmiRyVo41U3uTESxZKRJYOV9IqUhl4HiS85D5oa/bD6l84rhx95J 52/hUoApoStr8tSk0tzXk1/UWJEQ3fUZ7XoXqgWq+c1uMgbgZftAAqAw7Xvmc88t tf69FqBzw5dpf8Kg0I7iAggBpBLIWGJwnlVvcmLY4UUE1cDFfZstMBSXaS155LQF VClK+ytE2Ef8oAB9/nlMXgg4b3WXlD0PRQDRTNIwCAzdvsuu+S0q64xLtw/JYnLl KPcLWfwXy+kRhT+vtUWtqMjjkSQ3e6rJDybK11vgYIAWm/Xfk5s8i0g7Bgmnf3wp JGYUAR285gY0XMqL6LmU =vBo4 -----END PGP SIGNATURE-----

2 4

[Maria-developers] Please review: MDEV-5528 Command line variable to choose MariaDB-5.3 vs MySQL-5.6 temporal data formats
by Alexander Barkov 24 Oct '14

24 Oct '14

Hi, Please review my patch for MDEV-5528. It also fixes a bug: MDEV-6649 Different warnings for TIME and TIME(N) when @@old_mode=zero_date_time_cast Thanks.

2 8

[Maria-developers] INET6, data type plugins
by Alexander Barkov 09 Oct '14

09 Oct '14

Hello all, I have mostly implemented the server side part of the fixed length 16-byte INET6 data type (only some clean-ups left). MDEV-274 The data type for IPv6/IPv4 addresses in MariaDB https://mariadb.atlassian.net/browse/MDEV-274 I made it in a very new style, which is ready to data type plugins, as described in: MDEV-4912 Add a plugin to field types (column types) https://mariadb.atlassian.net/browse/MDEV-4912 It is done by introducing a new abstract super-class Type_handler with a number of virtual public methods (around 50 public methods at this point). To implement a new data type one just needs to implement an instantiable class derived from Type_handler and register the new data type in a special global registry. Everything start to work automatically, including (but not limited to): - correct comparison - hybrid type functions like COALESCE, LEAST - aggregate functions MAX() and MIN() - CAST to the new type - ORDER BY, both for field and for items - conversion to the standard data types - indexing and index search The definition of the instantiable class that actually implements INET6 looks like this: class Type_handler_inet6: public Type_handler { private: class Field_inet6: public Field { ... }; class Arg_comparator_inet6: public Arg_comparator { ... }; class Item_typecast_inet6: public Item_func { ... }; public: // various access methods }; It looks very nice for me so far :) I have some questions about the remaining things: 1. In my current patch the new data type INET6 is not loaded dynamically from a shared library. It's implemented inside mysqld. Is it Okey? Or would we prefer to make it as a plugin from the beginning? (in this case there's some more work left: the dynamic loader code, client-server protocol, PS protocol). 2. In case of pluggable data types we'll need some changes in the client-server protocol, to transfer data type names (and probably some more metadata) to the client side in some textual form. Any ideas on that? 3. PS protocol obviously needs two ways to bind data: in text format and in binary format. For example, in case of INET6 it would be nice to be able to bind parameters and fetch values using: - text representation "ffff::ffff" - and binary representation 0xFFFF000000000000000000000000FFFF. I can see some ways to do this: a. Every pluggable data type uses its own type code in MYSQL_BIND: char ip6[16]= "\xff\xff...\xff"; MYSQL_BIND bind; bind.buffer_type= 128; // This is INET6 bind.buffer= ip6; bind.length= sizeof(ip6); but the code (128) may vary between servers, depending on how the plugin is installed. b. Or we could introduce a singe new type code MYSQL_TYPE_RAW: char ip6[16]= "\xff\xff...\xff"; MYSQL_BIND bind; bind.buffer_type= MYSQL_TYPE_RAW; bind.buffer= ip6; bind.length= sizeof(ip6); The server should be able to treat a single type code with different data types without any problems, because it knows the parameter context. But this will not allow to mix values of different types though, for example insert an INET4 value into an INET6 column (or vice verse) won't be possible. c. Or we could support both "native" type codes and MYSQL_TYPE_RAW. 4. Do we need the INET4 data type (fixed length, 4 bytes)? 5. Do we need a PostgreSQL-alike INET data type which is able to store both INET4 and INET6 values (I guess this one should be variable length: 4-bytes for INET4 and 16 bytes for INET6 values. Any feedback would be appreciated. The current patch (attached) will be polished. but the general idea should already be clear from the current version. Thanks.

3 4

[Maria-developers] MariaDB Debian packaging: pending merge to upstream
by Otto Kekäläinen 03 Oct '14

03 Oct '14

Hello! Now when the rehearsal is done using simple spelling error fix patches, I now posted the first big merge request about my 5.5/debian upon the upstream 5.5 branch. Issue at https://mariadb.atlassian.net/browse/MDEV-6284 Merge request at https://code.launchpad.net/~otto/maria/maria-fix-debpkg-5.5/+merge/221576 I also have these old merge request still open: https://code.launchpad.net/~otto/maria/maria-fix-debian-copyright-trunk/+me… https://code.launchpad.net/~otto/maria/maria-fix-debian-copyright-5.5/+merg… The sooner somebody has can merge these the less likely it is that there will be merge conflicts.. Thanks! -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 6

[Maria-developers] [GSoC2014] Self-tuning cost coefficients
by Anshu Avinash 28 Sep '14

28 Sep '14

Hi all, I have written a documentation for framework which I have created for gathering and calculating cost coefficients for optimizer, as part of my GSoC project. It is available at http://igniting.in/gsoc2014/2014/08/20/documentation/. Any suggestions/reviews are welcome. Regards Anshu Avinash

3 3

Re: [Maria-developers] [Maria-discuss] MariaDB encryption
by Jonas Oreland 23 Sep '14

23 Sep '14

Hi again, > What is the type of license of your code? I asked internally about license, and it seems like we releasing dual gpl2/apache licensed code. > I would like to know, which interfaces from maria-DB you are using. I don't 100% understand the question. We didn't write any actual encryption code, but used the one provided in openssl. Other than that, we didn't really "use interfaces", but rather added/modified functionality/interfaces here and there. Can you be more specific ? /Jonas On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck <elmar(a)eperiesi-beck.de > wrote: > Hi! > We (eperi) would be glad to do a joined work with Google. > Our solution works with MS-SQL, Oracle and other DBs and we are currently > porting it to MariaDB - and - as Monty said - its never to late to put some > sources together and make the best for the open source community. > > What is the type of license of your code? > > Jonas, I am looking forward to connect to you directly. > > Regards > Elmar > > Hi! > > > Hi Jonas, > > (same Jonas we know from NDBCLUSTER? :-) Good to see you again) > > > > On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso(a)google.com> wrote: > > > >> Hi there, > >> I read this blog post > >> > http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.… > >> and wanted to inform you that we at Google has developed > on-disk/block-level encryption for Innodb, aria (as used by temporary > tables), binlogs and temp-files. > >> The code is not yet published, but we expect it to be within a few > weeks or so. > >> We (of course?) think that it would be better if you instead of > developing new code > >> spent the time testing/reviewing ours. > > We are out course happy to do this! > > >> I'm happy to answer questions on the topic, > >> and will let you know once we've published it. > > The main question I have about the Innodb encryption is if it based on the > compression code we did for fusion-io? > The idea we had on our side was that by using the new compression hooks we > could add encryption with very little changes to the Innodb code. > Looking forward to when you are ready to publish the code so we can > discuss your changes in detail. > > > This is great news! > > > > From what I gather, from Monty's blog post (and a 1:1 we had some time > back), this is something done by a partner/external company that has a > mostly OSS solution, that we should integrate into 10.1 > > Yes, that's correct. It I would have known that Google was working on > encryption I would have included them in my discussions with eperi. > Fortunately it's not yet too late to do this. > I am sure eperi would like to work on the Google code as a base! > > > That said, Google's release of something that works for InnoDB, Aria, > binlogs, temp files (and presumably not too hard to add for MyISAM) is > something we should definitely review and target for 10.1 > > Yes! > > Regards, > Monty > > >

3 21

[Maria-developers] no information about references - mysql bug 73757
by Roberto Spadim 04 Sep '14

04 Sep '14

Hi guys, there's a bug report about no precise information to solve a problem: http://bugs.mysql.com/bug.php?id=73757 test case: create table t11 (f1 integer primary key) engine innodb; alter table t11 add constraint c1 foreign key (f1) references t1(f1); /* Erro SQL (1005): Can't create table `teste`.`#sql-3f90_25c30` (errno: 150 "Foreign key constraint is incorrectly formed") */ should be better explain that reference table t1 don't exists or something like it should i report at jira, or just mysql bug database is ok? -- Roberto Spadim SPAEmpresarial Eng. Automação e Controle

2 2

[Maria-developers] Query in size
by Kavneet Kaur 04 Sep '14

04 Sep '14

Hi all, I just when through a revision~maria-captains/mariadb-java-client/trunk : /src/test/java/org/mariadb/jdbc/BigQueryTest.java (revision 516)Big Query activated in this code contains a character array of size 20000000. This is something that can be modified. All big queries possible cannot occupy this space. This is sort of leading to more of wastage of memory. Can anyone help me understand why this size has been selected when the same could be done at a size (50-90)% lesser. Regards Kavneet Kaur

2 1

[Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)
by Otto Kekäläinen 01 Sep '14

01 Sep '14

Hello! 2014-02-04 20:41 GMT+02:00 Daniel Bartholomew <dbart(a)mariadb.com>: > On Tue, Feb 4, 2014 at 5:39 AM, Kristian Nielsen > <knielsen(a)knielsen-hq.org> wrote: >>> And while you are at it, please also consider setting up an instance >>> 'kvm-deb-sid-pbuilder-amd64' that provides a Debian unstable >> >> Daniel, can you prepare new sets of VMs for Ubuntu trusty and Debian sid? >> Then I can try to setup Buildbot to do these builds. > > Ok, will do. I should have them up and running before the end of the week. Any updates on this?

3 22

Re: [Maria-developers] [Commits] eecac19: MDEV-6513 deprecate engine_condition_pushdown value of the @@optimizer_switch
by Oleksandr Byelkin 28 Aug '14

28 Aug '14

Hi! Everything is OK. Except one small note in two places. But it is OK to push as is. 26.08.2014 17:17, serg(a)mariadb.org пишет: > revision-id: eecac1976469d513062a0741005afc357b897871 > parent(s): f1c1c04a34f39bcd6622b9886fb089dd41e51103 > committer: Sergei Golubchik > branch nick: maria > timestamp: 2014-08-25 23:13:37 +0200 > message: > > MDEV-6513 deprecate engine_condition_pushdown value of the @@optimizer_switch > > * ignore the OPTIMIZER_SWITCH_ENGINE_CONDITION_PUSHDOWN bit > * issue a deprecation warning on 'engine_condition_pushdown=on' > * remove unused remains of the old pre-5.5 engine_condition_pushdown variable > > --- [skip] > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -197,17 +197,11 @@ enum enum_alter_inplace_result { > #define HA_RECORD_MUST_BE_CLEAN_ON_WRITE (1ULL << 41) > > /* > - Table condition pushdown must be performed regardless of > - 'engine_condition_pushdown' setting. > - > - This flag is aimed at storage engines that come with "special" predicates > - that can only be evaluated inside the storage engine. > - For example, when one does > - select * from sphinx_table where query='{fulltext_query}' > - then the "query=..." condition must be always pushed down into storage > - engine. > + This storage engine supports condition pushdown > */ > -#define HA_MUST_USE_TABLE_CONDITION_PUSHDOWN (1ULL << 42) > +#define HA_CAN_TABLE_CONDITION_PUSHDOWN (1ULL << 42) > +/* old name for the same flag */ > +#define HA_MUST_USE_TABLE_CONDITION_PUSHDOWN HA_CAN_TABLE_CONDITION_PUSHDOWN > > /** > The handler supports read before write removal optimization > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 57d2cc6..fe52a85 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -8574,18 +8574,6 @@ mysqld_get_one_option(int optid, > } > break; > #endif /* defined(ENABLED_DEBUG_SYNC) */ > - case OPT_ENGINE_CONDITION_PUSHDOWN: > - /* > - The last of --engine-condition-pushdown and --optimizer_switch on > - command line wins (see get_options(). > - */ > - if (global_system_variables.engine_condition_pushdown) > - global_system_variables.optimizer_switch|= > - OPTIMIZER_SWITCH_ENGINE_CONDITION_PUSHDOWN; > - else > - global_system_variables.optimizer_switch&= > - ~OPTIMIZER_SWITCH_ENGINE_CONDITION_PUSHDOWN; > - break; > case OPT_LOG_ERROR: > /* > "No --log-error" == "write errors to stderr", > @@ -8964,10 +8952,6 @@ static int get_options(int *argc_ptr, char ***argv_ptr) > &extra_connection_count); > #endif > > - global_system_variables.engine_condition_pushdown= > - MY_TEST(global_system_variables.optimizer_switch & > - OPTIMIZER_SWITCH_ENGINE_CONDITION_PUSHDOWN); > - > opt_readonly= read_only; > > /* > diff --git a/sql/mysqld.h b/sql/mysqld.h > index 21c020b..7a0a75c 100644 > --- a/sql/mysqld.h > +++ b/sql/mysqld.h > @@ -554,7 +554,6 @@ enum options_mysqld > OPT_DEBUG_SYNC_TIMEOUT, > OPT_DELAY_KEY_WRITE_ALL, > OPT_DEPRECATED_OPTION, > - OPT_ENGINE_CONDITION_PUSHDOWN, > OPT_IGNORE_DB_DIRECTORY, > OPT_ISAM_LOG, > OPT_KEY_BUFFER_SIZE, > diff --git a/sql/records.cc b/sql/records.cc > index 1b230c4..cbcdb36 100644 > --- a/sql/records.cc > +++ b/sql/records.cc > @@ -287,8 +287,8 @@ bool init_read_record(READ_RECORD *info,THD *thd, TABLE *table, > thd->variables.read_buff_size); > } > /* Condition pushdown to storage engine */ > - if ((thd->variables.optimizer_switch & > - OPTIMIZER_SWITCH_ENGINE_CONDITION_PUSHDOWN) && > + if ((table->file->ha_table_flags() & > + HA_MUST_USE_TABLE_CONDITION_PUSHDOWN) && > select && select->cond && > (select->cond->used_tables() & table->map) && > !table->file->pushed_cond) > diff --git a/sql/sql_class.h b/sql/sql_class.h > index f96110e..c42a87a 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -597,7 +597,6 @@ typedef struct system_variables > my_bool tx_read_only; > my_bool low_priority_updates; > my_bool query_cache_wlock_invalidate; > - my_bool engine_condition_pushdown; > my_bool keep_files_on_create; > > my_bool old_mode; > diff --git a/sql/sql_priv.h b/sql/sql_priv.h > index 5dc1918..9578c2f 100644 > --- a/sql/sql_priv.h > +++ b/sql/sql_priv.h > @@ -199,7 +199,7 @@ template <class T> bool valid_buffer_range(T jump, > #define OPTIMIZER_SWITCH_INDEX_MERGE_SORT_UNION (1ULL << 2) > #define OPTIMIZER_SWITCH_INDEX_MERGE_INTERSECT (1ULL << 3) > #define OPTIMIZER_SWITCH_INDEX_MERGE_SORT_INTERSECT (1ULL << 4) > -#define OPTIMIZER_SWITCH_ENGINE_CONDITION_PUSHDOWN (1ULL << 5) > +#define deprecated_ENGINE_CONDITION_PUSHDOWN (1ULL << 5) > #define OPTIMIZER_SWITCH_INDEX_COND_PUSHDOWN (1ULL << 6) > #define OPTIMIZER_SWITCH_DERIVED_MERGE (1ULL << 7) > #define OPTIMIZER_SWITCH_DERIVED_WITH_KEYS (1ULL << 8) > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 05c88a5..f061b51 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -9561,10 +9561,8 @@ make_join_select(JOIN *join,SQL_SELECT *select,COND *cond) > if (tab->table) > { > tab->table->file->pushed_cond= NULL; > - if (((thd->variables.optimizer_switch & > - OPTIMIZER_SWITCH_ENGINE_CONDITION_PUSHDOWN) || > - (tab->table->file->ha_table_flags() & > - HA_MUST_USE_TABLE_CONDITION_PUSHDOWN)) && > + if ((tab->table->file->ha_table_flags() & > + HA_MUST_USE_TABLE_CONDITION_PUSHDOWN) && It might be that it is better to use 'new' name for the flag in the lines we have changed. > !first_inner_tab) > { > COND *push_cond= > @@ -23598,10 +23596,8 @@ void JOIN_TAB::save_explain_data(Explain_table_access *eta, table_map prefix_tab > { > const COND *pushed_cond= tab->table->file->pushed_cond; > > - if (((thd->variables.optimizer_switch & > - OPTIMIZER_SWITCH_ENGINE_CONDITION_PUSHDOWN) || > - (tab->table->file->ha_table_flags() & > - HA_MUST_USE_TABLE_CONDITION_PUSHDOWN)) && > + if ((tab->table->file->ha_table_flags() & > + HA_MUST_USE_TABLE_CONDITION_PUSHDOWN) && The same. > pushed_cond) > { > eta->push_extra(ET_USING_WHERE_WITH_PUSHED_CONDITION); > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 7f7c725..db764d7 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -2181,13 +2181,15 @@ export const char *optimizer_switch_names[]= > "exists_to_in", > "default", NullS > }; > -/** propagates changes to @@engine_condition_pushdown */ > static bool fix_optimizer_switch(sys_var *self, THD *thd, > enum_var_type type) > { > SV *sv= (type == OPT_GLOBAL) ? &global_system_variables : &thd->variables; > - sv->engine_condition_pushdown= > - MY_TEST(sv->optimizer_switch & OPTIMIZER_SWITCH_ENGINE_CONDITION_PUSHDOWN); > + if (sv->optimizer_switch & deprecated_ENGINE_CONDITION_PUSHDOWN) > + push_warning_printf(current_thd, Sql_condition::WARN_LEVEL_WARN, > + ER_WARN_DEPRECATED_SYNTAX_NO_REPLACEMENT, > + ER(ER_WARN_DEPRECATED_SYNTAX_NO_REPLACEMENT), > + "engine_condition_pushdown=on"); > return false; > } > static Sys_var_flagset Sys_optimizer_switch( > diff --git a/storage/spider/mysql-test/spider/bg/r/spider_fixes.result b/storage/spider/mysql-test/spider/bg/r/spider_fixes.result > index 6db61fa..3033586 100644 > [skip]

2 1