July 2014 - developers - lists.mariadb.org

[Maria-developers] INET6, data type plugins
by Alexander Barkov 09 Oct '14

09 Oct '14

Hello all, I have mostly implemented the server side part of the fixed length 16-byte INET6 data type (only some clean-ups left). MDEV-274 The data type for IPv6/IPv4 addresses in MariaDB https://mariadb.atlassian.net/browse/MDEV-274 I made it in a very new style, which is ready to data type plugins, as described in: MDEV-4912 Add a plugin to field types (column types) https://mariadb.atlassian.net/browse/MDEV-4912 It is done by introducing a new abstract super-class Type_handler with a number of virtual public methods (around 50 public methods at this point). To implement a new data type one just needs to implement an instantiable class derived from Type_handler and register the new data type in a special global registry. Everything start to work automatically, including (but not limited to): - correct comparison - hybrid type functions like COALESCE, LEAST - aggregate functions MAX() and MIN() - CAST to the new type - ORDER BY, both for field and for items - conversion to the standard data types - indexing and index search The definition of the instantiable class that actually implements INET6 looks like this: class Type_handler_inet6: public Type_handler { private: class Field_inet6: public Field { ... }; class Arg_comparator_inet6: public Arg_comparator { ... }; class Item_typecast_inet6: public Item_func { ... }; public: // various access methods }; It looks very nice for me so far :) I have some questions about the remaining things: 1. In my current patch the new data type INET6 is not loaded dynamically from a shared library. It's implemented inside mysqld. Is it Okey? Or would we prefer to make it as a plugin from the beginning? (in this case there's some more work left: the dynamic loader code, client-server protocol, PS protocol). 2. In case of pluggable data types we'll need some changes in the client-server protocol, to transfer data type names (and probably some more metadata) to the client side in some textual form. Any ideas on that? 3. PS protocol obviously needs two ways to bind data: in text format and in binary format. For example, in case of INET6 it would be nice to be able to bind parameters and fetch values using: - text representation "ffff::ffff" - and binary representation 0xFFFF000000000000000000000000FFFF. I can see some ways to do this: a. Every pluggable data type uses its own type code in MYSQL_BIND: char ip6[16]= "\xff\xff...\xff"; MYSQL_BIND bind; bind.buffer_type= 128; // This is INET6 bind.buffer= ip6; bind.length= sizeof(ip6); but the code (128) may vary between servers, depending on how the plugin is installed. b. Or we could introduce a singe new type code MYSQL_TYPE_RAW: char ip6[16]= "\xff\xff...\xff"; MYSQL_BIND bind; bind.buffer_type= MYSQL_TYPE_RAW; bind.buffer= ip6; bind.length= sizeof(ip6); The server should be able to treat a single type code with different data types without any problems, because it knows the parameter context. But this will not allow to mix values of different types though, for example insert an INET4 value into an INET6 column (or vice verse) won't be possible. c. Or we could support both "native" type codes and MYSQL_TYPE_RAW. 4. Do we need the INET4 data type (fixed length, 4 bytes)? 5. Do we need a PostgreSQL-alike INET data type which is able to store both INET4 and INET6 values (I guess this one should be variable length: 4-bytes for INET4 and 16 bytes for INET6 values. Any feedback would be appreciated. The current patch (attached) will be polished. but the general idea should already be clear from the current version. Thanks.

3 4

[Maria-developers] MariaDB Debian packaging: pending merge to upstream
by Otto Kekäläinen 03 Oct '14

03 Oct '14

Hello! Now when the rehearsal is done using simple spelling error fix patches, I now posted the first big merge request about my 5.5/debian upon the upstream 5.5 branch. Issue at https://mariadb.atlassian.net/browse/MDEV-6284 Merge request at https://code.launchpad.net/~otto/maria/maria-fix-debpkg-5.5/+merge/221576 I also have these old merge request still open: https://code.launchpad.net/~otto/maria/maria-fix-debian-copyright-trunk/+me… https://code.launchpad.net/~otto/maria/maria-fix-debian-copyright-5.5/+merg… The sooner somebody has can merge these the less likely it is that there will be merge conflicts.. Thanks! -- Check out our blog at http://seravo.fi/blog and follow @ottokekalainen

2 6

Re: [Maria-developers] [Maria-discuss] MariaDB encryption
by Jonas Oreland 23 Sep '14

23 Sep '14

Hi again, > What is the type of license of your code? I asked internally about license, and it seems like we releasing dual gpl2/apache licensed code. > I would like to know, which interfaces from maria-DB you are using. I don't 100% understand the question. We didn't write any actual encryption code, but used the one provided in openssl. Other than that, we didn't really "use interfaces", but rather added/modified functionality/interfaces here and there. Can you be more specific ? /Jonas On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck <elmar(a)eperiesi-beck.de > wrote: > Hi! > We (eperi) would be glad to do a joined work with Google. > Our solution works with MS-SQL, Oracle and other DBs and we are currently > porting it to MariaDB - and - as Monty said - its never to late to put some > sources together and make the best for the open source community. > > What is the type of license of your code? > > Jonas, I am looking forward to connect to you directly. > > Regards > Elmar > > Hi! > > > Hi Jonas, > > (same Jonas we know from NDBCLUSTER? :-) Good to see you again) > > > > On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso(a)google.com> wrote: > > > >> Hi there, > >> I read this blog post > >> > http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.… > >> and wanted to inform you that we at Google has developed > on-disk/block-level encryption for Innodb, aria (as used by temporary > tables), binlogs and temp-files. > >> The code is not yet published, but we expect it to be within a few > weeks or so. > >> We (of course?) think that it would be better if you instead of > developing new code > >> spent the time testing/reviewing ours. > > We are out course happy to do this! > > >> I'm happy to answer questions on the topic, > >> and will let you know once we've published it. > > The main question I have about the Innodb encryption is if it based on the > compression code we did for fusion-io? > The idea we had on our side was that by using the new compression hooks we > could add encryption with very little changes to the Innodb code. > Looking forward to when you are ready to publish the code so we can > discuss your changes in detail. > > > This is great news! > > > > From what I gather, from Monty's blog post (and a 1:1 we had some time > back), this is something done by a partner/external company that has a > mostly OSS solution, that we should integrate into 10.1 > > Yes, that's correct. It I would have known that Google was working on > encryption I would have included them in my discussions with eperi. > Fortunately it's not yet too late to do this. > I am sure eperi would like to work on the Google code as a base! > > > That said, Google's release of something that works for InnoDB, Aria, > binlogs, temp files (and presumably not too hard to add for MyISAM) is > something we should definitely review and target for 10.1 > > Yes! > > Regards, > Monty > > >

3 21

[Maria-developers] New MariaDB builds: regular trusty and special kvm-deb-sid-pbuilder-amd and (Re: MariaDB)
by Otto Kekäläinen 01 Sep '14

01 Sep '14

Hello! 2014-02-04 20:41 GMT+02:00 Daniel Bartholomew <dbart(a)mariadb.com>: > On Tue, Feb 4, 2014 at 5:39 AM, Kristian Nielsen > <knielsen(a)knielsen-hq.org> wrote: >>> And while you are at it, please also consider setting up an instance >>> 'kvm-deb-sid-pbuilder-amd64' that provides a Debian unstable >> >> Daniel, can you prepare new sets of VMs for Ubuntu trusty and Debian sid? >> Then I can try to setup Buildbot to do these builds. > > Ok, will do. I should have them up and running before the end of the week. Any updates on this?

3 22

Re: [Maria-developers] [GSoC] Optimize mysql-test-runs - Results of new strategy
by Pablo Estrada 19 Aug '14

19 Aug '14

Hello Elena and all, First, addressing the previous email: Looking at the dump, I see it can also happen that the dump contains > several records for a pair platform/bbnum. I am not sure why it happens, I > think it shouldn't, might be a bug in buildbot and/or configuration, or > environmental problems. Anyway, due to the way we store output files, they > can well override each other in this case, thus for several platform/bbnum > record you will have only one file. I suppose that's what was hard to > resolve, sorry about that. > No worries ; ). There are several cases where platform and build number are the same. The system just names the files as follows: <platform>_<build_id>-log-test_1-stdio <platform>_<build_id>-log-test_2-stdio ..... <platform>_<build_id>-log-test_5-stdio These files seem to correspond temporarily with the test runs (*test_1-stdio) belongs to the first test_run of the same plt-bnum, and so on. Unfortunately, there are some cases where there are more test_runs on the dump than files available, and this means that it's impossible to be sure which file belongs to which test_run exactly. > You should consider skipped tests, at least for now. Your logic that they > are skipped because they can't be run is generally correct; unfortunately, > MTR first produces the *full* list of tests to run, and determines whether > a test can be run or not on a later stage, when it starts running the > tests. Your tool will receive the initial test list, and I'm not sure it's > realistic to re-write MTR so that it takes into account limitations that > cause skipping tests before creating the list. > I see. Okay then, duly noted. Possibly it's better to skip a test run altogether if there is no input > list for it; it would be definitely the best if there were 5K (or whatever > slice you are currently using) of continuous test runs with input lists; if > it so happens that there are lists for some branches but not others, you > can skip the branch entirely. > This doesn't seem like a good option. Recall drops seriously, and the test_runs that have a corresponding file don't seem to have a special pattern, and tend to have long spaces between them, so the information becomes irrelevant, and seemingly, not useful. > The core module should take as parameters > - list of tests to choose from, > - size of the running set (%), > - branch/platform (if we use them in the end), > and produce a new list of tests of the size of the running set. > > The wrapper module should > - read the list of tests from the outside world (for now, from a file), > - receive branch/platform as command-line options, > - have the running set size set as an easily changeable constant or as a > configuration parameter, > > and return the list of tests -- lets say for now, in the form of <test > suite>.<test name>, blank-separated, e.g. > main.select innondb.create-index ... > >> >> I am almost done 'translating' the code into a solution that divides it in 'core' and 'wrapper'. There are a few bugs that I still haven't figured out, but I believe I can iron those out pretty soon. I will also incorporate the percentage rather than fixed running_set. Now, regarding the state of the project (and the recall numbers that I am able to achieve so far), here are some observations: - Unfortunately, I am running out of ideas to try to improve recall. I tried tuning some parameters, giving more weight to ones or others, etc. I still wasn't able to push recall beyond ~87% on the strategy that uses file correlations. For what I've seen, some failures are just extremely hard to predict. - The strategy that uses only a weighted average of the failure frequency achieves a higher recall, but for a shorter time. The recall decays quickly afterwards. I may try to add some file-correlations to this strategy, to see if the recall can be sustained for a longer term. - There is one problem that I see regarding the data and the potential real-world implementation of the program: By verifying the recall with the historical data, we run the possibility of 'expecting' overfitting... so the results regarding the errors found when comparing to the historical data, and the results that could have been obtained by a real-world implementation are potentially different. A possible way to address that issue would require modifying the buildbot to gather more data over a longer term. So having said that, I am looking for some advice in the following regards: - I will try to take a step back from the new strategy, and see how I can adapt the original strategy to prevent the recall function from declining so sharply with time. - I will also spend some time keeping a codebase that adjusts better to the model that we need for the implementation. I will upload code soon. All suggestions are welcome. - Nonetheless, I feel that more data would allow to improve the algorithm greatly. Is it possible to prepare logging into the buildbot that would allow for more precise data collection? A slower, more iterative process, working closer with the buildbot and doing more detailed data collection might deliver better results. (I understand that this would probably influence the time-scope of the project) Let me know what you think about my suggestions. Regards Pablo

2 22

[Maria-developers] [GSoC] Introduction Mail
by Anshu Avinash 04 Aug '14

04 Aug '14

Hi all, I have been accepted for the project "Self-Tuning Optimizer". I will be doing this project under the mentorship of Sergei Golubchik. There is already a task created which will track the progress for this project: https://mariadb.atlassian.net/browse/MDEV-350 . I'll start with getting familiar with the code base, reading some literature on cost estimation in optimizer and interacting with community. About me: I am currently a 4th year student pursuing B.Tech - M.Tech dual degree in Computer Science and Engineering from Indian Institute of Technology (IIT) – Kanpur, India. This would be my first GSoC project. I had done an internship at Facebook last year. I'll also be maintaining a blog at http://igniting.in . I'm really excited to be a part of mariadb and I hope to make a significant contribution to it. Regards Anshu Avinash

5 48

[Maria-developers] Review request MDEV-6274 Collation usage statistics (for feedback plugin)
by Alexander Barkov 30 Jul '14

30 Jul '14

Hello Sergei, Please review a patch for MDEV-6274. Thanks.

2 3

Re: [Maria-developers] [Commits] Rev 4227: MDEV-6441: memory leak in file:///home/bell/maria/bzr/work-maria-5.5-MDEV-6441/
by Igor Babaev 29 Jul '14

29 Jul '14

Sanja, Correct the comment for the commit and push. The only question I have concerns another comment: +# MDEV-5515: sub-bug nets of 3rd execution crash ^ Is this an intended word? Regards, Igor. On 07/28/2014 05:36 AM, sanja(a)askmonty.org wrote: > At file:///home/bell/maria/bzr/work-maria-5.5-MDEV-6441/ > > ------------------------------------------------------------ > revno: 4227 > revision-id: sanja(a)askmonty.org-20140728123541-3dh3f20p41iyrefm > parent: sanja(a)askmonty.org-20140715093734-42nrug10i0xw8hvb > committer: sanja(a)askmonty.org > branch nick: work-maria-5.5-MDEV-6441 > timestamp: Mon 2014-07-28 15:35:41 +0300 > message: > MDEV-6441: memory leak > > Whole mysql_derived_prepare() was executed on statement memory. Now it is executed on runtime meory and all bugs induced bu this fixed. mysql_derived_prepare() was executed on the statement memory. Now it is executed on the runtime memory. All bugs induced by this were fixed. > > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Performance: question about strings created in sql_lex.cc
by Alexander Barkov 29 Jul '14

29 Jul '14

Hello, When parsing an SQL query, the function get_text() in sql_lex.cc always makes a copy of the string, allocating new memory using lip->m_thd->alloc() and copying the original string from the SQL query fragment to this new memory. Note, this happens even if the string does not have any escaped characters! I'm seeing some performance bottleneck here. It would be nice to avoid copying. Why could not we point a non-escaped string to the original SQL query fragment where the string is actually written? Are there any situations when the original SQL query is not accessible any more, but Item_string created at parse time is still in use? Are there any other things that are created in sql_yacc.yy using the TEXT_STRING token, which could exist longer than the original SQL query? Thanks.

1 0

[Maria-developers] merge 10.0-connect -> 10.0
by Alexander Barkov 29 Jul '14

29 Jul '14

Hello Olivier, I tried to merge 10.0-connect into the main 10.0, but two mtr tests failed in the merged tree. Note, they don't fail in 10.0-connect, so I guess something have changed in the main tree which makes your latest changes in 10.0-connect incompatible. Please find the log attached. I have no idea how to fix the failures. Can I push the merge into 10.0-connect, so you can fix the problems? Also, I noticed that you changed the sources outside of the "/storage/connect" directory: > --- sql/sql_partition.cc 2014-03-26 21:25:38 +0000 > +++ sql/sql_partition.cc 2014-07-29 04:44:43 +0000 > @@ -3195,13 +3195,26 @@ uint32 get_partition_id_cols_list_for_en > (list_index - 1)*num_columns, > nparts, left_endpoint, > include_endpoint))); > - > +#if 0 > if (!left_endpoint) > { > /* Set the end after this list tuple if not already after the last. */ > if (list_index < part_info->num_parts) > list_index++; > } > +#else > + if (!left_endpoint && list_index < part_info->num_list_values) > + { > + /* > + Set the end after this list tuple if it is not already after the last > + and it matches. > + */ > + int cmp = cmp_rec_and_tuple_prune(list_col_array + list_index*num_columns, > + nparts, left_endpoint, include_endpoint); > + if (cmp >= 0) > + list_index++; > + } > +#endif > > DBUG_RETURN(list_index); > } Can you please comment why do you need this change? Did you discuss this change with anyone? Thanks.

1 0