- developers - lists.mariadb.org

[Maria-developers] Python Storage Engine
by Kim Ebert 02 Nov '15

02 Nov '15

Greetings All, I wanted to share the beginnings of a project. This is a Python Storage Engine. The idea is to plumb in the necessary C to Python calls to allow a storage engine to be written in Python instead of C or C++. This may be valuable to those who want to prototype a database engine in Python, or if the performance is sufficient for the use case, for production. I haven't taken things to far yet, but I think it gives a rough idea of how to move forward with adding the other calls. In my particular case I'm using it to prototype connecting to a search API, like Solr, to provide an SQL like interface. I did have a quick question. I want to license it as a more permissive license, but since I'm using the example engine, does that mean that I need to use the GPL license? Or can I go with the more permissive MIT-new license? https://github.com/perfectsearch/mariadb-plugin-python-engine Thoughts and/or feedback are/is appreciated. Thanks, Kim

3 2

Re: [Maria-developers] MDEV-8692 prefschema test failures on ARM (on Debian build system)
by Sergei Golubchik 28 Oct '15

28 Oct '15

Hi, Alexander! On Oct 23, Alexander Barkov wrote: > Hi, > > - the variant with MY_PERF_CYCLES generally works, but does not fix the > test failure. What do you mean by that? How does it "work" if it does not fix the test failure? > - the variant with MY_CLOCK_REALTIME_FOR_CYCLES works fine > and fixed the problem. > > But it's not really "CYCLES". Perhaps this is not a problem, > because on some other platforms we also fallback cycles to nanoseconds, That commit 49667f0 falls back from cycles to nanoseconds explicitly. So it's ok not to support cycles, as far as I understand, and there's no need to do nanoseconds in place of cycles like your MY_CLOCK_REALTIME_FOR_CYCLES. By the way, why 49667f0 didn't fix the problem? What happens on ARMv7 - no 'cycles' timer at all in perfschema, or there is one and it works incorrectly? > diff --git a/mysql-test/suite/perfschema/t/func_file_io.test b/mysql-test/suite/perfschema/t/func_file_io.test > index 2055fa9..5894cb0 100644 > --- a/mysql-test/suite/perfschema/t/func_file_io.test > +++ b/mysql-test/suite/perfschema/t/func_file_io.test > @@ -43,6 +43,11 @@ TRUNCATE TABLE performance_schema.events_waits_current; > > SELECT * FROM t1 WHERE id = 1; > > +--vertical_results > +SELECT * FROM performance_schema.events_waits_history_long > +WHERE EVENT_NAME = 'wait/io/file/myisam/dfile'; > +--horizontal_results forgotten debug output? > + > SET @before_count = (SELECT SUM(TIMER_WAIT) > FROM performance_schema.events_waits_history_long > WHERE (EVENT_NAME = 'wait/io/file/myisam/dfile') > diff --git a/mysys/my_rdtsc.c b/mysys/my_rdtsc.c > index 028c7f8..a5ed808 100644 > --- a/mysys/my_rdtsc.c > +++ b/mysys/my_rdtsc.c > @@ -533,6 +590,10 @@ void my_timer_init(MY_TIMER_INFO *mti) > mti->cycles.routine= MY_TIMER_ROUTINE_ASM_GCC_SPARC32; > #elif defined(__sgi) && defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_SGI_CYCLE) > mti->cycles.routine= MY_TIMER_ROUTINE_SGI_CYCLE; > +#elif defined(MY_CLOCK_REALTIME_FOR_CYCLES) > + mti->cycles.routine= MY_TIMER_ROUTINE_CLOCK_GETTIME; > +#elif defined(MY_PERF_CYCLES) > + mti->cycles.routine= MY_TIMER_ROUTINE_SGI_CYCLE; I'd say that if we'll have it pushed, you'll need to add a dedicated MY_TIMER_ROUTINE_xxx for this, not reuse MY_TIMER_ROUTINE_SGI_CYCLE. > #elif defined(HAVE_SYS_TIMES_H) && defined(HAVE_GETHRTIME) > mti->cycles.routine= MY_TIMER_ROUTINE_GETHRTIME; > #else Regards, Sergei

1 0

Re: [Maria-developers] [Commits] 72bb358: MDEV-7930 Assertion `table_share->tmp_table != NO_TMP_TABLE || m_lock_type != 2' failed in handler::ha_index_read_map
by Sergey Petrunia 27 Oct '15

27 Oct '15

Hi Sanja, Please find some cosmetic feedback below. Ok to push after it has been addressed. On Mon, Oct 26, 2015 at 05:00:34PM +0100, sanja(a)mariadb.com wrote: > revision-id: 72bb358c2ba98e6e0d4a15b92af514a45751fe8d (mariadb-10.0.21-44-g72bb358) > parent(s): 18f7dfed179204dcfc02a27790e22bb9cc4e2e32 > committer: Oleksandr Byelkin > timestamp: 2015-10-26 16:59:40 +0100 > message: > > MDEV-7930 Assertion `table_share->tmp_table != NO_TMP_TABLE || m_lock_type != 2' failed in handler::ha_index_read_map > > In optimizing aggregate function do not try to touch tables from outer > query. > ... > diff --git a/sql/opt_sum.cc b/sql/opt_sum.cc > index fc3ce09..ad29229 100644 > --- a/sql/opt_sum.cc > +++ b/sql/opt_sum.cc > @@ -341,7 +341,8 @@ int opt_sum_query(THD *thd, > there are no outer joins. > */ > if (!conds && !((Item_sum_count*) item)->get_arg(0)->maybe_null && > - !outer_tables && maybe_exact_count) > + !outer_tables && maybe_exact_count && > + ((item->used_tables() & OUTER_REF_TABLE_BIT) == 0)) > { > if (!is_exact_count) > { > @@ -369,7 +370,8 @@ int opt_sum_query(THD *thd, > indexes to find the key. > */ > Item *expr=item_sum->get_arg(0); > - if (expr->real_item()->type() == Item::FIELD_ITEM) > + if (((expr->used_tables() & OUTER_REF_TABLE_BIT) == 0) && > + expr->real_item()->type() == Item::FIELD_ITEM) > { > uchar key_buff[MAX_KEY_LENGTH]; > TABLE_REF ref; > @@ -394,16 +396,20 @@ int opt_sum_query(THD *thd, > const_result= 0; > break; > } > + /* > + Constant table is already read and the only row is the max > + and the min > + */ I think this comment is not relevant anymore? It's part of the first variant of the fix, which is no longer in the patch. Please remove it. > if (!(error= table->file->ha_index_init((uint) ref.key, 1))) > - error= (is_max ? > + error= (is_max ? > get_index_max_value(table, &ref, range_fl) : > get_index_min_value(table, &ref, item_field, range_fl, > prefix_len)); > > /* Verify that the read tuple indeed matches the search key */ > - if (!error && reckey_in_range(is_max, &ref, item_field->field, > - conds, range_fl, prefix_len)) > - error= HA_ERR_KEY_NOT_FOUND; > + if (!error && reckey_in_range(is_max, &ref, item_field->field, > + conds, range_fl, prefix_len)) > + error= HA_ERR_KEY_NOT_FOUND; These are just endspace changes. Can you remove them? > table->disable_keyread(); > table->file->ha_index_end(); > if (error) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Several CVE's in Oracle MySQL, is MariaDB vulnerable?
by Christian Rebischke 26 Oct '15

26 Oct '15

Hello everyone, Sorry when I am on the wrong mailinglist. I wanted to submit a bugreport or issue for this but I can't signup for your JIRA because of license reasons. I am from the Archlinux-Security Team and want to ask if mariadb in the actual version is vulnerable to the following CVEs: CVE-2015-4913 CVE-2015-4910 CVE-2015-4905 CVE-2015-4904 CVE-2015-4895 CVE-2015-4890 CVE-2015-4879 CVE-2015-4870 CVE-2015-4862 CVE-2015-4864 CVE-2015-4861 CVE-2015-4858 CVE-2015-4836 CVE-2015-4833 CVE-2015-4830 CVE-2015-4826 CVE-2015-4819 CVE-2015-4815 CVE-2015-4807 CVE-2015-4802 CVE-2015-4800 CVE-2015-4792 CVE-2015-4791 CVE-2015-4766 I hope you can help me. Best regards -------------------------------------------------------------- Christian Rebischke Member of Archlinux CVE-Monitoring Team Website : www.nullday.de Twitter : @sh1bumi Jabber : shibumi(a)jabber.ccc.de PGP : 0xDFE2060D Fingerprint: 6DAF 7B80 8F9D F251 3962 0000 D214 61E3 DFE2 060D --------------------------------------------------------------

4 13

[Maria-developers] crosscompiling in buildroot: cannot find -laio and other link error
by Sylvain Raybaud 23 Oct '15

23 Oct '15

Hi I'm trying to get the latest version of mariadb-galera (10.0.21) into buildroot. I'm running into two errors which I have a feeling are related. But I may be wrong. These problems arise during the build of the host variant of the package. I get the two following errors (build does not stop at the first one because they happen in parallel jobs): 1: Linking CXX shared module ha_connect.so /home/sylvain/git/br/output/host/usr/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib/libxml2.so: error adding symbols: File in wrong format --> for host variant ha_connect.so should be linked against /home/sylvain/git/br/output/host/usr/lib/libxml2.so 2: Linking CXX shared module ha_innodb.so /usr/bin/ld: cannot find -laio The thing is, libaio and libxml2 are both present in /home/sylvain/git/br/output/host/usr/lib/. The link command present in storage/connect/CMakeFiles/connect.dir/link.txt (http://pastebin.com/Qn1ZCw2X) contains: /home/sylvain/git/br/output/host/usr/arm-buildroot-linux-uclibcgnueabi/sysroot/usr/lib/libxml2.so which is incorrect, should be: /home/sylvain/git/br/output/host/usr/lib/libxml2.so The link command present in storage/innobase/CMakeFiles/innobase.dir/link.txt (http://pastebin.com/it6W9WVs) does not contain -L/home/sylvain/git/br/output/host/usr/lib/. I think it should. At least, when I manually run this command adding -L/home/sylvain/git/br/output/host/usr/lib, it works. I'm really not sure how these link.txt are generated. I checked connect/CMakeLists.txt and innobase/CMakeLists.txt and could not find anything obvious problem. Beside, toplevel CMakeCache.txt does contain: CMAKE_EXE_LINKER_FLAGS:STRING=-L/home/sylvain/git/br/output/host/lib -L/home/sylvain/git/br/output/host/usr/lib -Wl,-rpath,/home/sylvain/git/br/output/host/usr/lib which seems correct. I don't know why this setting does not reach storage/innobase and I'm also not sure why ha_connect.so tries to link against target variant of libxml2. It basically comes down to have cmake look for shared libraries in the right place, but I don't seem to be able to determine which lever to pull. Any hint as to how to achieve that is therefore welcome. Cheers, -- Sylvain Raybaud www.green-communications.fr

2 7

[Maria-developers] innodb holepunch compression vs the filesystem
by MARK CALLAGHAN 21 Oct '15

21 Oct '15

InnoDB punch-hole compression is very bad for a filesystem when done on every (or most) page writes. That feature was not designed to be used in that manner and generates far too much in-memory metadata in the filesystem. If nothing else it leads to multi-minute unlink() system calls. Yet both MariaDB and MySQL proudly exclaim this as a feature. Are you going to use the same kludge as MySQL (do unlink() in a background thread) to avoid locking up InnoDB for many minutes during drop table, maybe leading to InnoDB killing mysqld process? Why didn't you catch this problem via testing or reading my blog posts? https://mariadb.com/kb/en/mariadb/compression/ https://blog.mariadb.org/mariadb-10-1-is-stable-ga/ https://bugs.mysql.com/bug.php?id=78277 http://smalldatum.blogspot.com/2015/08/first-day-with-innodb-transparent-pa… http://smalldatum.blogspot.com/2015/09/second-day-with-innodb-transparent-p… http://smalldatum.blogspot.com/2015/09/third-day-with-innodb-transparent-pa… http://smalldatum.blogspot.com/2015/10/wanted-file-system-on-which-innodb.h… -- Mark Callaghan mdcallag(a)gmail.com

2 1

Re: [Maria-developers] Deadlock in Parallel Replication with FLUSH TABLE WITH READ LOCK.
by Kristian Nielsen 20 Oct '15

20 Oct '15

> I enabled // replication on the slave that has MyISAM tables. > The first results are good, but I ended up with what looks like a deadlock. > I opened support # 10289 for "Deadlock in Parallel Replication with > FLUSH TABLE WITH READ LOCK". There is a patch for this: https://lists.launchpad.net/maria-developers/msg08675.html But no reply on the review request so far. - Kristian.

1 0

[Maria-developers] Code review process
by Otto Kekäläinen 19 Oct '15

19 Oct '15

Hello! During the developer meetup there was a discussion about Gerrit. OpenStack has quite a nice documentation how to contribute, including the git review command and gerrit workflow: http://docs.openstack.org/infra/manual/developers.html -- Otto Kekäläinen MariaDB Foundation

1 0

Re: [Maria-developers] [Maria-discuss] Moving from Launchpad lists to something else... ?
by Tom Worster 19 Oct '15

19 Oct '15

On 10/15/15, 5:03 AM, "Maria-discuss on behalf of Colin Charles" <maria-discuss-bounces+fsb=thefsb.org(a)lists.launchpad.net on behalf of colin(a)mariadb.org> wrote: >> On 15 Oct 2015, at 05:35, Jean Weisbuch <jean(a)phpnet.org> wrote: >> >> To be able to give the best answer we must first know what is >>motivating the move from LP. > >So the reason we were thinking of ditching launchpad for mailing lists: >1. When there was a MariaDB-Manager google group, that group got a lot of >user posts about MariaDB Server. That group now no longer exists, but it >gave us a hint that people are interested in a Google group > >2. We moved everything from Launchpad (bugs to Jira, code to Github) and >the only thing remaining is mailing lists > >3. The MariaDB Knowledgebase: https://mariadb.com/kb/en/ used to have an >OpenID login that would be useful if you had a LP account, but OpenID is >apparently going away (or gone), so there’s less use for LP > >4. LaunchPad itself seems to be “hard” for people to signup (see #1). Good points. Making it easy for people to get involved is important, but if that's the idea then moving to Google Groups isn't a great solution. Apart from it being old and clunky owing to Google's neglect, the insurmountable problem is that it's still an email list at core. Personally, I like email lists best but I'm very old and have been in computing since the 70s, and I accept that most of the rest of the world has moved on. Another problem the Maria community might consider together with the discuss email list is the low quality and participation in the web Q&A of the Knowledge Base. That sometimes makes me sad. There is now a decent open-source system that can function as email list replacement and provide web and mobile Q&A, as well as other stuff: https://www.discourse.org I've used it at New Relic and it seems pretty good. It's got a lot of good stuff and I want to highlight just two. 1. "Mailing List Support - Opt into a special mode where all messages are sent to you via email, exactly like a mailing list. Start new topics via email." 2. You can host it yourself or pay someone you trust to do so. It's not a "free" cloud service where you pay by feeding the internet advertising/surveillance economy. I don't know if Discourse is right for Maria but I wouldn't mention it if I didn't think it might help build a bigger and more active support community, which I long for. Tom

2 1

[Maria-developers] JSON in MariaDB 10.2
by Colin Charles 14 Oct '15

14 Oct '15

Hi all, We just had a session here at the MariaDB Server developer’s meeting in Amsterdam on JSON in MariaDB 10.2. Before forgetting while things are still fresh in the mind, here are the notes from it (in Markdown format). cheers, -colin # JSON in MariaDB * Virtual columns with functional indexes (i.e. generated columns like in 5.7) can be implemented in 2-3 weeks (Sergei) * Eric: Concern about using non-standard function names (functions in 5.7 are `JSN_INSERT` whereas the standard uses `JSON_QUERY`) * Suggestion: Use standard names when available, create UDF to match MySQL names that can be loaded as desired. (or vice versa -- do what MySQL 5.7 does and have UDFs to follow the standard) * Come up with Usable JSON Test suite * Employee DB in JSON? * Good tests exist in DocStore (from the 5.6 facebook tree) * Monty believes we should have more strictly typed (DocStore for example says `unique key doc_c(doc.address.zipcode as int, c)`). Monty thinks it should be optional to keep compatibility with 5.7. This is also similar to the JSON proposed standard (where you have to choose between lax or strict mode) * Colin: Might want to convert JSON document to a relational table (This is part of the standard as `JSON_TABLE`) * Colin: Do we support something similar to what FB has or what oracle has? * SergeyP: We need to follow oracle to maintain compatibility. * Colin: We need to do this soon (i.e. since 5.7 is coming and its going to be a problem for 10.1 not having it; so 10.2 needs it as soon as possible). Everyone generally agrees this is a good idea. # References * [SQL SUPPORT FOR JSON](http://jtc1bigdatasg.nist.gov/_workshop/08_SQL_Support_for_JSON_abstr… * [WL#8132: JSON datatype and binary storage format](https://dev.mysql.com/worklog/task/?id=8132) * [WL#7909: Server side JSON functions](https://dev.mysql.com/worklog/task/?id=7909) * [WL#8249: JSON comparator](https://dev.mysql.com/worklog/task/?id=8249) * [WL#8170: Expression analyzer for GC](https://dev.mysql.com/worklog/task/?id=8170) * [WL#8149: B-tree Index Support on non-materialized virtual columns](https://dev.mysql.com/worklog/task/?id=8149) * [WL#8227: Support SEs to create index on virtual generated columns](https://dev.mysql.com/worklog/task/?id=8227) * [WL#8114: Don't store virtual generated columns in database](https://dev.mysql.com/worklog/task/?id=8114) * [JSON Labs Release: Native JSON Data Type and Binary Format](http://mysqlserverteam.com/json-labs-release-native-json-data-type-… * [JSON Labs Release: JSON Functions, Part 1 — Manipulation JSON Data](http://mysqlserverteam.com/json-labs-release-json-functions-part-1-ma… * [JSON Labs Release: JSON Functions, Part 2 — Querying JSON Data](http://mysqlserverteam.com/mysql-5-7-lab-release-json-functions-part-… * [JSON Labs Release: Effective Functional Indexes in InnoDB](http://mysqlserverteam.com/json-labs-release-effective-functional-i… * [11.6 The JSON Data Type](https://dev.mysql.com/doc/refman/5.7/en/json.html) * [DocStore: Document Database for MySQL at Facebook](https://www.percona.com/live/mysql-conference-2015/sites/default/… * [JSON Support in MySQL](https://www.percona.com/live/mysql-conference-2015/sites/default/fil… # Thanks Thanks to Bryan Alsdorf for writing some of the notes during the session -- Colin Charles, Chief Evangelist, MariaDB Corporation blog: http://bytebot.net/blog/ | t: +1-347-903-3201 | Skype: colincharles | Twitter: @bytebot

1 0