August 2014 - developers - lists.mariadb.org

Re: [Maria-developers] Learning about MariaDB, Getting Started
by Kavneet Kaur 26 Aug '14

26 Aug '14

Hi, James, I am aiming to become am interested in data security and reducing query time by increasing efficiency and computational speed. Coding and optimizing codes are also cup of my tea. How can I contribute this in MariaDB as it's the only open source database software that attracted me. Regards,Kavneet Kaur Date: Fri, 22 Aug 2014 04:44:45 -0700 From: actionmessage(a)yahoo.com Subject: Re: [Maria-developers] Learning about MariaDB, Getting Started To: kavneetk(a)outlook.com; maria-developers(a)lists.launchpad.net; ashish1294(a)gmail.com Hi Kavneet and Ashish. Depends on what you're interested in working on, but here's some suggestions.Direct link to currently open worklogs:https://mariadb.atlassian.net/browse/MDEV-6629?filter=-4Three easy MDEVs and some other areas to work on:- MDEV-6584 - not using tcp_fastopen- MDEV-6555 - sql_log_bin should be SESSION_ONLY- MDEV-5696 - Graceful MariaDB shutdown at Windows restart- test and see if fixes are needed for SSL, Galera, Cassandra, GTID, IPv6 or partition features. "Howto Add a New Command to the MySQL Server" Tutorial http://www.jebriggs.com/blog/2013/05/howto-add-a-new-command-to-the-mysql-s…, James. From: Kavneet Kaur <kavneetk(a)outlook.com> To: "maria-developers(a)lists.launchpad.net" <maria-developers(a)lists.launchpad.net> Sent: Thursday, August 21, 2014 11:49 PM Subject: [Maria-developers] Learning about MariaDB Hi! I am highly interested in subject of database. I am also interested in becoming a research developer. Worked on MS SQL. Want to try MariaDB. May I get the guidance about this product as i'm totally new to it. Is there any specific way to begin? _______________________________________________Mailing list: https://launchpad.net/~maria-developersPost to : maria-developers(a)lists.launchpad.netUnsubscribe : https://launchpad.net/~maria-developersMore help : https://help.launchpad.net/ListHelp

3 6

[Maria-developers] Please review: 10% performance improvement for INSERT INTO t1 (blob) VALUES ('long value')
by Alexander Barkov 26 Aug '14

26 Aug '14

Hi, I think I found and fixed a bottleneck in how string literals are processed. The attached patch improves INSERT performance for strings, and especially for long blob and text values. Insertion time for 10MByte blob 1024 times (10GByte of data total) reduced on my laptop from 300 seconds to 274 seconds, which gives near 10% pure time improvement. What the patch changes: Before the patch : 1. The function get_text() is sql_lex.cc allocated an unescaped copy of every string literal (unescaping backslashes and double quotes, if any). Strangely, copying happened even if there were no really any escapes in the string. 2, The syntax parser in sql_yacc.yy created Item_string using the new unescaped buffer. Furthermore, in case of a multi-byte connection character set, the constructor for Item_string made another loop on the unescaped buffer, to calculate length in characters, which is needed to set max_length properly. After the patch: Item is now created using directly the SQL fragment, without making a copy. Even for escaped values!!! Length in characters is also calculated during the very first pass in get_text(), without any additional loops in the Item constructors. Unescaping is now done in the very end, when the value is actually needed: - Either in Field::store(). This happens when the string value is used in: INSERT INTO t1 VALUES('string'); Unescaping is done directly to the Field buffer, without any intermediary temporary storage. - Or in val_str(), if the string value is used elsewhere (in SELECT list, functions, operators, etc). The unescaped value is cached, to make sure that unescaping is done only one time, even if the value is needed for multiple rows, like in: SELECT * FROM t1 WHERE a='string with backslash or quote escapes'; The patch also fixes a few bugs related to string conversion/copying (some of them were reported on JIRA earlier, and some of them I found while working on the patch). Please also find "a.c" attached. I used it to test the improvement. I used a RelWithDebInfo compiled binary for tests. Note, "perf" and "valgrind --tool=cachegrind" also demonstrate a good improvement (reduced CPU and memory usage). I can send their statistics in a separate letter, if anyone is interested. Please review the patch. Thanks.

1 0

[Maria-developers] List affecting CVEs at mariadb.com
by Otto Kekäläinen 26 Aug '14

26 Aug '14

Hello Daniel (and others), The usual changelogs[1] and relese notes[2] don't seem to contain CVE identifiers, or even a separate section about fixed security issues For the downstream security teams if would be reassuring if the CVE information would be easily available. For example if the security teams follow the CVE news and they for example know or suspect that CVE-2014-4260 affects MariaDB, it would be nice to see if it is already fixed or what version it was fixed in, so downstream security teams can organize and prioritize their patching and release work. Do you have any suggestion how to address this? Should we maybe have a separate wiki page, e.g. https://mariadb.com/kb/en/mariadb/cve/ that would have a table of CVEs and MariaDB 5.5/10.0/Galera versions where they are fixed? Or should just each release notes include a subsection "Security" with these details? Something else? Of course we need to consider timing issues, e.g. a security issue fixed in MariaDB might get publicity and a CVE only later when Oracle releases it, and in those cases old release notes need to be upgraded to include the CVE identifiers. [1] https://mariadb.com/kb/en/mariadb-10013-changelog/ [2] https://mariadb.com/kb/en/mariadb-10013-release-notes/ (To be exact, googling for 'mariadb cve' does give one hit at mariadb.com in the 5.3.12 release notes)

3 3

Re: [Maria-developers] [Commits] Rev 4271: MDEV-6601 Assertion `!thd->in_active_multi_stmt_transa ction() || thd->in_multi_stmt_transaction_mode()' failed on executing a stored procedure with commit in lp:~maria-captains/maria/5.5
by Alexey Botchkov 25 Aug '14

25 Aug '14

Hi, Sergei. I'd add SERVER_STATUS_LAST_ROW_SENT to the status_backup_mask. It's not necessary as I don't see it affecting anything, but just to make a sign that we don't want this status to go higher out of the call. Best regards. HF 25.08.2014 10:49, Sergei Golubchik wrote: > At lp:~maria-captains/maria/5.5 > > ------------------------------------------------------------ > revno: 4271 > revision-id: sergii(a)pisem.net-20140825064904-h8isjhv963kbxlzy > parent: knielsen(a)knielsen-hq.org-20140813134639-wk760plnzg5wu4x8 > fixes bug: https://mariadb.atlassian.net/browse/MDEV-6601 > committer: Sergei Golubchik <sergii(a)pisem.net> > branch nick: 5.5 > timestamp: Mon 2014-08-25 08:49:04 +0200 > message: > MDEV-6601 Assertion `!thd->in_active_multi_stmt_transa ction() || thd->in_multi_stmt_transaction_mode()' failed on executing a stored procedure with commit > > Don't restore the whole of thd->server_status after a routine invocation, > only restore SERVER_STATUS_CURSOR_EXISTS, as --ps --embedded needs. > In particular, don't restore SERVER_STATUS_IN_TRANS. > === modified file 'mysql-test/r/sp-bugs.result' > --- a/mysql-test/r/sp-bugs.result 2014-01-24 12:50:18 +0000 > +++ b/mysql-test/r/sp-bugs.result 2014-08-25 06:49:04 +0000 > @@ -268,3 +268,9 @@ END $$ > CALL test_5531(1); > DROP PROCEDURE test_5531; > DROP TABLE t1; > +create procedure sp() begin > +commit; > +end| > +start transaction; > +call sp(); > +drop procedure sp; > > === modified file 'mysql-test/t/sp-bugs.test' > --- a/mysql-test/t/sp-bugs.test 2014-01-24 12:50:18 +0000 > +++ b/mysql-test/t/sp-bugs.test 2014-08-25 06:49:04 +0000 > @@ -285,3 +285,16 @@ DELIMITER ;$$ > CALL test_5531(1); > DROP PROCEDURE test_5531; > DROP TABLE t1; > + > +# > +# MDEV-6601 Assertion `!thd->in_active_multi_stmt_transa ction() || thd->in_multi_stmt_transaction_mode()' failed on executing a stored procedure with commit > +# > +delimiter |; > +create procedure sp() begin > + commit; > +end| > +delimiter ;| > +start transaction; > +call sp(); > +drop procedure sp; > + > > === modified file 'sql/sp_head.cc' > --- a/sql/sp_head.cc 2014-04-15 13:17:47 +0000 > +++ b/sql/sp_head.cc 2014-08-25 06:49:04 +0000 > @@ -1224,6 +1224,7 @@ sp_head::execute(THD *thd, bool merge_da > Item_change_list old_change_list; > String old_packet; > uint old_server_status; > + const uint status_backup_mask= SERVER_STATUS_CURSOR_EXISTS; > Reprepare_observer *save_reprepare_observer= thd->m_reprepare_observer; > Object_creation_ctx *saved_creation_ctx; > Warning_info *saved_warning_info; > @@ -1358,7 +1359,7 @@ sp_head::execute(THD *thd, bool merge_da > It is probably safe to use same thd->convert_buff everywhere. > */ > old_packet.swap(thd->packet); > - old_server_status= thd->server_status; > + old_server_status= thd->server_status & status_backup_mask; > > /* > Switch to per-instruction arena here. We can do it since we cleanup > @@ -1488,7 +1489,7 @@ sp_head::execute(THD *thd, bool merge_da > thd->spcont->pop_all_cursors(); // To avoid memory leaks after an error > > /* Restore all saved */ > - thd->server_status= old_server_status; > + thd->server_status= (thd->server_status & ~status_backup_mask) | old_server_status; > old_packet.swap(thd->packet); > DBUG_ASSERT(thd->change_list.is_empty()); > old_change_list.move_elements_to(&thd->change_list); > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Getting Started
by Ashish Kedia 23 Aug '14

23 Aug '14

Hi all, I am 3rd Year undergraduate student majoring in Information Technology from India. I am new to MariaDB however I wish to contribute. I am well acquainted with C++ and other tools required. Can someone help me find small tasks/bugs/projects to get started with ? Regards, Ashish Kedia, 3rd Year Undergraduate Student, Dept. of Information Technology, National Institute of Technology Karnataka, Surathkal

3 2

[Maria-developers] Closing bugs as Incomplete/Cannot reproduce requires FixVersion?
by Sergey Petrunia 23 Aug '14

23 Aug '14

Hi Serg, (writing to you because I think it was you who set Jira's rules for Fix Version field). I'm trying to close MDEV-6032 as "Incomplete" ("Cannot reproduce" would also work). This resolution is meant to say that the issue is not really resolved. And Jira's "Close Issue" dialog gives me this error: Field Fix Version/s with actual value does not match regular expression \[([0-9]+\.[0-9]+\.[0-9]+.*|N/A)\] I think it doesn't make sense. I even think that "Fix Version" must be empty for bugs that are closed with resolution other than "Fixed" or "Duplicate". Do you agree with this reasoning? If yes, is it possible to fix Jira to follow this logic? BR Sergei -- Sergei Petrunia, Software Developer MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Learning about MariaDB
by Kavneet Kaur 22 Aug '14

22 Aug '14

Hi! I am highly interested in subject of database. I am also interested in becoming a research developer. Worked on MS SQL. Want to try MariaDB. May I get the guidance about this product as i'm totally new to it. Is there any specific way to begin?

1 0

[Maria-developers] [GSoC] Pencils down
by sriram patil 21 Aug '14

21 Aug '14

Hi All, Following is the link to my pencils down blog post. You can go through it even if you have not followed my blog posts before. http://goo.gl/DqBgDe Thanks a lot for all your feedback and suggestions. Sriram.

3 3

[Maria-developers] [GSOC2014] UseServerPrepStmts - Connector/J
by Puneet Dewan 20 Aug '14

20 Aug '14

Hi Everyone, I have pushed the code and it can be found on the following link: https://code.launchpad.net/~team-java-connector/mariadb-java-client/mariadb… Thanks for the support. :) Please review and give suggestions. Regards Puneet.

1 0

Re: [Maria-developers] [GSoC] Optimize mysql-test-runs - Results of new strategy
by Pablo Estrada 19 Aug '14

19 Aug '14

Hello Elena and all, First, addressing the previous email: Looking at the dump, I see it can also happen that the dump contains > several records for a pair platform/bbnum. I am not sure why it happens, I > think it shouldn't, might be a bug in buildbot and/or configuration, or > environmental problems. Anyway, due to the way we store output files, they > can well override each other in this case, thus for several platform/bbnum > record you will have only one file. I suppose that's what was hard to > resolve, sorry about that. > No worries ; ). There are several cases where platform and build number are the same. The system just names the files as follows: <platform>_<build_id>-log-test_1-stdio <platform>_<build_id>-log-test_2-stdio ..... <platform>_<build_id>-log-test_5-stdio These files seem to correspond temporarily with the test runs (*test_1-stdio) belongs to the first test_run of the same plt-bnum, and so on. Unfortunately, there are some cases where there are more test_runs on the dump than files available, and this means that it's impossible to be sure which file belongs to which test_run exactly. > You should consider skipped tests, at least for now. Your logic that they > are skipped because they can't be run is generally correct; unfortunately, > MTR first produces the *full* list of tests to run, and determines whether > a test can be run or not on a later stage, when it starts running the > tests. Your tool will receive the initial test list, and I'm not sure it's > realistic to re-write MTR so that it takes into account limitations that > cause skipping tests before creating the list. > I see. Okay then, duly noted. Possibly it's better to skip a test run altogether if there is no input > list for it; it would be definitely the best if there were 5K (or whatever > slice you are currently using) of continuous test runs with input lists; if > it so happens that there are lists for some branches but not others, you > can skip the branch entirely. > This doesn't seem like a good option. Recall drops seriously, and the test_runs that have a corresponding file don't seem to have a special pattern, and tend to have long spaces between them, so the information becomes irrelevant, and seemingly, not useful. > The core module should take as parameters > - list of tests to choose from, > - size of the running set (%), > - branch/platform (if we use them in the end), > and produce a new list of tests of the size of the running set. > > The wrapper module should > - read the list of tests from the outside world (for now, from a file), > - receive branch/platform as command-line options, > - have the running set size set as an easily changeable constant or as a > configuration parameter, > > and return the list of tests -- lets say for now, in the form of <test > suite>.<test name>, blank-separated, e.g. > main.select innondb.create-index ... > >> >> I am almost done 'translating' the code into a solution that divides it in 'core' and 'wrapper'. There are a few bugs that I still haven't figured out, but I believe I can iron those out pretty soon. I will also incorporate the percentage rather than fixed running_set. Now, regarding the state of the project (and the recall numbers that I am able to achieve so far), here are some observations: - Unfortunately, I am running out of ideas to try to improve recall. I tried tuning some parameters, giving more weight to ones or others, etc. I still wasn't able to push recall beyond ~87% on the strategy that uses file correlations. For what I've seen, some failures are just extremely hard to predict. - The strategy that uses only a weighted average of the failure frequency achieves a higher recall, but for a shorter time. The recall decays quickly afterwards. I may try to add some file-correlations to this strategy, to see if the recall can be sustained for a longer term. - There is one problem that I see regarding the data and the potential real-world implementation of the program: By verifying the recall with the historical data, we run the possibility of 'expecting' overfitting... so the results regarding the errors found when comparing to the historical data, and the results that could have been obtained by a real-world implementation are potentially different. A possible way to address that issue would require modifying the buildbot to gather more data over a longer term. So having said that, I am looking for some advice in the following regards: - I will try to take a step back from the new strategy, and see how I can adapt the original strategy to prevent the recall function from declining so sharply with time. - I will also spend some time keeping a codebase that adjusts better to the model that we need for the implementation. I will upload code soon. All suggestions are welcome. - Nonetheless, I feel that more data would allow to improve the algorithm greatly. Is it possible to prepare logging into the buildbot that would allow for more precise data collection? A slower, more iterative process, working closer with the buildbot and doing more detailed data collection might deliver better results. (I understand that this would probably influence the time-scope of the project) Let me know what you think about my suggestions. Regards Pablo

2 22