- developers - lists.mariadb.org

Re: [Maria-developers] [Commits] 56d339111a4: MDEV-16374: Filtered shows 0 for materilization scan for a semi join, which makes optimizer always picks
by Sergey Petrunia 05 Jun '18

05 Jun '18

Hi Varun, On Mon, Jun 04, 2018 at 12:34:49PM +0530, Varun wrote: > revision-id: 56d339111a4e5c868ffa3b1f2538c5f101f67767 (mariadb-10.0.30-366-g56d339111a4) > parent(s): a61724a3ca187f6eb44fc67948acb76a94efa783 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-06-04 12:26:56 +0530 > message: > > MDEV-16374: Filtered shows 0 for materilization scan for a semi join, which makes optimizer always picks > materialization scan over materialization lookup > > For non-mergeable semi-joins we don't store the estimates of the IN subquery in table->file->stats.records. > In the function TABLE_LIST::fetch_number_of_rows, we store the number of rows in the tables > (estimates in case of derived table/views). > Currently we don't store the estimates for non-mergeable semi-joins, which leads to a problem of selecting > materialization scan over materialization lookup. > Fixed this by storing these estimated appropriately > * This is a change in the optimizer. Should we really do it in 10.0? In my opinion, this should go into 10.3 or 10.4. Let's discuss this, and I think Igor may have something to say, too. * The patch updates selectivity.test but not selectivity_innodb.result (which includes selectivity.test). * Please see my last comments at https://jira.mariadb.org/browse/MDEV-16374 . Here is an alternative patch (it passes the testsuite, and produces the same result for selectivity* tes): diff --git a/sql/sql_select.cc b/sql/sql_select.cc index 1285835..884997c 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -6602,7 +6602,7 @@ double matching_candidates_in_table(JOIN_TAB *s, bool with_found_constraint, { TABLE *table= s->table; double sel= table->cond_selectivity; - double table_records= (double)table->stat_records(); + double table_records= s->records; // psergey-10 dbl_records= table_records * sel; return dbl_records; } What do you think about it? I would like to get Igor's opinion also because I'm not certain which member variable should be changed here. (Why does derived table modify table->stats.records while JTBM semi-joins modify JOIN_TAB::records? These seem like two solutions for the same problem?) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Proposal for Negative Grants Project
by Rutuja Surve 04 Jun '18

04 Jun '18

Hello, Please find the proposal for the Negative Grants project attached along with this e-mail. It would be great to hear your opinions and suggestions on the approach proposed here, before delving into its implementation. Some context about the project: *The current privilege system allows access in this manner:GRANT SELECT ON some_database.* TO a_user@%;Any revoke would revoke access from the entire database. We want to support a revoke that would disable select on a specific table, like:REVOKE SELECT ON some_database.secret_table FROM a_user@%;Reference:[1] https://jira.mariadb.org/browse/MDEV-14443 <https://jira.mariadb.org/browse/MDEV-14443>[2] https://mariadb.com/kb/en/library/google-summer-of-code-2018/ <https://mariadb.com/kb/en/library/google-summer-of-code-2018/>Thanks,Rutuja*

3 6

[Maria-developers] Conservative parallel slave is "too optimistic" for certain engines
by Sergey Petrunia 01 Jun '18

01 Jun '18

Hello, This comes from MDEV-16242. == Symptoms == When one runs a parallel slave (mode=conservative) and replicates DML for MyRocks table without a Primary Key, replication may stop with a ER_KEY_NOT_FOUND error. This may happen even if the queries were run on the master sequentially. == A detail about conservative replication == Suppose the master runs these two statements, sequentially: INSERT INTO t1 VALUES (5),(6),(7); DELETE FROM t1; Parallel slave may schedule the INSERT to Thread1, and the DELETE to Thread2. In conservative parallel replication, the DELETE "will wait" to be executed after the INSERT. One may expect that "will wait" here means that DELETE execution does not start until the INSERT has committed. But it's more subtle than that: actually, DELETE execution will start as soon as the INSERT is ready to do the prepare/commit steps. I assume this was done to increase the parallelism: initial phases of DELETE can be ran in parallel with commit/prepare steps of the INSERT. This is safe to do: 1. The INSERT has acquired locks for all rows it is about to touch. DELETE will not be able to prevent INSERT from committing. 2. The DELETE starts its execution on a database that doesn't include the results of the INSERT (the INSERT has not committed yet!). But this is fine, because the DELETE locks the rows it is about to modify. If it attempts to access a row that the INSERT is about to insert, it will block on a lock until the INSERT finishes. == Applying this to MyRocks == The critical part is #2. It works if the DELETE command will acquire row locks for rows it about to DELETE. This is normally true: If the storage engine supports gap locks, attempting to read a locked gap will cause DELETE to wait for INSERT If the storage engine doesn't support gap locks, it will use Row-Based-Replication. RBR's Delete_rows_event includes Primary Key value of the row that should be deleted. Doing a point lookup on PK will cause the DELETE to block on the row lock. The remaining case is - storage engine that doesn't support gap locks - Row-based-replication of table without primary key In this case, finding the row to delete is done by scanning the table until we find the row where all columns match. The scan does not see the rows that the INSERT is about to INSERT, and DELETE stops with an error, "we could not find the row to delete". == Possible solutions == 1. Implement gap locking in MyRocks. This will take some time. 2. Change parallel slave to wait *for commit*. This should only be done if tables that are updated do not support Gap Locking. This is hard, it will require making risky changes on the SQL layer. 3. Disallow parallel slave execution for tables without Gap Locking and without primary key. (Looks most promising but I am not sure if it is sufficient). BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 2

[Maria-developers] Test failure overview for Debian/Ubuntu on 10.3 master branch (June 1st 2018 status report)
by Otto Kekäläinen 01 Jun '18

01 Jun '18

Hello! I reported here about the test failure status in early April and here is a follow-up. My old emails can be found here: https://lists.launchpad.net/maria-developers/msg11171.html https://lists.launchpad.net/maria-developers/msg11170.html If one looks at the picture from April and compares to the status today, the situation is a bit sad as we have even less green than before: https://lists.launchpad.net/maria-developers/pngcpxTB4wjTW.png Current issues we have in addition to the usual mtr test failures are: - On multiple bionic builds the upgrade fails to run with no apparent reason. The builder script does not seem to find any previous installation of anything, so that probably is missing from the images: + dpkg -l + grep -iE 'maria|mysql|galera' + true Thus the build fails. Example log: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-bionic-amd64/bui lds/196/steps/upgrade2/logs/stdio - On bionic and xenial aarch64 mysqld fails to start. In syslog we can see May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 0 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGN ORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION',' NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_O PTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANS I','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES' ,'STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES',' ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_N OT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH','EMPT Y_STRING_IS_NULL','SIMULTANEOUS_ASSIGNMENT'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGN ORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION',' NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_O PTIONS','NO_TABLE_OPTIONS','NO May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: _FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALU May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 0 [ERROR] mysqld: Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler. May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 6 [ERROR] InnoDB: Column last_update in table `mysql`.`innodb_table_stats` is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch). May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 6 [ERROR] InnoDB: Fetch of persistent statistics requested for table `mysql`.`gtid_slave_pos` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. http://buildbot.askmonty.org/buildbot/builders/kvm-deb-xenial-aarch64/b uilds/277/steps/upgrade1/logs/syslog - stretch amd64 fails to compile due to librocksdblib.a(compaction_job.cc.o): In function `rocksdb::ZSTD_TrainDictionary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long)': ./builddir/storage/rocksdb/./storage/rocksdb/rocksdb/util/compression.h :812: undefined reference to `ZDICT_trainFromBuffer' ./builddir/storage/rocksdb/./storage/rocksdb/rocksdb/util/compression.h :813: undefined reference to `ZDICT_isError' collect2: error: ld returned 1 exit status http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/bu ilds/3986/steps/compile/logs/stdio - jessie amd64 builds to compile due to /usr/bin/ld.bfd.real: /tmp/ccOIwjFo.ltrans0.ltrans.o: relocation R_X86_64_PC32 against symbol `toku_product_name_strings' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld.bfd.real: final link failed: Bad value collect2: error: ld returned 1 exit status http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/bui lds/4298/steps/compile/logs/stdio This -fPIC issue might be related to a change in hardening flags I "forward-ported" from Debian and I will look into it. All the other issues are things I cannot do anything about.. :( On the bright side I can report that there are at least no build regressions on Launchpad with current 10.3: https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.3/+build s?build_text=&build_state=all - Otto

1 0

[Maria-developers] With_sum_func_cache
by Alexander Barkov 31 May '18

31 May '18

Hello Sanja, I recently worked on MDEV-16309 and had hard time to find which Item classes in the hierarchy: - have always with_sum_func==true - have always with_sum_func==false - have variable with_sum_func To make it sure, before actually working on MDEV-16309, I had to create a separate working tree and did with Item::with_sum_func the same change that we previously did for Item::with_subselect in: MDEV-14517 Cleanup for Item::with_subselect and Item::has_subquery() (which you reviewed) - I find the code easier to read this way (instead of searching for all possible assignments, one can search who overrides the method) - Also, some Item can become smaller in size. It's pity to throw the patch away. So perhaps we could just apply it. Can you please have a look? Thanks. I the meanwhile I'll create a new MDEV for it (with a similar description to MDEV-14517)

2 2

[Maria-developers] Open and close a TABLE_LIST at random place in code
by Eugene Kosov 31 May '18

31 May '18

Hello. I need to see a table structure while executing some query (DELETE in my case, but there will be more query types in future). I have a TABLE_LIST which came from a parser. Now I'm opening it with open_table_only_view_structure(), examine table structure and close it, trying to prevent any side effects. And there is a lot of them. Current code looks like this: if (open_tables_only_view_structure(thd, table_list, false)) { return true; } // examine table structure close_mysql_tables(thd); for (Sroutine_hash_entry *e = thd->lex->sroutines_list.first;e; e=e->next) e->mdl_request.ticket= NULL; for (TABLE_LIST *it= table_list; it; it= it->next_global) { it->table= NULL; it->reinit_before_use(thd); //it->next_global= NULL; it->derived_result= NULL; it->derived= NULL; it->view= NULL; it->view_tables= NULL; it->view_sctx= NULL; it->file_version= 0; it->mariadb_version= 0; it->updatable_view= 0; it->view_suid= 0; it->derived_type= 0; it->open_type= OT_BASE_ONLY; it->view_creation_ctx= 0; // it->set_table_ref_id(TABLE_REF_NULL, 0); } Is there a way to close a TABLE_LIST gracefully without touching a lot of it's fields? Or even maybe it's possible to open a table in a simpler manner? -- Eugene

2 3

Re: [Maria-developers] [Commits] de722585c1b: MDEV-16225: wrong resultset from query with semijoin=on
by Sergey Petrunia 27 May '18

27 May '18

Hi Varun, On Thu, May 24, 2018 at 03:17:02AM +0530, Varun wrote: > revision-id: de722585c1b2ad2af99bd1749e8566ea77838764 (mariadb-5.5.56-219-gde722585c1b) > parent(s): 1ada4afb0a51f7283b6187a95019ec2cb80c8a0b > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-05-24 03:09:42 +0530 > message: > > MDEV-16225: wrong resultset from query with semijoin=on > > For non-semi-join subquery optimization we do a cost based decision between > Materialisation and IN -> EXIST transformation. The issue in this case is that for IN->EXIST transformation > we run JOIN::reoptimize with the IN->EXISt conditions and we come up with a new query plan. But when we compare > the cost with Materialization, we make the decision to chose Materialization so we need to restore the query plan > for Materilization. The restoring of query plan is not handled correctly. > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index d6d269a700f..b8d43b9797e 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -23805,6 +23805,22 @@ void JOIN::restore_query_plan(Join_plan_state *restore_from) > } > > } > + else > + { > + /* > + After running JOIN::reoptimize for IN->EXISTS > + transformation we might add elements to the keyuse array, > + The original query plan was that the keyuse array had no elements, so we > + should restore that state here. > + */ > + while(keyuse.elements) > + delete_dynamic_element(&keyuse, 0); I'm wondering, why do we need the above loop and not just delete everything with delete_dynamic(). But if I don't miss anything, this is not significant. Ok to push either the above variant or one with delete_dynamic(). > + for (uint i= 0; i < table_count; i++) > + { > + join_tab[i].keyuse= NULL; > + join_tab[i].checked_keys.clear_all(); > + } > + } > memcpy((uchar*) best_positions, (uchar*) restore_from->best_positions, > sizeof(POSITION) * (table_count + 1)); > /* Restore SJM nests */ > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Maria-discuss] Query ambiguity
by Karthick Subramanian 17 May '18

17 May '18

Check whether you really need a comma here: *metrics_5.id <http://metrics_5.id/>,* *(SELECT * Also in your screen shot, AS 'LT CPC`, there is no comma and you use derived table in SELECT and trying to use JOIN in the FROM using this. Sorry to say this, I think you can either break this logic into small simple chunks and use SP to derive the logic in case you are not much comfortable writing a complex SQL. I think there is lot of syntax errors on it. On Thu, May 17, 2018 at 10:05 AM, Dev C <chauhan.devsur(a)gmail.com> wrote: > Here is the full code and picture on what is required. > > SELECT > campaigns_5.name AS `Campaign Name`, > adgroups_5.name AS `Adgroup Name`, > STNegData1W.query, > keywords_5.matchType AS `Match Type`, > Sum(metrics_5.clicks) AS `LT Clicks`, > Sum(metrics_5.costs) AS `LT Spend`, > Sum(metrics_5.orders) AS `LT Orders`, > metrics_5.sales AS `LT Sales`, > If(Sum(metrics_5.sales)=0,0,Sum(metrics_5.costs)/Sum(metrics_5.sales)) > AS `LT ACOS`, > If(Sum(clicks)=0,0,Sum(costs)/Sum(clicks)) AS `LT CPC` > FROM (((keywords_5 INNER JOIN campaigns_5 ON keywords_5.campaignId = > campaigns_5.campaignId) INNER JOIN adgroups_5 ON keywords_5.adGroupId = > adgroups_5.adGroupId) INNER JOIN STNegData1W ON keywords_5.keywordId = > STNegData1W.keywordId) INNER JOIN metrics_5 ON STNegData1W.metricId = *metrics_5.id > <http://metrics_5.id>,* > *(SELECT * > STNegData4W.query, > STNegData4W.metricId, > STNegData4W.keywordId, > metrics_5.impressions > FROM > (SELECT searchterm_5.query, searchterm_5.metricId, > searchterm_5.keywordId, Sum(metrics_5.orders) AS `4W Orders` > FROM searchterm_5 INNER JOIN metrics_5 ON searchterm_5.metricId = > metrics_5.id > GROUP BY searchterm_5.query, searchterm_5.metricId, > searchterm_5.keywordId, metrics_5.created_at > HAVING (((Sum(metrics_5.orders))=0) AND ((metrics_5.created_at) > > DATE_SUB('2018-05-10',INTERVAL 28 DAY)))) AS STNegData4W > INNER JOIN > metrics_5 ON STNegData4W.metricId = metrics_5.id > GROUP BY STNegData4W.query, STNegData4W.metricId, STNegData4W.keywordId, > metrics_5.impressions, metrics_5.created_at > HAVING (((metrics_5.impressions)>0) AND ((metrics_5.created_at)> > DATE_SUB('2018-05-10', INTERVAL 7 DAY)))) AS STNegData1W > GROUP BY campaigns_5.name, adgroups_5.name, STNegData1W.query, > keywords_5.matchType, metrics_5.sales, campaigns_5.user_market_id, Right( > campaigns_5.name,4) > HAVING (((Sum(metrics_5.clicks))>5) AND ((campaigns_5.user_market_id)=12) > AND ((Right(campaigns_5.name,4))="PFUS")) > > On Thu, May 17, 2018 at 4:06 PM, Pantelis Theodosiou <ypercube(a)gmail.com> > wrote: > >> What Rhys said, please post queries as text, not as images. >> >> The issue is likely due to the way you structured your query. You have >> put a derived table in a random place (in the SELECT list). >> >> Please try to read about CTEs. It will help you design those complex >> queries better and make them more readable and easy to edit: >> https://mariadb.com/kb/en/library/with/ >> >> Best regards >> >> Pantelis Theodosiou >> >> On Thu, May 17, 2018 at 11:23 AM, <Rhys.Campbell(a)swisscom.com> wrote: >> >>> You're not making a lot of sense here. Please explain the "ambiguity". >>> That's a fairly complex query so you should include as text. >>> >>> >>> >>> "Does not work" <- In the image. It helps if you include error message. >>> >>> >>> >>> *From:* Maria-discuss [mailto:maria-discuss-bounces+rhys.campbell= >>> swisscom.com(a)lists.launchpad.net] *On Behalf Of *Dev C >>> *Sent:* 17 May 2018 06:41 >>> *To:* maria-discuss(a)lists.launchpad.net; Maria Developers < >>> maria-developers(a)lists.launchpad.net> >>> *Subject:* [Maria-discuss] Query ambiguity >>> >>> >>> >>> Hello >>> >>> >>> >>> I have query ambiguity as attached. Please help me on how to solve that. >>> >>> >>> >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~maria-discuss >>> Post to : maria-discuss(a)lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~maria-discuss >>> More help : https://help.launchpad.net/ListHelp >>> >>> >> > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp > >

1 0

Re: [Maria-developers] [Maria-discuss] Query ambiguity
by Pantelis Theodosiou 17 May '18

17 May '18

What Rhys said, please post queries as text, not as images. The issue is likely due to the way you structured your query. You have put a derived table in a random place (in the SELECT list). Please try to read about CTEs. It will help you design those complex queries better and make them more readable and easy to edit: https://mariadb.com/kb/en/library/with/ Best regards Pantelis Theodosiou On Thu, May 17, 2018 at 11:23 AM, <Rhys.Campbell(a)swisscom.com> wrote: > You're not making a lot of sense here. Please explain the "ambiguity". > That's a fairly complex query so you should include as text. > > > > "Does not work" <- In the image. It helps if you include error message. > > > > *From:* Maria-discuss [mailto:maria-discuss-bounces+rhys.campbell= > swisscom.com(a)lists.launchpad.net] *On Behalf Of *Dev C > *Sent:* 17 May 2018 06:41 > *To:* maria-discuss(a)lists.launchpad.net; Maria Developers < > maria-developers(a)lists.launchpad.net> > *Subject:* [Maria-discuss] Query ambiguity > > > > Hello > > > > I have query ambiguity as attached. Please help me on how to solve that. > > > > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp > >

2 1

Re: [Maria-developers] [Maria-discuss] Interval in MYsql query
by Peter Laursen 16 May '18

16 May '18

Use the DATE_ADD( ) or DATE_SUB() function with INTERVAL as explained here: https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html#functi… -- Peter. On Wed, May 16, 2018 at 4:07 PM, Dev C <chauhan.devsur(a)gmail.com> wrote: > Hello > > If I use below query > (metrics_5.created_at)>'5/10/2018'- 28 ) > it gives me some result > but if i use below query > (metrics_5.created_at)>'5/10/2018'- INTERVAL 28 DAY ) > > It does not give me any result, any one know why ? > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp > >

1 0