- developers - lists.mariadb.org

[Maria-developers] FOR UPDATE behavior change in 10.3-MDEV-11953
by Alexander Barkov 12 Apr '18

12 Apr '18

Hi Sanja, and Igor, (resending with Igor on CC) I noticed a FOR UPDATE behavior change in 10.3-MDEV-11953 In 10.3 I run this script in a client session: CREATE OR REPLACE TABLE t1 (a INT NOT NULL PRIMARY KEY); CREATE OR REPLACE TABLE t2 (a INT NOT NULL PRIMARY KEY); INSERT INTO t1 VALUES (1); INSERT INTO t2 VALUES (2); BEGIN; SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE; Notice no COMMIT yet! It returns this result: +---+ | a | +---+ | 1 | | 2 | +---+ Now I open a new console, start a new client session and run this script: BEGIN; SELECT * FROM t2 FOR UPDATE; The second session gets locked, as expected. Now I return to the first console and run "COMMIT;". The second console gets unlocked and returns +---+ | a | +---+ | 2 | +---+ Now if I do the same in 10.3-MDEV-11953, the second session does not get locked, it returns the result immediately, even before I typed "COMMIT;" in the first session. Note, if in 10.3-MDEV-11953 I start a transaction with a simpler query (without UNION) in the first session: BEGIN; SELECT * FROM t2 FOR UPDATE; and run the same script in the second session BEGIN; SELECT * FROM t2 FOR UPDATE; then the second session gets locked as expected. Is this behavior change expected? Thanks!

2 1

Re: [Maria-developers] FOR UPDATE behavior change in 10.3-MDEV-11953
by Alexander Barkov 12 Apr '18

12 Apr '18

Hi Igor, Please do "reply all" when replying. On 04/12/2018 01:28 AM, Igor Babaev wrote: > Alexander, > > If you have the same behaviour you have to use > > SELECT * FROM t1 UNION (SELECT * FROM t2 FOR UPDATE). Thanks. This worked fine for me. > > Otherwise it's considered as > > (SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE). What does this syntax mean 10.3-MDEV-11953? Which records should be locked? From t1, from t2, or from both? It seems that it locks nothing. If so, shouldn't this be disallowed? This query (without parentheses) also locks nothing: SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE; > > MySQL manual says that 'FOR UPDATE' must follow ORDER BY, > > but it does not tell you how to interpret FOR UPDATE in unions. The old behavior was to interpret UNION parts separately. This locked only t2 in 10.3: SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE; In 10.3-MDEV-11953 it locks nothing. Upgrade looks dangerous. Can we do this way: 1. In 10.4: disallow FOR UPDATE applied to the entire UNION. 2. In 10.3: Add a warning which tells that this feature is deprecated and suggests to use parentheses around union parts when a non-parenthesized union term is used with FOR UPDATE ? > > Anyway you can't use anymore: > SELECT c FROM t1 ORDER BY a > UNION > SELECT c FROM t2 ORDER BY b > ORDER BY c. > > You have to put it so: > (SELECT c FROM t1 ORDER BY a) > UNION > (SELECT c FROM t2 ORDER BY b) > ORDER BY c. > > (see the standard). I like this change. > > It means that the ORDER BY clause followed a union relates to the union. > The same is true for 'FOR UPDATE' because it follows optional ORDER BY/ > LIMIT clauses. > > > Regards, > Igor. > > > On 04/11/2018 09:10 AM, Alexander Barkov wrote: >> Hi Sanja, and Igor, >> >> >> (resending with Igor on CC) >> >> >> I noticed a FOR UPDATE behavior change in 10.3-MDEV-11953 >> >> In 10.3 I run this script in a client session: >> >> CREATE OR REPLACE TABLE t1 (a INT NOT NULL PRIMARY KEY); >> CREATE OR REPLACE TABLE t2 (a INT NOT NULL PRIMARY KEY); >> INSERT INTO t1 VALUES (1); >> INSERT INTO t2 VALUES (2); >> BEGIN; >> SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE; >> >> Notice no COMMIT yet! It returns this result: >> +---+ >> | a | >> +---+ >> | 1 | >> | 2 | >> +---+ >> >> Now I open a new console, start a new client session and run this script: >> >> BEGIN; >> SELECT * FROM t2 FOR UPDATE; >> >> The second session gets locked, as expected. >> >> Now I return to the first console and run "COMMIT;". >> The second console gets unlocked and returns >> +---+ >> | a | >> +---+ >> | 2 | >> +---+ >> >> >> Now if I do the same in 10.3-MDEV-11953, the second session does not get >> locked, it returns the result immediately, even before I typed "COMMIT;" >> in the first session. >> >> >> Note, if in 10.3-MDEV-11953 I start a transaction with a simpler query >> (without UNION) in the first session: >> >> BEGIN; >> SELECT * FROM t2 FOR UPDATE; >> >> and run the same script in the second session >> >> BEGIN; >> SELECT * FROM t2 FOR UPDATE; >> >> then the second session gets locked as expected. >> >> >> Is this behavior change expected? >> >> >> Thanks! >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~maria-developers >> Post to : maria-developers(a)lists.launchpad.net >> Unsubscribe : https://launchpad.net/~maria-developers >> More help : https://help.launchpad.net/ListHelp >>

2 3

[Maria-developers] Issue with setting up of MariaDB and solve issue MDEV-14896
by Aman Jain 11 Apr '18

11 Apr '18

Hello developers I am setting up the codebase for MariaDB on my system to understand the codebase and solve some issues. But I am facing a problem which I am not able to solve when I run "make" command. Error log: https://pastebin.com/u5fAhMHX OS: MacOS 10.13.3 Python version: 2.7.14 Please let me know how can I set up so that I can run and test the changes. Also, I want to solve issue MDEV-14896. Please connect me to the owner of the issue so that I can discuss the changes. ( https://jira.mariadb.org/browse/MDEV-14896 ) Thanks Aman Jain

3 5

[Maria-developers] 10.3-MDEV-11953
by Alexander Barkov 11 Apr '18

11 Apr '18

Hi Sanja and Igor, In 10.3-MDEV-11953 you're adding a number of main_select_push() / pop_select() around "expr" and other rules that contain "expr". How do you decide when push/pop is needed. Why, for example, LIMIT and ORDER clause don't have their own push/pop. They do create some expressions (Items). What is the rule? Thanks.

1 0

[Maria-developers] FOR UPDATE behavior change in 10.3-MDEV-11953
by Alexander Barkov 11 Apr '18

11 Apr '18

Hi Sanja, I noticed a FOR UPDATE behavior change in 10.3-MDEV-11953 In 10.3 I run this script in a client session: CREATE OR REPLACE TABLE t1 (a INT NOT NULL PRIMARY KEY); CREATE OR REPLACE TABLE t2 (a INT NOT NULL PRIMARY KEY); INSERT INTO t1 VALUES (1); INSERT INTO t2 VALUES (2); BEGIN; SELECT * FROM t1 UNION SELECT * FROM t2 FOR UPDATE; Notice no COMMIT yet! It returns this result: +---+ | a | +---+ | 1 | | 2 | +---+ Now I open a new console, start a new client session and run this script: BEGIN; SELECT * FROM t2 FOR UPDATE; The second session gets locked, as expected. Now I return to the first console and run "COMMIT;". The second console gets unlocked and returns +---+ | a | +---+ | 2 | +---+ Now if I do the same in 10.3-MDEV-11953, the second session does not get locked, it returns the result immediately, even before I typed "COMMIT;" in the first session. Note, if in 10.3-MDEV-11953 I start a transaction with a simpler query (without UNION) in the first session: BEGIN; SELECT * FROM t2 FOR UPDATE; and run the same script in the second session BEGIN; SELECT * FROM t2 FOR UPDATE; then the second session gets locked as expected. Is this behavior change expected? Thanks!

1 0

Re: [Maria-developers] 57673bc: MDEV-14024 PCRE2
by Sergei Golubchik 11 Apr '18

11 Apr '18

Hi, Alexey! On Apr 03, Alexey Botchkov wrote: > revision-id: 57673bcfeecf51d1dd8f4907adc8ca11d84796d3 (mariadb-10.3.5-76-g57673bc) > parent(s): d614d4b2e61552a5b74259f91b3e14b23707743f > committer: Alexey Botchkov > timestamp: 2018-04-03 02:59:51 +0400 > message: > > MDEV-14024 PCRE2. > > server ocde switched to the PCRE2 features. ^^^ s/ocde/code/ > diff --git a/cmake/pcre.cmake b/cmake/pcre.cmake > index 4c11392..271dd06 100644 > --- a/cmake/pcre.cmake > +++ b/cmake/pcre.cmake > @@ -5,24 +5,16 @@ SET(WITH_PCRE "auto" CACHE STRING > > + IF(NOT HAVE_PCRE2 OR WITH_PCRE STREQUAL "bundled") > IF (WITH_PCRE STREQUAL "system") > - MESSAGE(FATAL_ERROR "system pcre is not found or unusable") > + MESSAGE(FATAL_ERROR "system pcre2-8 is not found or unusable") better "pcre2-8 library" or simply "pcre2" > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index 89aa307..08aa293 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -5481,15 +5481,6 @@ int Regexp_processor_pcre::default_regex_flags() > return default_regex_flags_pcre(current_thd); > } > > -void Regexp_processor_pcre::set_recursion_limit(THD *thd) > -{ > - long stack_used; > - DBUG_ASSERT(thd == current_thd); > - stack_used= available_stack_size(thd->thread_stack, &stack_used); > - m_pcre_extra.match_limit_recursion= > - (ulong)((my_thread_stack_size - STACK_MIN_SIZE - stack_used)/my_pcre_frame_size); > -} > - > > /** > Convert string to lib_charset, if needed. > @@ -5537,19 +5528,32 @@ bool Regexp_processor_pcre::compile(String *pattern, bool send_error) > if (!(pattern= convert_if_needed(pattern, &pattern_converter))) > return true; > > - m_pcre= pcre_compile(pattern->c_ptr_safe(), m_library_flags, > - &pcreErrorStr, &pcreErrorOffset, NULL); > + m_pcre= pcre2_compile((PCRE2_SPTR8) pattern->ptr(), pattern->length(), > + m_library_flags, > + &pcreErrorNumber, &pcreErrorOffset, NULL); > > if (m_pcre == NULL) > { > if (send_error) > { > char buff[MAX_FIELD_WIDTH]; > - my_snprintf(buff, sizeof(buff), "%s at offset %d", pcreErrorStr, pcreErrorOffset); > + int lmsg= pcre2_get_error_message(pcreErrorNumber, > + (PCRE2_UCHAR8 *)buff, sizeof(buff)); > + if (lmsg >= 0) > + my_snprintf(buff+lmsg, sizeof(buff)-lmsg, > + " at offset %d", pcreErrorOffset); if lmsg < 0, does it guarantee that the buffer is 0-terminated? > my_error(ER_REGEXP_ERROR, MYF(0), buff); > } > return true; > } > + > + m_pcre_match_data= pcre2_match_data_create_from_pattern(m_pcre, NULL); > + if (m_pcre_match_data == NULL) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + return true; > + } > + > return false; > } > > @@ -5570,124 +5574,43 @@ bool Regexp_processor_pcre::compile(Item *item, bool send_error) > */ > void Regexp_processor_pcre::pcre_exec_warn(int rc) const > { ... > - push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > - ER_REGEXP_ERROR, ER_THD(thd, ER_REGEXP_ERROR), errmsg); > + int errlen= pcre2_get_error_message(rc, buf, sizeof(buf)); > + if (errlen >= 0) > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_REGEXP_ERROR, ER_THD(thd, ER_REGEXP_ERROR), buf); perhaps, issue some generic message if pcre2_get_error_message fails? > } > @@ -5697,14 +5620,12 @@ bool Regexp_processor_pcre::exec(String *str, int offset, > { > if (!(str= convert_if_needed(str, &subject_converter))) > return true; > - m_pcre_exec_rc= pcre_exec_with_warn(m_pcre, &m_pcre_extra, > - str->c_ptr_safe(), str->length(), > - offset, 0, > - m_SubStrVec, array_elements(m_SubStrVec)); > + m_pcre_exec_rc= pcre_exec_with_warn(m_pcre, m_pcre_match_data, > + str->ptr(), str->length(), offset, 0); > if (m_pcre_exec_rc > 0) > { > uint i; > - for (i= 0; i < n_result_offsets_to_convert; i++) > + for (i= 0; i < n_result_offsets_to_convert && i < m_pcre_exec_rc-1; i++) hmm, what does that mean? that both n_result_offsets_to_convert < m_pcre_exec_rc-1 and m_pcre_exec_rc-1 < n_result_offsets_to_convert are possible? > { > /* > Convert byte offset into character offset. > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index 3d11a22..d3f52ed 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -25,7 +25,8 @@ > > #include "item_func.h" /* Item_int_func, Item_bool_func */ > #define PCRE_STATIC 1 /* Important on Windows */ > -#include "pcre.h" /* pcre header file */ > +#define PCRE2_CODE_UNIT_WIDTH 8 you sure, you want it here and not, say, in one universally-included headers like my_config.h? You don't define it in other places now, so they include different function names and macros. > +#include "pcre2.h" /* pcre header file */ > #include "item.h" > > extern Item_result item_cmp_type(Item_result a,Item_result b); > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index fe2b9c8..dcf0996 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -111,7 +111,7 @@ > #include "sp_rcontext.h" > #include "sp_cache.h" > #include "sql_reload.h" // reload_acl_and_cache > -#include "pcre.h" > +#include "pcre2.h" do you still need to include pcre header here? > > #ifdef HAVE_POLL_H > #include <poll.h> > diff --git a/storage/mroonga/CMakeLists.txt b/storage/mroonga/CMakeLists.txt > index 5d8e8c1..57d12a1 100644 > --- a/storage/mroonga/CMakeLists.txt > +++ b/storage/mroonga/CMakeLists.txt > @@ -189,8 +189,8 @@ else() > set(MYSQL_VARIANT "MySQL") > endif() > > -if(EXISTS "${MYSQL_SOURCE_DIR}/pcre") > - set(MYSQL_REGEX_INCLUDE_DIR "${MYSQL_SOURCE_DIR}/pcre") > +if(EXISTS "${MYSQL_SOURCE_DIR}/pcre2") > + set(MYSQL_REGEX_INCLUDE_DIR "${MYSQL_SOURCE_DIR}/pcre2/src") eh, is that enough? there were a lot more changes in the server, than just correcting the path. > else() > set(MYSQL_REGEX_INCLUDE_DIR "${MYSQL_SOURCE_DIR}/regex") > endif() Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] FYI: 10.3 master branch head builds OK Launchpad all architectures
by Otto Kekäläinen 11 Apr '18

11 Apr '18

Hello! Just for the record, I've done some small tweaks to 10.3 so that it now builds perfectly and passes build tests on all Launchpad architectures that I have access to. https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.3/+build s?build_text=&build_state=all Let's hope it stays like this for the rest of the current 10.3 cycle and will not regress by before it is announced GA.. On buildbot the Debian sid and Ubuntu Bionic builders pass now. There are however some sporadic failures left. http://buildbot.askmonty.org/buildbot/grid?branch=10.3&category=main&ca tegory=experimental On Travis everything else passes apart from one test, which has been consistently failing for some time. Everything was green in January when I wrote the protected branches blog post, so this one is a consistently failing regression. https://travis-ci.org/MariaDB/server/builds/362466962 - Otto

2 5

[Maria-developers] Temporary table bug
by Badrul Chowdhury 09 Apr '18

09 Apr '18

Hi, Is anyone actively working on https://jira.mariadb.org/browse/MDEV-12459? If not, I would like to take a look. -- Thanks, Badrul

2 2

[Maria-developers] Test failure overview for Debian/Ubuntu on 10.3 master branch
by Otto Kekäläinen 05 Apr '18

05 Apr '18

(posting this here so to ensure it is recorded in mailing list archive) On Travis the consistently failing tests are: main.type_datetime_hires main.select_pkeycache main.select main.select_jcl6 main.type_float main.func_str main.type_time_hires main.type_timestamp_hires Log: https://api.travis-ci.org/v3/job/362466968/log.txt On buildbot the consistelty failing tests for Debian/Ubuntu are: galera.galera_sst_mysqldump Log: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-x86/build s/4033/steps/mtr/logs/stdio and http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-ppc64le/b uilds/4546/steps/mtr/logs/stdio Single ones: encryption.innodb_encryption Log: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-sid-x86/builds/6 560/steps/mtr/logs/stdio rocksdb.drop_index_inplace Log: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-ppc64le/b uilds/4546/steps/mtr/logs/stdio - Otto

1 0

Re: [Maria-developers] Regarding bugs (mysqldump)
by Vicențiu Ciorbaru 01 Apr '18

01 Apr '18

Hi Rutuja! Great analysis again and lets share this with the maria-developers mailing list as well. I've tested your proposed change and the change for dump_all_tables_in_db seems to not be necessary. If you look at how dump_all_tables_in_db gets called, it's followed by dump_all_views_in_db. The actual view creation happens in dump_all_views_in_db, while dump_all_tables_in_db will create some dummy tables for views. We create dummy tables first, so as to avoid having to check dependencies within views. There's more details in the comments of why we do that in the code of get_table_structure: /* Create a table with the same name as the view and with columns of the same name in order to satisfy views that depend on this view. The table will be removed when the actual view is created. The properties of each column, are not preserved in this temporary table, because they are not necessary. This will not be necessary once we can determine dependencies between views and can simply dump them in the appropriate order. */ For dump_selected_tables, your proposed change should be sufficient and according to my tests it works. Please submit a pull request for this. Looking forward to having this fixed soon. :) Unrelated to your analysis: There are plenty of edge cases where views won't work: For example: If we have 2 databases and one view uses something from a different database, if you're not careful about the order of things, it's going to fail. To create a "truly" always working dump one would have to create a full dependency graph and then do a topological sort for all objects in the database that depend on the objects we're dumping. This is outside of the scope of this bug, however it could prove to be a "better way to dump stuff". Regards, Vicentiu On Sun, 1 Apr 2018 at 06:28 Rutuja Surve <rutuja.r.surve(a)gmail.com> wrote: > Hello, > Could you review my following approach regarding the mysqldump bug ( > https://jira.mariadb.org/browse/MDEV-15021) > > I went through the code and observed the following: > 1. dump_all_tables_in_db calls dump_table first (for the list of all > tables), then dump_table calls get_view_structure. > 2. get_view_structure dumps the view for the corresponding table to > mysqldump > 3. After this call, dump_all_tables_in_db calls dump_routines_for_db > > dump_selected_tables also behaves in the exact same way as dump_all_tables_in_db, > following steps 1,2,3, except that it does it for a list of given tables > instead of doing it for all tables. > > We want the routines to get dumped before the views for the dump to become > importable. > There are two solutions for this: > 1. Make dump_all_tables_in_db call dump_routines_for_db before it calls > dump_table. This will dump routines first and views later. > This can be done by shifting the call to dump_routines_for_db before the > loop in which dump_table is called. I.E: > Shift : > if (opt_routines && mysql_get_server_version(mysql) >= 50009) > { > DBUG_PRINT("info", ("Dumping routines for database %s", database)); > *dump_routines_for_db*(database); > } > Before this loop: > while ((table= getTableName(0))) > { > char *end= strmov(afterdot, table); > if (include_table((uchar*) hash_key, end - hash_key)) > { > *dump_table*(table,database); > my_free(order_by); > .... > } > } > > We should do this for both dump_all_tables_in_db and dump_selected_tables. > > 2. Approach 2: > As suggested in the jira thread, we can use negative logic to call > dump_table on tables we intend to skip. > This appears to be a workaround. In the place where dump_all_tables_in_db > is called, we'd rather call dump_selected_tables > with a list of tables that don't use the routines we'll be listing after > that. This would fix the issue but is not a good approach. > > If dump_table doesn't alter the values for opt_routines, the first > approach should fix the issue. > > Thanks, > Rutuja > > > > On Fri, Mar 23, 2018 at 10:36 PM, Rutuja Surve <rutuja.r.surve(a)gmail.com> > wrote: > >> Hello, >> With respect to https://jira.mariadb.org/browse/MDEV-15021 : >> From what I understood from the thread, since the routines are placed >> below the tables/views, they cannot be used in views while importing. >> I went through the problematic_dump.sql file that's attached there. What >> does the '@' in the following mean: >> SET @saved_cs_client = @@character_set_client; >> (I'm not familiar with this syntax) >> Also, what does the following piece of code do: (what does create definer >> mean and does this function simply return 1 as there are no other >> statements after begin? This is a routine defined below test_view) >> CREATE DEFINER=`root`@`localhost` FUNCTION `test_function`() RETURNS >> int(11) >> BEGIN >> >> RETURN 1; >> END ;; >> DELIMITER ; >> From my understanding, test_view uses test_function and test_view2 would >> use test_function2. >> In the mysqldump.c file, I went through the dump_routines_for_db >> function. We'd require to modify that in order to fix the bug. >> >> From the beginner bugs list, I found these 2 bugs that are also related >> to mysql dump that I could take up: >> 1. https://jira.mariadb.org/browse/MDEV-9883 >> (Flag for letting mysqldump support the 'auto' option for character set). >> Where in the code base can I find the file where the ' --default-character-set' >> flag is defined?) >> 2. https://jira.mariadb.org/browse/MDEV-8765 >> The solution for this has already been proposed in the thread. For >> mysqldump to not corrupt 4 byte UTF8 data, "change >> include/my_global.h:#define MYSQL_UNIVERSAL_CLIENT_CHARSET to utf8mb4" >> >> Thanks, >> Rutuja >> > >

1 0