September 2018 - developers - lists.mariadb.org

[Maria-developers] Windows line endings in MariaDB sources again (
by Otto Kekäläinen 24 Jul '19

24 Jul '19

Hello! While trying to release 10.0.26 for Debian and Ubuntu I noticed git chokes again on Windows line endings. As our .gitattributes states that by default git should clean up the line endings, why do we keep getting new Windows line endings in new code? mariadb-10.0$ cat .gitattributes # Set the default behavior, in case people don't have core.autocrlf set. * text=auto ... storage/connect/mysql-test/connect/std_data/*.txt -text mariadb-10.0.26$ find . | xargs file | grep CRLF ./storage/connect/mysql-test/connect/std_data/boyswin.txt: ASCII text, with CRLF line terminators ./storage/connect/mysql-test/connect/std_data/expenses.txt: ASCII text, with CRLF line terminators ./storage/connect/mysql-test/connect/std_data/emp.txt: ASCII text, with CRLF line terminators ./mysql-test/std_data/loaddata7.dat: ASCII text, with CRLF line terminators ./mysql-test/r/loadxml.result: ASCII text, with CRLF, LF line terminators ./mysql-test/r/perror-win.result: ASCII text, with CRLF, LF line terminators ./mysql-test/r/mysql_binary_mode.result: ASCII English text, with CRLF, LF line terminators ./mysql-test/r/func_regexp_pcre.result: UTF-8 Unicode C++ program text, with CRLF, CR, LF line terminators ./pcre/testdata/grepoutputN: ASCII text, with CRLF, CR, LF line terminators ./pcre/testdata/greppatN4: ASCII text, with CRLF line terminators This leads for me to: dpkg-source: info: building mariadb-10.0 using existing ./mariadb-10.0_10.0.26.orig.tar.gz dpkg-source: info: local changes detected, the modified files are: mariadb-10.0/storage/connect/mysql-test/connect/std_data/boyswin.txt mariadb-10.0/storage/connect/mysql-test/connect/std_data/emp.txt mariadb-10.0/storage/connect/mysql-test/connect/std_data/expenses.txt dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/mariadb-10.0_10.0.26-1.diff.Yzwwhw Which means I have to do extra work with importing the upstream sources.. - Otto

2 4

[Maria-developers] Correct parsing of BINLOG 'XXX' with comment inside
by Sachin Setia 19 Oct '18

19 Oct '18

Hi Bar, Currently if we have generated the sql file using mysqlbinlog -v and if we have big statement binlog , then mysqlbinlog add comment in between of Binlog 'XXX' statement , but unfortunately base64_decode does not understands it, and it through error since # is not a base64 char This patches solves this. There are 2 patches one for 5.5 and 2nd for 10.0 and above Please review it.

4 7

Re: [Maria-developers] 9b453c1ebf4: MDEV-10963 Fragmented BINLOG query
by Sergei Golubchik 18 Oct '18

18 Oct '18

Hi, Andrei! On Sep 07, andrei.elkin(a)pp.inet.fi wrote: > revision-id: 9b453c1ebf47a20ea1436de3284f8810d0e64c4c (mariadb-10.1.34-8-g9b453c1ebf4) > parent(s): c09a8b5b36edb494e2bcc93074c06e26cd9f2b92 > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-09-07 20:36:16 +0300 > message: > > MDEV-10963 Fragmented BINLOG query > > The problem was originally stated in > http://bugs.mysql.com/bug.php?id=82212 > The size of an base64-encoded Rows_log_event exceeds its > vanilla byte representation in 4/3 times. > When a binlogged event size is about 1GB mysqlbinlog generates > a BINLOG query that can't be send out due to its size. > > It is fixed with fragmenting the BINLOG argument C-string into > (approximate) halves when the base64 encoded event is over 1GB size. > The mysqlbinlog in such case puts out > > SET @binlog_fragment_0='base64-encoded-fragment_0'; > SET @binlog_fragment_1='base64-encoded-fragment_1'; > BINLOG 2, 'binlog_fragment'; No, that's totally ridiculous. A statement that internally walks the list of user variables and collects values from variables named according to a specific pattern, seriously? Please, make it BINLOG CONCAT(@binlog_fragment_0, @binlog_fragment_1) that'll work with no questions asked, everybody understands what it means. The parser doesn't need to accept an arbitrary expression here, it'd be simpler and safer to hard-code the syntax as above. A couple of minor generic comments below, but I've stopped reviewing, because the code will change anyway. > to represent a big BINLOG 'base64-encoded-"total"'. > Two more statements are composed to promptly release memory > SET @binlog_fragment_0=NULL; > SET @binlog_fragment_1=NULL; > > diff --git a/client/mysqlbinlog.cc b/client/mysqlbinlog.cc > index 9753125dd67..f0689631cfb 100644 > --- a/client/mysqlbinlog.cc > +++ b/client/mysqlbinlog.cc > @@ -56,7 +56,13 @@ Rpl_filter *binlog_filter= 0; > > #define BIN_LOG_HEADER_SIZE 4 > #define PROBE_HEADER_LEN (EVENT_LEN_OFFSET+4) > - > +/* > + 2 fragments can always represent near 1GB row-based base64-encoded event as > + two strings each of size less than max(max_allowed_packet). > + Bigger number of fragments does not safe from potential need to tune (increase) > + @@max_allowed_packet before to process the fragments. So 2 is safe and enough. > +*/ > +#define BINLOG_ROWS_EVENT_ENCODED_FRAGMENTS 2 > > #define CLIENT_CAPABILITIES (CLIENT_LONG_PASSWORD | CLIENT_LONG_FLAG | CLIENT_LOCAL_FILES) > > @@ -71,6 +77,9 @@ ulong bytes_sent = 0L, bytes_received = 0L; > ulong mysqld_net_retry_count = 10L; > ulong open_files_limit; > ulong opt_binlog_rows_event_max_size; > +#ifndef DBUG_OFF > +ulong opt_binlog_rows_event_max_encoded_size; > +#endif Try to avoid unnecessary ifdefs. Here you could've defined the variable unconditionally and suppressed the warning with __attribute__((unused)), of you could've used it unconditionally as a splitting threshold below and only hidden the command line option. > uint test_flags = 0; > static uint opt_protocol= 0; > static FILE *result_file; > @@ -813,7 +822,14 @@ write_event_header_and_base64(Log_event *ev, FILE *result_file, > > /* Write header and base64 output to cache */ > ev->print_header(head, print_event_info, FALSE); > - ev->print_base64(body, print_event_info, FALSE); > + > + /* the assert states the only current use case for the function */ > + DBUG_ASSERT(print_event_info->base64_output_mode == > + BASE64_OUTPUT_ALWAYS); 1. don't document the obvious 2. don't wrap the line if it fits in 80-char line > + > + ev->print_base64(body, print_event_info, > + print_event_info->base64_output_mode != > + BASE64_OUTPUT_DECODE_ROWS); > > /* Read data from cache and write to result file */ > if (copy_event_cache_to_file_and_reinit(head, result_file) || > @@ -1472,6 +1490,15 @@ that may lead to an endless loop.", > "This value must be a multiple of 256.", > &opt_binlog_rows_event_max_size, &opt_binlog_rows_event_max_size, 0, > GET_ULONG, REQUIRED_ARG, UINT_MAX, 256, ULONG_MAX, 0, 256, 0}, > +#ifndef DBUG_OFF > + {"binlog-row-event-max-encoded-size", 0, all debug command-line options must start with "debug-", that is, either rename it to "debug-binlog-row-event-max-encoded-size" or make it non-debug. > + "The maximum size of base64-encoded rows-event in one BINLOG pseudo-query " > + "instance. When the computed actual size exceeds the limit " > + "the BINLOG's argument string is fragmented in two.", > + &opt_binlog_rows_event_max_encoded_size, > + &opt_binlog_rows_event_max_encoded_size, 0, > + GET_ULONG, REQUIRED_ARG, UINT_MAX/4, 256, ULONG_MAX, 0, 256, 0}, > +#endif > {"verify-binlog-checksum", 'c', "Verify checksum binlog events.", > (uchar**) &opt_verify_binlog_checksum, (uchar**) &opt_verify_binlog_checksum, > 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

[Maria-developers] CONC-312; caching_sha2_plugin
by Michal Schorm 02 Oct '18

02 Oct '18

Hello, Can I get some explanation, why was https://jira.mariadb.org/browse/CONC-312 closed as WON'T FIX ? -- MySQL 8.0 implemented "caching_sha2_password" as its default authentification plugin / method. MariaDB doesn't have support for it so mariadb-connector-c cannot be used for MySQL 8. That makes problems for nearly every Linux OS, beacuse they chose MariaDB over MySQL as the default implemetation. I'd like to know the MariaDB stance to it. Do you see it as an issue as well? Are there any plans to deal with it? How? Michal Schorm -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

2 2

Re: [Maria-developers] e11e0e6c8b9: MDEV-17167 - InnoDB: Failing assertion: table->get_ref_count() == 0 upon
by Sergey Vojtovich 24 Sep '18

24 Sep '18

Hi Sergei, On Mon, Sep 24, 2018 at 03:02:40PM +0200, Sergei Golubchik wrote: > Hi, Sergey! > > On Sep 12, Sergey Vojtovich wrote: > > revision-id: e11e0e6c8b9a85ca7f16c6b9b28fb505d464a96e (mariadb-10.3.7-186-ge11e0e6c8b9) > > parent(s): f2f661b848c02ebd7c444ba645b26416d3e4d2ad > > author: Sergey Vojtovich > > committer: Sergey Vojtovich > > timestamp: 2018-09-12 16:36:45 +0400 > > message: > > > > MDEV-17167 - InnoDB: Failing assertion: table->get_ref_count() == 0 upon > > truncating a temporary table > > > > TRUNCATE expects only one TABLE instance (which is used by TRUNCATE > > itself) to be open. However this requirement wasn't enforced after > > "MDEV-5535: Cannot reopen temporary table". > > > > Fixed by closing unused table instances before performing TRUNCATE. > > > diff --git a/sql/sql_class.h b/sql/sql_class.h > > index 6256522bf5c..e7dcd997e05 100644 > > --- a/sql/sql_class.h > > +++ b/sql/sql_class.h > > @@ -4612,6 +4612,7 @@ class THD :public Statement, > > > > TMP_TABLE_SHARE* save_tmp_table_share(TABLE *table); > > void restore_tmp_table_share(TMP_TABLE_SHARE *share); > > + void close_unused_temporary_table_instances(const TABLE_LIST *tl); > > > > private: > > /* Whether a lock has been acquired? */ > > diff --git a/sql/sql_truncate.cc b/sql/sql_truncate.cc > > index 201825d4593..695fcb538f9 100644 > > --- a/sql/sql_truncate.cc > > +++ b/sql/sql_truncate.cc > > @@ -401,6 +401,8 @@ bool Sql_cmd_truncate_table::truncate_table(THD *thd, TABLE_LIST *table_ref) > > /* In RBR, the statement is not binlogged if the table is temporary. */ > > binlog_stmt= !thd->is_current_stmt_binlog_format_row(); > > > > + thd->close_unused_temporary_table_instances(table_ref); > > Is it only TRUNCATE that needs it? What about ALTER? REPAIR? Other > similar commands? Right, do you think it is worth to exapnd scope of this bug and look for other possible issues? > > +*/ > > + > > +void THD::close_unused_temporary_table_instances(const TABLE_LIST *tl) > > +{ > > + TMP_TABLE_SHARE *share= find_tmp_table_share(tl); > > + > > + if (share) > > + { > > + Share_free_tables::List purge_tables; > > + All_share_tables_list::Iterator tables_it(share->all_tmp_tables); > > + > > + while (TABLE *table= tables_it++) > > + { > > + if (table->query_id == 0) > > + purge_tables.push_front(table); > > + } > > + > > + while (TABLE *table= purge_tables.pop_front()) > > + { > > + share->all_tmp_tables.remove(table); > > + free_temporary_table(table); > > + } > > Why are you doing it in two loops? Because free_temporary_table() > invalidates the iterator? Not free_temporary_table(), but rather share->all_tmp_tables.remove(table). OTOH it seems to be safe to remove current element with it++, but not with ++it. I'll try to remove extra loop if it works. Thanks, Sergey

2 1

[Maria-developers] unsubscribe
by Valentin Rakush 17 Sep '18

17 Sep '18

-- Best Regards, Valentin Rakush.

1 0

[Maria-developers] Compiling default MariaDB with OpenSSL in Windows
by Braiam Peguero 14 Sep '18

14 Sep '18

Hi, Following Georg Richter recommendations for compiling MariaDB on Windows 64bit, with OpenSSL using the following cmake commands: cmake .. -DWITH_SSL=system -G "Visual Studio 15 2017 Win64" cmake --build . --config relwithdebinfo --target package However, several recipes fail to build, for the following errors: "builddir\mysys_ssl\mysys_ssl.vcxproj" (default target) (8) -> (ClCompile target) -> c:\users\braiam\src\mariadb-10.3.9\mysys_ssl\my_crypt.cc(42): error C3861: 'bzero': identifier not found [C:\Users\br aiam\src\mariadb-10.3.9\bld\mysys_ssl\mysys_ssl.vcxproj] and "builddir\package.vcxproj" (default target) (1) -> "builddir\ALL_BUILD.vcxproj" (default target) (3) -> "builddir\libmariadb\sha256_password.vcxproj" (default target) (205) -> (Link target) -> sha256_pw.obj : error LNK2019: unresolved external symbol BIO_free referenced in function auth_sha256_client [C:\User s\braiam\src\mariadb-10.3.9\bld\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol BIO_new_mem_buf referenced in function auth_sha256_client [ builddir\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol RSA_size referenced in function auth_sha256_client [C:\User s\braiam\src\mariadb-10.3.9\bld\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol RSA_public_encrypt referenced in function auth_sha256_clien t [builddir\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol RSA_free referenced in function auth_sha256_client [C:\User s\braiam\src\mariadb-10.3.9\bld\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol PEM_read_bio_RSA_PUBKEY referenced in function auth_sha256_ client [builddir\libmariadb\sha256_password.vcxproj] sha256_pw.obj : error LNK2019: unresolved external symbol ERR_clear_error referenced in function auth_sha256_client [ builddir\libmariadb\sha256_password.vcxproj] builddir\libmariadb\RelWithDebInfo\sha256_password.dll : fatal error LNK1120: 7 unresol ved externals [builddir\libmariadb\sha256_password.vcxproj] Using the OpenSSL 1.1.0i 64bit binaries from Shinning Light Productions. Using the 1.0.2p version produce even more linking errors. I also used FireDaemon but failed. Is there a recommended Windows binary for compiling? Or do I need to compile my own openssl? -- Braiam Peguero

2 3

Re: [Maria-developers] [Commits] 3c7e2540f63: MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion
by Sergey Petrunia 12 Sep '18

12 Sep '18

Hi Varun, Looking at the testcase name, I have found Serg's patch for MDEV-4285. His patch skips sorting if the keys have zero length. He does it at a late stage, in Filesort_buffer::sort_buffer(). His patch doesn't cover the example of MDEV-17020, because the testcase in MDEV-17020 uses LIMIT and so uses a Priority Queue (and not Filesort_buffer::sort_buffer which uses qsort). However, your patch doesn't cover all cases either. There are other ways to produce zero-length keys. I have found this one: SELECT * FROM t1 ORDER BY 'foo', substring(pk, 0) LIMIT 2; it crashes in the same way as the test for MDEV-17020 used to do. Possible options: 1. Find all Items that might produce zero-length keys. It's difficult not to miss anything. 2. Instead of #1, add a universal logic "if sort key length is 0, don't sort". Is this doable? 3. Extend the Serg's approach. We cannot use his code directly ("dont sort"), because Priority Queue API doesn't easily allow it, but perhaps we could add a comparison function that will allow to compare zero-length keys? Or make them 1-byte long with a dummy value (do we need the sorting to be stable??) On Fri, Aug 31, 2018 at 12:18:08AM +0530, Varun wrote: > revision-id: 3c7e2540f63c6801a0d0a27fbea7c934bb61221f (mariadb-10.0.36-13-g3c7e2540f63) > parent(s): b3c320bb0b93e516cda4db277cfa3efeef48c988 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-08-31 00:14:20 +0530 > message: > > MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion > > We have a case here where the the entire sort key is empty and so sorting is not needed. > The crash happens because filesort expects the sort key to have length > 0. > The case here is like ORDER BY CONVERT(a,char(0)), which if evaluated is a constant so > can be removed from the ORDER BY clause > > --- > mysql-test/r/order_by_zerolength-4285.result | 28 +++++++++++++++++----------- > mysql-test/t/order_by_zerolength-4285.test | 13 +++++++++++++ > sql/item_timefunc.h | 6 ++++++ > 3 files changed, 36 insertions(+), 11 deletions(-) > > diff --git a/mysql-test/r/order_by_zerolength-4285.result b/mysql-test/r/order_by_zerolength-4285.result > index f60ce7d90c7..f77b284df3c 100644 > --- a/mysql-test/r/order_by_zerolength-4285.result > +++ b/mysql-test/r/order_by_zerolength-4285.result > @@ -12,15 +12,21 @@ pk > 8 > 9 > 10 > -Warnings: > -Warning 1292 Truncated incorrect CHAR(0) value: '1' > -Warning 1292 Truncated incorrect CHAR(0) value: '2' > -Warning 1292 Truncated incorrect CHAR(0) value: '3' > -Warning 1292 Truncated incorrect CHAR(0) value: '4' > -Warning 1292 Truncated incorrect CHAR(0) value: '5' > -Warning 1292 Truncated incorrect CHAR(0) value: '6' > -Warning 1292 Truncated incorrect CHAR(0) value: '7' > -Warning 1292 Truncated incorrect CHAR(0) value: '8' > -Warning 1292 Truncated incorrect CHAR(0) value: '9' > -Warning 1292 Truncated incorrect CHAR(0) value: '10' > +drop table t1; > +# > +# MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion > +# > +set @save_sql_mode= @@sql_mode; > +SET @@sql_mode= ''; > +CREATE TABLE t1 (pk INT PRIMARY KEY); > +INSERT INTO t1 VALUES (1),(2); > +explain > +SELECT * FROM t1 ORDER BY 'foo', CONVERT(pk, CHAR(0)) LIMIT 2; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index NULL PRIMARY 4 NULL 2 Using index > +SELECT * FROM t1 ORDER BY 'foo', Cast(pk as CHAR(0)) LIMIT 2; > +pk > +1 > +2 > +set @@sql_mode= @save_sql_mode; > drop table t1; > diff --git a/mysql-test/t/order_by_zerolength-4285.test b/mysql-test/t/order_by_zerolength-4285.test > index 2fb58edd36d..f03d528320c 100644 > --- a/mysql-test/t/order_by_zerolength-4285.test > +++ b/mysql-test/t/order_by_zerolength-4285.test > @@ -6,3 +6,16 @@ insert into t1 values (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); > select * from t1 order by now(), cast(pk as char(0)); > drop table t1; > > +--echo # > +--echo # MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion > +--echo # > + > +set @save_sql_mode= @@sql_mode; > +SET @@sql_mode= ''; > +CREATE TABLE t1 (pk INT PRIMARY KEY); > +INSERT INTO t1 VALUES (1),(2); > +explain > +SELECT * FROM t1 ORDER BY 'foo', CONVERT(pk, CHAR(0)) LIMIT 2; > +SELECT * FROM t1 ORDER BY 'foo', Cast(pk as CHAR(0)) LIMIT 2; > +set @@sql_mode= @save_sql_mode; > +drop table t1; > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index 80b98758fb7..461c3feed8c 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -903,6 +903,12 @@ class Item_char_typecast :public Item_str_func > String *val_str(String *a); > void fix_length_and_dec(); > void print(String *str, enum_query_type query_type); > + table_map used_tables() const > + { > + if (!cast_length) > + return (table_map) 0L; > + return Item_str_func::used_tables(); > + } > }; > > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 17540ad: MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>.
by Sergei Golubchik 04 Sep '18

04 Sep '18

Hi, Alexey! On Sep 03, Alexey Botchkov wrote: > revision-id: 17540adc625ff4ac0f00aa10796cb6e763838d18 (mariadb-10.1.35-25-g17540ad) > parent(s): 34e026f35a7fbeeaec412547ee207c880944ae68 > committer: Alexey Botchkov > timestamp: 2018-09-03 16:49:46 +0400 > message: > > MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>. > > LOCK view WRITE shouldn't block FLUSH view. > So we set the view's mdl_request type to it's tables. > > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 6bd6b6a..d7a2a6b 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1534,8 +1534,7 @@ bool mysql_make_view(THD *thd, TABLE_SHARE *share, TABLE_LIST *table, > for (tbl= view_main_select_tables; tbl; tbl= tbl->next_local) > { > tbl->lock_type= table->lock_type; > - tbl->mdl_request.set_type((tbl->lock_type >= TL_WRITE_ALLOW_WRITE) ? > - MDL_SHARED_WRITE : MDL_SHARED_READ); > + tbl->mdl_request.set_type(table->mdl_request.type); > } I agree it's looks like a proper fix. But I'm a bit cautious about possible side effects. A more conservative fix would be only change the lock level for TL_NO_READ_WRITE, like tbl->lock_type == TL_NO_READ_WRITE ? TL_NO_READ_WRITE : tbl->lock_type >= TL_WRITE_ALLOW_WRITE ? MDL_SHARED_WRITE : MDL_SHARED_READ (or with an if(), for readability). But I don't see whether it'll make any difference as compared to your fix. Please, run all tests locally, and I mean *all*, big, ps, embedded. And see if you can think of any cases where this change would cause a change in behavior. After that - ok to push! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] 2 pull requests
by Ming Lin 04 Sep '18

04 Sep '18

Hi Sergey, Could you take a look at these 2 PRs? MDEV-16768: https://github.com/MariaDB/server/pull/837 You approved this one, but not merged yet. MDEV-15816: https://github.com/MariaDB/server/pull/849 This one was already merged by facebook. Thanks, Ming

1 1