- developers - lists.mariadb.org

[Maria-developers] unbundle RocksDB
by Michal Schorm 13 Jun '18

13 Jun '18

Hello, Since we have a rocksdb package ready in Fedora, I tried to unbundle it, using the system package instead. I wasn't successful though. I haven't found any CMake argument which seemed related and from the build log it seems, the code tries to link against the static libraries in any case. Before I'll start to dig further, I'm stopping by on this list, making sure it is possible. - is it currently supported? - have I overlooked any related CMake argument? Links: MariaDB in Fedora: https://src.fedoraproject.org/rpms/mariadb/tree/master -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

3 3

Re: [Maria-developers] d9913834ceb: MDEV-14014 Multi-Slave Replication Fail: bogus data in log event
by Sergei Golubchik 13 Jun '18

13 Jun '18

Hi, andrei.elkin! On May 30, andrei.elkin(a)pp.inet.fi wrote: > revision-id: d9913834cebe4ed8494b3d76762cd882788a8fd5 (mariadb-10.1.33-28-gd9913834ceb) > parent(s): c1698e8dc50f0c342c1a6886426ba1d43396cd1e > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-05-30 20:06:44 +0300 > message: > > MDEV-14014 Multi-Slave Replication Fail: bogus data in log event > > MDEV-7257 made a dump thread to read from binlog concurrently with > writers as long as the read bytes are below a water-mark > (MYSQL_BIN_LOG::binlog_end_pos). However it appeared to be possible a > dump thread reader reach out for bytes past the water mark through a > feature of IO_CACHE that fills in the internal buffer and while doing > so it could read what the reader is not supposed to see (the bytes > above MYSQL_BIN_LOG::binlog_end_pos). > > The issue is fixed with constraining the IO_CACHE buffer fill to respect > the watermark. > An added test simulates potentially unconstrained buffer fill and an > assert guards this is not the case anymore. What about a much simpler one-liner fix: --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -2549,6 +2549,7 @@ static int send_events(binlog_send_info *info, IO_CACHE* return 1; info->last_pos= linfo->pos; + log->end_of_file= end_pos; error= Log_event::read_log_event(log, packet, info->fdev, opt_master_verify_checksum ? info->current_checksum_alg : BINLOG_CHECKSUM_ALG_OFF); It doesn't change IO_CACHE structure and doesn't add any overhead to the IO_CACHE. Did you already test that your commit actually fixes the issue? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] 20f3479: MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>.
by Sergei Golubchik 12 Jun '18

12 Jun '18

Hi, Alexey! On Jun 11, Alexey Botchkov wrote: > revision-id: 20f347974f6a9b4e2c65a5ca79aefb8e118ba6cd (mariadb-10.1.33-33-g20f3479) > parent(s): 3627dd7f6a1cf5bd7151ff23290d64cb6dffea90 > committer: Alexey Botchkov > timestamp: 2018-06-11 14:58:46 +0400 > message: > > MDEV-15890 Strange error message if you try to FLUSH TABLES <view> after LOCK TABLES <view>. > > The reload_acl_and_cache() now looks into the VIEW definition > to check the involved tables if the locked_tables_mode. That's a pretty neat solution :) But it seems that - contrary to what I expected - that MariaDB actually distinguishes between locking a view a locking all its tables. Consider this test case: ---- create table t1 (a int); create view v1 as select * from t1; lock table v1 write; insert v1 values (1); select * from t1; unlock tables; drop view v1; drop table t1; ---- this works and inserts 1 just fine. If you change the lock table statement to lock t1, it'll with Table 'v1' was not locked with LOCK TABLES It'd be great if FLUSH TABLES will exhibit the same behavior, it needs to know if the view is locked, not only if all underlying tables are locked. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Maria-discuss] MariaDB crashes because of "long semaphore wait" after migrating from 10.1 to 10.3
by Vicențiu Ciorbaru 12 Jun '18

12 Jun '18

Hi! I have cc'ed the maria-developers mailing list as this is potentially a dev issue and also the devs responsible for Windows and InnoDB development. There might be some people on vacation right now, but this should get looked into in a week or two at most. Let's find a solution for your problem. :) Vicentiu On Sun, 10 Jun 2018, 19:11 TD, <tdcf(a)gmx.net> wrote: > Hi, > > I have a Windows server that has been running MariaDB 10.1 successfully > for over a year. The server remains mostly idle for long times with some > read access, but occasionally there are transactions that add data (about > 500k rows per commit). There can be up to 10 such transactions (one per > database) at the same time and during those times the server is under quite > some load (the code processing the data resides on the same server as the > database). We're using an 8 core AMD Ryzen, 32 GB RAM and a Samsung SSD. > > Like I said, this worked fine with MariaDB 10.1 all the time. > > We're currently in the process of setting up a new server (specs similar > to the ones above) and because MariaDB 10.3 has been released recently, I > was curious and installed it. The database now constantly shuts down > because of "[FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We > intentionally crash the server because it appears to be hung." whenever > there is one of the load situations described above. If I install MariaDB > 10.1 on the new server, it runs as fine as the old one. The crashes also > happen when I start with a clean database (no data migrated from 10.1 > databases). > > I'm not a huge MariaDB config pro, as it's only a tool to store the data, > it just needs to run with reasonable performance and has always worked > until now. As far as I know and can judge from comments in the file, the > my.ini in use is (mostly?) based on my-innodb-heavy-4G.cnf, but coworkers > might have made modifications. I've attached the contents below. > > I don't know how I can debug this issue further. Windows event viewer > contains several log warnings with the text "InnoDB: A long semaphore > wait:" but no further information why this wait happens. I couldn't find > any other error logs. I guess it is some configuration error (I read about > MariaDB switching from XtraDB to InnoDB, so I guess that the issue might be > related). > > Migrating to 10.3 is not an absolute must, but it would be nice to be able > to fix the problem. > > Any help would be greatly appreciated and if you need more information, > I'll gladly try to provide it. > > Thanks, > Tom. > > my.ini: > [mysqld] > max_connections = 100 > max_allowed_packet = 16M > binlog_cache_size = 1M > max_heap_table_size = 64M > read_buffer_size = 2M > read_rnd_buffer_size = 16M > sort_buffer_size = 8M > join_buffer_size = 8M > thread_cache_size = 10 > query_cache_size = 64M > query_cache_limit = 2M > thread_stack = 240K > tmp_table_size = 64M > slow_query_log > expire_logs_days = 14 > log-bin = mysql-bin > character-set-server=utf8 > collation-server=utf8_general_ci > > #*** MyISAM Specific options > key_buffer_size = 32M > bulk_insert_buffer_size = 64M > myisam_sort_buffer_size = 128M > myisam_max_sort_file_size = 10G > myisam_repair_threads = 1 > myisam_recover > > # *** INNODB Specific options *** > #innodb_fatal_semaphore_wait_threshold = 6000 #tried that, didn't help > #innodb_adaptive_hash_index = 0 #tried that (suggested in some stack > overflow post, didn't help) > innodb_buffer_pool_size = 2G > innodb_data_file_path = ibdata1:10M:autoextend > innodb_write_io_threads = 8 > innodb_read_io_threads = 8 > innodb_thread_concurrency = 16 > innodb_log_buffer_size = 8M > innodb_log_file_size = 256M > innodb_log_files_in_group = 3 > innodb_max_dirty_pages_pct = 10 > innodb_io_capacity = 2000 > innodb_lock_wait_timeout = 120 > > [mysql] > no-auto-rehash > > [myisamchk] > key_buffer_size = 512M > sort_buffer_size = 512M > read_buffer = 8M > write_buffer = 8M > > [mysqld_safe] > open-files-limit = 8192 > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp >

3 2

[Maria-developers] MariaDB Developers Unconference in a few weeks!
by Otto Kekäläinen 08 Jun '18

08 Jun '18

Hello everybody! The next developers unconference will be in a few weeks: https://mariadb.org/2018-2-developers-unconference-in-finland/ There are still open slots so we encourage you to sign up now. https://www.meetup.com/MariaDB-Developers/events/249024232/ We have plenty of break-out rooms this time and there is space, so we also encourage you to suggest sessions/discussions you would like to have to ian(a)mariadb.org. Around mid-summer is a great time to visit Finland. The nights that don't get dark are worth experiencing! Getting to the venue in Tampere., Finlnd is also very easy. https://mariadb.org/travel-tips-to-tampere-finland/#why-visit Hope to see you there! - Otto

1 0

Re: [Maria-developers] [Commits] 56d339111a4: MDEV-16374: Filtered shows 0 for materilization scan for a semi join, which makes optimizer always picks
by Sergey Petrunia 05 Jun '18

05 Jun '18

Hi Varun, On Mon, Jun 04, 2018 at 12:34:49PM +0530, Varun wrote: > revision-id: 56d339111a4e5c868ffa3b1f2538c5f101f67767 (mariadb-10.0.30-366-g56d339111a4) > parent(s): a61724a3ca187f6eb44fc67948acb76a94efa783 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-06-04 12:26:56 +0530 > message: > > MDEV-16374: Filtered shows 0 for materilization scan for a semi join, which makes optimizer always picks > materialization scan over materialization lookup > > For non-mergeable semi-joins we don't store the estimates of the IN subquery in table->file->stats.records. > In the function TABLE_LIST::fetch_number_of_rows, we store the number of rows in the tables > (estimates in case of derived table/views). > Currently we don't store the estimates for non-mergeable semi-joins, which leads to a problem of selecting > materialization scan over materialization lookup. > Fixed this by storing these estimated appropriately > * This is a change in the optimizer. Should we really do it in 10.0? In my opinion, this should go into 10.3 or 10.4. Let's discuss this, and I think Igor may have something to say, too. * The patch updates selectivity.test but not selectivity_innodb.result (which includes selectivity.test). * Please see my last comments at https://jira.mariadb.org/browse/MDEV-16374 . Here is an alternative patch (it passes the testsuite, and produces the same result for selectivity* tes): diff --git a/sql/sql_select.cc b/sql/sql_select.cc index 1285835..884997c 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -6602,7 +6602,7 @@ double matching_candidates_in_table(JOIN_TAB *s, bool with_found_constraint, { TABLE *table= s->table; double sel= table->cond_selectivity; - double table_records= (double)table->stat_records(); + double table_records= s->records; // psergey-10 dbl_records= table_records * sel; return dbl_records; } What do you think about it? I would like to get Igor's opinion also because I'm not certain which member variable should be changed here. (Why does derived table modify table->stats.records while JTBM semi-joins modify JOIN_TAB::records? These seem like two solutions for the same problem?) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Proposal for Negative Grants Project
by Rutuja Surve 04 Jun '18

04 Jun '18

Hello, Please find the proposal for the Negative Grants project attached along with this e-mail. It would be great to hear your opinions and suggestions on the approach proposed here, before delving into its implementation. Some context about the project: *The current privilege system allows access in this manner:GRANT SELECT ON some_database.* TO a_user@%;Any revoke would revoke access from the entire database. We want to support a revoke that would disable select on a specific table, like:REVOKE SELECT ON some_database.secret_table FROM a_user@%;Reference:[1] https://jira.mariadb.org/browse/MDEV-14443 <https://jira.mariadb.org/browse/MDEV-14443>[2] https://mariadb.com/kb/en/library/google-summer-of-code-2018/ <https://mariadb.com/kb/en/library/google-summer-of-code-2018/>Thanks,Rutuja*

3 6

[Maria-developers] Conservative parallel slave is "too optimistic" for certain engines
by Sergey Petrunia 01 Jun '18

01 Jun '18

Hello, This comes from MDEV-16242. == Symptoms == When one runs a parallel slave (mode=conservative) and replicates DML for MyRocks table without a Primary Key, replication may stop with a ER_KEY_NOT_FOUND error. This may happen even if the queries were run on the master sequentially. == A detail about conservative replication == Suppose the master runs these two statements, sequentially: INSERT INTO t1 VALUES (5),(6),(7); DELETE FROM t1; Parallel slave may schedule the INSERT to Thread1, and the DELETE to Thread2. In conservative parallel replication, the DELETE "will wait" to be executed after the INSERT. One may expect that "will wait" here means that DELETE execution does not start until the INSERT has committed. But it's more subtle than that: actually, DELETE execution will start as soon as the INSERT is ready to do the prepare/commit steps. I assume this was done to increase the parallelism: initial phases of DELETE can be ran in parallel with commit/prepare steps of the INSERT. This is safe to do: 1. The INSERT has acquired locks for all rows it is about to touch. DELETE will not be able to prevent INSERT from committing. 2. The DELETE starts its execution on a database that doesn't include the results of the INSERT (the INSERT has not committed yet!). But this is fine, because the DELETE locks the rows it is about to modify. If it attempts to access a row that the INSERT is about to insert, it will block on a lock until the INSERT finishes. == Applying this to MyRocks == The critical part is #2. It works if the DELETE command will acquire row locks for rows it about to DELETE. This is normally true: If the storage engine supports gap locks, attempting to read a locked gap will cause DELETE to wait for INSERT If the storage engine doesn't support gap locks, it will use Row-Based-Replication. RBR's Delete_rows_event includes Primary Key value of the row that should be deleted. Doing a point lookup on PK will cause the DELETE to block on the row lock. The remaining case is - storage engine that doesn't support gap locks - Row-based-replication of table without primary key In this case, finding the row to delete is done by scanning the table until we find the row where all columns match. The scan does not see the rows that the INSERT is about to INSERT, and DELETE stops with an error, "we could not find the row to delete". == Possible solutions == 1. Implement gap locking in MyRocks. This will take some time. 2. Change parallel slave to wait *for commit*. This should only be done if tables that are updated do not support Gap Locking. This is hard, it will require making risky changes on the SQL layer. 3. Disallow parallel slave execution for tables without Gap Locking and without primary key. (Looks most promising but I am not sure if it is sufficient). BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 2

[Maria-developers] Test failure overview for Debian/Ubuntu on 10.3 master branch (June 1st 2018 status report)
by Otto Kekäläinen 01 Jun '18

01 Jun '18

Hello! I reported here about the test failure status in early April and here is a follow-up. My old emails can be found here: https://lists.launchpad.net/maria-developers/msg11171.html https://lists.launchpad.net/maria-developers/msg11170.html If one looks at the picture from April and compares to the status today, the situation is a bit sad as we have even less green than before: https://lists.launchpad.net/maria-developers/pngcpxTB4wjTW.png Current issues we have in addition to the usual mtr test failures are: - On multiple bionic builds the upgrade fails to run with no apparent reason. The builder script does not seem to find any previous installation of anything, so that probably is missing from the images: + dpkg -l + grep -iE 'maria|mysql|galera' + true Thus the build fails. Example log: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-bionic-amd64/bui lds/196/steps/upgrade2/logs/stdio - On bionic and xenial aarch64 mysqld fails to start. In syslog we can see May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 0 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGN ORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION',' NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_O PTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANS I','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES' ,'STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES',' ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_N OT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH','EMPT Y_STRING_IS_NULL','SIMULTANEOUS_ASSIGNMENT'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGN ORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION',' NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_O PTIONS','NO_TABLE_OPTIONS','NO May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: _FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALU May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 0 [ERROR] mysqld: Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler. May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 6 [ERROR] InnoDB: Column last_update in table `mysql`.`innodb_table_stats` is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch). May 28 16:11:25 ubuntu-xenial-aarch64 mysqld: 2018-05-28 16:11:25 6 [ERROR] InnoDB: Fetch of persistent statistics requested for table `mysql`.`gtid_slave_pos` but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. http://buildbot.askmonty.org/buildbot/builders/kvm-deb-xenial-aarch64/b uilds/277/steps/upgrade1/logs/syslog - stretch amd64 fails to compile due to librocksdblib.a(compaction_job.cc.o): In function `rocksdb::ZSTD_TrainDictionary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long)': ./builddir/storage/rocksdb/./storage/rocksdb/rocksdb/util/compression.h :812: undefined reference to `ZDICT_trainFromBuffer' ./builddir/storage/rocksdb/./storage/rocksdb/rocksdb/util/compression.h :813: undefined reference to `ZDICT_isError' collect2: error: ld returned 1 exit status http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/bu ilds/3986/steps/compile/logs/stdio - jessie amd64 builds to compile due to /usr/bin/ld.bfd.real: /tmp/ccOIwjFo.ltrans0.ltrans.o: relocation R_X86_64_PC32 against symbol `toku_product_name_strings' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld.bfd.real: final link failed: Bad value collect2: error: ld returned 1 exit status http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/bui lds/4298/steps/compile/logs/stdio This -fPIC issue might be related to a change in hardening flags I "forward-ported" from Debian and I will look into it. All the other issues are things I cannot do anything about.. :( On the bright side I can report that there are at least no build regressions on Launchpad with current 10.3: https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.3/+build s?build_text=&build_state=all - Otto

1 0

[Maria-developers] With_sum_func_cache
by Alexander Barkov 31 May '18

31 May '18

Hello Sanja, I recently worked on MDEV-16309 and had hard time to find which Item classes in the hierarchy: - have always with_sum_func==true - have always with_sum_func==false - have variable with_sum_func To make it sure, before actually working on MDEV-16309, I had to create a separate working tree and did with Item::with_sum_func the same change that we previously did for Item::with_subselect in: MDEV-14517 Cleanup for Item::with_subselect and Item::has_subquery() (which you reviewed) - I find the code easier to read this way (instead of searching for all possible assignments, one can search who overrides the method) - Also, some Item can become smaller in size. It's pity to throw the patch away. So perhaps we could just apply it. Can you please have a look? Thanks. I the meanwhile I'll create a new MDEV for it (with a similar description to MDEV-14517)

2 2