September 2019 - discuss - lists.mariadb.org

Re: [Maria-discuss] Rocks, toku and some performance considerations.
by pslawek83 12 Sep '19

12 Sep '19

Hi Guys, @Jonas thanks for sharing the link. I was just using default config which cames with mariadb 10..4 plus i increased cache size to 6GB, so that's the only option which is in the file: plugin-load-add=ha_rocksdb.so rocksdb_block_cache_size=61784 Basically i have just started trying it, so not very familiar with it and for my tests i just waited for all the background tasks for rocks to end so the server load is 0. No bloom filter... just simple SELECT sum(x) from y SELECT sum(x) from y WHERE index in Z then did 2 same queries to make sure all blocks are in the cache on some 100 million rows table. Basically what surprised me the most is that innodb was able to do around 180mb/s reads on this vm and it wasn't putting load on IO (1-2% WA on samsung 850 PRO), with rocks i had around 20mb/s and over 50% cpu time spent in i/o wait, so maybe i'll try debian 10 and totally fresh install. Might be that something is "broken" on debian9... @ Reinis ah really bad news, hopefully it won't be removed soon, it is so great I was actually hoping for it to become new innodb especially that its characteristics seems to be much more in tune with new hardware (many cpu cores, a lot of ram, ssd storage) than innodb and internally it seems to be "better b tree" so it's similar as well. Can you tell quickly why you were not able to use rocks? Thanks again! Dnia 12 września 2019 15:09 Jonas Krauss <jkrauss82(a)gmail.com> napisał(a): Hello, how did you configure myrocks/rocksdb? Have you set up bloom filters? Maybe you could post your rocksdb options from the my.cnf in use here. This link was helpful to me when setting up myrocks: smalldatum.blogspot.com smalldatum.blogspot.com Regards Jonas Am Do., 12. Sept. 2019 um 14:44 Uhr schrieb jocelyn fournier < jocelyn.fournier(a)gmail.com >: Hi! It seems MariaDB is dropping TokuDB in 10.5. See jira.mariadb.org jira.mariadb.org BR, Jocelyn Fournier > Le 12 sept. 2019 à 14:31, pslawek83 < pslawek83(a)o2.pl > a écrit : > > Hi Everyone, i got recently interested in rocks and have a couple of questions if anyone else is doing/done some migrations to this engine: > > 1. Noticed that compared to toku - I/O overhead for range SELECT's with cold caches is much higher, especially on vmware I/O utilization (WAit) goes over 50% of available CPU on standard cfq scheduler and 4 cores. Changing it to noop helps and utilization drops around tenfold to 5-10%. It doesn't matter if we're doing full table scan or range scan with index. Basically i thought for LSM lists there'll be almost no IOPs and I/O utilization will be non-existing because all you need to do is reading a long, continuous block on disk (we're using SSDs). Any reason for that or am i doing something incorrectly? (default config + 6gb rocks key cache) Any reason it's working so badly with cfq and much better with noop? > > 2. Now with keys 100% cached - rocks is still around 2-3 times slower than toku for range scans, and even considerably slower than innodb. From what i understand keycache for rocks stores uncompressed keys, so is there some performance issue eg. with copying data from global keycache to local storage and is it going to be fixed or that's something related to how rocks works internally and there's no way of making it work better in the future? > > 3. In general it was a huge surprise that toku which is (at least it seems) so complicated to implement, and is based on very complex variation of b tree is having so much better read performance and so much better i/o characteristics... than "simple" (from what i understand) sorted list, which could be probably just read in bulk like a simple log file... > > 4. I found some info that having rocks databases which are over 100gb in size is not recommended (and it seems this is tiny... we were able to work with myisam tables which were close to 2TB). Also merging data could make the DB to end up being twice as big for some time. Are there any plans of implementing one-file-per-table like for inno? > > 5. What's the future of toku? I understand that percona is considering dropping it, will you take developement if that will happen or are you going to obsolete it and focus on rocks? From what i saw it seems that rocks and toku have totally different characteristics. So rocks is decent for point-reads (seems faster than toku when number of row reads is low)... and it seems to require less memory than toku, but for range scans it isn't going nowhere close. So both engines may have completely different use cases (eg. toku is great for long running statistical queries and servers with a lot of memory) > > Thanks > ______________________________ > Mailing list: launchpad.net launchpad.net > Post to : maria-discuss(a)lists.launchpad. > Unsubscribe : launchpad.net launchpad.net > More help : help.launchpad.net help.launchpad.net ______________________________ Mailing list: launchpad.net launchpad.net Post to : maria-discuss(a)lists.launchpad. Unsubscribe : launchpad.net launchpad.net More help : help.launchpad.net help.launchpad.net

1 0

[Maria-discuss] Json validation of unicode escaped strings
by manoj 12 Sep '19

12 Sep '19

I am trying to insert json data in to a table with json check constraint. The non-ascii characters are unicode escaped. This fails in cases where the string in the json data has unicode points in non-BMP plane, e.g U+1F449. Error: ERROR 4025 (23000): CONSTRAINT `js` failed for `t`.`t` The same json works if the unicode characters are encoded in utf-8 and not escaped. I think I can confirm that this is a json validation error with json_valid. MariaDB [t]> show create table t; +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | t | CREATE TABLE `t` ( `id` int(11) NOT NULL AUTO_INCREMENT, `js` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL CHECK (json_valid(`js`)), PRIMARY KEY (`id`) ) ENGINE=MyISAM AUTO_INCREMENT=19 DEFAULT CHARSET=utf8mb4 | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) MariaDB [t]> select json_valid('{"test": "\\ud83d\\ude0b"}'); +------------------------------------------+ | json_valid('{"test": "\\ud83d\\udc49"}') | +------------------------------------------+ | 0 | +------------------------------------------+ 1 row in set (0.00 sec) This is on Maria 10.3 MariaDB [t]> select version(); +----------------+ | version() | +----------------+ | 10.3.8-MariaDB | +----------------+ So, my questions, are unicode escaped strings in json supported? Is this a bug or a limitation? I have searched for this issue and have failed to come up with anything. Thanks, Manoj

2 1

[Maria-discuss] Rocks, toku and some performance considerations.
by pslawek83 12 Sep '19

12 Sep '19

Hi Everyone, i got recently interested in rocks and have a couple of questions if anyone else is doing/done some migrations to this engine: 1. Noticed that compared to toku - I/O overhead for range SELECT's with cold caches is much higher, especially on vmware I/O utilization (WAit) goes over 50% of available CPU on standard cfq scheduler and 4 cores. Changing it to noop helps and utilization drops around tenfold to 5-10%. It doesn't matter if we're doing full table scan or range scan with index. Basically i thought for LSM lists there'll be almost no IOPs and I/O utilization will be non-existing because all you need to do is reading a long, continuous block on disk (we're using SSDs). Any reason for that or am i doing something incorrectly? (default config + 6gb rocks key cache) Any reason it's working so badly with cfq and much better with noop? 2. Now with keys 100% cached - rocks is still around 2-3 times slower than toku for range scans, and even considerably slower than innodb. From what i understand keycache for rocks stores uncompressed keys, so is there some performance issue eg. with copying data from global keycache to local storage and is it going to be fixed or that's something related to how rocks works internally and there's no way of making it work better in the future? 3. In general it was a huge surprise that toku which is (at least it seems) so complicated to implement, and is based on very complex variation of b tree is having so much better read performance and so much better i/o characteristics... than "simple" (from what i understand) sorted list, which could be probably just read in bulk like a simple log file... 4. I found some info that having rocks databases which are over 100gb in size is not recommended (and it seems this is tiny... we were able to work with myisam tables which were close to 2TB). Also merging data could make the DB to end up being twice as big for some time. Are there any plans of implementing one-file-per-table like for inno? 5. What's the future of toku? I understand that percona is considering dropping it, will you take developement if that will happen or are you going to obsolete it and focus on rocks? From what i saw it seems that rocks and toku have totally different characteristics. So rocks is decent for point-reads (seems faster than toku when number of row reads is low)... and it seems to require less memory than toku, but for range scans it isn't going nowhere close. So both engines may have completely different use cases (eg. toku is great for long running statistical queries and servers with a lot of memory) Thanks

5 4

[Maria-discuss] unsubscribe
by c.monty＠web.de 12 Sep '19

12 Sep '19

1 0

Re: [Maria-discuss] fsync alternative
by pslawek83 12 Sep '19

12 Sep '19

Hi Jure, maybe you can try using toku engine. I noticed that with large keycaches syncing it to disk can sometimes take up to a minute for myisam tables. Also innodb is generally heavy on I/O. Toku on the other hand is heavier on CPU/memory but seems very light on I/O (from my experience), so maybe that would help... Rocksdb should be light on random I/O access as it's using LSM, haven't used it in production yet and not sure how mature it is but may work... You can try to disable binlogs/fsync almost altogether if you don't care about data consistency, it's very unrecommended. And if you're using replication skip the 3'rd line because it'll break it, if you're not using replication remove _xxx. SET GLOBAL innodb_flush_log_at_trx_commit = 0; SET GLOBAL sync_binlog = 0; SET GLOBAL log_bin_xxx = 0; But i believe SAN is really the issue. Benchmark this SAN (throughput / iops)... if not conclusive try running from local SSD on raid10 or something similar... raid5/6 works horribly with SSDs and databases in general due to trim support and write amplification (not sure what raid you're using)... www.percona.com www.percona.com www.percona.com www.percona.com For kernel's pagecache/fsync comment, i think it's working as expected and you're missing write-behind not OS file cache... I guess without fsync it'd be totally impossible to have the data in consistent state after a crash, so you could not only loose couple of second's worth of data but whole table, as writes could be postponed and re-ordered, especially on SSDs... So for example if you'd do copy-on-write to make changes atomic, the pointer to data block could be written before the data block is (fully) there on disk and you'd get a structure pointing to not-existing or random data. To be sure you're having anything which is making sense in your files you'd need to 1. write your data 2. ensure it's on the device in full (fsync) 3. atomically change the data pointer from OLD to NEW data, which is probably using assumption that block-aligned writes below 512b should be atomic, or some log-like structure Maria is probably doing it little differently, but concept is for sure similar if you don't want your software to be bound to some single, specific, probably very expensive and propertialy mix of software and hardware which can support some basic atomic I/O. For IO over network... actually had a huge issues even with running simple readonly sqlite over network-attached I/O via NFS. I believe such kind of setup when your throughput is high will at least need dedicated 10Gbps network link. And still SAN/NAS can have "normal" throughput on 10Gb, but latency will probably be much higher (so slower fsync's). I'd try to move this SQL server outside of VM and to local storage. Best, Slawomir. Dnia 7 września 2019 21:38 Jure Sah <e(a)juresah.si> napisał(a): Hi, On 7. 09. 19 21:03, Jan Steinman wrote: I’m using an inexpensive Mac Mini, maxed out with RAM, and a 2GB SSD, running NOTHING but MariaDB. I even run it headless, which means all the UI processes stay in sleep(3). When I was having web server performance issues, that was the one thing that improved things the most. And that was after wasting a lot of time trying to tweak MariaDB variables. I work for an ISP. The system they use is connected to a dedicated SSD RAID array on SAN. On the particular virtual machine the MariaDB is on the same server as the Apache webserver. The main issue is that the website gets a lot of traffic. It has some CMS-based website on it and the session table is grinding away constantly. Normally it's not a problem, but once when a malfunction on the SAN network caused degraded performance (not downtime mind you!), with the usual traffic the website was simply down with Gateway timeout from memcached. The workaround was to temporarily move the database to a ramdisk (tmpfs). My employer considers stepping outside the recommendations of the open source community to be not worth the risk, and the issue was resolved since. However I know that I shouldn't have had to make the workaround with the ramdisk, because the kernel page cache was supposed to take care of that by itself. The reason this doesn't work is the fsyncs used by MariaDB, which effectively disables any advantage offered by the page cache. I think that is a great shame so I am trying to see if there is anything I can do about that. Perhaps there is a better alternative out there but nobody has as of yet tried to make it work. I'm also not entirely convinced there is a reasonable argument for being so particular about fsyncing everything on typical server hardware (servers are usually UPS backed). The argument about battery-backed RAID controllers, while this is something we have, is something I find particularly unreasonable, since as far as I know fsync isn't even supported by most RAID controller drivers in Linux. There's an email on the Kernel mailing list that it does exist, but that Linux doesn't use it. Disabling fsync boosts performance of a typical MySQL server by something like a factor of 3. Are you really sure about that? I don’t know the MySQL implementation details, but typically, sync(2) is run from a dedicated thread per file, and so should not block anything, unless you’ve run out of threads or something, in which case, tuning may help. I found the measurement by some blogger on the internet. I'm not sure what hardware they tried this on, it seems to me it would depend. In the aforementioned malfunction it was pretty clear that MariaDB's performance is capped by storage. It makes sense too, if you are fsyncing on write operations, it's going to block if the storage is slow. I'm not excluding the possibility that there is a setting here that is making my system behave the way that it is. LP, Jure ______________________________ Mailing list: launchpad.net launchpad.net Post to : maria-discuss(a)lists.launchpad. Unsubscribe : launchpad.net launchpad.net More help : help.launchpad.net help.launchpad.net

1 0

[Maria-discuss] Thread State: Unlocking tables
by mariadb＠biblestuph.com 10 Sep '19

10 Sep '19

Can someone fill me in as to what's happening behind the scenes when a thread has the state "Unlocking tables?" I looked here but did not see it listed: https://mariadb.com/kb/en/library/general-thread-states/ Specifically, I sometimes see threads spending a lot of time in this state, and am looking to do some troubleshooting as to the cause of the delay. I.E., what could cause a thread to be delayed while in that state. They're (currently) MyISAM tables if it makes a difference. Thanks! Dan

2 4

Re: [Maria-discuss] fsync alternative
by Jure Sah 07 Sep '19

07 Sep '19

Hi, On 7. 09. 19 21:03, Jan Steinman wrote: > I’m using an inexpensive Mac Mini, maxed out with RAM, and a 2GB SSD, > running NOTHING but MariaDB. I even run it headless, which means all > the UI processes stay in sleep(3). When I was having web server > performance issues, that was the one thing that improved things the > most. And that was after wasting a lot of time trying to tweak MariaDB > variables. I work for an ISP. The system they use is connected to a dedicated SSD RAID array on SAN. On the particular virtual machine the MariaDB is on the same server as the Apache webserver. The main issue is that the website gets a lot of traffic. It has some CMS-based website on it and the session table is grinding away constantly. Normally it's not a problem, but once when a malfunction on the SAN network caused degraded performance (not downtime mind you!), with the usual traffic the website was simply down with Gateway timeout from memcached. The workaround was to temporarily move the database to a ramdisk (tmpfs). My employer considers stepping outside the recommendations of the open source community to be not worth the risk, and the issue was resolved since. However I know that I shouldn't have had to make the workaround with the ramdisk, because the kernel page cache was supposed to take care of that by itself. The reason this doesn't work is the fsyncs used by MariaDB, which effectively disables any advantage offered by the page cache. I think that is a great shame so I am trying to see if there is anything I can do about that. Perhaps there is a better alternative out there but nobody has as of yet tried to make it work. I'm also not entirely convinced there is a reasonable argument for being so particular about fsyncing everything on typical server hardware (servers are usually UPS backed). The argument about battery-backed RAID controllers, while this is something we have, is something I find particularly unreasonable, since as far as I know fsync isn't even supported by most RAID controller drivers in Linux. There's an email on the Kernel mailing list that it does exist, but that Linux doesn't use it. > >> Disabling fsync boosts performance of a typical MySQL server >> by something like a factor of 3. > > Are you really sure about that? I don’t know the MySQL implementation > details, but typically, sync(2) is run from a dedicated thread per > file, and so should not block anything, unless you’ve run out of > threads or something, in which case, tuning may help. I found the measurement by some blogger on the internet. I'm not sure what hardware they tried this on, it seems to me it would depend. In the aforementioned malfunction it was pretty clear that MariaDB's performance is capped by storage. It makes sense too, if you are fsyncing on write operations, it's going to block if the storage is slow. I'm not excluding the possibility that there is a setting here that is making my system behave the way that it is. LP, Jure

2 1

[Maria-discuss] stability of 10.4.7
by Tom Ryan 07 Sep '19

07 Sep '19

We had been using 10.2.x for some time in a galera config with wonderful stability. A disk issue caused us to temporarily have to utilize a single 10.3.x instance (again with no stability issues). After we resolved our disk issue, we decided to move to 10.4.7 (since it was the second stable version) and listed on the website as the "stable version". Unfortunately, nodes in the cluster crash just about everyday. one node usually fails and then another node drops, with errors like 190906 14:37:59 [ERROR] mysqld got signal 11 ; Sep 6 14:37:59 node-2 mysqld[1846]: This could be because you hit a bug. It is also possible that this binary Sep 6 14:37:59 node-2 mysqld[1846]: or one of the libraries it was linked against is corrupt, improperly built, Sep 6 14:37:59 node-2 mysqld[1846]: or misconfigured. This error can also be caused by malfunctioning hardware. Sep 6 14:37:59 node-2 mysqld[1846]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Sep 6 14:37:59 node-2 mysqld[1846]: We will try our best to scrape up some info that will hopefully help Sep 6 14:37:59 node-2 mysqld[1846]: diagnose the problem, but since we have already crashed, Sep 6 14:37:59 node-2 mysqld[1846]: something is definitely wrong and this may fail. Sep 6 14:37:59 node-2 mysqld[1846]: Server version: 10.4.7-MariaDB-1:10.4.7+maria~xenial-log Sep 6 14:37:59 node-2 mysqld[1846]: key_buffer_size=134217728 Sep 6 14:37:59 node-2 mysqld[1846]: read_buffer_size=2097152 Sep 6 14:37:59 node-2 mysqld[1846]: max_used_connections=30 Sep 6 14:37:59 node-2 mysqld[1846]: max_threads=4002 Sep 6 14:37:59 node-2 mysqld[1846]: thread_count=45 Sep 6 14:37:59 node-2 mysqld[1846]: It is possible that mysqld could use up to Sep 6 14:37:59 node-2 mysqld[1846]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24817728 K bytes of memory Sep 6 14:37:59 node-2 mysqld[1846]: Hope that's ok; if not, decrease some variables in the equation. Sep 6 14:37:59 node-2 mysqld[1846]: Thread pointer: 0x7f32052a0348 Sep 6 14:37:59 node-2 mysqld[1846]: Attempting backtrace. You can use the following information to find out Sep 6 14:37:59 node-2 mysqld[1846]: where mysqld died. If you see no messages after this, something went Sep 6 14:37:59 node-2 mysqld[1846]: terribly wrong... Sep 6 14:37:59 node-2 mysqld[1846]: stack_bottom = 0x7f3271446dd8 thread_stack 0x49000 Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5644f71dcf4e] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_fatal_signal+0x307)[0x5644f6c70b07] Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f33e141d890] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handler_index_cond_check+0x8d)[0x5644f6c7b1fd] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x566623)[0x5644f6973623] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xbb4ba8)[0x5644f6fc1ba8] This is getting old and is unlike our usual mariadb experience. Looking at the jira for 10.4.8, it appears to fix many problems in 10.4 and I'm hopeful it solves this. In the meantime, any ideas or suggestions for things to look for? Thanks!

2 3

[Maria-discuss] 'Reverse Replication" Rollback Strategy
by Webert de Souza Lima 04 Sep '19

04 Sep '19

Hello, I'm currently working on migrating a Galera 10.3 cluster (A) to Amazon RDS. (B) In order to minimize impacts and downtime, the plan is to have a standard MySQL Slave on RDS (B) to sync data, then do the switch to it and stop the replication A->B. The main concern is a catastrophic scenario on B where a rollback to A is needed. How to bring back only new data that now sits on B to the old cluster A? One approach I saw is to, before switching applications to the new server, stop the replication A->B and enable replication B->A. This is my question here. Before the switch, the data will be the same on A and B but B will not be part of the cluster A, so things like a binlog position change. Is it possible to enable this opposite replication B->A without having to extract a whole dump of B and restoring it on A ? Thanks Regards, Webert Lima *Berlin, Germany* *IRC NICK - WebertRLZ*

1 0

[Maria-discuss] 10.3.x: unknown variable 'innodb_support_xa=1'
by Reindl Harald 03 Sep '19

03 Sep '19

unknown variable 'innodb_support_xa=1' would you funny guys consider things like deperectaion warnings in previous releases or at least offer an option to change all these fucking "i don't know a option because it was removed or you did not compile something in and hence i refuse to start the server" to warnings?

6 23