October 2019 - discuss - lists.mariadb.org

Re: [Maria-discuss] Performance tuning sought for MariaDB
by pslawek83 07 Oct '19

07 Oct '19

Innodb buffer pool should be probably around 10-20GB if you're going to use just one single innodb table. Currently you are probably killing mariadb with these 1.3k threads. First you can try to limit threads running simultaneusly to 5-30 because you're not goint to have any performance benefit from going above certain (rather low) point, you'll only produce mutex contention so the server will became overloaded with thread coordination code rather than doing any real work. 50 threads would be probably too much already, and you have 20 times that ;) Not sure if you're doing single op per transaction but without changes to code you probably won't be able to make this app run considerably faster anyway. Because small ops, like couter increment will never be fast in SQL so you'll need some intermediate "aggregation" step to issue as few sql commands and as few transactions as possible. In your C app you probably need to preprocess and aggregate the data. You take eg. 100k operations and process them in memory. So eg. insert, modify and set on same PK would became just single set. Then you can further divide these 100k operations into 1k-ops transactions. At the end you can parallelize by starting many instances of the application and spreading the keys over using CRC32. So probably what you'll need to do is: 1. aggregate as many operations as possible 2. divide to N chunks 3. read all affected keys for chunk X in single query 4. modify data in memory 5. output a transaction 6. goto 3 untill all chunks are done Dnia 7 października 2019 18:17 JCA <1.41421(a)gmail.com> napisał(a): I am running MariaDB 10.0.34 on a Slackware 14.2 system. I have a C application that interacts with MariaDB in the following way: 1. Read data from a FIFO. 2. Insert the data into a table in a MariaDB database, if absent, or modify it according to certain specific criteria otherwise. This works as expected. The problem that I have is that data are being written to the FIFO at a fast rate. In order to be able to keep up, at any given time my application reads the data available at the FIFO, and spawns a thread to process the chunk of data just read. It is in this thread that all the database interaction takes place. In order to deal with this, I have the following entries in my /etc/my.cnf file: # this is read by the standalone daemon and embedded servers [server] # this is only for the mysqld standalone daemon [mysqld] # thread_handling=pool-of-thread # log=/var/log/mysqld.log # this is only for embedded server [embedded] # This group is only read by MariaDB servers, not by MySQL. # If you use the same .cnf file for MySQL and MariaDB, # you can put MariaDB-only options here [mariadb] # log=/var/log/mysqld.log general_log_file = /var/lib/mysql/mysql.log # general_log = 1 # transaction-isolation = READ-COMMITTED # key_buffer = 1280M # 128MB for every 1GB of RAM # sort_buffer_size = 1M # 1MB for every 1GB of RAM # read_buffer_size = 1M # 1MB for every 1GB of RAM # read_rnd_buffer_size = 1M # 1MB for every 1GB of RAM # thread_concurrency = 24 # Based on the number of CPUs # so make it CPU*2 # thread-handling=pool-of-thread # innodb_flush_log_at_trx_commit != 1 # open_files_limit = 50000 thread-handling=pool-of-thread max_connections = 1000 table_open_cache = 800 query_cache_type = 0 innodb_buffer_pool_size = 512M innodb_buffer_pool_instances = 10 innodb_adaptive_hash_index_par = 20 innodb_lock_wait_timeout = 5000 With this, my application can keep up with the FIFO writer, but - depending on the circumstances - my database can't. As I am writing this, there are over 1300 threads connected to my database; any command that I issue at the mysql CLI takes over one minute to return. I am keeping track on how long each thread takes to complete, and that is of the order of hundreds of seconds - sometimes thousands. Each thread is itself simple, in that it just issues a couple of simple MariaDB commands. Currently my table consists of 1.6 million entries, and growing - on this basis, I expect that things will get only worse. Each entry,however, will never require more than a couple of hundred bytes of storage. The operations that can be undertaken on entries are insertion, deletion and modification, the latter being straightforward - like e.g. incrementing a counter or replacing a short string. My system has 24 GB of RAM and 12 cores. Occasionally all the cores are fully busy with MariaDB activity, but most of the time barely one or two are. I am a newbie when it comes to interacting with MariaDB - please, bear with me. I know I must use a single database and a single table. I also know - because of the nature of the data that are being written to the FIFO - that the probability for two different threads to be operating on the same entry in the table at the same time is negligible - i.e. for all practical purposes, that will not happen. What I need is advice on how to configure my instance of MariaDB to perform optimally in the scenario above. In particular, I would like for it to make better use of all the cores available - in essence, to parallelize the database operations as much as possible. Feedback from the experts will be much appreciated. ______________________________ Mailing list: launchpad.net launchpad.net Post to : maria-discuss(a)lists.launchpad. Unsubscribe : launchpad.net launchpad.net More help : help.launchpad.net help.launchpad.net

2 1

[Maria-discuss] Benchmarking MariaDB/InnoDB using TATP - Writers slow down readers?
by Aarati K 07 Oct '19

07 Oct '19

I'm benchmarking MariaDB 10.4 with TATP benchmark. The InnoDB config file has the following parameters set: [mysqld] innodb_data_home_dir=/usr/local/mysql/data innodb_undo_directory=/usr/local/mysql/data innodb_undo_log_truncate=1 innodb_undo_tablespaces=3 innodb_flush_method=O_DIRECT innodb_page_size=65536 When I run just the read-mix of TATP (i.e, the read queries alone), I get a result as follows: I 2019-10-06 16:06:06 STATISTICS 90% response time (us), by transaction I 2019-10-06 16:06:06 STATISTICS GET_SUBSCRIBER_DATA: 744 I 2019-10-06 16:06:06 STATISTICS GET_NEW_DESTINATION: 1323 I 2019-10-06 16:06:06 STATISTICS GET_ACCESS_DATA: 738 I 2019-10-06 16:06:06 CONTROL MQTh for test run is 6772 Focusing on the first query, GET_SUBSCRIBER_DATA, it has a 90% response time of 744us. However, when I run the R90/W10 mix (i.e, 90% read requests, 10% write requests), the result is as follows: I 2019-10-07 01:23:27 STATISTICS 90% response time (us), by transaction I 2019-10-07 01:23:27 STATISTICS GET_SUBSCRIBER_DATA: 3771 I 2019-10-07 01:23:27 STATISTICS GET_NEW_DESTINATION: 4712 I 2019-10-07 01:23:27 STATISTICS GET_ACCESS_DATA: 3687 I 2019-10-07 01:23:27 STATISTICS UPDATE_SUBSCRIBER_DATA: 18801 I 2019-10-07 01:23:27 STATISTICS UPDATE_LOCATION: 18831 I 2019-10-07 01:23:27 STATISTICS INSERT_CALL_FORWARDING: 20017 I 2019-10-07 01:23:27 STATISTICS DELETE_CALL_FORWARDING: 19027 I 2019-10-07 01:23:27 CONTROL MQTh for test run is 1464 The response time of GET_SUBSCRIBER_DATA is 3771 us, which is much slower. Queries are generated probabilistically, and keys are chosen uniformly at random. However, what I don't understand is, how are the writers slowing down the readers by that much, especially when MariaDB/InnoDB has *row-level locking*, to avoid this issue of writers blocking readers. Are there any configuration parameters that need to be tweaked?

1 0

[Maria-discuss] MariaDB 10.4.8
by Thomas Plant 04 Oct '19

04 Oct '19

Hello, we have a client VM with MariaDB 10.4.8 which crashes always on the same query (it's in the log). Can someone help me to debug the problem, or is it already known problem? Server ist a CentOS 7 patched up to date and MariaDB 10.4.8 from the MariaDB repositories. Log contains the following: 191003 9:43:19 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.8-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=1 max_threads=302 thread_count=7 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 795622 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x55d12dc0ed28 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ff2fd21dcf0 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d12b70483e] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55d12b19ae0f] sigaction.c:0(__restore_rt)[0x7ff3153325f0] /usr/sbin/mysqld(handler_index_cond_check+0xbf)[0x55d12b1a56af] /usr/sbin/mysqld(+0x59ee80)[0x55d12ae94e80] /usr/sbin/mysqld(+0xb340cd)[0x55d12b42a0cd] /usr/sbin/mysqld(+0xa52c87)[0x55d12b348c87] /usr/sbin/mysqld(+0xa52edf)[0x55d12b348edf] /usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x10f)[0x55d12b19f8ff] /usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x1c)[0x55d12b2cde9c] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1a9)[0x55d12afc7dd9] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xb9c)[0x55d12afea4cc] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x55d12afea723] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x55d12afe8a26] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55d12afe9597] /usr/sbin/mysqld(+0x58dd2a)[0x55d12ae83d2a] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4591)[0x55d12af90641] /usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x4b6)[0x55d12afa8d06] /usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x92)[0x55d12afa8ec2] /usr/sbin/mysqld(+0x6b3827)[0x55d12afa9827] /usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x27)[0x55d12afa98b7] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1625)[0x55d12af97d85] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55d12af99aac] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55d12b07718a] /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d12b07726d] pthread_create.c:0(start_thread)[0x7ff31532ae65] /lib64/libc.so.6(clone+0x6d)[0x7ff3136cb88d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x55d12dc24b38): select * from `users` where (`role` = ? or `role` = ?) and exists (select * from `email_templates` inner join `users_email` on `email_template s`.`id` = `users_email`.`email_template_id` where `users`.`id` = `users_email`.`user_id` and `email_template_id` = ?) and `users`.`deleted_at` is null Connection ID (thread ID): 9 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdow n=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_ merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_increme ntal=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition _pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15028 15028 processes Max open files 16364 16364 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15028 15028 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core Thanks for any hint. Kind Regards, Thomas

3 6