On Fri, Sep 6, 2019 at 5:56 PM Sergei Golubchik <serg@mariadb.org> wrote:
Hi, Tom!
On Sep 06, Tom Ryan wrote:
Sep 6 14:37:59 node-2 mysqld[1846]: stack_bottom = 0x7f3271446dd8
thread_stack 0x49000
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5644f71dcf4e] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_fatal_signal+0x307)[0x5644f6c70b07] Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f33e141d890] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handler_index_cond_check+0x8d)[0x5644f6c7b1fd] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x566623)[0x5644f6973623] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xbb4ba8)[0x5644f6fc1ba8]
This looks truncated. Could you post a complete stack trace please?
Sep 6 14:37:59 node-2 mysqld[1846]: 190906 14:37:59 [ERROR] mysqld got signal 11 ; Sep 6 14:37:59 node-2 mysqld[1846]: This could be because you hit a bug. It is also possible that this binary Sep 6 14:37:59 node-2 mysqld[1846]: or one of the libraries it was linked against is corrupt, improperly built, Sep 6 14:37:59 node-2 mysqld[1846]: or misconfigured. This error can also be caused by malfunctioning hardware. Sep 6 14:37:59 node-2 mysqld[1846]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Sep 6 14:37:59 node-2 mysqld[1846]: We will try our best to scrape up some info that will hopefully help Sep 6 14:37:59 node-2 mysqld[1846]: diagnose the problem, but since we have already crashed, Sep 6 14:37:59 node-2 mysqld[1846]: something is definitely wrong and this may fail. Sep 6 14:37:59 node-2 mysqld[1846]: Server version: 10.4.7-MariaDB-1:10.4.7+maria~xenial-log Sep 6 14:37:59 node-2 mysqld[1846]: key_buffer_size=134217728 Sep 6 14:37:59 node-2 mysqld[1846]: read_buffer_size=2097152 Sep 6 14:37:59 node-2 mysqld[1846]: max_used_connections=30 Sep 6 14:37:59 node-2 mysqld[1846]: max_threads=4002 Sep 6 14:37:59 node-2 mysqld[1846]: thread_count=45 Sep 6 14:37:59 node-2 mysqld[1846]: It is possible that mysqld could use up to Sep 6 14:37:59 node-2 mysqld[1846]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24817728 K bytes of memory Sep 6 14:37:59 node-2 mysqld[1846]: Hope that's ok; if not, decrease some variables in the equation. Sep 6 14:37:59 node-2 mysqld[1846]: Thread pointer: 0x7f32052a0348 Sep 6 14:37:59 node-2 mysqld[1846]: Attempting backtrace. You can use the following information to find out Sep 6 14:37:59 node-2 mysqld[1846]: where mysqld died. If you see no messages after this, something went Sep 6 14:37:59 node-2 mysqld[1846]: terribly wrong... Sep 6 14:37:59 node-2 mysqld[1846]: stack_bottom = 0x7f3271446dd8 thread_stack 0x49000 Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5644f71dcf4e] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_fatal_signal+0x307)[0x5644f6c70b07] Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f33e141d890] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handler_index_cond_check+0x8d)[0x5644f6c7b1fd] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x566623)[0x5644f6973623] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xbb4ba8)[0x5644f6fc1ba8] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xad1cfa)[0x5644f6edecfa] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xad1fa3)[0x5644f6edefa3] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x12f)[0x5644f6c75acf] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x22)[0x5644f6d988e2] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x164)[0x5644f6a97034] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xc9f)[0x5644f6ab85af] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x5644f6ab8833] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x5644 f6ab6b26] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0xf6)[0x5644f6ab7456] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x55756d)[0x5644f696456d] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x7371)[0x5644f6a61741] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x23a)[0x5644f6a632da] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x656b59)[0x5644f6a63b59] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x2633)[0x5644f6a66ae3] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5644f6a670ac] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x281)[0x5644f6b43691] Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5644f6b437bd] Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f33e14126db] Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f33e081088f] Sep 6 14:37:59 node-2 mysqld[1846]: Trying to get some variables. Sep 6 14:37:59 node-2 mysqld[1846]: Some pointers may be invalid and cause the dump to abort. Sep 6 14:37:59 node-2 mysqld[1846]: Query (0x7f3205209000): SELECT field_data_field_weight.field_weight_value AS field_data_field_weight_field_weight_value, node.nid AS nid, DATE_FORMAT((DATE_A DD('19700101', INTERVAL node.created SECOND) + INTERVAL -14400 SECOND), '%Y%m') AS node_created_month FROM node node LEFT JOIN field_data_field_weight field_data_field_weight ON node.nid = fiel d_data_field_weight.entity_id AND (field_data_field_weight.entity_type = 'node' AND field_data_field_weight.deleted = '0') WHERE (( (node.status = 1 OR (node.uid = 0 AND 0 <> 0 AND 0 = 1) OR 0 = 1) AND (node.nid IN (SELECT tn.nid AS nid FROM taxonomy_index tn WHERE ( (tn.tid = '3') ))) ))AND ( EXISTS (SELECT na.nid AS nid FROM node_access na WHERE (( (na.gid = '0') AND (na.realm = 'all') )OR( (na.gid = '0') AND (na.realm = 'content_access_author') )OR( (na.gid = '1') AND (na.realm = 'content_access_rid') ))AND (na.grant_view
= '1') AND (node.nid = na.nid) )) ORDER BY no
de_created_month DESC, field_data_field_weight_field_weight_value DESC LIMIT 15 OFFSET 0 Sep 6 14:37:59 node-2 mysqld[1846]: Connection ID (thread ID): 15972 Sep 6 14:37:59 node-2 mysqld[1846]: Status: NOT_KILLED Sep 6 14:37:59 node-2 mysqld[1846]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_conditio n_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_ma tch_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on, optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subqu ery=on,rowid_filter=on,condition_pushdown_from_having=on Sep 6 14:37:59 node-2 mysqld[1846]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Sep 6 14:37:59 node-2 mysqld[1846]: information that should help you find out what is causing the crash. Sep 6 14:37:59 node-2 mysqld[1846]: Writing a core file... Sep 6 14:37:59 node-2 mysqld[1846]: Working directory at /var/lib/mysql Sep 6 14:37:59 node-2 mysqld[1846]: Resource Limits: Sep 6 14:37:59 node-2 mysqld[1846]: Limit Soft Limit Hard Limit Units Sep 6 14:37:59 node-2 mysqld[1846]: Max cpu time unlimited unlimited seconds Sep 6 14:37:59 node-2 mysqld[1846]: Max file size unlimited unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max data size unlimited unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max stack size 8388608 unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max core file size 0 unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max resident set unlimited unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max processes 128431 128431 processes Sep 6 14:37:59 node-2 mysqld[1846]: Max open files 16364 16364 files Sep 6 14:37:59 node-2 mysqld[1846]: Max locked memory 16777216 16777216 bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max address space unlimited unlimited bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max file locks unlimited unlimited locks Sep 6 14:37:59 node-2 mysqld[1846]: Max pending signals 128431 128431 signals Sep 6 14:37:59 node-2 mysqld[1846]: Max msgqueue size 819200 819200 bytes Sep 6 14:37:59 node-2 mysqld[1846]: Max nice priority 0 0 Sep 6 14:37:59 node-2 mysqld[1846]: Max realtime priority 0 0 Sep 6 14:37:59 node-2 mysqld[1846]: Max realtime timeout unlimited unlimited us Sep 6 14:37:59 node-2 mysqld[1846]: Core pattern: |/usr/share/apport/apport %p %s %c %d %P Sep 6 14:37:59 node-2 kernel: [31368.222207] show_signal_msg: 7 callbacks suppressed Sep 6 14:37:59 node-2 kernel: [31368.222210] mysqld[3138]: segfault at 0 ip 00005644f6c7b1fd sp 00007f32714417b0 error 4 here's one from this morning Sep 7 05:12:07 node-1 mysqld[18950]: This could be because you hit a bug. It is also possible that this binary Sep 7 05:12:07 node-1 mysqld[18950]: or one of the libraries it was linked against is corrupt, improperly built, Sep 7 05:12:07 node-1 mysqld[18950]: or misconfigured. This error can also be caused by malfunctioning hardware. Sep 7 05:12:07 node-1 mysqld[18950]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Sep 7 05:12:07 node-1 mysqld[18950]: We will try our best to scrape up some info that will hopefully help Sep 7 05:12:07 node-1 mysqld[18950]: diagnose the problem, but since we have already crashed, Sep 7 05:12:07 node-1 mysqld[18950]: something is definitely wrong and this may fail. Sep 7 05:12:07 node-1 mysqld[18950]: Server version: 10.4.7-MariaDB-1:10.4.7+maria~bionic-log Sep 7 05:12:07 node-1 mysqld[18950]: key_buffer_size=134217728 Sep 7 05:12:07 node-1 mysqld[18950]: read_buffer_size=2097152 Sep 7 05:12:07 node-1 mysqld[18950]: max_used_connections=27 Sep 7 05:12:07 node-1 mysqld[18950]: max_threads=4002 Sep 7 05:12:07 node-1 mysqld[18950]: thread_count=42 Sep 7 05:12:07 node-1 mysqld[18950]: It is possible that mysqld could use up to Sep 7 05:12:07 node-1 mysqld[18950]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24817728 K bytes of memory Sep 7 05:12:07 node-1 mysqld[18950]: Hope that's ok; if not, decrease some variables in the equation. Sep 7 05:12:07 node-1 mysqld[18950]: Thread pointer: 0x7f1adc00b618 Sep 7 05:12:07 node-1 mysqld[18950]: Attempting backtrace. You can use the following information to find out Sep 7 05:12:07 node-1 mysqld[18950]: where mysqld died. If you see no messages after this, something went Sep 7 05:12:07 node-1 mysqld[18950]: terribly wrong... Sep 7 05:12:07 node-1 mysqld[18950]: stack_bottom = 0x7f199fecadd8 thread_stack 0x49000 Sep 7 05:12:07 node-1 mysqld[18950]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a2e3675cee] Sep 7 05:12:07 node-1 mysqld[18950]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55a2e30ed085] Sep 7 05:12:07 node-1 mysqld[18950]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f1b115d0890] Sep 7 05:12:07 node-1 kernel: [84233.145921] mysqld[26579]: segfault at 0 ip 000055a2e30f7500 sp 00007f19905b9e10 error 4 in mysqld[55a2e2896000+13d6000] Sep 7 05:12:07 node-1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV Sep 7 05:12:07 node-1 systemd[1]: mariadb.service: Failed with result 'signal'.
And please tell exactly what binaries you're using. Version, architecture, Linux distribution.
ubuntu 18.04 mariadb respository /usr/sbin/mysqld: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=cdf04892e30322f0ee59a1aa604e6087ad040231, stripped
This is getting old and is unlike our usual mariadb experience.
Looking at the jira for 10.4.8, it appears to fix many problems in 10.4 and I'm hopeful it solves this.
In the meantime, any ideas or suggestions for things to look for?
10.4.8 should be out any day now. But if I'd know more about the crash, I could tell whether it's fixed there or not.
Regards, Sergei