Hi, Tom!
On Sep 06, Tom Ryan wrote:
>
> Sep 6 14:37:59 node-2 mysqld[1846]: stack_bottom = 0x7f3271446dd8 thread_stack 0x49000
> Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5644f71dcf4e]
> Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_fatal_signal+0x307)[0x5644f6c70b07]
> Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f33e141d890]
> Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handler_index_cond_check+0x8d)[0x5644f6c7b1fd]
> Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x566623)[0x5644f6973623]
> Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xbb4ba8)[0x5644f6fc1ba8]
This looks truncated. Could you post a complete stack trace please?
Sep 6 14:37:59 node-2 mysqld[1846]: 190906 14:37:59 [ERROR] mysqld got signal 11 ;
Sep 6 14:37:59 node-2 mysqld[1846]: This could be because you hit a bug. It is also possible that this binary
Sep 6 14:37:59 node-2 mysqld[1846]: or one of the libraries it was linked against is corrupt, improperly built,
Sep 6 14:37:59 node-2 mysqld[1846]: or misconfigured. This error can also be caused by malfunctioning hardware.
Sep 6 14:37:59 node-2 mysqld[1846]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Sep 6 14:37:59 node-2 mysqld[1846]: We will try our best to scrape up some info that will hopefully help
Sep 6 14:37:59 node-2 mysqld[1846]: diagnose the problem, but since we have already crashed,
Sep 6 14:37:59 node-2 mysqld[1846]: something is definitely wrong and this may fail.
Sep 6 14:37:59 node-2 mysqld[1846]: Server version: 10.4.7-MariaDB-1:10.4.7+maria~xenial-log
Sep 6 14:37:59 node-2 mysqld[1846]: key_buffer_size=134217728
Sep 6 14:37:59 node-2 mysqld[1846]: read_buffer_size=2097152
Sep 6 14:37:59 node-2 mysqld[1846]: max_used_connections=30
Sep 6 14:37:59 node-2 mysqld[1846]: max_threads=4002
Sep 6 14:37:59 node-2 mysqld[1846]: thread_count=45
Sep 6 14:37:59 node-2 mysqld[1846]: It is possible that mysqld could use up to
Sep 6 14:37:59 node-2 mysqld[1846]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24817728 K bytes of memory
Sep 6 14:37:59 node-2 mysqld[1846]: Hope that's ok; if not, decrease some variables in the equation.
Sep 6 14:37:59 node-2 mysqld[1846]: Thread pointer: 0x7f32052a0348
Sep 6 14:37:59 node-2 mysqld[1846]: Attempting backtrace. You can use the following information to find out
Sep 6 14:37:59 node-2 mysqld[1846]: where mysqld died. If you see no messages after this, something went
Sep 6 14:37:59 node-2 mysqld[1846]: terribly wrong...
Sep 6 14:37:59 node-2 mysqld[1846]: stack_bottom = 0x7f3271446dd8 thread_stack 0x49000
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5644f71dcf4e]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_fatal_signal+0x307)[0x5644f6c70b07]
Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f33e141d890]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handler_index_cond_check+0x8d)[0x5644f6c7b1fd]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x566623)[0x5644f6973623]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xbb4ba8)[0x5644f6fc1ba8]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xad1cfa)[0x5644f6edecfa]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xad1fa3)[0x5644f6edefa3]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x12f)[0x5644f6c75acf]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x22)[0x5644f6d988e2]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x164)[0x5644f6a97034]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xc9f)[0x5644f6ab85af]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x5644f6ab8833]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x5644
f6ab6b26]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0xf6)[0x5644f6ab7456]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x55756d)[0x5644f696456d]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x7371)[0x5644f6a61741]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x23a)[0x5644f6a632da]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x656b59)[0x5644f6a63b59]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x2633)[0x5644f6a66ae3]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5644f6a670ac]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x281)[0x5644f6b43691]
Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5644f6b437bd]
Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f33e14126db]
Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f33e081088f]
Sep 6 14:37:59 node-2 mysqld[1846]: Trying to get some variables.
Sep 6 14:37:59 node-2 mysqld[1846]: Some pointers may be invalid and cause the dump to abort.
Sep 6 14:37:59 node-2 mysqld[1846]: Query (0x7f3205209000): SELECT field_data_field_weight.field_weight_value AS field_data_field_weight_field_weight_value, node.nid AS nid, DATE_FORMAT((DATE_A
DD('19700101', INTERVAL node.created SECOND) + INTERVAL -14400 SECOND), '%Y%m') AS node_created_month FROM node node LEFT JOIN field_data_field_weight field_data_field_weight ON node.nid = fiel
d_data_field_weight.entity_id AND (field_data_field_weight.entity_type = 'node' AND field_data_field_weight.deleted = '0') WHERE (( (node.status = 1 OR (node.uid = 0 AND 0 <> 0 AND 0 = 1) OR 0 =
1) AND (node.nid IN (SELECT tn.nid AS nid FROM taxonomy_index tn WHERE ( (tn.tid = '3') ))) ))AND ( EXISTS (SELECT na.nid AS nid FROM node_access na WHERE (( (na.gid = '0') AND (na.realm =
'all') )OR( (na.gid = '0') AND (na.realm = 'content_access_author') )OR( (na.gid = '1') AND (na.realm = 'content_access_rid') ))AND (na.grant_view >= '1') AND (node.nid = na.nid) )) ORDER BY no
de_created_month DESC, field_data_field_weight_field_weight_value DESC LIMIT 15 OFFSET 0
Sep 6 14:37:59 node-2 mysqld[1846]: Connection ID (thread ID): 15972
Sep 6 14:37:59 node-2 mysqld[1846]: Status: NOT_KILLED
Sep 6 14:37:59 node-2 mysqld[1846]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_conditio
n_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_ma
tch_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,
optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subqu
ery=on,rowid_filter=on,condition_pushdown_from_having=on
Sep 6 14:37:59 node-2 mysqld[1846]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Sep 6 14:37:59 node-2 mysqld[1846]: information that should help you find out what is causing the crash.
Sep 6 14:37:59 node-2 mysqld[1846]: Writing a core file...
Sep 6 14:37:59 node-2 mysqld[1846]: Working directory at /var/lib/mysql
Sep 6 14:37:59 node-2 mysqld[1846]: Resource Limits:
Sep 6 14:37:59 node-2 mysqld[1846]: Limit Soft Limit Hard Limit Units
Sep 6 14:37:59 node-2 mysqld[1846]: Max cpu time unlimited unlimited seconds
Sep 6 14:37:59 node-2 mysqld[1846]: Max file size unlimited unlimited bytes
Sep 6 14:37:59 node-2 mysqld[1846]: Max data size unlimited unlimited bytes
Sep 6 14:37:59 node-2 mysqld[1846]: Max stack size 8388608 unlimited bytes
Sep 6 14:37:59 node-2 mysqld[1846]: Max core file size 0 unlimited bytes
Sep 6 14:37:59 node-2 mysqld[1846]: Max resident set unlimited unlimited bytes
Sep 6 14:37:59 node-2 mysqld[1846]: Max processes 128431 128431 processes
Sep 6 14:37:59 node-2 mysqld[1846]: Max open files 16364 16364 files
Sep 6 14:37:59 node-2 mysqld[1846]: Max locked memory 16777216 16777216 bytes
Sep 6 14:37:59 node-2 mysqld[1846]: Max address space unlimited unlimited bytes
Sep 6 14:37:59 node-2 mysqld[1846]: Max file locks unlimited unlimited locks
Sep 6 14:37:59 node-2 mysqld[1846]: Max pending signals 128431 128431 signals
Sep 6 14:37:59 node-2 mysqld[1846]: Max msgqueue size 819200 819200 bytes
Sep 6 14:37:59 node-2 mysqld[1846]: Max nice priority 0 0
Sep 6 14:37:59 node-2 mysqld[1846]: Max realtime priority 0 0
Sep 6 14:37:59 node-2 mysqld[1846]: Max realtime timeout unlimited unlimited us
Sep 6 14:37:59 node-2 mysqld[1846]: Core pattern: |/usr/share/apport/apport %p %s %c %d %P
Sep 6 14:37:59 node-2 kernel: [31368.222207] show_signal_msg: 7 callbacks suppressed
Sep 6 14:37:59 node-2 kernel: [31368.222210] mysqld[3138]: segfault at 0 ip 00005644f6c7b1fd sp 00007f32714417b0 error 4
Sep 7 05:12:07 node-1 mysqld[18950]: This could be because you hit a bug. It is also possible that this binary
Sep 7 05:12:07 node-1 mysqld[18950]: or one of the libraries it was linked against is corrupt, improperly built,
Sep 7 05:12:07 node-1 mysqld[18950]: or misconfigured. This error can also be caused by malfunctioning hardware.
Sep 7 05:12:07 node-1 mysqld[18950]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Sep 7 05:12:07 node-1 mysqld[18950]: We will try our best to scrape up some info that will hopefully help
Sep 7 05:12:07 node-1 mysqld[18950]: diagnose the problem, but since we have already crashed,
Sep 7 05:12:07 node-1 mysqld[18950]: something is definitely wrong and this may fail.
Sep 7 05:12:07 node-1 mysqld[18950]: Server version: 10.4.7-MariaDB-1:10.4.7+maria~bionic-log
Sep 7 05:12:07 node-1 mysqld[18950]: key_buffer_size=134217728
Sep 7 05:12:07 node-1 mysqld[18950]: read_buffer_size=2097152
Sep 7 05:12:07 node-1 mysqld[18950]: max_used_connections=27
Sep 7 05:12:07 node-1 mysqld[18950]: max_threads=4002
Sep 7 05:12:07 node-1 mysqld[18950]: thread_count=42
Sep 7 05:12:07 node-1 mysqld[18950]: It is possible that mysqld could use up to
Sep 7 05:12:07 node-1 mysqld[18950]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24817728 K bytes of memory
Sep 7 05:12:07 node-1 mysqld[18950]: Hope that's ok; if not, decrease some variables in the equation.
Sep 7 05:12:07 node-1 mysqld[18950]: Thread pointer: 0x7f1adc00b618
Sep 7 05:12:07 node-1 mysqld[18950]: Attempting backtrace. You can use the following information to find out
Sep 7 05:12:07 node-1 mysqld[18950]: where mysqld died. If you see no messages after this, something went
Sep 7 05:12:07 node-1 mysqld[18950]: terribly wrong...
Sep 7 05:12:07 node-1 mysqld[18950]: stack_bottom = 0x7f199fecadd8 thread_stack 0x49000
Sep 7 05:12:07 node-1 mysqld[18950]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a2e3675cee]
Sep 7 05:12:07 node-1 mysqld[18950]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55a2e30ed085]
Sep 7 05:12:07 node-1 mysqld[18950]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f1b115d0890]
Sep 7 05:12:07 node-1 kernel: [84233.145921] mysqld[26579]: segfault at 0 ip 000055a2e30f7500 sp 00007f19905b9e10 error 4 in mysqld[55a2e2896000+13d6000]
Sep 7 05:12:07 node-1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
Sep 7 05:12:07 node-1 systemd[1]: mariadb.service: Failed with result 'signal'.
And please tell exactly what binaries you're using. Version,
architecture, Linux distribution.
/usr/sbin/mysqld: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=cdf04892e30322f0ee59a1aa604e6087ad040231, stripped
> This is getting old and is unlike our usual mariadb experience.
>
> Looking at the jira for 10.4.8, it appears to fix many problems in
> 10.4 and I'm hopeful it solves this.
>
> In the meantime, any ideas or suggestions for things to look for?
10.4.8 should be out any day now. But if I'd know more about the crash,
I could tell whether it's fixed there or not.
Regards,
Sergei