On Fri, Sep 6, 2019 at 5:56 PM Sergei Golubchik <serg@mariadb.org> wrote:
Hi, Tom!

On Sep 06, Tom Ryan wrote:
>
> Sep 6 14:37:59 node-2 mysqld[1846]: stack_bottom = 0x7f3271446dd8 thread_stack 0x49000
> Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5644f71dcf4e]
> Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_fatal_signal+0x307)[0x5644f6c70b07]
> Sep 6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f33e141d890]
> Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handler_index_cond_check+0x8d)[0x5644f6c7b1fd]
> Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x566623)[0x5644f6973623]
> Sep 6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xbb4ba8)[0x5644f6fc1ba8]

This looks truncated. Could you post a complete stack trace please?

Sep  6 14:37:59 node-2 mysqld[1846]: 190906 14:37:59 [ERROR] mysqld got signal 11 ;

Sep  6 14:37:59 node-2 mysqld[1846]: This could be because you hit a bug. It is also possible that this binary

Sep  6 14:37:59 node-2 mysqld[1846]: or one of the libraries it was linked against is corrupt, improperly built,

Sep  6 14:37:59 node-2 mysqld[1846]: or misconfigured. This error can also be caused by malfunctioning hardware.

Sep  6 14:37:59 node-2 mysqld[1846]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs

Sep  6 14:37:59 node-2 mysqld[1846]: We will try our best to scrape up some info that will hopefully help

Sep  6 14:37:59 node-2 mysqld[1846]: diagnose the problem, but since we have already crashed,

Sep  6 14:37:59 node-2 mysqld[1846]: something is definitely wrong and this may fail.

Sep  6 14:37:59 node-2 mysqld[1846]: Server version: 10.4.7-MariaDB-1:10.4.7+maria~xenial-log

Sep  6 14:37:59 node-2 mysqld[1846]: key_buffer_size=134217728

Sep  6 14:37:59 node-2 mysqld[1846]: read_buffer_size=2097152

Sep  6 14:37:59 node-2 mysqld[1846]: max_used_connections=30

Sep  6 14:37:59 node-2 mysqld[1846]: max_threads=4002

Sep  6 14:37:59 node-2 mysqld[1846]: thread_count=45

Sep  6 14:37:59 node-2 mysqld[1846]: It is possible that mysqld could use up to

Sep  6 14:37:59 node-2 mysqld[1846]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24817728 K  bytes of memory

Sep  6 14:37:59 node-2 mysqld[1846]: Hope that's ok; if not, decrease some variables in the equation.

Sep  6 14:37:59 node-2 mysqld[1846]: Thread pointer: 0x7f32052a0348

Sep  6 14:37:59 node-2 mysqld[1846]: Attempting backtrace. You can use the following information to find out

Sep  6 14:37:59 node-2 mysqld[1846]: where mysqld died. If you see no messages after this, something went

Sep  6 14:37:59 node-2 mysqld[1846]: terribly wrong...

Sep  6 14:37:59 node-2 mysqld[1846]: stack_bottom = 0x7f3271446dd8 thread_stack 0x49000

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5644f71dcf4e]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_fatal_signal+0x307)[0x5644f6c70b07]

Sep  6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f33e141d890]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handler_index_cond_check+0x8d)[0x5644f6c7b1fd]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x566623)[0x5644f6973623]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xbb4ba8)[0x5644f6fc1ba8]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xad1cfa)[0x5644f6edecfa]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0xad1fa3)[0x5644f6edefa3]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x12f)[0x5644f6c75acf]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x22)[0x5644f6d988e2]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x164)[0x5644f6a97034]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xc9f)[0x5644f6ab85af]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x5644f6ab8833]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x5644

f6ab6b26]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0xf6)[0x5644f6ab7456]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x55756d)[0x5644f696456d]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x7371)[0x5644f6a61741]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x23a)[0x5644f6a632da]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(+0x656b59)[0x5644f6a63b59]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x2633)[0x5644f6a66ae3]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5644f6a670ac]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x281)[0x5644f6b43691]

Sep  6 14:37:59 node-2 mysqld[1846]: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x5644f6b437bd]

Sep  6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f33e14126db]

Sep  6 14:37:59 node-2 mysqld[1846]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f33e081088f]

Sep  6 14:37:59 node-2 mysqld[1846]: Trying to get some variables.

Sep  6 14:37:59 node-2 mysqld[1846]: Some pointers may be invalid and cause the dump to abort.

Sep  6 14:37:59 node-2 mysqld[1846]: Query (0x7f3205209000): SELECT field_data_field_weight.field_weight_value AS field_data_field_weight_field_weight_value, node.nid AS nid, DATE_FORMAT((DATE_A

DD('19700101', INTERVAL node.created SECOND) + INTERVAL -14400 SECOND), '%Y%m') AS node_created_month FROM  node node LEFT JOIN field_data_field_weight field_data_field_weight ON node.nid = fiel

d_data_field_weight.entity_id AND (field_data_field_weight.entity_type = 'node' AND field_data_field_weight.deleted = '0') WHERE (( (node.status = 1 OR (node.uid = 0 AND 0 <> 0 AND 0 = 1) OR 0 =

 1) AND (node.nid IN  (SELECT tn.nid AS nid FROM  taxonomy_index tn WHERE ( (tn.tid = '3') ))) ))AND ( EXISTS  (SELECT na.nid AS nid FROM  node_access na WHERE (( (na.gid = '0') AND (na.realm = 

'all') )OR( (na.gid = '0') AND (na.realm = 'content_access_author') )OR( (na.gid = '1') AND (na.realm = 'content_access_rid') ))AND (na.grant_view >= '1') AND (node.nid = na.nid) ))  ORDER BY no

de_created_month DESC, field_data_field_weight_field_weight_value DESC LIMIT 15 OFFSET 0

Sep  6 14:37:59 node-2 mysqld[1846]: Connection ID (thread ID): 15972

Sep  6 14:37:59 node-2 mysqld[1846]: Status: NOT_KILLED

Sep  6 14:37:59 node-2 mysqld[1846]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_conditio

n_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_ma

tch_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,

optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subqu

ery=on,rowid_filter=on,condition_pushdown_from_having=on

Sep  6 14:37:59 node-2 mysqld[1846]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

Sep  6 14:37:59 node-2 mysqld[1846]: information that should help you find out what is causing the crash.

Sep  6 14:37:59 node-2 mysqld[1846]: Writing a core file...

Sep  6 14:37:59 node-2 mysqld[1846]: Working directory at /var/lib/mysql

Sep  6 14:37:59 node-2 mysqld[1846]: Resource Limits:

Sep  6 14:37:59 node-2 mysqld[1846]: Limit                     Soft Limit           Hard Limit           Units

Sep  6 14:37:59 node-2 mysqld[1846]: Max cpu time              unlimited            unlimited            seconds

Sep  6 14:37:59 node-2 mysqld[1846]: Max file size             unlimited            unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max data size             unlimited            unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max stack size            8388608              unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max core file size        0                    unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max resident set          unlimited            unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max processes             128431               128431               processes

Sep  6 14:37:59 node-2 mysqld[1846]: Max open files            16364                16364                files

Sep  6 14:37:59 node-2 mysqld[1846]: Max locked memory         16777216             16777216             bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max address space         unlimited            unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max file locks            unlimited            unlimited            locks

Sep  6 14:37:59 node-2 mysqld[1846]: Max pending signals       128431               128431               signals

Sep  6 14:37:59 node-2 mysqld[1846]: Max msgqueue size         819200               819200               bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max nice priority         0                    0

Sep  6 14:37:59 node-2 mysqld[1846]: Max realtime priority     0                    0

Sep  6 14:37:59 node-2 mysqld[1846]: Max realtime timeout      unlimited            unlimited            us

Sep  6 14:37:59 node-2 mysqld[1846]: Core pattern: |/usr/share/apport/apport %p %s %c %d %P

Sep  6 14:37:59 node-2 kernel: [31368.222207] show_signal_msg: 7 callbacks suppressed

Sep  6 14:37:59 node-2 kernel: [31368.222210] mysqld[3138]: segfault at 0 ip 00005644f6c7b1fd sp 00007f32714417b0 error 4


here's one from this morning

Sep  7 05:12:07 node-1 mysqld[18950]: This could be because you hit a bug. It is also possible that this binary

Sep  7 05:12:07 node-1 mysqld[18950]: or one of the libraries it was linked against is corrupt, improperly built,

Sep  7 05:12:07 node-1 mysqld[18950]: or misconfigured. This error can also be caused by malfunctioning hardware.

Sep  7 05:12:07 node-1 mysqld[18950]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs

Sep  7 05:12:07 node-1 mysqld[18950]: We will try our best to scrape up some info that will hopefully help

Sep  7 05:12:07 node-1 mysqld[18950]: diagnose the problem, but since we have already crashed,

Sep  7 05:12:07 node-1 mysqld[18950]: something is definitely wrong and this may fail.

Sep  7 05:12:07 node-1 mysqld[18950]: Server version: 10.4.7-MariaDB-1:10.4.7+maria~bionic-log

Sep  7 05:12:07 node-1 mysqld[18950]: key_buffer_size=134217728

Sep  7 05:12:07 node-1 mysqld[18950]: read_buffer_size=2097152

Sep  7 05:12:07 node-1 mysqld[18950]: max_used_connections=27

Sep  7 05:12:07 node-1 mysqld[18950]: max_threads=4002

Sep  7 05:12:07 node-1 mysqld[18950]: thread_count=42

Sep  7 05:12:07 node-1 mysqld[18950]: It is possible that mysqld could use up to

Sep  7 05:12:07 node-1 mysqld[18950]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24817728 K  bytes of memory

Sep  7 05:12:07 node-1 mysqld[18950]: Hope that's ok; if not, decrease some variables in the equation.

Sep  7 05:12:07 node-1 mysqld[18950]: Thread pointer: 0x7f1adc00b618

Sep  7 05:12:07 node-1 mysqld[18950]: Attempting backtrace. You can use the following information to find out

Sep  7 05:12:07 node-1 mysqld[18950]: where mysqld died. If you see no messages after this, something went

Sep  7 05:12:07 node-1 mysqld[18950]: terribly wrong...

Sep  7 05:12:07 node-1 mysqld[18950]: stack_bottom = 0x7f199fecadd8 thread_stack 0x49000

Sep  7 05:12:07 node-1 mysqld[18950]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a2e3675cee]

Sep  7 05:12:07 node-1 mysqld[18950]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55a2e30ed085]

Sep  7 05:12:07 node-1 mysqld[18950]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f1b115d0890]

Sep  7 05:12:07 node-1 kernel: [84233.145921] mysqld[26579]: segfault at 0 ip 000055a2e30f7500 sp 00007f19905b9e10 error 4 in mysqld[55a2e2896000+13d6000]

Sep  7 05:12:07 node-1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV

Sep  7 05:12:07 node-1 systemd[1]: mariadb.service: Failed with result 'signal'.

 

And please tell exactly what binaries you're using. Version,
architecture, Linux distribution.


ubuntu 18.04
mariadb respository

/usr/sbin/mysqld: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=cdf04892e30322f0ee59a1aa604e6087ad040231, stripped



 
> This is getting old and is unlike our usual mariadb experience.
>
> Looking at the jira for 10.4.8, it appears to fix many problems in
> 10.4 and I'm hopeful it solves this.
>
> In the meantime, any ideas or suggestions for things to look for?

10.4.8 should be out any day now. But if I'd know more about the crash,
I could tell whether it's fixed there or not.

Regards,
Sergei