New subject: [Maria-discuss] Troubleshooting Statement That Crashes MariaDB (mysqld got signal 11)

4 Nov 2021

      Hello,

I have a nothing fancy SQL statement that crashes MariaDB 10.3.31.  The 
error log reports "mysqld got signal 11" with the below included mini 
core dump.

The statement caused no issues with my prior install of 10.2.xx. With 
10.3, the statement runs the majority of the time.  The vast majority of 
the time.

I have never dealt with such an issue with MariaDB, and am unsure where 
to start with troubleshooting this.  Any tips?

FYI, there is a corresponding bug recorded here: 
https://jira.mariadb.org/browse/MDEV-26835

Thanks,

Mike

211014 16:13:53 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.3.31-MariaDB-1:10.3.31+maria~focal-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=587
max_threads=752
thread_count=81
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 
1784362 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f13e83b3768
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f1644e96dd8 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x32)[0x55d5a2f6f3a2]
/usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x55d5a2a46e8d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f27569fc3c0]
/usr/sbin/mysqld(+0x978f4f)[0x55d5a2bdbf4f]
/usr/sbin/mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x50)[0x55d5a2a4c6d0]
/usr/sbin/mysqld(_Z16rr_from_pointersP11READ_RECORD+0x3e)[0x55d5a2b7cb2e]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x18e)[0x55d5a28869be]
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xbee)[0x55d5a28b0dce]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x37)[0x55d5a28b1137]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xf2)[0x55d5a28b1282]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x14b)[0x55d5a28b1bcb]
/usr/sbin/mysqld(+0x5de9f1)[0x55d5a28419f1]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x5414)[0x55d5a284f6e4]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1f3)[0x55d5a2852253]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x105d)[0x55d5a285462d]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x12d)[0x55d5a28565dd]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x246)[0x55d5a293f186]
/usr/sbin/mysqld(handle_one_connection+0x3f)[0x55d5a293f35f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f27569f0609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f2756915293]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f13e84a8b70): SELECT * FROM ((
             #
             # get postings from last statement and postings not on 
statements
             #
             SELECT a.*, COALESCE(ci.id, i.id) as invoice_id
             FROM ddx_lab_801317.postings a
             LEFT JOIN ddx_lab_801317.cases_invoices ci
             ON a.ref_type = 'CASE_INVOICE' AND a.ref_id = ci.id
             LEFT JOIN ddx_lab_801317.invoices i
             ON a.ref_type = 'INVOICE' AND a.ref_id = i.id
             WHERE (
                 a.statement_date IS NULL
                 OR a.statement_date = '2021-09-27'
             )
          AND a.practice_id='19686'
             ORDER BY post_date
         ) UNION (
             #
             # get postings that have remaining balances
             #
             SELECT DISTINCT a.*, COALESCE(ci.id, i.id) as invoice_id
             FROM ddx_lab_801317.postings a
             LEFT JOIN ddx_lab_801317.cases_invoices ci
             ON a.ref_type = 'CASE_INVOICE' AND a.ref_id = ci.id
             LEFT JOIN ddx_lab_801317.invoices i
             ON a.ref_type = 'INVOICE' AND a.ref_id = i.id
             LEFT JOIN ddx_lab_801317.postings b
                 ON a.ref_type = b.ref_type
                 AND a.ref_id = b.ref_id
                 AND a.id != b.id

             WHERE NOT (a.type = 'CREDIT' OR a.type = 'PAYMENT')
             AND ((SELECT SUM(b.amount)) IS NULL OR (SELECT 
SUM(a.amount)) + (SELECT SUM(b.amount)) > 0)
          AND a.practice_id='19686'
             GROUP BY a.id
             ORDER BY post_date
         ) UNION (
             #
             # get partial payments and credits from older statements
             #
             SELECT a.*, COALESCE(ci.id, i.id) as invoice_id
             FROM ddx_lab_801317.postings a
             LEFT JOIN ddx_lab_801317.cases_invoices ci
             ON a.ref_type = 'CASE_INVOICE' AND a.ref_id = ci.id
             LEFT JOIN ddx_lab_801317.invoices i
             ON a.ref_type = 'INVOICE' AND a.ref_id = i.id
             WHERE (
                 a.statement_date < '2021-09-27'
             )
             AND (a.type='PAYMENT' OR a.type='CREDIT')
          AND a.practice_id='19686' )) AS t1 ORDER BY post_date ASC, 
invoice_id ASC

Connection ID (thread ID): 21014956
Status: NOT_KILLED

Optimizer switch: 
index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

The manual page at 
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ 
contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited unlimited            seconds
Max file size             unlimited unlimited            bytes
Max data size             unlimited unlimited            bytes
Max stack size            8388608 unlimited            bytes
Max core file size        0 unlimited            bytes
Max resident set          unlimited unlimited            bytes
Max processes             unlimited unlimited            processes
Max open files            1048576 1048576              files
Max locked memory         unlimited unlimited            bytes
Max address space         unlimited unlimited            bytes
Max file locks            unlimited unlimited            locks
Max pending signals       514974 514974               signals
Max msgqueue size         819200 819200               bytes
Max nice priority         0 0
Max realtime priority     0 0
Max realtime timeout      unlimited unlimited            us
Core pattern: core

[Maria-discuss] Troubleshooting Statement That Crashes MariaDB (mysqld got signal 11)

Michael Caplan

Gordan Bobic

Michael Caplan

tags

participants (2)