Thanks Sergie

Been a while since I've worked with core dumps.  Gdd is telling me the crash files are not a recognised file format.  Installed systemd-coredump instead and got this from coredumpctl dump --output=core

           PID: 45641 (mariadbd)
           UID: 124 (mysql)
           GID: 134 (mysql)
        Signal: 11 (SEGV)
     Timestamp: Sat 2025-06-07 14:43:13 UTC (3s ago)
  Command Line: /usr/sbin/mariadbd
    Executable: /usr/sbin/mariadbd
 Control Group: /system.slice/mariadb.service
          Unit: mariadb.service
         Slice: system.slice
       Boot ID: 73cd144f6e9d4838b017b061212a9eb5
    Machine ID: e7027c3ebde74f4a870b311db52087a0
      Hostname: debbie.e-learndesign.scot
       Storage: /var/lib/systemd/coredump/core.mariadbd.124.73cd144f6e9d4838b017b061212a9eb5.45641.1749307393000000.zst (present)
  Size on Disk: 2.0M
       Package: mariadb/1:11.4.7+maria~ubu2404
      build-id: 082d7769ab844a0690ee7a7fbf3888d73ca70585
       Message: Process 45641 (mariadbd) of user 124 dumped core.
                
                Module file_key_management.so from deb mariadb-1:11.4.7+maria~ubu2404.amd64
                Module libzstd.so.1 from deb libzstd-1.5.5+dfsg2-2build1.1.amd64
                Module libgcc_s.so.1 from deb gcc-14-14.2.0-4ubuntu2~24.04.amd64
                Module libstdc++.so.6 from deb gcc-14-14.2.0-4ubuntu2~24.04.amd64
                Module libsystemd.so.0 from deb systemd-255.4-1ubuntu8.6.amd64
                Module mariadbd from deb mariadb-1:11.4.7+maria~ubu2404.amd64
                Stack trace of thread 45641:
                #0  0x0000702f1cc4553b __kill (libc.so.6 + 0x4553b)
                #1  0x00005fcc1521fa76 handle_fatal_signal (mariadbd + 0xaefa76)
                #2  0x0000702f1cc45330 __restore_rt (libc.so.6 + 0x45330)
                #3  0x00005fcc14d66615 n/a (mariadbd + 0x636615)
                #4  0x00005fcc15773036 n/a (mariadbd + 0x1043036)
                #5  0x00005fcc155f468d n/a (mariadbd + 0xec468d)
                #6  0x00005fcc14d52c5a n/a (mariadbd + 0x622c5a)
                #7  0x00005fcc154cb926 n/a (mariadbd + 0xd9b926)
                #8  0x00005fcc1522217c _Z24ha_initialize_handlertonPv (mariadbd + 0xaf217c)
                #9  0x00005fcc14f465b3 n/a (mariadbd + 0x8165b3)
                #10 0x00005fcc14f467a7 n/a (mariadbd + 0x8167a7)
                #11 0x00005fcc14f4ce3c _Z11plugin_initPiPPci (mariadbd + 0x81ce3c)
                #12 0x00005fcc14debd65 n/a (mariadbd + 0x6bbd65)
                #13 0x00005fcc14deff79 _Z11mysqld_mainiPPc (mariadbd + 0x6bff79)
                #14 0x0000702f1cc2a1ca __libc_start_call_main (libc.so.6 + 0x2a1ca)
                #15 0x0000702f1cc2a28b __libc_start_main_impl (libc.so.6 + 0x2a28b)
                #16 0x00005fcc14dc9865 _start (mariadbd + 0x699865)
                
                Stack trace of thread 45645:
                #0  0x0000702f1cc98d71 __futex_abstimed_wait_common64 (libc.so.6 + 0x98d71)
                #1  0x0000702f1cc9bc8e __pthread_cond_wait_common (libc.so.6 + 0x9bc8e)
                #2  0x00005fcc14d70685 psi_cond_timedwait (mariadbd + 0x640685)
                #3  0x00005fcc14d2a5e8 n/a (mariadbd + 0x5fa5e8)
                #4  0x00005fcc15423953 n/a (mariadbd + 0xcf3953)
                #5  0x0000702f1cc9caa4 start_thread (libc.so.6 + 0x9caa4)
                #6  0x0000702f1cd29c3c __clone3 (libc.so.6 + 0x129c3c)
                
                Stack trace of thread 45643:
                #0  0x0000702f1cc98d71 __futex_abstimed_wait_common64 (libc.so.6 + 0x98d71)
                #1  0x0000702f1cc9bc8e __pthread_cond_wait_common (libc.so.6 + 0x9bc8e)
                #2  0x00005fcc14d70685 psi_cond_timedwait (mariadbd + 0x640685)
                #3  0x00005fcc14d70758 n/a (mariadbd + 0x640758)
                #4  0x00005fcc15423953 n/a (mariadbd + 0xcf3953)
                #5  0x0000702f1cc9caa4 start_thread (libc.so.6 + 0x9caa4)
                #6  0x0000702f1cd29c3c __clone3 (libc.so.6 + 0x129c3c)
                
                Stack trace of thread 45646:
                #0  0x0000702f1db76ab0 n/a (liburing.so.2 + 0x2ab0)
                #1  0x0000702f1db770ad __io_uring_get_cqe (liburing.so.2 + 0x30ad)
                #2  0x00005fcc15615de8 n/a (mariadbd + 0xee5de8)
                #3  0x0000702f1d0ecdb4 n/a (libstdc++.so.6 + 0xecdb4)
                #4  0x0000702f1cc9caa4 start_thread (libc.so.6 + 0x9caa4)
                #5  0x0000702f1cd29c3c __clone3 (libc.so.6 + 0x129c3c)
                
                Stack trace of thread 45648:
                #0  0x0000702f1cc98d71 __futex_abstimed_wait_common64 (libc.so.6 + 0x98d71)
                #1  0x0000702f1cc9b7ed __pthread_cond_wait_common (libc.so.6 + 0x9b7ed)
                #2  0x00005fcc155c4675 n/a (mariadbd + 0xe94675)
                #3  0x0000702f1d0ecdb4 n/a (libstdc++.so.6 + 0xecdb4)
                #4  0x0000702f1cc9caa4 start_thread (libc.so.6 + 0x9caa4)
                #5  0x0000702f1cd29c3c __clone3 (libc.so.6 + 0x129c3c)
                
                Stack trace of thread 45647:
                #0  0x0000702f1cd1b4cd __GI___poll (libc.so.6 + 0x11b4cd)
                #1  0x00005fcc155bf52d n/a (mariadbd + 0xe8f52d)
                #2  0x0000702f1d0ecdb4 n/a (libstdc++.so.6 + 0xecdb4)
                #3  0x0000702f1cc9caa4 start_thread (libc.so.6 + 0x9caa4)
                #4  0x0000702f1cd29c3c __clone3 (libc.so.6 + 0x129c3c)
                ELF object binary architecture: AMD x86-64


Before this I also installed mariadb-server-dbgsym and the entry in the error log looks like:

2025-06-07 14:47:21 0 [Note] Starting MariaDB 11.4.7-MariaDB-ubu2404-log source revision 118cfcf82107188f2295631193658b2ef94f4f3f server_uid 6IX+WeD2vga5XxJDcZUY249RO+s= as process 52898
2025-06-07 14:47:21 0 [Note] InnoDB: Compressed tables use zlib 1.3
2025-06-07 14:47:21 0 [Note] InnoDB: Number of transaction pools: 1
2025-06-07 14:47:21 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2025-06-07 14:47:21 0 [Note] InnoDB: Using liburing
2025-06-07 14:47:21 0 [Note] InnoDB: innodb_buffer_pool_size_max=22528m, innodb_buffer_pool_size=22528m
2025-06-07 14:47:21 0 [Note] InnoDB: Initialized memory pressure event listener
2025-06-07 14:47:21 0 [Note] InnoDB: Completed initialization of buffer pool
2025-06-07 14:47:21 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2025-06-07 14:47:21 0 [Note] InnoDB: Resetting space id's in the doublewrite buffer
2025-06-07 14:47:21 0 [Note] InnoDB: End of log at LSN=43439159458895
2025-06-07 14:47:21 0 [Note] InnoDB: Opened 3 undo tablespaces
2025-06-07 14:47:21 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2025-06-07 14:47:21 0 [Warning] InnoDB: Cannot free the unused segments in system tablespace because a previous shutdown was not with innodb_fast_shutdown=0 or XA PREPARE transactions exist
2025-06-07 14:47:22 0 [Note] InnoDB: System tablespace defragmentation process starts
2025-06-07 14:47:22 0 [Note] InnoDB: Moving the data from extents 15552 through 2452672
250607 14:47:22 [ERROR] /usr/sbin/mariadbd got signal 11 ;
Sorry, we probably made a mistake, and this is a bug.

Your assistance in bug reporting will enable us to fix this for the next release.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report
a bug on https://jira.mariadb.org/.

Please include the information from the server start above, to the end of the
information below.

Server version: 11.4.7-MariaDB-ubu2404-log source revision: 118cfcf82107188f2295631193658b2ef94f4f3f

The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
contains instructions to obtain a better version of the backtrace below.
Following these instructions will help MariaDB developers provide a fix quicker.

Attempting backtrace. Include this in the bug report.
(note: Retrieving this information may fail)

Thread pointer: 0x0
stack_bottom = 0x0 thread_stack 0x30000
/usr/sbin/mariadbd(my_print_stacktrace+0x30)[0x63a1fcb51560]
/usr/sbin/mariadbd(handle_fatal_signal+0x2a1)[0x63a1fc6eaae1]
libc_sigaction.c:0(__restore_rt)[0x714484045330]
/usr/sbin/mariadbd(+0x636615)[0x63a1fc231615]
/usr/sbin/mariadbd(+0x1043036)[0x63a1fcc3e036]
/usr/sbin/mariadbd(+0xec468d)[0x63a1fcabf68d]
/usr/sbin/mariadbd(+0x622c5a)[0x63a1fc21dc5a]
/usr/sbin/mariadbd(+0xd9b926)[0x63a1fc996926]
/usr/sbin/mariadbd(_Z24ha_initialize_handlertonPv+0x8c)[0x63a1fc6ed17c]
/usr/sbin/mariadbd(+0x8165b3)[0x63a1fc4115b3]
/usr/sbin/mariadbd(+0x8167a7)[0x63a1fc4117a7]
/usr/sbin/mariadbd(_Z11plugin_initPiPPci+0xbfc)[0x63a1fc417e3c]
/usr/sbin/mariadbd(+0x6bbd65)[0x63a1fc2b6d65]
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0xf79)[0x63a1fc2baf79]
x86/libc-start.c:74(__libc_start_call_main)[0x71448402a1ca]
csu/libc-start.c:128(call_init)[0x71448402a28b]
/usr/sbin/mariadbd(_start+0x25)[0x63a1fc294865]
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits (excludes unlimited resources):
Limit                     Soft Limit           Hard Limit           Units     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max processes             139959               139959               processes
Max open files            1048576              1048576              files     
Max locked memory         524288               524288               bytes     
Max pending signals       139959               139959               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h

Kernel version: Linux version 6.8.0-60-generic (buildd@lcy02-amd64-054) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025

2025-06-07 14:47:29 0 [Note] Starting MariaDB 11.4.7-MariaDB-ubu2404-log source revision 118cfcf82107188f2295631193658b2ef94f4f3f server_uid 6IX+WeD2vga5XxJDcZUY249RO+s= as process 53111
2025-06-07 14:47:29 0 [Note] InnoDB: Compressed tables use zlib 1.3
2025-06-07 14:47:29 0 [Note] InnoDB: Number of transaction pools: 1
2025-06-07 14:47:29 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2025-06-07 14:47:29 0 [Note] InnoDB: Using liburing
2025-06-07 14:47:29 0 [Note] InnoDB: innodb_buffer_pool_size_max=22528m, innodb_buffer_pool_size=22528m
2025-06-07 14:47:29 0 [Note] InnoDB: Initialized memory pressure event listener
2025-06-07 14:47:29 0 [Note] InnoDB: Completed initialization of buffer pool
2025-06-07 14:47:29 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2025-06-07 14:47:29 0 [Note] InnoDB: Resetting space id's in the doublewrite buffer
2025-06-07 14:47:29 0 [Note] InnoDB: End of log at LSN=43439159458895
2025-06-07 14:47:29 0 [Note] InnoDB: Opened 3 undo tablespaces
2025-06-07 14:47:29 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2025-06-07 14:47:29 0 [Warning] InnoDB: Cannot free the unused segments in system tablespace because a previous shutdown was not with innodb_fast_shutdown=0 or XA PREPARE transactions exist
2025-06-07 14:47:30 0 [Note] InnoDB: System tablespace defragmentation process starts
2025-06-07 14:47:30 0 [Note] InnoDB: Moving the data from extents 15552 through 2452672
250607 14:47:30 [ERROR] /usr/sbin/mariadbd got signal 11 ;
Sorry, we probably made a mistake, and this is a bug.

Your assistance in bug reporting will enable us to fix this for the next release.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report
a bug on https://jira.mariadb.org/.

Please include the information from the server start above, to the end of the
information below.

Server version: 11.4.7-MariaDB-ubu2404-log source revision: 118cfcf82107188f2295631193658b2ef94f4f3f

The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
contains instructions to obtain a better version of the backtrace below.
Following these instructions will help MariaDB developers provide a fix quicker.

Attempting backtrace. Include this in the bug report.
(note: Retrieving this information may fail)

Thread pointer: 0x0
stack_bottom = 0x0 thread_stack 0x30000
/usr/sbin/mariadbd(my_print_stacktrace+0x30)[0x64ab42e73560]
/usr/sbin/mariadbd(handle_fatal_signal+0x2a1)[0x64ab42a0cae1]
libc_sigaction.c:0(__restore_rt)[0x7ebd14245330]


Open file limits are off in this.  /etc/security/limits.conf has unlimited for files for mysql and systemd config file has LimitNOFILE=2097152.  Currently trying to track down why that is.

Kind regards

Derick


On 07/06/2025 15:07, Sergei Golubchik via discuss wrote:
Hi, Derick,

On Jun 07, Derick Turner via discuss wrote:
Hi all,

Patching one of our spare servers this morning and ran into a couple of 
issues.  The first was a complaint in the service status:
...
Thread pointer: 0x0
stack_bottom = 0x0 thread_stack 0x30000
/usr/sbin/mariadbd(my_print_stacktrace+0x30)[0x5a41189db560]
/usr/sbin/mariadbd(handle_fatal_signal+0x2a1)[0x5a4118574ae1]
/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7650bf845330]
/usr/sbin/mariadbd(+0x636615)[0x5a41180bb615]
/usr/sbin/mariadbd(+0x1043036)[0x5a4118ac8036]
/usr/sbin/mariadbd(+0xec468d)[0x5a411894968d]
/usr/sbin/mariadbd(+0x622c5a)[0x5a41180a7c5a]
/usr/sbin/mariadbd(+0xd9b926)[0x5a4118820926]
/usr/sbin/mariadbd(_Z24ha_initialize_handlertonPv+0x8c)[0x5a411857717c]
/usr/sbin/mariadbd(+0x8165b3)[0x5a411829b5b3]
/usr/sbin/mariadbd(+0x8167a7)[0x5a411829b7a7]
/usr/sbin/mariadbd(_Z11plugin_initPiPPci+0xbfc)[0x5a41182a1e3c]
/usr/sbin/mariadbd(+0x6bbd65)[0x5a4118140d65]
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0xf79)[0x5a4118144f79]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7650bf82a1ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7650bf82a28b]
/usr/sbin/mariadbd(_start+0x25)[0x5a411811e865]
Writing a core file...
Working directory at /var/lib/mysql

Is there anything I'm missing before I raise this as a bug?
yes, please install dbgsym packages to get a better stack trace.
or load the core dump (that was written above) in gdb and run
`thread apply all bt`. Someone will ask you do it after you raise a bug
anyway, so you can as well be prepared.

Regards,
Sergei
Chief Architect, MariaDB Server
and security@mariadb.org
_______________________________________________
discuss mailing list -- discuss@lists.mariadb.org
To unsubscribe send an email to discuss-leave@lists.mariadb.org
-- 
Derick Turner - He/Him

Managing Director
e-Learn Design Ltd | Scotland's Moodle Partner
T +44 (0)845 474 4512 | M +44 (0)7808 068 797 | www.e-learndesign.co.uk