Hi,
We're use mariadb 10.1.3 as a main database of our system.
We migrated data from develpment server to product server a week ago.
Below is our error case :
MariaDB shutdown with signal 6 when some query executes. (insert or
update...)
The error log is below.
------------------------------------------------------------------------
----------------------------------
...
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0
in file /data001/masvc01/ibdata1 encrypted.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key
management plugin or used key_id 286 is not found or used encryption
algorithm or method does not match.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace
as missing. You may drop this table or install correct key management
plugin and key file.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0
in file /data001/masvc01/ibdata1 encrypted.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key
management plugin or used key_id 286 is not found or used encryption
algorithm or method does not match.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace
as missing. You may drop this table or install correct key management
plugin and key file.
2016-07-18 09:44:14 7f38f3c36b00 InnoDB: Assertion failure in thread
139882584566528 in file buf0buf.cc line 4803
...
...
------------------------------------------------------------------------
-----------------------------------------------
We check out system innodb table space with innochecksum.
The result is below.
------------------------------------------------------------------------
-----
/engn001/masvc01/mysql/bin/innochecksum -v ibdata1
InnoDB offline file checksum utility.
Variables (--variable-name=value)
and boolean options {FALSE|TRUE} Value (after reading options)
---------------------------------
----------------------------------------
verbose TRUE
debug FALSE
skip-corrupt FALSE
count FALSE
start-page 0
end-page 0
page 0
per-page-details FALSE
leaf FALSE
merge 0
Table is uncompressed
Page size is 16384
file ibdata1_20160718 = 3221225472 bytes (196608 pages)...
InnoChecksum; checking pages in range 0 to 196607
Fail; page 0 invalid (fails innodb and crc32 checksum)
------------------------------------------------------------------------
----
We hava ibdata1, ibdata2, ibdata3 and ibdata4 and results of
innochecksum are same.
We compiled mariadb with debug option and test under same enviroment.
Below is the mysql.trace when db crash.
-------------------------------------------------
========================================================================
==================================================
T@10 : | | | | | | | | | | | | | | | | | >print_buffer_to_file
T@10 : | | | | | | | | | | | | | | | | | | enter: buffer: InnoDB:
Marking tablespace as missing. You may drop this table or install
correct key management plugin and key file.
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log
(0x15b4b60) locking
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log
(0x15b4b60) locked
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log
(0x15b4b60) unlocking
T@10 : | | | | | | | | | | | | | | | | |
mysql_audit_acquire_plugins T@10 : | | | | | | | | | | | | | | | | | |
lock (0x5dcaeb00) locking T@10 : | | | | | | | | | | | | | | | | | | >my_multi_malloc T@10 : | | | | | | | | | | | | | | | | | | | >my_malloc T@10 : | | | | | | | | | | | | | | | | | | | | my: size: 208 my_flags: 24 T@10 : | | | | | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e8415a710 T@10 : | | | | | | | | | | | | | | | | | | | my_hash_init T@10 : | | | | | | | | | | | | | | | | | | | enter: hash: 0x7f9e8415a710 size: 128 T@10 : | | | | | | | | | | | | | | | | | | | >init_dynamic_array2 T@10 : | | | | | | | | | | | | | | | | | | | | >my_malloc
--------------------------------------------------- So we export data from mariadb and import to new table space. After migration, we shutdown mariadb and restart. but signal 11 occurred and db crash. Below is our error log. ------------------------------------------------------------------------ -------------------------- 2016-07-19 8:55:09 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Normal shutdown 2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Killing the scheduler thread, thread id 1 2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Waiting for the scheduler thread to reply 2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Stopped 2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Purging the queue. 0 events 160719 8:55:09 server_audit: STOPPED 2016-07-19 8:55:09 140556627830528 [Note] InnoDB: FTS optimize thread exiting. 2016-07-19 8:55:09 140556344748800 [Note] InnoDB: Starting shutdown... 2016-07-19 8:55:09 140556648810240 [Warning] InnoDB: Dumping buffer pool(s) to /data001/masvc01/ib_buffer_pool 2016-07-19 8:55:10 140556648810240 [Warning] InnoDB: Buffer pool(s) dump completed at 160719 8:55:10 2016-07-19 8:55:15 140556344748800 [Note] InnoDB: Shutdown completed; log sequence number 399145649446 2016-07-19 8:55:15 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Shutdown complete 160719 08:55:16 mysqld_safe mysqld from pid file /engn001/masvc01/mysql/mysqld.pid ended 160719 08:57:05 mysqld_safe Starting mysqld daemon with databases from /data001/masvc01 2016-07-19 8:57:05 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld (mysqld 10.1.14-MariaDB) starting as process 2608 ... 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: The InnoDB memory heap is disabled 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Memory barrier is not used 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Compressed tables use zlib 1.2.3 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using Linux native AIO 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using SSE crc32 instructions 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Initializing buffer pool, size = 24.0G 2016-07-19 8:57:08 140304014350176 [Note] InnoDB: Completed initialization of buffer pool 2016-07-19 8:57:10 140304014350176 [Note] InnoDB: Highest supported file format is Barracuda. 2016-07-19 8:57:13 140304014350176 [Note] InnoDB: 128 rollback segment(s) are active. 2016-07-19 8:57:13 140304014350176 [Note] InnoDB: Waiting for purge to start 2016-07-19 8:57:13 140304014350176 [Note] InnoDB: Percona XtraDB (http://www.percona.com http://www.percona.com/ ) 5.6.29-76.2 started; log sequence number 399145649446 2016-07-19 8:57:13 140273638147840 [Note] InnoDB: Dumping buffer pool(s) not yet started 2016-07-19 08:57:13 7f94005f8700 InnoDB: Loading buffer pool(s) from /data001/masvc01/ib_buffer_pool 2016-07-19 8:57:13 140304014350176 [Note] Plugin 'FEEDBACK' is disabled. 160719 8:57:13 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED. 2016-07-19 8:57:13 140304014350176 [Note] Server socket created on IP: '::'. 2016-07-19 8:57:13 140304014350176 [Warning] 'user' entry 'root@lgedglap02v' ignored in --skip-name-resolve mode. 2016-07-19 8:57:13 140304014350176 [Warning] 'user' entry '@lgedglap02v' ignored in --skip-name-resolve mode. 2016-07-19 8:57:13 140304014350176 [Warning] 'proxies_priv' entry '@% root@lgedglap02v' ignored in --skip-name-resolve mode. 2016-07-19 8:57:13 140273595853568 [Note] Event Scheduler: scheduler thread started with id 1 2016-07-19 8:57:13 140304014350176 [Note] Reading of all Master_info entries succeded 2016-07-19 8:57:13 140304014350176 [Note] Added new Master_info '' to hash table 2016-07-19 8:57:13 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld: ready for connections. Version: '10.1.14-MariaDB' socket: '/engn001/masvc01/mysql/mysqld.sock' port: 3310 MariaDB Server 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted. 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match. 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file. 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted. 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match. 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file. 160719 8:57:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.1.14-MariaDB key_buffer_size=67108864 read_buffer_size=50331648 max_used_connections=3 max_threads=102 thread_count=4 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10094612 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x48400 /engn001/masvc01/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0xc028ae] /engn001/masvc01/mysql/bin/mysqld(handle_fatal_signal+0x464)[0x7651f4] /lib64/libpthread.so.0[0x3ffe40f7e0] /engn001/masvc01/mysql/bin/mysqld[0x8fa6d3] /engn001/masvc01/mysql/bin/mysqld[0xa3ba47] /engn001/masvc01/mysql/bin/mysqld[0xa51a2e] /engn001/masvc01/mysql/bin/mysqld[0xa52bc9] /engn001/masvc01/mysql/bin/mysqld[0xa433ff] /engn001/masvc01/mysql/bin/mysqld[0xa440b7] /lib64/libpthread.so.0[0x3ffe407aa1] /lib64/libc.so.6(clone+0x6d)[0x3ffe0e893d] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. ..... ------------------------------------------------------------------------ -------------------------------------------------- I can't find solution for it. Is it a bug or hardware fault ? Regards. 김경남 차장 CTO 아키텍처담당 Post Architecture팀 02-2099-6177 / 010-2208-3006 _____ 상기 메시지와 첨부화일 내에는 비밀정보가 포함되어 있을 수 있으며, 지정된 수신자에 한하여 조회 및 사용될 수 있습니다. 만약 송신자의 실수로 인하여 상기 메시지를 수신하였다면, 송신자에게 메시지를 반송해 주시고, 원본 메시지와 모든 사본을 폐기해 주시기 바랍니다. 상기 메시지의 전체 또는 일부에 대해 무단 열람, 사용, 공개, 배포하는 것은 금지되어 있습니다.(주)LG CNS. This message and its attachments may contain confidential information, and they are intended to be viewed or used by only the individuals specified in the message. If you have received this message in an error from the sender, please contact the sender immediately to notify the error and delete all of the message and its copies. It is prohibited to view, use, make public and/or distribute part or whole of this message without written permission.