Hi,
We're use mariadb 10.1.3 as a main database of our system.
We migrated data from develpment server to product server a week ago.
Below is our error case :
MariaDB shutdown with signal 6 when some query executes. (insert or update...)
The error log is below.
----------------------------------------------------------------------------------------------------------
...
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match.
2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
2016-07-18 09:44:14 7f38f3c36b00 InnoDB: Assertion failure in thread 139882584566528 in file buf0buf.cc line 4803
...
...
-----------------------------------------------------------------------------------------------------------------------
We check out system innodb table space with innochecksum.
The result is below.
-----------------------------------------------------------------------------
/engn001/masvc01/mysql/bin/innochecksum -v ibdata1
InnoDB offline file checksum utility.
Variables (--variable-name=value)
and boolean options {FALSE|TRUE} Value (after reading options)
--------------------------------- ----------------------------------------
verbose TRUE
debug FALSE
skip-corrupt FALSE
count FALSE
start-page 0
end-page 0
page 0
per-page-details FALSE
leaf FALSE
merge 0
Table is uncompressed
Page size is 16384
file ibdata1_20160718 = 3221225472 bytes (196608 pages)...
InnoChecksum; checking pages in range 0 to 196607
Fail; page 0 invalid (fails innodb and crc32 checksum)
----------------------------------------------------------------------------
We hava ibdata1, ibdata2, ibdata3 and ibdata4 and results of innochecksum are same.
We compiled mariadb with debug option and test under same enviroment.
Below is the mysql.trace when db crash.
-------------------------------------------------
==========================================================================================================================
T@10 : | | | | | | | | | | | | | | | | | >print_buffer_to_file
T@10 : | | | | | | | | | | | | | | | | | | enter: buffer: InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) locking
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) locked
T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) unlocking
T@10 : | | | | | | | | | | | | | | | | | <print_buffer_to_file
T@10 : | | | | | | | | | | | | | | | | <vprint_msg_to_log
T@10 : | | | | | | | | | | | | | | | <sql_print_error
T@10 : | | | | | | | | | | | | | | | >my_malloc
T@10 : | | | | | | | | | | | | | | | | my: size: 4096 my_flags: 16
T@10 : | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e84159690
T@10 : | | | | | | | | | | | | | | | <my_malloc
T@10 : | | | | | | | | | | | | | | | >push_warning_printf
T@10 : | | | | | | | | | | | | | | | | enter: warning: 192
T@10 : | | | | | | | | | | | | | | | | >push_warning
T@10 : | | | | | | | | | | | | | | | | | enter: code: 192, msg: Table in tablespace 0 encrypted.However key management plugin or used key_id 290 is not found or used encryption algorithm or method does not match. Can't continue opening the table.
T@10 : | | | | | | | | | | | | | | | | | >THD::raise_condition
T@10 : | | | | | | | | | | | | | | | | | | >mysql_audit_acquire_plugins
T@10 : | | | | | | | | | | | | | | | | | | <mysql_audit_acquire_plugins
T@10 : | | | | | | | | | | | | | | | | | | mutex: l_perm->lock (0x5dcaeb00) locking
T@10 : | | | | | | | | | | | | | | | | | | >my_multi_malloc
T@10 : | | | | | | | | | | | | | | | | | | | >my_malloc
T@10 : | | | | | | | | | | | | | | | | | | | | my: size: 208 my_flags: 24
T@10 : | | | | | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e8415a710
T@10 : | | | | | | | | | | | | | | | | | | | <my_malloc
T@10 : | | | | | | | | | | | | | | | | | | <my_multi_malloc
T@10 : | | | | | | | | | | | | | | | | | | >my_hash_init
T@10 : | | | | | | | | | | | | | | | | | | | enter: hash: 0x7f9e8415a710 size: 128
T@10 : | | | | | | | | | | | | | | | | | | | >init_dynamic_array2
T@10 : | | | | | | | | | | | | | | | | | | | | >my_malloc
---------------------------------------------------------------------------------------------------------------------------
So we export data from mariadb and import to new table space.
After migration, we shutdown mariadb and restart. but signal 11 occurred and db crash.
Below is our error log.
--------------------------------------------------------------------------------------------------
2016-07-19 8:55:09 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Normal shutdown
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Killing the scheduler thread, thread id 1
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Waiting for the scheduler thread to reply
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Stopped
2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Purging the queue. 0 events
160719 8:55:09 server_audit: STOPPED
2016-07-19 8:55:09 140556627830528 [Note] InnoDB: FTS optimize thread exiting.
2016-07-19 8:55:09 140556344748800 [Note] InnoDB: Starting shutdown...
2016-07-19 8:55:09 140556648810240 [Warning] InnoDB: Dumping buffer pool(s) to /data001/masvc01/ib_buffer_pool
2016-07-19 8:55:10 140556648810240 [Warning] InnoDB: Buffer pool(s) dump completed at 160719 8:55:10
2016-07-19 8:55:15 140556344748800 [Note] InnoDB: Shutdown completed; log sequence number 399145649446
2016-07-19 8:55:15 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Shutdown complete
160719 08:55:16 mysqld_safe mysqld from pid file /engn001/masvc01/mysql/mysqld.pid ended
160719 08:57:05 mysqld_safe Starting mysqld daemon with databases from /data001/masvc01
2016-07-19 8:57:05 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld (mysqld 10.1.14-MariaDB) starting as process 2608 ...
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: The InnoDB memory heap is disabled
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Memory barrier is not used
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Compressed tables use zlib 1.2.3
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using Linux native AIO
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using SSE crc32 instructions
2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Initializing buffer pool, size = 24.0G
2016-07-19 8:57:08 140304014350176 [Note] InnoDB: Completed initialization of buffer pool
2016-07-19 8:57:10 140304014350176 [Note] InnoDB: Highest supported file format is Barracuda.
2016-07-19 8:57:13 140304014350176 [Note] InnoDB: 128 rollback segment(s) are active.
2016-07-19 8:57:13 140304014350176 [Note] InnoDB: Waiting for purge to start
2016-07-19 8:57:13 140304014350176 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.29-76.2 started; log sequence number 399145649446
2016-07-19 8:57:13 140273638147840 [Note] InnoDB: Dumping buffer pool(s) not yet started
2016-07-19 08:57:13 7f94005f8700 InnoDB: Loading buffer pool(s) from /data001/masvc01/ib_buffer_pool
2016-07-19 8:57:13 140304014350176 [Note] Plugin 'FEEDBACK' is disabled.
160719 8:57:13 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED.
2016-07-19 8:57:13 140304014350176 [Note] Server socket created on IP: '::'.
2016-07-19 8:57:13 140304014350176 [Warning] 'user' entry 'root@lgedglap02v' ignored in --skip-name-resolve mode.
2016-07-19 8:57:13 140304014350176 [Warning] 'user' entry '@lgedglap02v' ignored in --skip-name-resolve mode.
2016-07-19 8:57:13 140304014350176 [Warning] 'proxies_priv' entry '@% root@lgedglap02v' ignored in --skip-name-resolve mode.
2016-07-19 8:57:13 140273595853568 [Note] Event Scheduler: scheduler thread started with id 1
2016-07-19 8:57:13 140304014350176 [Note] Reading of all Master_info entries succeded
2016-07-19 8:57:13 140304014350176 [Note] Added new Master_info '' to hash table
2016-07-19 8:57:13 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld: ready for connections.
Version: '10.1.14-MariaDB' socket: '/engn001/masvc01/mysql/mysqld.sock' port: 3310 MariaDB Server
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match.
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted.
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match.
2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file.
160719 8:57:47 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.1.14-MariaDB
key_buffer_size=67108864
read_buffer_size=50331648
max_used_connections=3
max_threads=102
thread_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10094612 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
/engn001/masvc01/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0xc028ae]
/engn001/masvc01/mysql/bin/mysqld(handle_fatal_signal+0x464)[0x7651f4]
/lib64/libpthread.so.0[0x3ffe40f7e0]
/engn001/masvc01/mysql/bin/mysqld[0x8fa6d3]
/engn001/masvc01/mysql/bin/mysqld[0xa3ba47]
/engn001/masvc01/mysql/bin/mysqld[0xa51a2e]
/engn001/masvc01/mysql/bin/mysqld[0xa52bc9]
/engn001/masvc01/mysql/bin/mysqld[0xa433ff]
/engn001/masvc01/mysql/bin/mysqld[0xa440b7]
/lib64/libpthread.so.0[0x3ffe407aa1]
/lib64/libc.so.6(clone+0x6d)[0x3ffe0e893d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
.....
--------------------------------------------------------------------------------------------------------------------------
I can't find solution for it.
Is it a bug or hardware fault ?
Regards.
|