[Maria-discuss] One node out of cluster keeps crashing

Hello everyone, MariaDB-Galera-server-10.0.17-1.el7.centos.x86_64 MariaDB-client-10.0.17-1.el7.centos.x86_64 galera-25.3.9-1.rhel7.el7.centos.x86_64 We have a three node MariaDB-galera cluster, and one of our nodes keeps crashing. The other two nodes have been running fine for weeks without issues. They all have the exact same specifications. I have even rebuilt this node from ground up, and the new node still crashed. Could someone take a look at the following logs and help me figure out what is wrong, and how can we avoid this in future? ===================================== 2015-03-30 10:13:05 7f5be57fc700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 19 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 116110 srv_active, 0 srv_shutdown, 343879 srv_idle srv_master_thread log flush and writes: 459986 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 56843 OS WAIT ARRAY INFO: signal count 804274 Mutex spin waits 1039838, rounds 335818, OS waits 4493 RW-shared spins 393078, rounds 1863774, OS waits 46608 RW-excl spins 34511, rounds 1424782, OS waits 5051 Spin rounds per wait: 0.32 mutex, 4.74 RW-shared, 41.28 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 3156054 Purge done for trx's n:o < 3155992 undo n:o < 0 state: running but idle History list length 814 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 3155294, not started MySQL thread id 246025, OS thread handle 0x7f5d84124700, query id 32716796 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up ---TRANSACTION 3154332, not started MySQL thread id 245589, OS thread handle 0x7f5d4c963700, query id 32642939 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up ---TRANSACTION 3150705, not started MySQL thread id 245568, OS thread handle 0x7f5d840db700, query id 32777452 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up ---TRANSACTION 3152515, not started MySQL thread id 245295, OS thread handle 0x7f5d851ff700, query id 32560015 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up ---TRANSACTION 3155566, not started MySQL thread id 245281, OS thread handle 0x7f5d86d23700, query id 32774720 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up ---TRANSACTION 3156038, not started TOO MANY LOCKS PRINTED FOR THIS TRX: SUPPRESSING FURTHER PRINTS -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 294306 OS file reads, 2485633 OS file writes, 532545 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 63, seg size 65, 28112 merges merged operations: insert 42583, delete mark 939, delete 29 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 121964244699 Log flushed up to 121964244699 Pages flushed up to 121964244699 Last checkpoint at 121964244699 Max checkpoint age 216721613 Checkpoint age target 209949063 Modified age 0 Checkpoint age 0 0 pending log writes, 0 pending chkp writes 153001 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 6263013376; in additional pool allocated 0 Total memory allocated by read views 3288 Internal hash tables (constant factor + variable factor) Adaptive hash index 423323504 (96884488 + 326439016) Page hash 757784 (buffer pool 0 only) Dictionary cache 25112304 (24222544 + 889760) File system 854680 (812272 + 42408) Lock system 15172408 (15139192 + 33216) Recovery system 0 (0 + 0) Dictionary memory allocated 889760 Buffer pool size 373496 Buffer pool size, bytes 6119358464 Free buffers 34210 Database pages 319362 Old database pages 118034 Modified db pages 0 Percent of dirty pages(LRU & free pages): 0.000 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 1891, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 294208, created 25154, written 2272207 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 319362, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 46687 Buffer pool size, bytes 764919808 Free buffers 4184 Database pages 40015 Old database pages 14791 Modified db pages 0 Percent of dirty pages(LRU & free pages): 0.000 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 237, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 36928, created 3087, written 294016 LRU len: 40015, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 46687 Buffer pool size, bytes 764919808 Free buffers 4881 Database pages 39304 Old database pages 14528 Modified db pages 0 Percent of dirty pages(LRU & free pages): 0.000 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 232, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 36418, created 2886, written 189961 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 39304, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 46687 Buffer pool size, bytes 764919808 Free buffers 4278 Database pages 39906 Old database pages 14748 Modified db pages 0 Percent of dirty pages(LRU & free pages): 0.000 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 215, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 36782, created 3124, written 154398 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 39906, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 46687 Buffer pool size, bytes 764919808 Free buffers 4286 Database pages 39917 Old database pages 14754 Modified db pages 0 Percent of dirty pages(LRU & free pages): 0.000 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 247, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 36327, created 3590, written 392683 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 39917, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 46687 Buffer pool size, bytes 764919808 Free buffers 4215 Database pages 39956 Old database pages 14769 Modified db pages 0 Percent of dirty pages(LRU & free pages): 0.000 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 276, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 36833, created 3123, written 258687 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 39956, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 46687 Buffer pool size, bytes 764919808 Free buffers 4082 Database pages 40127 Old database pages 14831 Modified db pages 0 Percent of dirty pages(LRU & free pages): 0.000 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 243, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 36928, created 3199, written 252207 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 40127, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 46687 Buffer pool size, bytes 764919808 Free buffers 4010 Database pages 40224 Old database pages 14860 Modified db pages 0 Percent of dirty pages(LRU & free pages): 0.000 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 223, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 37120, created 3104, written 527443 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 40224, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 46687 Buffer pool size, bytes 764919808 Free buffers 4274 Database pages 39913 Old database pages 14753 Modified db pages 0 Percent of dirty pages(LRU & free pages): 0.000 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 218, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 36872, created 3041, written 202812 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 39913, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 2 read views open inside InnoDB 8 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 8 out of 1000 descriptors used ---OLDEST VIEW--- Normal read view Read view low limit trx n:o 3156042 Read view up limit trx id 3155989 Read view low limit trx id 3156042 Read view individually stored trx ids: Read view trx id 3155989 Read view trx id 3156029 Read view trx id 3156038 ----------------- Main thread process no. 2488, id 140032660715264, state: sleeping Number of rows inserted 1182044, updated 1504189, deleted 4658, read 2328285511 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 0, updated 0, deleted 0, read 0 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long WSREP: BF lock wait long