Hello everyone,
MariaDB-Galera-server-10.0.17-1.el7.centos.x86_64
MariaDB-client-10.0.17-1.el7.centos.x86_64
galera-25.3.9-1.rhel7.el7.centos.x86_64
We have a three node MariaDB-galera cluster, and one of our nodes keeps
crashing. The other two nodes have been running fine for weeks without
issues. They all have the exact same specifications. I have even
rebuilt this node from ground up, and the new node still crashed.
Could someone take a look at the following logs and help me figure out
what is wrong, and how can we avoid this in future?
=====================================
2015-03-30 10:13:05 7f5be57fc700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 19 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 116110 srv_active, 0 srv_shutdown, 343879
srv_idle
srv_master_thread log flush and writes: 459986
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 56843
OS WAIT ARRAY INFO: signal count 804274
Mutex spin waits 1039838, rounds 335818, OS waits 4493
RW-shared spins 393078, rounds 1863774, OS waits 46608
RW-excl spins 34511, rounds 1424782, OS waits 5051
Spin rounds per wait: 0.32 mutex, 4.74 RW-shared, 41.28 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 3156054
Purge done for trx's n:o < 3155992 undo n:o < 0 state: running but idle
History list length 814
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 3155294, not started
MySQL thread id 246025, OS thread handle 0x7f5d84124700, query id
32716796 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up
---TRANSACTION 3154332, not started
MySQL thread id 245589, OS thread handle 0x7f5d4c963700, query id
32642939 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up
---TRANSACTION 3150705, not started
MySQL thread id 245568, OS thread handle 0x7f5d840db700, query id
32777452 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up
---TRANSACTION 3152515, not started
MySQL thread id 245295, OS thread handle 0x7f5d851ff700, query id
32560015 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up
---TRANSACTION 3155566, not started
MySQL thread id 245281, OS thread handle 0x7f5d86d23700, query id
32774720 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up
---TRANSACTION 3156038, not started
TOO MANY LOCKS PRINTED FOR THIS TRX: SUPPRESSING FURTHER PRINTS
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer
thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
294306 OS file reads, 2485633 OS file writes, 532545 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 63, seg size 65, 28112 merges
merged operations:
insert 42583, delete mark 939, delete 29
discarded operations:
insert 0, delete mark 0, delete 0
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 121964244699
Log flushed up to 121964244699
Pages flushed up to 121964244699
Last checkpoint at 121964244699
Max checkpoint age 216721613
Checkpoint age target 209949063
Modified age 0
Checkpoint age 0
0 pending log writes, 0 pending chkp writes
153001 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 6263013376; in additional pool allocated 0
Total memory allocated by read views 3288
Internal hash tables (constant factor + variable factor)
Adaptive hash index 423323504 (96884488 + 326439016)
Page hash 757784 (buffer pool 0 only)
Dictionary cache 25112304 (24222544 + 889760)
File system 854680 (812272 + 42408)
Lock system 15172408 (15139192 + 33216)
Recovery system 0 (0 + 0)
Dictionary memory allocated 889760
Buffer pool size 373496
Buffer pool size, bytes 6119358464
Free buffers 34210
Database pages 319362
Old database pages 118034
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1891, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 294208, created 25154, written 2272207
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 319362, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 46687
Buffer pool size, bytes 764919808
Free buffers 4184
Database pages 40015
Old database pages 14791
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 237, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36928, created 3087, written 294016
LRU len: 40015, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 46687
Buffer pool size, bytes 764919808
Free buffers 4881
Database pages 39304
Old database pages 14528
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 232, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36418, created 2886, written 189961
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 39304, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 46687
Buffer pool size, bytes 764919808
Free buffers 4278
Database pages 39906
Old database pages 14748
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 215, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36782, created 3124, written 154398
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 39906, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 46687
Buffer pool size, bytes 764919808
Free buffers 4286
Database pages 39917
Old database pages 14754
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 247, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36327, created 3590, written 392683
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 39917, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 46687
Buffer pool size, bytes 764919808
Free buffers 4215
Database pages 39956
Old database pages 14769
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 276, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36833, created 3123, written 258687
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 39956, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 46687
Buffer pool size, bytes 764919808
Free buffers 4082
Database pages 40127
Old database pages 14831
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 243, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36928, created 3199, written 252207
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 40127, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 46687
Buffer pool size, bytes 764919808
Free buffers 4010
Database pages 40224
Old database pages 14860
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 223, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 37120, created 3104, written 527443
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 40224, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 46687
Buffer pool size, bytes 764919808
Free buffers 4274
Database pages 39913
Old database pages 14753
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 218, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36872, created 3041, written 202812
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 39913, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
2 read views open inside InnoDB
8 RW transactions active inside InnoDB
0 RO transactions active inside InnoDB
8 out of 1000 descriptors used
---OLDEST VIEW---
Normal read view
Read view low limit trx n:o 3156042
Read view up limit trx id 3155989
Read view low limit trx id 3156042
Read view individually stored trx ids:
Read view trx id 3155989
Read view trx id 3156029
Read view trx id 3156038
-----------------
Main thread process no. 2488, id 140032660715264, state: sleeping
Number of rows inserted 1182044, updated 1504189, deleted 4658, read
2328285511
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
