Hi, All
Recently, We use Sysbench Test MariaDB 10.1.13. In Test ,We found server lock and can't write .
Testing environment:
server1: 10.2.1.242 (HP OEM Fusion IO PCIE Card 3.2TB )
server2: 10.2.1.239 (ScanDisk Fusion IO PCIE Card 3.2TB)
sysbench test script:
sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua --num-threads=16 --mysql-host=localhost --mysql-port=3306 --mysql-socket=/data/mysql2/data/mysql.sock --mysql-user=root --mysql-password=123456 --mysql-db=test --max-time=0 --max-requests=0 --report-interval=1 --oltp-tables-count=16 run
when wsrep_on = on , after run a few minutes or a few hours, both server1 and server2 were stucked, all write stoped ,only select statement run normal :
................
[ 815s] threads: 16, tps: 6176.00, reads: 86431.00, writes: 24699.00, response time: 3.37ms (95%), errors: 0.00, reconnects: 0.00
[ 816s] threads: 16, tps: 6064.99, reads: 84907.86, writes: 24256.96, response time: 3.66ms (95%), errors: 0.00, reconnects: 0.00
[ 817s] threads: 16, tps: 6097.00, reads: 85394.05, writes: 24394.02, response time: 3.54ms (95%), errors: 0.00, reconnects: 0.00
[ 818s] threads: 16, tps: 6131.97, reads: 85819.52, writes: 24529.86, response time: 3.65ms (95%), errors: 0.00, reconnects: 0.00
[ 819s] threads: 16, tps: 6143.04, reads: 86036.63, writes: 24575.18, response time: 3.44ms (95%), errors: 0.00, reconnects: 0.00
[ 820s] threads: 16, tps: 5819.80, reads: 81532.22, writes: 23332.20, response time: 3.60ms (95%), errors: 0.00, reconnects: 0.00
[ 821s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects: 0.00
[ 822s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects: 0.00
[ 823s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects: 0.00
[ 824s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects: 0.00
[ 825s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects: 0.00
[ 826s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects: 0.00
................
In error log file not error info , after execute pstack `pidof mysqld` > pstack.log, sysbench run normal again , and start write data.
Every time have this problem, in pstack.log file contains a few lock:
#0 0x00007f2b04fe0334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f2b04fdb5d8 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f2b04fdb4a7 in pthread_mutex_lock () from /lib64/libpthread.so.0
....................
When set wsrep_on = off, Sysbench test MariaDB 10.1.13 normal.
As contrast ,We download Percona-XtraDB-Cluster-5.6.28-rel76.1-25.14.1.Linux.x86_64.ssl101.tar.gz and run sysbench test, found same problem too.
We can't confirm , is it hardware or galera question ? please help analyze ,thank you.
Reference link: