Hi, All

     Recently, We use Sysbench  Test  MariaDB 10.1.13.  In Test ,We found server lock and can't write . 

Testing environment:

server1:  10.2.1.242    (HP OEM Fusion IO PCIE Card  3.2TB )

server2:  10.2.1.239    (ScanDisk Fusion IO PCIE Card 3.2TB)

sysbench test script:

 sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua --num-threads=16  --mysql-host=localhost --mysql-port=3306 --mysql-socket=/data/mysql2/data/mysql.sock --mysql-user=root --mysql-password=123456 --mysql-db=test --max-time=0 --max-requests=0  --report-interval=1 --oltp-tables-count=16 run

when wsrep_on = on ,  after run a few minutes or a few hours, both server1 and server2  were stucked, all write stoped ,only select statement run normal  :

................

[ 815s] threads: 16, tps: 6176.00, reads: 86431.00, writes: 24699.00, response time: 3.37ms (95%), errors: 0.00, reconnects:  0.00

[ 816s] threads: 16, tps: 6064.99, reads: 84907.86, writes: 24256.96, response time: 3.66ms (95%), errors: 0.00, reconnects:  0.00

[ 817s] threads: 16, tps: 6097.00, reads: 85394.05, writes: 24394.02, response time: 3.54ms (95%), errors: 0.00, reconnects:  0.00

[ 818s] threads: 16, tps: 6131.97, reads: 85819.52, writes: 24529.86, response time: 3.65ms (95%), errors: 0.00, reconnects:  0.00

[ 819s] threads: 16, tps: 6143.04, reads: 86036.63, writes: 24575.18, response time: 3.44ms (95%), errors: 0.00, reconnects:  0.00

[ 820s] threads: 16, tps: 5819.80, reads: 81532.22, writes: 23332.20, response time: 3.60ms (95%), errors: 0.00, reconnects:  0.00

[ 821s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00

[ 822s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00

[ 823s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00

[ 824s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00

[ 825s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00

[ 826s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time: 0.00ms (95%), errors: 0.00, reconnects:  0.00

................

In error log file not error info , after execute pstack `pidof mysqld` > pstack.log,  sysbench run normal again , and start write data.

Every time have this problem,  in pstack.log file contains a few lock: 

#0  0x00007f2b04fe0334 in __lll_lock_wait () from /lib64/libpthread.so.0

#1  0x00007f2b04fdb5d8 in _L_lock_854 () from /lib64/libpthread.so.0

#2  0x00007f2b04fdb4a7 in pthread_mutex_lock () from /lib64/libpthread.so.0

....................

When set wsrep_on = off,  Sysbench test MariaDB 10.1.13 normal.

As contrast ,We download Percona-XtraDB-Cluster-5.6.28-rel76.1-25.14.1.Linux.x86_64.ssl101.tar.gz   and run sysbench test, found same problem too.

We can't confirm , is it hardware or galera question ?   please help analyze ,thank you.

Reference link:

https://en.wikibooks.org/wiki/Linux_Applications_Debugging_Techniques/Deadlocks

http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them

--

杨建军

_______________________________________________
Mailing list: https://launchpad.net/~maria-discuss
Post to     : maria-discuss@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp