Hi, I repeated the exact problem. Firstly, you need to have this branch:

jan@jan-GE70-0NC-0ND ~/mysql/mariadb-patches $ bzr info
Standalone tree (format: 2a)
Location:
  branch root: .

Related branches:
  parent branch: bzr+ssh://bazaar.launchpad.net/~elenst/randgen/mariadb-patches/

Inside that directory I run following command :

perl ./runall-new.pl --grammar=conf/engines/engine_stress.yy --gendata=conf/engines/engine_stress.zz --duration=600 --queries=100M --threads=4 --galera=mmm --basedir=/home/jan/mysql/galera-10.0 --vardir=/home/jan/mysql/galera-test
--mysqld=--wsrep-provider=/usr/lib/libgalera_smm.so --debug --mysqld=--general-log=1
--mysqld=--general_log_file=/home/jan/mysql/galera-test/stmt.log --logfile=/home/jan/mysql/galera-test/rpg.log --valgrind

Correct log file attached.
Yes, I did run using valgrind, while it reported some uninitialized variables,
not really any memory corruptions.  Attached, the log from node0. Similarly,
Well, uninitialised variables are exactly what I would expect could lead to
things looking like you have shown so far. So if you have Valgrind errors, you
should definitely start by fixing them.

However, I see no messages from Valgrind in the log you attached?!?

So please, repeat the exact problem you get (assertion during rollback) while
running with Valgrind. Provide the exact command you used to get the error in
this particular run, and the correct log that shows the Valgrind errors about
uninitialised variables, and I can give more suggestions for how to fix.

debug=+d did not help. Where I should add some output to narrow down the place
where the corruption happens.

It did produce the output, not yet found the reason, some of the output just before corruption

MYSQL_BIN_LOG::flush_and_set_pending_rows_event(event): enter: event: 0x7f2a200bc800
MYSQL_BIN_LOG::flush_and_set_pending_rows_event(event): info: cache_mngr->pending(): 0x0
Rows_log_event::do_add_row_data: enter: row_data: 0x7f2a200cb248  length: 10
Rows_log_event::do_add_row_data: row_data: Memory: 0x7f2a200cb248  Bytes: (10)
F8 01 00 00 00 01 74 02 77 65
my_realloc: my: ptr: 0x0  size: 1024  my_flags: 80
my_malloc: my: size: 1024  my_flags: 80
my_malloc: exit: ptr: 0x7f2a200895e0
THD::binlog_query: enter: qtype: ROW  query: 'INSERT /*! IGNORE */ INTO table1_int_autoinc VALUES  (NULL, 't', 'we', NULL, NULL)'
MYSQL_BIN_LOG::flush_and_set_pending_rows_event(event): enter: event: 0x0
MYSQL_BIN_LOG::flush_and_set_pending_rows_event(event): info: cache_mngr->pending(): 0x7f2a200bc800
Log_event::write_header: enter: filepos: 65  length: 20 type: 23
MYSQL_BIN_LOG::flush_and_set_pending_rows_event(event): m_width: Memory: 0x7f2a882e0190  Bytes: (1)
05
MYSQL_BIN_LOG::flush_and_set_pending_rows_event(event): m_cols: Memory: 0x7f2a200bc8c8  Bytes: (1)
FF
MYSQL_BIN_LOG::flush_and_set_pending_rows_event(event): rows: Memory: 0x7f2a200895e0  Bytes: (10)
F8 01 00 00 00 01 74 02 77 65
my_free: my: ptr: 0x7f2a200895e0
sf_malloc_usable_size: exit: size: 1024  flags: 80
my_free: my: ptr: 0x7f2a200bc800
sf_malloc_usable_size: exit: size: 304  flags: 24
THD::binlog_query: debug: is_current_stmt_binlog_format_row: 1
mysql_insert: THD::enter_stage: /home/jan/mysql/galera-10.0/sql/sql_insert.cc:1078
alloc_root: enter: root: 0x39a4640
alloc_root: exit: ptr: 0x7f2a20005bf8
mysql_execute_command: THD::enter_stage: /home/jan/mysql/galera-10.0/sql/sql_parse.cc:5323
trans_register_ha: enter: stmt
ha_commit_trans: info: all: 0 thd->in_sub_stmt: 0 ha_info: 0x39a14c8 is_real_trans: 1
wsrep_run_wsrep_commit: wsrep: replicating commit
wsrep_run_wsrep_commit: mutex: LOCK_wsrep_thd (0x39a2fe0) locking
wsrep_run_wsrep_commit: mutex: LOCK_wsrep_thd (0x39a2fe0) locked
wsrep_run_wsrep_commit: mutex: LOCK_wsrep_replaying (0x18ed380) locking
my_malloc: my: size: 192  my_flags: 24
my_malloc: exit: ptr: 0x7f2a200bc800
my_hash_init: enter: hash: 0x7f2a200bc800  size: 128
my_malloc: my: size: 2048  my_flags: 0
my_malloc: exit: ptr: 0x7f2a20082e60
my_hash_init: enter: hash: 0x7f2a200bc860  size: 128
my_malloc: my: size: 2048  my_flags: 0
my_malloc: exit: ptr: 0x7f2a200b7f70
my_malloc: my: size: 56  my_flags: 56
my_malloc: exit: ptr: 0x7f2a200cedd0
add_to_locked_mutex: info: inserting 0x7f2a200cedd0  into  0x39a2fe0  (id: 284 -> 254)
wsrep_run_wsrep_commit: mutex: LOCK_wsrep_replaying (0x18ed380) locked
wsrep_run_wsrep_commit: mutex: LOCK_wsrep_replaying (0x18ed380) unlocking
wsrep_run_wsrep_commit: mutex: LOCK_wsrep_thd (0x39a2fe0) unlocking
reinit_io_cache: enter: cache: 0x7f2a200cf5e0 type: 1  seek_offset: 0  clear_cache: 0
my_malloc: my: size: 104  my_flags: 0
my_malloc: exit: ptr: 0x7f2a200bc940
my_free: my: ptr: 0x7f2a200bc940
sf_malloc_usable_size: exit: size: 104  flags: 0
wsrep_run_wsrep_commit: mutex: LOCK_wsrep_thd (0x39a2fe0) locking
wsrep_run_wsrep_commit: mutex: LOCK_wsrep_thd (0x39a2fe0) locked
wsrep_run_wsrep_commit: wsrep: replicating commit success
wsrep_run_wsrep_commit: mutex: LOCK_wsrep_thd (0x39a2fe0) unlocking
print_buffer_to_file: enter: buffer: WSREP: ::jan_BINLOG_COMMIT:: TRX: Pending (nil) pos 0, pos_in_file 139818903074288 current_pos 139818903139504 request_pos 139818903139608 write_pos 49062320

R:

--

Jan Lindström
Principal Engineer

MariaDB | MaxScale | skype: jan_p_lindstrom

www.skysql.com

Twitter Blog Facebook LinkedIn Google+