Re: [Maria-developers] 7107565: MDEV-10259 mysqld crash with certain statement length and...
Hi, Sachin! Great work! On Mar 27, Sachin Setiya wrote:
revision-id: 710756527ec8589bce33bc834a1d8e6604266b6d (mariadb-10.1.31-47-g7107565) parent(s): 8152c52e1af608da5d288a4e6fc82e2d804e64cd author: Sachin Setiya committer: Sachin Setiya timestamp: 2018-03-27 04:24:57 +0530 message:
MDEV-10259 mysqld crash with certain statement length and... order with Galera and encrypt-tmp-files=1
Problem:- If trans_cache (IO_CACHE) uses encrypted tmp file then on next DML server will crash.
Case:- Lets take a case , we have a table t1 , We try to do 2 inserts in t1 1. A really long insert so that trans_cache has to use temp_file 2. Just a small insert
Analysis:- Actually server crashes from inside of galera library. /lib64/libc.so.6(abort+0x175)[0x7fb5ba779dc5] /usr/lib64/galera/libgalera_smm.so(_ZN6galera3FSMINS_9TrxHandle5State... mysys/stacktrace.c:247(my_print_stacktrace)[0x7fb5a714940e] sql/signal_handler.cc:160(handle_fatal_signal)[0x7fb5a715c1bd] sql/wsrep_hton.cc:257(wsrep_rollback)[0x7fb5bcce923a] sql/wsrep_hton.cc:268(wsrep_rollback)[0x7fb5bcce9368] sql/handler.cc:1658(ha_rollback_trans(THD*, bool))[0x7fb5bcd4f41a] sql/handler.cc:1483(ha_commit_trans(THD*, bool))[0x7fb5bcd4f804]
but actual issue is not in galera but in mariadb, because for 2nd insert we should never call rollback. We are calling rollback because log_and_order fails it fails because write_cache fails , It fails because after reinit_io_cache(trans_cache) , my_b_tell says 0 so we look into tmp_file for data , which is obviously wrong since temp was used for previous insert and it no longer exist.
Add here something like: wsrep_write_cache_inc() reads the IO_CACHE in a loop, filling it with my_b_fill() until it returns "0 bytes read". Later MYSQL_BIN_LOG::write_cache() does the same. wsrep_write_cache_inc() assumes that reading a zero bytes past EOF leaves the old data in the cache. Let's do that.
diff --git a/sql/mf_iocache_encr.cc b/sql/mf_iocache_encr.cc index 149e6fe..41e1333 100644 --- a/sql/mf_iocache_encr.cc +++ b/sql/mf_iocache_encr.cc @@ -49,8 +49,7 @@ static int my_b_encr_read(IO_CACHE *info, uchar *Buffer, size_t Count)
if (pos_in_file == info->end_of_file) {
here, a comment: /* reading past EOF should not empty the cache */
- info->read_pos= info->read_end= info->buffer; - info->pos_in_file= pos_in_file; + info->read_pos= info->read_end; info->error= 0; DBUG_RETURN(MY_TEST(Count)); }
and add a test for it into unittest/sql/mf_iocache-t.cc. Something simple, like, write into a cache, reinit to READ, fill, fill, reinit to READ, verify the data is still there. Regards, Sergei Chief Architect MariaDB and security@mariadb.org
Hi serg! Thanks for the review! On Tue, May 8, 2018 at 2:51 PM, Sergei Golubchik <serg@mariadb.org> wrote:
Hi, Sachin!
Great work!
On Mar 27, Sachin Setiya wrote:
revision-id: 710756527ec8589bce33bc834a1d8e6604266b6d (mariadb-10.1.31-47-g7107565) parent(s): 8152c52e1af608da5d288a4e6fc82e2d804e64cd author: Sachin Setiya committer: Sachin Setiya timestamp: 2018-03-27 04:24:57 +0530 message:
MDEV-10259 mysqld crash with certain statement length and... order with Galera and encrypt-tmp-files=1
Problem:- If trans_cache (IO_CACHE) uses encrypted tmp file then on next DML server will crash.
Case:- Lets take a case , we have a table t1 , We try to do 2 inserts in t1 1. A really long insert so that trans_cache has to use temp_file 2. Just a small insert
Analysis:- Actually server crashes from inside of galera library. /lib64/libc.so.6(abort+0x175)[0x7fb5ba779dc5] /usr/lib64/galera/libgalera_smm.so(_ZN6galera3FSMINS_9TrxHandle5State... mysys/stacktrace.c:247(my_print_stacktrace)[0x7fb5a714940e] sql/signal_handler.cc:160(handle_fatal_signal)[0x7fb5a715c1bd] sql/wsrep_hton.cc:257(wsrep_rollback)[0x7fb5bcce923a] sql/wsrep_hton.cc:268(wsrep_rollback)[0x7fb5bcce9368] sql/handler.cc:1658(ha_rollback_trans(THD*, bool))[0x7fb5bcd4f41a] sql/handler.cc:1483(ha_commit_trans(THD*, bool))[0x7fb5bcd4f804]
but actual issue is not in galera but in mariadb, because for 2nd insert we should never call rollback. We are calling rollback because log_and_order fails it fails because write_cache fails , It fails because after reinit_io_cache(trans_cache) , my_b_tell says 0 so we look into tmp_file for data , which is obviously wrong since temp was used for previous insert and it no longer exist.
Add here something like:
wsrep_write_cache_inc() reads the IO_CACHE in a loop, filling it with my_b_fill() until it returns "0 bytes read". Later MYSQL_BIN_LOG::write_cache() does the same. wsrep_write_cache_inc() assumes that reading a zero bytes past EOF leaves the old data in the cache. Let's do that.
Done.
diff --git a/sql/mf_iocache_encr.cc b/sql/mf_iocache_encr.cc index 149e6fe..41e1333 100644 --- a/sql/mf_iocache_encr.cc +++ b/sql/mf_iocache_encr.cc @@ -49,8 +49,7 @@ static int my_b_encr_read(IO_CACHE *info, uchar *Buffer, size_t Count)
if (pos_in_file == info->end_of_file) {
here, a comment: /* reading past EOF should not empty the cache */
Done.
- info->read_pos= info->read_end= info->buffer; - info->pos_in_file= pos_in_file; + info->read_pos= info->read_end; info->error= 0; DBUG_RETURN(MY_TEST(Count)); }
and add a test for it into unittest/sql/mf_iocache-t.cc. Something simple, like, write into a cache, reinit to READ, fill, fill, reinit to READ, verify the data is still there.
Done.
Regards, Sergei Chief Architect MariaDB and security@mariadb.org
Updated Patch http://lists.askmonty.org/pipermail/commits/2018-May/012508.html -- Regards Sachin Setiya Software Engineer at MariaDB
HI serg! On Tue, May 8, 2018 at 5:16 PM, Sachin Setiya <sachin.setiya@mariadb.com> wrote:
Hi serg!
Thanks for the review!
On Tue, May 8, 2018 at 2:51 PM, Sergei Golubchik <serg@mariadb.org> wrote:
Hi, Sachin!
Great work!
revision-id: 710756527ec8589bce33bc834a1d8e6604266b6d (mariadb-10.1.31-47-g7107565) parent(s): 8152c52e1af608da5d288a4e6fc82e2d804e64cd author: Sachin Setiya committer: Sachin Setiya timestamp: 2018-03-27 04:24:57 +0530 message:
MDEV-10259 mysqld crash with certain statement length and... order with Galera and encrypt-tmp-files=1
Problem:- If trans_cache (IO_CACHE) uses encrypted tmp file then on next DML server will crash.
Case:- Lets take a case , we have a table t1 , We try to do 2 inserts in t1 1. A really long insert so that trans_cache has to use temp_file 2. Just a small insert
Analysis:- Actually server crashes from inside of galera library. /lib64/libc.so.6(abort+0x175)[0x7fb5ba779dc5] /usr/lib64/galera/libgalera_smm.so(_ZN6galera3FSMINS_9TrxHan
On Mar 27, Sachin Setiya wrote: dle5State...
mysys/stacktrace.c:247(my_print_stacktrace)[0x7fb5a714940e] sql/signal_handler.cc:160(handle_fatal_signal)[0x7fb5a715c1bd] sql/wsrep_hton.cc:257(wsrep_rollback)[0x7fb5bcce923a] sql/wsrep_hton.cc:268(wsrep_rollback)[0x7fb5bcce9368] sql/handler.cc:1658(ha_rollback_trans(THD*, bool))[0x7fb5bcd4f41a] sql/handler.cc:1483(ha_commit_trans(THD*, bool))[0x7fb5bcd4f804]
but actual issue is not in galera but in mariadb, because for 2nd insert we should never call rollback. We are calling rollback because log_and_order fails it fails because write_cache fails , It fails because after reinit_io_cache(trans_cache) , my_b_tell says 0 so we look into tmp_file for data , which is obviously wrong since temp was used for previous insert and it no longer exist.
Add here something like:
wsrep_write_cache_inc() reads the IO_CACHE in a loop, filling it with my_b_fill() until it returns "0 bytes read". Later MYSQL_BIN_LOG::write_cache() does the same. wsrep_write_cache_inc() assumes that reading a zero bytes past EOF leaves the old data in the cache. Let's do that.
Done.
diff --git a/sql/mf_iocache_encr.cc b/sql/mf_iocache_encr.cc index 149e6fe..41e1333 100644 --- a/sql/mf_iocache_encr.cc +++ b/sql/mf_iocache_encr.cc @@ -49,8 +49,7 @@ static int my_b_encr_read(IO_CACHE *info, uchar *Buffer, size_t Count)
if (pos_in_file == info->end_of_file) {
here, a comment: /* reading past EOF should not empty the cache */
Done.
- info->read_pos= info->read_end= info->buffer; - info->pos_in_file= pos_in_file; + info->read_pos= info->read_end; info->error= 0; DBUG_RETURN(MY_TEST(Count)); }
and add a test for it into unittest/sql/mf_iocache-t.cc. Something simple, like, write into a cache, reinit to READ, fill, fill, reinit to READ, verify the data is still there.
Done.
Regards, Sergei Chief Architect MariaDB and security@mariadb.org
Updated Patch http://lists.askmonty.org/pipermail/commits/2018- May/012508.html
Actually I forgot to add this part of code int main(int argc __attribute__((unused)),char *argv[]) { MY_INIT(argv[0]); - plan(29); + plan(44); This links to updated patch , http://lists.askmonty.org/pipermail/commits/2018-May/012509.html
-- Regards Sachin Setiya Software Engineer at MariaDB
-- Regards Sachin Setiya Software Engineer at MariaDB
participants (2)
-
Sachin Setiya -
Sergei Golubchik