Re: [Maria-developers] [Commits] 8ac272ab2c4: MDEV-12253: Buffer pool blocks are accessed after they have been freed

On Mon, Mar 20, 2017 at 3:59 PM, jan wrote:

diff --git a/mysql-test/suite/encryption/include/innodb-util.pl b/mysql-test/suite/encryption/include/innodb-util.pl index 241545dac18..5a7295682dd 100644 --- a/mysql-test/suite/encryption/include/innodb-util.pl +++ b/mysql-test/suite/encryption/include/innodb-util.pl @@ -4,8 +4,11 @@ # All the tables must be in the same database, you can call it like so: # ib_backup_tablespaces("test", "t1", "blah", ...).

+use strict; +use warnings; use File::Copy; use File::Spec; +use Fcntl qw(:DEFAULT :seek);

sub ib_normalize_path { my ($path) = @_; @@ -124,3 +127,32 @@ sub ib_restore_ibd_files { ib_restore_ibd_file($tmpd, $datadir, $db, $table); } } + +sub ib_corrupt_tablespace { + my ($db, $table) = @_; + my $datadir = $ENV{'MYSQLD_DATADIR'}; + my $page_size = $ENV{'INNODB_PAGE_SIZE'}; + my $ibd_file = sprintf("%s%s/%s.ibd", $datadir,$db,$table);

The sprintf() function and the separate parameters $db, $table look like obfuscation to me. Why not just write "$ENV{MYSQLD_DATADIR}/test/t1.ibd" when invoking this function in the .test file?

+ print "file: $ibd_file $page_size\n";

Why do you want to pollute the output with the page size? That would require rdiff files for different page sizes.

+# We try to corrupt FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION field + my $len; + + sysopen IBD_FILE, $ibd_file, O_RDWR || die "Unable to open $ibd_file"; + + while ($len = sysread(IBD_FILE, $_, $page_size)) + { + sysseek(IBD_FILE, -$len, SEEK_CUR); + substr($_, 26, 8) = pack("N",0xdeadbeef); + print "Corrupted page with $chunk\n"; + syswrite(IBD_FILE, $_, $len); + } + close IBD_FILE; +}

Where is the variable $chunk defined? Why does this script execute issue output to stdout for every iteration of the loop? That would seem to make the test depend on the number of pages in the file, and thus cause result differences when using a different innodb_page_size. If the sysread() function happens to be interrupted by a signal, it could return a different amount than 0 or $page_size bytes. Thus, the operation of the loop would be nondeterministic. I would rather use a simple copying loop with a regular expression replacement, instead of this in-place modification. I would just do something like this: --exec perl -ei 's|(.{26}).{8}(.$ENV{INNODB_PAGE_SIZE})|"$1".pack("N",0xdeadbeef),"$2"|seg' $MYSQLD_DATADIR/test/t1.ibd Shorter and faster, and using fewer system calls.

diff --git a/mysql-test/suite/encryption/r/innodb-force-corrupt.result b/mysql-test/suite/encryption/r/innodb-force-corrupt.result new file mode 100644 index 00000000000..bba2ec17bbc --- /dev/null +++ b/mysql-test/suite/encryption/r/innodb-force-corrupt.result @@ -0,0 +1,38 @@ +call mtr.add_suppression("InnoDB: Database page corruption on disk or a failed.*"); +call mtr.add_suppression("InnoDB: Corruption: Block in space_id .* in file .* corrupted."); +call mtr.add_suppression("InnoDB: Based on page type .*"); +call mtr.add_suppression("InnoDB: Unable to read tablespace .* page no .* into the buffer pool after .* attempts"); +CALL mtr.add_suppression("InnoDB: Error: Unable to read tablespace .* page no .* into the buffer pool after 100 attempts"); +CALL mtr.add_suppression("InnoDB: Database page corruption on disk or a failed"); +CALL mtr.add_suppression("InnoDB: Space .* file test/t1 read of page .*"); +CALL mtr.add_suppression("InnoDB: You may have to recover from a backup."); +CALL mtr.add_suppression("InnoDB: It is also possible that your operatingsystem has corrupted its own file cache."); +CALL mtr.add_suppression("InnoDB: and rebooting your computer removes the error."); +CALL mtr.add_suppression("InnoDB: If the corrupt page is an index page you can also try to"); +CALL mtr.add_suppression("InnoDB: fix the corruption by dumping, dropping, and reimporting"); +CALL mtr.add_suppression("InnoDB: the corrupt table. You can use CHECK"); +CALL mtr.add_suppression("InnoDB: TABLE to scan your table for corruption."); +CALL mtr.add_suppression("InnoDB: See also .* about forcing recovery."); +CALL mtr.add_suppression("InnoDB: However key management plugin or used key_version 3221605118 is not found or used encryption algorithm or method does not match."); +CALL mtr.add_suppression("Marking tablespace as missing. You may drop this table or install correct key management plugin and key file."); +CALL mtr.add_suppression("InnoDB: Block in space_id .* in file .* encrypted.");

Are all these suppressions really required? (The same problem is with multiple tests. +call mtr.add_suppression("mysqld: File .* not found .*");

The suppression is looking for a substring by default (no implied $ at the end of the regular expression). The .* at the end of the regexp is clutter.

+call mtr.add_suppression(".*");

This catch-all is absolutely not acceptable.

+--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +--shutdown_server +--source include/wait_until_disconnected.inc

Do not use the low-level commands directly. Use restart_mysqld.inc. See the encryption test cleanup that I did in 10.2 in the merge: https://github.com/MariaDB/server/commit/2d96d13ecdcab4ce16c0d8976753d8aa501...

++--write_file $MYSQLTEST_VARDIR/keys1.txt +1;770A8A65DA156D24EE2A093277530142 +4;770A8A65DA156D24EE2A093277530143 +EOF

This file can be created already before the shutdown (or restart). Why not use key files in std_data? What is the benefit of creating files with identical static content in multiple --suite=encryption tests? +--exec echo "restart:--innodb-encrypt-tables --innodb-stats-persistent

--plugin-load-add=file_key_management.so --file-key-management --file-key-management-filename=$MYSQLTEST_VARDIR/keys1.txt" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +--enable_reconnect +--source include/wait_until_connected_again.inc

This is missing disable_reconnect, which is part of restart_mysqld.inc.

+--exec echo "restart:--innodb-encrypt-tables --innodb-stats-persistent --plugin-load-add=file_key_management.so --file-key-management --file-key-management-filename=$MYSQLTEST_VARDIR/keys2.txt" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +--enable_reconnect +--source include/wait_until_connected_again.inc +--source include/restart_mysqld.inc

What is the purpose of restarting the server twice here?

+--error 1296 +select count(*) from t1 FORCE INDEX (test) where b like 'secret%'; +--error 1296 +select count(*) from t2 FORCE INDEX (test) where b like 'secret%'; +select count(*) from t3 FORCE INDEX (test) where b like 'secret%';

Do not use numeric error codes. Refer to include/mysqld_ername.h in the build directory for the symbolic codes. 1296 is ER_GET_ERRMSG. +--remove_file $MYSQLTEST_VARDIR/keys1.txt

+--remove_file $MYSQLTEST_VARDIR/keys2.txt + +--write_file $MYSQLTEST_VARDIR/keys1.txt +1;770A8A65DA156D24EE2A093277530142 +4;770A8A65DA156D24EE2A093277530143 +EOF

Why remove and rewrite the file with the identical content? Similar steps are being done in multiple encryption tests. Can some of the repetition be avoided? Or could we test multiple operations with one server restart? Instead of having one test to check if CHECK TABLE crashes when the keys are incorrect or not available, another to check if ALTER TABLE crashes, and so on, just restart the server once and then execute all the commands that are to be tested. That would make --suite=encryption run faster too.

+--exec echo "restart:--innodb-encrypt-tables --innodb-stats-persistent --plugin-load-add=file_key_management.so --file-key-management --file-key-management-filename=$MYSQLTEST_VARDIR/keys1.txt" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +--enable_reconnect +--source include/wait_until_connected_again.inc + +drop table t1,t2,t3; + +--remove_file $MYSQLTEST_VARDIR/keys1.txt

The server was left running, tied to keys1.txt. Shouldn’t we rather restart the server at the end of the test, to have the same configuration as it was at the start of the test?

diff --git a/mysql-test/suite/encryption/t/innodb-force-corrupt.test b/mysql-test/suite/encryption/t/innodb-force-corrupt.test new file mode 100644 index 00000000000..3127b8f4006 --- /dev/null +++ b/mysql-test/suite/encryption/t/innodb-force-corrupt.test @@ -0,0 +1,122 @@ +# +# MDEV-11759: Encryption code in MariaDB 10.1/10.2 causes compatibility problems +# + +-- source include/have_innodb.inc +-- source include/have_file_key_management_plugin.inc +# Don't test under valgrind, memory leaks will occur +source include/not_valgrind.inc;

Which memory leaks? Please run this with --valgrind and show.

+# Avoid CrashReporter popup on Mac +source include/not_crashrep.inc;

Why? The test should not crash.

+# Don't test under embedded +source include/not_embedded.inc;

+# Require InnoDB

+source include/have_innodb.inc;

I would omit this kind of redundant comments. I’d add a comment to explain that embedded server tests do not support restarting.

+# Test could open crash reporter on Windows +# if compiler set up +source include/not_windows.inc;

This is totally unacceptable. We must test under Windows on Buildbot.

+call mtr.add_suppression("InnoDB: Database page corruption on disk or a failed.*"); +call mtr.add_suppression("InnoDB: Corruption: Block in space_id .* in file .* corrupted."); +call mtr.add_suppression("InnoDB: Based on page type .*"); +call mtr.add_suppression("InnoDB: Unable to read tablespace .* page no .* into the buffer pool after .* attempts"); +CALL mtr.add_suppression("InnoDB: Error: Unable to read tablespace .* page no .* into the buffer pool after 100 attempts"); +CALL mtr.add_suppression("InnoDB: Database page corruption on disk or a failed"); +CALL mtr.add_suppression("InnoDB: Space .* file test/t1 read of page .*"); +CALL mtr.add_suppression("InnoDB: You may have to recover from a backup."); +CALL mtr.add_suppression("InnoDB: It is also possible that your operatingsystem has corrupted its own file cache."); +CALL mtr.add_suppression("InnoDB: and rebooting your computer removes the error."); +CALL mtr.add_suppression("InnoDB: If the corrupt page is an index page you can also try to"); +CALL mtr.add_suppression("InnoDB: fix the corruption by dumping, dropping, and reimporting"); +CALL mtr.add_suppression("InnoDB: the corrupt table. You can use CHECK"); +CALL mtr.add_suppression("InnoDB: TABLE to scan your table for corruption."); +CALL mtr.add_suppression("InnoDB: See also .* about forcing recovery."); +CALL mtr.add_suppression("InnoDB: However key management plugin or used key_version 3221605118 is not found or used encryption algorithm or method does not match."); +CALL mtr.add_suppression("Marking tablespace as missing. You may drop this table or install correct key management plugin and key file."); +CALL mtr.add_suppression("InnoDB: Block in space_id .* in file .* encrypted.");

Are all these suppressions really necessary? Some of the first 4 suppressions are matching same strings as some of the later suppressions.

+let $MYSQLD_DATADIR=`select @@datadir`; +let INNODB_PAGE_SIZE=`select @@innodb_page_size`; +let MYSQLD_DATADIR=`select @@datadir`;

Why is MYSQLD_DATADIR assigned both to an mtr internal variable and an environment variable? The latter should suffice.

+--source include/shutdown_mysqld.inc +--source include/wait_until_disconnected.inc

The wait_until_disconnected.inc is redundant and should be removed.

+--disable_result_log +--error 1296 +SELECT * FROM t1; +# +# There is change that index is marked as corrupted here +# +--error 0,1712 +SELECT * FROM t2;

chance != change. Why would t2 sometimes be not corrupted? In which way does it fail if the 0, is removed? Use symbolic error codes, not numeric ones.

+--error 1296 +SELECT * FROM t3; +--enable_result_log + +--source include/shutdown_mysqld.inc +--source include/wait_until_disconnected.inc

Again, the wait_until_disconnected.inc should be removed.

+ +--echo # Restore the original t1.ibd +--remove_file $MYSQLD_DATADIR/test/t1.ibd +--move_file $MYSQLD_DATADIR/test/t1.ibd.backup $MYSQLD_DATADIR/test/t1.ibd +--move_file $MYSQLD_DATADIR/test/t2.ibd.backup $MYSQLD_DATADIR/test/t2.ibd +--move_file $MYSQLD_DATADIR/test/t3.ibd.backup $MYSQLD_DATADIR/test/t3.ibd + +--source include/start_mysqld.inc + +DROP TABLE t1,t2,t3;

diff --git a/mysql-test/suite/encryption/t/innodb-redo-badkey.test b/mysql-test/suite/encryption/t/innodb-redo-badkey.test

new file mode 100644 index 00000000000..34ffeb1b59a --- /dev/null +++ b/mysql-test/suite/encryption/t/innodb-redo-badkey.test @@ -0,0 +1,121 @@ +-- source include/have_innodb.inc +-- source include/have_file_key_management_plugin.inc +# embedded does not support restart +-- source include/not_embedded.inc +# Don't test this under valgrind, memory leaks will occur. +--source include/not_valgrind.inc +# Avoid CrashReporter popup on Mac +--source include/not_crashrep.inc +--source include/not_windows.inc

The not_windows.inc is not acceptable. I would say that crashes or memory leaks are not acceptable either.

+--source ../../suite/innodb/include/no_checkpoint_start.inc + +--disable_warnings +SET GLOBAL innodb_file_format = `Barracuda`; +SET GLOBAL innodb_file_per_table = ON; +--enable_warnings + +create table t1(a int not null primary key auto_increment, c char(250), b blob) engine=innodb row_format=compressed encrypted=yes encryption_key_id=10; +create table t2(a int not null primary key auto_increment, c char(250), b blob) engine=innodb row_format=compressed; +create table t3(a int not null primary key auto_increment, c char(250), b blob) engine=innodb encrypted=yes encryption_key_id=10; +create table t4(a int not null primary key auto_increment, c char(250), b blob) engine=innodb; +CREATE INDEX test ON t1 (b(10)); +CREATE INDEX test ON t2 (b(10)); +CREATE INDEX test ON t3 (b(10)); +CREATE INDEX test ON t4 (b(10)); + +--disable_query_log +--let $i = 20 +begin; +while ($i) +{ + insert into t1(c,b) values (repeat('secret1',20), repeat('secret2',6000)); + dec $i; +} +commit; +--enable_query_log + +begin; +insert into t2 select * from t1; +insert into t3 select * from t1; +insert into t4 select * from t1; +commit; + +--remove_file $MYSQLTEST_VARDIR/keys1.txt +let SEARCH_FILE= $MYSQLTEST_VARDIR/log/my_restart.err; + +# +# We test redo log page read at recv_read_page using +# incorrect keys from std_data/keys.txt. If checkpoint +# happens we will skip this test. If no checkpoint +# happens, InnoDB refuses to start as used +# encryption key is incorrect. +# +SET GLOBAL innodb_buf_flush_list_now = 1; +begin; +update t1 set c = repeat('secret3', 20); +update t2 set c = repeat('secret4', 20); +update t3 set c = repeat('secret4', 20); +update t4 set c = repeat('secret4', 20); +insert into t1 (c,b) values (repeat('secret5',20), repeat('secret6',6000)); +insert into t2 (c,b) values (repeat('secret7',20), repeat('secret8',6000)); +insert into t3 (c,b) values (repeat('secret9',20), repeat('secre10',6000)); +insert into t4 (c,b) values (repeat('secre11',20), repeat('secre12',6000)); +COMMIT; +let $cleanup= drop table t1,t2,t3,t4; +--let CLEANUP_IF_CHECKPOINT= $cleanup; +--source ../../suite/innodb/include/no_checkpoint_end.inc

Do we really need to execute all this code while not allowing any redo log checkpoint? It would seem to me that this test would be skipped most of the time. Where is the file my_restart.err created? +--write_file $MYSQLTEST_VARDIR/keys1.txt

+1;770A8A65DA156D24EE2A093277530997 +10;770A8A65DA156D24EE2A093277530142 +EOF + +--echo # restart +--error 1 +-- source include/start_mysqld.inc +--source include/kill_mysqld.inc

This looks very suspicious. Why is the start and kill needed? And when would --source return an error? Why would we want to kill the server here? A shutdown would make the test more deterministic. I think that we would want to ensure that InnoDB refused to start up after the start_mysqld.inc.

diff --git a/mysql-test/suite/encryption/t/innodb-redo-nokeys.test b/mysql-test/suite/encryption/t/innodb-redo-nokeys.test new file mode 100644 index 00000000000..bfe5cd78e5b --- /dev/null +++ b/mysql-test/suite/encryption/t/innodb-redo-nokeys.test @@ -0,0 +1,98 @@ +-- source include/have_innodb.inc +-- source include/have_file_key_management_plugin.inc +# embedded does not support restart +-- source include/not_embedded.inc +# Don't test this under valgrind, memory leaks will occur. +--source include/not_valgrind.inc +# Avoid CrashReporter popup on Mac +--source include/not_crashrep.inc +--source include/not_windows.inc + +--write_file $MYSQLTEST_VARDIR/keys1.txt +1;770A8A65DA156D24EE2A093277530997 +10;770A8A65DA156D24EE2A093277530999 +EOF

This file is very similar to the other added tests. I would prefer to have one test that tests multiple things across a server restart, instead of having a large number of tests that duplicate a lot of boilerplate code and make it hard to follow the actual purpose of the test. Can you please merge some of the tests? +--source ../../suite/innodb/include/no_checkpoint_start.inc

+ +create table t1(a int not null primary key auto_increment, c char(200), b blob) engine=innodb row_format=compressed encrypted=yes encryption_key_id=10; +create table t2(a int not null primary key auto_increment, c char(200), b blob) engine=innodb row_format=compressed; +create table t3(a int not null primary key auto_increment, c char(200), b blob) engine=innodb encrypted=yes encryption_key_id=10; +create table t4(a int not null primary key auto_increment, c char(200), b blob) engine=innodb; +CREATE INDEX test ON t1 (b(10)); +CREATE INDEX test ON t2 (b(10)); +CREATE INDEX test ON t3 (b(10)); +CREATE INDEX test ON t4 (b(10)); + +--disable_query_log +--let $i = 20 +begin; +while ($i) +{ + insert into t1(c,b) values (repeat('secret1',20), repeat('secret2',6000)); + dec $i; +} +commit; +--enable_query_log + +begin; +insert into t2 select * from t1; +insert into t3 select * from t1; +insert into t4 select * from t1; +commit; + +--remove_file $MYSQLTEST_VARDIR/keys1.txt + +# +# We test redo log page read at recv_read_page using +# incorrect keys from std_data/keys.txt. If checkpoint +# happens we will skip this test. If no checkpoint +# happens, InnoDB refuses to start as used +# encryption key is not found. +# +SET GLOBAL innodb_flush_log_at_trx_commit=1; +begin; +update t1 set c = repeat('secret3', 20); +update t2 set c = repeat('secret4', 20); +update t3 set c = repeat('secret4', 20); +update t4 set c = repeat('secret4', 20); +insert into t1 (c,b) values (repeat('secret5',20), repeat('secret6',6000)); +insert into t2 (c,b) values (repeat('secret7',20), repeat('secret8',6000)); +insert into t3 (c,b) values (repeat('secret9',20), repeat('secre10',6000)); +insert into t4 (c,b) values (repeat('secre11',20), repeat('secre12',6000)); +COMMIT; +let $cleanup= drop table t1,t2,t3,t4; +--let CLEANUP_IF_CHECKPOINT= $cleanup; +--source ../../suite/innodb/include/no_checkpoint_end.inc

Is it really necessary to place the no_checkpoint_start.inc before the table creation statements? Why are there separate CREATE TABLE and CREATE INDEX statements for the empty tables?

diff --git a/mysql-test/suite/innodb/t/innodb_bug14147491.test b/mysql-test/suite/innodb/t/innodb_bug14147491.test index 5776b2c2e37..d16a44e8a55 100644 --- a/mysql-test/suite/innodb/t/innodb_bug14147491.test +++ b/mysql-test/suite/innodb/t/innodb_bug14147491.test @@ -7,6 +7,7 @@ call mtr.add_suppression("InnoDB: Database page corruption on disk or a failed.*"); call mtr.add_suppression("InnoDB: Corruption: Block in space_id .* in file .* corrupted."); call mtr.add_suppression("InnoDB: Based on page type .*"); +call mtr.add_suppression("InnoDB: Unable to read tablespace .* page no .* into the buffer pool after .* attempts");

# Don't test under valgrind, memory leaks will occur source include/not_valgrind.inc;

I think that InnoDB is already noisy enough. Please fix the code instead of adding one more warning that is not really helping users. @@ -4096,6 +4096,11 @@ btr_estimate_number_of_different_key_vals(

page = btr_cur_get_page(&cursor);

+ if (index->table->file_unreadable || index->table->corrupted) { + mtr_commit(&mtr); + goto exit_loop; + } + rec = page_rec_get_next(page_get_infimum_rec(page));

if (!page_rec_is_supremum(rec)) {

When is this code covered? Why are we not testing this before assigning page? That variable is not being used after the goto. In fact, the variable "page" should be declared inside the for loop body. diff --git a/storage/innobase/btr/btr0pcur.cc b/storage/innobase/btr/

btr0pcur.cc index 01d2e1bb8e2..237e4e8d109 100644 --- a/storage/innobase/btr/btr0pcur.cc +++ b/storage/innobase/btr/btr0pcur.cc @@ -416,6 +416,11 @@ btr_pcur_move_to_next_page( cursor->old_stored = BTR_PCUR_OLD_NOT_STORED;

page = btr_pcur_get_page(cursor); + + if (!page) { + return; + } + next_page_no = btr_page_get_next(page, mtr); space = buf_block_get_space(btr_pcur_get_block(cursor)); zip_size = buf_block_get_zip_size(btr_pcur_get_block(cursor)); @@ -425,11 +430,16 @@ btr_pcur_move_to_next_page( next_block = btr_block_get(space, zip_size, next_page_no, cursor->latch_mode, btr_pcur_get_btr_cur(cursor)->index, mtr); + + if (!next_block) { + return; + } + next_page = buf_block_get_frame(next_block); #ifdef UNIV_BTR_DEBUG ut_a(page_is_comp(next_page) == page_is_comp(page)); ut_a(btr_page_get_prev(next_page, mtr) - == buf_block_get_page_no(btr_pcur_get_block(cursor))); + == buf_block_get_page_no(btr_pcur_get_block(cursor))); #endif /* UNIV_BTR_DEBUG */ next_block->check_index_page_at_flush = TRUE;

Are these changes really necessary? In which tests would the added "return" statements be executed? These conditions look like they could hurt performance.

diff --git a/storage/innobase/buf/buf0buf.cc b/storage/innobase/buf/ buf0buf.cc index c7e2183fa7b..066a68da8c0 100644 --- a/storage/innobase/buf/buf0buf.cc +++ b/storage/innobase/buf/buf0buf.cc @@ -2358,7 +2358,18 @@ buf_page_get_zip( /* Page not in buf_pool: needs to be read from file */

ut_ad(!hash_lock); - buf_read_page(space, zip_size, offset, NULL); + dberr_t err = buf_read_page(space, zip_size, offset); + + if (err != DB_SUCCESS) { + ib_logf(IB_LOG_LEVEL_ERROR, + "Reading compressed page " ULINTPF + " from tablespace " ULINTPF + " failed with error: %s (%d).", + offset, space, ut_strerr(err), err); + + goto err_exit; + } +

#if defined UNIV_DEBUG || defined UNIV_BUF_DEBUG ut_a(++buf_dbg_counter % 5771 || buf_validate());

Return the error to the only caller of this function, and let that function report the error. With the patch as it is, 2 messages would be reported for the single error. Reporting the ut_strerr(err) alone is sufficient; nobody is interested in the enum dberr_t numeric values. DBUG_EXECUTE_IF(

"innodb_page_corruption_retries", - retries = BUF_PAGE_READ_MAX_RETRIES; + goto force_fail; );

This change is wrong. The idea of the debug instrumentation is to speed up the failure (not retry reading the corrupted page for 100 times). The test must not crash even when this instrumentation is not present.

+ /* Try to set table as corrupted instead of + asserting. */ + if (space > TRX_SYS_SPACE && + dict_set_corrupted_by_space(space)) { + return (NULL); }

Why should we commit suicide when a page in the system tablespace is corrupted? I would just return NULL from here in any case. - if (!bpage->encrypted) {

- ut_ad(buf_pool->n_pend_reads > 0); - buf_pool->n_pend_reads--; + /* After this point bpage can't be referenced. */ + buf_LRU_free_one_page(bpage);

- buf_pool_mutex_exit(buf_pool); - } + ut_ad(buf_pool->n_pend_reads > 0); + buf_pool->n_pend_reads--;

- return(ret); + buf_pool_mutex_exit(buf_pool); }

Elsewhere, MariaDB 10.1 is using atomic memory access for the field buf_pool->n_pend_reads. We must be consistent. If some reader or writer is using atomic memory access and no buf_pool->mutex at all, then every access must use atomic memory access, no matter if holding buf_pool->mutex. @@ -4708,31 +4696,12 @@ buf_page_io_complete(

/* If page space id is larger than TRX_SYS_SPACE (0), we will attempt to mark the corresponding table as corrupted instead of crashing server */ - if (bpage->space > TRX_SYS_SPACE - && buf_mark_space_corrupt(bpage)) { - return(false); + if (bpage->space > TRX_SYS_SPACE) { + buf_mark_space_corrupt(bpage); + return(err); } else { - if (!bpage->encrypted) { - ib_logf(IB_LOG_LEVEL_ERROR, - "Ending processing because of a corrupt database page."); - - ut_error; - } - - ib_push_warning(innobase_get_trx(), DB_DECRYPTION_FAILED, - "Table in tablespace %lu encrypted." - "However key management plugin or used key_id %lu is not found or" - " used encryption algorithm or method does not match." - " Can't continue opening the table.", - bpage->space, bpage->key_version); - - if (bpage->encrypted && bpage->space > TRX_SYS_SPACE) { - buf_mark_space_corrupt(bpage); - } else { - ut_error; - } - - return(false); + ib_logf(IB_LOG_LEVEL_FATAL, + "Ending processing because of a corrupt database page."); } } }

Again, why commit suicide if a page in the system tablespace cannot be read? I think that we should try our best to allow any data to be rescue, and not crash early. diff --git a/storage/innobase/buf/buf0flu.cc b/storage/innobase/buf/

buf0flu.cc index 3f03ee8256f..e805585ed1e 100644 --- a/storage/innobase/buf/buf0flu.cc +++ b/storage/innobase/buf/buf0flu.cc @@ -956,7 +956,10 @@ buf_flush_write_block_low(

/* true means we want to evict this page from the LRU list as well. */ - buf_page_io_complete(bpage, true); + dberr_t err = buf_page_io_complete(bpage, true); + + /* This is write path, encryption should not fail here. */ + ut_ad(err == DB_SUCCESS); }

/* Increment the counter of I/O operations used

I think I already noted in an earlier review that a warning for the unused variable "err" would be issued for the non-debug build. Please build both non-debug and debug, and pay attention to the warnings! @@ -444,23 +464,23 @@ High-level function which reads a page asynchronously

from a file to the buffer buf_pool if it is not already there. Sets the io_fix flag and sets an exclusive lock on the buffer frame. The flag is cleared and the x-lock released by the i/o-handler thread. -@return TRUE if page has been read in, FALSE in case of failure */ +@param[in] space Tablespace id +@param[in] offset Page no */ UNIV_INTERN -ibool +void buf_read_page_async( -/*================*/ - ulint space, /*!< in: space id */ - ulint offset) /*!< in: page number */ + ulint space, + ulint offset)

Add a function comment that this function is only used for restoring buffer pool dumps, and that we do not care if the page exists or if the read succeeded. @@ -728,16 +748,30 @@ buf_read_ahead_linear(

count += buf_read_page_low( &err, false, ibuf_mode, - space, zip_size, FALSE, tablespace_version, i, NULL); - if (err == DB_TABLESPACE_DELETED) { - ut_print_timestamp(stderr); - fprintf(stderr, - " InnoDB: Warning: in" - " linear readahead trying to access\n" - "InnoDB: tablespace %lu page %lu,\n" - "InnoDB: but the tablespace does not" - " exist or is just being dropped.\n", - (ulong) space, (ulong) i); + space, zip_size, FALSE, tablespace_version, i); + + switch(err) { + case DB_SUCCESS: + break; + case DB_TABLESPACE_DELETED: + ib_logf(IB_LOG_LEVEL_WARN, + "In random" + "readahead trying to access" + "tablespace " ULINTPF " page " ULINTPF + " but the tablespace does not" + " exist or is just being dropped.", + space, i); + break; + + case DB_DECRYPTION_FAILED: + ib_logf(IB_LOG_LEVEL_ERROR, + "In random readahead read page " ULINTPF + " from tablespace " ULINTPF + " but page is still encrypted.", + i, space); + break; + default: + ut_error; }

You seem to have missed my earlier review comment where I noted that the message about "linear readahead" is misleadingly replaced by messages mentioning "random readahead".

@@ -829,6 +863,13 @@ buf_read_ibuf_merge_pages( page_nos[i], zip_size, FALSE); } + + if (err == DB_DECRYPTION_FAILED) { + ib_logf(IB_LOG_LEVEL_ERROR, + "Read page " ULINTPF " from tablespace " ULINTPF + " for insert buffer but page was encrypted.", + page_nos[i], space_ids[i]); + } }

os_aio_simulated_wake_handler_threads();

It would be clearer to say in the error message that we were unable to decrypt the page.

@@ -2192,7 +2192,7 @@ dict_load_table_low(

(*table)->id = mach_read_from_8(field);

- (*table)->ibd_file_missing = FALSE; + (*table)->file_unreadable = FALSE;

return(NULL); } @@ -2379,7 +2379,7 @@ dict_load_table( "Table '%s' tablespace is set as discarded.", table_name);

- table->ibd_file_missing = TRUE; + table->file_unreadable = true;

} else if (!fil_space_for_table_exists_in_mem( table->space, name, false, true, heap, @@ -2387,7 +2387,7 @@ dict_load_table(

if (DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY)) { /* Do not bother to retry opening temporary tables. */ - table->ibd_file_missing = TRUE; + table->file_unreadable = true;

} else { if (!(ignore_err & DICT_ERR_IGNORE_RECOVER_LOCK)) { @@ -2422,7 +2422,7 @@ dict_load_table( /* We failed to find a sensible tablespace file */

- table->ibd_file_missing = TRUE; + table->file_unreadable = TRUE; } if (filepath) { mem_free(filepath);

This is not consistent. Sometimes you are using true, sometimes TRUE or FALSE for the unsigned:1 bit-field. Either way is fine, but be consistent.

diff --git a/storage/innobase/fil/fil0crypt.cc b/storage/innobase/fil/ fil0crypt.cc index 7bef91f47ff..6a54ec6df35 100644 --- a/storage/innobase/fil/fil0crypt.cc +++ b/storage/innobase/fil/fil0crypt.cc @@ -541,6 +541,11 @@ fil_parse_write_crypt_data( fil_space_release(space); }

+ /* Check is used key found from encryption plugin */ + if (crypt_data->should_encrypt() && !crypt_data->is_key_found()) { + return NULL; + } + return ptr; }

As far as I can understand, this is prematurely ending the redo log recovery, without returning any error! This is not acceptable. Please add a proper test for this and ensure that an error is being returned.

diff --git a/storage/innobase/fil/fil0fil.cc b/storage/innobase/fil/ fil0fil.cc index 627a63aaae0..079803eedf3 100644 --- a/storage/innobase/fil/fil0fil.cc +++ b/storage/innobase/fil/fil0fil.cc @@ -6292,6 +6292,8 @@ fil_aio_wait( mutex_enter(&fil_system->mutex);

fil_node_complete_io(fil_node, fil_system, type); + ulint purpose = fil_node->space->purpose; + ulint space_id = fil_node->space->id;

mutex_exit(&fil_system->mutex);

Why assign fil_node->space->purpose to a local variable? It is only referenced once.

@@ -6303,9 +6305,29 @@ fil_aio_wait( deadlocks in the i/o system. We keep tablespace 0 data files always open, and use a special i/o thread to serve insert buffer requests. */

- if (fil_node->space->purpose == FIL_TABLESPACE) { + if (purpose == FIL_TABLESPACE) { srv_set_io_thread_op_info(segment, "complete io for buf page"); - buf_page_io_complete(static_cast(message)); + buf_page_t* bpage = static_cast(message); + ulint offset = bpage ? bpage->offset : ULINT_UNDEFINED; + + dberr_t err = buf_page_io_complete(bpage); + + if (err != DB_SUCCESS) { + ib_log_level_t level = IB_LOG_LEVEL_FATAL; + + /* In crash recovery set log corruption on + and produce only an error to fail InnoDB startup. */ + if (recv_recovery_is_on()) { + recv_sys->found_corrupt_log = true; + level = IB_LOG_LEVEL_ERROR; + } + + ib_logf(level, + "%s operation failed for tablespace " + ULINTPF " offset " ULINTPF " error=%s (%d).", + type == OS_FILE_WRITE ? "Write" : "Read", + space_id, offset, ut_strerr(err),err); + } } else { srv_set_io_thread_op_info(segment, "complete io for log"); log_io_complete(static_cast(message));

Can this really ever fail for writes? Please add a debug assertion and simplify the message. There is no need to display the numeric value of dberr_t; the ut_strerr() suffices. diff --git a/storage/innobase/include/btr0pcur.ic

b/storage/innobase/include/btr0pcur.ic index 1cd13824542..87b48de6389 100644 --- a/storage/innobase/include/btr0pcur.ic +++ b/storage/innobase/include/btr0pcur.ic @@ -357,6 +357,11 @@ btr_pcur_move_to_next( return(FALSE); }

+ if (cursor->btr_cur.index->table->corrupted || + cursor->btr_cur.index->table->file_unreadable) { + return(FALSE); + } + btr_pcur_move_to_next_page(cursor, mtr);

return(TRUE);

Is this change really needed? Can we merge the "corrupted" flag to "file_unreadable" to make the check less expensive? diff --git a/storage/innobase/include/log0recv.h b/storage/innobase/include/

log0recv.h index 9ca1c46d72b..1ff5cf721bf 100644 --- a/storage/innobase/include/log0recv.h +++ b/storage/innobase/include/log0recv.h @@ -274,9 +274,10 @@ recv_sys_var_init(void); #endif /* !UNIV_HOTBACKUP */ /** Apply the hash table of stored log records to persistent data pages. @param[in] last_batch whether the change buffer merge will be - performed as part of the operation */ + performed as part of the operation +@return DB_SUCCESS or DB_DECRYPTION_FAILED */ UNIV_INTERN -void +dberr_t recv_apply_hashed_log_recs(bool last_batch);

Please try to keep the return value as void, or alternatively add MY_ATTRIBUTE((warn_unused_result)) and fix all callers. I do not think it is easy to fix log_checkpoint(),

diff --git a/storage/innobase/log/log0recv.cc b/storage/innobase/log/ log0recv.cc index 45fd42848e8..e56d4a46650 100644 --- a/storage/innobase/log/log0recv.cc +++ b/storage/innobase/log/log0recv.cc @@ -1298,6 +1298,10 @@ recv_parse_or_apply_log_rec_body( break; case MLOG_FILE_WRITE_CRYPT_DATA: ptr = const_cast(fil_parse_write_crypt_data(ptr, end_ptr, block)); + + if (!ptr) { + recv_sys->found_corrupt_log = TRUE; + } break; default: ptr = NULL;

If block==NULL, ptr=NULL is returned on EOF. We do not want to have random bogus failures in redo log recovery! Even if a redo log record happens to be split across multiple redo log segments, we must parse the whole redo log!

@@ -3549,7 +3553,7 @@ row_truncate_table_for_mysql( "create a new tablespace", table->name);

- table->ibd_file_missing = 1; + table->file_unreadable = 1; err = DB_ERROR; goto funct_exit; }

Here we are assigning yet another literal to the field. Earlier, FALSE, TRUE, and true have been observed. Be consistent. Or even better, add a method void dict_table_t::set_unreadable(bool unreadable = true) and use it.

diff --git a/storage/innobase/row/row0sel.cc b/storage/innobase/row/ row0sel.cc index dab1bc58db5..8f73ed4d49a 100644 --- a/storage/innobase/row/row0sel.cc +++ b/storage/innobase/row/row0sel.cc @@ -3714,11 +3714,12 @@ row_search_for_mysql(

return(DB_TABLESPACE_DELETED);

- } else if (prebuilt->table->ibd_file_missing) { + } else if (prebuilt->table->file_unreadable && + fil_space_get(prebuilt->table->space) == NULL) {

return(DB_TABLESPACE_NOT_FOUND);

- } else if (prebuilt->table->is_encrypted) { + } else if (prebuilt->table->file_unreadable) {

return(DB_DECRYPTION_FAILED); } else if (!prebuilt->index_usable) {

I would add an method bool dict_table_t::is_file_missing() const and use it instead of duplicating the code. Marko -- DON’T MISS M|17 April 11 - 12, 2017 The Conrad Hotel New York City https://m17.mariadb.com/ Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation