Hi Michael,

Thanks! In the stack trace, I see fsp_flags_try_adjust(), a function that I had to introduce to fix a file format compatibility problem that had been introduced in MariaDB 10.1.0:
MDEV-11623, commit ab1e6fefd in MariaDB 10.1.21, 10.2.4. The format compatibility problem mostly affects a non-default innodb_page_size.

However, the "root cause" for this slowdown is the validate=true in the following code in the function innobase_start_or_create_for_mysql():

            bool validate = recv_needed_recovery
                && srv_force_recovery == 0;

            dict_check_tablespaces_and_store_max_id(validate);

Normally, we should have validate=false, because no crash recovery should be needed. If the server was not shut down cleanly, then there should be a startup message "Starting crash recovery", and recv_needed_recovery should hold.

The variable srv_force_recovery is tied to the parameter innodb_force_recovery, which is 0 by default.

To me, this looks like it could be 'working as designed'. But, you did not share the server error log from the very start. Was there a crash recovery message? If not, then it is an interesting bug.

If you copied the data files from a file system snapshot from a running server, then it is very likely that crash recovery was needed. You could work around this by setting --innodb-force-recovery=1 when starting up. But beware that this will cause InnoDB to ignore certain errors.

Maybe a better idea would be to invoke "mariabackup --prepare" and "mariabackup --copy-back" in order to apply the redo log before starting up the server on the copied files. And while you are at it, you could also use "mariabackup --backup" and skip the file system snapshot.

Best regards,

Marko

On Tue, Jan 9, 2018 at 4:32 PM, Michael Caplan <michael@eggplant.ws> wrote:

Hi Marko,

Thanks for your patience with me. I think I have better material for you to look at now.

Best,

Mike

On 2018-01-09 03:17 AM, Marko Mäkelä wrote:

Michael,

Sorry, I now realize that you must likely have used a stripped executable, and therefore the symbols for the InnoDB functions are missing (displayed as ?? in the stack trace output from gdb). As far as I understand, gdb should be able to resolve the symbols if you install the separate debug symbol package:

sudo apt install mariadb-server-core-10.2-dbgsym

I tried to find the installation package corresponding to '10.2.11-MariaDB-10.2.11+maria~xenial-log' so that I would be able to resolve the stack trace manually. As a developer, normally I work with code that I build myself from the source code, and therefore I am a bit bad at giving advice that applies to the packaged code.

I downloaded and extracted the files from the following packages:
mariadb-server-10.2-dbgsym_10.2.11+maria~xenial_amd64.deb
mariadb-server-10.2_10.2.11+maria~xenial_amd64.deb
mariadb-server-core-10.2-dbgsym_10.2.11+maria~xenial_amd64.deb
mariadb-server-core-10.2_10.2.11+maria~xenial_amd64.deb

The server executable /usr/sbin/mysqld is included in the mysql-server-core package. Unfortunately I was unable to match that executable with the addresses from your gdb output. I did the following:

file mysqld

mysqld: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=7649cc717c0ee820185a633fd27692f3b9160bed, stripped

gdb mysqld

symbol-file ../lib/debug/.build-id/76/49cc717c0ee820185a633fd27692f3b9160bed.debug

followed by a number of gdb commands. I expected the least significant 12 bits of the addresses to match in the disassembly of ha_initialize_handlerton() and plugin_init(), but that was not the case. Maybe I did something wrong.

So, can you please install the debug symbol package and try again?

Best regards,

Marko

Marko Mäkelä, Lead Developer InnoDB
MariaDB Corporation

DON’T MISS

M|18

MariaDB User Conference

February 26 - 27, 2018

New York City

https://m18.mariadb.com/