Hi Michael, Thanks! In the stack trace, I see fsp_flags_try_adjust(), a function that I had to introduce to fix a file format compatibility problem that had been introduced in MariaDB 10.1.0: MDEV-11623 <https://jira.mariadb.org/browse/MDEV-11623>, commit ab1e6fefd <https://github.com/MariaDB/server/commit/ab1e6fefd869242d962cb91a006f37bb9ad534a7> in MariaDB 10.1.21, 10.2.4. The format compatibility problem mostly affects a non-default innodb_page_size. However, the "root cause" for this slowdown is the validate=true in the following code in the function innobase_start_or_create_for_mysql(): bool validate = recv_needed_recovery && srv_force_recovery == 0; dict_check_tablespaces_and_store_max_id(validate); Normally, we should have validate=false, because no crash recovery should be needed. If the server was not shut down cleanly, then there should be a startup message "Starting crash recovery", and recv_needed_recovery should hold. The variable srv_force_recovery is tied to the parameter innodb_force_recovery, which is 0 by default. To me, this looks like it could be 'working as designed'. But, you did not share the server error log from the very start. Was there a crash recovery message? If not, then it is an interesting bug. If you copied the data files from a file system snapshot from a running server, then it is very likely that crash recovery was needed. You could work around this by setting --innodb-force-recovery=1 when starting up. But beware that this will cause InnoDB to ignore certain errors. Maybe a better idea would be to invoke "mariabackup --prepare" and "mariabackup --copy-back" in order to apply the redo log before starting up the server on the copied files. And while you are at it, you could also use "mariabackup --backup" and skip the file system snapshot. Best regards, Marko On Tue, Jan 9, 2018 at 4:32 PM, Michael Caplan <michael@eggplant.ws> wrote:
Hi Marko,
Thanks for your patience with me. I think I have better material for you to look at now. Best,
Mike
On 2018-01-09 03:17 AM, Marko Mäkelä wrote:
Michael,
Sorry, I now realize that you must likely have used a stripped executable, and therefore the symbols for the InnoDB functions are missing (displayed as ?? in the stack trace output from gdb). As far as I understand, gdb should be able to resolve the symbols if you install the separate debug symbol package:
sudo apt install mariadb-server-core-10.2-dbgsym
I tried to find the installation package corresponding to '10.2.11-MariaDB-10.2.11+maria~xenial-log' so that I would be able to resolve the stack trace manually. As a developer, normally I work with code that I build myself from the source code, and therefore I am a bit bad at giving advice that applies to the packaged code.
I downloaded and extracted the files from the following packages: mariadb-server-10.2-dbgsym_10.2.11+maria~xenial_amd64.deb <http://mirror.netinch.com/pub/mariadb//mariadb-10.2.11/repo/ubuntu/pool/main/m/mariadb-10.2/mariadb-server-10.2-dbgsym_10.2.11+maria%7Exenial_amd64.deb> mariadb-server-10.2_10.2.11+maria~xenial_amd64.deb <http://mirror.netinch.com/pub/mariadb//mariadb-10.2.11/repo/ubuntu/pool/main/m/mariadb-10.2/mariadb-server-10.2_10.2.11+maria%7Exenial_amd64.deb> mariadb-server-core-10.2-dbgsym_10.2.11+maria~xenial_amd64.deb <http://mirror.netinch.com/pub/mariadb//mariadb-10.2.11/repo/ubuntu/pool/main/m/mariadb-10.2/mariadb-server-core-10.2-dbgsym_10.2.11+maria%7Exenial_amd64.deb> mariadb-server-core-10.2_10.2.11+maria~xenial_amd64.deb <http://mirror.netinch.com/pub/mariadb//mariadb-10.2.11/repo/ubuntu/pool/main/m/mariadb-10.2/mariadb-server-core-10.2_10.2.11+maria%7Exenial_amd64.deb>
The server executable /usr/sbin/mysqld is included in the mysql-server-core package. Unfortunately I was unable to match that executable with the addresses from your gdb output. I did the following:
file mysqld mysqld: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=7649cc717c0ee820185a633fd27692f3b9160bed, stripped
gdb mysqld symbol-file ../lib/debug/.build-id/76/49cc717c0ee820185a633fd27692f3 b9160bed.debug
followed by a number of gdb commands. I expected the least significant 12 bits of the addresses to match in the disassembly of ha_initialize_handlerton() and plugin_init(), but that was not the case. Maybe I did something wrong.
So, can you please install the debug symbol package and try again?
Best regards,
Marko
-- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/