[Maria-discuss] Orphaned TokuDB tables
Hello All, Sometimes, TokuDB gets left in a state where it thinks that parts of a table exist (as observed in TokuDB's file map), but not all of the underlying files (the TokuDB data files and/or the frm file) exist. This probably occurs due to the interaction between the non-transactional DDL that MySQL/MariaDB uses to create or drop tables, and the transactional TokuDB engine underneath. There is some bug there. So, what can one do to remove broken TokuDB file maps? There is an internal fractal tree called the tokudb.directory that names high level data names to an underlying file that the data is stored in. The tokudb_file_map information schema dumps out the contents of this tree. The key to this tree is the high level data name (called a dname). In TokuDB, this data name consists of the database name, the table name, and the index name. There is also a tree for the table (called status) that contains table wide metadata. I added the tokudb_delete_dnames_from_directory tool to the https://github.com/prohaska7/mariadb-server/tree/fix-tokudb-filemap branch. This standalone tool can be used to delete dnames from the tokudb.directory. Standalone means that MariaDB must have been previously shutdown cleanly (which puts the tokudb recovery logs in a nice clean state so that the tool does not have to run crash recovery). The input to the tool is the MariaDB data directory with nice and clean tokudb recovery logs along with a sequence of dname's. The tool transactionally deletes rows from the tokudb.directory fractal tree, which updates the recovery logs. Building the tool requires a MariaDB build environment a simple compile sequence something like this: gcc -g -o tokudb_delete_dnames_from_directory tokudb_delete_dnames_from_directory.cc -I$TFT/buildheader -L$TFT/src -ltokufractaltree Running the tool looks something like this: Get the dnames of interest from the tokudb_file_map. In this case, they are ./test/s-main and ./test/s-status. shutdown MariaDB ./tokudb_delete_dnames_from_directory ~/projects/mariadb-server-install/data ./test/s-main ./test/s-status
Hi, Rich! Shall we include it in packages together with ha_tokudb.so? Or is it too dangerous for that? On Aug 02, Rich Prohaska wrote:
Hello All,
Sometimes, TokuDB gets left in a state where it thinks that parts of a table exist (as observed in TokuDB's file map), but not all of the underlying files (the TokuDB data files and/or the frm file) exist. This probably occurs due to the interaction between the non-transactional DDL that MySQL/MariaDB uses to create or drop tables, and the transactional TokuDB engine underneath. There is some bug there.
I added the tokudb_delete_dnames_from_directory tool to the https://github.com/prohaska7/mariadb-server/tree/fix-tokudb-filemap branch. This standalone tool can be used to delete dnames from the tokudb.directory.
Regards, Sergei Chief Architect MariaDB and security@mariadb.org
On Tue, Aug 2, 2016 at 10:13 AM, Sergei Golubchik <serg@mariadb.org> wrote:
Hi, Rich!
Shall we include it in packages together with ha_tokudb.so? Or is it too dangerous for that?
It depends on how prevalent this problem is. If there are just a few instances of this problem, then maybe this tool should just be available for support people to use. However, if this problem starts to cost too much, then perhaps the tool could be integrated into mysqld instead of being a standalone tool. In the long term, transactional DDL in the mysql server is the way to go as it would close all of the holes. In addition, there probably is some bug in the interaction between the MySQL server and TokuDB WRT table creation and deletion that may need to be fixed. However, I was unable to find this bug when working at Tokutek.
On Aug 02, Rich Prohaska wrote:
Hello All,
Sometimes, TokuDB gets left in a state where it thinks that parts of a table exist (as observed in TokuDB's file map), but not all of the underlying files (the TokuDB data files and/or the frm file) exist. This probably occurs due to the interaction between the non-transactional DDL that MySQL/MariaDB uses to create or drop tables, and the transactional TokuDB engine underneath. There is some bug there.
I added the tokudb_delete_dnames_from_directory tool to the https://github.com/prohaska7/mariadb-server/tree/fix-tokudb-filemap branch. This standalone tool can be used to delete dnames from the tokudb.directory.
Regards, Sergei Chief Architect MariaDB and security@mariadb.org
participants (2)
-
Rich Prohaska
-
Sergei Golubchik