[Maria-discuss] Failure upon startup due to changes in /var/lib/mysql/databasename/
Hello, Thanks for the prior suggestions, they ended well. I found a behavior that I found a slight bit confounding. While I tested the server, one of the wikis I host was modified by a friend of mine. When I broke MariaDB, it was no longer feasible to start it, (to the degree that it could have taken days of tinkering), With MariaDB verified to be not running, I tarred the directory /var/lib/mysql/mywiki/ into a file mywiki_with_new_data.tar. I reinstalled MariaDB thoroughly to force the creation of a new /var/lib/mysql/ and /etc/mysql/ . I felt it was safe to recover the databases from the backup SQL file I had created on the currently running machine. Everything now works great, but the mywiki database doesn't reflect the changes anymore. So I stop the mysql process and then for the sake of recoverability, I tar up /var/lib/mysql/mywiki into a file called mywiki_without_new_data.tar. I then rm -rf /var/lib/mysql/mywiki and then untar the mywiki_with_new_data.tar in it's place. MariaDB will no longer start. If I again rm -rf /var/lib/mysql/mywiki and then untar mywiki_without_new_data.tar into the location, MariaDB starts without error. This isn't a code blue situation, my friend can recreate the change he made and that is the easiest solution, but others reading this might not have that luxury. I am also quite curious about what went wrong and how to repair the problem. I didn't think this was possible to fail. There is an interesting except from the errors described below. I'm not certain what exactly this means, or how to correct it. Given that MariaDB could have failed uncleanly when the mysql database was imported, recovery may be infeasible. (the wiki connected to MariaDB as root, it's bad, I know) <excerpt /var/lib/mysql/altdorf.err> InnoDB: Error: tablespace id is 235 in the data dictionary InnoDB: but in file ./mywiki/objectcache.ibd it is 254! 2017-06-07 11:45:08 5c7ff420 InnoDB: Assertion failure in thread 1551889440 in file fil0fil.cc line 638 InnoDB: We intentionally generate a memory trap. </excerpt> # service mysql start Job for mysql.service failed. See 'systemctl status mysql.service' and 'journalctl -xn' for details. # systemctl -l status mysql.service * mysql.service - LSB: Start and stop the mysql database server daemon Loaded: loaded (/etc/init.d/mysql) Active: failed (Result: exit-code) since Wed 2017-06-07 11:45:40 CEST; 1min 16s ago Process: 1692 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS) Process: 1753 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE) Jun 07 11:45:08 altdorf mysql[1753]: /etc/init.d/mysql: WARNING: /etc/mysql/my.cnf cannot be read. See README.Debian.gz ... (warning). Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: [61B blob data] Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")' Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists! Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: Jun 07 11:45:40 altdorf mysql[1753]: Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed! Jun 07 11:45:40 altdorf systemd[1]: mysql.service: control process exited, code=exited status=1 Jun 07 11:45:40 altdorf systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon. Jun 07 11:45:40 altdorf systemd[1]: Unit mysql.service entered failed state. # journalctl -xn -- Logs begin at Wed 2017-05-24 10:06:00 CEST, end at Wed 2017-06-07 11:45:40 CEST. -- Jun 07 11:45:08 altdorf /etc/init.d/mysql[1764]: WARNING: /etc/mysql/my.cnf cannot be read. See Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: 0 processes alive and '/usr/bin/mysqladmin --de Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: [61B blob data] Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: error: 'Can't connect to local MySQL server thr Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: Check that mysqld is running and that the socke Jun 07 11:45:40 altdorf /etc/init.d/mysql[2152]: Jun 07 11:45:40 altdorf mysql[1753]: Starting MariaDB database server: mysqld . . . . . . . . . Jun 07 11:45:40 altdorf systemd[1]: mysql.service: control process exited, code=exited status=1 Jun 07 11:45:40 altdorf systemd[1]: Failed to start LSB: Start and stop the mysql database serve -- Subject: Unit mysql.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit mysql.service has failed. -- -- The result is failed. Jun 07 11:45:40 altdorf systemd[1]: Unit mysql.service entered failed state. # cat /var/lib/mysql/aldorf.err (excerpts from file with a database name changed to protect user privacy) 170607 11:45:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 170607 11:45:08 [Note] /usr/sbin/mysqld (mysqld 10.0.30-MariaDB-0+deb8u2) starting as process 1851 ... 170607 11:45:08 [Note] InnoDB: Using mutexes to ref count buffer pool pages 170607 11:45:08 [Note] InnoDB: The InnoDB memory heap is disabled 170607 11:45:08 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 170607 11:45:08 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 170607 11:45:08 [Note] InnoDB: Compressed tables use zlib 1.2.8 170607 11:45:08 [Note] InnoDB: Using Linux native AIO 170607 11:45:08 [Note] InnoDB: Not using CPU crc32 instructions 170607 11:45:08 [Note] InnoDB: Initializing buffer pool, size = 128.0M 170607 11:45:08 [Note] InnoDB: Completed initialization of buffer pool 170607 11:45:08 [Note] InnoDB: Highest supported file format is Barracuda. 170607 11:45:08 [Note] InnoDB: 128 rollback segment(s) are active. 170607 11:45:08 [Note] InnoDB: Waiting for purge to start InnoDB: Error: tablespace id is 235 in the data dictionary InnoDB: but in file ./mywiki/objectcache.ibd it is 254! 2017-06-07 11:45:08 5c7ff420 InnoDB: Assertion failure in thread 1551889440 in file fil0fil.cc line 638 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 170607 11:45:08 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.30-MariaDB-0+deb8u2 key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=153 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 465934 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x48000 The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 170607 11:45:08 mysqld_safe mysqld from pid file /var/lib/mysql/altdorf.pid ended
Am 07.06.2017 um 12:34 schrieb Timothy D. Legg:
So I stop the mysql process and then for the sake of recoverability, I tar up /var/lib/mysql/mywiki into a file called mywiki_without_new_data.tar. I then rm -rf /var/lib/mysql/mywiki and then untar the mywiki_with_new_data.tar in it's place. MariaDB will no longer start. If I again rm -rf /var/lib/mysql/mywiki and then untar mywiki_without_new_data.tar into the location, MariaDB starts without error.
This isn't a code blue situation, my friend can recreate the change he made and that is the easiest solution, but others reading this might not have that luxury. I am also quite curious about what went wrong and how to repair the problem. I didn't think this was possible to fail.
There is an interesting except from the errors described below. I'm not certain what exactly this means, or how to correct it. Given that MariaDB could have failed uncleanly when the mysql database was imported, recovery may be infeasible. (the wiki connected to MariaDB as root, it's bad, I know)
<excerpt /var/lib/mysql/altdorf.err> InnoDB: Error: tablespace id is 235 in the data dictionary InnoDB: but in file ./mywiki/objectcache.ibd it is 254! 2017-06-07 11:45:08 5c7ff420 InnoDB: Assertion failure in thread 1551889440 in file fil0fil.cc line 638 InnoDB: We intentionally generate a memory trap. </excerpt>
you can't do that with InnoDB databases beaus eby design there exists a idiotic global table space even with "innodb_file_per_table" - why the whole server needs to crash with 10.1 and newer in such cases is a different story
When I broke MariaDB, it was no longer feasible to start it, (to the degree that it could have taken days of tinkering), With MariaDB verified to be not running, I tarred the directory /var/lib/mysql/mywiki/ into a file mywiki_with_new_data.tar.
I reinstalled MariaDB thoroughly to force the creation of a new /var/lib/mysql/ and /etc/mysql/ .
As Reindl Harald pointed out you can't do this with InnoDB since it has the 'ibdata' table namespace file. Afaik the only engine you can move/recreate table this way (binary) is MyISAM (and if I'm not wrong also Aria) all other engines more or less have some sort of namespace implementation and it's not possibly to move a single database/table to a different server just by copying the table files. p.s. I share similar opinions about the "innodb_file_per_table" not being enough and especially that in some cases the namespace file tends to grow beyond reason and there is no way to shrink/compact it other than a full data dump / deleting it and reimporting in a clean instance. rr
Transportable tablespaces is a possible, if fiddly, option... https://dev.mysql.com/doc/refman/5.7/en/tablespace-copying.html I think I've only ever done this between MariaDB instances but it should work with a source MySQL servers. Here's an example I did a while back... http://www.youdidwhatwithtsql.com/moving-innodb-database-flush-tables-export... Cheers, Rhys -----Original Message----- From: Maria-discuss [mailto:maria-discuss-bounces+rhys.campbell=swisscom.com@lists.launchpad.net] On Behalf Of Reinis Rozitis Sent: 07 June 2017 12:58 To: maria-discuss@lists.launchpad.net Subject: Re: [Maria-discuss] Failure upon startup due to changes in/var/lib/mysql/databasename/
When I broke MariaDB, it was no longer feasible to start it, (to the degree that it could have taken days of tinkering), With MariaDB verified to be not running, I tarred the directory /var/lib/mysql/mywiki/ into a file mywiki_with_new_data.tar.
I reinstalled MariaDB thoroughly to force the creation of a new /var/lib/mysql/ and /etc/mysql/ .
As Reindl Harald pointed out you can't do this with InnoDB since it has the 'ibdata' table namespace file. Afaik the only engine you can move/recreate table this way (binary) is MyISAM (and if I'm not wrong also Aria) all other engines more or less have some sort of namespace implementation and it's not possibly to move a single database/table to a different server just by copying the table files. p.s. I share similar opinions about the "innodb_file_per_table" not being enough and especially that in some cases the namespace file tends to grow beyond reason and there is no way to shrink/compact it other than a full data dump / deleting it and reimporting in a clean instance. rr _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
participants (4)
-
Reindl Harald
-
Reinis Rozitis
-
Rhys.Campbell@swisscom.com
-
Timothy D. Legg