[Maria-developers] Database corruption
Hello, We have a java tools to massively load data in a database (parallel load in different tables). With it, on bb-10.2-compatibility branch, I'm able to reproduce crash and sometimes database corruption: · on truncate (but delete works fine) · on parallel insert Crash seems occurred in innoDB code. See attached, logs of each case. Do you need more information ? Regards, Jérôme.
Hello Jerome, On 04/01/2017 04:17 PM, jerome brauge wrote:
Hello,
We have a java tools to massively load data in a database (parallel load in different tables).
With it, on bb-10.2-compatibility branch, I’m able to reproduce crash and sometimes database corruption:
· on truncate (but delete works fine)
· on parallel insert
Crash seems occurred in innoDB code.
See attached, logs of each case.
Do you need more information ?
Does the script use anything specific for sql_mode=ORACLE? Do you have a chance to test with the current main 10.2 branch?
Regards,
Jérôme.
I wonder if this could be caused by missing MDL protection around TRUNCATE TABLE, which would then incorrectly allow concurrent INSERT and TRUNCATE. One place where MDL is definitely missing is DROP DATABASE (reported as MDEV-11342 <https://jira.mariadb.org/browse/MDEV-11342>). That said, the InnoDB in MariaDB 10.2 is based on MySQL 5.7, and one difference between MySQL 5.6 and 5.7 is that InnoDB supports crash-safe atomic TRUNCATE TABLE for .ibd files (WL#6501 <http://dev.mysql.com/worklog/task/?id=6501>). Best regards, Marko -- DON’T MISS M|17 April 11 - 12, 2017 The Conrad Hotel New York City https://m17.mariadb.com/ Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation
Hello Marko, Jerome, Marko, thanks for your comments. On 04/03/2017 08:17 AM, Marko Mäkelä wrote:
I wonder if this could be caused by missing MDL protection around TRUNCATE TABLE, which would then incorrectly allow concurrent INSERT and TRUNCATE. One place where MDL is definitely missing is DROP DATABASE (reported as MDEV-11342 <https://jira.mariadb.org/browse/MDEV-11342>).
This is assigned to Sergey Vojtovich. Sergey is currently on vacation and will be back in the last decade of April. Marko, do you think it is something hard to fix? (Sorry, I'm not strong in this area of the code)
That said, the InnoDB in MariaDB 10.2 is based on MySQL 5.7, and one difference between MySQL 5.6 and 5.7 is that InnoDB supports crash-safe atomic TRUNCATE TABLE for .ibd files (WL#6501 <http://dev.mysql.com/worklog/task/?id=6501>).
Best regards,
Marko -- DON’T MISS M|17 April 11 - 12, 2017 The Conrad Hotel New York City https://m17.mariadb.com/
Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation
Hello Marko, Alexander, Thanks for your quick analysis. As I be able to reproduce crash with parallel "delete / insert", does it mean that "delete" act as truncate when there is no clause ? Feel free to ask me if you want me to test patches. Alexander, there is nothing specific to sql_mode=ORACLE. Best regards, Jérôme.
-----Message d'origine----- De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 3 avril 2017 06:24 À : Marko Mäkelä Cc : jerome brauge; MariaDB Developers (maria- developers@lists.launchpad.net); Sergey Vojtovich Objet : Re: [Maria-developers] Database corruption
Hello Marko, Jerome,
Marko, thanks for your comments.
On 04/03/2017 08:17 AM, Marko Mäkelä wrote:
I wonder if this could be caused by missing MDL protection around TRUNCATE TABLE, which would then incorrectly allow concurrent INSERT and TRUNCATE. One place where MDL is definitely missing is DROP DATABASE (reported as MDEV-11342 <https://jira.mariadb.org/browse/MDEV-11342>).
This is assigned to Sergey Vojtovich. Sergey is currently on vacation and will be back in the last decade of April.
Marko, do you think it is something hard to fix?
(Sorry, I'm not strong in this area of the code)
That said, the InnoDB in MariaDB 10.2 is based on MySQL 5.7, and one difference between MySQL 5.6 and 5.7 is that InnoDB supports crash-safe atomic TRUNCATE TABLE for .ibd files (WL#6501 <http://dev.mysql.com/worklog/task/?id=6501>).
Best regards,
Marko -- DON’T MISS M|17 April 11 - 12, 2017 The Conrad Hotel New York City https://m17.mariadb.com/
Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation
Hi Jérôme, Because InnoDB is a transactional storage engine when it comes to DML operations, DELETE FROM table (without a WHERE clause) is not equivalent to TRUNCATE TABLE. The reason is that you can ROLLBACK after DELETE, but you cannot ROLLBACK after TRUNCATE, which is a DDL operation (and DDL is not fully transactional). That said, I now have a better explanation for the problems that you have encountered. The branch bb-10.2-compatibility recently merged the branch 10.2 (which is close to what will be released as MariaDB Server 10.2.5). As part of this merge, a regression that was introduced in 10.1 was merged. That regression would cause a SIGSEGV during TRUNCATE TABLE (and presumably less likely during other DDL operations) when certain pages of the table are not in the InnoDB buffer pool. I pushed the fix (MDEV-12428 <https://jira.mariadb.org/browse/MDEV-12428>) to 10.1 and merged to 10.2 today. A merge of that to bb-10.2-compatibility should fix the issue. Best regards, Marko
Hello Marko, On 04/04/2017 10:13 AM, Marko Mäkelä wrote:
Hi Jérôme,
Because InnoDB is a transactional storage engine when it comes to DML operations, DELETE FROM table (without a WHERE clause) is not equivalent to TRUNCATE TABLE. The reason is that you can ROLLBACK after DELETE, but you cannot ROLLBACK after TRUNCATE, which is a DDL operation (and DDL is not fully transactional).
That said, I now have a better explanation for the problems that you have encountered.
The branch bb-10.2-compatibility recently merged the branch 10.2 (which is close to what will be released as MariaDB Server 10.2.5). As part of this merge, a regression that was introduced in 10.1 was merged. That regression would cause a SIGSEGV during TRUNCATE TABLE (and presumably less likely during other DDL operations) when certain pages of the table are not in the InnoDB buffer pool. I pushed the fix (MDEV-12428 <https://jira.mariadb.org/browse/MDEV-12428>) to 10.1 and merged to 10.2 today. A merge of that to bb-10.2-compatibility should fix the issue.
Please let me know when the fix is available in 10.2. I will propagate it to bb-10.2-compatibility asap. Thanks.
Best regards,
Marko
Hi Bar, On Tue, Apr 4, 2017 at 9:16 AM, Alexander Barkov <bar@mariadb.org> wrote:
Please let me know when the fix is available in 10.2. I will propagate it to bb-10.2-compatibility asap.
I merged it yesterday to 10.2. Both bb-10.2-compatibility and 10.3 will need the merge of https://github.com/MariaDB/server/commit/6d417a0bad205a6bacfee10dbc46dd631b0... or a later revision of 10.2. Marko -- DON’T MISS M|17 April 11 - 12, 2017 The Conrad Hotel New York City https://m17.mariadb.com/ Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation
Hi Marko, On 04/05/2017 01:18 PM, Marko Mäkelä wrote:
Hi Bar,
On Tue, Apr 4, 2017 at 9:16 AM, Alexander Barkov <bar@mariadb.org <mailto:bar@mariadb.org>> wrote:
Please let me know when the fix is available in 10.2. I will propagate it to bb-10.2-compatibility asap.
I merged it yesterday to 10.2. Both bb-10.2-compatibility and 10.3 will need the merge of https://github.com/MariaDB/server/commit/6d417a0bad205a6bacfee10dbc46dd631b0... or a later revision of 10.2.
Thanks for the update. I'm merging the latest 10.2 now.
Marko -- DON’T MISS M|17 April 11 - 12, 2017 The Conrad Hotel New York City https://m17.mariadb.com/
Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation
Hi Marko, Alexander, I confirm, I can load my database with no hang (with delete or truncate). Thank you very much. Regards, Jérôme.
-----Message d'origine----- De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mercredi 5 avril 2017 12:12 À : Marko Mäkelä Cc : jerome brauge; MariaDB Developers (maria- developers@lists.launchpad.net); Sergey Vojtovich Objet : Re: [Maria-developers] Database corruption
Hi Marko,
On 04/05/2017 01:18 PM, Marko Mäkelä wrote:
Hi Bar,
On Tue, Apr 4, 2017 at 9:16 AM, Alexander Barkov <bar@mariadb.org <mailto:bar@mariadb.org>> wrote:
Please let me know when the fix is available in 10.2. I will propagate it to bb-10.2-compatibility asap.
I merged it yesterday to 10.2. Both bb-10.2-compatibility and 10.3 will need the merge of
https://github.com/MariaDB/server/commit/6d417a0bad205a6bacfee10dbc 46d
d631b093e75 or a later revision of 10.2.
Thanks for the update. I'm merging the latest 10.2 now.
Marko -- DON’T MISS M|17 April 11 - 12, 2017 The Conrad Hotel New York City https://m17.mariadb.com/
Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation
Hello Jerome, Marko, I rebased bb-10.2-compatibility on top of the latest 10.2. Please clone again. (git pull won't work) Thanks. On 04/05/2017 01:18 PM, Marko Mäkelä wrote:
Hi Bar,
On Tue, Apr 4, 2017 at 9:16 AM, Alexander Barkov <bar@mariadb.org <mailto:bar@mariadb.org>> wrote:
Please let me know when the fix is available in 10.2. I will propagate it to bb-10.2-compatibility asap.
I merged it yesterday to 10.2. Both bb-10.2-compatibility and 10.3 will need the merge of https://github.com/MariaDB/server/commit/6d417a0bad205a6bacfee10dbc46dd631b0... or a later revision of 10.2.
Marko -- DON’T MISS M|17 April 11 - 12, 2017 The Conrad Hotel New York City https://m17.mariadb.com/
Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation
Hello Marko, Alexander, Thanks for your works. When this issue will be corrected on bb-10.2-compatibility, I replay my tests. Best regards, Jérôme. De : Marko Mäkelä [mailto:marko.makela@mariadb.com] Envoyé : mardi 4 avril 2017 08:14 À : jerome brauge Cc : Alexander Barkov; MariaDB Developers (maria-developers@lists.launchpad.net); Sergey Vojtovich Objet : Re: [Maria-developers] Database corruption Hi Jérôme, Because InnoDB is a transactional storage engine when it comes to DML operations, DELETE FROM table (without a WHERE clause) is not equivalent to TRUNCATE TABLE. The reason is that you can ROLLBACK after DELETE, but you cannot ROLLBACK after TRUNCATE, which is a DDL operation (and DDL is not fully transactional). That said, I now have a better explanation for the problems that you have encountered. The branch bb-10.2-compatibility recently merged the branch 10.2 (which is close to what will be released as MariaDB Server 10.2.5). As part of this merge, a regression that was introduced in 10.1 was merged. That regression would cause a SIGSEGV during TRUNCATE TABLE (and presumably less likely during other DDL operations) when certain pages of the table are not in the InnoDB buffer pool. I pushed the fix (MDEV-12428<https://jira.mariadb.org/browse/MDEV-12428>) to 10.1 and merged to 10.2 today. A merge of that to bb-10.2-compatibility should fix the issue. Best regards, Marko
participants (3)
-
Alexander Barkov -
jerome brauge -
Marko Mäkelä