June 2016 - discuss - lists.mariadb.org

Re: [Maria-discuss] Horrendeous InnoDB crash
by Mark 06 Jun '16

06 Jun '16

The log starts just with: 2016-06-06 16:39:16 7716 [Note] Server socket created on IP: '::'. 2016-06-06 16:39:16 7716 [Note] C:\xampp\mysql\bin\mysqld.exe: ready for connections. Version: '10.1.13-MariaDB' socket: '' port: 3306 mariadb.org binary distribution No errors at startup (database prefilled with manually run retrieval script). Then, at 16:47, when the scheduled retrieve task starts for the first time, the log adds: InnoDB: Error: trying to access page number 250370 in space 69, InnoDB: space name news/commentsxover, InnoDB: which is outside the tablespace bounds. InnoDB: Byte offset 0, len 16384, i/o type 10. InnoDB: If you get this error at mysqld startup, please check that InnoDB: your my.cnf matches the ibdata files that you have in the InnoDB: MySQL server. 2016-06-06 16:47:38 a74 InnoDB: Assertion failure in thread 2676 in file fil0fil.cc line 5866 So, with almost certainty, the scheduled task corrupts the database. Especially since I was now (unfortunately) able to replicate the same conditions for the crash. But there's nothing in that php script that could normally do that. (Especially since it runs manually just fine, without corrupting anything). The scheduled task runs as me, but is 'hidden', though (i.e., doesn't open a console). -----Original Message----- From: Chris Calender [mailto:chris.calender@mariadb.com] Sent: Monday, June 6, 2016 17:44 To: Mark <asarian(a)xs4all.nl>; maria-discuss(a)lists.launchpad.net Subject: Re: [Maria-discuss] Horrendeous InnoDB crash Hello, No need for the config file, you have corruption in your tablespace, hence the crash. It starts up here, at 16:39:16: > 2016-06-06 16:39:16 1e24 InnoDB: Warning: Using... > ... > 2016-06-06 16:39:16 7716 [Note] C:\xampp\mysql\bin\mysqld.exe: ready for > connections. > Version: '10.1.13-MariaDB' socket: '' port: 3306 mariadb.org binary distribution At the exact same time, we see the corruption (thus this corruption was both detected and mentioned at startup): > InnoDB: Error: trying to access page number 250370 in space 69, > InnoDB: space name spotweb/commentsxover, > InnoDB: which is outside the tablespace bounds. > InnoDB: Byte offset 0, len 16384, i/o type 10. > InnoDB: If you get this error at mysqld startup, please check that > InnoDB: your my.cnf matches the ibdata files that you have in the > InnoDB: MySQL server. Then, about 8 minutes later, you attempt to select from the same table that is corrupted ("spotweb.commentsxover"), hence the immediate assertion failure: > 2016-06-06 16:47:38 a74 InnoDB: Assertion failure in thread 2676 in > file fil0fil.cc line 5866 And the stack trace confirms the query: > Query (0x4b01740): SELECT messageid FROM commentsxover ORDER BY id DESC > LIMIT 5000 Since that is the only table referenced as being corrupted, you could probably get by with dropping it and then restoring it from a backup. If you do not have a current backup, then you will need to enable innodb_force_recovery (read below link for details), then dump from the table, dumping "around" the corruption using mysqldump with ORDER BY pk and LIMIT, and then once you have as much data as possible, then DROP the table, and import your dump. http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html Hope this helps! Best wishes, Chris On 6/6/2016 11:17 AM, Mark wrote: > And yet the same 9*&^%^&*()_ crash (see below). I'm beginning to get > real tired of this. > > The corruption starts almost immediately upon running a scheduled task > that's retrieving posts (at 16:47 hours). It's the same task I ran > manually first. There's absolutely no reason why a simple scheduled > post retrieval script should immediately corrupt MariaDB beyond all > recognition. The scheduled task consists of nothing more than a script > (that runs as me), like: > > c:\xampp\php\php.exe c:\xampp\htdocs\news\retrieve.php -force > > It runs every 30 mins. > > And no, no write-delays, or computer crashes that preceded this. > > 2016-06-06 15:52:27 3552 [Note] C:\xampp\mysql\bin\mysqld.exe: Normal > shutdown > > 2016-06-06 15:52:27 3552 [Note] Event Scheduler: Purging the queue. 0 > events > > 2016-06-06 15:52:27 3504 [Note] InnoDB: FTS optimize thread exiting. > > 2016-06-06 15:52:27 3552 [Note] InnoDB: Starting shutdown... > > 2016-06-06 15:52:29 3552 [Note] InnoDB: Shutdown completed; log > sequence number 6767908697 > > 2016-06-06 15:52:29 3552 [Note] C:\xampp\mysql\bin\mysqld.exe: > Shutdown complete > > 2016-06-06 16:39:16 1e24 InnoDB: Warning: Using > innodb_additional_mem_pool_size is DEPRECATED. This option may be > removed in future releases, together with the option > innodb_use_sys_malloc and with the InnoDB's internal memory allocator. > > 2016-06-06 16:39:16 7716 [Note] InnoDB: Using mutexes to ref count > buffer pool pages > > 2016-06-06 16:39:16 7716 [Note] InnoDB: The InnoDB memory heap is > disabled > > 2016-06-06 16:39:16 7716 [Note] InnoDB: Mutexes and rw_locks use > Windows interlocked functions > > 2016-06-06 16:39:16 7716 [Note] InnoDB: Memory barrier is not used > > 2016-06-06 16:39:16 7716 [Note] InnoDB: Compressed tables use zlib > 1.2.3 > > 2016-06-06 16:39:16 7716 [Note] InnoDB: Using generic crc32 > instructions > > 2016-06-06 16:39:16 7716 [Note] InnoDB: Initializing buffer pool, size > = 16.0M > > 2016-06-06 16:39:16 7716 [Note] InnoDB: Completed initialization of > buffer pool > > 2016-06-06 16:39:16 7716 [Note] InnoDB: Highest supported file format > is Barracuda. > > 2016-06-06 16:39:16 7716 [Note] InnoDB: 128 rollback segment(s) are active. > > 2016-06-06 16:39:16 7716 [Note] InnoDB: Waiting for purge to start > > 2016-06-06 16:39:16 7716 [Note] InnoDB: Percona XtraDB > (http://www.percona.com) 5.6.28-76.1 started; log sequence number > 6767908697 > > 2016-06-06 16:39:16 7100 [Note] InnoDB: Dumping buffer pool(s) not yet > started > > 2016-06-06 16:39:16 7716 [Note] Plugin 'FEEDBACK' is disabled. > > 2016-06-06 16:39:16 7716 [Note] Server socket created on IP: '::'. > > 2016-06-06 16:39:16 7716 [Note] C:\xampp\mysql\bin\mysqld.exe: ready > for connections. > > Version: '10.1.13-MariaDB' socket: '' port: 3306 mariadb.org binary > distribution > > InnoDB: Error: trying to access page number 250370 in space 69, > > InnoDB: space name spotweb/commentsxover, > > InnoDB: which is outside the tablespace bounds. > > InnoDB: Byte offset 0, len 16384, i/o type 10. > > InnoDB: If you get this error at mysqld startup, please check that > > InnoDB: your my.cnf matches the ibdata files that you have in the > > InnoDB: MySQL server. > > 2016-06-06 16:47:38 a74 InnoDB: Assertion failure in thread 2676 in > file fil0fil.cc line 5866 > > InnoDB: We intentionally generate a memory trap. > > InnoDB: Submit a detailed bug report to http://bugs.mysql.com. > > InnoDB: If you get repeated assertion failures or crashes, even > > InnoDB: immediately after the mysqld startup, there may be > > InnoDB: corruption in the InnoDB tablespace. Please refer to > > InnoDB: > http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html > > InnoDB: about forcing recovery. > > 160606 16:47:38 [ERROR] mysqld got exception 0x80000003 ; > > This could be because you hit a bug. It is also possible that this > binary > > or one of the libraries it was linked against is corrupt, improperly > built, > > or misconfigured. This error can also be caused by malfunctioning hardware. > > To report this bug, see https://mariadb.com/kb/en/reporting-bugs > > We will try our best to scrape up some info that will hopefully help > > diagnose the problem, but since we have already crashed, > > something is definitely wrong and this may fail. > > Server version: 10.1.13-MariaDB > > key_buffer_size=16777216 > > read_buffer_size=262144 > > max_used_connections=1 > > max_threads=1001 > > thread_count=1 > > It is possible that mysqld could use up to > > key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = > 787100 K bytes of memory > > Hope that's ok; if not, decrease some variables in the equation. > > Thread pointer: 0x0x4af7270 > > Attempting backtrace. You can use the following information to find > out > > where mysqld died. If you see no messages after this, something went > > terribly wrong... > > mysqld.exe!my_parameter_handler() > > mysqld.exe!my_mb_ctype_mb() > > mysqld.exe!?get_ctx@MDL_ticket@@QBEPAVMDL_context@@XZ() > > mysqld.exe!?get_ctx@MDL_ticket@@QBEPAVMDL_context@@XZ() > > mysqld.exe!?functype@Item_func_dyncol_create@@UBE?AW4Functype@Item_fun > c@@XZ() > > mysqld.exe!?get_ctx@MDL_ticket@@QBEPAVMDL_context@@XZ() > > mysqld.exe!??0Global_read_lock@@QAE@XZ() > > mysqld.exe!??0Global_read_lock@@QAE@XZ() > > mysqld.exe!?get_trg_event_map@Update_rows_log_event@@UAEEXZ() > > mysqld.exe!?get_trg_event_map@Update_rows_log_event@@UAEEXZ() > > mysqld.exe!?ha_open@handler@@QAEHPAUTABLE@@PBDHI@Z() > > mysqld.exe!?open_table_from_share@@YA?AW4open_frm_error@@PAVTHD@@PAUTA > BLE_SHARE@@PBDIIIPAUTABLE@@_N@Z() > > mysqld.exe!?open_table@@YA_NPAVTHD@@PAUTABLE_LIST@@PAVOpen_table_conte > xt@@@Z() > > mysqld.exe!?recover_from_failed_open@Open_table_context@@QAE_NXZ() > > mysqld.exe!?open_tables@@YA_NPAVTHD@@ABUDDL_options_st@@PAPAUTABLE_LIS > T@@PAIIPAVPrelocking_strategy@@@Z() > > mysqld.exe!?open_and_lock_tables@@YA_NPAVTHD@@ABUDDL_options_st@@PAUTA > BLE_LIST@@_NIPAVPrelocking_strategy@@@Z() > > mysqld.exe!??0Table_scope_and_contents_source_st@@QAE@ABU0@@Z() > > mysqld.exe!?mysql_execute_command@@YAHPAVTHD@@@Z() > > mysqld.exe!?mysql_parse@@YAXPAVTHD@@PADIPAVParser_state@@@Z() > > mysqld.exe!?dispatch_command@@YA_NW4enum_server_command@@PAVTHD@@PADI@ > Z() > > mysqld.exe!?do_command@@YA_NPAVTHD@@@Z() > > mysqld.exe!?threadpool_process_request@@YAHPAVTHD@@@Z() > > mysqld.exe!?tp_end@@YAXXZ() > > KERNEL32.DLL!SetUserGeoID() > > ntdll.dll!TpSimpleTryPost() > > ntdll.dll!EtwNotificationRegister() > > KERNEL32.DLL!BaseThreadInitThunk() > > ntdll.dll!RtlUnicodeStringToInteger() > > ntdll.dll!RtlUnicodeStringToInteger() > > Trying to get some variables. > > Some pointers may be invalid and cause the dump to abort. > > Query (0x4b01740): SELECT messageid FROM commentsxover ORDER BY id > DESC LIMIT 5000 > > Connection ID (thread ID): 2 > > Status: NOT_KILLED > > Optimizer switch: > index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_me > rge_intersection=on,index_merge_sort_intersection=off,engine_condition > _pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_wit > h_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=o > n,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on > ,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_ > join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,jo > in_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,tab > le_elimination=on,extended_keys=on,exists_to_in=on > > The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html > contains > > information that should help you find out what is causing the crash. > > *From:*Justin Swanhart [mailto:greenlion@gmail.com] > *Sent:* Monday, June 6, 2016 16:16 > *To:* Mark <asarian(a)xs4all.nl> > *Cc:* maria-discuss(a)lists.launchpad.net > *Subject:* Re: [Maria-discuss] Horrendeous InnoDB crash > > Hi, > > Do you have write-behind caching turned on? Do you have RAID card > without BBU? Did your computer crash? If the OS or hardware (like > IDE HDD cache) lies about actually writing to disk, InnoDB can become > corrupted, and this is not the fault of innodb. Be aware that if this > is a VM, many VM lie about disk writes, a "feature" you must turn off. > > Sent from my iPhone > > > On Jun 6, 2016, at 6:23 AM, Mark <asarian(a)xs4all.nl > <mailto:asarian@xs4all.nl>> wrote: > > Sorry I wiped the database; but it got to the point where MariaDB > immediately crashed upon startup (so the usual repair attempts were > futile, as they all kinda assume a MariaDB server that's up and > running). I have years of experience (FreeBSD) with myisam tables; > but a crash like this is a first for me, really, where the entire > database was foobarred to the point MariaDB wouldn't even start any > more. > > Thanks for creating the bug report. I shall investigate myself too, > in trying to find a way to integrate the MariaDB shutdown gracefully > (in UNIX this is all much simpler and straightforward). > > -Mark > > *From:*Vladislav Vaintroub [mailto:vvaintroub@gmail.com] > *Sent:* Monday, June 6, 2016 11:39 > *To:* Mark <asarian(a)xs4all.nl <mailto:asarian@xs4all.nl>>; > maria-discuss(a)lists.launchpad.net > <mailto:maria-discuss@lists.launchpad.net> > *Subject:* Re: [Maria-discuss] Horrendeous InnoDB crash > > Yes, it is possible that Windows does not give enough time to > MariaDB to shut down gracefully. I just created a task for this one > here https://jira.mariadb.org/browse/MDEV-10183. > > Still, Innodb should handle this situation gracefully, and recover > on startup, so what you see is a bug. It is a pity that you wiped > down the database, it would be useful for a bug report. > > On 06.06.2016 11:08, Mark wrote: > > Got a huge crash today, right after initializing my first > MariaDB database (see below). Got several more errors about > tables having crashed later on, and MariaDB wouldn't even start > up any more (I wound up wiping the entire database). > > So, my question is, what could cause MariaDB to fail so > horrendeously?! I though InnoDB was supposed to be *better* than > myisam!? Is is because maybe Windows 10 doesn't give MariaDB > enough time to shut down gracefully? Could it be because a > Windows scheduler job (potentially) aborts the fetcher script > when it's still running? (Again, I though InnoDB was supposed to > be transaction-safe). > > In its current state, MariaDB is completely unusable for me. > > And no, I don't have hard disk errors. :) > > Seriously, though, I could use some major insight! > > Thanks. > > 2016-06-06 10:32:15 2552 [Note] Server socket created on IP: '::'. > > 2016-06-06 10:32:15 2552 [Note] C:\xampp\mysql\bin\mysqld.exe: > ready for connections. > > Version: '10.1.13-MariaDB' socket: '' port: 3306 mariadb.org > <http://mariadb.org> binary distribution > > InnoDB: Error: trying to access page number 251010 in space > 69, > > InnoDB: space name news/commentsxover, > > InnoDB: which is outside the tablespace bounds. > > InnoDB: Byte offset 0, len 16384, i/o type 10. > > InnoDB: If you get this error at mysqld startup, please check > that > > InnoDB: your my.cnf matches the ibdata files that you have in > the > > InnoDB: MySQL server. > > 2016-06-06 08:47:46 e08 InnoDB: Assertion failure in thread > 3592 in file fil0fil.cc <http://fil0fil.cc> line 5866 > > InnoDB: We intentionally generate a memory trap. > > InnoDB: Submit a detailed bug report to http://bugs.mysql.com. > > InnoDB: If you get repeated assertion failures or crashes, > even > > InnoDB: immediately after the mysqld startup, there may be > > InnoDB: corruption in the InnoDB tablespace. Please refer to > > InnoDB: > > http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html > > InnoDB: about forcing recovery. > > 160606 8:47:46 [ERROR] mysqld got exception 0x80000003 ; > > This could be because you hit a bug. It is also possible that > this binary > > or one of the libraries it was linked against is corrupt, > improperly built, > > or misconfigured. This error can also be caused by > malfunctioning hardware. > > > > > > _______________________________________________ > > Mailing list:https://launchpad.net/~maria-discuss > > Post to :maria-discuss@lists.launchpad.net > <mailto:maria-discuss@lists.launchpad.net> > > Unsubscribe :https://launchpad.net/~maria-discuss > > More help :https://help.launchpad.net/ListHelp > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > <mailto:maria-discuss@lists.launchpad.net> > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp > > > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp > -- Chris Calender, M.S., Principal Support Engineer MariaDB Corporation

1 0

Re: [Maria-discuss] Horrendeous InnoDB crash
by Federico Razzoli 06 Jun '16

06 Jun '16

InnoDB is different from MyISAM. It crashed intentionally, as stated in the log you posted, to avoid data corruption propagation. Take a look at this page, so next time you can try to repair tables: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html Regards, Federico -------------------------------------------- Lun 6/6/16, Mark <asarian(a)xs4all.nl> ha scritto: Oggetto: Re: [Maria-discuss] Horrendeous InnoDB crash A: maria-discuss(a)lists.launchpad.net Data: Lunedì 6 giugno 2016, 12:23 #yiv2481875527 #yiv2481875527 -- _filtered #yiv2481875527 {font-family:Wingdings;panose-1:5 0 0 0 0 0 0 0 0 0;} _filtered #yiv2481875527 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv2481875527 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv2481875527 {font-family:Consolas;panose-1:2 11 6 9 2 2 4 3 2 4;} #yiv2481875527 #yiv2481875527 p.yiv2481875527MsoNormal, #yiv2481875527 li.yiv2481875527MsoNormal, #yiv2481875527 div.yiv2481875527MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:11.0pt;color:black;} #yiv2481875527 a:link, #yiv2481875527 span.yiv2481875527MsoHyperlink {color:#0563C1;text-decoration:underline;} #yiv2481875527 a:visited, #yiv2481875527 span.yiv2481875527MsoHyperlinkFollowed {color:#954F72;text-decoration:underline;} #yiv2481875527 p {margin-right:0in;margin-left:0in;font-size:11.0pt;color:black;} #yiv2481875527 pre {margin:0in;margin-bottom:.0001pt;font-size:10.0pt;color:black;} #yiv2481875527 p.yiv2481875527MsoListParagraph, #yiv2481875527 li.yiv2481875527MsoListParagraph, #yiv2481875527 div.yiv2481875527MsoListParagraph {margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;font-size:11.0pt;color:black;} #yiv2481875527 p.yiv2481875527msonormal0, #yiv2481875527 li.yiv2481875527msonormal0, #yiv2481875527 div.yiv2481875527msonormal0 {margin-right:0in;margin-left:0in;font-size:12.0pt;color:black;} #yiv2481875527 span.yiv2481875527EmailStyle19 {color:windowtext;} #yiv2481875527 span.yiv2481875527HTMLPreformattedChar {color:black;} #yiv2481875527 span.yiv2481875527EmailStyle22 {color:windowtext;} #yiv2481875527 .yiv2481875527MsoChpDefault {font-size:10.0pt;} _filtered #yiv2481875527 {margin:1.0in 1.0in 1.0in 1.0in;} #yiv2481875527 div.yiv2481875527WordSection1 {} #yiv2481875527 _filtered #yiv2481875527 {} _filtered #yiv2481875527 {} _filtered #yiv2481875527 {} _filtered #yiv2481875527 {font-family:Wingdings;} _filtered #yiv2481875527 {font-family:Symbol;} _filtered #yiv2481875527 {} _filtered #yiv2481875527 {font-family:Wingdings;} _filtered #yiv2481875527 {font-family:Symbol;} _filtered #yiv2481875527 {} _filtered #yiv2481875527 {font-family:Wingdings;} #yiv2481875527 ol {margin-bottom:0in;} #yiv2481875527 ul {margin-bottom:0in;} #yiv2481875527 Sorry I wiped the database; but it got to the point where MariaDB immediately crashed upon startup (so the usual repair attempts were futile, as they all kinda assume a MariaDB server that’s up and running). I have years of experience (FreeBSD) with myisam tables; but a crash like this is a first for me, really, where the entire database was foobarred to the point MariaDB wouldn’t even start any more. Thanks for creating the bug report. I shall investigate myself too, in trying to find a way to integrate the MariaDB shutdown gracefully (in UNIX this is all much simpler and straightforward). - Mark From: Vladislav Vaintroub [mailto:vvaintroub@gmail.com] Sent: Monday, June 6, 2016 11:39 To: Mark <asarian(a)xs4all.nl>; maria-discuss(a)lists.launchpad.net Subject: Re: [Maria-discuss] Horrendeous InnoDB crash Yes, it is possible that Windows does not give enough time to MariaDB to shut down gracefully. I just created a task for this one here https://jira.mariadb.org/browse/MDEV-10183.Still, Innodb should handle this situation gracefully, and recover on startup, so what you see is a bug. It is a pity that you wiped down the database, it would be useful for a bug report. On 06.06.2016 11:08, Mark wrote:Got a huge crash today, right after initializing my first MariaDB database (see below). Got several more errors about tables having crashed later on, and MariaDB wouldn't even start up any more (I wound up wiping the entire database). So, my question is, what could cause MariaDB to fail so horrendeously?! I though InnoDB was supposed to be *better* than myisam!? Is is because maybe Windows 10 doesn't give MariaDB enough time to shut down gracefully? Could it be because a Windows scheduler job (potentially) aborts the fetcher script when it's still running? (Again, I though InnoDB was supposed to be transaction-safe). In its current state, MariaDB is completely unusable for me. And no, I don't have hard disk errors. :) Seriously, though, I could use some major insight! Thanks. 2016-06-06 10:32:15 2552 [Note] Server socket created on IP: '::'.2016-06-06 10:32:15 2552 [Note] C:\xampp\mysql\bin\mysqld.exe: ready for connections.Version: '10.1.13-MariaDB' socket: '' port: 3306 mariadb.org binary distributionInnoDB: Error: trying to access page number 251010 in space 69,InnoDB: space name news/commentsxover,InnoDB: which is outside the tablespace bounds.InnoDB: Byte offset 0, len 16384, i/o type 10.InnoDB: If you get this error at mysqld startup, please check thatInnoDB: your my.cnf matches the ibdata files that you have in theInnoDB: MySQL server.2016-06-06 08:47:46 e08 InnoDB: Assertion failure in thread 3592 in file fil0fil.cc line 5866InnoDB: We intentionally generate a memory trap.InnoDB: Submit a detailed bug report to http://bugs.mysql.com.InnoDB: If you get repeated assertion failures or crashes, evenInnoDB: immediately after the mysqld startup, there may beInnoDB: corruption in the InnoDB tablespace. Please refer toInnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.htmlInnoDB: about forcing recovery.160606 8:47:46 [ERROR] mysqld got exception 0x80000003 ;This could be because you hit a bug. It is also possible that this binaryor one of the libraries it was linked against is corrupt, improperly built,or misconfigured. This error can also be caused by malfunctioning hardware. _______________________________________________Mailing list: https://launchpad.net/~maria-discussPost to : maria-discuss(a)lists.launchpad.netUnsubscribe : https://launchpad.net/~maria-discussMore help : https://help.launchpad.net/ListHelp -----Segue allegato----- _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss(a)lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

1 0

[Maria-discuss] Galera Cluster on FreeBSD
by Niklaas Baudet von Gersdorff 03 Jun '16

03 Jun '16

Hello, I'm trying to setup a galera cluster on FreeBSD. I followed the instructions on [1] but every time I issue SHOW STATUS LIKE 'wsrep_cluster_size'; I get +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | wsrep_cluster_size | 0 | +--------------------+-------+ 1 row in set (0.00 sec) So, the cluster is not working. This is the configuration of the node that I want to set as primary: -----------------------------------8<----------------------------------- [mysqld] datadir=/var/db/mysql socket=/tmp/mysql.sock user=mysql binlog_format=ROW bind-address=0.0.0.0 default_storage_engine=innodb innodb_autoinc_lock_mode=2 innodb_flush_log_at_trx_commit=0 innodb_buffer_pool_size=122M wsrep_provider=/usr/local/lib/libgalera_smm.so wsrep_provider_options="gcache.size=300M; gcache.page_size=300M" wsrep_cluster_name="klaas_cluster" wsrep_cluster_address="gcomm://" #wsrep_cluster_address="gcomm://10.1.7.1,10.1.7.2,10.3.7.1" wsrep_sst_method=rsync [mysql] wsrep_cluster_name=klaas_cluster wsrep_cluster_address="gcomm://10.1.7.1,10.1.7.2,10.3.7.1" wsrep_node_name=box-hlm-01 wsrep_node_address="10.1.7.1" [mysql_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid ----------------------------------->8----------------------------------- On the other two nodes the configuration is the same, except (1) only the second `wsrep_cluster_address` exists; (2) `wsrep_node_address` differs, obviously. Instead of using an "empty" `wsrep_cluster_address` I also tried setting mysql_args="--wsrep-new-cluster" in /etc/rc.conf without getting the desired result. The logs lack any relevant warnings or errors: -----------------------------------8<----------------------------------- 160602 07:59:44 mysqld_safe mysqld from pid file /var/db/mysql/mariadb1.box-hlm-01.klaas.pid ended 160602 08:00:34 mysqld_safe Starting mysqld daemon with databases from /var/db/mysql 2016-06-02 8:00:34 34426872832 [Note] /usr/local/libexec/mysqld (mysqld 10.1.13-MariaDB) starting as process 22580 ... 2016-06-02 8:00:34 34426872832 [Warning] You need to use --log-bin to make --binlog-format work. 2016-06-02 8:00:34 34426872832 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2016-06-02 8:00:34 34426872832 [Note] InnoDB: The InnoDB memory heap is disabled 2016-06-02 8:00:34 34426872832 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-06-02 8:00:34 34426872832 [Note] InnoDB: Memory barrier is not used 2016-06-02 8:00:34 34426872832 [Note] InnoDB: Compressed tables use zlib 1.2.8 2016-06-02 8:00:34 34426872832 [Note] InnoDB: Using SSE crc32 instructions 2016-06-02 8:00:34 34426872832 [Note] InnoDB: Initializing buffer pool, size = 122.0M 2016-06-02 8:00:34 34426872832 [Note] InnoDB: Completed initialization of buffer pool 2016-06-02 8:00:34 34426872832 [Note] InnoDB: Highest supported file format is Barracuda. 2016-06-02 8:00:34 34426872832 [Note] InnoDB: 128 rollback segment(s) are active. 2016-06-02 8:00:34 34426872832 [Note] InnoDB: Waiting for purge to start 2016-06-02 8:00:34 34426872832 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.28-76.1 started; log sequence number 1616955 2016-06-02 8:00:34 34426872832 [Note] Plugin 'FEEDBACK' is disabled. 2016-06-02 8:00:34 34426899456 [Note] InnoDB: Dumping buffer pool(s) not yet started 2016-06-02 8:00:34 34426872832 [Note] Server socket created on IP: '0.0.0.0'. 2016-06-02 8:00:34 34426872832 [Note] /usr/local/libexec/mysqld: ready for connections. Version: '10.1.13-MariaDB' socket: '/tmp/mysql.sock' port: 3306 FreeBSD Ports ----------------------------------->8----------------------------------- Any help is very much appreciated. Niklaas 1: http://galeracluster.com/documentation-webpages/index.html

3 6

[Maria-discuss] New Question: Consistently able to crash MariaDB 10.0.25 and 10.1.4 - Signal 11
by AskMonty KB 02 Jun '16

02 Jun '16

Hello, A new question has been asked in "MariaDB Community" by gaurav_sharma_g1. Please answer it at http://mariadb.com/kb/en/consistently-able-to-crash-mariadb-10025-and-1014-… as the person asking the question may not be subscribed to the mailing list. -------------------------------- We are able to consistently reproduce the crash of MariaDB with the following query: {{{ drop procedure if exists sp_getServerAndAgentHeartbeats; DELIMITER $$ CREATE PROCEDURE sp_getServerAndAgentHeartbeats() begin select allof.* from (select bothof.Name fullname, bothof.instanceid name, bothof.DMSUrl id, bothof.ts heartbeat, bothof.tz, bothof.local_time, bothof.pdfagentsetup, bothof.origin, bothof.dmsid, bothof.DealerID dealerid, bothof.server_group, bothof.hasNotes from ( select d.Name, dlrurl.DealerID, DMSUrl, DMSSuff, psh.* , (select OptionValue from DealerSetupOption where DealerID = d.ID and OptionKey = 'Dealer.timezone.id') tz , (select timestampadd(minute, (480 + OptionValue), current_timestamp()) from DealerSetupOption where DealerID = d.ID and OptionKey = 'Dealer.timezone' ) local_time , (select OptionValue from DealerSetupOption where DealerID = d.ID and OptionKey = 'pdfserver.settings') pdfagentsetup , "server" origin , (select DMSID from DealerDMS where DealerID = d.ID) dmsid , (select guid from dms_staging_db.server_groups sg where sg.ip = DMSUrl and find_in_set(DMSSuff, sg.suffix) > 0 limit 1) server_group , IF((select count(*) from dms_staging_db.server_notes sn where sn.uuid = server_group) >= 1,'true','false') hasNotes from (select DealerID, OptionValue as DMSUrl from DealerSetupOption dso1 where OptionKey = 'Dealer.Pdf.Source.Url') dlrurl inner join (select DealerID, OptionValue as DMSSuff from DealerSetupOption dso1 where OptionKey = 'ADP.SessionSuffix') dlrsuff on dlrsuff.DealerID = dlrurl.DealerID inner join Dealer d on dlrurl.DealerID = d.ID inner join PdfServerHeartbeat psh on psh.dealerurl = DMSUrl and psh.instanceid = concat_ws('', '_', DMSSuff) where coalesce(DMSUrl, '') !='' and coalesce(DMSSuff, '') !='' ) bothof union ( select d.Name fullname , aih.DealerIdentifier name , aih.DealerMarker id , aih.lastgotdata heartbeat , (select OptionValue from DealerSetupOption where DealerID = d.ID and OptionKey = 'Dealer.timezone.id') tz , (select timestampadd(minute, (480 + OptionValue), current_timestamp()) from DealerSetupOption where DealerID = d.ID and OptionKey = 'Dealer.timezone' ) local_time , (select OptionValue from DealerSetupOption where DealerID = d.ID and OptionKey = 'pdfserver.settings') pdfagentsetup , "agent" origin , (select DMSID from DealerDMS where DealerID = d.ID) dmsid , d.ID DealerID , (select guid from dms_staging_db.server_groups sg where sg.ip = aih.DealerMarker and find_in_set(aih.DealerIdentifier, sg.suffix) > 0 limit 1) server_group , IF((select count(*) from dms_staging_db.server_notes sn where sn.uuid = server_group) >= 1,'true','false') hasNotes from (select DealerMarker, DealerIdentifier, count(1) pcount, max(ts) lastgotdata from AgentInstanceHeartbeat where Instance != 'pdf_server' group by DealerMarker, DealerIdentifier ) aih inner join ( select dlrurl.DealerID, DMSUrl, DMSSuff from (select DealerID, OptionValue as DMSUrl from DealerSetupOption dso1 where OptionKey = 'Dealer.Pdf.Source.Url') dlrurl inner join (select DealerID, OptionValue as DMSSuff from DealerSetupOption dso1 where OptionKey = 'ADP.SessionSuffix') dlrsuff on dlrsuff.DealerID = dlrurl.DealerID where coalesce(DMSUrl, '') !='' and coalesce(DMSSuff, '') !='' ) dsodata on (aih.DealerMarker = dsodata.DMSUrl and aih.DealerIdentifier = dsodata.DMSSuff) inner join Dealer d on (dsodata.DealerID = d.ID) where d.isValid = 1 ) order by 1 asc) allof; end$$ }}} **Our developer wrote this query as a modification to an existing query and upon executing the above SP, we consistently see the following stack trace in the MariaDB logs:** {{{ Thread pointer: 0x0x7f301afa0008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2fdce88140 thread_stack 0x48400 /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0x7f48c34c197b] /usr/sbin/mysqld(handle_fatal_signal+0x475)[0x7f48c3020825] /lib64/libpthread.so.0(+0xf7e0)[0x7f48c26267e0] /usr/sbin/mysqld(_ZN17Dependency_marker11visit_fieldEP10Item_field+0x4b)[0x7f48c304765b] /usr/sbin/mysqld(_ZN10Item_field30enumerate_field_refs_processorEPh+0x12)[0x7f48c302d612] mysys/stacktrace.c:247(my_print_stacktrace)[0x7f48c3043c43] sql/signal_handler.cc:160(handle_fatal_signal)[0x7f48c30a2690] sql/item.cc:6707(Dependency_marker::visit_field(Item_field*))[0x7f48c303ddc4] sql/item_subselect.cc:329(Item_subselect::enumerate_field_refs_processor(unsigned char*))[0x7f48c303ef5e] sql/item.cc:5054(Item_field::fix_fields(THD*, Item**))[0x7f48c3040098] /usr/sbin/mysqld(_ZN9Item_func10fix_fieldsEP3THDPP4Item+0x1a7)[0x7f48c30732a7] sql/item_func.cc:208(Item_func::fix_fields(THD*, Item**))[0x7f48c2e713cb] sql/sql_base.cc:8629(setup_conds(THD*, TABLE_LIST*, List<TABLE_LIST>&, Item**))[0x7f48c2ef5160] sql/sql_select.cc:646(setup_without_group)[0x7f48c30a5b44] sql/item_subselect.cc:259(Item_subselect::fix_fields(THD*, Item**))[0x7f48c30a9a0d] sql/item_func.cc:208(Item_func::fix_fields(THD*, Item**))[0x7f48c30732a7] sql/item_func.cc:208(Item_func::fix_fields(THD*, Item**))[0x7f48c30732a7] sql/sql_base.cc:7901(setup_fields(THD*, Item**, List<Item>&, enum_mark_columns, List<Item>*, bool))[0x7f48c2e70c13] sql/sql_select.cc:791(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x7f48c2ef50a2] sql/sql_union.cc:454(st_select_lex_unit::prepare(THD*, select_result*, unsigned long))[0x7f48c2f400ec] sql/sql_derived.cc:671(mysql_derived_prepare(THD*, LEX*, TABLE_LIST*))[0x7f48c2e8e570] sql/sql_derived.cc:195(mysql_handle_single_derived(LEX*, TABLE_LIST*, unsigned int))[0x7f48c2e8f327] sql/sql_lex.cc:3820(st_select_lex::handle_derived(LEX*, unsigned int))[0x7f48c2ea46b7] sql/table.cc:7126(TABLE_LIST::handle_derived(LEX*, unsigned int))[0x7f48c2f55767] sql/sql_lex.cc:3820(st_select_lex::handle_derived(LEX*, unsigned int))[0x7f48c2ea46b7] sql/sql_select.cc:711(JOIN::prepare(Item***, TABLE_LIST*, unsigned int, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x7f48c2ef4e31] sql/sql_select.cc:3429(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x7f48c2f01aed] sql/sql_select.cc:384(handle_select(THD*, LEX*, select_result*, unsigned long))[0x7f48c2f04bfd] /usr/sbin/mysqld(+0x451102)[0x7f48c2ea7102] sql/sql_parse.cc:5895(execute_sqlcom_select)[0x7f48c2eb33bf] sql/sp_head.cc:3215(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x7f48c311c9be] sql/sp_head.cc:2979(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x7f48c312243e] sql/sp_head.cc:3129(sp_instr_stmt::execute(THD*, unsigned int*))[0x7f48c312293d] sql/sp_head.cc:1319(sp_head::execute(THD*, bool))[0x7f48c311f91c] sql/sp_head.cc:2104(sp_head::execute_procedure(THD*, List<Item>*))[0x7f48c3120cc6] sql/sql_parse.cc:2405(do_execute_sp)[0x7f48c2eb3b30] sql/sp_head.cc:3215(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x7f48c311c9be] sql/sp_head.cc:2979(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x7f48c312243e] sql/sp_head.cc:3129(sp_instr_stmt::execute(THD*, unsigned int*))[0x7f48c312293d] sql/sp_head.cc:1319(sp_head::execute(THD*, bool))[0x7f48c311f91c] sql/sp_head.cc:2104(sp_head::execute_procedure(THD*, List<Item>*))[0x7f48c3120cc6] sql/sql_parse.cc:2405(do_execute_sp)[0x7f48c2eb3b30] sql/sql_parse.cc:7314(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x7f48c2eb68fd] sql/sql_parse.cc:1486(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x7f48c2eb955b] sql/sql_parse.cc:1109(do_command(THD*))[0x7f48c2eb9b0b] sql/sql_connect.cc:1350(do_handle_one_connection(THD*))[0x7f48c2f7569f] sql/sql_connect.cc:1264(handle_one_connection)[0x7f48c2f757f7] /lib64/libpthread.so.0(+0x7aa1)[0x7f48c261eaa1] /lib64/libc.so.6(clone+0x6d)[0x7f48c0b03aad] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f302e2a8020): is an invalid pointer Connection ID (thread ID): 264 Status: NOT_KILLED }}} After this MariaDB crashes and restarts. **For now we have reverted the changes back to the old code which is shown below:** {{{ drop procedure if exists sp_getServerAndAgentHeartbeats; DELIMITER $$ CREATE PROCEDURE sp_getServerAndAgentHeartbeats() begin select allof.*, IF((select count(*) from dms_staging_db.server_notes sn where sn.uuid = concat(allof.dealerid,'-notes')) >= 1,'true','false') hasNotes from (select bothof.Name fullname, bothof.instanceid name, bothof.DMSUrl id, bothof.ts heartbeat, bothof.tz, bothof.local_time, bothof.pdfagentsetup, bothof.origin, bothof.dmsid, bothof.DealerID dealerid from ( ( select d.Name, dlrurl.DealerID, DMSUrl, DMSSuff, psh.* , (select OptionValue from DealerSetupOption where DealerID = d.ID and OptionKey = 'Dealer.timezone.id') tz , (select timestampadd(minute, (480 + OptionValue), current_timestamp()) from DealerSetupOption where DealerID = d.ID and OptionKey = 'Dealer.timezone' ) local_time , (select OptionValue from DealerSetupOption where DealerID = d.ID and OptionKey = 'pdfserver.settings') pdfagentsetup , "server" origin , (select DMSID from DealerDMS where DealerID = d.ID) dmsid from (select DealerID, OptionValue as DMSUrl from DealerSetupOption dso1 where OptionKey = 'Dealer.Pdf.Source.Url') dlrurl inner join (select DealerID, OptionValue as DMSSuff from DealerSetupOption dso1 where OptionKey = 'ADP.SessionSuffix') dlrsuff on dlrsuff.DealerID = dlrurl.DealerID inner join Dealer d on dlrurl.DealerID = d.ID inner join PdfServerHeartbeat psh on psh.dealerurl = DMSUrl and psh.instanceid = concat_ws('', '_', DMSSuff) where coalesce(DMSUrl, '') !='' and coalesce(DMSSuff, '') !='' ) union ( select d.Name, data.* , (select OptionValue from DealerSetupOption where DealerID = d.ID and OptionKey = 'Dealer.timezone.id') tz , (select timestampadd(minute, (480 + OptionValue), current_timestamp()) from DealerSetupOption where DealerID = d.ID and OptionKey = 'Dealer.timezone' ) local_time , (select OptionValue from DealerSetupOption where DealerID = d.ID and OptionKey = 'pdfserver.settings') pdfagentsetup , "server" origin , (select DMSID from DealerDMS where DealerID = d.ID) dmsid from ( select replace(dealerurl, 'dealer-', '') DealerID, dealerurl DMSUrl, null DMSSuff, psh1.* from PdfServerHeartbeat psh1 where dealerurl like 'dealer-%' ) data inner join Dealer d on data.DealerID = d.ID ) ) bothof union ( select d.Name fullname , aih.DealerIdentifier name , aih.DealerMarker id , aih.lastgotdata heartbeat , (select OptionValue from DealerSetupOption where DealerID = d.ID and OptionKey = 'Dealer.timezone.id') tz , (select timestampadd(minute, (480 + OptionValue), current_timestamp()) from DealerSetupOption where DealerID = d.ID and OptionKey = 'Dealer.timezone' ) local_time , (select OptionValue from DealerSetupOption where DealerID = d.ID and OptionKey = 'pdfserver.settings') pdfagentsetup , "agent" origin , (select DMSID from DealerDMS where DealerID = d.ID) dmsid , d.ID DealerID from (select DealerMarker, DealerIdentifier, count(1) pcount, max(ts) lastgotdata from AgentInstanceHeartbeat where Instance != 'pdf_server' group by DealerMarker, DealerIdentifier ) aih inner join ( select dlrurl.DealerID, DMSUrl, DMSSuff from (select DealerID, OptionValue as DMSUrl from DealerSetupOption dso1 where OptionKey = 'Dealer.Pdf.Source.Url') dlrurl inner join (select DealerID, OptionValue as DMSSuff from DealerSetupOption dso1 where OptionKey = 'ADP.SessionSuffix') dlrsuff on dlrsuff.DealerID = dlrurl.DealerID where coalesce(DMSUrl, '') !='' and coalesce(DMSSuff, '') !='' ) dsodata on (aih.DealerMarker = dsodata.DMSUrl and aih.DealerIdentifier = dsodata.DMSSuff) inner join Dealer d on (dsodata.DealerID = d.ID) where d.isValid = 1 ) order by 1 asc) allof; end$$ }}} -------------------------------- To view or answer this question please visit: http://mariadb.com/kb/en/consistently-able-to-crash-mariadb-10025-and-1014-…

1 0

[Maria-discuss] Fwd: Undelivered Mail Returned to Sender
by Reindl Harald 02 Jun '16

02 Jun '16

what is that for a bullshit? MariaDB newsletters not use proper mail-headers to not trigger autoresponders? in 2016? seriously? with a invalid sender? seriously? in 2016? -------- Weitergeleitete Nachricht -------- Betreff: Undelivered Mail Returned to Sender Datum: Wed, 1 Jun 2016 19:53:55 +0200 (CEST) Von: Mail Delivery System <postmaster(a)thelounge.net> An: h.reindl(a)thelounge.net This is the mail system at host mail.thelounge.net I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below For assistance, forward this report to <postmaster(a)thelounge.net> You can delete your own text from the attached returned message <no-reply(a)mariaDB.com>: host aspmx.l.google.com[173.194.65.27] said: 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 https://support.google.com/mail/answer/6596 d83si44788327wmf.53 - gsmtp (in reply to RCPT TO command)

1 0

[Maria-discuss] Backing up tables with data at rest encryption
by Lee Bennett 01 Jun '16

01 Jun '16

Hi We are in the process of setting up data at rest encryption encryption with MariaDB 10.1.13 so we can encrypt specific table. This works without any problem however when I try and backup with Percona Xtrabackup it fails saying table appears to be corrupted: 60601 14:53:46 [01] ...done 160601 14:53:46 [01] Streaming ./mytest/customers_big.ibd [01] xtrabackup: Database page corruption detected at page 1, retrying... [01] xtrabackup: Database page corruption detected at page 1, retrying... 160601 14:53:46 >> log scanned up to (3146117051) [01] xtrabackup: Database page corruption detected at page 1, retrying... [01] xtrabackup: Database page corruption detected at page 1, retrying... [01] xtrabackup: Database page corruption detected at page 1, retrying... [01] xtrabackup: Database page corruption detected at page 1, retrying... [01] xtrabackup: Database page corruption detected at page 1, retrying... [01] xtrabackup: Database page corruption detected at page 1, retrying... [01] xtrabackup: Database page corruption detected at page 1, retrying... [01] xtrabackup: Error: failed to read page after 10 retries. File ./mytest/customers_big.ibd seems to be corrupted. [01] xtrabackup: Error: xtrabackup_copy_datafile() failed. [01] xtrabackup: Error: failed to copy datafile. This says to me that Xtrabackup isn’t compatible with MariaDB encryption however the documents kind of give the impression it is: https://mariadb.com/kb/en/mariadb/data-at-rest-encryption/ <https://mariadb.com/kb/en/mariadb/data-at-rest-encryption/> Regards Lee

3 3

[Maria-discuss] Long email about a replication issue
by Rhys.Campbell＠swisscom.com 01 Jun '16

01 Jun '16

Hi All, This weekend I had to repair a replication problem on one of our clusters. I've attempted to get to the root cause but not sure where I can go from here. Replication broke on Friday night because of foreign key errors. I initially retrieved the correct row, restarted replication, which then broke again due to a different tables FK error. I fixed this and then started replication again. All was well for an hour and then it broke again with a similar issue. We called it a night and decided to address on Saturday AM as there was no immediate production impact. The next day I discovered that records were missing from a bunch of tables so I recovered the cluster using a backup. I initially recovered the slaves attached to M2 from a backup but using the binlogs from M2. This caused the slaves to break. So instead I then recovered from the binlog on M1, before pointing them back to M2. This worked fine and everything in the cluster is currently well. Clearly something is wrong with the binlogs of M2. This is the first time I have seen this issue on the cluster that has been running fine for the last year. There have been no recent changes in the version of MariaDB or the applications using the cluster. We are running version 10.0.23-MariaDB-log. We are running replication with GTIDs, semi-sync repl between masters, and we run regular pt-table-checksum jobs. slave_sql_verify_checksum is on but master_sql_verify_checksum is off. The setup is M1<->M2 (with attached slaves). M1 is the active master receiving all writes. Access is controlled through an F5 and I don't think any errant transactions have occurred on the inactive master (M2). I've checked this by grepping the binlogs for the M2 server_id. The initial associated record that broke replication was attached to a "user" table record. This user was created on Friday at 16:21PM. Replication broke around 11:30PM that night. The user record had a GTID of GTID 0-1-36823254 (recovered from M1) I've looked into the appropriate binlog from M2... The head mysqlbinlog mysql-bin.002133 | grep GTID | head #160521 5:02:32 server id 1 end_log_pos 392 CRC32 0x97a118a8 GTID 0-1-36613622 #160521 5:02:32 server id 1 end_log_pos 768 CRC32 0x1859f6fb GTID 0-1-36613623 The tail... mysqlbinlog mysql-bin.002133 | grep GTID | tail #160528 11:58:04 server id 1 end_log_pos 103889368 CRC32 0x66e7c623 GTID 0-1-36851211 #160528 11:58:04 server id 1 end_log_pos 103971892 CRC32 0xe55c4842 GTID 0-1-36851212 If I grep for the specific GTID on M2 I get nothing... mysqlbinlog mysql-bin.002133 | grep "GTID 0-1-36823254"; <- Note the GTID is between the head & tail range. If I grep for this record by email address I also get nothing. So I must conclude this record (and a bunch of others), never got to master 2. We have pretty extensive monitoring of these servers and we should have been alerted to replication lag, locking etc but we got nothing until replication broke due to the FK errors. You would expect replication to break here because of a gap in the GTIDs. This did not happen and I'm almost certain that GTID replication could not have been deactivated and the positions messed around with. I'm unsure of where to go now. Any ideas? Any thoughts are appreciated. Regards, Rhys

2 2