@joocelyn Here is the my.cfg file @justin I will check that post. Thanks [mysqld] # ------------------------------------------------------------------------------- # ++++ General # ------------------------------------------------------------------------------- datadir = /var/lib/mysql/data pid-file = /var/run/mysqld/mysqld.pid socket = /var/lib/mysql/mysql.sock tmpdir = /var/lib/mysql/tmp port = 3306 # ------------------------------------------------------------------------------- # ++++ Logging # ------------------------------------------------------------------------------- log-error = /var/lib/mysql/log/mysqld-error.log long_query_time = 10 slow-query-log-file = /var/lib/mysql/log/mysqld-queries_slow.log log-slave-updates log-bin = /var/lib/mysql/log/bin/slave-bin log-warnings = 1 max_relay_log_size = 200M relay_log_space_limit = 25000M # ------------------------------------------------------------------------------- # ++++ Network # ------------------------------------------------------------------------------- max_connections = 3000 max_connect_errors = 1000 wait_timeout = 120 connect_timeout = 30 interactive_timeout = 3600 slave_net_timeout = 120 back_log = 50 max_allowed_packet = 128M # ------------------------------------------------------------------------------- # ++++ Misc # ------------------------------------------------------------------------------- # Text Searchs ft_min_word_len = 2 plugin-load = ha_tokudb # ------------------------------------------------------------------------------- # ++++ Threads # ------------------------------------------------------------------------------- thread_concurrency = 8 thread_cache = 64 # ------------------------------------------------------------------------------- # ++++ Memory # ------------------------------------------------------------------------------- # Tables table_cache = 4096 tmp_table_size = 256M # Memory per Thread sort_buffer_size = 8M read_buffer_size = 4M read_rnd_buffer_size = 16M # Query Cache query_cache_type = 1 query_cache_limit = 2M query_cache_size = 64M # ------------------------------------------------------------------------------- # ++++ MyISAM Parameters # ------------------------------------------------------------------------------- key_buffer_size = 1024M myisam_sort_buffer_size = 64M myisam_recover = FORCE,BACKUP # ------------------------------------------------------------------------------- # ++++ InnoDB Parameters # ------------------------------------------------------------------------------- # General innodb_data_home_dir = /var/lib/mysql/innodb innodb_log_group_home_dir = /var/lib/mysql/innodblogs innodb_file_per_table innodb_data_file_path = ibdata1:100M:autoextend innodb_status_file = ib_status innodb_autoextend_increment = 10M innodb_support_xa = 0 innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_flush_log_at_trx_commit = 2 # Memory innodb_buffer_pool_size = 6G innodb_additional_mem_pool_size = 8M innodb_open_files = 512 # Logging innodb_log_buffer_size = 8M innodb_log_file_size = 256M innodb_log_files_in_group = 2 # ------------------------------------------------------------------------------- # ++++ Replication : SLAVE Profile # ------------------------------------------------------------------------------- #skip-slave-start server-id = 124388 relay-log = /var/lib/mysql/log/replication/slave-bin relay-log-info-file = /var/lib/mysql/log/replication/ slave-log.info master-info-file = /var/lib/mysql/log/replication/ master-log.info max_binlog_size = 20971520 read-only = 1 [mysqldump] quick max_allowed_packet = 128M [isamchk] key_buffer = 256M sort_buffer_size = 256M read_buffer = 2M write_buffer = 2M [myisamchk] key_buffer = 256M sort_buffer_size = 256M read_buffer = 2M write_buffer = 2M [mysqlhotcopy] interactive-timeout On Tue, Mar 3, 2015 at 12:56 PM, Justin Swanhart wrote:

Hi,

You might try to find the source of the termination with this:

http://www.percona.com/blog/2014/07/18/systemtap-solves-phantom-mysqld-sigte...

On Tue, Mar 3, 2015 at 8:45 AM, jocelyn fournier < jocelyn.fournier@gmail.com> wrote:

...

Hi,

Could you send your my.cnf ?

Thanks, Jocelyn

Le 03/03/2015 16:24, Gabriel Sosa a écrit :

Hello,

I've been a proudly user of mariadb 5.5.x for a long time now but given the nice feature setI decided to give mariadb 10.x a try.

I took one of our current slaves running mariadb 5.5.x (on centos 6.5) and followed the upgrade steps using yum and ran *mysql_upgrade*. The upgrade ran without any trouble...then I moved a huge table (about 1B records right now) from InnoDB to TokuDB.

Now, every couple of hours I find that the replication is far behind the master and the reason is because this server keeps checking tables marked as crashed....

I can't seems to find any indicator of OOM killer in the system logs NOR anything related to that in the mysql log, the only clue I have is:

--------------- *150303 10:08:57 mysqld_safe Number of processes running now: 0* *150303 10:08:57 mysqld_safe mysqld restarted* 150303 10:08:58 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release. 150303 10:08:59 [Warning] option 'innodb-status-file': boolean value 'ib_status' wasn't recognized. Set to OFF. 150303 10:08:59 [Warning] option 'innodb-autoextend-increment': unsigned value 10485760 adjusted to 1000 150303 10:08:59 [Note] InnoDB: Using mutexes to ref count buffer pool pages 150303 10:08:59 [Note] InnoDB: The InnoDB memory heap is disabled 150303 10:08:59 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 150303 10:08:59 [Note] InnoDB: Memory barrier is not used 150303 10:08:59 [Note] InnoDB: Compressed tables use zlib 1.2.3 150303 10:08:59 [Note] InnoDB: Using Linux native AIO 150303 10:08:59 [Note] InnoDB: Using CPU crc32 instructions 150303 10:08:59 [Note] InnoDB: Initializing buffer pool, size = 6.0G 150303 10:08:59 [Note] InnoDB: Completed initialization of buffer pool 150303 10:08:59 [Note] InnoDB: Highest supported file format is Barracuda. 150303 10:08:59 [Note] InnoDB: Log scan progressed past the checkpoint lsn 14178608217047 150303 10:08:59 [Note] InnoDB: Database was not shutdown normally! 150303 10:08:59 [Note] InnoDB: Starting crash recovery. 150303 10:08:59 [Note] InnoDB: Reading tablespace information from the .ibd files... 150303 10:09:05 [Note] InnoDB: Restoring possible half-written data pages 150303 10:09:05 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 14178613459456 InnoDB: Doing recovery: scanned up to log sequence number 14178618702336 InnoDB: Doing recovery: scanned up to log sequence number 14178623945216 InnoDB: Doing recovery: scanned up to log sequence number 14178629188096 InnoDB: Doing recovery: scanned up to log sequence number 14178634430976 InnoDB: Doing recovery: scanned up to log sequence number 14178639673856 InnoDB: Doing recovery: scanned up to log sequence number 14178644916736 ...... ...... ...... ...... 150303 10:09:14 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 16433109, file name slave-bin.852141 InnoDB: Last MySQL binlog file position 0 16784214, file name /var/lib/mysql/log/bin/slave-bin.005211 150303 10:09:34 [Note] InnoDB: 128 rollback segment(s) are active. 150303 10:09:34 [Note] InnoDB: Waiting for purge to start 150303 10:09:34 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 14178834821658 Tue Mar 3 10:09:36 2015 TokuFT recovery starting in env /var/lib/mysql/data/ Tue Mar 3 10:09:36 2015 TokuFT recovery scanning backward from 1330297 Tue Mar 3 10:09:36 2015 TokuFT recovery bw_end_checkpoint at 1330297 timestamp 1425395277901416 xid 1330281 (bw_newer) Tue Mar 3 10:09:36 2015 TokuFT recovery bw_begin_checkpoint at 1330281 timestamp 1425395273487081 (bw_between) Tue Mar 3 10:09:36 2015 TokuFT recovery turning around at begin checkpoint 1330281 time 4414335 Tue Mar 3 10:09:36 2015 TokuFT recovery starts scanning forward to 1330297 from 1330281 left 16 (fw_between) Tue Mar 3 10:09:36 2015 TokuFT recovery closing 14 dictionaries Tue Mar 3 10:09:36 2015 TokuFT recovery making a checkpoint Tue Mar 3 10:09:36 2015 TokuFT recovery done 150303 10:09:36 [Note] Recovering after a crash using /var/lib/mysql/log/bin/slave-bin

---------------

The odd thing is that the 5.5.x version was working just fine (since about a year) in the same hardware...nothing changed in that front.

Any clue?

Thank you in advance

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

-- Gabriel Sosa Sometimes the questions are complicated and the answers are simple. -- Dr. Seuss

Hi, Presumably, if the server is using too much ram it will a) crash and leave a stack trace (not happening) or b) invoke the OOMPK which leaves a trace in syslog, which also, presumably, is not happening, since she can find no trace of OOMPK activity. So memory is a good guess (output of free -m would be useful) I doubt it is memory related do to lack of evidence for it. --Justin On Tue, Mar 3, 2015 at 9:58 AM, jocelyn fournier

wrote:

Hi,

No variables to tweak the tokudb memory allocation ? By default tokudb_cache_size takes half of the total system memory. With 6G allocated to InnoDB buffer pool, and 1G to the MyISAM key buffer, are you sure you have enough memory for TokuDB ?

Jocelyn

Le 03/03/2015 17:37, Gabriel Sosa a écrit :

@joocelyn Here is the my.cfg file

@justin I will check that post. Thanks

[mysqld]

# ------------------------------------------------------------------------------- # ++++ General # ------------------------------------------------------------------------------- datadir = /var/lib/mysql/data pid-file = /var/run/mysqld/mysqld.pid socket = /var/lib/mysql/mysql.sock tmpdir = /var/lib/mysql/tmp port = 3306

# ------------------------------------------------------------------------------- # ++++ Logging # ------------------------------------------------------------------------------- log-error = /var/lib/mysql/log/mysqld-error.log long_query_time = 10 slow-query-log-file = /var/lib/mysql/log/mysqld-queries_slow.log log-slave-updates log-bin = /var/lib/mysql/log/bin/slave-bin log-warnings = 1

max_relay_log_size = 200M relay_log_space_limit = 25000M

# ------------------------------------------------------------------------------- # ++++ Network # ------------------------------------------------------------------------------- max_connections = 3000 max_connect_errors = 1000 wait_timeout = 120 connect_timeout = 30 interactive_timeout = 3600 slave_net_timeout = 120 back_log = 50 max_allowed_packet = 128M

# ------------------------------------------------------------------------------- # ++++ Misc # ------------------------------------------------------------------------------- # Text Searchs ft_min_word_len = 2 plugin-load = ha_tokudb

# ------------------------------------------------------------------------------- # ++++ Threads # ------------------------------------------------------------------------------- thread_concurrency = 8 thread_cache = 64

# ------------------------------------------------------------------------------- # ++++ Memory # ------------------------------------------------------------------------------- # Tables table_cache = 4096 tmp_table_size = 256M # Memory per Thread sort_buffer_size = 8M read_buffer_size = 4M read_rnd_buffer_size = 16M # Query Cache query_cache_type = 1 query_cache_limit = 2M query_cache_size = 64M

# ------------------------------------------------------------------------------- # ++++ MyISAM Parameters # ------------------------------------------------------------------------------- key_buffer_size = 1024M myisam_sort_buffer_size = 64M myisam_recover = FORCE,BACKUP

# ------------------------------------------------------------------------------- # ++++ InnoDB Parameters # ------------------------------------------------------------------------------- # General innodb_data_home_dir = /var/lib/mysql/innodb innodb_log_group_home_dir = /var/lib/mysql/innodblogs innodb_file_per_table innodb_data_file_path = ibdata1:100M:autoextend innodb_status_file = ib_status innodb_autoextend_increment = 10M innodb_support_xa = 0 innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_flush_log_at_trx_commit = 2 # Memory innodb_buffer_pool_size = 6G innodb_additional_mem_pool_size = 8M innodb_open_files = 512 # Logging innodb_log_buffer_size = 8M innodb_log_file_size = 256M innodb_log_files_in_group = 2

# ------------------------------------------------------------------------------- # ++++ Replication : SLAVE Profile # ------------------------------------------------------------------------------- #skip-slave-start server-id = 124388 relay-log = /var/lib/mysql/log/replication/slave-bin relay-log-info-file = /var/lib/mysql/log/replication/ slave-log.info master-info-file = /var/lib/mysql/log/replication/ master-log.info max_binlog_size = 20971520

read-only = 1

[mysqldump] quick max_allowed_packet = 128M

[isamchk] key_buffer = 256M sort_buffer_size = 256M read_buffer = 2M write_buffer = 2M

[myisamchk] key_buffer = 256M sort_buffer_size = 256M read_buffer = 2M write_buffer = 2M

[mysqlhotcopy] interactive-timeout

On Tue, Mar 3, 2015 at 12:56 PM, Justin Swanhart wrote:

...

Hi,

You might try to find the source of the termination with this:

http://www.percona.com/blog/2014/07/18/systemtap-solves-phantom-mysqld-sigte...

On Tue, Mar 3, 2015 at 8:45 AM, jocelyn fournier < jocelyn.fournier@gmail.com> wrote:

...

Hi,

Could you send your my.cnf ?

Thanks, Jocelyn

Le 03/03/2015 16:24, Gabriel Sosa a écrit :

Hello,

I've been a proudly user of mariadb 5.5.x for a long time now but given the nice feature setI decided to give mariadb 10.x a try.

I took one of our current slaves running mariadb 5.5.x (on centos 6.5) and followed the upgrade steps using yum and ran *mysql_upgrade*. The upgrade ran without any trouble...then I moved a huge table (about 1B records right now) from InnoDB to TokuDB.

Now, every couple of hours I find that the replication is far behind the master and the reason is because this server keeps checking tables marked as crashed....

I can't seems to find any indicator of OOM killer in the system logs NOR anything related to that in the mysql log, the only clue I have is:

--------------- *150303 10:08:57 mysqld_safe Number of processes running now: 0* *150303 10:08:57 mysqld_safe mysqld restarted* 150303 10:08:58 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release. 150303 10:08:59 [Warning] option 'innodb-status-file': boolean value 'ib_status' wasn't recognized. Set to OFF. 150303 10:08:59 [Warning] option 'innodb-autoextend-increment': unsigned value 10485760 adjusted to 1000 150303 10:08:59 [Note] InnoDB: Using mutexes to ref count buffer pool pages 150303 10:08:59 [Note] InnoDB: The InnoDB memory heap is disabled 150303 10:08:59 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 150303 10:08:59 [Note] InnoDB: Memory barrier is not used 150303 10:08:59 [Note] InnoDB: Compressed tables use zlib 1.2.3 150303 10:08:59 [Note] InnoDB: Using Linux native AIO 150303 10:08:59 [Note] InnoDB: Using CPU crc32 instructions 150303 10:08:59 [Note] InnoDB: Initializing buffer pool, size = 6.0G 150303 10:08:59 [Note] InnoDB: Completed initialization of buffer pool 150303 10:08:59 [Note] InnoDB: Highest supported file format is Barracuda. 150303 10:08:59 [Note] InnoDB: Log scan progressed past the checkpoint lsn 14178608217047 150303 10:08:59 [Note] InnoDB: Database was not shutdown normally! 150303 10:08:59 [Note] InnoDB: Starting crash recovery. 150303 10:08:59 [Note] InnoDB: Reading tablespace information from the .ibd files... 150303 10:09:05 [Note] InnoDB: Restoring possible half-written data pages 150303 10:09:05 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 14178613459456 InnoDB: Doing recovery: scanned up to log sequence number 14178618702336 InnoDB: Doing recovery: scanned up to log sequence number 14178623945216 InnoDB: Doing recovery: scanned up to log sequence number 14178629188096 InnoDB: Doing recovery: scanned up to log sequence number 14178634430976 InnoDB: Doing recovery: scanned up to log sequence number 14178639673856 InnoDB: Doing recovery: scanned up to log sequence number 14178644916736 ...... ...... ...... ...... 150303 10:09:14 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 16433109, file name slave-bin.852141 InnoDB: Last MySQL binlog file position 0 16784214, file name /var/lib/mysql/log/bin/slave-bin.005211 150303 10:09:34 [Note] InnoDB: 128 rollback segment(s) are active. 150303 10:09:34 [Note] InnoDB: Waiting for purge to start 150303 10:09:34 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 14178834821658 Tue Mar 3 10:09:36 2015 TokuFT recovery starting in env /var/lib/mysql/data/ Tue Mar 3 10:09:36 2015 TokuFT recovery scanning backward from 1330297 Tue Mar 3 10:09:36 2015 TokuFT recovery bw_end_checkpoint at 1330297 timestamp 1425395277901416 xid 1330281 (bw_newer) Tue Mar 3 10:09:36 2015 TokuFT recovery bw_begin_checkpoint at 1330281 timestamp 1425395273487081 (bw_between) Tue Mar 3 10:09:36 2015 TokuFT recovery turning around at begin checkpoint 1330281 time 4414335 Tue Mar 3 10:09:36 2015 TokuFT recovery starts scanning forward to 1330297 from 1330281 left 16 (fw_between) Tue Mar 3 10:09:36 2015 TokuFT recovery closing 14 dictionaries Tue Mar 3 10:09:36 2015 TokuFT recovery making a checkpoint Tue Mar 3 10:09:36 2015 TokuFT recovery done 150303 10:09:36 [Note] Recovering after a crash using /var/lib/mysql/log/bin/slave-bin

---------------

The odd thing is that the 5.5.x version was working just fine (since about a year) in the same hardware...nothing changed in that front.

Any clue?

Thank you in advance

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

-- Gabriel Sosa Sometimes the questions are complicated and the answers are simple. -- Dr. Seuss

Well, you can actually choose to either lower the tokudb_cache_size value to avoid to eat all the memory, or convert your other innodb tables to tokudb, if it makes sense :) (you can have the master on InnoDB, and start converting the tables to TokuDB on the slave to check all is working properly) Jocelyn Le 03/03/2015 22:26, Gabriel Sosa a écrit :

using systemtap I got

[Tue Mar 3 18:48:46 2015] SIGKILL was sent to ps (pid:9763) by processes uid:0 [Tue Mar 3 18:49:46 2015] SIGKILL was sent to ps (pid:9824) by processes uid:0 *[Tue Mar 3 18:50:44 2015] SIGKILL was sent to mysqld (pid:7597) by mysqld uid:498* [Tue Mar 3 18:50:46 2015] SIGKILL was sent to ps (pid:9890) by processes uid:0 [Tue Mar 3 18:51:46 2015] SIGKILL was sent to ps (pid:9997) by processes uid:0 [Tue Mar 3 18:52:35 2015] SIGKILL was sent to cdm (pid:10045) by cdm uid:0

I saw this several times but I wasn't able to find (in this case) which process has the PID 498. @jocelyn, you are right...to tweaks at all for tokudb (my bad here), although I wasn't expecting such /naive/ approach, I mean, I could have not enable the tokudb plugin and all other tables using the other engines would work just fine (maybe just slow though)

free -m returns about 47M free...which isn't much...

the issue is that if I need to provide tokudb more RAM I will need to take if from the InnoDB buffer pool, which we use a lot...the main reason to use tokudb was to "compress" some huge tables that are not transactional nor readed all the time.

On Tue, Mar 3, 2015 at 4:06 PM, Justin Swanhart mailto:greenlion@gmail.com> wrote:

Hi,

Presumably, if the server is using too much ram it will a) crash and leave a stack trace (not happening) or b) invoke the OOMPK which leaves a trace in syslog, which also, presumably, is not happening, since she can find no trace of OOMPK activity.

So memory is a good guess (output of free -m would be useful) I doubt it is memory related do to lack of evidence for it.

--Justin

On Tue, Mar 3, 2015 at 9:58 AM, jocelyn fournier mailto:jocelyn.fournier@gmail.com> wrote:

Hi,

No variables to tweak the tokudb memory allocation ? By default tokudb_cache_size takes half of the total system memory. With 6G allocated to InnoDB buffer pool, and 1G to the MyISAM key buffer, are you sure you have enough memory for TokuDB ?

Jocelyn

Le 03/03/2015 17:37, Gabriel Sosa a écrit :

...

@joocelyn Here is the my.cfg file

@justin I will check that post. Thanks

[mysqld]

# ------------------------------------------------------------------------------- # ++++ General # ------------------------------------------------------------------------------- datadir = /var/lib/mysql/data pid-file = /var/run/mysqld/mysqld.pid socket = /var/lib/mysql/mysql.sock tmpdir = /var/lib/mysql/tmp port = 3306

# ------------------------------------------------------------------------------- # ++++ Logging # ------------------------------------------------------------------------------- log-error = /var/lib/mysql/log/mysqld-error.log long_query_time = 10 slow-query-log-file = /var/lib/mysql/log/mysqld-queries_slow.log log-slave-updates log-bin = /var/lib/mysql/log/bin/slave-bin log-warnings = 1

max_relay_log_size = 200M relay_log_space_limit = 25000M

# ------------------------------------------------------------------------------- # ++++ Network # ------------------------------------------------------------------------------- max_connections = 3000 max_connect_errors = 1000 wait_timeout = 120 connect_timeout = 30 interactive_timeout = 3600 slave_net_timeout = 120 back_log = 50 max_allowed_packet = 128M

# ------------------------------------------------------------------------------- # ++++ Misc # ------------------------------------------------------------------------------- # Text Searchs ft_min_word_len = 2 plugin-load = ha_tokudb

# ------------------------------------------------------------------------------- # ++++ Threads # ------------------------------------------------------------------------------- thread_concurrency = 8 thread_cache = 64

# ------------------------------------------------------------------------------- # ++++ Memory # ------------------------------------------------------------------------------- # Tables table_cache = 4096 tmp_table_size = 256M # Memory per Thread sort_buffer_size = 8M read_buffer_size = 4M read_rnd_buffer_size = 16M # Query Cache query_cache_type = 1 query_cache_limit = 2M query_cache_size = 64M

# ------------------------------------------------------------------------------- # ++++ MyISAM Parameters # ------------------------------------------------------------------------------- key_buffer_size = 1024M myisam_sort_buffer_size = 64M myisam_recover = FORCE,BACKUP

# ------------------------------------------------------------------------------- # ++++ InnoDB Parameters # ------------------------------------------------------------------------------- # General innodb_data_home_dir = /var/lib/mysql/innodb innodb_log_group_home_dir = /var/lib/mysql/innodblogs innodb_file_per_table innodb_data_file_path = ibdata1:100M:autoextend innodb_status_file = ib_status innodb_autoextend_increment = 10M innodb_support_xa = 0 innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_flush_log_at_trx_commit = 2 # Memory innodb_buffer_pool_size = 6G innodb_additional_mem_pool_size = 8M innodb_open_files = 512 # Logging innodb_log_buffer_size = 8M innodb_log_file_size = 256M innodb_log_files_in_group = 2

# ------------------------------------------------------------------------------- # ++++ Replication : SLAVE Profile # ------------------------------------------------------------------------------- #skip-slave-start server-id = 124388 relay-log = /var/lib/mysql/log/replication/slave-bin relay-log-info-file = /var/lib/mysql/log/replication/slave-log.info http://slave-log.info master-info-file = /var/lib/mysql/log/replication/master-log.info http://master-log.info max_binlog_size = 20971520

read-only = 1

[mysqldump] quick max_allowed_packet = 128M

[isamchk] key_buffer = 256M sort_buffer_size = 256M read_buffer = 2M write_buffer = 2M

[myisamchk] key_buffer = 256M sort_buffer_size = 256M read_buffer = 2M write_buffer = 2M

[mysqlhotcopy] interactive-timeout

On Tue, Mar 3, 2015 at 12:56 PM, Justin Swanhart mailto:greenlion@gmail.com> wrote:

Hi,

You might try to find the source of the termination with this: http://www.percona.com/blog/2014/07/18/systemtap-solves-phantom-mysqld-sigte...

On Tue, Mar 3, 2015 at 8:45 AM, jocelyn fournier mailto:jocelyn.fournier@gmail.com> wrote:

Hi,

Could you send your my.cnf ?

Thanks, Jocelyn

Le 03/03/2015 16:24, Gabriel Sosa a écrit :

...

Hello,

I've been a proudly user of mariadb 5.5.x for a long time now but given the nice feature setI decided to give mariadb 10.x a try.

I took one of our current slaves running mariadb 5.5.x (on centos 6.5) and followed the upgrade steps using yum and ran *mysql_upgrade*. The upgrade ran without any trouble...then I moved a huge table (about 1B records right now) from InnoDB to TokuDB.

Now, every couple of hours I find that the replication is far behind the master and the reason is because this server keeps checking tables marked as crashed....

I can't seems to find any indicator of OOM killer in the system logs NOR anything related to that in the mysql log, the only clue I have is:

--------------- *150303 10:08:57 mysqld_safe Number of processes running now: 0* *150303 10:08:57 mysqld_safe mysqld restarted* 150303 10:08:58 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release. 150303 10:08:59 [Warning] option 'innodb-status-file': boolean value 'ib_status' wasn't recognized. Set to OFF. 150303 10:08:59 [Warning] option 'innodb-autoextend-increment': unsigned value 10485760 adjusted to 1000 150303 10:08:59 [Note] InnoDB: Using mutexes to ref count buffer pool pages 150303 10:08:59 [Note] InnoDB: The InnoDB memory heap is disabled 150303 10:08:59 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 150303 10:08:59 [Note] InnoDB: Memory barrier is not used 150303 10:08:59 [Note] InnoDB: Compressed tables use zlib 1.2.3 150303 10:08:59 [Note] InnoDB: Using Linux native AIO 150303 10:08:59 [Note] InnoDB: Using CPU crc32 instructions 150303 10:08:59 [Note] InnoDB: Initializing buffer pool, size = 6.0G 150303 10:08:59 [Note] InnoDB: Completed initialization of buffer pool 150303 10:08:59 [Note] InnoDB: Highest supported file format is Barracuda. 150303 10:08:59 [Note] InnoDB: Log scan progressed past the checkpoint lsn 14178608217047 150303 10:08:59 [Note] InnoDB: Database was not shutdown normally! 150303 10:08:59 [Note] InnoDB: Starting crash recovery. 150303 10:08:59 [Note] InnoDB: Reading tablespace information from the .ibd files... 150303 10:09:05 [Note] InnoDB: Restoring possible half-written data pages 150303 10:09:05 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 14178613459456 InnoDB: Doing recovery: scanned up to log sequence number 14178618702336 InnoDB: Doing recovery: scanned up to log sequence number 14178623945216 InnoDB: Doing recovery: scanned up to log sequence number 14178629188096 InnoDB: Doing recovery: scanned up to log sequence number 14178634430976 InnoDB: Doing recovery: scanned up to log sequence number 14178639673856 InnoDB: Doing recovery: scanned up to log sequence number 14178644916736 ...... ...... ...... ...... 150303 10:09:14 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 16433109, file name slave-bin.852141 InnoDB: Last MySQL binlog file position 0 16784214, file name /var/lib/mysql/log/bin/slave-bin.005211 150303 10:09:34 [Note] InnoDB: 128 rollback segment(s) are active. 150303 10:09:34 [Note] InnoDB: Waiting for purge to start 150303 10:09:34 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 14178834821658 Tue Mar 3 10:09:36 2015 TokuFT recovery starting in env /var/lib/mysql/data/ Tue Mar 3 10:09:36 2015 TokuFT recovery scanning backward from 1330297 Tue Mar 3 10:09:36 2015 TokuFT recovery bw_end_checkpoint at 1330297 timestamp 1425395277901416 xid 1330281 (bw_newer) Tue Mar 3 10:09:36 2015 TokuFT recovery bw_begin_checkpoint at 1330281 timestamp 1425395273487081 (bw_between) Tue Mar 3 10:09:36 2015 TokuFT recovery turning around at begin checkpoint 1330281 time 4414335 Tue Mar 3 10:09:36 2015 TokuFT recovery starts scanning forward to 1330297 from 1330281 left 16 (fw_between) Tue Mar 3 10:09:36 2015 TokuFT recovery closing 14 dictionaries Tue Mar 3 10:09:36 2015 TokuFT recovery making a checkpoint Tue Mar 3 10:09:36 2015 TokuFT recovery done 150303 10:09:36 [Note] Recovering after a crash using /var/lib/mysql/log/bin/slave-bin

---------------

The odd thing is that the 5.5.x version was working just fine (since about a year) in the same hardware...nothing changed in that front.

Any clue?

Thank you in advance

_______________________________________________ Mailing list:https://launchpad.net/~maria-discuss https://launchpad.net/%7Emaria-discuss Post to :maria-discuss@lists.launchpad.net mailto:maria-discuss@lists.launchpad.net Unsubscribe :https://launchpad.net/~maria-discuss https://launchpad.net/%7Emaria-discuss More help :https://help.launchpad.net/ListHelp

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss https://launchpad.net/%7Emaria-discuss Post to : maria-discuss@lists.launchpad.net mailto:maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss https://launchpad.net/%7Emaria-discuss More help : https://help.launchpad.net/ListHelp

-- Gabriel Sosa Sometimes the questions are complicated and the answers are simple. -- Dr. Seuss

-- Gabriel Sosa Sometimes the questions are complicated and the answers are simple. -- Dr. Seuss

Thank you all for your replies... @Reindl thanks for pointing that out. Now, reading again the output of free -m and knowing that tokudb will try to get much as 50% of the server's memory, I will surely need to tweak some values since free + cached memory doesn't meet the need [root@db ~]# free -m total used free shared buffers cached Mem: 7840 7793 47 0 23 3572 -/+ buffers/cache: 4197 3642 Swap: 2047 1494 553 I will work on tweaking the tokudb_cache_size and the innodb_buffer_pool_size parameters, I will let you know from there. Thank you On Tue, Mar 3, 2015 at 6:47 PM, Reindl Harald wrote:

Am 03.03.2015 um 22:26 schrieb Gabriel Sosa:

...

using systemtap I got

[Tue Mar 3 18:48:46 2015] SIGKILL was sent to ps (pid:9763) by processes uid:0 [Tue Mar 3 18:49:46 2015] SIGKILL was sent to ps (pid:9824) by processes uid:0 *[Tue Mar 3 18:50:44 2015] SIGKILL was sent to mysqld (pid:7597) by mysqld uid:498* [Tue Mar 3 18:50:46 2015] SIGKILL was sent to ps (pid:9890) by processes uid:0 [Tue Mar 3 18:51:46 2015] SIGKILL was sent to ps (pid:9997) by processes uid:0 [Tue Mar 3 18:52:35 2015] SIGKILL was sent to cdm (pid:10045) by cdm uid:0

I saw this several times but I wasn't able to find (in this case) which process has the PID 498. @jocelyn, you are right...to tweaks at all for tokudb (my bad here), although I wasn't expecting such /naive/ approach, I mean, I could have not enable the tokudb plugin and all other tables using the other engines would work just fine (maybe just slow though)

free -m returns about 47M free...which isn't much...

you need to learn read the output

it don't matter how much is free, what matters is if the allocated memory is used, shared, buffers or even cache - cache don't matter because it is re-used if a application requests memory

until that happens a sane OS is using *100 %* of the available memory for caching - that below don't mean there are only 577 MB free, i can with no issue start a application requesting 4 GB ram because the 11086 MB cache can be re-used at every moment in time

[root@srv-rhsoft:~]$ free -m total used free shared buffers cached Mem: 15923 15345 577 239 1020 11086 -/+ buffers/cache: 3238 12684 Swap: 3025 6 3018

_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

-- Gabriel Sosa Sometimes the questions are complicated and the answers are simple. -- Dr. Seuss