[Maria-developers] Fwd: Total No of events in event group.
Hi Kristian! On Tue, Jan 10, 2017 at 4:41 PM, Kristian Nielsen <knielsen@knielsen-hq.org> wrote:
Sachin Setiya <sachin.setiya@mariadb.com> writes:
Can we do something like send total no of Rows_log_events in a event group to slave. We can write this info in Gtid_log_event. Reason for doing this is that on slave we can show how much progress we have made.
My immediate impression is that this sounds like a bad idea.
It increases the binlog size without adding any real information - after all the event count can be determined easily by just reading the events.
You mean counting the all events in event group without applying them.
It also requires careful design and testing for impact on forwards- and backwards compatibility.
And event count is not a very meaningful progress indicator. One event group could have 10 Rows_log_event each with a single row operation. While another could have a single Rows_log_event that contains a hundred row operations.
Okay, So it there a better way to show progress in slave ?
And why only rows event? What about Query_log_events in a transaction in statement-based replication? We can generalize this idea to Query_log_events also.
- Kristian. Regards sachin
Sachin Setiya <sachin.setiya@mariadb.com> writes:
Okay, So it there a better way to show progress in slave ?
You have not explained what you mean by "showing progress". Doesn't SHOW SLAVE STATUS already show event-by-event progress in the relay log position? For progress inside a row event, for example one could display the offset of current row operation as a percentage of total event size. Be sure to consider the performance impact of such status updates. Why does replication slave need different progress monitoring than any other thread doing queriess? I mean, more visibility and monitoring is nice, sure. But it should start with a general approach and design. Not by randomly adding bits and pieces to various binlog events (or other places) without a clear idea what is being achieved, and why... - Kristian.
Hi Kristian! On Tue, Jan 10, 2017 at 5:37 PM, Kristian Nielsen <knielsen@knielsen-hq.org> wrote:
Sachin Setiya <sachin.setiya@mariadb.com> writes:
Okay, So it there a better way to show progress in slave ?
You have not explained what you mean by "showing progress".
Doesn't SHOW SLAVE STATUS already show event-by-event progress in the relay log position?
For progress inside a row event, for example one could display the offset of current row operation as a percentage of total event size. Be sure to consider the performance impact of such status updates.
Why does replication slave need different progress monitoring than any other thread doing queriess?
I mean, more visibility and monitoring is nice, sure. But it should start with a general approach and design. Not by randomly adding bits and pieces to various binlog events (or other places) without a clear idea what is being achieved, and why...
- Kristian. This task is related to mdev-7409 / mdev-4557. I have created a patch for this mdev. but I was thinking for monitoring command 'SHOW PROCESSLIST' in Progress column can we show how much % of the events from Event group Slave has applied so that user can get a better monitoring. If I apply my patch I get only this. MariaDB [d]> show processlist; +----+-------------+-----------+------+---------+------+-------------------------------------------------+------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +----+-------------+-----------+------+---------+------+-------------------------------------------------+------------------+----------+ | 3 | system user | | d | Connect | 0 | Write_rows_log_event::write_row(-1) on table t1 | NULL | 0.000 | | 4 | system user | | NULL | Connect | 121 | Waiting for master to send event | NULL | 0.000 | | 7 | root | localhost | d | Query | 0 | init | show processlist | 0.000 | +----+-------------+-----------+------+---------+------+-------------------------------------------------+------------------+----------+
Progress is always zero. So I was thinking if somehow in advace we can know how many events are there in event group we can show progress to user by calculating how much % of events from Event group we have executed. Regards sachin
Sachin Setiya <sachin.setiya@mariadb.com> writes:
I have created a patch for this mdev. but I was thinking for monitoring command 'SHOW PROCESSLIST' in Progress column can we show how much % of the events from Event group Slave has applied so that user can get a better monitoring. If I apply my patch I get only this. MariaDB [d]> show processlist; +----+-------------+-----------+------+---------+------+-------------------------------------------------+------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +----+-------------+-----------+------+---------+------+-------------------------------------------------+------------------+----------+ | 3 | system user | | d | Connect | 0 | Write_rows_log_event::write_row(-1) on table t1 | NULL | 0.000 | | 4 | system user | | NULL | Connect | 121 | Waiting for master to send event | NULL | 0.000 | | 7 | root | localhost | d | Query | 0 | init | show processlist | 0.000 | +----+-------------+-----------+------+---------+------+-------------------------------------------------+------------------+----------+
Progress is always zero.
I think the "progress" feature was implemented by Monty (Cc:'ed). If I understand correctly, "progress" means progress of one query, not of a complete transaction. So surely, it should be the same for a replication thread? If you were to count events in an event group, you would get progress of the transaction. And it would be a very crude metric, since typically different events can take very different amounts of time to execute. You could should be the percentage of one row event that has been completed? I think the row event is completely in-memory, so that should be easy. I think there can be multiple row events for one source statement on the master, so it might also not perfectly match the normal meaning of "progress". Did you consider the performance impact of updating the progress? I would worry that updating eg. for every row operation would be prohibitively expensive. - Kristian.
participants (2)
-
Kristian Nielsen -
Sachin Setiya