[Maria-developers] Tungsten Replicator and MariaDB
Hi All, This is a heads-up that we are actively testing Tungsten Replicator against the latest MariaDB build and hope to complete initial certification this week. So far there are no problems. We are also going to add a feature to use Maria for our catalog tables instead of InnoDB. Tungsten Replicator builds are GPL V2, by the way. This brings up a question for the Maria dev team-what are your plans, if any, for replication support in MariaDB? In particular, are there any plans that would affect binlog formats? Thanks, Robert P.s., I don't see a location on AskMonty.org to post builds for related products. Any hints where to go? We would like to post our open source offerings in a location that is easy for MariaDB users to find. -- Robert Hodges, CTO, Continuent, Inc. Email: robert.hodges@continuent.com Mobile: +1-510-501-3728 Skype: hodgesrm
Hi Robert On 12/06/2009, at 1:45 AM, Robert Hodges wrote:
This is a heads-up that we are actively testing Tungsten Replicator against the latest MariaDB build and hope to complete initial certification this week. So far there are no problems. We are also going to add a feature to use Maria for our catalog tables instead of InnoDB. Tungsten Replicator builds are GPL V2, by the way.
This brings up a question for the Maria dev team—what are your plans, if any, for replication support in MariaDB? In particular, are there any plans that would affect binlog formats?
I'd expect the patches for binlog checksums and global transaction IDs; but you'll be adding those in for 5.4 anyway.
P.s., I don’t see a location on AskMonty.org to post builds for related products. Any hints where to go? We would like to post our open source offerings in a location that is easy for MariaDB users to find.
Files or announcements/links? ourdelta.org could do some of it. Cheers, Arjen. -- Arjen Lentz, Director @ Open Query (http://openquery.com) Affordable Training and ProActive Support for MySQL & related technologies Follow our blog at http://openquery.com/blog/ OurDelta: free enhanced builds for MySQL @ http://ourdelta.org
Hi Arjen, Thanks for the response. I really appreciate the guidance-are these patches to your knowledge already in the current 5.4 builds? We have not done any work yet to accommodate 5.4 format changes. I need to check these as it's our policy to try to support over time a reasonably complete set of the MySQL builds, up to and including significant variants like Drizzle. We would also gladly post file announcements on ourdelta.org. We can also add your builds to the list of MySQL builds against which we certify. As long as there are no unexpected binlog changes it should be pretty straightforward. Cheers, Robert On 6/11/09 4:24 PM PDT, "Arjen Lentz" <arjen@openquery.com> wrote: Hi Robert On 12/06/2009, at 1:45 AM, Robert Hodges wrote:
This is a heads-up that we are actively testing Tungsten Replicator against the latest MariaDB build and hope to complete initial certification this week. So far there are no problems. We are also going to add a feature to use Maria for our catalog tables instead of InnoDB. Tungsten Replicator builds are GPL V2, by the way.
This brings up a question for the Maria dev team-what are your plans, if any, for replication support in MariaDB? In particular, are there any plans that would affect binlog formats?
I'd expect the patches for binlog checksums and global transaction IDs; but you'll be adding those in for 5.4 anyway.
P.s., I don't see a location on AskMonty.org to post builds for related products. Any hints where to go? We would like to post our open source offerings in a location that is easy for MariaDB users to find.
Files or announcements/links? ourdelta.org could do some of it. Cheers, Arjen. -- Arjen Lentz, Director @ Open Query (http://openquery.com) Affordable Training and ProActive Support for MySQL & related technologies Follow our blog at http://openquery.com/blog/ OurDelta: free enhanced builds for MySQL @ http://ourdelta.org -- Robert Hodges, CTO, Continuent, Inc. Email: robert.hodges@continuent.com Mobile: +1-510-501-3728 Skype: hodgesrm
Hi Robert, On Thu, Jun 11, 2009 at 08:45:08AM -0700, Robert Hodges wrote:
This brings up a question for the Maria dev team-what are your plans, if any, for replication support in MariaDB? In particular, are there any plans that would affect binlog formats?
So far we don't have any planned or in-development features that would have an effect on replication or binlog format. MariaDB's replication is expected to be the same as MySQL replication. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog
Hi Sergey On 19/06/2009, at 8:12 PM, Sergey Petrunya wrote:
On Thu, Jun 11, 2009 at 08:45:08AM -0700, Robert Hodges wrote:
This brings up a question for the Maria dev team-what are your plans, if any, for replication support in MariaDB? In particular, are there any plans that would affect binlog formats?
So far we don't have any planned or in-development features that would have an effect on replication or binlog format. MariaDB's replication is expected to be the same as MySQL replication.
This is the real world, and pragmatism is expected. Things are broken in the land of MySQL replication, and thus we should fix it. The fixes are actually available, so why even bother coming up with excuses? Yep it changes/extends the binlog format. Tough! It's an upgrade; just like any other, a newer server should be able to be a slave to an older master. But this is important stuff! For instance, adding checksums to the binlog prevents a whole load of trouble. Not having it in there since the start (yes SashaP, I know you're guilty - plus all those who reviewed your code at the time) is a 10 year embarassment. Let's get it right, now, shall we? Thanks Cheers, Arjen. -- Arjen Lentz, Director @ Open Query (http://openquery.com) Affordable Training and ProActive Support for MySQL & related technologies Follow our blog at http://openquery.com/blog/ OurDelta: free enhanced builds for MySQL @ http://ourdelta.org
On Fri, Jun 19, 2009 at 5:46 AM, Arjen Lentz<arjen@openquery.com> wrote:
Hi Sergey
On 19/06/2009, at 8:12 PM, Sergey Petrunya wrote:
On Thu, Jun 11, 2009 at 08:45:08AM -0700, Robert Hodges wrote:
This brings up a question for the Maria dev team-what are your plans, if any, for replication support in MariaDB? In particular, are there any plans that would affect binlog formats?
So far we don't have any planned or in-development features that would have an effect on replication or binlog format. MariaDB's replication is expected to be the same as MySQL replication.
You can add support for binlog checksums without breaking compatibility because the format is extensible. Alas, extensibility is broken because of bugs, so when those bugs are fixed I think you can do this. However ...
This is the real world, and pragmatism is expected. Things are broken in the land of MySQL replication, and thus we should fix it. The fixes are actually available, so why even bother coming up with excuses?
Yep it changes/extends the binlog format. Tough! It's an upgrade; just like any other, a newer server should be able to be a slave to an older master. But this is important stuff! For instance, adding checksums to the binlog prevents a whole load of trouble. Not having it in there since the start (yes SashaP, I know you're guilty - plus all those who reviewed your code at the time) is a 10 year embarassment. Let's get it right, now, shall we? Thanks
Time is finite and the work to do on MySQL is not. It is not that hard to do this feature, but who has the time to do it? And if nobody has the time for it, then who is funding it? Does MariaDB have a list of pending work ranked by priority? -- Mark Callaghan mdcallag@gmail.com
Hi All, I have the same question as Mark, namely: what is the prioritized list of possible improvements to replication? To answer Mark's other question, improving replication is work that we (Continuent) would be willing to help fund. Improvements in this area seem like a great way for MariaDB to differentiate itself. We would like to help create the list of likely areas for improvement both in terms of benefits to users as well as how feasible fixes are. Here are a few items that I have run into during work in the MySQL binlog. They may be different from other people as we are parsing the binlog directly, which is not something that the average user does regularly. * Event checksums. Binlog corruption does not happen too often but it's bad when it does. I got it last week during a customer demo. :( * Column names in table map events. Right now it's just numbers. Not good for filtering SQL or replication to other databases. * Keys. Update events in row replication use before/after images but do not specify keys. This is problematic in a number of cases. Global transaction IDs and semi-synchronous replication support would also be high on my list of feature additions for general usability. Fixes for serious bugs would also be very welcome. Cheers, Robert P.s., Mark, do you have a pointer to the extensibility bug(s)? On 6/20/09 8:00 AM PDT, "MARK CALLAGHAN" <mdcallag@gmail.com> wrote:
On Fri, Jun 19, 2009 at 5:46 AM, Arjen Lentz<arjen@openquery.com> wrote:
Hi Sergey
On 19/06/2009, at 8:12 PM, Sergey Petrunya wrote:
On Thu, Jun 11, 2009 at 08:45:08AM -0700, Robert Hodges wrote:
This brings up a question for the Maria dev team-what are your plans, if any, for replication support in MariaDB? In particular, are there any plans that would affect binlog formats?
So far we don't have any planned or in-development features that would have an effect on replication or binlog format. MariaDB's replication is expected to be the same as MySQL replication.
You can add support for binlog checksums without breaking compatibility because the format is extensible. Alas, extensibility is broken because of bugs, so when those bugs are fixed I think you can do this. However ...
This is the real world, and pragmatism is expected. Things are broken in the land of MySQL replication, and thus we should fix it. The fixes are actually available, so why even bother coming up with excuses?
Yep it changes/extends the binlog format. Tough! It's an upgrade; just like any other, a newer server should be able to be a slave to an older master. But this is important stuff! For instance, adding checksums to the binlog prevents a whole load of trouble. Not having it in there since the start (yes SashaP, I know you're guilty - plus all those who reviewed your code at the time) is a 10 year embarassment. Let's get it right, now, shall we? Thanks
Time is finite and the work to do on MySQL is not. It is not that hard to do this feature, but who has the time to do it? And if nobody has the time for it, then who is funding it?
Does MariaDB have a list of pending work ranked by priority?
-- Mark Callaghan mdcallag@gmail.com
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
-- Robert Hodges, CTO, Continuent, Inc. Email: robert.hodges@continuent.com Mobile: +1-510-501-3728 Skype: hodgesrm
On Sat, Jun 20, 2009 at 11:16 AM, Robert Hodges<robert.hodges@continuent.com> wrote:
Hi All,
I have the same question as Mark, namely: what is the prioritized list of possible improvements to replication? To answer Mark's other question, improving replication is work that we (Continuent) would be willing to help fund. Improvements in this area seem like a great way for MariaDB to differentiate itself. We would like to help create the list of likely areas for improvement both in terms of benefits to users as well as how feasible fixes are.
Here are a few items that I have run into during work in the MySQL binlog. They may be different from other people as we are parsing the binlog directly, which is not something that the average user does regularly.
* Event checksums. Binlog corruption does not happen too often but it's bad when it does. I got it last week during a customer demo. :(
* Column names in table map events. Right now it's just numbers. Not good for filtering SQL or replication to other databases.
* Keys. Update events in row replication use before/after images but do not specify keys. This is problematic in a number of cases.
Global transaction IDs and semi-synchronous replication support would also be high on my list of feature additions for general usability. Fixes for serious bugs would also be very welcome.
Cheers, Robert
P.s., Mark, do you have a pointer to the extensibility bug(s)?
Justin or Mats might know the bug numbers. The fix for this should be included in the launchpad branch Justin created for global group ids -- https://code.launchpad.net/~jtolmer/mysql-server/global-trx-ids -- Mark Callaghan mdcallag@gmail.com
On Sat, Jun 20, 2009 at 12:20 PM, MARK CALLAGHAN <mdcallag@gmail.com> wrote:
On Sat, Jun 20, 2009 at 11:16 AM, Robert Hodges<robert.hodges@continuent.com> wrote:
Hi All,
I have the same question as Mark, namely: what is the prioritized list of possible improvements to replication? To answer Mark's other question, improving replication is work that we (Continuent) would be willing to help fund. Improvements in this area seem like a great way for MariaDB to differentiate itself. We would like to help create the list of likely areas for improvement both in terms of benefits to users as well as how feasible fixes are.
Here are a few items that I have run into during work in the MySQL binlog. They may be different from other people as we are parsing the binlog directly, which is not something that the average user does regularly.
* Event checksums. Binlog corruption does not happen too often but it's bad when it does. I got it last week during a customer demo. :(
* Column names in table map events. Right now it's just numbers. Not good for filtering SQL or replication to other databases.
* Keys. Update events in row replication use before/after images but do not specify keys. This is problematic in a number of cases.
Global transaction IDs and semi-synchronous replication support would also be high on my list of feature additions for general usability. Fixes for serious bugs would also be very welcome.
Cheers, Robert
P.s., Mark, do you have a pointer to the extensibility bug(s)?
Sounds like you're asking about 37466 <http://bugs.mysql.com/?id=37466>.
Justin or Mats might know the bug numbers. The fix for this should be included in the launchpad branch Justin created for global group ids -- https://code.launchpad.net/~jtolmer/mysql-server/global-trx-ids
Yes, my launchpad branch has the fix for 37466. There are actually several places which needed to be fixed including changing the slave to no longer write stop events to the relay log. My branch also has several other fixes; event checksums, global transaction ID based recovery of replication positions after a slave crash and my fixes for 38826 <http://bugs.mysql.com/?id=38826> and 39325<http://bugs.mysql.com/?id=39325> which are a bit different from the official ones. Justin
-- Mark Callaghan mdcallag@gmail.com
On Fri, Jun 19, 2009 at 10:46:04PM +1000, Arjen Lentz wrote:
Hi Sergey
On 19/06/2009, at 8:12 PM, Sergey Petrunya wrote:
On Thu, Jun 11, 2009 at 08:45:08AM -0700, Robert Hodges wrote:
This brings up a question for the Maria dev team-what are your plans, if any, for replication support in MariaDB? In particular, are there any plans that would affect binlog formats?
So far we don't have any planned or in-development features that would have an effect on replication or binlog format. MariaDB's replication is expected to be the same as MySQL replication.
This is the real world, and pragmatism is expected. Things are broken in the land of MySQL replication, and thus we should fix it. The fixes are actually available, so why even bother coming up with excuses?
Oops. I've totally forgot about those. Was thinking of what we ourselves have already in development. About those patches, I have no idea. BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog
hi!
"Arjen" == Arjen Lentz <arjen@openquery.com> writes:
Arjen> Hi Sergey Arjen> On 19/06/2009, at 8:12 PM, Sergey Petrunya wrote:
On Thu, Jun 11, 2009 at 08:45:08AM -0700, Robert Hodges wrote:
This brings up a question for the Maria dev team-what are your plans, if any, for replication support in MariaDB? In particular, are there any plans that would affect binlog formats?
So far we don't have any planned or in-development features that would have an effect on replication or binlog format. MariaDB's replication is expected to be the same as MySQL replication.
Arjen> This is the real world, and pragmatism is expected. Arjen> Things are broken in the land of MySQL replication, and thus we should Arjen> fix it. My view is that we at Monty Program Ab don't currently have plans to rewrite the MySQL replication. We do however plan to fix issues and bugs in the current replication and apply some of the important patches that are floating around. Arjen> The fixes are actually available, so why even bother coming up with Arjen> excuses? What excuses? Arjen> Yep it changes/extends the binlog format. Tough! It's an upgrade; just Arjen> like any other, a newer server should be able to be a slave to an Arjen> older master. Yes. Arjen> But this is important stuff! For instance, adding checksums to the Arjen> binlog prevents a whole load of trouble. Not having it in there since Arjen> the start (yes SashaP, I know you're guilty - plus all those who Arjen> reviewed your code at the time) is a 10 year embarassment. Let's get Arjen> it right, now, shall we? Arjen> Thanks As far as I know, it should be possible to add checksum without breaking things for old masters/slaves. Need to verify if this is the case. Regards, Monty
Hi all, I would like to post a clarification as my first message may have created some unintended confusion. We are working with Monty and Ethan O¹Rafferty to figure out how to work together on our respective releases, which includes where to post them. So my question at the end is not relevant for this list. The other question about replication features still is a relevant question. I'll follow up on the other replies. Cheers, Robert On 6/11/09 8:45 AM PDT, "Robert Hodges" <robert.hodges@continuent.com> wrote:
Hi All,
This is a heads-up that we are actively testing Tungsten Replicator against the latest MariaDB build and hope to complete initial certification this week. So far there are no problems. We are also going to add a feature to use Maria for our catalog tables instead of InnoDB. Tungsten Replicator builds are GPL V2, by the way.
This brings up a question for the Maria dev team<what are your plans, if any, for replication support in MariaDB? In particular, are there any plans that would affect binlog formats?
Thanks, Robert
P.s., I don¹t see a location on AskMonty.org to post builds for related products. Any hints where to go? We would like to post our open source offerings in a location that is easy for MariaDB users to find.
-- Robert Hodges, CTO, Continuent, Inc. Email: robert.hodges@continuent.com Mobile: +1-510-501-3728 Skype: hodgesrm
participants (6)
-
Arjen Lentz -
Justin Tolmer -
MARK CALLAGHAN -
Michael Widenius -
Robert Hodges -
Sergey Petrunya