[Maria-developers] API Documentaion for Replication Features
Hi, I have a need for replicating data from multiple masters (independent data going to different tables). I have looked into the following with no success: Tungsten: Slow, CPU hungry, outrageously bloated, and unacceptably unreliable (replication will stall silently at times, and the only way to resume it is to offline and online the replication service). mysql-replication-listener: Just about got it to work with local binlogs, but haven't managed to get it to connect via TCP to a MySQL server. So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: 1) Slave TCP connections to the master and binlog fetching 2) Binlog parsing 3) Config file parsing In the longer term I'd like to write a patch that adds multiple-master functionality into MySQL itself. For configuring it, I was thinking about the following WRT configuration in my.cnf with multiple section instances, e.g.: [replicator] server-id=12345 master-host=server1 master-port=3306 master-user=user1 master-password=password1 replicate-do-table=db1.table1 [replicator] server-id=12345 master-host=server2 master-port=3306 master-user=user2 master-password=password2 replicate-do-table=db2.table2 etc. Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand. TIA. Gordan
Hi, Gordan! On Nov 29, Gordan Bobic wrote:
Hi,
I have a need for replicating data from multiple masters (independent data going to different tables). ... So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: ... Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand.
You might want to look at https://kb.askmonty.org/en/multi-source-replication/ or http://mysql.taobao.org/index.php/Patch_source_code#Multi-master_replication Regards, Sergei
On 11/29/2012 02:53 PM, Sergei Golubchik wrote:
Hi, Gordan!
On Nov 29, Gordan Bobic wrote:
Hi,
I have a need for replicating data from multiple masters (independent data going to different tables). ... So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: ... Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand.
You might want to look at https://kb.askmonty.org/en/multi-source-replication/
That looks _exactly_ like what I was looking for. I'll take the 10.0 Alpha for a spin and try it out. Many thanks. :) BTW, should I expect any particular issues when replicating from MySQL 5.5 Cluster to MariaDB 10? Gordan
Hi Gordan, I have tried to replicate from mysql5.5 to MariaDB 10.0 by multi-source replication feature, I have not found any problems, it runs well. Thanks, Lixun Sent from my iPhone On 2012-11-29, at 下午11:38, Gordan Bobic <gordan@bobich.net> wrote:
On 11/29/2012 02:53 PM, Sergei Golubchik wrote:
Hi, Gordan!
On Nov 29, Gordan Bobic wrote:
Hi,
I have a need for replicating data from multiple masters (independent data going to different tables). ... So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: ... Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand.
You might want to look at https://kb.askmonty.org/en/multi-source-replication/
That looks _exactly_ like what I was looking for. I'll take the 10.0 Alpha for a spin and try it out.
Many thanks. :)
BTW, should I expect any particular issues when replicating from MySQL 5.5 Cluster to MariaDB 10?
Gordan
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Thanks Lixun. How is the replicate-[do|ignore]-[table|database] functionality handled with multiple masters where I only want specific tables from specific masters so that the feeds don't clobber each other? These don't appear to be settable in the CHANGE MASTER statement, and I cannot find any documentation on how to set them unambiguously in the config file. Any hints? Gordan On 11/29/2012 03:54 PM, Lixun Peng wrote:
Hi Gordan,
I have tried to replicate from mysql5.5 to MariaDB 10.0 by multi-source replication feature, I have not found any problems, it runs well.
Thanks, Lixun
Sent from my iPhone
On 2012-11-29, at 下午11:38, Gordan Bobic<gordan@bobich.net> wrote:
On 11/29/2012 02:53 PM, Sergei Golubchik wrote:
Hi, Gordan!
On Nov 29, Gordan Bobic wrote:
Hi,
I have a need for replicating data from multiple masters (independent data going to different tables). ... So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: ... Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand.
You might want to look at https://kb.askmonty.org/en/multi-source-replication/
That looks _exactly_ like what I was looking for. I'll take the 10.0 Alpha for a spin and try it out.
Many thanks. :)
BTW, should I expect any particular issues when replicating from MySQL 5.5 Cluster to MariaDB 10?
Gordan
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Hi Gordan, I'm not changed the replicate-[do|ignore]-[table|database] functionality code, so I think it should work for all slave threads, but it can't specify each master to use specific filter. On Fri, Nov 30, 2012 at 12:55 AM, Gordan Bobic <gordan@bobich.net> wrote:
Thanks Lixun.
How is the replicate-[do|ignore]-[table|database] functionality handled with multiple masters where I only want specific tables from specific masters so that the feeds don't clobber each other? These don't appear to be settable in the CHANGE MASTER statement, and I cannot find any documentation on how to set them unambiguously in the config file.
Any hints?
Gordan
On 11/29/2012 03:54 PM, Lixun Peng wrote:
Hi Gordan,
I have tried to replicate from mysql5.5 to MariaDB 10.0 by multi-source replication feature, I have not found any problems, it runs well.
Thanks, Lixun
Sent from my iPhone
On 2012-11-29, at 下午11:38, Gordan Bobic<gordan@bobich.net> wrote:
On 11/29/2012 02:53 PM, Sergei Golubchik wrote:
Hi, Gordan!
On Nov 29, Gordan Bobic wrote:
Hi,
I have a need for replicating data from multiple masters (independent data going to different tables). ... So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: ... Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand.
You might want to look at https://kb.askmonty.org/en/multi-source-replication/
That looks _exactly_ like what I was looking for. I'll take the 10.0 Alpha for a spin and try it out.
Many thanks. :)
BTW, should I expect any particular issues when replicating from MySQL 5.5 Cluster to MariaDB 10?
Gordan
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
-- Senior MySQL Developer @ Taobao.com Mobile Phone: +86 18658156856 (Hangzhou) Gtalk: penglixun(at)gmail.com Twitter: http://www.twitter.com/plinux Blog: http://www.penglixun.com
Hi!
"Lixun" == Lixun Peng <penglixun@gmail.com> writes:
Lixun> Hi Gordan, Lixun> I'm not changed the replicate-[do|ignore]-[table|database] functionality Lixun> code, so I think it should work for all slave threads, but it can't specify Lixun> each master to use specific filter. Yes, the replication filters are global for now. It would probably be 2-3 days of work to make these per master. The main questions is only how to define this in the config file. We could probably use the same thing we use for key buffers: master_name.replicate-do-db=... Do you want some information of how to develop this or do you think you can find someone to sponsor this? Regards, Monty
Hi Monty,
"Lixun" == Lixun Peng<penglixun@gmail.com> writes:
Lixun> Hi Gordan, Lixun> I'm not changed the replicate-[do|ignore]-[table|database] functionality Lixun> code, so I think it should work for all slave threads, but it can't specify Lixun> each master to use specific filter.
Yes, the replication filters are global for now.
It would probably be 2-3 days of work to make these per master.
The main questions is only how to define this in the config file. We could probably use the same thing we use for key buffers:
master_name.replicate-do-db=...
Do you want some information of how to develop this or do you think you can find someone to sponsor this?
Thanks for the info. At the moment, I'm interested in some pointers on how to develop this at the moment. On a related note, is there anything like the sql_slave_skip_counter variable implemented on a per-master basis? Or is this something else that would need separating out? I know I could just change the relevant master info file (carefully) with the new log position, but that's somewhat inconvenient and requires restarting the server. Gordan
Hi Gordan, I think you can found the answer from doc : https://kb.askmonty.org/en/multi-source-replication/ like this: set @@default_master_connection=''; show status like 'Slave_running'; set @@default_master_connection='other_connection'; show status like 'Slave_running'; but some parameters only can write in my.cnf and can't be changed in running, we have not implement them temporarily. Thanks, Lixun On Fri, Nov 30, 2012 at 7:46 PM, Gordan Bobic <gordan@bobich.net> wrote:
Hi Monty,
"Lixun" == Lixun Peng<penglixun@gmail.com> writes:
>
Lixun> Hi Gordan, Lixun> I'm not changed the replicate-[do|ignore]-[table|**database] functionality Lixun> code, so I think it should work for all slave threads, but it can't specify Lixun> each master to use specific filter.
Yes, the replication filters are global for now.
It would probably be 2-3 days of work to make these per master.
The main questions is only how to define this in the config file. We could probably use the same thing we use for key buffers:
master_name.replicate-do-db=..**.
Do you want some information of how to develop this or do you think you can find someone to sponsor this?
Thanks for the info. At the moment, I'm interested in some pointers on how to develop this at the moment.
On a related note, is there anything like the sql_slave_skip_counter variable implemented on a per-master basis? Or is this something else that would need separating out? I know I could just change the relevant master info file (carefully) with the new log position, but that's somewhat inconvenient and requires restarting the server.
Gordan
-- Senior MySQL Developer @ Taobao.com Mobile Phone: +86 18658156856 (Hangzhou) Gtalk: penglixun(at)gmail.com Twitter: http://www.twitter.com/plinux Blog: http://www.penglixun.com
Hi Monty, On Fri, Nov 30, 2012 at 4:01 PM, Michael Widenius <monty@askmonty.org>wrote:
Hi!
"Lixun" == Lixun Peng <penglixun@gmail.com> writes:
Lixun> Hi Gordan, Lixun> I'm not changed the replicate-[do|ignore]-[table|database] functionality Lixun> code, so I think it should work for all slave threads, but it can't specify Lixun> each master to use specific filter.
Yes, the replication filters are global for now.
It would probably be 2-3 days of work to make these per master.
The main questions is only how to define this in the config file. We could probably use the same thing we use for key buffers:
master_name.replicate-do-db=...
I think this way is ok. I agree with you.
Do you want some information of how to develop this or do you think you can find someone to sponsor this?
I can try to implement it. Are you doing it? Thanks, Lixun -- Senior MySQL Developer @ Taobao.com Mobile Phone: +86 18658156856 (Hangzhou) Gtalk: penglixun(at)gmail.com Twitter: http://www.twitter.com/plinux Blog: http://www.penglixun.com
participants (4)
-
Gordan Bobic -
Lixun Peng -
Michael Widenius -
Sergei Golubchik