[Maria-developers] API Documentaion for Replication Features

newer
Re: [Maria-developers] How to get...

older
[Maria-developers] MariaDB 5.5.28a...

Gordan Bobic

29 Nov 2012 29 Nov '12

3:10 p.m.

Hi, I have a need for replicating data from multiple masters (independent data going to different tables). I have looked into the following with no success: Tungsten: Slow, CPU hungry, outrageously bloated, and unacceptably unreliable (replication will stall silently at times, and the only way to resume it is to offline and online the replication service). mysql-replication-listener: Just about got it to work with local binlogs, but haven't managed to get it to connect via TCP to a MySQL server. So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: 1) Slave TCP connections to the master and binlog fetching 2) Binlog parsing 3) Config file parsing In the longer term I'd like to write a patch that adds multiple-master functionality into MySQL itself. For configuring it, I was thinking about the following WRT configuration in my.cnf with multiple section instances, e.g.: [replicator] server-id=12345 master-host=server1 master-port=3306 master-user=user1 master-password=password1 replicate-do-table=db1.table1 [replicator] server-id=12345 master-host=server2 master-port=3306 master-user=user2 master-password=password2 replicate-do-table=db2.table2 etc. Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand. TIA. Gordan

Show replies by date

Sergei Golubchik

29 Nov 29 Nov

3:53 p.m.

Hi, Gordan! On Nov 29, Gordan Bobic wrote:

...

Hi,

I have a need for replicating data from multiple masters (independent data going to different tables). ... So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: ... Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand.

You might want to look at https://kb.askmonty.org/en/multi-source-replication/ or http://mysql.taobao.org/index.php/Patch_source_code#Multi-master_replication Regards, Sergei

Gordan Bobic

4:38 p.m.

On 11/29/2012 02:53 PM, Sergei Golubchik wrote:

...

Hi, Gordan!

On Nov 29, Gordan Bobic wrote:

...
Hi,

I have a need for replicating data from multiple masters (independent data going to different tables). ... So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: ... Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand.

You might want to look at https://kb.askmonty.org/en/multi-source-replication/

That looks _exactly_ like what I was looking for. I'll take the 10.0 Alpha for a spin and try it out. Many thanks. :) BTW, should I expect any particular issues when replicating from MySQL 5.5 Cluster to MariaDB 10? Gordan

Lixun Peng

4:54 p.m.

Hi Gordan, I have tried to replicate from mysql5.5 to MariaDB 10.0 by multi-source replication feature, I have not found any problems, it runs well. Thanks, Lixun Sent from my iPhone On 2012-11-29, at 下午11:38, Gordan Bobic wrote:

...

On 11/29/2012 02:53 PM, Sergei Golubchik wrote:

...
Hi, Gordan!

On Nov 29, Gordan Bobic wrote:

...
Hi,

I have a need for replicating data from multiple masters (independent data going to different tables). ... So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: ... Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand.

You might want to look at https://kb.askmonty.org/en/multi-source-replication/

That looks _exactly_ like what I was looking for. I'll take the 10.0 Alpha for a spin and try it out.

Many thanks. :)

BTW, should I expect any particular issues when replicating from MySQL 5.5 Cluster to MariaDB 10?

Gordan

_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp

Gordan Bobic

5:55 p.m.

Thanks Lixun. How is the replicate-[do|ignore]-[table|database] functionality handled with multiple masters where I only want specific tables from specific masters so that the feeds don't clobber each other? These don't appear to be settable in the CHANGE MASTER statement, and I cannot find any documentation on how to set them unambiguously in the config file. Any hints? Gordan On 11/29/2012 03:54 PM, Lixun Peng wrote:

...

Hi Gordan,

I have tried to replicate from mysql5.5 to MariaDB 10.0 by multi-source replication feature, I have not found any problems, it runs well.

Thanks, Lixun

Sent from my iPhone

On 2012-11-29, at 下午11:38, Gordan Bobic wrote:

...
On 11/29/2012 02:53 PM, Sergei Golubchik wrote:

...
Hi, Gordan!

On Nov 29, Gordan Bobic wrote:

...
Hi,

I have a need for replicating data from multiple masters (independent data going to different tables). ... So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: ... Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand.

You might want to look at https://kb.askmonty.org/en/multi-source-replication/

That looks _exactly_ like what I was looking for. I'll take the 10.0 Alpha for a spin and try it out.

Many thanks. :)

BTW, should I expect any particular issues when replicating from MySQL 5.5 Cluster to MariaDB 10?

Gordan

_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp

Lixun Peng

30 Nov 30 Nov

2:28 a.m.

Hi Gordan, I'm not changed the replicate-[do|ignore]-[table|database] functionality code, so I think it should work for all slave threads, but it can't specify each master to use specific filter. On Fri, Nov 30, 2012 at 12:55 AM, Gordan Bobic wrote:

...

Thanks Lixun.

How is the replicate-[do|ignore]-[table|database] functionality handled with multiple masters where I only want specific tables from specific masters so that the feeds don't clobber each other? These don't appear to be settable in the CHANGE MASTER statement, and I cannot find any documentation on how to set them unambiguously in the config file.

Any hints?

Gordan

On 11/29/2012 03:54 PM, Lixun Peng wrote:

...
Hi Gordan,

I have tried to replicate from mysql5.5 to MariaDB 10.0 by multi-source replication feature, I have not found any problems, it runs well.

Thanks, Lixun

Sent from my iPhone

On 2012-11-29, at 下午11:38, Gordan Bobic wrote:

...
On 11/29/2012 02:53 PM, Sergei Golubchik wrote:

...
Hi, Gordan!

On Nov 29, Gordan Bobic wrote:

...
Hi,

I have a need for replicating data from multiple masters (independent data going to different tables). ... So what I was hoping to do is write something based on the existing MySQL code since at least that should "just work". The problem is - I am completely unfamiliar with MySQL internals, and I haven't found any function level API documentation. So, would anyone here be kind enough to point me at the specific bits of code in MySQL that handle: ... Any pointers would be greatly appreciated. If somebody here is already undertaking a similar effort, I would rather like to lend a hand.

You might want to look at https://kb.askmonty.org/en/multi-source-replication/

That looks _exactly_ like what I was looking for. I'll take the 10.0 Alpha for a spin and try it out.

Many thanks. :)

BTW, should I expect any particular issues when replicating from MySQL 5.5 Cluster to MariaDB 10?

Gordan

_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp

_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp

-- Senior MySQL Developer @ Taobao.com Mobile Phone: +86 18658156856 (Hangzhou) Gtalk: penglixun(at)gmail.com Twitter: http://www.twitter.com/plinux Blog: http://www.penglixun.com

Michael Widenius

9:01 a.m.

Hi!

...

...
...
...
...
"Lixun" == Lixun Peng writes:

Lixun> Hi Gordan, Lixun> I'm not changed the replicate-[do|ignore]-[table|database] functionality Lixun> code, so I think it should work for all slave threads, but it can't specify Lixun> each master to use specific filter. Yes, the replication filters are global for now. It would probably be 2-3 days of work to make these per master. The main questions is only how to define this in the config file. We could probably use the same thing we use for key buffers: master_name.replicate-do-db=... Do you want some information of how to develop this or do you think you can find someone to sponsor this? Regards, Monty

Gordan Bobic

12:46 p.m.

Hi Monty,

...

...
...
...
...
...
"Lixun" == Lixun Peng writes:

Lixun> Hi Gordan, Lixun> I'm not changed the replicate-[do|ignore]-[table|database] functionality Lixun> code, so I think it should work for all slave threads, but it can't specify Lixun> each master to use specific filter.

Yes, the replication filters are global for now.

It would probably be 2-3 days of work to make these per master.

The main questions is only how to define this in the config file. We could probably use the same thing we use for key buffers:

master_name.replicate-do-db=...

Do you want some information of how to develop this or do you think you can find someone to sponsor this?

Thanks for the info. At the moment, I'm interested in some pointers on how to develop this at the moment. On a related note, is there anything like the sql_slave_skip_counter variable implemented on a per-master basis? Or is this something else that would need separating out? I know I could just change the relevant master info file (carefully) with the new log position, but that's somewhat inconvenient and requires restarting the server. Gordan

Lixun Peng

3 Dec 3 Dec

9:52 a.m.

Hi Gordan, I think you can found the answer from doc : https://kb.askmonty.org/en/multi-source-replication/ like this: set @@default_master_connection=''; show status like 'Slave_running'; set @@default_master_connection='other_connection'; show status like 'Slave_running'; but some parameters only can write in my.cnf and can't be changed in running, we have not implement them temporarily. Thanks, Lixun On Fri, Nov 30, 2012 at 7:46 PM, Gordan Bobic wrote:

...

Hi Monty,

"Lixun" == Lixun Peng writes:

...
...
...
...
...
>

Lixun> Hi Gordan, Lixun> I'm not changed the replicate-[do|ignore]-[table|**database] functionality Lixun> code, so I think it should work for all slave threads, but it can't specify Lixun> each master to use specific filter.

Yes, the replication filters are global for now.

It would probably be 2-3 days of work to make these per master.

The main questions is only how to define this in the config file. We could probably use the same thing we use for key buffers:

master_name.replicate-do-db=..**.

Do you want some information of how to develop this or do you think you can find someone to sponsor this?

Thanks for the info. At the moment, I'm interested in some pointers on how to develop this at the moment.

On a related note, is there anything like the sql_slave_skip_counter variable implemented on a per-master basis? Or is this something else that would need separating out? I know I could just change the relevant master info file (carefully) with the new log position, but that's somewhat inconvenient and requires restarting the server.

Gordan

-- Senior MySQL Developer @ Taobao.com Mobile Phone: +86 18658156856 (Hangzhou) Gtalk: penglixun(at)gmail.com Twitter: http://www.twitter.com/plinux Blog: http://www.penglixun.com

Lixun Peng

9:54 a.m.

Hi Monty, On Fri, Nov 30, 2012 at 4:01 PM, Michael Widenius wrote:

...

Hi!

...
...
...
...
...
"Lixun" == Lixun Peng writes:

Lixun> Hi Gordan, Lixun> I'm not changed the replicate-[do|ignore]-[table|database] functionality Lixun> code, so I think it should work for all slave threads, but it can't specify Lixun> each master to use specific filter.

Yes, the replication filters are global for now.

It would probably be 2-3 days of work to make these per master.

The main questions is only how to define this in the config file. We could probably use the same thing we use for key buffers:

master_name.replicate-do-db=...

I think this way is ok. I agree with you.

...

Do you want some information of how to develop this or do you think you can find someone to sponsor this?

I can try to implement it. Are you doing it? Thanks, Lixun -- Senior MySQL Developer @ Taobao.com Mobile Phone: +86 18658156856 (Hangzhou) Gtalk: penglixun(at)gmail.com Twitter: http://www.twitter.com/plinux Blog: http://www.penglixun.com

4297

Age (days ago)

4301

Last active (days ago)

List overview

Download

9 comments

4 participants

participants (4)

Gordan Bobic
Lixun Peng
Michael Widenius
Sergei Golubchik

[Maria-developers] API Documentaion for Replication Features

tags

participants (4)