[Maria-developers] External replication proceeding
Monty: I received your email and am re-posting to the maria developers list. It is truly amazing that maria developers are HAPPY with replication as it stands right now. binary logs, Single SLAVE reading from individual masters, bla-bla-bla..... All the current replication code is like walking on the edge of a cliff, anything goes wrong and the replication process STOPS with an error. And a HUMAN BEING must wake up from their blissful sleep to get the process started again and figure out what happened. And I just LOVE having to: LOCK TABLES; RESET MASTER SET SLAVE OFF; MYSQLDUMP; COPY FILES TO SLAVE; SET MASTER TO 'sql_log_bin=45678945"; SET SLAVE ON; SECONDS BEHIND MASTER... WRITE A THESIS FOR MY PHD IN ROCKET SCIENCE Oh my GOD!!!!!! What an absolute nightmare of a design, it works, just not well. I have not made an in depth study of galera, but I refuse to make replication another research project, replication should be EASY, auto-healing, self analyzing, N-MASTER and with simple source code. I will open source the files that I use to change the MYSQLD daemon. The best way to debug MariaDB on Linux is to run the program in Netbeans 7.1.2. You can even give it command line parameters to work against a currently installed system with the data/conf files in place. Your trace idea is very nice, but there is nothing like STEPPING THROUGH the code in NetBeans/GDB in a nice IDE instead of cryptic command line GDB. Cheers, md On 1/2/2013 5:21 AM, Michael Widenius wrote:
Hi!
A better place to send emails like this is to maria-developers@lists.launchpad.net; That way you can get all MariaDB developers to answer you questions.
"md" == md <md@rpzdesign.com> writes: md> Hello: md> Does anybody know the name(s) of the source files I could find the code md> that processes the Table/Procedure Drop/Create/Update, Alter Table, etc md> statements.
md> I want to intercept these SQL statement calls in mysqld before md> they get sent to the specific storage engine like Aria or Xtradb.
sql_parse.cc proably contains the code you are looking for. Another place to look is in the the current replication code, ie the sql_repl.cc
md> I am working on an asynchronous external replication daemon md> that handles all of the data and structure synchronization md> replication issues (N-master-master, Hot Insert Master additions, DDL md> changes, etc)
Sounds very intersting. Is this a open source project ?
md> This is one of the missing pieces of the puzzle for MariaDB. md> (oh yeah, mysql does not exist to me anymore, only Aria/Xtradb)
:)
md> I need to have the mysqld daemon issue a signal that a structural change md> has taken place and the external replication daemon will pick it up md> and handle all the changes processing across ALL the masters (WITHOUT A md> HUMAN BEING INVOLVED!!!!)
md> I figured I would ask since most of you are fluent in the source md> code and just know where everything is, you would know right away.
That's ok, but best way to get quick responce is to ask on the maria-developers list or at #maria channel on Freenode.
md> I am using MariaDB 5.5.28a from source.
On thing you can do is:
./BUILD/compile-pentium-debug-max mysqld --debug
And then start 'mysql' and execute a CREATE TABLE command.
You can find then find a full trace of all executed functions in: /tmp/mysqld.trace
That's probably one of the best ways to learn how MariaDB works.
See:
https://kb.askmonty.org/en/creating-a-trace-file/
Regards, Monty
Hi
I received your email and am re-posting to the maria developers list.
It is truly amazing that maria developers are HAPPY with replication as it stands right now. binary logs, Single SLAVE reading from individual masters, bla-bla-bla.....
You are absolutely correct in that the classic replication mechanism with the binary log is by no means perfect, and in particular setup of a new slave can be a complete pest. That said, it was first implemented around 1999 and has served the web world extremely well. In fact, it was implemented on behalf of one particular client (non-recurring engineering). In development, inevitably priorities have to be set. As classic replication is a known entity for people, they can generally cope with its quirks. So rather than seeing it as a "must fix immediately", it's regarded as a "it'd be nice to replace this, over time". If it were to be designed and implemented today, particularly with the knowledge of how replication is getting used in the real world, and without any restrictions on budget/scope from a particular client, I'm sure it'd look very different - and no developer here would disagree with that basis analysis of the situation. In practical terms, Galera is one possible replacement - I say possible, because it works differently and therefore does not cover all specific use cases of classic replication. It is unlikely that any single alternative can replace all use cases, so I'd envisage that classic replication will continue to be used even if most people move to another mechanism. Galera can take care of most uses cases, and its implementation takes care of initialisation of new nodes and resynchronisation. The people at Codership identified the problems, worked on a new architecture, and delivered the Galera code to MariaDB. I am very impressed with their work and ongoing commitment. I suggest you do take a look at what Galera does. Does it satisfy the requirements for replication that you set? Based on your own email where you listed your requirements, I'm suggesting that it might. If it needs improving, why not contribute to the Galera code? Should you feel that something else is required after all, please go right ahead and create it - as you know MariaDB gladly accepts contributions. I look forward to seeing your code contributions. Regards, Arjen. (MariaDB captain) -- Arjen Lentz, Exec.Director @ Open Query (http://openquery.com) Australian peace of mind for your MySQL/MariaDB infrastructure. Follow us at http://openquery.com/blog/ & http://twitter.com/openquery
Hi!
"md" == md <md@rpzdesign.com> writes:
md> Monty: md> I received your email and am re-posting to the maria developers list. md> It is truly amazing that maria developers are HAPPY with replication as md> it stands right now. md> binary logs, Single SLAVE reading from individual masters, bla-bla-bla..... We are constantly working on making replication better. However with few resources it does take some time. - MariaDB 5.5 added group commit, which did speed up replication 3-5x for many common cases. - MariaDB 10.0 support already multiple slaves. - We are working on two different patches for parallel replication (one with taobao, on our own) that. The binary log is actually not a bad idea as it makes it easy to shift binary logs around and also support multiple storage engines. md> All the current replication code is like walking on the edge of a cliff, md> anything goes wrong md> and the replication process STOPS with an error. And a HUMAN BEING must md> wake up from their md> blissful sleep to get the process started again and figure out what md> happened. There are some improvement coming for that in MySQL 5.6 and MariaDB 10.0 md> And I just LOVE having to: md> LOCK TABLES; md> RESET MASTER md> SET SLAVE OFF; md> MYSQLDUMP; md> COPY FILES TO SLAVE; md> SET MASTER TO 'sql_log_bin=45678945"; md> SET SLAVE ON; md> SECONDS BEHIND MASTER... md> WRITE A THESIS FOR MY PHD IN ROCKET SCIENCE md> Oh my GOD!!!!!! This will be fixed in MySQL 5.6 and MariaDB 10.1 md> What an absolute nightmare of a design, it works, just not well. md> I have not made an in depth study of galera, but I refuse to make md> replication another research project, md> replication should be EASY, auto-healing, self analyzing, N-MASTER and md> with simple source code. Yes, it should. However to do that is a LOT of work and can't just be done over night. md> I will open source the files that I use to change the MYSQLD daemon. md> The best way to debug MariaDB on Linux is to run the program in Netbeans md> 7.1.2. md> You can even give it command line parameters to work against md> a currently installed system with the data/conf files in place. That is how I work daily on Linux; It's trivial to setup in any development environment. (Just one simple .my.cnf file in your home directory with the right paths and things works) md> Your trace idea is very nice, but there is nothing like STEPPING THROUGH md> the code md> in NetBeans/GDB in a nice IDE instead of cryptic command line GDB. I am using ddd for debugging code. However for understanding code flows and finding bugs that involve wrong data, the trace file is MUCH better than any debugger. Regards, Monty
participants (3)
-
Arjen Lentz -
md＠rpzdesign.com -
Michael Widenius