[Maria-developers] [GSoC] MDEV-4674 Port InnoDB memcached interface to MariaDB
Hello. My name is Piotr Jurkiewicz. I am a last year MSc student at AGH University of Science and Technology in Krakow, Poland. I already contributed to MySQL InnoDB Memcached plugin. I know it well, from user and developer points of view. I have been using MySQL InnoDB Memcached plugin since half year in a production system I administrate at my university. However, I must admit that quality of this plugin is rather poor. I have run several times into bugs, from the simple ones, like incorrect results of arithmetic operations, to the serious, like crashes of the whole MySQL process. I reported some of these bugs to Oracle and contributed patches fixing them [1] [2]. However, the more I was diving into the plugin code, the more I was realizing that the problem is not only buggy code and insufficient testing. The core mistakes had been made at the very beginning, when plugin architecture was set up. The most prominent result of these mistakes is lack of access security. After connecting to Memcached port, a user has access to all databases and tables exported by the plugin. Fixing this would require complete remodeling of plugin architecture and the way users configure and use it. This is not possible in MySQL, as it would break existing deployments. Therefore, I was very excited, when I saw that porting of InnoDB Memcached plugin is among MariaDB proposed GSoC ideas. This gives a possibility to build the plugin in a proper way. During the last months I have been thinking a lot how this plugin should be structured. I think I worked out a clear concept of that. With the financing from Google I will be able to dedicate sufficient amount of time for its implementation, what should result in fully featured, secure and robust Memcached plugin for MariaDB. You can review current version of my proposal in the Google Melange system: https://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2015/... As for now, it is only visible for MariaDB organization members, who are registered in Melange system. On March 27th, after student application deadline, I will make it visible for non-registered users as well and send its copy to this list. I am waiting for your remarks and hints. [1]: http://bugs.mysql.com/bug.php?id=74874 [2]: http://bugs.mysql.com/bug.php?id=75864 Piotr
Would it make sense to build a memcache plugin on top of the handler interface rather directly into innodb ? That way it "just work" for any storage engine. /Jonas On Tue, Mar 24, 2015 at 1:14 PM, Piotr Jurkiewicz < piotr.jerzy.jurkiewicz@gmail.com> wrote:
Hello.
My name is Piotr Jurkiewicz. I am a last year MSc student at AGH University of Science and Technology in Krakow, Poland. I already contributed to MySQL InnoDB Memcached plugin. I know it well, from user and developer points of view.
I have been using MySQL InnoDB Memcached plugin since half year in a production system I administrate at my university. However, I must admit that quality of this plugin is rather poor. I have run several times into bugs, from the simple ones, like incorrect results of arithmetic operations, to the serious, like crashes of the whole MySQL process. I reported some of these bugs to Oracle and contributed patches fixing them [1] [2]. However, the more I was diving into the plugin code, the more I was realizing that the problem is not only buggy code and insufficient testing. The core mistakes had been made at the very beginning, when plugin architecture was set up. The most prominent result of these mistakes is lack of access security. After connecting to Memcached port, a user has access to all databases and tables exported by the plugin. Fixing this would require complete remodeling of plugin architecture and the way users configure and use it. This is not possible in MySQL, as it would break existing deployments.
Therefore, I was very excited, when I saw that porting of InnoDB Memcached plugin is among MariaDB proposed GSoC ideas. This gives a possibility to build the plugin in a proper way. During the last months I have been thinking a lot how this plugin should be structured. I think I worked out a clear concept of that. With the financing from Google I will be able to dedicate sufficient amount of time for its implementation, what should result in fully featured, secure and robust Memcached plugin for MariaDB.
You can review current version of my proposal in the Google Melange system: https://www.google-melange.com/gsoc/proposal/review/ student/google/gsoc2015/piotrjurkiewicz/5743522325987328
As for now, it is only visible for MariaDB organization members, who are registered in Melange system. On March 27th, after student application deadline, I will make it visible for non-registered users as well and send its copy to this list.
I am waiting for your remarks and hints.
[1]: http://bugs.mysql.com/bug.php?id=74874 [2]: http://bugs.mysql.com/bug.php?id=75864
Piotr
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
On 24 Mar 2015, at 22:06, Jonas Oreland <jonaso@google.com> wrote:
Would it make sense to build a memcache plugin on top of the handler interface rather directly into innodb ? That way it "just work" for any storage engine.
Would that not reduce the performance that the InnoDB memcached plugin now offers? Also, if the idea is to make it more secure, and differ from MySQL's implementation (i.e. make it better), if we can get similar performance from the HANDLER interface, why not? (I was under the impression we just wanted to port it, to have feature compatibility; but can't argue with better) -- Colin Charles, Chief Evangelist, MariaDB Corporation blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles
On 2015-03-24 15:52, Colin Charles wrote:
Would that not reduce the performance that the InnoDB memcached plugin now offers?
It is an interesting question. For some reason original plugin authors decided to build it directly on InnoDB API, rather than on Handler API. I think it's all a matter of trade-offs. Having direct interfaces using engine-specific APIs (like MySQL plugin does it now): + (probably) higher performance can be achieved + will explore different approach then MariaDB HandlerSocket plugin - requires creating interface for every engine Having universal Handler API interface: + requires a lot less work to introduce support for a new engine - (probably) lower performance can be achieved - makes Memcached plugin in fact duplicate of HandlerSocket plugin, but speaking different wire protocol It is possible to combine the best of both worlds: 1. Leave InnoDB memcached plugin interface, as it is already done. (And eventually port it to XtraDB, as it would be relatively straightforward.) 2. Create Handler API memcached plugin interface. As a result, InnoDB tables will be accessible using InnoDB API or Handler API (configurable in containers table?) Other engines tables will be accessible only using Handler API. This will allow performance comparisons between these two approaches in case of InnoDB tables. After some time it would eventually clarify which approach is better. Piotr
On Tue, Mar 24, 2015 at 3:52 PM, Colin Charles <colin@mariadb.org> wrote:
On 24 Mar 2015, at 22:06, Jonas Oreland <jonaso@google.com> wrote:
Would it make sense to build a memcache plugin on top of the handler interface rather directly into innodb ? That way it "just work" for any storage engine.
Would that not reduce the performance that the InnoDB memcached plugin now offers?
i think that performance reduction would be minimal *iff* providing equally good implementations. however, it's easier to make it on top of a API that you know better...i think that is why innodb made it on top of innodb.
Also, if the idea is to make it more secure, and differ from MySQL's implementation (i.e. make it better), if we can get similar performance from the HANDLER interface, why not? (I was under the impression we just wanted to port it, to have feature compatibility; but can't argue with better)
i have no opinion about "why", just made a suggestion on "how". /Jonas
This is a really interesting discussion, since the model we are describing resembles closely to HandlerSocket Plugin (since it is build upon Handler API instead of Storage Engine) it led me to research some more about Benchmarking test involving Memcached + InnoDB vs HandlerSocket. With *MySql version 5.6* Benchmarking test had given to significant advantage to HandlerSocket (processing 750,000 Qps) over Memcached InnoDB plugin (420,000 QPS). here: https://github.com/entering/miscellaneous/blob/master/benchmarks/handler-soc... *But*, with in the recent test made my MySql team on *MySql version 5.7*, there was significant improvement with Memcached - InnoDB plugin now able to process (*1,000,000 Query/sec (QPS)) , *so in my view point it will be better to work with direct Storage Engine access instead of building it on top of Handler API. Link to the recent benchmarking test with version 5.7: http://dimitrik.free.fr/blog/archives/2013/11/mysql-performance-over-1m-qps-... Also as an excellent point raised by *Piotr* making Memcache Plugin will duplicate of HandlerSocket plugin, but speaking different wire protocol , which definitely won't give us much advantage . For more feature by Mysql 5.7 InnoDB Memcached plugin : http://dev.mysql.com/doc/refman/5.7/en/innodb-memcached-intro.html Will be happy to hear any counter questions/points about it. --Aalekh On Wed, Mar 25, 2015 at 6:09 AM, Jonas Oreland <jonaso@google.com> wrote:
On Tue, Mar 24, 2015 at 3:52 PM, Colin Charles <colin@mariadb.org> wrote:
On 24 Mar 2015, at 22:06, Jonas Oreland <jonaso@google.com> wrote:
Would it make sense to build a memcache plugin on top of the handler interface rather directly into innodb ? That way it "just work" for any storage engine.
Would that not reduce the performance that the InnoDB memcached plugin now offers?
i think that performance reduction would be minimal *iff* providing equally good implementations. however, it's easier to make it on top of a API that you know better...i think that is why innodb made it on top of innodb.
Also, if the idea is to make it more secure, and differ from MySQL's implementation (i.e. make it better), if we can get similar performance from the HANDLER interface, why not? (I was under the impression we just wanted to port it, to have feature compatibility; but can't argue with better)
i have no opinion about "why", just made a suggestion on "how".
/Jonas
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
On 2015-03-25 03:29, Aalekh Nigam wrote:
With *MySql version 5.6* Benchmarking test had given to significant advantage to HandlerSocket (processing 750,000 Qps) over Memcached InnoDB plugin (420,000 QPS). here: https://github.com/entering/miscellaneous/blob/master/benchmarks/handler-soc...
In the test, which results you cite, Yoshinori Matsunobu was comparing HandlerSocket to standalone Memcached, not InnoDB Memcached plugin. See: http://yoshinorimatsunobu.blogspot.pt/2010/10/using-mysql-as-nosql-story-for... .
Link to the recent benchmarking test with version 5.7: http://dimitrik.free.fr/blog/archives/2013/11/mysql-performance-over-1m-qps-...
You cannot compare these benchmarks, because Yoshinori and DimitriK were using very different machines. Yoshinori was performing his tests on 2.5GHz x 8 core Nehalem, whereas DimitriK was using 2Ghz * 48 core machine. Piotr
As student application deadline is over, I have made my proposal public: http://www.google-melange.com/gsoc/proposal/public/google/gsoc2015/piotrjurk... Everyone should be able to see it now, without logging in to the Melange system. Piotr
Thank for pointing this out, I have now attempted my own Benchmarking test under Amazon EC2 Environment , which concluded into following: On Running the two plugins with 10,000 Queries: - Get Operation with HandlerSocket was completed in 1.20499610901 Sec, i.e; making 8,299 QPS (Queries per second) - Get Operation with InnoDB Memcached Plugin was completed in .93115 seconds i.e; making 10,739 QPS - Select Query for MySQL took 333.834 seconds which is considerably low Also you guys can read more about this experiment in my blog post here: http://mimdl.tumblr.com/post/115308901909/benchmarking-handlersocket-mysql-i... --Aalekh On Mon, Mar 30, 2015 at 4:45 AM, Piotr Jurkiewicz < piotr.jerzy.jurkiewicz@gmail.com> wrote:
As student application deadline is over, I have made my proposal public:
http://www.google-melange.com/gsoc/proposal/public/google/ gsoc2015/piotrjurkiewicz/5743522325987328
Everyone should be able to see it now, without logging in to the Melange system.
Piotr
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
participants (4)
-
Aalekh Nigam -
Colin Charles -
Jonas Oreland -
Piotr Jurkiewicz