Hi,

From experience working on two storage engines on MySQL and being main developer for the second one, and one "storage engine" on big company, I would say MySQL storage engine API has some legacy problems and clearly some missing functionality. e.g.

- Some part of the storage engines are always the same, parameters are not fully given to functions or you really can't return proper error messages

- There is interface to get e.g. row type, but still you need to hack handler.cc/handler.h because some values are hard coded there (show create table)

- Storage engine API clearly shows that transaction concept has being introduced after the first API version was designed, still you can't really return errors on commit operation

- Storage engine can announce some that it can do some functionality to upper layer, but this is very limited. Storage engines should have more control what is done on MySQL parser/optimiser level and what is done on storage engine layer. E.g. for columnar storage engine, I would love to have opportunity to inform MySQL that I can do join's myself with all conditions pushed down to attributes, but that I can't do e.g. external sorting.

- For in-memory storage engine there should be API where MySQL data type and in-memory data types are exactly the same to avoid conversions, similarly in-memory buffer pool should contain directly usable types for MySQL.

Thus, as a conclusion, we really would need a better and cleaner API where storage engine has more control.

R: Jan

The storage engine API has never been my favorite part of MySQL. I haven't written a storage engine but I used to maintain a custom storage engine and migrated that from 4.0 to 5.0 so I had some experience with the API.

Is there a different API that would make it easier for innovation to come to MySQL/MariaDB? For example there is a lot going on in the write-optimized database space. Until recently we had nothing for MySQL open-source. Now we have open source TokuDB (great news). I haven't read their source but I have been told they use something that was derived from the BerkeleyDB API.

Would the BerkeleyDB API make it easier to get new storage engines into MySQL? For example, we don't have an engine optimized for in-memory workloads. And while InnoDB will have great cache hit rates when all data is in-memory. It might write much more data to disk than required.

Do we need a better API or is the current code good enough? I hope to hear from people who have done serious storage engine work.

--
Mark Callaghan
mdcallag@gmail.com
_______________________________________________
Mailing list: https://launchpad.net/~maria-developers
Post to     : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp