I've worked on the TokuDB storage engine for quite a while now. I have had many experiences over the years, so I guess it's hard to know where to begin. I guess I will start small, and if the conversation evolves, I can contribute more thoughts. I think the current API is really good, as evidenced by the fact that many storage engines have used it to plug into MySQL. The two areas that I see we can really benefit from are the following: - documentation of the API, discussing clearly what the contracts for the functions are. There are still several functions I am not fully clear on, but am glad that they "just work". This includes documentation on the locking mechanisms. For instance, clear documentation on what external_lock, store_lock, and other related functions do would be great. - general API cleanup. In too many places, the functions just assume things, such as the buffer passed into write_row is the same as table->record[0]. In other instances, the well defined way to answer certain questions is not obvious. For instance: - If I want information on the fields, do I use the table object or table_share? - There seem to be several ways to determine if a field is NULL - There is redundant information in the KEY* structure and the fields that make up the KEY. I recall when working on MySQL 5.1, this information was inconsistent and it led to bugs, such as MySQL bug 37292 - I am sure there are plenty of other instances of this that I can find. - Transactions. The API seems to not be designed with transactions taken into account. Transactions are started in some cases of external_lock, or in start_stmt. Transactions are committed either in the handlerton's commit function or in some cases in external_lock. A clear API that takes transactions into consideration would be quite helpful. Those are my early thoughts. I am happy to share more if people are interested. -Zardosht On Tue, Aug 13, 2013 at 6:41 PM, MARK CALLAGHAN <mdcallag@gmail.com> wrote:
The storage engine API has never been my favorite part of MySQL. I haven't written a storage engine but I used to maintain a custom storage engine and migrated that from 4.0 to 5.0 so I had some experience with the API.
Is there a different API that would make it easier for innovation to come to MySQL/MariaDB? For example there is a lot going on in the write-optimized database space. Until recently we had nothing for MySQL open-source. Now we have open source TokuDB (great news). I haven't read their source but I have been told they use something that was derived from the BerkeleyDB API.
Would the BerkeleyDB API make it easier to get new storage engines into MySQL? For example, we don't have an engine optimized for in-memory workloads. And while InnoDB will have great cache hit rates when all data is in-memory. It might write much more data to disk than required.
Do we need a better API or is the current code good enough? I hope to hear from people who have done serious storage engine work.
-- Mark Callaghan mdcallag@gmail.com
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp