I've worked on the TokuDB storage engine for quite a while now. I have
had many experiences over the years, so I guess it's hard to know
where to begin. I guess I will start small, and if the conversation
evolves, I can contribute more thoughts. I think the current API is
really good, as evidenced by the fact that many storage engines have
used it to plug into MySQL. The two areas that I see we can really
benefit from are the following:
- documentation of the API, discussing clearly what the contracts for
the functions are. There are still several functions I am not fully
clear on, but am glad that they "just work". This includes
documentation on the locking mechanisms. For instance, clear
documentation on what external_lock, store_lock, and other related
functions do would be great.
- general API cleanup. In too many places, the functions just assume
things, such as the buffer passed into write_row is the same as
table->record[0]. In other instances, the well defined way to answer
certain questions is not obvious. For instance:
- If I want information on the fields, do I use the table object or
table_share?
- There seem to be several ways to determine if a field is NULL
- There is redundant information in the KEY* structure and the
fields that make up the KEY. I recall when working on MySQL 5.1, this
information was inconsistent and it led to bugs, such as MySQL bug
37292
- I am sure there are plenty of other instances of this that I can find.
- Transactions. The API seems to not be designed with transactions
taken into account. Transactions are started in some cases of
external_lock, or in start_stmt. Transactions are committed either in
the handlerton's commit function or in some cases in external_lock. A
clear API that takes transactions into consideration would be quite
helpful.
Those are my early thoughts. I am happy to share more if people are interested.
-Zardosht
On Tue, Aug 13, 2013 at 6:41 PM, MARK CALLAGHAN
The storage engine API has never been my favorite part of MySQL. I haven't written a storage engine but I used to maintain a custom storage engine and migrated that from 4.0 to 5.0 so I had some experience with the API.
Is there a different API that would make it easier for innovation to come to MySQL/MariaDB? For example there is a lot going on in the write-optimized database space. Until recently we had nothing for MySQL open-source. Now we have open source TokuDB (great news). I haven't read their source but I have been told they use something that was derived from the BerkeleyDB API.
Would the BerkeleyDB API make it easier to get new storage engines into MySQL? For example, we don't have an engine optimized for in-memory workloads. And while InnoDB will have great cache hit rates when all data is in-memory. It might write much more data to disk than required.
Do we need a better API or is the current code good enough? I hope to hear from people who have done serious storage engine work.
-- Mark Callaghan mdcallag@gmail.com
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp