On Fri, Dec 16, 2022 at 3:46 AM Jonathan M. Wilbur <jonathan@wilbur.space> wrote:
Sorry to ask this question here, since it is not strictly MariaDB-related, but I need the knowledge of people that develop with databases: is there significant overhead to using a DBMS versus a storage engine directly?
I was thinking about this recently. For some predetermined access patterns, it might pay off to translate the SQL statements directly into executable code that would directly access the buffer pool (page cache) of the storage engine. In terms of InnoDB, the generated code could work on the level of mini-transactions (atomic access to multiple buffer pages). The "class handler" interface that HandlerSocket uses still involves conversions between row formats, for example. I did not look into details, but https://github.com/KowalskiThomas/LLVMSQLite seemed interesting to me. Of course, we have to keep in mind that SQLite is a much simpler architecture, maybe not at all suitable for high-concurrency applications, but it could make sense in the embedded space or in some read-mostly settings. In a general purpose database, some just-in-time compilation could help with more complex queries. It would be a trade-off between memory management, time to compile, and time to run the compiled vs. interpreted code path. For a stream of simple arbitrary queries, interpreting might win. Best regards, Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation