[Maria-developers] WL#3859: Plug-in Service API discussion
While we were discussing a generic I/O layer in MariaDB, Sergei brought up the issue of WL#3859. This worklog carries a principal issue that I wanted to mention and discuss. Sergei, if I understood you correctly, you said that Sun MySQL would never take a new internal API that does not follow the WL#3859 framework. Correct? My understanding (which may or may not be correct) is that at least part of MySQL/Sun wants to develop the storage engine plugin layer (and all other plugin layers) to be a complete ABI, that is complete binary compatibility. So that the user can obtain a pre-compiled plugin independently and load it into the server without risk of crashing if plugin is compiled against a different source version of the server (but may get an error message if versions are incompatible). This basically means that all data and code needs to be exported via the WL#3859 framework, not using the system linker as is currently done. Advantages include - Avoiding crash on incompatible version is good. - Supports selling proprietary (non Open Source) plugins. - Works on Windows where the system linker cannot export symbols from executables (though there is a standard workaround of just compiling the server into a library that is called from a stub main executable). Disadvantages include - Supports selling proprietary (non Open Source) plugins (yes, can be an advantage or disadvantage depending on point-of-view ;-). - Will tend to discourage making desirable changes, as the procedure becomes more complicated (more work to track versioned API, less flexibitily in what C++ language constructs can be used); and there will be the desire to minimize binary incompatibility. It would be great if some of you still inside Sun could confirm or reject that this is the way things are. This raises at least two important questions for MariaDB: 1. Do we want to adopt this direction, moving all existing exposed server internals into this framework, and implementing any new ones within it as well? Or reserve the right to use old-style direct linking where it makes more sense? (We can of course in any case use the WL#3859 framework where it makes sense, the issue is the principle of using it for everything eventually). 2. It seems impossible that we can keep the Sun/MySQL API versions and the MariaDB API versions 100% synchronised. So how will we handle version compatibility (ie. Sun/MySQL version 0x0109 may be different from Maria version 0x0109 due to different paces of development)? Will all plugins need to track two different versions, one for Sun/MySQL and one for MariaDB? Or will we somehow coordinate between the two teams to avoid version conflicts? Any opinions? My personal opinion is that I am sceptical if this worklog is a good idea; most of the big projects of this kind that have been successful have used the old MySQL approach (Apache, Perl, the Linux kernel). Though I'm not 100% decided one way or the other yet. But more importantly I think we need to have a conscious decision on how to deal with this in MariaDB, as it is a fundamental design issue that will impact many other desicions along the way. - Kristian.
Hi, Kristian! On Feb 27, Kristian Nielsen wrote:
While we were discussing a generic I/O layer in MariaDB, Sergei brought up the issue of WL#3859. This worklog carries a principal issue that I wanted to mention and discuss.
Sergei, if I understood you correctly, you said that Sun MySQL would never take a new internal API that does not follow the WL#3859 framework. Correct?
I won't speak for the whole "Sun MySQL" that will "never" do something. But the current goal is that everything that needs to be exported will be exported as a service.
My understanding (which may or may not be correct) is that at least part of MySQL/Sun wants to develop the storage engine plugin layer (and all other plugin layers) to be a complete ABI, that is complete binary compatibility. So that the user can obtain a pre-compiled plugin independently and load it into the server without risk of crashing if plugin is compiled against a different source version of the server (but may get an error message if versions are incompatible).
Yes.
This basically means that all data and code needs to be exported via the WL#3859 framework, not using the system linker as is currently done.
Yes.
Advantages include
- Avoiding crash on incompatible version is good. - Supports selling proprietary (non Open Source) plugins. - Works on Windows where the system linker cannot export symbols from executables (though there is a standard workaround of just compiling the server into a library that is called from a stub main executable).
Disadvantages include
- Supports selling proprietary (non Open Source) plugins (yes, can be an advantage or disadvantage depending on point-of-view ;-). - Will tend to discourage making desirable changes, as the procedure becomes more complicated (more work to track versioned API, less flexibitily in what C++ language constructs can be used); and there will be the desire to minimize binary incompatibility.
On the opposite, it will be easier to do changes, because versioning will make them safe and we could have old and new versions supported at the same time - that we we can add new features to a service without causing binary incompatibility with the existing plugins.
It would be great if some of you still inside Sun could confirm or reject that this is the way things are.
That's the direction we want to move (and are moving) in.
This raises at least two important questions for MariaDB:
1. Do we want to adopt this direction, moving all existing exposed server internals into this framework, and implementing any new ones within it as well? Or reserve the right to use old-style direct linking where it makes more sense? (We can of course in any case use the WL#3859 framework where it makes sense, the issue is the principle of using it for everything eventually).
2. It seems impossible that we can keep the Sun/MySQL API versions and the MariaDB API versions 100% synchronised. So how will we handle version compatibility (ie. Sun/MySQL version 0x0109 may be different from Maria version 0x0109 due to different paces of development)? Will all plugins need to track two different versions, one for Sun/MySQL and one for MariaDB? Or will we somehow coordinate between the two teams to avoid version conflicts?
Any opinions?
My personal opinion is that I am sceptical if this worklog is a good idea; most of the big projects of this kind that have been successful have used the old MySQL approach (Apache, Perl, the Linux kernel). Though I'm not 100% decided one way or the other yet.
But more importantly I think we need to have a conscious decision on how to deal with this in MariaDB, as it is a fundamental design issue that will impact many other desicions along the way.
Regards / Mit vielen Grüßen, Sergei -- __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sergei Golubchik <serg@sun.com> / /|_/ / // /\ \/ /_/ / /__ Principal Software Engineer/Server Architect /_/ /_/\_, /___/\___\_\___/ Sun Microsystems GmbH, HRB München 161028 <___/ Sonnenallee 1, 85551 Kirchheim-Heimstetten Geschäftsführer: Thomas Schroeder, Wolfgang Engels, Dr. Roland Boemer Vorsitzender des Aufsichtsrates: Martin Häring
Sergei Golubchik <serg@mysql.com> writes:
On Feb 27, Kristian Nielsen wrote:
This basically means that all data and code needs to be exported via the WL#3859 framework, not using the system linker as is currently done.
Yes.
- Will tend to discourage making desirable changes, as the procedure becomes more complicated (more work to track versioned API, less flexibitily in what C++ language constructs can be used); and there will be the desire to minimize binary incompatibility.
On the opposite, it will be easier to do changes, because versioning will make them safe and we could have old and new versions supported at the same time - that we we can add new features to a service without causing binary incompatibility with the existing plugins.
That's the direction we want to move (and are moving) in.
Thanks for the info Sergei, that helped quite a bit. At first I though this could introduce limitations. I am thinking for example of a thing like condition pushdown, where storage engine plugins essentially have access to the complete item tree. Having to wrap everything in item in an abstration layer doesn't seem attractive, but now I'm starting to see that it does not necessarily have to be done that way. In principle under WL#3859 we could just publish an API that exposes the item tree and bump the version whenever anything changes in the implementation. (of course having a cleaner condition pushdown API could be a good thing in itself, but the important thing is that the WL#3859 interface is flexible and does not restrict anything). The ability to check at load time for compatibility (rather than crash) is certainly attractive...
2. It seems impossible that we can keep the Sun/MySQL API versions and the MariaDB API versions 100% synchronised. So how will we handle version compatibility (ie. Sun/MySQL version 0x0109 may be different from Maria version 0x0109 due to different paces of development)? Will all plugins need to track two different versions, one for Sun/MySQL and one for MariaDB? Or will we somehow coordinate between the two teams to avoid version conflicts?
I talked to Mats Kindahl, he said the code is not pushed anywhere available yet. But I plan to take a closer look when it is, and think more about the versioning then. Maybe it is as simple as just saying MySQL, MariaDB, Drizzle etc. are not plugin binary compatible. Thanks, - Kristian.
Hi, Kristian! On Mar 11, Kristian Nielsen wrote:
Sergei Golubchik <serg@mysql.com> writes:
- Will tend to discourage making desirable changes, as the procedure becomes more complicated (more work to track versioned API, less flexibitily in what C++ language constructs can be used); and there will be the desire to minimize binary incompatibility.
On the opposite, it will be easier to do changes, because versioning will make them safe and we could have old and new versions supported at the same time - that we we can add new features to a service without causing binary incompatibility with the existing plugins.
That's the direction we want to move (and are moving) in.
Thanks for the info Sergei, that helped quite a bit.
At first I though this could introduce limitations. I am thinking for example of a thing like condition pushdown, where storage engine plugins essentially have access to the complete item tree. Having to wrap everything in item in an abstration layer doesn't seem attractive, but now I'm starting to see that it
Still that's what Timour is working on :)
does not necessarily have to be done that way. In principle under WL#3859 we could just publish an API that exposes the item tree and bump the version whenever anything changes in the implementation. (of course having a cleaner condition pushdown API could be a good thing in itself, but the important thing is that the WL#3859 interface is flexible and does not restrict anything).
Right. Regards / Mit vielen Grüßen, Sergei -- __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sergei Golubchik <serg@sun.com> / /|_/ / // /\ \/ /_/ / /__ Principal Software Engineer/Server Architect /_/ /_/\_, /___/\___\_\___/ Sun Microsystems GmbH, HRB München 161028 <___/ Sonnenallee 1, 85551 Kirchheim-Heimstetten Geschäftsführer: Thomas Schroeder, Wolfgang Engels, Dr. Roland Boemer Vorsitzender des Aufsichtsrates: Martin Häring
participants (2)
-
Kristian Nielsen -
Sergei Golubchik