Re: [Maria-discuss] client library shared library version changed in 5.5.10
Hi, Monty!
On Mar 18, Michael Widenius wrote:
"MARK" == MARK CALLAGHAN mdcallag@gmail.com writes:
I think there was discussion about how client library shared library versions during the Lisbon meeting.
From http://dev.mysql.com/doc/refman/5.5/en/news-5-5-10.html
Incompatible Change: The shared library version of the client library was increased to 18 to reflect ABI changes, and avoid compatibility problems with the client library in MySQL 5.1. Note that this is an incompatible change between 5.5.10 and earlier 5.5 versions, so client programs that use the 5.5 client library should be recompiled against the 5.5.10 client library.
Serg, as you have now probably have full insight into the MariaDB 5.5 code, is it possible to keep the API compatible with MariaDB 5.1 / MySQL 5.1 to not force all applications to have to be recompiled when upgrading to 5.5 ?
The API incompatibility was caused by migration to CMake. If you want to keep libraries compatible, if's something that Wlad may need to look into.
Regards, Sergei
-----Original Message----- From: Sergei Golubchik [mailto:serg@askmonty.org] Sent: Sonntag, 27. März 2011 21:26 To: monty@askmonty.org Cc: MARK CALLAGHAN; maria-discuss@lists.launchpad.net; wlad@montyprogram.com Subject: Re: [Maria-discuss] client library shared library version changed
in
5.5.10
Hi, Monty!
On Mar 18, Michael Widenius wrote:
> "MARK" == MARK CALLAGHAN mdcallag@gmail.com writes:
I think there was discussion about how client library shared library versions during the Lisbon meeting.
From http://dev.mysql.com/doc/refman/5.5/en/news-5-5-10.html
Incompatible Change: The shared library version of the client library was increased to 18 to reflect ABI changes, and avoid compatibility problems with the client library in MySQL 5.1. Note that this is an incompatible change between 5.5.10 and earlier 5.5 versions, so client programs that use the 5.5 client library should be recompiled against the 5.5.10 client library.
Serg, as you have now probably have full insight into the MariaDB 5.5 code, is it possible to keep the API compatible with MariaDB 5.1 / MySQL 5.1 to not force all applications to have to be recompiled when upgrading to 5.5 ?
The API incompatibility was caused by migration to CMake. If you want to keep libraries compatible, if's something that Wlad may need to look into.
I do not really think it was CMake that broke it. The "official" ABI has not changed, however some people claim that API and ABI are 2 different things ;)
For 5.5 the discussion starts here http://bugs.mysql.com/bug.php?id=59078 ([9 Feb 2:26] Clint Byrum, IIRC he works for Debian). His argument that ABI has changed "dramatically" is that for example an exported symbol named TERMINATE has gone, while new symbols PSI_hook and PSI_server were added. He uses "nm" for his argumentation.
Clint's claim is that every time an exported symbol is changed/removed or added, shared library version must be bumped. Unix linkers export every global function and variable by default. Which means, if one renames a global variable as part of cleanup, the ABI is changed. Changes like that can happen really in any point release, so I see no other way to maintain the 100% ABI compatibility, rather than restricting visibility to only symbols defined by client API (i.e mysql_xxx stuff) . On some Unixes, it is possible to do with linker version script .On Windows symbols were restricted to the official API from the start on , using with the .def file.
This would be a clean solution from my point of view. However, should we go for it, this would result in incompatibilities with Oracle's MySQL, and this has to be worked out somehow (best thing is probably contribute visibility-restricting patch, if they are at all interested) . Besides , Unix users need to be warned that shared and static library are not interchangeable and that it will not be possible to access e.g mysys/strings/dbug functionality using the shared library. Hopefully most of the programs are written using officially documented client API functions, so there won't be many surprises.
Regards, Wlad
Hi Vlad, all
On 28/03/2011, at 8:18 AM, Vladislav Vaintroub wrote:
For 5.5 the discussion starts here http://bugs.mysql.com/bug.php?id=59078 ([9 Feb 2:26] Clint Byrum, IIRC he works for Debian). His argument that ABI has changed "dramatically" is that for example an exported symbol named TERMINATE has gone, while new symbols PSI_hook and PSI_server were added. He uses "nm" for his argumentation.
Clint's claim is that every time an exported symbol is changed/ removed or added, shared library version must be bumped. Unix linkers export every global function and variable by default. Which means, if one renames a global variable as part of cleanup, the ABI is changed. Changes like that can happen really in any point release, so I see no other way to maintain the 100% ABI compatibility, rather than restricting visibility to only symbols defined by client API (i.e mysql_xxx stuff) . On some Unixes, it is possible to do with linker version script .On Windows symbols were restricted to the official API from the start on , using with the .def file.
This would be a clean solution from my point of view. However, should we go for it, this would result in incompatibilities with Oracle's MySQL, and this has to be worked out somehow (best thing is probably contribute visibility-restricting patch, if they are at all interested) .
I think it makes sense to use a linker version script just like using .def on Windows. There's a clearly defined (and documented) set of functions and variables, and nothing else should be exposed.
Obviously if some environments don't support doing this for a library then there's not much we can do (apart from not having global variables! ;-) but otherwise it works out fine and also satisfies Clint's concerns. I figure libmysqlclient is not the only thing Debian has reviewed, so apparently the many other libraries do apply a methodology that restricts undesired symbol exposure.
So, no need to invent the wheel there; if others do this, it'll be plainly visible. Look at another library and see how it's done there. Do the same and indeed let's submit that cleanup back to Oracle in a public bugreport.
Besides , Unix users need to be warned that shared and static library are not interchangeable and that it will not be possible to access e.g mysys/strings/dbug functionality using the shared library. Hopefully most of the programs are written using officially documented client API functions, so there won't be many surprises.
They should, otherwise all kinds of things can go wrong anyway.
Regards, Arjen.
Hi, Vladislav!
From http://dev.mysql.com/doc/refman/5.5/en/news-5-5-10.html
>
Incompatible Change: The shared library version of the client library was increased to 18 to reflect ABI changes, and avoid compatibility problems with the client library in MySQL 5.1. Note that this is an incompatible change between 5.5.10 and earlier 5.5 versions, so client programs that use the 5.5 client library should be recompiled against the 5.5.10 client library.
Serg, as you have now probably have full insight into the MariaDB 5.5 code, is it possible to keep the API compatible with MariaDB 5.1 / MySQL 5.1 to not force all applications to have to be recompiled when upgrading to 5.5 ?
The API incompatibility was caused by migration to CMake. If you want to keep libraries compatible, if's something that Wlad may need to look into.
I do not really think it was CMake that broke it. The "official" ABI has not changed, however some people claim that API and ABI are 2 different things ;)
The main source of the breakage is migration to CMake: making libmysqlclient.so in 5.5 cmake builds don't use version script, and don't use -Wl,--whole-archive.
I've run the gcc -shared line manually in 5.5 with the version script, whole-archive, and the version changed to 16. And mysql client from 5.1 happily loaded the resulted library.
Regards, Sergei
P.S. I had to do one more change, related to removed (from 5.5) safemalloc, I added this to libmysql.c:
void my_no_flags_free(void *ptr) { free(ptr); }
but as we are going to restore safemalloc in our 5.5, this won't be an issue.
P.P.S.: the api was extended in 5.5, true. but it's not a reason to break backward compatibility - it was extended in a perfectly backward compatible way.
-----Original Message----- From: Sergei Golubchik [mailto:serg@askmonty.org] Sent: Montag, 28. März 2011 10:46 To: Vladislav Vaintroub Cc: monty@askmonty.org; maria-discuss@lists.launchpad.net Subject: Re: [Maria-discuss] client library shared library version changed
in
5.5.10
Hi, Vladislav!
From http://dev.mysql.com/doc/refman/5.5/en/news-5-5-10.html >>
Incompatible Change: The shared library version of the client library was increased to 18 to reflect ABI changes, and avoid compatibility problems with the client library in MySQL 5.1. Note that this is an incompatible change between 5.5.10 and earlier 5.5 versions, so client programs that use the 5.5 client library should be recompiled against the 5.5.10 client library.
Serg, as you have now probably have full insight into the MariaDB
5.5
code, is it possible to keep the API compatible with MariaDB 5.1 / MySQL 5.1 to not force all applications to have to be recompiled
when
upgrading to 5.5 ?
The API incompatibility was caused by migration to CMake. If you want to keep libraries compatible, if's something that Wlad may need to look into.
I do not really think it was CMake that broke it. The "official" ABI
has
not changed, however some people claim that API and ABI are 2 different things ;)
The main source of the breakage is migration to CMake: making libmysqlclient.so in 5.5 cmake builds don't use version script, and don't use -Wl,--whole-archive.
Yep, version script is likely a factor. Well spotted, thanks!
I've run the gcc -shared line manually in 5.5 with the version script, whole-archive, and the version changed to 16. And mysql client from 5.1 happily loaded the resulted library.
I think you would agree that our own utilities (mysql client for example) were not designed with well-designed shared libraries or backward compatibility in mind. Except officially documented API, they use a whole bunch of undocumented mysys/dbug/strings functionality. That stuff is not sacred, it is not found in the API header files, and it is subject to changes and refactoring, as you also noticed in case of missing my_no_flags_free(). Client utilities cannot link to shared client library on Windows and even on Linux they require "export everything"-strategy, which you fix with non-portable -Wl,--whole-archive. Personally, I would not support any attempts to link our client utilities with the shared library. We do not do it, and others should not attempt it either
Wlad
Hi, Vladislav!
The main source of the breakage is migration to CMake: making libmysqlclient.so in 5.5 cmake builds don't use version script, and don't use -Wl,--whole-archive.
Yep, version script is likely a factor. Well spotted, thanks!
I've run the gcc -shared line manually in 5.5 with the version script, whole-archive, and the version changed to 16. And mysql client from 5.1 happily loaded the resulted library.
I think you would agree that our own utilities (mysql client for example) were not designed with well-designed shared libraries or backward compatibility in mind. Except officially documented API, they use a whole bunch of undocumented mysys/dbug/strings functionality. That stuff is not sacred, it is not found in the API header files, and it is subject to changes and refactoring, as you also noticed in case of missing my_no_flags_free(). Client utilities cannot link to shared client library on Windows and even on Linux they require "export everything"-strategy, which you fix with non-portable -Wl,--whole-archive. Personally, I would not support any attempts to link our client utilities with the shared library. We do not do it, and others should not attempt it either
I think I'd prefer our command-line tools (excluding mysqlbinlog) to be just "normal" client, using our client shared library.
I don't like -Wl,--whole-archive and prefer a controlled export approach. But I think (not sure, and I hasn't tried) that the only problem is my_getopt - which de-facto was always part of the API, but in mysql-5.5 it's not exported. Our command-line utilites use it. I suspect that adding my_getopt function to the CLIENT_API_FUNCTIONS list, would've solved the issue. my_getopt has a great general applicability, I can easily see other clients that may want to use it (I would certainly use it if I were writing MySQL client).
And we need to restore the version script. It's still there - in libmysql/libmysql.ver.in - but it's not used. Which made the bug#3074 to re-appear.
Regards, Sergei
"Vladislav Vaintroub" wlad@montyprogram.com writes:
For 5.5 the discussion starts here http://bugs.mysql.com/bug.php?id=59078 ([9 Feb 2:26] Clint Byrum, IIRC he works for Debian). His argument that ABI has changed "dramatically" is that for example an exported symbol named TERMINATE has gone, while new symbols PSI_hook and PSI_server were added. He uses "nm" for his argumentation.
I think you misread him. His main argument I think is that a program compiled/linked against the 5.1 version of libmysqlclient does not work with the 5.5 version of the library. This breaks established standards for how Unix .so version numbering works with respect to compatibility, where same major version number implies backwards compatibility.
It's sad if we introduced binary incompatibility only due to accidentally building in a different way... however, I fear that if now MySQL/Oracle already decided to bump the major version number, then it will only cause even more confusion if we did something different in MariaDB...
- Kristian.
participants (4)
-
Arjen Lentz -
Kristian Nielsen -
Sergei Golubchik -
Vladislav Vaintroub