[Maria-developers] Shared plugin library build question
I am looking at a difference in the HP-UX vs. Linux mariadb build and was wondering if someone could help me understand why something is set up the way it is. The problem I see is that on HP-UX, when I do a mariadb build I do not get shared plugin libraries like ha_federated. Looking into why this was happening I found that when the plugin libraries like ha_federated are built they have a dependency on libsqlservices and libsqlservices is built only as an archive library. HP-UX doesn't support building shared libaries that depend on archive libraries and so the build of the shared plugin libraries fails when no shared version of libsqlservices is found. So my main question is: why is libsqlservices built only as an archive library? The other question is: if libsqlservices has to be an archive only library, is there is another way to link it in to executables, such as linking it in explicitly when building an executable instead of counting on the shared plugin libraries to bring it in indirectly by making it a dependent library? Steve Ellcey sje@cup.hp.com
Hi, Steve! On Sep 22, Steve Ellcey wrote:
I am looking at a difference in the HP-UX vs. Linux mariadb build and was wondering if someone could help me understand why something is set up the way it is.
The problem I see is that on HP-UX, when I do a mariadb build I do not get shared plugin libraries like ha_federated. Looking into why this was happening I found that when the plugin libraries like ha_federated are built they have a dependency on libsqlservices and libsqlservices is built only as an archive library. HP-UX doesn't support building shared libaries that depend on archive libraries and so the build of the shared plugin libraries fails when no shared version of libsqlservices is found.
Oops. It's a problem.
So my main question is: why is libsqlservices built only as an archive library?
Yes.
The other question is: if libsqlservices has to be an archive only library, is there is another way to link it in to executables, such as linking it in explicitly when building an executable instead of counting on the shared plugin libraries to bring it in indirectly by making it a dependent library?
No. I'll explain what it is and how it works. In MariaDB there is a concept of "Services". A Service is simply a group of related function implemented in the server that logically belong together. A plugin uses them normally, by calling, say, somefunc(1,2,3). But "somefunc" for dynamic plugins is defined (in services.h) as #define somefunc(A,B,C) some_service->func(A,B,C) and if a plugin uses somefunc(), it will need some_service symbol. Now, libmysqlservices provides it, defined as void *some_service = (void*)1; This number is the *version of the service some_services". And it's not really 1, I've just used it as an example. When a plugin is loaded into the server, the server changes the value of the "some_service" symbol in the plugin to point to the structure with the function pointers, so when some_service->func(1,2,3) is finally executed, it works normally, as expected from an ordinary function call. But before replacing the value of the "some_service" symbol the server checks the version number to verify that the number stored in the plugin (version of the service that plugin requires) is compatible with the version of this service as provided by the server. In case of a mismatch the server will refuse to load a plugin. The trick is - because of the way a linker works, there may be hundreds of services (numbers) in libmysqlservices.a, but only those that the plugin really uses will go into the plugin dynamic library, and any incompatibilities in other services won't prevent plugin from being loaded. It's automatic dependency tracking. Anyway, for all this to work, the server must not be linked with libmysqlservices.a. But every dynamic plugin must be - because every dynamic plugin must carry version numbers for all services that it uses, versions of the library with which a plugin was compiled, not versions with which it will be run. Because one can compile a plugin with one version of the MariaDB and use with another - and it's up to the server to detect a possible incompatibility. At the moment I don't have a working solution that will work for for HP-UX :( I need to try some ideas and see which one will work. Regards, Sergei
Steve, Are you talking about PIC requirements, when you say " HP-UX doesn't support building shared libaries that depend on archive libraries". In MySQL 5.5, linking static to shared feature was used extensively (as a replacement for libtool's "convenience libraries", static library was compiled with platform dependent PIC flag), and it worked on all major Unixes, I'm pretty sure it worked on HPUX/Itanium. I think the trick for Maria 5.3 could be to compile the sqlservices library with the PIC flag (+Z or +z or something similar, if I recall this correctly). I do not have access to HPUX machines anymore, but you might want to give it a try.
-----Original Message----- From: maria-developers- bounces+wlad=montyprogram.com@lists.launchpad.net [mailto:maria- developers-bounces+wlad=montyprogram.com@lists.launchpad.net] On Behalf Of Steve Ellcey Sent: Donnerstag, 22. September 2011 23:12 To: maria-developers@lists.launchpad.net Subject: [Maria-developers] Shared plugin library build question
I am looking at a difference in the HP-UX vs. Linux mariadb build and was wondering if someone could help me understand why something is set up the way it is.
The problem I see is that on HP-UX, when I do a mariadb build I do not get shared plugin libraries like ha_federated. Looking into why this was happening I found that when the plugin libraries like ha_federated are built they have a dependency on libsqlservices and libsqlservices is built only as an archive library. HP-UX doesn't support building shared libaries that depend on archive libraries and so the build of the shared plugin libraries fails when no shared version of libsqlservices is found.
So my main question is: why is libsqlservices built only as an archive library? The other question is: if libsqlservices has to be an archive only library, is there is another way to link it in to executables, such as linking it in explicitly when building an executable instead of counting on the shared plugin libraries to bring it in indirectly by making it a dependent library?
Steve Ellcey sje@cup.hp.com
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
On Fri, 2011-09-23 at 00:10 +0200, Vladislav Vaintroub wrote:
Steve, Are you talking about PIC requirements, when you say " HP-UX doesn't support building shared libaries that depend on archive libraries".
No I was thinking of a dependent library list, after building mariadb on linux, in the storage/federated/.libs directory, there is ha_federated.so, the shared library and there is ha_federated.la, the libtool library file which contains: dependency_libs=' -L/wsp/sje/mariadb/linux/obj/libservices -lmysqlservices -lcrypt -lnsl -lpthread -lrt' I think this causes libtool, when building something that uses ha_federated.so to also link in libmysqlservices (and crypt, etc). libmsqlservices would only need to be PIC if it were copied into ha_federated.so and I don't think that is happening is it? If so then it would need to be PIC and with my build, the libsqlservices code is not compiled with the PIC options.
In MySQL 5.5, linking static to shared feature was used extensively (as a replacement for libtool's "convenience libraries", static library was compiled with platform dependent PIC flag), and it worked on all major Unixes, I'm pretty sure it worked on HPUX/Itanium.
I think the trick for Maria 5.3 could be to compile the sqlservices library with the PIC flag (+Z or +z or something similar, if I recall this correctly). I do not have access to HPUX machines anymore, but you might want to give it a try.
I think compiling with PIC should only be needed if libsqlservices code is intended to be linked into a shared library and I am not sure if that is the case or not. Here is the error message that libtool gives me during the build: *** Warning: linker path does not have real file for library -lmysqlservices. *** I have the capability to make that library automatically link in when *** you link to this library. But I can only do this if you have a *** shared version of the library, which you do not appear to have *** because I did check the linker path looking for a file starting *** with libmysqlservices and none of the candidates passed a file format test *** using a file magic. Last file checked: /wsp/sje/mariadb/isvn/mariadb-project/5.3.1.beta/obj_gcc/libservices/libmysqlservices.a *** Warning: libtool could not satisfy all declared inter-library *** dependencies of module ha_federated. Therefore, libtool will create *** a static module, that should work as long as the dlopening *** application is linked with the -dlopen flag. libtool: link: ar cru .libs/ha_federated.a .libs/ha_federated_la-ha_federated.o .libs/ha_federated_la-string.o Steve Ellcey sje@cup.hp.com
-----Original Message----- From: Steve Ellcey [mailto:sje@cup.hp.com] Sent: Freitag, 23. September 2011 00:37 To: Vladislav Vaintroub Cc: maria-developers@lists.launchpad.net Subject: Re: [Maria-developers] Shared plugin library build question
Hi Steve,
I think compiling with PIC should only be needed if libsqlservices code is intended to be linked into a shared library and I am not sure if that is the case or not.
It is intended.
Here is the error message that libtool gives me during the build:
*** Warning: linker path does not have real file for library -lmysqlservices. *** I have the capability to make that library automatically link in when *** you link to this library. But I can only do this if you have a *** shared version of the library, which you do not appear to have *** because I did check the linker path looking for a file starting *** with libmysqlservices and none of the candidates passed a file format test *** using a file magic. Last file checked: /wsp/sje/mariadb/isvn/mariadb- project/5.3.1.beta/obj_gcc/libservices/libmysqlservices.a
Ok, libtool is being smart here.. What happens if you change the definition of ha_federated_la_LDFLAGS in federated/Makefile.am to ha_federated_la_LDFLAGS = -module -rpath $(pkgplugindir) $(top_builddir)/libservices/libmysqlservices.a (Disclaimer, I have not tried it myself, the intention is to get rid of -l<libname> in favor of full path to the static library. This I think has more chances to be successful)
Hi, Vladislav! On Sep 23, Vladislav Vaintroub wrote:
-----Original Message----- From: Steve Ellcey [mailto:sje@cup.hp.com] Sent: Freitag, 23. September 2011 00:37 To: Vladislav Vaintroub Cc: maria-developers@lists.launchpad.net Subject: Re: [Maria-developers] Shared plugin library build question
I think compiling with PIC should only be needed if libsqlservices code is intended to be linked into a shared library and I am not sure if that is the case or not.
It is intended.
Here is the error message that libtool gives me during the build:
*** Warning: linker path does not have real file for library -lmysqlservices. *** I have the capability to make that library automatically link in when *** you link to this library. But I can only do this if you have a *** shared version of the library, which you do not appear to have *** because I did check the linker path looking for a file starting *** with libmysqlservices and none of the candidates passed a file format test *** using a file magic. Last file checked: /wsp/sje/mariadb/isvn/mariadb- project/5.3.1.beta/obj_gcc/libservices/libmysqlservices.a
Ok, libtool is being smart here..
What happens if you change the definition of ha_federated_la_LDFLAGS in federated/Makefile.am to ha_federated_la_LDFLAGS = -module -rpath $(pkgplugindir) $(top_builddir)/libservices/libmysqlservices.a (Disclaimer, I have not tried it myself, the intention is to get rid of -l<libname> in favor of full path to the static library. This I think has more chances to be successful)
I've just tried. It produces *** Warning: Linking the shared library ha_example.la against the *** static library ../../libservices/libmysqlservices.a is not portable! but seems to work nevertheless. The main question was - whether dependency tracking works, that is, only those services that are really used should get into the final binary, not the complete libmysqlservices.a. And it does work (verified with nm). Regards, Sergei
On Fri, 2011-09-23 at 00:59 +0200, Vladislav Vaintroub wrote:
Ok, libtool is being smart here..
What happens if you change the definition of ha_federated_la_LDFLAGS in federated/Makefile.am to ha_federated_la_LDFLAGS = -module -rpath $(pkgplugindir) $(top_builddir)/libservices/libmysqlservices.a (Disclaimer, I have not tried it myself, the intention is to get rid of -l<libname> in favor of full path to the static library. This I think has more chances to be successful)
Well, the build works and I get a shared library but when I run the test suite all my federated tests failed. Looking at the log file from the testing I see: 110923 14:22:01 [ERROR] Can't open shared library '/wsp/sje/mariadb/isvn2/mariadb-project/5.3.1.beta/install/lib/mysql/plugin/ha_federatedx.so' (errno: 2 Unsatisfied data symbol 'my_snprintf_service' in load module '/wsp/sje/mariadb/isvn2/mariadb-project/5.3.1.beta/install/lib/mysq) 110923 14:22:01 [ERROR] /wsp/sje/mariadb/isvn2/mariadb-project/5.3.1.beta/install/libexec/mysqld: unknown option '--federated' 110923 14:22:01 [ERROR] Aborting Steve Ellcey sje@cup.hp.com
Hi, Steve! On Sep 23, Steve Ellcey wrote:
On Fri, 2011-09-23 at 00:59 +0200, Vladislav Vaintroub wrote:
Ok, libtool is being smart here..
What happens if you change the definition of ha_federated_la_LDFLAGS in federated/Makefile.am to ha_federated_la_LDFLAGS = -module -rpath $(pkgplugindir) $(top_builddir)/libservices/libmysqlservices.a (Disclaimer, I have not tried it myself, the intention is to get rid of -l<libname> in favor of full path to the static library. This I think has more chances to be successful)
Well, the build works and I get a shared library but when I run the test suite all my federated tests failed. Looking at the log file from the testing I see:
110923 14:22:01 [ERROR] Can't open shared library '/wsp/sje/mariadb/isvn2/mariadb-project/5.3.1.beta/install/lib/mysql/plugin/ha_federatedx.so' (errno: 2 Unsatisfied data symbol 'my_snprintf_service' in load module '/wsp/sje/mariadb/isvn2/mariadb-project/5.3.1.beta/install/lib/mysq) 110923 14:22:01 [ERROR] /wsp/sje/mariadb/isvn2/mariadb-project/5.3.1.beta/install/libexec/mysqld: unknown option '--federated' 110923 14:22:01 [ERROR] Aborting
So, apparently, it didn't work :( This thread - http://lists.gnu.org/archive/html/libtool/2010-07/msg00031.html seems to suggest that the limitation is in libtool only. The reporter wrote that after patching libtool, everything links and works correctly. We're moving to mysql-5.5 codebase, and it can be built with cmake only, not with autotools. I wonder whether cmake has the same problem that libtool has on hp-ux. Regards, Sergei
-----Original Message----- From: Sergei Golubchik [mailto:serg@askmonty.org] Sent: Freitag, 23. September 2011 23:40 To: Steve Ellcey Cc: Vladislav Vaintroub; maria-developers@lists.launchpad.net Subject: Re: [Maria-developers] Shared plugin library build question
We're moving to mysql-5.5 codebase, and it can be built with cmake only, not with autotools. I wonder whether cmake has the same problem that libtool has on hp-ux.
It is a while since I have touched HPUX last time, but I recall it worked well with CMake, also wrt plugins. Minor annoyances of the compiler (default for pre-ANSI for-loop behavior in C++) and CMake (preference to create shared libraries as .sl rather than .so), are fixed in cmake/os/HPUX.cmake in 5.5 codebase. I also recall on some reason it was always necessary to set CFLAGS=+DD64 CXXFLAGS=+DD64 prior to running CMake, else PA-RISC libraries were produced, or something like that.
Regards, Sergei
On Sat, 2011-09-24 at 00:07 +0200, Vladislav Vaintroub wrote:
We're moving to mysql-5.5 codebase, and it can be built with cmake only, not with autotools. I wonder whether cmake has the same problem that libtool has on hp-ux.
Is there any estimate of when this switch to cmake is going to happen? Steve Ellcey sje@cup.hp.com
Hi, Steve! On Sep 23, Steve Ellcey wrote:
On Sat, 2011-09-24 at 00:07 +0200, Vladislav Vaintroub wrote:
We're moving to mysql-5.5 codebase, and it can be built with cmake only, not with autotools. I wonder whether cmake has the same problem that libtool has on hp-ux.
Is there any estimate of when this switch to cmake is going to happen?
Try lp:maria/5.5 It's still a bit rough around the edges, but it mostly works and is good enough to see whether you'll have this libmysqlservices.a problem or not. Regards, Sergei
On Sun, 2011-11-06 at 15:45 +0100, Sergei Golubchik wrote:
Try lp:maria/5.5
It's still a bit rough around the edges, but it mostly works and is good enough to see whether you'll have this libmysqlservices.a problem or not.
Regards, Sergei
Sergei, I have downloaded these sources and done a trial build. I have run into a couple of problems so far. When compiling 5.5/mysys/base64.c I get an error because it is including <base64.h> and finding /usr/local/include/base64.h before 5.5/include/base64.h. This causes an error because the base64_encode in that header does not match the one in mariadb. I assume we are using the wrong base64.h because mysys/CMakeFiles/CMakeDirectoryInformation.cmake has: SET(CMAKE_C_INCLUDE_PATH "include" "/usr/local/include" "/wsp/sje/mariadb/bazaar/repo/5.5/include" "/wsp/sje/mariadb/bazaar/repo/5.5/mysys" ) where /usr/local/include is listed before the 5.5/include and 5.5/mysys directories. Unfortunately, CMakeDirectoryInformation.cmake is generated by Cmake and I am not sure where it gets its information from about the order of the include files. FYI: I am building in an object directory that is separate from the source directory, I don't know if that is related to the problem or not. The other problem I ran into is one that the older sources have too but I don't think I mentioned it before. In mysys/my_getsystime.c there is code that is dependent on HAVE_CLOCK_GETTIME, this macro is true for HP-UX because we do have this function, but we do not have the macro CLOCK_MONOTONIC (just CLOCK_REALTIME, CLOCK_VIRTUAL, and CLOCK_PROFILE). I don't think CLOCK_MONOTONIC is part of the POSIX clock_gettime definition and in our sources I just worked around by undef'ing HAVE_CLOCK_GETTIME. Steve Ellcey sje@cup.hp.com
-----Original Message----- From: Steve Ellcey [mailto:sje@cup.hp.com] Sent: Dienstag, 8. November 2011 22:51 To: Sergei Golubchik Cc: Vladislav Vaintroub; maria-developers@lists.launchpad.net Subject: Re: [Maria-developers] Shared plugin library build question
On Sun, 2011-11-06 at 15:45 +0100, Sergei Golubchik wrote:
<skip>
I have downloaded these sources and done a trial build. I have run into a couple of problems so far. When compiling 5.5/mysys/base64.c I get an error because it is including <base64.h> and finding /usr/local/include/base64.h before 5.5/include/base64.h. This causes an error because the base64_encode in that header does not match the one in mariadb. I assume we are using the wrong base64.h because mysys/CMakeFiles/CMakeDirectoryInformation.cmake has:
SET(CMAKE_C_INCLUDE_PATH "include" "/usr/local/include" "/wsp/sje/mariadb/bazaar/repo/5.5/include" "/wsp/sje/mariadb/bazaar/repo/5.5/mysys" )
where /usr/local/include is listed before the 5.5/include and 5.5/mysys directories. Unfortunately, CMakeDirectoryInformation.cmake is generated by Cmake and I am not sure where it gets its information from about the order of the include files.
Include directories in cmake are manipulated by INCLUDE_DIRECTORIES directive http://www.cmake.org/cmake/help/cmake-2-8-docs.html#command:include_director... . It has an option for appending , prepending and an for SYSTEM headers( I'm not sure about exact effect of this one) mysys/CMakeLists.txt has at the very top of the file INCLUDE_DIRECTORIES(${ZLIB_INCLUDE_DIR} ${CMAKE_SOURCE_DIR}/include ${CMAKE_SOURCE_DIR}/mysys) I assume ZLIB_INCLUDE_DIR is /usr/local/include, since we use system zlib if it is available. You can try to change order here and for example move ${CMAKE_SOURCE_DIR}/include to the start of the list. I think the best option for MySQL, and for MariaDB is to rename own base64.h and function prototypes in a way that it does not conflict with system headers on any system it is meant to be run on. Relying on the include order is fragile, I also do not see how it would not solve the problem with same function name different prototypes - how can we always reliably force use our base64_encode rather than system one ? FYI: I am building in an object
directory that is separate from the source directory, I don't know if that is related to the problem or not.
It should not be a problem.
On Tue, 2011-11-08 at 23:33 +0100, Vladislav Vaintroub wrote:
Include directories in cmake are manipulated by INCLUDE_DIRECTORIES directive http://www.cmake.org/cmake/help/cmake-2-8-docs.html#command:include_director... . It has an option for appending , prepending and an for SYSTEM headers( I'm not sure about exact effect of this one)
mysys/CMakeLists.txt has at the very top of the file INCLUDE_DIRECTORIES(${ZLIB_INCLUDE_DIR} ${CMAKE_SOURCE_DIR}/include ${CMAKE_SOURCE_DIR}/mysys)
I assume ZLIB_INCLUDE_DIR is /usr/local/include, since we use system zlib if it is available. You can try to change order here and for example move ${CMAKE_SOURCE_DIR}/include to the start of the list.
Thanks, changing the order of the includes in this file (putting ZLIB_INCLUDE_DIR last) fixed that problem. Now I die when compiling storage/sphinx/ha_sphinx.cc. That file calls localtime_r but it doesn't seem to include <time.h> where localtime_r is declared, so the C++ compiler dies with: 5.5/storage/sphinx/ha_sphinx.cc:911:34: error: 'localtime_r' was not declared in this scope I am not sure if time.h should be included directly in this file or if one it's includes (my_sys.h or something) should be including time.h. Steve Ellcey sje@cup.hp.com
On Tue, 2011-11-08 at 15:19 -0800, Steve Ellcey wrote:
Thanks, changing the order of the includes in this file (putting ZLIB_INCLUDE_DIR last) fixed that problem. Now I die when compiling storage/sphinx/ha_sphinx.cc. That file calls localtime_r but it doesn't seem to include <time.h> where localtime_r is declared, so the C++ compiler dies with:
5.5/storage/sphinx/ha_sphinx.cc:911:34: error: 'localtime_r' was not declared in this scope
I am not sure if time.h should be included directly in this file or if one it's includes (my_sys.h or something) should be including time.h.
Steve Ellcey sje@cup.hp.com
Woops, following up to my own email, it looks like time.h is getting included it is just that we aren't defining _REENTRENT so we don't see the definition from time.h. I am not sure if this is a mariadb issue or an HP-UX issue. I will investigate it some more. Steve Ellcey sje@cup.hp.com
-----Original Message----- From: Steve Ellcey [mailto:sje@cup.hp.com] Sent: Mittwoch, 9. November 2011 00:31 To: Vladislav Vaintroub Cc: maria-developers@lists.launchpad.net; 'Sergei Golubchik' Subject: Re: [Maria-developers] Shared plugin library build question
On Tue, 2011-11-08 at 15:19 -0800, Steve Ellcey wrote:
Thanks, changing the order of the includes in this file (putting ZLIB_INCLUDE_DIR last) fixed that problem. Now I die when compiling storage/sphinx/ha_sphinx.cc. That file calls localtime_r but it doesn't seem to include <time.h> where localtime_r is declared, so the C++ compiler dies with:
5.5/storage/sphinx/ha_sphinx.cc:911:34: error: 'localtime_r' was not declared in this scope
I am not sure if time.h should be included directly in this file or if one it's includes (my_sys.h or something) should be including time.h.
Steve Ellcey sje@cup.hp.com
Woops, following up to my own email, it looks like time.h is getting included it is just that we aren't defining _REENTRENT so we don't see the definition from time.h. I am not sure if this is a mariadb issue or an HP-UX issue. I will investigate it some more.
If you stuck, a simple way to fix it might be to add -D_REENTRANT to cmake/os/HP-UX.cmake , so everything is compiled with it. The line ADD_DEFINITIONS(-D_REENTRANT) would do the trick.
On Wed, 2011-11-09 at 00:54 +0100, Vladislav Vaintroub wrote:
If you stuck, a simple way to fix it might be to add -D_REENTRANT to cmake/os/HP-UX.cmake , so everything is compiled with it. The line ADD_DEFINITIONS(-D_REENTRANT)
would do the trick.
I added 'ADD_DEFINITIONS(-pthread)' instead, the -pthread flag causes the compiler to define _REENTRANT. I don't see any cmake configure code that tries to determine if -pthread is needed when compiling or linking and I don't see its use in any of the other cmake files, I find this surprising. The Mariadb 5.3.2 configure script certainly has code to see if -pthread (or -mt or -threads) should be used or not. I am also not sure if this is all I need for linking since we want to use -pthread to link in the libpthread library. Do flags added with ADD_DEFINITIONS get used on a link line or is there another rule I should use in addition to ADD_DEFINITIONS? I now die when compiling client/mysql.cc. HIST_ENTRY does not seem to be defined but HAVE_HIST_ENTRY is. This might be another include file issue, I am not sure yet. Steve Ellcey sje@cup.hp.com
-----Original Message----- From: Steve Ellcey [mailto:sje@cup.hp.com] Sent: Mittwoch, 9. November 2011 01:53 To: Vladislav Vaintroub Cc: 'Sergei Golubchik'; maria-developers@lists.launchpad.net Subject: Re: [Maria-developers] Shared plugin library build question
On Wed, 2011-11-09 at 00:54 +0100, Vladislav Vaintroub wrote:
If you stuck, a simple way to fix it might be to add -D_REENTRANT to cmake/os/HP-UX.cmake , so everything is compiled with it. The line ADD_DEFINITIONS(-D_REENTRANT)
would do the trick.
I added 'ADD_DEFINITIONS(-pthread)' instead, the -pthread flag causes the compiler to define _REENTRANT. I don't see any cmake configure code that tries to determine if -pthread is needed when compiling or linking and I don't see its use in any of the other cmake files, I find this surprising. The Mariadb 5.3.2 configure script certainly has code to see if -pthread (or -mt or -threads) should be used or not.
I am also not sure if this is all I need for linking since we want to use -pthread to link in the libpthread library. Do flags added with ADD_DEFINITIONS get used on a link line or is there another rule I should use in addition to ADD_DEFINITIONS?
The flags here are added to compile flags. If compiler is also the linker, you'll get them in the link line too. Use make VERBOSE=1 to see the full command line, or you can find the full link line in the CMakeFiles/<target>.dir/link.txt , as described here http://forge.mysql.com/wiki/CMake#Tips_for_developers You should not use any other rule for threads, it is taken care automatically by FIND_PACKAGE(Threads) and LINK_LIBRARIES(${CMAKE_THREAD_LIBS_INIT}) inside configure.cmake. The documentation on FIND_PACKAGE and different modules for might not be a strongest point of CMake, but if you googling for CMAKE_THREAD_LIBS_INIT brings good hits.
I now die when compiling client/mysql.cc. HIST_ENTRY does not seem to be defined but HAVE_HIST_ENTRY is. This might be another include file issue, I am not sure yet.
Oh, this is libedit vs readline. It is the most complicated stuff in current cmake setup (cmake/readline.cmake), that unfortunately got ported 1-1 from autotools, and the code in autotools was already bad originally and it has not got prettier with CMake , and it was my very first CMake code , and it remained as ugly as it was at the start. But to the problem There are different possibilities to use system readline, system libedit, bundled readline, bundled libedit , the default is bundled libedit (frankly I've got no clue why we or MySQL have a choice of 2 almost equivalent libs, but anyway) The cmake flags that enable one or another are -DWITH_READLINE={1,0} -DWITH_LIBEDIT={1,0} If none is set (equivalent to both are 0), system library is searched, with readline preference, and libedit backoff. I have to admit , I do not understand the logic in cmake/readline.cmake fully, so this is also from my memory mostly. One of those libs should be new otherwise there are problems with header files, and HIST_ENTRY might well be this problem. What you can definitely do right now, is to remove CMakeCache.txt and rerun cmake <path-to-source> -DWITH_LIBEDIT=0 -DWITH_READLINE=0 and rebuild mysql client with "make mysql" If that fails, try the cmake <path-to-source> -DWITH_LIBEDIT=0 -DWITH_READLINE=1 . Once we find the variant that works on HPUX, I'd prefer to hardcode that in cmake/os/HP-UX.cmake (if memory serves me right, only readline is workable on HPUX, not libedit, but it has been some time since I have compiled 5.5 on HPUX )
Steve Ellcey sje@cup.hp.com
On Wed, 2011-11-09 at 02:50 +0100, Vladislav Vaintroub wrote:
Oh, this is libedit vs readline. It is the most complicated stuff in current cmake setup (cmake/readline.cmake), that unfortunately got ported 1-1 from autotools, and the code in autotools was already bad originally and it has not got prettier with CMake , and it was my very first CMake code , and it remained as ugly as it was at the start.
But to the problem
There are different possibilities to use system readline, system libedit, bundled readline, bundled libedit , the default is bundled libedit (frankly I've got no clue why we or MySQL have a choice of 2 almost equivalent libs, but anyway)
The cmake flags that enable one or another are -DWITH_READLINE={1,0} -DWITH_LIBEDIT={1,0}
If none is set (equivalent to both are 0), system library is searched, with readline preference, and libedit backoff. I have to admit , I do not understand the logic in cmake/readline.cmake fully, so this is also from my memory mostly. One of those libs should be new otherwise there are problems with header files, and HIST_ENTRY might well be this problem.
What you can definitely do right now, is to remove CMakeCache.txt and rerun cmake <path-to-source> -DWITH_LIBEDIT=0 -DWITH_READLINE=0
That did not work, cmake died with: CMake Error at cmake/readline.cmake:223 (MESSAGE): Cannot find system readline or libedit libraries.Use WITH_READLINE or WITH_LIBEDIT
If that fails, try the cmake <path-to-source> -DWITH_LIBEDIT=0 -DWITH_READLINE=1 .
This one did work.
Once we find the variant that works on HPUX, I'd prefer to hardcode that in cmake/os/HP-UX.cmake (if memory serves me right, only readline is workable on HPUX, not libedit, but it has been some time since I have compiled 5.5 on HPUX )
It looks like READLINE is the way to go. And with this change added to the earlier ones I got the build to complete with no errors. I haven't tried running the test suite yet, but I will do that next. This build was done with GCC, I want to also try building with the HP compiler to see how that works. Steve Ellcey sje@cup.hp.com
-----Original Message----- From: Steve Ellcey [mailto:sje@cup.hp.com] Sent: Mittwoch, 9. November 2011 18:37 To: Vladislav Vaintroub Cc: 'Sergei Golubchik'; maria-developers@lists.launchpad.net Subject: RE: [Maria-developers] Shared plugin library build question
It looks like READLINE is the way to go. And with this change added to the earlier ones I got the build to complete with no errors. I haven't tried running the test suite yet, but I will do that next. This build was done with GCC, I want to also try building with the HP compiler to see how that works.
A word of warning upfront before trying HP compiler. Unless xlc has changed its defaults in the last year or two, you'd need to set CFLAGS and CXXFLAGS env.vars prior to CMake so it compiles for Itanium. it was either +DD64 or maybe +DSitanium2 or something similar. Otherwise the build would produce PA-RISC binaries (those binaries are runnable on the HPUX Itanium, due to some emulation layer, but it is likely not what people expect)
Steve Ellcey sje@cup.hp.com
-----Original Message----- From: Vladislav Vaintroub [mailto:wlad@montyprogram.com] Sent: Mittwoch, 9. November 2011 19:13 To: 'sje@cup.hp.com' Cc: 'Sergei Golubchik'; 'maria-developers@lists.launchpad.net' Subject: RE: [Maria-developers] Shared plugin library build question
A word of warning upfront before trying HP compiler. Unless xlc has changed its defaults in the last year or two, you'd need to set CFLAGS and CXXFLAGS env.vars prior to CMake so it compiles for Itanium.
Scratch xlc, that was AIX, not HPUX... But the rest of the warning applies :)
it was either +DD64 or maybe +DSitanium2 or something similar. Otherwise the build would produce PA-RISC binaries (those binaries are runnable on the HPUX Itanium, due to some emulation layer, but it is likely not what people expect)
Steve Ellcey sje@cup.hp.com
On Wed, 2011-11-09 at 19:25 +0100, Vladislav Vaintroub wrote:
A word of warning upfront before trying HP compiler. Unless xlc has changed its defaults in the last year or two, you'd need to set CFLAGS and CXXFLAGS env.vars prior to CMake so it compiles for Itanium.
Scratch xlc, that was AIX, not HPUX... But the rest of the warning applies :)
it was either +DD64 or maybe +DSitanium2 or something similar. Otherwise the build would produce PA-RISC binaries (those binaries are runnable on the HPUX Itanium, due to some emulation layer, but it is likely not what people expect)
Well, I tried building Mariadb with the HP compiler and I ran into a problem with anonymous unions. In 5.3.2 the anonymous unions (and anonymous struct) in st_dynamic_column_value were changed to have names. This change does not seem to be in 5.5. I don't know if that is intentional or accidental but it makes it more difficult to compile the 5.5 code with the HP compiler. Steve Ellcey sje@cup.hp.com
Hi, Steve! On Nov 11, Steve Ellcey wrote:
On Wed, 2011-11-09 at 19:25 +0100, Vladislav Vaintroub wrote:
A word of warning upfront before trying HP compiler. Unless xlc has changed its defaults in the last year or two, you'd need to set CFLAGS and CXXFLAGS env.vars prior to CMake so it compiles for Itanium.
Scratch xlc, that was AIX, not HPUX... But the rest of the warning applies :)
it was either +DD64 or maybe +DSitanium2 or something similar. Otherwise the build would produce PA-RISC binaries (those binaries are runnable on the HPUX Itanium, due to some emulation layer, but it is likely not what people expect)
Well, I tried building Mariadb with the HP compiler and I ran into a problem with anonymous unions. In 5.3.2 the anonymous unions (and anonymous struct) in st_dynamic_column_value were changed to have names.
This change does not seem to be in 5.5. I don't know if that is intentional or accidental but it makes it more difficult to compile the 5.5 code with the HP compiler.
I've merged latest changes from 5.3 (including the one that gives names to anonymous unions) into 5.5 only few days ago. It is still not pushed, because few optimizer test cases are broken. I'm investigating. Regards, Sergei
On Wed, 2011-11-09 at 09:36 -0800, Steve Ellcey wrote:
It looks like READLINE is the way to go. And with this change added to the earlier ones I got the build to complete with no errors. I haven't tried running the test suite yet, but I will do that next. This build was done with GCC, I want to also try building with the HP compiler to see how that works.
Steve Ellcey sje@cup.hp.com
FYI: I ran the test suite and got 2521 passes, 493 skipped, 30 disabled, and 41 fails. A bunch of federated tests failed with: 111109 11:19:27 [ERROR] /wsp/sje/mariadb/bazaar/repo/install/bin/mysqld: unknown option '--federated' 111109 11:19:27 [ERROR] Aborting I wonder if this is related to the shared library issue? When building 5.3.2 I used '--with-plugins=max-no-ndb' on the configure. Is there a CMake equivalent for this? There seem to be some other failing tests where I have garbage in the output, I haven't investigated these but the failures are along the lines of this difference: -mysql.user authentication_string 42 NULL NO text 65535 65535 NULL NULL utf8 utf8_bin text +mysql.user authentication_string 42 NULL NO text11)e,'Y')'Insert','Update','References')J 65535 65535 NULL NULL utf8 utf8_bin text Has anyone else seen failures like this? Steve Ellcey sje@cup.hp.com
-----Original Message----- From: Steve Ellcey [mailto:sje@cup.hp.com] Sent: Mittwoch, 9. November 2011 22:02 To: Vladislav Vaintroub Cc: maria-developers@lists.launchpad.net; 'Sergei Golubchik' Subject: Re: [Maria-developers] Shared plugin library build question
On Wed, 2011-11-09 at 09:36 -0800, Steve Ellcey wrote:
It looks like READLINE is the way to go. And with this change added to the earlier ones I got the build to complete with no errors. I haven't tried running the test suite yet, but I will do that next. This build was done with GCC, I want to also try building with the HP compiler to see how that works.
Steve Ellcey sje@cup.hp.com
FYI: I ran the test suite and got 2521 passes, 493 skipped, 30 disabled, and 41 fails.
A bunch of federated tests failed with:
111109 11:19:27 [ERROR] /wsp/sje/mariadb/bazaar/repo/install/bin/mysqld: unknown option '--federated' 111109 11:19:27 [ERROR] Aborting
I wonder if this is related to the shared library issue? When building 5.3.2 I used '--with-plugins=max-no-ndb' on the configure. Is there a CMake equivalent for this?
I assume this is a bug in mysql-test-run.pl . Currently, plugins are not loaded unless you build in the source directory. Could you apply patch for 'mysql-test/mysql-test-run.pl' from here http://lists.askmonty.org/pipermail/commits/2011-October/002493.html and see if this helps? Please make sure to apply diff ONLY for 'mysql-test/mysql-test-run.pl' , not the whole thing
There seem to be some other failing tests where I have garbage in the output, I haven't investigated these but the failures are along the lines of this difference:
-mysql.user authentication_string 42 NULL NO text 65535 65535 NULL NULL utf8 utf8_bin text +mysql.user authentication_string 42 NULL NO text11)e,'Y')'Insert','Update','References')J 65535 65535 NULL NULL utf8 utf8_bin text
Has anyone else seen failures like this?
Yes, I have seen that on Windows. information_schema queries, with junk always appended at the end of "text" or "blob"
Steve Ellcey sje@cup.hp.com
On Wed, 2011-11-09 at 22:40 +0100, Vladislav Vaintroub wrote:
I assume this is a bug in mysql-test-run.pl . Currently, plugins are not loaded unless you build in the source directory. Could you apply patch for 'mysql-test/mysql-test-run.pl' from here http://lists.askmonty.org/pipermail/commits/2011-October/002493.html and see if this helps? Please make sure to apply diff ONLY for 'mysql-test/mysql-test-run.pl' , not the whole thing
This did not seem to help. I changed mysql-test-run.pl but I still get the federated test failures with: 111109 14:55:20 [ERROR] /wsp/sje/mariadb/bazaar/repo/install/bin/mysqld: unknown option '--federated' 111109 14:55:20 [ERROR] Aborting When I look in the lib/plugin directory I see ha_federated.so and ha_federatedx.so but no federated.so. Steve Ellcey sje@cup.hp.com
-----Original Message----- From: Steve Ellcey [mailto:sje@cup.hp.com] Sent: Donnerstag, 10. November 2011 00:09 To: Vladislav Vaintroub Cc: 'Sergei Golubchik'; maria-developers@lists.launchpad.net Subject: Re: [Maria-developers] Shared plugin library build question
On Wed, 2011-11-09 at 22:40 +0100, Vladislav Vaintroub wrote:
I assume this is a bug in mysql-test-run.pl . Currently, plugins are not loaded unless you build in the source directory. Could you apply patch for 'mysql-test/mysql-test- run.pl' from here http://lists.askmonty.org/pipermail/commits/2011-October/002493.html and see if this helps? Please make sure to apply diff ONLY for 'mysql-test/mysql-test-run.pl' , not the whole thing
This did not seem to help. I changed mysql-test-run.pl but I still get the federated test failures with:
111109 14:55:20 [ERROR] /wsp/sje/mariadb/bazaar/repo/install/bin/mysqld: unknown option '--federated' 111109 14:55:20 [ERROR] Aborting
When I look in the lib/plugin directory I see ha_federated.so and ha_federatedx.so but no federated.so.
Yes, storage engines built as shared libraries are always "ha_" prefixed (historical reasons). Why they are not loaded, or whether they are attempted to load, or why mysqld barfs, I cannot tell at the moment . But from 5.3's (Windows-only) cmake I see that we had always built federatedx statically into the server, thus I hope a rebuild with cmake . -DWITH_FEDERATEDX_STORAGE_ENGINE=1 could finally solve your problems.
Steve Ellcey sje@cup.hp.com
Hi, Steve! On Nov 09, Steve Ellcey wrote:
On Wed, 2011-11-09 at 09:36 -0800, Steve Ellcey wrote:
It looks like READLINE is the way to go. And with this change added to the earlier ones I got the build to complete with no errors. I haven't tried running the test suite yet, but I will do that next. This build was done with GCC, I want to also try building with the HP compiler to see how that works.
Steve Ellcey sje@cup.hp.com
FYI: I ran the test suite and got 2521 passes, 493 skipped, 30 disabled, and 41 fails.
A bunch of federated tests failed with:
111109 11:19:27 [ERROR] /wsp/sje/mariadb/bazaar/repo/install/bin/mysqld: unknown option '--federated' 111109 11:19:27 [ERROR] Aborting
I wonder if this is related to the shared library issue? When building 5.3.2 I used '--with-plugins=max-no-ndb' on the configure. Is there a CMake equivalent for this?
Yes, here're the alternatives: * you can use BUILD/compile-*-debug-max-no-ndb it'll work as before. * you can run ./configure --with-plugins=max-no-ndb it's a perl wrapper over cmake, that provides some (limited) autoconf's configure compatibility. * you can run cmake -DWITH_MAX_NO_NDB=1 * you can run ccmake - it's a ncurses menu tool, that allows you to see all configuration variables and change them as you wish. There are graphical cmake configurators too.
There seem to be some other failing tests where I have garbage in the output, I haven't investigated these but the failures are along the lines of this difference:
-mysql.user authentication_string 42 NULL NO text 65535 65535 NULL NULL utf8 utf8_bin text +mysql.user authentication_string 42 NULL NO text11)e,'Y')'Insert','Update','References')J 65535 65535 NULL NULL utf8 utf8_bin text
Has anyone else seen failures like this?
Yes, I did (not the same, but similar). I haven't investigated it yet. Regards, Sergei
On Wed, 2011-11-09 at 23:54 +0100, Sergei Golubchik wrote:
Hi, Steve!
There seem to be some other failing tests where I have garbage in the output, I haven't investigated these but the failures are along the lines of this difference:
-mysql.user authentication_string 42 NULL NO text 65535 65535 NULL NULL utf8 utf8_bin text +mysql.user authentication_string 42 NULL NO text11)e,'Y')'Insert','Update','References')J 65535 65535 NULL NULL utf8 utf8_bin text
Has anyone else seen failures like this?
Yes, I did (not the same, but similar). I haven't investigated it yet.
Regards, Sergei
FYI: I was building in 32 bit mode, when I switched to 64 bit mode, these errors when away and I am down to 21 failures. 10 federated failures plus these: main.ps w2 [ fail ] main.select_pkeycache w2 [ fail ] main.not_embedded_server w4 [ fail ] main.mysqld--help-notwin w4 [ fail ] sys_vars.binlog_dbug_fsync_sleep_basic w1 [ fail ] sys_vars.debug_crc_break_basic w1 [ fail ] main.file_contents w2 [ fail ] sys_vars.plugin_dir_basic w5 [ fail ] main.select w4 [ fail ] main.select_jcl6 w4 [ fail ] percona.percona_server_variables_release w2 [ fail ] main.not_embedded_server is a known problem on IA64 HP-UX due to stack checking. I am going to try Vlad's change to mysql-test-run.pl first to see if that makes the federated failures go away and then I will try using -DWITH_MAX_NO_NDB=1 to see what effect that has. Steve Ellcey sje@cup.hp.com
On Wed, 2011-11-09 at 23:54 +0100, Sergei Golubchik wrote:
Yes, here're the alternatives:
* you can run cmake -DWITH_MAX_NO_NDB=1
So this resulted in some compilation errors in code that wasn't getting compiled before for 5.5, though I had seen these issues in 5.3.2. In storage/pbxt/src/xt_config.h I used __hpux to set XT_HPUX and then in storage/pbxt/src/xt_defs.h I used that ifdef to set u_int8_t, etc. like is already done for XT_SOLARIS. In storage/pbxt/src/filesys_xt.cc (xt_dir_is_file) I used XT_HPUX to use the code under the XT_SOLARIS ifdef. In storage/pbxt/src/thread_xt.cc (xt_throw_signal) I used XT_HPUX to use the code that is under the XT_WIN ifdef. I also replaced the atoll call in storage/pbxt/src/database_xt.cc with a strtoll call. Once I had done these changes, I was able to get a good build and I ran the testsuite and the federated tests all passed but a bunch of pbxt tests failed and I had never seen those fail with 5.3.2. These failures were all of the type: CURRENT_TEST: pbxt.odbc mysqltest: At line 19: query 'insert into t1 SET A=NULL,B=1' failed: 1467: Failed to read auto-increment value from storage engine Other then those failures, the only problems I saw were the 11 failures I mentioned in my earlier email. Steve Ellcey sje@cup.hp.com
participants (3)
-
Sergei Golubchik -
Steve Ellcey -
Vladislav Vaintroub