Query on Building Maria DB for different Compression Algorithm
[AMD Official Use Only - General]
Hi,
I am building MariaDB from source code and have queries on linking of compression algorithm.
Before building the source code of MariaDB, I am exporting the path for SNAPPY or LZMA or the algorithm which is currently it supports.
Then I build the code with - cmake -B . -DCMAKE_INSTALL_PREFIX=$PWD/maria_bin -DPLUGIN_MROONGA=NO -DPLUGIN_ROCKSDB=NO cmake --build . -j cmake -install .
It compiled the code and installed in the specified location.
When I checked the lib/plugin folder it has generated, provider_lzma.so or provider_snappy.so files and when I did ldd on that .so file, it has given the path from where it is fetching the liblzma.so.1 or libsnappy.so.1 file.
Query:
1. The path which ldd shows for provider_lzma.so or provider_snappy.so is taking form default path, and it is not from the path which I am exporting the library and bin path - could you please explain me the correct steps. 2. Is there any way or any Flag we can set to switch between different compression method during build time itself, and in which log we can check it has linked properly. 3. How to benchmark the linked compression method and how we can see it is using the linked compression method from the custom path of .so files.
Kindly help on above query.
Thanks Rahul Raj
On 18.07.2023 09:09, Raj, Rahul via discuss wrote:
[AMD Official Use Only - General]
I am building MariaDB from source code and have queries on linking of compression algorithm.
Before building the source code of MariaDB, I am exporting the path for SNAPPY or LZMA or the algorithm which is currently it supports.
How do you do that?
When I checked the lib/plugin folder it has generated, provider_lzma.so or provider_snappy.so files and when I did ldd on that .so file, it has given the path from where it is fetching the liblzma.so.1 or libsnappy.so.1 file.
So that works. cmake uses system search paths to locate include files and libraries by default.
1. The path which ldd shows for provider_lzma.so or provider_snappy.so is taking form default path, and it is not from the path which I am exporting the library and bin path – could you please explain me the correct steps.
There are typically two cmake options for each compression method. For example for snappy:
SNAPPY_INCLUDE_DIRS SNAPPY_LIBRARIES
if you want cmake to use a specific implementation of snappy, you must specify those on the cmake command line:
cmake ... -DSNAPPY_INCLUDE_DIRS="/foo/include" -DSNAPPY_LIBRARIES="/foo/lib"
ccmake lists all the possible cmake options in advanced mode.
2. Is there any way or any Flag we can set to switch between different compression method during build time itself, and in which log we can check it has linked properly.
cmake spits out messages to stdout, i.e. on my system for snappy:
-- Found Snappy: /usr/lib/x86_64-linux-gnu/libsnappy.so
and at the end of configuring cmake also lists all found packages on stderr. The compressions libraries are in the optional section:
-- The following OPTIONAL packages have been found:
* BZip2
* LZ4 (required version >= 1.6)
* LZO
* Snappy
You can parse that output. And/or use ldd on the plugins in $PLUGINDIR
3. How to benchmark the linked compression method and how we can see it is using the linked compression method from the custom path of .so files.
Use any benchmark you see fit. I.e. you can try to reproduce Perconas published numbers. They have published a lot on compression.
Most benchmarks do poorly for compression since they use generated data (i.e. random strings - not compressible). Benchmarks thus fail to make the real world behavior visible. For that you need real world data.
HTH, XL
[AMD Official Use Only - General]
Thank you, Axel! for quick response on my query, it will help me compiling the compression algorithm without exporting the library path.
Query: 1. Out of the multiple compression methods that it finds and links, how do I know which one it uses at runtime while testing or benchmarking MariaDB? For Ex- I just want to test only Snappy method and benchmark it, but while compiling it is saying BZIP2, LZ4, LZMA, Snappy etc., Found.
2. Is there a way that it links only one method at a time, and rest method it says NOT FOUND even it is present in the system.
Thanks & Regards Rahul Raj
-----Original Message----- From: Axel Schwenke axel@askmonty.org Sent: Tuesday, July 18, 2023 4:16 PM To: Raj, Rahul Rahul.Raj@amd.com; discuss@lists.mariadb.org Subject: Re: [MariaDB discuss] Query on Building Maria DB for different Compression Algorithm
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
On 18.07.2023 09:09, Raj, Rahul via discuss wrote:
[AMD Official Use Only - General]
I am building MariaDB from source code and have queries on linking of compression algorithm.
Before building the source code of MariaDB, I am exporting the path for SNAPPY or LZMA or the algorithm which is currently it supports.
How do you do that?
When I checked the lib/plugin folder it has generated, provider_lzma.so or provider_snappy.so files and when I did ldd on that .so file, it has given the path from where it is fetching the liblzma.so.1 or libsnappy.so.1 file.
So that works. cmake uses system search paths to locate include files and libraries by default.
1. The path which ldd shows for provider_lzma.so or provider_snappy.so is taking form default path, and it is not from the path which I am exporting the library and bin path - could you please explain me the correct steps.
There are typically two cmake options for each compression method. For example for snappy:
SNAPPY_INCLUDE_DIRS SNAPPY_LIBRARIES
if you want cmake to use a specific implementation of snappy, you must specify those on the cmake command line:
cmake ... -DSNAPPY_INCLUDE_DIRS="/foo/include" -DSNAPPY_LIBRARIES="/foo/lib"
ccmake lists all the possible cmake options in advanced mode.
2. Is there any way or any Flag we can set to switch between different compression method during build time itself, and in which log we can check it has linked properly.
cmake spits out messages to stdout, i.e. on my system for snappy:
-- Found Snappy: /usr/lib/x86_64-linux-gnu/libsnappy.so
and at the end of configuring cmake also lists all found packages on stderr. The compressions libraries are in the optional section:
-- The following OPTIONAL packages have been found:
* BZip2
* LZ4 (required version >= 1.6)
* LZO
* Snappy
You can parse that output. And/or use ldd on the plugins in $PLUGINDIR
3. How to benchmark the linked compression method and how we can see it is using the linked compression method from the custom path of .so files.
Use any benchmark you see fit. I.e. you can try to reproduce Perconas published numbers. They have published a lot on compression.
Most benchmarks do poorly for compression since they use generated data (i.e. random strings - not compressible). Benchmarks thus fail to make the real world behavior visible. For that you need real world data.
HTH, XL
On 18.07.2023 14:44, Raj, Rahul wrote:
[AMD Official Use Only - General]
Thank you, Axel!
1. Out of the multiple compression methods that it finds and links, how do I know which one it uses at runtime while testing or benchmarking MariaDB?
When you create a compressed table, you can specify the compression algorithm to use. RTFM:
https://mariadb.com/kb/en/innodb-page-compression/
and generally
https://mariadb.com/kb/en/optimization-and-tuning-compression/
2. Is there a way that it links only one method at a time, and rest method it says NOT FOUND even it is present in the system.
Not really. Of course you can build the server without a certain plugin. Or you (re)move the respective plugin from $PLUGINDIR and restart the server.
BR, XL
Hi, Axel,
On Jul 18, Axel Schwenke via discuss wrote:
2. Is there a way that it links only one method at a time, and rest method it says NOT FOUND even it is present in the system.
Not really. Of course you can build the server without a certain plugin. Or you (re)move the respective plugin from $PLUGINDIR and restart the server.
Or simply don't load a plugin. That's why it is a plugin, not a built-in functionality. Or disable it with --disable-<pluginname>.
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
[AMD Official Use Only - General]
Hi,
I am building MariaDB from source code and have queries on linking of compression algorithm.
Query:
1. Can I stop building zlib algorithm? which is building by default, even after passing different plugin like LZ4, snappy etc.,. Any argument in cmake that can be passed to stop compiling ZLIB algorithm.
1. Getting below error when linking 3rd party LZ4 library: In the below scenario build compilation is successfully. When trying to run below command to populate data folder to start server, it is throwing error.
I have tried loading LD_LIBRARY_PATH, even that also produce the same result.
#sudo ./scripts/mariadb-install-db --srcdir=/home/mariadb-10.11.3 --plugin-dir=/home/mariadb-10.11.3/maria_bin/lib/plugin --user=mysql
2023-08-02 17:56:58 0 [ERROR] mariadbd: Can't open shared library '/home/mariadb-10.11.3/maria_bin/lib/plugin/provider_lz4.so' (errno: 2, liblz4.so: cannot open shared object file: No such file or directory)
2023-08-02 17:56:58 0 [ERROR] Couldn't load plugins from 'provider_lz4.so'.
Kindly help on above query.
Thanks Rahul Raj
On 04.08.2023 11:57, Raj, Rahul via discuss wrote:
1. Can I stop building zlib algorithm? which is building by default, even after passing different plugin like LZ4, snappy etc.,. Any argument in cmake that can be passed to stop compiling ZLIB algorithm.
No. ZLIB is used internally, for example the COMPRESS() SQL function and the compressed client-server protocol. If no zlib is installed, the bundled (with MariaDB source) library is used.
2. Getting below error when linking 3rd party LZ4 library: In the below scenario build compilation is successfully. When trying to run below command to populate data folder to start server, it is throwing error.
*2023-08-02 17:56:58 0 [ERROR] mariadbd: Can't open shared library '/home/mariadb-10.11.3/maria_bin/lib/plugin/provider_lz4.so' (errno: 2, liblz4.so: cannot open shared object file: No such file or directory)*
The error message (from the MariaDB error log) says that liblz4.so could not be loaded by the mariadbd process. Where is it?
I have tried loading LD_LIBRARY_PATH, even that also produce the same result.
LD_LIBRARY_PATH is the right solution if liblz4.so is not in standard path. It must however be set in the environment from which the mariadbd process is started. Try this:
#sudo LD_LIBRARY_PATH=/path/to/liblz4 ./scripts/mariadb-install-db --srcdir=/home/mariadb-10.11.3 --plugin-dir=/home/mariadb-10.11.3/maria_bin/lib/plugin --user=mysql
It is probably easier to add the path containing liblz4.so to the standard search path for the dynamic linker. This would be in /etc/ld.so.conf or - depending on your Linux distribution - /etc/ld.so.conf.d/.
[AMD Official Use Only - General]
Hi Axel,
I followed your suggestion, and all the error has resolved but it didn't launch the Mariadb server. It throws below error:
#sudo ./support-files/mysql.server start Starting MariaDB .230810 08:15:40 mysqld_safe Logging to '/mysql_data/mysql/lib-03.err'. 230810 08:15:40 mysqld_safe Starting mariadbd daemon with databases from /mysql_data/mysql ./support-files/mysql.server: 264: kill: No such process
ERROR!
Thanks Rahul Raj
-----Original Message----- From: Axel Schwenke axel@askmonty.org Sent: Sunday, August 6, 2023 8:01 PM To: Raj, Rahul Rahul.Raj@amd.com; discuss@lists.mariadb.org Subject: Re: [MariaDB discuss] Re: Query on Building Maria DB for different Compression Algorithm
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
On 04.08.2023 11:57, Raj, Rahul via discuss wrote:
1. Can I stop building zlib algorithm? which is building by default, even after passing different plugin like LZ4, snappy etc.,. Any argument in cmake that can be passed to stop compiling ZLIB algorithm.
No. ZLIB is used internally, for example the COMPRESS() SQL function and the compressed client-server protocol. If no zlib is installed, the bundled (with MariaDB source) library is used.
2. Getting below error when linking 3rd party LZ4 library: In the below scenario build compilation is successfully. When trying to run below command to populate data folder to start server, it is throwing error.
*2023-08-02 17:56:58 0 [ERROR] mariadbd: Can't open shared library '/home/mariadb-10.11.3/maria_bin/lib/plugin/provider_lz4.so' (errno: 2, liblz4.so: cannot open shared object file: No such file or directory)*
The error message (from the MariaDB error log) says that liblz4.so could not be loaded by the mariadbd process. Where is it?
I have tried loading LD_LIBRARY_PATH, even that also produce the same result.
LD_LIBRARY_PATH is the right solution if liblz4.so is not in standard path. It must however be set in the environment from which the mariadbd process is started. Try this:
#sudo LD_LIBRARY_PATH=/path/to/liblz4 ./scripts/mariadb-install-db --srcdir=/home/mariadb-10.11.3 --plugin-dir=/home/mariadb-10.11.3/maria_bin/lib/plugin --user=mysql
It is probably easier to add the path containing liblz4.so to the standard search path for the dynamic linker. This would be in /etc/ld.so.conf or - depending on your Linux distribution - /etc/ld.so.conf.d/.
What are the contents of mysql/lib-03.err? I suspect you need to add LD_LIBRARY_PATH to the script. Also, Based on how you created the datadir, you may have to run mariadbd as root:
LD_LIBRARY_PATH=<path-to-liblz4> ./mysql.server start --user=root
On Thu, 10 Aug 2023 at 11:54, Raj, Rahul via discuss < discuss@lists.mariadb.org> wrote:
[AMD Official Use Only - General]
Hi Axel,
I followed your suggestion, and all the error has resolved but it didn't launch the Mariadb server. It throws below error:
#sudo ./support-files/mysql.server start Starting MariaDB .230810 08:15:40 mysqld_safe Logging to '/mysql_data/mysql/lib-03.err'. 230810 08:15:40 mysqld_safe Starting mariadbd daemon with databases from /mysql_data/mysql ./support-files/mysql.server: 264: kill: No such process
ERROR!
Thanks Rahul Raj
-----Original Message----- From: Axel Schwenke axel@askmonty.org Sent: Sunday, August 6, 2023 8:01 PM To: Raj, Rahul Rahul.Raj@amd.com; discuss@lists.mariadb.org Subject: Re: [MariaDB discuss] Re: Query on Building Maria DB for different Compression Algorithm
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
On 04.08.2023 11:57, Raj, Rahul via discuss wrote:
1. Can I stop building zlib algorithm? which is building by default,
even
after passing different plugin like LZ4, snappy etc.,. Any argument
in
cmake that can be passed to stop compiling ZLIB algorithm.
No. ZLIB is used internally, for example the COMPRESS() SQL function and the compressed client-server protocol. If no zlib is installed, the bundled (with MariaDB source) library is used.
2. Getting below error when linking 3rd party LZ4 library: In the below scenario build compilation is successfully. When trying to run below command to populate data folder to start server, it is throwing
error.
*2023-08-02 17:56:58 0 [ERROR] mariadbd: Can't open shared library '/home/mariadb-10.11.3/maria_bin/lib/plugin/provider_lz4.so' (errno: 2, liblz4.so: cannot open shared object file: No such file or directory)*
The error message (from the MariaDB error log) says that liblz4.so could not be loaded by the mariadbd process. Where is it?
I have tried loading LD_LIBRARY_PATH, even that also produce the same
result.
LD_LIBRARY_PATH is the right solution if liblz4.so is not in standard path. It must however be set in the environment from which the mariadbd process is started. Try this:
#sudo LD_LIBRARY_PATH=/path/to/liblz4 ./scripts/mariadb-install-db --srcdir=/home/mariadb-10.11.3 --plugin-dir=/home/mariadb-10.11.3/maria_bin/lib/plugin --user=mysql
It is probably easier to add the path containing liblz4.so to the standard search path for the dynamic linker. This would be in /etc/ld.so.conf or - depending on your Linux distribution - /etc/ld.so.conf.d/. _______________________________________________ discuss mailing list -- discuss@lists.mariadb.org To unsubscribe send an email to discuss-leave@lists.mariadb.org
[AMD Official Use Only - General]
Thank You Vicentiu!!! It worked.
Mysql/lib-03.err contains: 230810 11:49:22 mysqld_safe mysqld from pid file /mysql_data/mysql/lib-daytonax-03.pid ended 230810 11:49:39 mysqld_safe Starting mariadbd daemon with databases from /mysql_data/mysql 2023-08-10 11:49:39 0 [Note] Starting MariaDB 10.11.3-MariaDB source revision 0bb31039f54bd6a0dc8f0fc7d40e6b58a51998b0 as process 3271335 2023-08-10 11:49:39 0 [Warning] option 'table_open_cache_instances': unsigned value 128 adjusted to 64 2023-08-10 11:49:40 0 [ERROR] mariadbd: Can't open shared library '/home/MYSQL/mariadb-10.11.3/maria_bin/lib/plugin/provider_lz4.so' (errno: 2, liblz4.so: cannot open shared object file: No such file or directory) 2023-08-10 11:49:40 0 [ERROR] Couldn't load plugins from 'provider_lz4.so'. 2023-08-10 11:49:40 0 [Warning] mariadbd: Couldn't allocate 130023424 bytes (Large/HugeTLB memory page size 2097152); errno 12; continuing to smaller size 2023-08-10 11:49:40 0 [ERROR] mariadbd: InnoDB: compression algorithm lz4 (2) is not available. Please, load the corresponding provider plugin. 2023-08-10 11:49:40 0 [ERROR] Plugin 'InnoDB' init function returned error. 2023-08-10 11:49:40 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2023-08-10 11:49:40 0 [Note] Plugin 'FEEDBACK' is disabled. 2023-08-10 11:49:40 0 [Warning] 'default-authentication-plugin' is MySQL 5.6 / 5.7 compatible option. To be implemented in later versions. 2023-08-10 11:49:40 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files. 2023-08-10 11:49:40 0 [Warning] 'innodb-thread-concurrency' was removed. It does nothing now and exists only for compatibility with old my.cnf files. 2023-08-10 11:49:40 0 [ERROR] /home/MYSQL/mariadb-10.11.3/maria_bin/bin/mariadbd: unknown variable 'provider_lz4=force_plus_permanent' 2023-08-10 11:49:40 0 [ERROR] Aborting 230810 11:49:40 mysqld_safe mysqld from pid file /mysql_data/mysql/lib-daytonax-03.pid ended
It worked with: #sudo LD_LIBRARY_PATH=<path-to-liblz4> ./mysql.server start --user=mysql
Thank you again for all your inputs.
General question: Is there a way that can used instead of setting LD_LIBRARY_PATH each time when using it form different location. I mean to say one-time setting.
In above case I have to use LD_LIBRARY_PATH for each run of scripts.
Regards Rahul Raj From: Vicențiu Ciorbaru vicentiu@mariadb.org Sent: Thursday, August 10, 2023 3:08 PM To: Raj, Rahul Rahul.Raj@amd.com Cc: axel@askmonty.org; discuss@lists.mariadb.org Subject: Re: [MariaDB discuss] Re: Query on Building Maria DB for different Compression Algorithm
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
What are the contents of mysql/lib-03.err? I suspect you need to add LD_LIBRARY_PATH to the script. Also, Based on how you created the datadir, you may have to run mariadbd as root:
LD_LIBRARY_PATH=<path-to-liblz4> ./mysql.server start --user=root
On Thu, 10 Aug 2023 at 11:54, Raj, Rahul via discuss <discuss@lists.mariadb.orgmailto:discuss@lists.mariadb.org> wrote: [AMD Official Use Only - General]
Hi Axel,
I followed your suggestion, and all the error has resolved but it didn't launch the Mariadb server. It throws below error:
#sudo ./support-files/mysql.server start Starting MariaDB .230810 08:15:40 mysqld_safe Logging to '/mysql_data/mysql/lib-03.err'. 230810 08:15:40 mysqld_safe Starting mariadbd daemon with databases from /mysql_data/mysql ./support-files/mysql.server: 264: kill: No such process
ERROR!
Thanks Rahul Raj
-----Original Message----- From: Axel Schwenke <axel@askmonty.orgmailto:axel@askmonty.org> Sent: Sunday, August 6, 2023 8:01 PM To: Raj, Rahul <Rahul.Raj@amd.commailto:Rahul.Raj@amd.com>; discuss@lists.mariadb.orgmailto:discuss@lists.mariadb.org Subject: Re: [MariaDB discuss] Re: Query on Building Maria DB for different Compression Algorithm
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
On 04.08.2023 11:57, Raj, Rahul via discuss wrote:
1. Can I stop building zlib algorithm? which is building by default, even after passing different plugin like LZ4, snappy etc.,. Any argument in cmake that can be passed to stop compiling ZLIB algorithm.
No. ZLIB is used internally, for example the COMPRESS() SQL function and the compressed client-server protocol. If no zlib is installed, the bundled (with MariaDB source) library is used.
2. Getting below error when linking 3rd party LZ4 library: In the below scenario build compilation is successfully. When trying to run below command to populate data folder to start server, it is throwing error.
*2023-08-02 17:56:58 0 [ERROR] mariadbd: Can't open shared library '/home/mariadb-10.11.3/maria_bin/lib/plugin/provider_lz4.so' (errno: 2, liblz4.so: cannot open shared object file: No such file or directory)*
The error message (from the MariaDB error log) says that liblz4.so could not be loaded by the mariadbd process. Where is it?
I have tried loading LD_LIBRARY_PATH, even that also produce the same result.
LD_LIBRARY_PATH is the right solution if liblz4.so is not in standard path. It must however be set in the environment from which the mariadbd process is started. Try this:
#sudo LD_LIBRARY_PATH=/path/to/liblz4 ./scripts/mariadb-install-db --srcdir=/home/mariadb-10.11.3 --plugin-dir=/home/mariadb-10.11.3/maria_bin/lib/plugin --user=mysql
It is probably easier to add the path containing liblz4.so to the standard search path for the dynamic linker. This would be in /etc/ld.so.conf or - depending on your Linux distribution - /etc/ld.so.conf.d/. _______________________________________________ discuss mailing list -- discuss@lists.mariadb.orgmailto:discuss@lists.mariadb.org To unsubscribe send an email to discuss-leave@lists.mariadb.orgmailto:discuss-leave@lists.mariadb.org
participants (4)
-
Axel Schwenke -
Raj, Rahul -
Sergei Golubchik -
Vicențiu Ciorbaru