[Maria-discuss] Which binaries include support for compression zstd and lz4?
Hello all,
I would like to have the most recent MariaDB 10.5.4 with the MyRocks plugin and support for LZ4 *and* ZSTD compression on a Debian-based Linux distro.
So far I have found:
- Ubuntu 18.04, MariaDB 10.2.26 binaries support LZ4 and ZSTD (not final in rocksdb) - Debian 10, MariaDB 10.4.13 supports only LZ4 - Ubuntu 20.04, MariaDB 10.3 binaries from vanilla apt repositories (not those from the repository configuration tool on mariadb.org) support LZ4 and ZSTD (not final) - Ubuntu 20.04, MariaDB 10.5.4 from mirror.klaus-uwe.me supports only LZ4 - Ubuntu 20.04, MariaDB 10.5.4 from ftp.utexas.edu supports only LZ4 (I assume that mirrors are really mirrors and not individual compiled versions, however I have tried out of curiosity)
Is there any way to tell which compression algos are supported by which releases on which distros?
To get ZSTD support I have tried to compile 10.5.4 on Ubuntu 20.04 from source following this guide https://mariadb.com/kb/en/building-mariadb-on-ubuntu/ but it failed somewhere when it was trying to compile the column store. I am no expert in compiling and I would rather use binaries, if available.
Thanks in advance
Jonas
Hello!
Ubuntu 18.04, MariaDB 10.2.26 binaries support LZ4 and ZSTD (not final in rocksdb)
What do you mean by "not final in RocksDB"?
Is there any way to tell which compression algos are supported by which releases on which distros?
Not at the moment. The page https://mariadb.com/kb/en/innodb-page-compression/#configuring-the-innodb-pa... lists what options in theory are available, but in reality it varies on what was enabled in the build.
RocksDB compression in a separate thing, and documented at https://mariadb.com/kb/en/myrocks-and-data-compression/
In Debian.org (and Ubuntu) repositories you can find MariaDB 10.3 which supports both lz4 and zstd both in InnoDB and in RocksDB.
We have work in progress to upstream this support to the upstream MariaDB 10.5 so that users would have the option to choose these whenever they want. In fact, there has been a PR (https://github.com/MariaDB/server/pull/1582) pending for some weeks now but there is a shortage of manpower to review and test everything.
If your company Stockpulse.de has a support contract with MariaDB.com I recommend you notify your contact person that this feature is important for you.
- Otto
Hi, Jonas!
On Jun 27, Jonas Krauss wrote:
Hello all,
I would like to have the most recent MariaDB 10.5.4 with the MyRocks plugin and support for LZ4 *and* ZSTD compression on a Debian-based Linux distro.
So far I have found:
- Ubuntu 18.04, MariaDB 10.2.26 binaries support LZ4 and ZSTD (not final in rocksdb) - Debian 10, MariaDB 10.4.13 supports only LZ4 - Ubuntu 20.04, MariaDB 10.3 binaries from vanilla apt repositories (not those from the repository configuration tool on mariadb.org) support LZ4 and ZSTD (not final) - Ubuntu 20.04, MariaDB 10.5.4 from mirror.klaus-uwe.me supports only LZ4 - Ubuntu 20.04, MariaDB 10.5.4 from ftp.utexas.edu supports only LZ4 (I assume that mirrors are really mirrors and not individual compiled versions, however I have tried out of curiosity)
It seems that zstd support has disappeared recently because of a bug introduced as a merge mistake.
I've fixed it, so the next release should have zstd again. And we're adding tests to make sure this won't happen again. Sorry for this.
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
The zstd bugfix is the second cimmit in my PR has been open for 20 days. I hope Serg you cherry picked it instead of remaking it
https://github.com/MariaDB/server/commit/b4a532b27bf1ab1fd1ff2a5db9638630010...
la 27. kesäk. 2020 klo 19.58 Sergei Golubchik serg@mariadb.org kirjoitti:
Hi, Jonas!
On Jun 27, Jonas Krauss wrote:
Hello all,
I would like to have the most recent MariaDB 10.5.4 with the MyRocks
plugin
and support for LZ4 *and* ZSTD compression on a Debian-based Linux
distro.
So far I have found:
- Ubuntu 18.04, MariaDB 10.2.26 binaries support LZ4 and ZSTD (not
final
in rocksdb) - Debian 10, MariaDB 10.4.13 supports only LZ4 - Ubuntu 20.04, MariaDB 10.3 binaries from vanilla apt repositories
(not
those from the repository configuration tool on mariadb.org) support
LZ4
and ZSTD (not final) - Ubuntu 20.04, MariaDB 10.5.4 from mirror.klaus-uwe.me supports
only LZ4
- Ubuntu 20.04, MariaDB 10.5.4 from ftp.utexas.edu supports only LZ4
(I
assume that mirrors are really mirrors and not individual compiled versions, however I have tried out of curiosity)
It seems that zstd support has disappeared recently because of a bug introduced as a merge mistake.
I've fixed it, so the next release should have zstd again. And we're adding tests to make sure this won't happen again. Sorry for this.
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
Hi, Otto!
I did remake it, didn't know it was in your PR. My fix was almost exactly the same as yours, down to missing underscores in the error message. I've had one change inside Findzstd.cmake, but that's the only difference.
In my defense, I've done it in 10.3, where the bug was introduced.
My commits are still in the stage branch, so if you'd like I can push your commit instead.
On Jun 27, Otto Kekäläinen wrote:
The zstd bugfix is the second cimmit in my PR has been open for 20 days. I hope Serg you cherry picked it instead of remaking it
https://github.com/MariaDB/server/commit/b4a532b27bf1ab1fd1ff2a5db9638630010...
la 27. kesäk. 2020 klo 19.58 Sergei Golubchik serg@mariadb.org kirjoitti:
It seems that zstd support has disappeared recently because of a bug introduced as a merge mistake.
I've fixed it, so the next release should have zstd again. And we're adding tests to make sure this won't happen again. Sorry for this.
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
Hi Sergei,
I've fixed it, so the next release should have zstd again.
And we're adding tests to make sure this won't happen again. Sorry for this.
No problem, I am glad this has been taken care of already. Will wait for the next release then.
Otto:
If you are a coder and want to spend development time yourself, then I
suggest you contribute to the PR review instead of spending time
building sources and maintaining repositories of your own.
As a matter of fact I am a coder, however I am not sure if I could be of help here (depends on the used languages and complexity of the code base). As we at Stockpulse are relying on MariaDB and have a good experience in general with the database I would be glad to contribute in one way or another. So I will have a look at the PR and see if I would be able to provide support, instead of trying to compile from source. Thanks for the advice.
Best
Jonas
Am Sa., 27. Juni 2020 um 19:16 Uhr schrieb Sergei Golubchik < serg@mariadb.org>:
Hi, Otto!
I did remake it, didn't know it was in your PR. My fix was almost exactly the same as yours, down to missing underscores in the error message. I've had one change inside Findzstd.cmake, but that's the only difference.
In my defense, I've done it in 10.3, where the bug was introduced.
My commits are still in the stage branch, so if you'd like I can push your commit instead.
On Jun 27, Otto Kekäläinen wrote:
The zstd bugfix is the second cimmit in my PR has been open for 20 days.
I
hope Serg you cherry picked it instead of remaking it
https://github.com/MariaDB/server/commit/b4a532b27bf1ab1fd1ff2a5db9638630010...
la 27. kesäk. 2020 klo 19.58 Sergei Golubchik serg@mariadb.org
kirjoitti:
It seems that zstd support has disappeared recently because of a bug introduced as a merge mistake.
I've fixed it, so the next release should have zstd again. And we're adding tests to make sure this won't happen again. Sorry for this.
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
Hello Jonas!
If you are a coder and want to spend development time yourself, then I
suggest you contribute to the PR review instead of spending time building sources and maintaining repositories of your own.
As a matter of fact I am a coder, however I am not sure if I could be of help here (depends on the used languages and complexity of the code base). As we at Stockpulse are relying on MariaDB and have a good experience in general with the database I would be glad to contribute in one way or another. So I will have a look at the PR and see if I would be able to provide support, instead of trying to compile from source. Thanks for the advice.
You don't need to code yourself or understand the code base, you can already help by looking at the commit messages, inline comments, SQL commands and other parts you do understand and comment on them. I also created a PPA for you so you can install the binaries on Ubuntu 20.04 (but only as a test) and verify that the version in the PR fixes the issues you reported: https://github.com/MariaDB/server/pull/1582#issuecomment-650629202
Any thumbs up or comments to the PR will help push it forward in the review pipeline, as external confirmation is always encouraging to the reviewer that the PR is correct and valid.
- Otto
I did remake it, didn't know it was in your PR. My fix was almost exactly the same as yours, down to missing underscores in the error message. I've had one change inside Findzstd.cmake, but that's the only difference.
In my defense, I've done it in 10.3, where the bug was introduced.
My commits are still in the stage branch, so if you'd like I can push your commit instead.
Yes, I would appreciate that. In general, all of my PRs that have multiple commits are built so that each commit is pretty much self-standing and you are free to cherry-pick commits from them if you don't want to merge the entire PR.
There are also other PRs where I have fixed or polished things from your previous commits. I know you are very busy with many things, but maybe it would after all save you some time to review those PRs and merge them in full or partially to avoid yourself having to spend time on fixing issues that are already fixed. For example today I self-assigned myself a MDEV you had assigned and the fix is pending for review already.
participants (3)
-
Jonas Krauss -
Otto Kekäläinen -
Sergei Golubchik