Re: [Maria-developers] Sphinx Storage engine for MariaDB
Hello Michael, Looks like I've overlooked this email back then. :( Peter pinged me about Sphinx vs Maria status recently and I just found it. Well, hopefully better late than never! Sunday, June 7, 2009, 1:18:30 PM, you wrote: MW> Andrew, what are the possible drawbacks you can see with having MW> Sphinx to be a part of MairaDB for a user that is not using Sphinx? Can't think of any. SphinxSE is a mere client and as such does not allocate any big RAM buffers or other resources. MW> I assume that if Sphinx is not enabled, it will not take any MW> resources. MW> If Sphinx is enabled but not used, what are the resorces it would use? Pretty much none, AFAIK. --- On an unrelated note, we're working on so called RT backend here, and it will allow all the normal CRUD operations in run time (as opposed to only having reads against a static fulltext index that we have now). When it's done it'll be also technically possible to integrate it too - and do so tighter by embedding the library instead of just talking to Sphinx searchd over network. Sphinx searchd can now talk MySQL protocol and supports basic SQL syntax. So for "just" full text tasks end users don't really need the integrated version. However it still might possibly be useful in certain use cases. To keep FT index in (better) sync with DB data, avoid overheads of double network roundtrips for additional processing, avoid hassles of keeping two connections and manually managing two open transactions, etc. So I wonder what'd be your opinion about the integration - whether it seems useful at all, and if yes, whether network client or embedded library route seems better. -- Best regards, Andrew mailto:shodan@shodan.ru
Hi Andrew, all, On 28/07/2009, at 12:43 PM, Andrew Aksyonoff wrote:
Looks like I've overlooked this email back then. :( Peter pinged me about Sphinx vs Maria status recently and I just found it. Well, hopefully better late than never!
Sunday, June 7, 2009, 1:18:30 PM, you wrote: MW> Andrew, what are the possible drawbacks you can see with having MW> Sphinx to be a part of MairaDB for a user that is not using Sphinx?
Can't think of any. SphinxSE is a mere client and as such does not allocate any big RAM buffers or other resources.
MW> I assume that if Sphinx is not enabled, it will not take any MW> resources.
MW> If Sphinx is enabled but not used, what are the resorces it would use?
Pretty much none, AFAIK.
In 5.1 it's a pluggable. So the question is, do we want to compile it in, or compile it "with" and leave it as a pluggable. Then a user just has to explicitly INSTALL PLUGIN once, and otherwise there's no RAM use at all. Compiling it in has the advantage of everybody having it enabled, which make starting to use it a tiny bit easier. But it may be close to a moot point, in 5.1 Cheers, Arjen. -- Arjen Lentz, Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget. Follow our blog at http://openquery.com/blog/ OurDelta: free enhanced builds for MySQL @ http://ourdelta.org
Hi!
"Arjen" == Arjen Lentz <arjen@openquery.com> writes:
Arjen> Hi Andrew, all, Arjen> On 28/07/2009, at 12:43 PM, Andrew Aksyonoff wrote:
Looks like I've overlooked this email back then. :( Peter pinged me about Sphinx vs Maria status recently and I just found it. Well, hopefully better late than never!
Sunday, June 7, 2009, 1:18:30 PM, you wrote: MW> Andrew, what are the possible drawbacks you can see with having MW> Sphinx to be a part of MairaDB for a user that is not using Sphinx?
Can't think of any. SphinxSE is a mere client and as such does not allocate any big RAM buffers or other resources.
MW> I assume that if Sphinx is not enabled, it will not take any MW> resources.
MW> If Sphinx is enabled but not used, what are the resorces it
would use?
Pretty much none, AFAIK.
Arjen> In 5.1 it's a pluggable. Arjen> So the question is, do we want to compile it in, or compile it "with" Arjen> and leave it as a pluggable. Arjen> Then a user just has to explicitly INSTALL PLUGIN once, and otherwise Arjen> there's no RAM use at all. Arjen> Compiling it in has the advantage of everybody having it enabled, Arjen> which make starting to use it a tiny bit easier. Arjen> But it may be close to a moot point, in 5.1 I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible. Later we can consider if we want to have 'mariadb-min, mariadb and mariadb-max' versions that have different compilation/configure options. Regards, Monty
Hi Monty, all On 30/07/2009, at 6:02 PM, Michael Widenius wrote:
Arjen> In 5.1 it's a pluggable. Arjen> So the question is, do we want to compile it in, or compile it "with" Arjen> and leave it as a pluggable. Arjen> Then a user just has to explicitly INSTALL PLUGIN once, and otherwise Arjen> there's no RAM use at all.
Arjen> Compiling it in has the advantage of everybody having it enabled, Arjen> which make starting to use it a tiny bit easier. Arjen> But it may be close to a moot point, in 5.1
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.
Sounds good, that was the exisitng plan for the 5.1 OurDelta builds we've been working with. The hassle was that some engines first objected to getting built in- tree, including PBXT and FederatedX. So that ate some time.
Later we can consider if we want to have 'mariadb-min, mariadb and mariadb-max' versions that have different compilation/configure options.
I don't think that makes sense. Since in 5.1 they're pluggable, we can later on just build them as a pluggable and people can INSTALL PLUGIN if they need the particular component. No need for different builds, that's just confusing (and a heck of a lot more -unnecessary- work) on the build end. Cheers, Arjen. -- Arjen Lentz, Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget. Follow our blog at http://openquery.com/blog/ OurDelta: free enhanced builds for MySQL @ http://ourdelta.org
Hi!
"Arjen" == Arjen Lentz <arjen@openquery.com> writes:
<cut>
Later we can consider if we want to have 'mariadb-min, mariadb and mariadb-max' versions that have different compilation/configure options.
Arjen> I don't think that makes sense. Since in 5.1 they're pluggable, we can Arjen> later on just build them as a pluggable and people can INSTALL PLUGIN Arjen> if they need the particular component. No need for different builds, Arjen> that's just confusing (and a heck of a lot more -unnecessary- work) on Arjen> the build end. There is still resons to have different builds: - It's easier to disable large features, for example geometrical objects, replication, mathematical functions by using defines than doing plugins. - From a distribution point of view, it's good to have a smaller distribution with only the critically needed plugins. But I agree, a plugin system where people could easily download plugins and get them automaticly installed would be preferable for those cases where it's possible. Regards, Monty
Hi, Michael! On Jul 30, Michael Widenius wrote:
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.
On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed. Regards / Mit vielen Grüßen, Sergei -- __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sergei Golubchik <serg@sun.com> / /|_/ / // /\ \/ /_/ / /__ Principal Software Engineer/Server Architect /_/ /_/\_, /___/\___\_\___/ Sun Microsystems GmbH, HRB München 161028 <___/ Sonnenallee 1, 85551 Kirchheim-Heimstetten Geschäftsführer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Häring
Hi Sergei On 06/08/2009, at 4:48 PM, Sergei Golubchik wrote:
On Jul 30, Michael Widenius wrote:
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.
On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed.
We considered this earlier. It'd work for new installs, but not for existing ones so they'd have them all disabled until manually added. It's fine if that's what you intended, but it's important to realise this consequence. Cheers, Arjen. -- Arjen Lentz, Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget. Follow our blog at http://openquery.com/blog/ OurDelta: free enhanced builds for MySQL @ http://ourdelta.org
Hi, Arjen! On Aug 06, Arjen Lentz wrote:
On 06/08/2009, at 4:48 PM, Sergei Golubchik wrote:
On Jul 30, Michael Widenius wrote:
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.
On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed.
We considered this earlier. It'd work for new installs, but not for existing ones so they'd have them all disabled until manually added.
That's not too different from other changes in system tables. mysql_fix_privilege_tables is generally recommended after an upgrade anyway. Regards / Mit vielen Grüßen, Sergei -- __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sergei Golubchik <serg@sun.com> / /|_/ / // /\ \/ /_/ / /__ Principal Software Engineer/Server Architect /_/ /_/\_, /___/\___\_\___/ Sun Microsystems GmbH, HRB München 161028 <___/ Sonnenallee 1, 85551 Kirchheim-Heimstetten Geschäftsführer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Häring
Hi Sergei On 07/08/2009, at 5:31 AM, Sergei Golubchik wrote:
On Aug 06, Arjen Lentz wrote:
On 06/08/2009, at 4:48 PM, Sergei Golubchik wrote:
On Jul 30, Michael Widenius wrote:
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.
On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed.
We considered this earlier. It'd work for new installs, but not for existing ones so they'd have them all disabled until manually added.
That's not too different from other changes in system tables. mysql_fix_privilege_tables is generally recommended after an upgrade anyway.
Ah, so you want to put it in the mysql_fix_privilege_tables rather than in mysql_install_db? That can work. Monty, what do you reckon? Cheers, Arjen. -- Arjen Lentz, Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget. Follow our blog at http://openquery.com/blog/ OurDelta: free enhanced builds for MySQL @ http://ourdelta.org
Hi!
"Sergei" == Sergei Golubchik <sergii@pisem.net> writes:
Sergei> Hi, Arjen! Sergei> On Aug 06, Arjen Lentz wrote:
On 06/08/2009, at 4:48 PM, Sergei Golubchik wrote:
On Jul 30, Michael Widenius wrote:
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.
On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed.
We considered this earlier. It'd work for new installs, but not for existing ones so they'd have them all disabled until manually added.
Sergei> That's not too different from other changes in system tables. Sergei> mysql_fix_privilege_tables is generally recommended after an upgrade Sergei> anyway. mysql_fix_privilege_tables changes tables to be the newest version. We can't do this with the plugins, as we don't know for an old installation what plugins the user wanted to install. Just letting mysql_fix_privilege_tables install all plugins will thus not work. Regards, Monty
Hi, Michael! On Aug 10, Michael Widenius wrote:
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.
On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed.
We considered this earlier. It'd work for new installs, but not for existing ones so they'd have them all disabled until manually added.
That's not too different from other changes in system tables. mysql_fix_privilege_tables is generally recommended after an upgrade anyway.
mysql_fix_privilege_tables changes tables to be the newest version.
We can't do this with the plugins, as we don't know for an old installation what plugins the user wanted to install.
All of them, of course. If all plugins will be compiled in - they'll be all installed. Installing them all in mysql_fix_privilege_tables will produce basically the same effect. Regards / Mit vielen Grüßen, Sergei -- __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sergei Golubchik <serg@sun.com> / /|_/ / // /\ \/ /_/ / /__ Principal Software Engineer/Server Architect /_/ /_/\_, /___/\___\_\___/ Sun Microsystems GmbH, HRB München 161028 <___/ Sonnenallee 1, 85551 Kirchheim-Heimstetten Geschäftsführer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Häring
Hi!
"Sergei" == Sergei Golubchik <sergii@pisem.net> writes:
<cut>
We can't do this with the plugins, as we don't know for an old installation what plugins the user wanted to install.
Sergei> All of them, of course. Sergei> If all plugins will be compiled in - they'll be all installed. Sergei> Installing them all in mysql_fix_privilege_tables will produce basically Sergei> the same effect. This doesn't work as we don't know which plugins that should be installed for an old installation. For an old installation, an user may already decided that some plugins should be installed and others should not. Adding things that he doesn't want in mysql_fix_privilege_tables is not an option. Another thing is that he may have used the source installation himself and compiled things into the source. In this case switching to a our distribution where things needs to be loaded would cause extra work for him. Regards, Monty
Hi, Michael! On Aug 16, Michael Widenius wrote:
Hi!
"Sergei" == Sergei Golubchik <sergii@pisem.net> writes:
<cut>
We can't do this with the plugins, as we don't know for an old installation what plugins the user wanted to install.
All of them, of course. If all plugins will be compiled in - they'll be all installed. Installing them all in mysql_fix_privilege_tables will produce basically the same effect.
This doesn't work as we don't know which plugins that should be installed for an old installation.
For an old installation, an user may already decided that some plugins should be installed and others should not. Adding things that he doesn't want in mysql_fix_privilege_tables is not an option.
Another thing is that he may have used the source installation himself and compiled things into the source. In this case switching to a our distribution where things needs to be loaded would cause extra work for him.
I don't understand what you're talking about. How could installing everything via mysql.plugin be worse than installing everything by compiling everything in ? Can you come up with an example of an "old installation" where installing plugins from mysql_fix_privilege_tables is "not an option" ? Regards / Mit vielen Grüßen, Sergei -- __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sergei Golubchik <serg@sun.com> / /|_/ / // /\ \/ /_/ / /__ Principal Software Engineer/Server Architect /_/ /_/\_, /___/\___\_\___/ Sun Microsystems GmbH, HRB München 161028 <___/ Sonnenallee 1, 85551 Kirchheim-Heimstetten Geschäftsführer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Häring
Andrew, If you do local integration would you be able to use remote servers as well or only local indexes ? I think short term the integration you have is good the long term it is good to be able to have full text search functionality for data stored in any engine handled by Sphinx or something like that but that is a long way off.
So I wonder what'd be your opinion about the integration - whether it seems useful at all, and if yes, whether network client or embedded library route seems better.
-- Best regards, Andrew mailto:shodan@shodan.ru
-- Peter Zaitsev, CEO, Percona Inc. Tel: +1 888 401 3401 ext 501 Skype: peter_zaitsev 24/7 Emergency Line +1 888 401 3401 ext 911
Hello Peter, Tuesday, July 28, 2009, 7:14:28 PM, you wrote: PZ> If you do local integration would you be able to use remote PZ> servers as well or only local indexes ? It's all about handling local full text indexes from within MySQL without having to have a separate Sphinx instance. PZ> I think short term the integration you have is good the long term PZ> it is good to be able to have full text search functionality for PZ> data stored in any engine handled by Sphinx or something like that PZ> but that is a long way off. Yep, but maybe not that long... maybe closer than we think :) -- Best regards, Andrew mailto:shodan@shodan.ru
participants (5)
-
Andrew Aksyonoff -
Arjen Lentz -
Michael Widenius -
Peter Zaitsev -
Sergei Golubchik