Re: [Maria-developers] Sphinx Storage engine for MariaDB

Andrew Aksyonoff

28 Jul 2009 28 Jul '09

4:43 a.m.

Hello Michael, Looks like I've overlooked this email back then. :( Peter pinged me about Sphinx vs Maria status recently and I just found it. Well, hopefully better late than never! Sunday, June 7, 2009, 1:18:30 PM, you wrote: MW> Andrew, what are the possible drawbacks you can see with having MW> Sphinx to be a part of MairaDB for a user that is not using Sphinx? Can't think of any. SphinxSE is a mere client and as such does not allocate any big RAM buffers or other resources. MW> I assume that if Sphinx is not enabled, it will not take any MW> resources. MW> If Sphinx is enabled but not used, what are the resorces it would use? Pretty much none, AFAIK. --- On an unrelated note, we're working on so called RT backend here, and it will allow all the normal CRUD operations in run time (as opposed to only having reads against a static fulltext index that we have now). When it's done it'll be also technically possible to integrate it too - and do so tighter by embedding the library instead of just talking to Sphinx searchd over network. Sphinx searchd can now talk MySQL protocol and supports basic SQL syntax. So for "just" full text tasks end users don't really need the integrated version. However it still might possibly be useful in certain use cases. To keep FT index in (better) sync with DB data, avoid overheads of double network roundtrips for additional processing, avoid hassles of keeping two connections and manually managing two open transactions, etc. So I wonder what'd be your opinion about the integration - whether it seems useful at all, and if yes, whether network client or embedded library route seems better. -- Best regards, Andrew mailto:shodan@shodan.ru

Show replies by date

Arjen Lentz

28 Jul 28 Jul

5:52 a.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi Andrew, all, On 28/07/2009, at 12:43 PM, Andrew Aksyonoff wrote:

...

Looks like I've overlooked this email back then. :( Peter pinged me about Sphinx vs Maria status recently and I just found it. Well, hopefully better late than never!

Sunday, June 7, 2009, 1:18:30 PM, you wrote: MW> Andrew, what are the possible drawbacks you can see with having MW> Sphinx to be a part of MairaDB for a user that is not using Sphinx?

Can't think of any. SphinxSE is a mere client and as such does not allocate any big RAM buffers or other resources.

MW> I assume that if Sphinx is not enabled, it will not take any MW> resources.

MW> If Sphinx is enabled but not used, what are the resorces it would use?

Pretty much none, AFAIK.

In 5.1 it's a pluggable. So the question is, do we want to compile it in, or compile it "with" and leave it as a pluggable. Then a user just has to explicitly INSTALL PLUGIN once, and otherwise there's no RAM use at all. Compiling it in has the advantage of everybody having it enabled, which make starting to use it a tiny bit easier. But it may be close to a moot point, in 5.1 Cheers, Arjen. -- Arjen Lentz, Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget. Follow our blog at http://openquery.com/blog/ OurDelta: free enhanced builds for MySQL @ http://ourdelta.org

Michael Widenius

30 Jul 30 Jul

10:02 a.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi!

...

...
...
...
...
"Arjen" == Arjen Lentz writes:

Arjen> Hi Andrew, all, Arjen> On 28/07/2009, at 12:43 PM, Andrew Aksyonoff wrote:

...

...
Looks like I've overlooked this email back then. :( Peter pinged me about Sphinx vs Maria status recently and I just found it. Well, hopefully better late than never!

Sunday, June 7, 2009, 1:18:30 PM, you wrote: MW> Andrew, what are the possible drawbacks you can see with having MW> Sphinx to be a part of MairaDB for a user that is not using Sphinx?

Can't think of any. SphinxSE is a mere client and as such does not allocate any big RAM buffers or other resources.

MW> I assume that if Sphinx is not enabled, it will not take any MW> resources.

...
MW> If Sphinx is enabled but not used, what are the resorces it

...
would use?

Pretty much none, AFAIK.

Arjen> In 5.1 it's a pluggable. Arjen> So the question is, do we want to compile it in, or compile it "with" Arjen> and leave it as a pluggable. Arjen> Then a user just has to explicitly INSTALL PLUGIN once, and otherwise Arjen> there's no RAM use at all. Arjen> Compiling it in has the advantage of everybody having it enabled, Arjen> which make starting to use it a tiny bit easier. Arjen> But it may be close to a moot point, in 5.1 I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible. Later we can consider if we want to have 'mariadb-min, mariadb and mariadb-max' versions that have different compilation/configure options. Regards, Monty

Arjen Lentz

6 Aug 6 Aug

1:08 a.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi Monty, all On 30/07/2009, at 6:02 PM, Michael Widenius wrote:

...

Arjen> In 5.1 it's a pluggable. Arjen> So the question is, do we want to compile it in, or compile it "with" Arjen> and leave it as a pluggable. Arjen> Then a user just has to explicitly INSTALL PLUGIN once, and otherwise Arjen> there's no RAM use at all.

Arjen> Compiling it in has the advantage of everybody having it enabled, Arjen> which make starting to use it a tiny bit easier. Arjen> But it may be close to a moot point, in 5.1

I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.

Sounds good, that was the exisitng plan for the 5.1 OurDelta builds we've been working with. The hassle was that some engines first objected to getting built in- tree, including PBXT and FederatedX. So that ate some time.

...

Later we can consider if we want to have 'mariadb-min, mariadb and mariadb-max' versions that have different compilation/configure options.

I don't think that makes sense. Since in 5.1 they're pluggable, we can later on just build them as a pluggable and people can INSTALL PLUGIN if they need the particular component. No need for different builds, that's just confusing (and a heck of a lot more -unnecessary- work) on the build end. Cheers, Arjen. -- Arjen Lentz, Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget. Follow our blog at http://openquery.com/blog/ OurDelta: free enhanced builds for MySQL @ http://ourdelta.org

Michael Widenius

16 Aug 16 Aug

3:53 p.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi!

...

...
...
...
...
"Arjen" == Arjen Lentz writes:

<cut>

...

...
Later we can consider if we want to have 'mariadb-min, mariadb and mariadb-max' versions that have different compilation/configure options.

Arjen> I don't think that makes sense. Since in 5.1 they're pluggable, we can Arjen> later on just build them as a pluggable and people can INSTALL PLUGIN Arjen> if they need the particular component. No need for different builds, Arjen> that's just confusing (and a heck of a lot more -unnecessary- work) on Arjen> the build end. There is still resons to have different builds: - It's easier to disable large features, for example geometrical objects, replication, mathematical functions by using defines than doing plugins. - From a distribution point of view, it's good to have a smaller distribution with only the critically needed plugins. But I agree, a plugin system where people could easily download plugins and get them automaticly installed would be preferable for those cases where it's possible. Regards, Monty

Sergei Golubchik

6 Aug 6 Aug

8:48 a.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi, Michael! On Jul 30, Michael Widenius wrote:

...

I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.

On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed. Regards / Mit vielen Grüßen, Sergei -- __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sergei Golubchik / /|_/ / // /\ \/ /_/ / /__ Principal Software Engineer/Server Architect /_/ /_/\_, /___/\___\_\___/ Sun Microsystems GmbH, HRB München 161028 <___/ Sonnenallee 1, 85551 Kirchheim-Heimstetten Geschäftsführer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Häring

Arjen Lentz

2:39 p.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi Sergei On 06/08/2009, at 4:48 PM, Sergei Golubchik wrote:

...

On Jul 30, Michael Widenius wrote:

...
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.

On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed.

We considered this earlier. It'd work for new installs, but not for existing ones so they'd have them all disabled until manually added. It's fine if that's what you intended, but it's important to realise this consequence. Cheers, Arjen. -- Arjen Lentz, Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget. Follow our blog at http://openquery.com/blog/ OurDelta: free enhanced builds for MySQL @ http://ourdelta.org

Sergei Golubchik

9:31 p.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi, Arjen! On Aug 06, Arjen Lentz wrote:

...

On 06/08/2009, at 4:48 PM, Sergei Golubchik wrote:

...
On Jul 30, Michael Widenius wrote:

...
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.

On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed.

We considered this earlier. It'd work for new installs, but not for existing ones so they'd have them all disabled until manually added.

That's not too different from other changes in system tables. mysql_fix_privilege_tables is generally recommended after an upgrade anyway. Regards / Mit vielen Grüßen, Sergei -- __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sergei Golubchik / /|_/ / // /\ \/ /_/ / /__ Principal Software Engineer/Server Architect /_/ /_/\_, /___/\___\_\___/ Sun Microsystems GmbH, HRB München 161028 <___/ Sonnenallee 1, 85551 Kirchheim-Heimstetten Geschäftsführer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Häring

Arjen Lentz

7 Aug 7 Aug

1:51 a.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi Sergei On 07/08/2009, at 5:31 AM, Sergei Golubchik wrote:

...

On Aug 06, Arjen Lentz wrote:

...
On 06/08/2009, at 4:48 PM, Sergei Golubchik wrote:

...
On Jul 30, Michael Widenius wrote:

...
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.

On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed.

We considered this earlier. It'd work for new installs, but not for existing ones so they'd have them all disabled until manually added.

That's not too different from other changes in system tables. mysql_fix_privilege_tables is generally recommended after an upgrade anyway.

Ah, so you want to put it in the mysql_fix_privilege_tables rather than in mysql_install_db? That can work. Monty, what do you reckon? Cheers, Arjen. -- Arjen Lentz, Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget. Follow our blog at http://openquery.com/blog/ OurDelta: free enhanced builds for MySQL @ http://ourdelta.org

Michael Widenius

10 Aug 10 Aug

1:47 p.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi!

...

...
...
...
...
"Sergei" == Sergei Golubchik writes:

Sergei> Hi, Arjen! Sergei> On Aug 06, Arjen Lentz wrote:

...

...
On 06/08/2009, at 4:48 PM, Sergei Golubchik wrote:

...
On Jul 30, Michael Widenius wrote:

...
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.

On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed.

We considered this earlier. It'd work for new installs, but not for existing ones so they'd have them all disabled until manually added.

Sergei> That's not too different from other changes in system tables. Sergei> mysql_fix_privilege_tables is generally recommended after an upgrade Sergei> anyway. mysql_fix_privilege_tables changes tables to be the newest version. We can't do this with the plugins, as we don't know for an old installation what plugins the user wanted to install. Just letting mysql_fix_privilege_tables install all plugins will thus not work. Regards, Monty

Sergei Golubchik

11 Aug 11 Aug

5:43 p.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi, Michael! On Aug 10, Michael Widenius wrote:

...

...
...
...
...
I think we should compile all the plugin in (but not enable all by default), at least for now, to make MariaDB as easy to use as possible.

On the opposite, I think you should compile them all out. But preinstall them, that is add necessary lines to mysql.plugin table from mysql_install_db. The user experience will be pretty much the same as if everything was compiled in, but one would be able to uninstall plugins easily, if needed.

We considered this earlier. It'd work for new installs, but not for existing ones so they'd have them all disabled until manually added.

...
That's not too different from other changes in system tables. mysql_fix_privilege_tables is generally recommended after an upgrade anyway.

mysql_fix_privilege_tables changes tables to be the newest version.

We can't do this with the plugins, as we don't know for an old installation what plugins the user wanted to install.

All of them, of course. If all plugins will be compiled in - they'll be all installed. Installing them all in mysql_fix_privilege_tables will produce basically the same effect. Regards / Mit vielen Grüßen, Sergei -- __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sergei Golubchik / /|_/ / // /\ \/ /_/ / /__ Principal Software Engineer/Server Architect /_/ /_/\_, /___/\___\_\___/ Sun Microsystems GmbH, HRB München 161028 <___/ Sonnenallee 1, 85551 Kirchheim-Heimstetten Geschäftsführer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Häring

Michael Widenius

16 Aug 16 Aug

3:57 p.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi!

...

...
...
...
...
"Sergei" == Sergei Golubchik writes:

<cut>

...

...
We can't do this with the plugins, as we don't know for an old installation what plugins the user wanted to install.

Sergei> All of them, of course. Sergei> If all plugins will be compiled in - they'll be all installed. Sergei> Installing them all in mysql_fix_privilege_tables will produce basically Sergei> the same effect. This doesn't work as we don't know which plugins that should be installed for an old installation. For an old installation, an user may already decided that some plugins should be installed and others should not. Adding things that he doesn't want in mysql_fix_privilege_tables is not an option. Another thing is that he may have used the source installation himself and compiled things into the source. In this case switching to a our distribution where things needs to be loaded would cause extra work for him. Regards, Monty

Sergei Golubchik

7:53 p.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hi, Michael! On Aug 16, Michael Widenius wrote:

...

Hi!

...
...
...
...
...
"Sergei" == Sergei Golubchik writes:

<cut>

...
...
We can't do this with the plugins, as we don't know for an old installation what plugins the user wanted to install.

...
All of them, of course. If all plugins will be compiled in - they'll be all installed. Installing them all in mysql_fix_privilege_tables will produce basically the same effect.

This doesn't work as we don't know which plugins that should be installed for an old installation.

For an old installation, an user may already decided that some plugins should be installed and others should not. Adding things that he doesn't want in mysql_fix_privilege_tables is not an option.

Another thing is that he may have used the source installation himself and compiled things into the source. In this case switching to a our distribution where things needs to be loaded would cause extra work for him.

I don't understand what you're talking about. How could installing everything via mysql.plugin be worse than installing everything by compiling everything in ? Can you come up with an example of an "old installation" where installing plugins from mysql_fix_privilege_tables is "not an option" ? Regards / Mit vielen Grüßen, Sergei -- __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sergei Golubchik / /|_/ / // /\ \/ /_/ / /__ Principal Software Engineer/Server Architect /_/ /_/\_, /___/\___\_\___/ Sun Microsystems GmbH, HRB München 161028 <___/ Sonnenallee 1, 85551 Kirchheim-Heimstetten Geschäftsführer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel Vorsitzender des Aufsichtsrates: Martin Häring

Peter Zaitsev

28 Jul 28 Jul

5:14 p.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Andrew, If you do local integration would you be able to use remote servers as well or only local indexes ? I think short term the integration you have is good the long term it is good to be able to have full text search functionality for data stored in any engine handled by Sphinx or something like that but that is a long way off.

...

So I wonder what'd be your opinion about the integration - whether it seems useful at all, and if yes, whether network client or embedded library route seems better.

-- Best regards, Andrew mailto:shodan@shodan.ru

-- Peter Zaitsev, CEO, Percona Inc. Tel: +1 888 401 3401 ext 501 Skype: peter_zaitsev 24/7 Emergency Line +1 888 401 3401 ext 911

Andrew Aksyonoff

5:54 p.m.

New subject: [Maria-developers] Sphinx Storage engine for MariaDB

Hello Peter, Tuesday, July 28, 2009, 7:14:28 PM, you wrote: PZ> If you do local integration would you be able to use remote PZ> servers as well or only local indexes ? It's all about handling local full text indexes from within MySQL without having to have a separate Sphinx instance. PZ> I think short term the integration you have is good the long term PZ> it is good to be able to have full text search functionality for PZ> data stored in any engine handled by Sphinx or something like that PZ> but that is a long way off. Yep, but maybe not that long... maybe closer than we think :) -- Best regards, Andrew mailto:shodan@shodan.ru

5501

Age (days ago)

5520

Last active (days ago)

List overview

Download

14 comments

5 participants

participants (5)

Andrew Aksyonoff
Arjen Lentz
Michael Widenius
Peter Zaitsev
Sergei Golubchik

Re: [Maria-developers] Sphinx Storage engine for MariaDB

tags

participants (5)