[Maria-developers] request for comment: OQGRAPH in 5.1 packages

Arjen Lentz

30 Oct 2009 30 Oct '09

9:35 p.m.

Hi all, fellow Maria captains in particular (but naturally anybody here can comment) We'd like to include the plugin for OQGRAPH engine in the 5.1 packages we're just about to build. It would not be pulled in like the xtradb/pbxt engines, but be compiled separately and not loaded in by default (people will have to do INSTALL PLUGIN) so that as long as it's not loaded, it can have no influence on the running of mysqld. The rationale is this. From the experience with PBXT, people really want/need binaries/ packages before they will try things. If 5.1 binaries had had PBXT plugin sitting there, lots more people would have tried it earlier, filed bugreports and feedback, and Paul would have been where he is now much quicker. With MariaDB pulling it in it's ok now, but it's just a darn waste and pity of the earlier time. Since 5.1's plugin infrastructure still requires a plugin to be compiled against close to exact the original mysqld source, the only way to ensure that is to compile them from the same source at the same time, next to eachother. So that's what I'm proposing. Nonsense like the feature preview builds that Sun/MySQL did just make no sense in the real world, people can't use that. So while sticking new plugins in a future version like 5.2 appears sensible, it doesn't actually help in getting the code out there and used which is of course the only way to get feedback and bugreports. The ability to have plugins distributed but not loaded is the key here, it allows us to get stuff out and those who want to try it can, without destabilising anything for those who don't. (On the practical side, since it's essentially separate it would get added during the source tarball prep in the builds, so no action required inside the maria tree) With the packages getting prepped now, I realise it's a tad short notice. Don't feel rushed, but do please reply when you read this, otherwise it might get lost in your email pile (I know my friends, as I know myself ;-) Thanks Cheers, Arjen.

Show replies by date

Hakan Kuecuekyilmaz

1 Nov 1 Nov

1:15 p.m.

Arjen, all, I like the idea of putting community developed storage engines into our releases. On 30.10.2009, at 21:35, Arjen Lentz wrote:

...

Hi all, fellow Maria captains in particular (but naturally anybody here can comment)

We'd like to include the plugin for OQGRAPH engine in the 5.1 packages we're just about to build. It would not be pulled in like the xtradb/pbxt engines, but be compiled separately and not loaded in by default (people will have to do INSTALL PLUGIN) so that as long as it's not loaded, it can have no influence on the running of mysqld.

It would not be loaded by default, but if you load the engine, then it surely will have influence on mysqld.

...

The rationale is this. From the experience with PBXT, people really want/need binaries/ packages before they will try things. If 5.1 binaries had had PBXT plugin sitting there, lots more people would have tried it earlier, filed bugreports and feedback, and Paul would have been where he is now much quicker. With MariaDB pulling it in it's ok now, but it's just a darn waste and pity of the earlier time.

I would like to be strict in our release policy. Once we declare a release beta, then we should be feature complete and no new features should be added. Otherwise, there will be n "cool", "harmless", "trivial", and so on additions people may want to add.

...

Since 5.1's plugin infrastructure still requires a plugin to be compiled against close to exact the original mysqld source, the only way to ensure that is to compile them from the same source at the same time, next to eachother. So that's what I'm proposing.

The plugin infrastructure and storage engine interface is not as clean as one would guess. Therefore a new engine adds new risks to the run time.

...

Nonsense like the feature preview builds that Sun/MySQL did just make no sense in the real world, people can't use that. So while sticking new plugins in a future version like 5.2 appears sensible, it doesn't actually help in getting the code out there and used which is of course the only way to get feedback and bugreports. The ability to have plugins distributed but not loaded is the key here, it allows us to get stuff out and those who want to try it can, without destabilising anything for those who don't.

I don't see any need for feature preview stuff. We will have at least a stable branch and a development branch. I think that we aim for way shorter alpha to beta release cycles; 3 to 6 months would be nice.

...

(On the practical side, since it's essentially separate it would get added during the source tarball prep in the builds, so no action required inside the maria tree)

With the packages getting prepped now, I realise it's a tad short notice. Don't feel rushed, but do please reply when you read this, otherwise it might get lost in your email pile (I know my friends, as I know myself ;-) Thanks

You are right, it would not matter, if nobody loads that engine. But people will load it and it can lead to instabilities. My answer is: no new features after beta. Best regards, Hakan -- Hakan Küçükyılmaz, QA/Benchmark Engineer, Stuttgart/Germany Monty Program Ab, http://askmonty.org/ Skype: hakank_ Phone: +49 171 1919839

Alex Yurchenko

2:30 p.m.

...

On 30.10.2009, at 21:35, Arjen Lentz wrote: <skip>

...
The rationale is this. From the experience with PBXT, people really want/need binaries/ packages before they will try things. If 5.1 binaries had had PBXT plugin sitting there, lots more people would have tried it earlier, filed bugreports and feedback, and Paul would have been where he is now much quicker. With MariaDB pulling it in it's ok now, but it's just a darn waste and pity of the earlier time.

I think Arjen makes a pretty good point here. Plugin architecture offers a remarkable opportunity to safely advertise and promote new stuff. The

On Sun, 1 Nov 2009 13:15:31 +0100, Hakan Kuecuekyilmaz wrote: <skip> possibility to easily try new promising plugins without a hassle of downloading and compiling sources would surely be welcomed by the users and add value to MariaDB. In some cases it might be a decisive factor in choosing between MySQL and MariaDB. <skip>

...

You are right, it would not matter, if nobody loads that engine. But people will load it and it can lead to instabilities.

If people don't want to try this engine, they will hardly load it. If people want to try this engine, then some of them will download sources, compile, install - "and it can lead to instabilities". You just make their lifes harder by not including the binary.

...

My answer is: no new features after beta.

If you call plugin a "feature", what sort of "plugin" is that? The whole point of a plugin concept is to get rid of "features", isn't it? Sincerely, Alex -- Alexey Yurchenko, Codership Oy, www.codership.com Skype: alexey.yurchenko, Phone: +358-400-516-011

Henrik Ingo

2 Nov 2 Nov

9:50 a.m.

Noting also Hakan's and Alex' comments... On Fri, Oct 30, 2009 at 10:35 PM, Arjen Lentz wrote:

...

We'd like to include the plugin for OQGRAPH engine in the 5.1 packages we're just about to build. It would not be pulled in like the xtradb/pbxt engines, but be compiled separately and not loaded in by default (people will have to do INSTALL PLUGIN) so that as long as it's not loaded, it can have no influence on the running of mysqld.

Someone would of course have to look at the OQGRAPH engine itself to have an opinion on "worthyness". That is not something for me.

...

The rationale is this. From the experience with PBXT, people really want/need binaries/packages before they will try things. If 5.1 binaries had had PBXT plugin sitting there, lots more people would have tried it earlier, filed bugreports and feedback, and Paul would have been where he is now much quicker. With MariaDB pulling it in it's ok now, but it's just a darn waste and pity of the earlier time.

Since 5.1's plugin infrastructure still requires a plugin to be compiled against close to exact the original mysqld source, the only way to ensure that is to compile them from the same source at the same time, next to eachother. So that's what I'm proposing.

Yes. We have some ideas how to enable an even richer ecosystem of plugins than what is possible now. Adding storage engines is fine, but potentially people could start producing hundreds of plugins, and we don't want to have on giant RPM with hundreds of megs of plugins. But that is for the future, for now what you say makes sense.

...

Nonsense like the feature preview builds that Sun/MySQL did just make no sense in the real world, people can't use that. So while sticking new plugins in a future version like 5.2 appears sensible, it doesn't actually help in getting the code out there and used which is of course the only way to get feedback and bugreports. The ability to have plugins distributed but not loaded is the key here, it allows us to get stuff out and those who want to try it can, without destabilising anything for those who don't.

I'm not sure I understand how you meant this to happen, but as I see it we have been working on building even "what we have now". So until that is working, the answer would be a firm no. (I was on vacation last week, but I see we have a beta out now for some platforms, so I take it your proposal is meant for the "what you'd like to do next" discussion, so possibly we are in full agreement here.)

...

(On the practical side, since it's essentially separate it would get added during the source tarball prep in the builds, so no action required inside the maria tree)

Everything that is needed to produce the MariaDB builds must exist in the maria launchpad tree (except for the VM images where the build happened, but these should also be available). Things must not have been "pulled in from here and there". The other thing, why I'm writing this at all, is that you are probably not aware that our plans for 5.2 is for it to be a really short release cycle, following essentially immediately upon 5.1. So the idea since August has been to not touch 5.1 more than "make it work" and put all new stuff in 5.2, and even for 5.2 things have to be 95% ready, no major new features built from scratch, that is for 5.4. henrik -- email: henrik.ingo@avoinelama.fi tel: +358-40-5697354 www: www.avoinelama.fi/~hingo book: www.openlife.cc

Michael Widenius

3:01 p.m.

Hi!

...

...
...
...
...
"Arjen" == Arjen Lentz writes:

Arjen> Hi all, fellow Maria captains in particular (but naturally anybody Arjen> here can comment) Arjen> We'd like to include the plugin for OQGRAPH engine in the 5.1 packages Arjen> we're just about to build. Arjen> It would not be pulled in like the xtradb/pbxt engines, but be Arjen> compiled separately and not loaded in by default (people will have to Arjen> do INSTALL PLUGIN) so that as long as it's not loaded, it can have no Arjen> influence on the running of mysqld. Arjen> The rationale is this. Arjen> From the experience with PBXT, people really want/need binaries/ Arjen> packages before they will try things. If 5.1 binaries had had PBXT Arjen> plugin sitting there, lots more people would have tried it earlier, Arjen> filed bugreports and feedback, and Paul would have been where he is Arjen> now much quicker. With MariaDB pulling it in it's ok now, but it's Arjen> just a darn waste and pity of the earlier time. Arjen> Since 5.1's plugin infrastructure still requires a plugin to be Arjen> compiled against close to exact the original mysqld source, the only Arjen> way to ensure that is to compile them from the same source at the same Arjen> time, next to eachother. Arjen> So that's what I'm proposing. Arjen> Nonsense like the feature preview builds that Sun/MySQL did just make Arjen> no sense in the real world, people can't use that. So while sticking Arjen> new plugins in a future version like 5.2 appears sensible, it doesn't Arjen> actually help in getting the code out there and used which is of Arjen> course the only way to get feedback and bugreports. The ability to Arjen> have plugins distributed but not loaded is the key here, it allows us Arjen> to get stuff out and those who want to try it can, without Arjen> destabilising anything for those who don't.

...

From my point of view, I think it's ok that we add 'alpha' storage engines, that are not loaded by default, to the MariaDB tree.

In my view, an engine that is just distributed with MariaDB will not downgrade the overall quality of MariaDB itself. I would however like to suggest that when we impose the following restrictions to any engine code that are to be distributed with MariaDB: - The engine should be 'useful for a large number of people'. - The engine will not cause any delays (except build & test run times) when doing MariaDB releases. - The code needs to compile without any warnings or errors. - The code should work on all platforms. - Any hard bugs (server crashes or security issues) should be solved ASAP (preferably within 1 week) - Any reasonable-to-fix bugs should be solved promptly (within 2 weeks or before next MariaDB release) - There should be reasonable tests for the engine to ensure that it works. If OQGRAPH satisfies the above requirements (or Arjen promises that all issues will be taken care of), I am ok to add it to MariaDB 5.1 Comments? Regards, Monty PS: Yes, I know that this means we should try to get sphinx into MariaDB 5.1 too ASAP.

Peter Laursen

3:14 p.m.

+ there should be documentation of reasonable standard available for such engine. On Mon, Nov 2, 2009 at 15:01, Michael Widenius wrote:

...

Hi!

...
...
...
...
...
"Arjen" == Arjen Lentz writes:

Arjen> Hi all, fellow Maria captains in particular (but naturally anybody Arjen> here can comment)

Arjen> We'd like to include the plugin for OQGRAPH engine in the 5.1 packages Arjen> we're just about to build. Arjen> It would not be pulled in like the xtradb/pbxt engines, but be Arjen> compiled separately and not loaded in by default (people will have to Arjen> do INSTALL PLUGIN) so that as long as it's not loaded, it can have no Arjen> influence on the running of mysqld.

Arjen> The rationale is this. Arjen> From the experience with PBXT, people really want/need binaries/ Arjen> packages before they will try things. If 5.1 binaries had had PBXT Arjen> plugin sitting there, lots more people would have tried it earlier, Arjen> filed bugreports and feedback, and Paul would have been where he is Arjen> now much quicker. With MariaDB pulling it in it's ok now, but it's Arjen> just a darn waste and pity of the earlier time.

Arjen> Since 5.1's plugin infrastructure still requires a plugin to be Arjen> compiled against close to exact the original mysqld source, the only Arjen> way to ensure that is to compile them from the same source at the same Arjen> time, next to eachother. Arjen> So that's what I'm proposing.

Arjen> Nonsense like the feature preview builds that Sun/MySQL did just make Arjen> no sense in the real world, people can't use that. So while sticking Arjen> new plugins in a future version like 5.2 appears sensible, it doesn't Arjen> actually help in getting the code out there and used which is of Arjen> course the only way to get feedback and bugreports. The ability to Arjen> have plugins distributed but not loaded is the key here, it allows us Arjen> to get stuff out and those who want to try it can, without Arjen> destabilising anything for those who don't.

...
From my point of view, I think it's ok that we add 'alpha' storage engines, that are not loaded by default, to the MariaDB tree.

In my view, an engine that is just distributed with MariaDB will not downgrade the overall quality of MariaDB itself.

I would however like to suggest that when we impose the following restrictions to any engine code that are to be distributed with MariaDB:

- The engine should be 'useful for a large number of people'. - The engine will not cause any delays (except build & test run times) when doing MariaDB releases. - The code needs to compile without any warnings or errors. - The code should work on all platforms. - Any hard bugs (server crashes or security issues) should be solved ASAP (preferably within 1 week) - Any reasonable-to-fix bugs should be solved promptly (within 2 weeks or before next MariaDB release) - There should be reasonable tests for the engine to ensure that it works.

If OQGRAPH satisfies the above requirements (or Arjen promises that all issues will be taken care of), I am ok to add it to MariaDB 5.1

Comments?

Regards, Monty

PS: Yes, I know that this means we should try to get sphinx into MariaDB 5.1 too ASAP.

_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp

Kristian Nielsen

4 Nov 4 Nov

1:40 p.m.

Arjen Lentz writes:

...

packages before they will try things. If 5.1 binaries had had PBXT plugin sitting there, lots more people would have tried it earlier, filed bugreports and feedback, and Paul would have been where he is now much quicker. With MariaDB pulling it in it's ok now, but it's just a darn waste and pity of the earlier time.

I also think this is a very clear description of an important point. ++

...

(On the practical side, since it's essentially separate it would get added during the source tarball prep in the builds, so no action required inside the maria tree)

It is _much_ better to get it inside the tree. Everything in MariaDB development revolves around the bzr tree. Development, buildbot, testing etc. etc. If it's not in the tree, it will be broken by some MariaDB change. And you will discover it only during release work. And we will have to delay the release or omit ograph from that release, and generally waste lots of time and effort. And you will be on your own to fix things. If it's in the tree we will catch problems immediately and have the longest possible time to plan and fix. Of course it's your code so it's your call in the end. But I just don't understand why you would _not_ want it in the tree. Is there some specific concern(s) that we could perhaps address? We can use bzr merge-into (like with xtradb) so you can still maintain a stand-alone tree. We accept storage engines like this into the tree without any copyright assignment, just the GPL 2 is fine. I know I'm preaching in-tree and Buildbot to you all the time :-). But it really is the way development needs to be done to scale to the level we want and utilise the limited resources we have. - Kristian.

Arjen Lentz

2:37 p.m.

Hi Kristian, Hakan, On 04/11/2009, at 10:40 PM, Kristian Nielsen wrote:

...

Arjen Lentz writes:

...
packages before they will try things. If 5.1 binaries had had PBXT plugin sitting there, lots more people would have tried it earlier, filed bugreports and feedback, and Paul would have been where he is now much quicker. With MariaDB pulling it in it's ok now, but it's just a darn waste and pity of the earlier time.

I also think this is a very clear description of an important point.

++

thanks The PBXT thing has irked me for years now, and while PBXT is now in MariaDB it'll be great to see a general solution to the issue.

...

...
(On the practical side, since it's essentially separate it would get added during the source tarball prep in the builds, so no action required inside the maria tree)

It is _much_ better to get it inside the tree.

Everything in MariaDB development revolves around the bzr tree. Development, buildbot, testing etc. etc.

If it's not in the tree, it will be broken by some MariaDB change. And you will discover it only during release work. And we will have to delay the release or omit ograph from that release, and generally waste lots of time and effort. And you will be on your own to fix things.

If it's in the tree we will catch problems immediately and have the longest possible time to plan and fix.

Of course it's your code so it's your call in the end. But I just don't understand why you would _not_ want it in the tree.

Is there some specific concern(s) that we could perhaps address?

We can use bzr merge-into (like with xtradb) so you can still maintain a stand-alone tree.

We accept storage engines like this into the tree without any copyright assignment, just the GPL 2 is fine.

I know I'm preaching in-tree and Buildbot to you all the time :-). But it really is the way development needs to be done to scale to the level we want and utilise the limited resources we have.

I'm fine with having OQGRAPH in the MariaDB tree, on a merge-into basis, but I do feel it's important to address all the issues here: OQGRAPH is very new. since it's not a general purpose storage engine the maturity curve is theoretically different; but Open Query does not want to get some special treatment for whatever reason. Whatever set of "rules" we decide on here should apply to all plugins. I also appreciate the comments made by for instance Hakan, who feels that once a tree is in its beta phase, no new code (not even non- loaded plugins) should be added. While those plugins would not be installed by default, they'd still be quite present. So this is a perfectly valid position to take. Since we're building packages, it's possible to make say a mariadb- xxxx package that contains just the plugin xxxx. In that scenario, it does not actually matter whether the plugin comes from the mariadb tree or not, it would be compiled against it anyway. To make this decently workable, we need to sort out the "need an entire mariadb source tree to build the plugin" problem. We filed a wishlist bug for this, https://bugs.launchpad.net/maria/+bug/470580 which also includes a patch by Antony Curtis. It creates a mysql- glob.h during the build process, which we could put into a mariadb- plugin-dev package. Then, some mariadb-xxxx packages can be build in the main mariadb build, and some can be separate. It also allows developers to work on plugins sanely. It cleans up the entire dev environment for this - it's not 100% pretty as it's a nasty big include, but it's a good step given the current mess. Now to return to the original trail, perhaps Hakan would be ok with plugins being in separate packages, then there is no chance someone could have code installed that is not necessarily production ready, if they haven't explicitly chosen to install it. The problem I'm trying to resolve here is still essentially the PBXT issue. It's really important that new plugins get tested so they can feedback and bugreports, and that only happens if they're available in easy binary form. However, having them available for the current production version (or anything >= beta) as well as alpha stage versions, is important. The cycle from alpha to final is still sufficiently long that this will hinder the development cycle of new plugins. So, if we can agree on a way that allows new plugins to become available in binary form to beta or even final versions of MariaDB, I think that would be a great win and really invigorate the plugin ecosystem. Hakan, what do you reckon? Regards, Arjen. -- Arjen Lentz, Exec.Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget. Follow our blog at http://openquery.com/blog/ OurDelta: packages for MySQL and MariaDB @ http://ourdelta.org

Henrik Ingo

5 Nov 5 Nov

2:44 p.m.

On Wed, Nov 4, 2009 at 3:37 PM, Arjen Lentz wrote:

...

I also appreciate the comments made by for instance Hakan, who feels that once a tree is in its beta phase, no new code (not even non-loaded plugins) should be added. While those plugins would not be installed by default, they'd still be quite present. So this is a perfectly valid position to take.

Since we're building packages, it's possible to make say a mariadb-xxxx package that contains just the plugin xxxx. In that scenario, it does not actually matter whether the plugin comes from the mariadb tree or not, it would be compiled against it anyway. To make this decently workable, we need to sort out the "need an entire mariadb source tree to build the plugin" problem.

...

Now to return to the original trail, perhaps Hakan would be ok with plugins being in separate packages, then there is no chance someone could have code installed that is not necessarily production ready, if they haven't explicitly chosen to install it. The problem I'm trying to resolve here is still essentially the PBXT issue. It's really important that new plugins get tested so they can feedback and bugreports, and that only happens if they're available in easy binary form. However, having them available for the current production version (or anything >= beta) as well as alpha stage versions, is important. The cycle from alpha to final is still sufficiently long that this will hinder the development cycle of new plugins. So, if we can agree on a way that allows new plugins to become available in binary form to beta or even final versions of MariaDB, I think that would be a great win and really invigorate the plugin ecosystem.

Hakan, what do you reckon?

Just to throw another idea out there, we've had talks about re-introducing something like mariadb-max packaging. The idea would be that a basic mariadb install should not only be stable, but for most people should be reasonably small. Having a separate package that then includes more engines and other plugins, would allow users to choose to stick with just the standard package, or the -max package as they prefer. I think even InnoDB was first introduced in mysql-max binaries (?), so it is not a bad precedent to follow. henrik -- email: henrik.ingo@avoinelama.fi tel: +358-40-5697354 www: www.avoinelama.fi/~hingo book: www.openlife.cc

Arjen Lentz

10:53 p.m.

Hi Henrik On 05/11/2009, at 11:44 PM, Henrik Ingo wrote:

...

Just to throw another idea out there, we've had talks about re-introducing something like mariadb-max packaging. The idea would be that a basic mariadb install should not only be stable, but for most people should be reasonably small. Having a separate package that then includes more engines and other plugins, would allow users to choose to stick with just the standard package, or the -max package as they prefer.

Interesting idea! some notes below

...

I think even InnoDB was first introduced in mysql-max binaries (?), so it is not a bad precedent to follow.

Yes it was, however to the present day people still believe that MySQL 3.23 and 4.0 did not have InnoDB. And of course later the MaxDB (aka SAP DB) confusion was added to that. A giant fail in product clarity. Currently, the builds are all actually -max. So they would have to be modified to be "standard" so we can add new things to -max, or we create a new "bleeding edge" group. Antony was commenting to me the other day how it was a pity that his nice architecture for plugin groups had never been utilised beyond -max! In the OurDelta 5.0 builds we have a -sail build (aka sailing close to the wind) which admittedly is not particularly descriptive, but neither does it cause confusion. If people don't know they won't use it, and if they do know they use it when they want. The -sail build simply has a few more patches on top of the basic set. This includes some experimental InnoDB patches from Yasufumi, and now also OQGRAPH. It's a nice way to introduce new code without disrupting existing stability for those who prefer that. So yes, conceptually we could do something similar with MariaDB, but we need different naming. The -max thing is polluted. But, if we're only adding additional plugins, it does not need to be a different main package, but simply an "extra" package that people can install optionally. It has the same effect as my suggested mariadb- pluginname packaging, except it's a big bundle instead of all separate. Cheers, Arjen. -- Arjen Lentz, Exec.Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget. Follow our blog at http://openquery.com/blog/ OurDelta: packages for MySQL and MariaDB @ http://ourdelta.org

Peter Laursen

11:11 p.m.

Let me add that the 'mysql-essentials' download package is very popular with Windows users (it includes InnoDB, though!). I think MariaDB should be careful not to make the default download too big. Not that bandwidth and disk space is a big problem for most users nowadays. But it seems people want a small package for evaluation. MariaDB download is/will be significantly larger than a 'parallel' MySQL and that itself can be a 'stopper' for popularization. 98% of users do not care about tests and rarely used clients and scripts. They want *the basic database functionality* only (but don't mistake: I am a big PBXT fan and that should be included as default - as it already is with the XAMPP 'bundle' package and has been for around 6 months or so). . Peter Webyog . On Thu, Nov 5, 2009 at 22:53, Arjen Lentz wrote:

...

Hi Henrik

On 05/11/2009, at 11:44 PM, Henrik Ingo wrote:

...
Just to throw another idea out there, we've had talks about re-introducing something like mariadb-max packaging. The idea would be that a basic mariadb install should not only be stable, but for most people should be reasonably small. Having a separate package that then includes more engines and other plugins, would allow users to choose to stick with just the standard package, or the -max package as they prefer.

Interesting idea! some notes below

I think even InnoDB was first introduced in mysql-max binaries (?), so

...
it is not a bad precedent to follow.

Yes it was, however to the present day people still believe that MySQL 3.23 and 4.0 did not have InnoDB. And of course later the MaxDB (aka SAP DB) confusion was added to that. A giant fail in product clarity.

Currently, the builds are all actually -max. So they would have to be modified to be "standard" so we can add new things to -max, or we create a new "bleeding edge" group. Antony was commenting to me the other day how it was a pity that his nice architecture for plugin groups had never been utilised beyond -max!

In the OurDelta 5.0 builds we have a -sail build (aka sailing close to the wind) which admittedly is not particularly descriptive, but neither does it cause confusion. If people don't know they won't use it, and if they do know they use it when they want. The -sail build simply has a few more patches on top of the basic set. This includes some experimental InnoDB patches from Yasufumi, and now also OQGRAPH. It's a nice way to introduce new code without disrupting existing stability for those who prefer that.

So yes, conceptually we could do something similar with MariaDB, but we need different naming. The -max thing is polluted. But, if we're only adding additional plugins, it does not need to be a different main package, but simply an "extra" package that people can install optionally. It has the same effect as my suggested mariadb-pluginname packaging, except it's a big bundle instead of all separate.

Cheers, Arjen. -- Arjen Lentz, Exec.Director @ Open Query (http://openquery.com) Exceptional Services for MySQL at a fixed budget.

Follow our blog at http://openquery.com/blog/ OurDelta: packages for MySQL and MariaDB @ http://ourdelta.org

_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp

Kristian Nielsen

9 Nov 9 Nov

11:31 a.m.

Arjen Lentz writes:

...

We filed a wishlist bug for this, https://bugs.launchpad.net/maria/+bug/470580 which also includes a patch by Antony Curtis. It creates a mysql- glob.h during the build process, which we could put into a mariadb- plugin-dev package.

Then, some mariadb-xxxx packages can be build in the main mariadb build, and some can be separate. It also allows developers to work on plugins sanely. It cleans up the entire dev environment for this - it's not 100% pretty as it's a nasty big include, but it's a good step given the current mess.

I took a look at this (sorry for slight delay, have been swamped with lots of small tasks the last couple of days). I very much agree with the idea of having a plugin-dev package that is sufficient for building plugins. It seems it should be doable with not too much work. I know we already did some fixes in the MariaDB tree to make this possible. Basically some configure/CFLAGS options could affect the binary interface (think it was DEBUG / SAFEMALLOC flags). These were moved into config.h so that it is not necessary to guess the right values of these options to build a plugin that will load without crashing. We can do more of these kinds of fixes as needed. I do not understand the approach taken in the patch though. Seems to me to be the wrong approach? Though I could be misunderstanding, the patch doesn't contain much in the way of explanation / comments yet. So I was expecting to see somthing that collected a set of include files needed to build (storage engine) plugins, and putting them somewhere plugins could reference them. Instead, the patch seems to run the C pre-processor on mysql_priv.h, I assume to get a _single_ .h file with everything? And the result is passed through some Perl filter and munged in various interesting ways. So I guess my first question is why to do it this roundabout way at all? Why not just collect the necessary .h files? Seems much cleaner, simpler, easier to maintain, etc. If there are some issues with this, maybe we could instead fix these issues in the source? Thanks, - Kristian.

...

=== modified file 'sql/Makefile.am' --- old/sql/Makefile.am 2009-09-15 10:46:35 +0000 +++ new/sql/Makefile.am 2009-11-02 09:19:32 +0000 @@ -20,6 +20,7 @@ MYSQLBASEdir= $(prefix) MYSQLLIBdir= $(pkglibdir) pkgplugindir = $(pkglibdir)/plugin +globincldir = $(pkgincludedir)/mysql/$(VERSION) INCLUDES = @ZLIB_INCLUDES@ \ -I$(top_builddir)/include -I$(top_srcdir)/include \ -I$(top_srcdir)/regex -I$(srcdir) $(openssl_includes) \ @@ -29,6 +30,7 @@ libexec_PROGRAMS = mysqld EXTRA_PROGRAMS = gen_lex_hash bin_PROGRAMS = mysql_tzinfo_to_sql +globincl_HEADERS = mysql-glob.h

noinst_LTLIBRARIES= libndb.la \ udf_example.la @@ -179,6 +181,12 @@ ./gen_lex_hash$(EXEEXT) > $@-t $(MV) $@-t $@

+mysql-glob.h: mysql_priv.h + $(CXXCPP) -x c++-header $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \ + $(AM_CPPFLAGS) $(CPPFLAGS) $(AM_CXXFLAGS) $(CXXFLAGS) \ + -CC -dDI $^ | $(PERL) $(srcdir)/make-mysql-glob.pl > $@-t + $(MV) $@-t $@ + # For testing of udf_example.so udf_example_la_SOURCES= udf_example.c udf_example_la_LDFLAGS= -module -rpath $(pkglibdir)

=== added file 'sql/make-mysql-glob.pl' --- old/sql/make-mysql-glob.pl 1970-01-01 00:00:00 +0000 +++ new/sql/make-mysql-glob.pl 2009-11-02 09:39:43 +0000 @@ -0,0 +1,179 @@ +#!/usr/bin/perl + +use strict; + +my $current_file; +my $current_line; +my $current_path; + +my @command_line= (); + +my $until_file; +my $until_line; + +my $last_include; + +my $echo= \&echo_on; +my $echo_line; + +my $first_define; +my $define_suffix = sub { }; + +sub canon($) +{ + my ($path)= @_; + my $tmp= $path; + $tmp =~ s@/[^/]+/\.\./@/@; + return canon($tmp) if $tmp ne $path; + return $path; +} + +$first_define = sub { + my ($line) = @_; + if ($line =~ m/^#define\s+([^\s]+)/) + { + my ($e,$d)= ($echo, $1); + $first_define = sub { return 0; }; + &$e("#ifndef MYSQL_GLOB_H\n#define MYSQL_GLOB_H\n#ifdef $d\n#error cannot mix headers\n#endif\n$line\n"); + foreach (@command_line) + { + &$e($_); + } + + $define_suffix= sub + { + &$e("#endif /* MYSQL_GLOB_H / $d */\n"); + }; + + return 1; + } + return 0; +}; + +sub echo_on($) +{ + my ($line) = @_; + + return if &$first_define($line); + return if $line =~ m/^#define\s+SHOW_FUNC\s+/; + + print $echo_line if defined $echo_line; + print $line; + undef $echo_line; +} + +sub echo_off($) +{ + my ($line)= @_; + if (($current_file eq $until_file) && ($current_line == $until_line)) + { + $echo= \&echo_on; + undef $until_file; + undef $until_line; + undef $echo_line; + echo_on($line) if !($line =~ m/^# /); + } +} + +sub echo_cmdline($) +{ + my ($line)= @_; + push @command_line, $line if + ($current_file ne $until_file) || ($current_line != $until_line); + echo_off($line); +} + +sub test_builtin($$) +{ + ($until_line, $until_file)= @_ if !defined($until_file); + $echo= \&echo_off; + return 0; +} + +sub test_cmdline($$) +{ + ($until_line, $until_file)= @_ if !defined($until_file); + $echo= \&echo_cmdline; + return 0; +} + +sub test_incpath($$) +{ + if ($current_line == 1) + { + ($until_line, $until_file)= @_ if !defined($until_file); + if ($current_line == 1) + { + undef $echo_line; + &$echo($last_include); + $echo= \&echo_off; + } + } + return 0; +} + +while (<>) +{ + my ($last_line, $last_file)= ($current_line, $current_file); + my $test= sub { return 1; }; + + if (m/^# (\d+) "(.*)"/) + { + ($current_line, $current_file)= (int $1,$2); + + if ($current_file =~ m/\/$/) + { + $current_path= $`; + ($current_line, $current_file)= ($last_line,$last_file); + next; + } + + $echo_line= $_; + + $test= sub + { + my $saved_echo= $echo; + $echo= sub + { + my ($line)= @_; + $echo= $saved_echo; + echo_off($line); + }; + return 1; + }; + + $test= sub { return test_builtin($last_line, $last_file); } if + $current_file eq '<built-in>'; + $test= sub { return test_cmdline($last_line, $last_file); } if + $current_file eq '<command line>'; + $test= sub { return test_incpath($last_line, $last_file); } if + $current_file =~ m/^\//; + + if (!($current_file =~ m/^\//)) + { + my $new_path= canon($current_path.$current_file); + $echo_line =~ s/"$current_file"/"$new_path"/; + } + $echo_line=~ s/^# /#line /; + + } + + if (m/^#include /) + { + $test= sub { + my $saved_echo= $echo; + $echo= sub { + ($last_include)= @_; + $echo= $saved_echo; + }; + return 1; + }; + } + + &$echo($_) if &$test(); + $current_line++; +} + +&$define_suffix(); + +exit 0;

5417

Age (days ago)

5427

Last active (days ago)

List overview

Download

11 comments

7 participants

participants (7)

Alex Yurchenko
Arjen Lentz
Hakan Kuecuekyilmaz
Henrik Ingo
Kristian Nielsen
Michael Widenius
Peter Laursen

[Maria-developers] request for comment: OQGRAPH in 5.1 packages

tags

participants (7)