- developers - lists.mariadb.org

[Maria-developers] Progress (by Knielsen): Buildbot upgrade test from MariaDB 5.1.42->newest (108)
by worklog-noreply＠askmonty.org 14 May '10

14 May '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Buildbot upgrade test from MariaDB 5.1.42->newest CREATION DATE..: Fri, 19 Mar 2010, 07:48 SUPERVISOR.....: Knielsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Other TASK ID........: 108 (http://askmonty.org/worklog/?tid=108) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 25 (hours remain) ORIG. ESTIMATE.: 25 PROGRESS NOTES: -=-=(Knielsen - Fri, 14 May 2010, 06:42)=-=- This task was done in MWL#118 (I forgot about this one, so added a new, duplicate task). Reported zero hours worked. Estimate unchanged. -=-=(Knielsen - Fri, 19 Mar 2010, 07:52)=-=- Low Level Design modified. --- /tmp/wklog.108.old.3290 2010-03-19 07:52:59.000000000 +0000 +++ /tmp/wklog.108.new.3290 2010-03-19 07:52:59.000000000 +0000 @@ -1 +1,11 @@ +Tasks needed to do this: + + - For all the .deb kvm builders, set up a new virtual machine image. Based on + -serial or -install, pre-install the appropriate MariaDB 5.1.42 package + with some data. Similar to how the -upgrade images are set up with MySQL + pre-installed. + + - Add a test step in the Buildbot configuration that runs another upgrade + test, similar to the existing upgrade test, but with the images with + MariaDB preinstalled rather than MySQL. DESCRIPTION: Buildbot currently does automatic upgrade test for .debs from the official MySQL 5.0/5.1 package to the newest MariaDB package. In addition to this, we need an upgrade test from older MariaDB release. Probably 5.1.42, the first stable release, is the one to use. This is particularly important for the 5.2/5.3 trees, to test that upgrade from 5.1->5.2/5.3 works. LOW-LEVEL DESIGN: Tasks needed to do this: - For all the .deb kvm builders, set up a new virtual machine image. Based on -serial or -install, pre-install the appropriate MariaDB 5.1.42 package with some data. Similar to how the -upgrade images are set up with MySQL pre-installed. - Add a test step in the Buildbot configuration that runs another upgrade test, similar to the existing upgrade test, but with the images with MariaDB preinstalled rather than MySQL. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Buildbot MariaDB->MariaDB upgrade testing (118)
by worklog-noreply＠askmonty.org 14 May '10

14 May '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Buildbot MariaDB->MariaDB upgrade testing CREATION DATE..: Wed, 12 May 2010, 13:48 SUPERVISOR.....: Knielsen IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Other TASK ID........: 118 (http://askmonty.org/worklog/?tid=118) VERSION........: STATUS.........: Complete PRIORITY.......: 60 WORKED HOURS...: 5 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 8 PROGRESS NOTES: -=-=(Knielsen - Fri, 14 May 2010, 06:41)=-=- Version updated. --- /tmp/wklog.118.old.18369 2010-05-14 06:41:01.000000000 +0000 +++ /tmp/wklog.118.new.18369 2010-05-14 06:41:01.000000000 +0000 @@ -1 +1 @@ -Server-9.x + -=-=(Knielsen - Fri, 14 May 2010, 06:41)=-=- Status updated. --- /tmp/wklog.118.old.18369 2010-05-14 06:41:01.000000000 +0000 +++ /tmp/wklog.118.new.18369 2010-05-14 06:41:01.000000000 +0000 @@ -1 +1 @@ -Un-Assigned +Complete -=-=(Knielsen - Fri, 14 May 2010, 06:40)=-=- Installed the new virtual images, using 5.1.42 from OurDelta (for all except lucid, which is new and so has no 5.1.42 ourdelta package and also needed a fixed set of packages due to changes in the lucid MySQL packaging). Added a new upgrade2 step in the Buildbot configuration to test this. Updated Buildbot wiki documentation. Worked 5 hours and estimate 0 hours remain (original estimate decreased by 3 hours). DESCRIPTION: Create an additional test step for Buildbot to check that upgrading from one version of MariaDB to another works ok. We already have testing of upgrade from distro-official MySQL .debs to our MariaDB .debs. What we need is for each Debian/Ubuntu, a new KVM virtual image with MariaDB 5.1.42 (First GA release) pre-installed, just like the existing update test uses images with MySQL pre-installed. Then the Buildbot configuration must be updated to add another upgrade test step, just like the existing one but using the images with MariaDB pre-installed. Also, the setup of the new images must be added to the existing documentation: http://askmonty.org/wiki/BuildBot::package http://askmonty.org/wiki/BuildBot:vm-setup ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Progress (by Knielsen): Buildbot MariaDB->MariaDB upgrade testing (118)
by worklog-noreply＠askmonty.org 14 May '10

14 May '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Buildbot MariaDB->MariaDB upgrade testing CREATION DATE..: Wed, 12 May 2010, 13:48 SUPERVISOR.....: Knielsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Other TASK ID........: 118 (http://askmonty.org/worklog/?tid=118) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 5 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 8 PROGRESS NOTES: -=-=(Knielsen - Fri, 14 May 2010, 06:40)=-=- Installed the new virtual images, using 5.1.42 from OurDelta (for all except lucid, which is new and so has no 5.1.42 ourdelta package and also needed a fixed set of packages due to changes in the lucid MySQL packaging). Added a new upgrade2 step in the Buildbot configuration to test this. Updated Buildbot wiki documentation. Worked 5 hours and estimate 0 hours remain (original estimate decreased by 3 hours). DESCRIPTION: Create an additional test step for Buildbot to check that upgrading from one version of MariaDB to another works ok. We already have testing of upgrade from distro-official MySQL .debs to our MariaDB .debs. What we need is for each Debian/Ubuntu, a new KVM virtual image with MariaDB 5.1.42 (First GA release) pre-installed, just like the existing update test uses images with MySQL pre-installed. Then the Buildbot configuration must be updated to add another upgrade test step, just like the existing one but using the images with MariaDB pre-installed. Also, the setup of the new images must be added to the existing documentation: http://askmonty.org/wiki/BuildBot::package http://askmonty.org/wiki/BuildBot:vm-setup ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Summary of discussion about MariaDB future and release plans
by Kristian Nielsen 13 May '10

13 May '10

Hi, With all the european MariaDB people stuck in California, there is an unexpected opportunity for extra face-to-face meetups. So today I had the opportunity to enjoy lunch at a nice sushi place in Santa Clara with Henrik, Timour, Peter, Vadim, and Sanja. Here are a couple of key points from the discussion: 1. We need to consider carefully how we handle keeping the MariaDB product stable for production use. Peter had the point that the critical thing is avoiding behaviour change. This means that if some version of MariaDB works sufficiently different that the user need to work on their application to upgrade, it is a blocker to adoption. Different in this context means different from any previous version of MariaDB, but it _also_ means any difference from MySQL! I think I can see that this is actually more important than mere bugs/regressions. If a bug slips through, we can fix it, and if we can fix it quick enough the user may even be happy with us. But if the behaviour is qualitatively different (and such difference can not easily be turned off), that is a real problem for the user. We discussed a lot around this for the subquery optimisations. This is a "scary feature", as it has the very real potential to change execution plans, occasionally for the worse. As Timour pointed out, there is for each new optimiser feature a server variable to enable/disable it. So we need to very clearly state in the release notes, under "upgrading to MariaDB x.y", that there are optimiser changes, and give the exact list of variable settings that will make the optimiser run in "MySQL 5.1 mode". For other features this is simpler. For storage engines, just disable the engine and it will have no impact. For possibly replication plugins, again don't use the plugin and there should be no difference in behaviour. But again, we need to very carefully think about this and communicate/document this. For other features again we might need to re-think this point, making sure we do not change behaviour in a non reversible way. I think this is a good observation from those with a deep knowledge of the operational side of Databases. 2. On a note related to stability, I think we need to carefully avoid the mistakes from the MySQL release model. Basically, we need to have regular releases (6-12 month cycles). This is all-important! Much more important than any single feature, however big. We must _never_ push a feature into a tree if it is not ready. Better make an empty release! So this means it is actually wrong to say that subqueries will be in MariaDB 5.3. That should not be the plan. The subqueries will be in the first release made after they are ready. Which maybe 5.3, may be something else. I think we very nearly made the same mistake with 5.2. We had a feature list, and while some of them were "rolling", we also had features that were decided in advance that they _must_ be pushed at a certain date. I strongly believe this is wrong. We should have made _all_ features rolling. Unless we are extremely careful with this, sooner or later there will be sufficient pressure that we will push unfinished stuff into a release, causing _all_ of that release to be a failure. By waiting with the unfinished feature, only that feature will "fail", and by being careful with releases, the next release will be close anyway. (Yes, "finished" cannot mean bug free. But it can mean "as good as we can reasonably make it".) 3. The final point I took from the discussion was related to version numbering. We in the MariaDB team of course know that we rock, and that we will change the world tomorrow, or at least as soon as planes will take us back to Europe :-). But the reality is that for now, we are an appendix to MySQL with MariaDB 5.1 and 5.2. It thus can be argued that this should be reflected in the version numbering, to avoid confusion. Peter made the point that he would like to see MariaDB 5.1 and 5.2 versioning be made so that it is clear that there is a base MySQL version plus some well-defined set of changes. (This is how XtraDB release numbering works). So we currently have MariaDB 5.1.44, which is equivalent to MySQL 5.1.44 plus some set of safe patches. But MariaDB 5.2.0 is actually the same, just with some additional, also safe patches. So rather than go to MariaDB 5.2, we could imagine something like this: MariaDB 5.1.44-1 Current MariaDB 5.1.44 MariaDB 5.1.44-2 Current MariaDB 5.2.0 MariaDB 5.1.44-1.1 In case we need to do a bugfix for 5.1.44-1 MariaDB 5.1.46-1 Next time we merge MySQL 5.1.46 into MariaDB -1 MariaDB 5.1.46-2 Next time we merge MySQL 5.1.46 into MariaDB -2 I need to think more before I fully make up my mind about these points one way or the other, but I think these are in any case interesting points to consider. - Kristian.

7 17

[Maria-developers] bzr commit into Mariadb 5.2, with Maria 2.0:maria/5.2 branch (igor:2790)
by Igor Babaev 13 May '10

13 May '10

#At lp:maria/5.2 based on revid:igor@askmonty.org-20100512040958-yc654eq2zdfo8l8m 2790 Igor Babaev 2010-05-12 Cleanup modified: sql/sql_derived.cc === modified file 'sql/sql_derived.cc' --- a/sql/sql_derived.cc 2010-05-12 04:09:58 +0000 +++ b/sql/sql_derived.cc 2010-05-13 06:59:14 +0000 @@ -159,12 +159,7 @@ mysql_handle_single_derived(LEX *lex, TA uint phase_flag= DT_INIT << phase; if (phase_flag > phases) break; -#if 0 - if (!(phases & phase_flag) || - derived->merged_for_insert && phase_flag != DT_REINIT) -#else if (!(phases & phase_flag)) -#endif continue; /* Skip derived tables to which the phase isn't applicable. */ if (phase_flag != DT_PREPARE && @@ -605,19 +600,11 @@ bool mysql_derived_prepare(THD *thd, LEX bool res= FALSE; // Skip already prepared views/DT -#if 0 - if (!unit || unit->prepared || derived->merged_for_insert) -#else if (!unit || unit->prepared) -#endif DBUG_RETURN(FALSE); /* It's a target view for an INSERT, create field translation only. */ -#if 0 - if (derived->skip_prepare_derived && !derived->is_multitable()) -#else if (derived->merged_for_insert) -#endif { res= derived->create_field_translation(thd); DBUG_RETURN(res);

1 0

[Maria-developers] Updated (by Knielsen): Use Buildbot to populate apt/yum repositories (117)
by worklog-noreply＠askmonty.org 12 May '10

12 May '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Use Buildbot to populate apt/yum repositories CREATION DATE..: Wed, 12 May 2010, 07:04 SUPERVISOR.....: Knielsen IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Other TASK ID........: 117 (http://askmonty.org/worklog/?tid=117) VERSION........: Server-9.x STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 4 ESTIMATE.......: 16 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: -=-=(Knielsen - Wed, 12 May 2010, 21:20)=-=- High-Level Specification modified. --- /tmp/wklog.117.old.21448 2010-05-12 21:20:34.000000000 +0000 +++ /tmp/wklog.117.new.21448 2010-05-12 21:20:34.000000000 +0000 @@ -1,5 +1,5 @@ -As for signing, I think it may be possible/best to do the signing outside of -buildbot, as a separate process. There are some advantages to this: +The signing of packages can be done outside of Buildbot, as a separate +process. There are some advantages to this: - Security: the private key can be kept less exposed when it is not included in the buildbot infrastructure. @@ -9,9 +9,6 @@ - Generally reducing the complexity of the buildbot setup. -This of course requires that it is possible to sign the packages after the -actual build. - ---- Here is how to sign the .rpms. @@ -42,20 +39,37 @@ ---- -For .deb, I *think* we are using secure apt, which does not actually sign the -packages, rather it signs the "Release" file which is created when the -repository is set up. So in this case again there is no problem doing the -signing outside of the build itself (in fact that is the way it must be). +For .deb, it is not the individual .deb that is signed, it is the +repository. Here is one way to generate a signed repository, using reprepro. -Found two tools that can help with building and signing apt repositories: -reprepro (seems to be the newest, recommended) and apt-ftparchive. +The ourdelta/bakery signing stuff needs to be copied to ~/.gnupg ----- +mkdir repo # or whatever +cd repo +mkdir conf +cat >conf/distributions <<END +Origin: MariaDB +Label: MariaDB +Codename: hardy +Architectures: amd64 +Components: mariadb-ourdelta +Description: MariaDB test Repository +SignWith: autosign(a)ourdelta.org +END +for i in `find /home/buildbot/debs/ -name '*.deb'` ; do reprepro --basedir=. +includedeb hardy $i ; done + +The corrosponding line for /etc/apt/sources.list: -ToDO: Figure out how to handle the mysql-client-core issue on lucid. Arjen -suggested splitting up so we have this package ourselves, or maybe it can be -handled with replace/provide/conflict dependencies. + deb file:///home/buildbot/repo hardy mariadb-ourdelta + +This works for multiple distributions, by adding more sections to the +conf/distributions file. + +---- -ToDo: Figure out exactly what files/directory structure needs to be uploaded -(asked Peter, awaiting reply). +For the mysql-client-core-5.1 issue, the solution is to split the +mariadb-client-5.1 (and 5.2) package similarly into +mariadb-client-core-5.1. The mariadb-client-core-5.1 package then provides: +mysql-client-core-5.1. -=-=(Knielsen - Wed, 12 May 2010, 18:25)=-=- High-Level Specification modified. --- /tmp/wklog.117.old.12634 2010-05-12 18:25:58.000000000 +0000 +++ /tmp/wklog.117.new.12634 2010-05-12 18:25:58.000000000 +0000 @@ -12,9 +12,35 @@ This of course requires that it is possible to sign the packages after the actual build. -For .rpm this seems to be easy (from reading, didn't try yet): +---- + +Here is how to sign the .rpms. + +Copy in the ourdelta/bakery signing stuff to ~/.gnupg and ~/.rpmmacros. + +Run + + rpm --addsign *.rpm + +That's all! This can be tested by creating a local yum repository: - rpm --addsign <packages> + createrepo <dir> + +(where <dir> contains the signed .rpms). Then create the file +/etc/yum.repos.d/localmaria.repo: + +[localmaria] +name=Local MariaDB repo +baseurl=file:///home/buildbot/rpms +gpgcheck=1 +enabled=1 +gpgkey=http://master.ourdelta.org/deb/ourdelta.gpg + +Now this should work to install MariaDB: + + sudo yum install MariaDB-server + +---- For .deb, I *think* we are using secure apt, which does not actually sign the packages, rather it signs the "Release" file which is created when the -=-=(Knielsen - Wed, 12 May 2010, 07:14)=-=- High-Level Specification modified. --- /tmp/wklog.117.old.401 2010-05-12 07:14:27.000000000 +0000 +++ /tmp/wklog.117.new.401 2010-05-12 07:14:27.000000000 +0000 @@ -1 +1,35 @@ +As for signing, I think it may be possible/best to do the signing outside of +buildbot, as a separate process. There are some advantages to this: + + - Security: the private key can be kept less exposed when it is not included + in the buildbot infrastructure. + + - It is good to have one step of human intervention before actually signing + and releasing packages. + + - Generally reducing the complexity of the buildbot setup. + +This of course requires that it is possible to sign the packages after the +actual build. + +For .rpm this seems to be easy (from reading, didn't try yet): + + rpm --addsign <packages> + +For .deb, I *think* we are using secure apt, which does not actually sign the +packages, rather it signs the "Release" file which is created when the +repository is set up. So in this case again there is no problem doing the +signing outside of the build itself (in fact that is the way it must be). + +Found two tools that can help with building and signing apt repositories: +reprepro (seems to be the newest, recommended) and apt-ftparchive. + +---- + +ToDO: Figure out how to handle the mysql-client-core issue on lucid. Arjen +suggested splitting up so we have this package ourselves, or maybe it can be +handled with replace/provide/conflict dependencies. + +ToDo: Figure out exactly what files/directory structure needs to be uploaded +(asked Peter, awaiting reply). -=-=(Knielsen - Wed, 12 May 2010, 07:06)=-=- Upgraded lucid VMs to the official release. Discussed with Arjen how to handle things. Did a lot of reading on how apt repositories work. Worked 4 hours and estimate 16 hours remain (original estimate unchanged). DESCRIPTION: Since the package building for MariaDB is now fully automated in Buildbot, it has been decided to use packages from Buildbot for the OurDelta apt and yum repositories. This worklog is about fixing/implementing anything that is missing to achieve this. - When doing a real release build, packages/repositories need to be signed, so that users will not get a warning about unauthenticated packages. This signing must only be done on official releases, not on daily builds (to avoid confusing one with the other). - Packages must be uploaded from the Buildbot host. The OurDelta infrastructure has a DropBox share that could be used for this, another option is to simply use rsync. - Ubuntu 10.04 "lucid" has been released, and we need to support that for packages, so the Buildbot VM for lucid must be upgraded to have the official release. - In Ubuntu 10.04, the official MySQL packages include a new package mysql-client-core, we currently have a conflict with this on install that we need to handle somehow. HIGH-LEVEL SPECIFICATION: The signing of packages can be done outside of Buildbot, as a separate process. There are some advantages to this: - Security: the private key can be kept less exposed when it is not included in the buildbot infrastructure. - It is good to have one step of human intervention before actually signing and releasing packages. - Generally reducing the complexity of the buildbot setup. ---- Here is how to sign the .rpms. Copy in the ourdelta/bakery signing stuff to ~/.gnupg and ~/.rpmmacros. Run rpm --addsign *.rpm That's all! This can be tested by creating a local yum repository: createrepo <dir> (where <dir> contains the signed .rpms). Then create the file /etc/yum.repos.d/localmaria.repo: [localmaria] name=Local MariaDB repo baseurl=file:///home/buildbot/rpms gpgcheck=1 enabled=1 gpgkey=http://master.ourdelta.org/deb/ourdelta.gpg Now this should work to install MariaDB: sudo yum install MariaDB-server ---- For .deb, it is not the individual .deb that is signed, it is the repository. Here is one way to generate a signed repository, using reprepro. The ourdelta/bakery signing stuff needs to be copied to ~/.gnupg mkdir repo # or whatever cd repo mkdir conf cat >conf/distributions <<END Origin: MariaDB Label: MariaDB Codename: hardy Architectures: amd64 Components: mariadb-ourdelta Description: MariaDB test Repository SignWith: autosign(a)ourdelta.org END for i in `find /home/buildbot/debs/ -name '*.deb'` ; do reprepro --basedir=. includedeb hardy $i ; done The corrosponding line for /etc/apt/sources.list: deb file:///home/buildbot/repo hardy mariadb-ourdelta This works for multiple distributions, by adding more sections to the conf/distributions file. ---- For the mysql-client-core-5.1 issue, the solution is to split the mariadb-client-5.1 (and 5.2) package similarly into mariadb-client-core-5.1. The mariadb-client-core-5.1 package then provides: mysql-client-core-5.1. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Use Buildbot to populate apt/yum repositories (117)
by worklog-noreply＠askmonty.org 12 May '10

12 May '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Use Buildbot to populate apt/yum repositories CREATION DATE..: Wed, 12 May 2010, 07:04 SUPERVISOR.....: Knielsen IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Other TASK ID........: 117 (http://askmonty.org/worklog/?tid=117) VERSION........: Server-9.x STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 4 ESTIMATE.......: 16 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: -=-=(Knielsen - Wed, 12 May 2010, 18:25)=-=- High-Level Specification modified. --- /tmp/wklog.117.old.12634 2010-05-12 18:25:58.000000000 +0000 +++ /tmp/wklog.117.new.12634 2010-05-12 18:25:58.000000000 +0000 @@ -12,9 +12,35 @@ This of course requires that it is possible to sign the packages after the actual build. -For .rpm this seems to be easy (from reading, didn't try yet): +---- + +Here is how to sign the .rpms. + +Copy in the ourdelta/bakery signing stuff to ~/.gnupg and ~/.rpmmacros. + +Run + + rpm --addsign *.rpm + +That's all! This can be tested by creating a local yum repository: - rpm --addsign <packages> + createrepo <dir> + +(where <dir> contains the signed .rpms). Then create the file +/etc/yum.repos.d/localmaria.repo: + +[localmaria] +name=Local MariaDB repo +baseurl=file:///home/buildbot/rpms +gpgcheck=1 +enabled=1 +gpgkey=http://master.ourdelta.org/deb/ourdelta.gpg + +Now this should work to install MariaDB: + + sudo yum install MariaDB-server + +---- For .deb, I *think* we are using secure apt, which does not actually sign the packages, rather it signs the "Release" file which is created when the -=-=(Knielsen - Wed, 12 May 2010, 07:14)=-=- High-Level Specification modified. --- /tmp/wklog.117.old.401 2010-05-12 07:14:27.000000000 +0000 +++ /tmp/wklog.117.new.401 2010-05-12 07:14:27.000000000 +0000 @@ -1 +1,35 @@ +As for signing, I think it may be possible/best to do the signing outside of +buildbot, as a separate process. There are some advantages to this: + + - Security: the private key can be kept less exposed when it is not included + in the buildbot infrastructure. + + - It is good to have one step of human intervention before actually signing + and releasing packages. + + - Generally reducing the complexity of the buildbot setup. + +This of course requires that it is possible to sign the packages after the +actual build. + +For .rpm this seems to be easy (from reading, didn't try yet): + + rpm --addsign <packages> + +For .deb, I *think* we are using secure apt, which does not actually sign the +packages, rather it signs the "Release" file which is created when the +repository is set up. So in this case again there is no problem doing the +signing outside of the build itself (in fact that is the way it must be). + +Found two tools that can help with building and signing apt repositories: +reprepro (seems to be the newest, recommended) and apt-ftparchive. + +---- + +ToDO: Figure out how to handle the mysql-client-core issue on lucid. Arjen +suggested splitting up so we have this package ourselves, or maybe it can be +handled with replace/provide/conflict dependencies. + +ToDo: Figure out exactly what files/directory structure needs to be uploaded +(asked Peter, awaiting reply). -=-=(Knielsen - Wed, 12 May 2010, 07:06)=-=- Upgraded lucid VMs to the official release. Discussed with Arjen how to handle things. Did a lot of reading on how apt repositories work. Worked 4 hours and estimate 16 hours remain (original estimate unchanged). DESCRIPTION: Since the package building for MariaDB is now fully automated in Buildbot, it has been decided to use packages from Buildbot for the OurDelta apt and yum repositories. This worklog is about fixing/implementing anything that is missing to achieve this. - When doing a real release build, packages/repositories need to be signed, so that users will not get a warning about unauthenticated packages. This signing must only be done on official releases, not on daily builds (to avoid confusing one with the other). - Packages must be uploaded from the Buildbot host. The OurDelta infrastructure has a DropBox share that could be used for this, another option is to simply use rsync. - Ubuntu 10.04 "lucid" has been released, and we need to support that for packages, so the Buildbot VM for lucid must be upgraded to have the official release. - In Ubuntu 10.04, the official MySQL packages include a new package mysql-client-core, we currently have a conflict with this on install that we need to handle somehow. HIGH-LEVEL SPECIFICATION: As for signing, I think it may be possible/best to do the signing outside of buildbot, as a separate process. There are some advantages to this: - Security: the private key can be kept less exposed when it is not included in the buildbot infrastructure. - It is good to have one step of human intervention before actually signing and releasing packages. - Generally reducing the complexity of the buildbot setup. This of course requires that it is possible to sign the packages after the actual build. ---- Here is how to sign the .rpms. Copy in the ourdelta/bakery signing stuff to ~/.gnupg and ~/.rpmmacros. Run rpm --addsign *.rpm That's all! This can be tested by creating a local yum repository: createrepo <dir> (where <dir> contains the signed .rpms). Then create the file /etc/yum.repos.d/localmaria.repo: [localmaria] name=Local MariaDB repo baseurl=file:///home/buildbot/rpms gpgcheck=1 enabled=1 gpgkey=http://master.ourdelta.org/deb/ourdelta.gpg Now this should work to install MariaDB: sudo yum install MariaDB-server ---- For .deb, I *think* we are using secure apt, which does not actually sign the packages, rather it signs the "Release" file which is created when the repository is set up. So in this case again there is no problem doing the signing outside of the build itself (in fact that is the way it must be). Found two tools that can help with building and signing apt repositories: reprepro (seems to be the newest, recommended) and apt-ftparchive. ---- ToDO: Figure out how to handle the mysql-client-core issue on lucid. Arjen suggested splitting up so we have this package ourselves, or maybe it can be handled with replace/provide/conflict dependencies. ToDo: Figure out exactly what files/directory structure needs to be uploaded (asked Peter, awaiting reply). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Knielsen): Buildbot MariaDB->MariaDB upgrade testing (118)
by worklog-noreply＠askmonty.org 12 May '10

12 May '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Buildbot MariaDB->MariaDB upgrade testing CREATION DATE..: Wed, 12 May 2010, 13:48 SUPERVISOR.....: Knielsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Other TASK ID........: 118 (http://askmonty.org/worklog/?tid=118) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 8 (hours remain) ORIG. ESTIMATE.: 8 PROGRESS NOTES: DESCRIPTION: Create an additional test step for Buildbot to check that upgrading from one version of MariaDB to another works ok. We already have testing of upgrade from distro-official MySQL .debs to our MariaDB .debs. What we need is for each Debian/Ubuntu, a new KVM virtual image with MariaDB 5.1.42 (First GA release) pre-installed, just like the existing update test uses images with MySQL pre-installed. Then the Buildbot configuration must be updated to add another upgrade test step, just like the existing one but using the images with MariaDB pre-installed. Also, the setup of the new images must be added to the existing documentation: http://askmonty.org/wiki/BuildBot::package http://askmonty.org/wiki/BuildBot:vm-setup ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2855: Build perl scripts in the correct directory
by noreply＠launchpad.net 12 May '10

12 May '10

------------------------------------------------------------ revno: 2855 committer: Bo Thorsen <bo(a)askmonty.org> branch nick: trunk-work timestamp: Wed 2010-05-12 14:33:10 +0200 message: Build perl scripts in the correct directory modified: scripts/CMakeLists.txt -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription

1 0

[Maria-developers] Patch to generate perl scripts in the right place
by Bo Thorsen 12 May '10

12 May '10

As discussed with Kristian Nielsen on IRC. Bo.

1 0