----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Use Buildbot to populate apt/yum repositories CREATION DATE..: Wed, 12 May 2010, 07:04 SUPERVISOR.....: Knielsen IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Other TASK ID........: 117 (http://askmonty.org/worklog/?tid=117) VERSION........: Server-9.x STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 4 ESTIMATE.......: 16 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: -=-=(Knielsen - Wed, 12 May 2010, 21:20)=-=- High-Level Specification modified. --- /tmp/wklog.117.old.21448 2010-05-12 21:20:34.000000000 +0000 +++ /tmp/wklog.117.new.21448 2010-05-12 21:20:34.000000000 +0000 @@ -1,5 +1,5 @@ -As for signing, I think it may be possible/best to do the signing outside of -buildbot, as a separate process. There are some advantages to this: +The signing of packages can be done outside of Buildbot, as a separate +process. There are some advantages to this: - Security: the private key can be kept less exposed when it is not included in the buildbot infrastructure. @@ -9,9 +9,6 @@ - Generally reducing the complexity of the buildbot setup. -This of course requires that it is possible to sign the packages after the -actual build. - ---- Here is how to sign the .rpms. @@ -42,20 +39,37 @@ ---- -For .deb, I *think* we are using secure apt, which does not actually sign the -packages, rather it signs the "Release" file which is created when the -repository is set up. So in this case again there is no problem doing the -signing outside of the build itself (in fact that is the way it must be). +For .deb, it is not the individual .deb that is signed, it is the +repository. Here is one way to generate a signed repository, using reprepro. -Found two tools that can help with building and signing apt repositories: -reprepro (seems to be the newest, recommended) and apt-ftparchive. +The ourdelta/bakery signing stuff needs to be copied to ~/.gnupg ----- +mkdir repo # or whatever +cd repo +mkdir conf +cat >conf/distributions <<END +Origin: MariaDB +Label: MariaDB +Codename: hardy +Architectures: amd64 +Components: mariadb-ourdelta +Description: MariaDB test Repository +SignWith: autosign@ourdelta.org +END +for i in `find /home/buildbot/debs/ -name '*.deb'` ; do reprepro --basedir=. +includedeb hardy $i ; done + +The corrosponding line for /etc/apt/sources.list: -ToDO: Figure out how to handle the mysql-client-core issue on lucid. Arjen -suggested splitting up so we have this package ourselves, or maybe it can be -handled with replace/provide/conflict dependencies. + deb file:///home/buildbot/repo hardy mariadb-ourdelta + +This works for multiple distributions, by adding more sections to the +conf/distributions file. + +---- -ToDo: Figure out exactly what files/directory structure needs to be uploaded -(asked Peter, awaiting reply). +For the mysql-client-core-5.1 issue, the solution is to split the +mariadb-client-5.1 (and 5.2) package similarly into +mariadb-client-core-5.1. The mariadb-client-core-5.1 package then provides: +mysql-client-core-5.1. -=-=(Knielsen - Wed, 12 May 2010, 18:25)=-=- High-Level Specification modified. --- /tmp/wklog.117.old.12634 2010-05-12 18:25:58.000000000 +0000 +++ /tmp/wklog.117.new.12634 2010-05-12 18:25:58.000000000 +0000 @@ -12,9 +12,35 @@ This of course requires that it is possible to sign the packages after the actual build. -For .rpm this seems to be easy (from reading, didn't try yet): +---- + +Here is how to sign the .rpms. + +Copy in the ourdelta/bakery signing stuff to ~/.gnupg and ~/.rpmmacros. + +Run + + rpm --addsign *.rpm + +That's all! This can be tested by creating a local yum repository: - rpm --addsign <packages> + createrepo <dir> + +(where <dir> contains the signed .rpms). Then create the file +/etc/yum.repos.d/localmaria.repo: + +[localmaria] +name=Local MariaDB repo +baseurl=file:///home/buildbot/rpms +gpgcheck=1 +enabled=1 +gpgkey=http://master.ourdelta.org/deb/ourdelta.gpg + +Now this should work to install MariaDB: + + sudo yum install MariaDB-server + +---- For .deb, I *think* we are using secure apt, which does not actually sign the packages, rather it signs the "Release" file which is created when the -=-=(Knielsen - Wed, 12 May 2010, 07:14)=-=- High-Level Specification modified. --- /tmp/wklog.117.old.401 2010-05-12 07:14:27.000000000 +0000 +++ /tmp/wklog.117.new.401 2010-05-12 07:14:27.000000000 +0000 @@ -1 +1,35 @@ +As for signing, I think it may be possible/best to do the signing outside of +buildbot, as a separate process. There are some advantages to this: + + - Security: the private key can be kept less exposed when it is not included + in the buildbot infrastructure. + + - It is good to have one step of human intervention before actually signing + and releasing packages. + + - Generally reducing the complexity of the buildbot setup. + +This of course requires that it is possible to sign the packages after the +actual build. + +For .rpm this seems to be easy (from reading, didn't try yet): + + rpm --addsign <packages> + +For .deb, I *think* we are using secure apt, which does not actually sign the +packages, rather it signs the "Release" file which is created when the +repository is set up. So in this case again there is no problem doing the +signing outside of the build itself (in fact that is the way it must be). + +Found two tools that can help with building and signing apt repositories: +reprepro (seems to be the newest, recommended) and apt-ftparchive. + +---- + +ToDO: Figure out how to handle the mysql-client-core issue on lucid. Arjen +suggested splitting up so we have this package ourselves, or maybe it can be +handled with replace/provide/conflict dependencies. + +ToDo: Figure out exactly what files/directory structure needs to be uploaded +(asked Peter, awaiting reply). -=-=(Knielsen - Wed, 12 May 2010, 07:06)=-=- Upgraded lucid VMs to the official release. Discussed with Arjen how to handle things. Did a lot of reading on how apt repositories work. Worked 4 hours and estimate 16 hours remain (original estimate unchanged). DESCRIPTION: Since the package building for MariaDB is now fully automated in Buildbot, it has been decided to use packages from Buildbot for the OurDelta apt and yum repositories. This worklog is about fixing/implementing anything that is missing to achieve this. - When doing a real release build, packages/repositories need to be signed, so that users will not get a warning about unauthenticated packages. This signing must only be done on official releases, not on daily builds (to avoid confusing one with the other). - Packages must be uploaded from the Buildbot host. The OurDelta infrastructure has a DropBox share that could be used for this, another option is to simply use rsync. - Ubuntu 10.04 "lucid" has been released, and we need to support that for packages, so the Buildbot VM for lucid must be upgraded to have the official release. - In Ubuntu 10.04, the official MySQL packages include a new package mysql-client-core, we currently have a conflict with this on install that we need to handle somehow. HIGH-LEVEL SPECIFICATION: The signing of packages can be done outside of Buildbot, as a separate process. There are some advantages to this: - Security: the private key can be kept less exposed when it is not included in the buildbot infrastructure. - It is good to have one step of human intervention before actually signing and releasing packages. - Generally reducing the complexity of the buildbot setup. ---- Here is how to sign the .rpms. Copy in the ourdelta/bakery signing stuff to ~/.gnupg and ~/.rpmmacros. Run rpm --addsign *.rpm That's all! This can be tested by creating a local yum repository: createrepo <dir> (where <dir> contains the signed .rpms). Then create the file /etc/yum.repos.d/localmaria.repo: [localmaria] name=Local MariaDB repo baseurl=file:///home/buildbot/rpms gpgcheck=1 enabled=1 gpgkey=http://master.ourdelta.org/deb/ourdelta.gpg Now this should work to install MariaDB: sudo yum install MariaDB-server ---- For .deb, it is not the individual .deb that is signed, it is the repository. Here is one way to generate a signed repository, using reprepro. The ourdelta/bakery signing stuff needs to be copied to ~/.gnupg mkdir repo # or whatever cd repo mkdir conf cat >conf/distributions <<END Origin: MariaDB Label: MariaDB Codename: hardy Architectures: amd64 Components: mariadb-ourdelta Description: MariaDB test Repository SignWith: autosign@ourdelta.org END for i in `find /home/buildbot/debs/ -name '*.deb'` ; do reprepro --basedir=. includedeb hardy $i ; done The corrosponding line for /etc/apt/sources.list: deb file:///home/buildbot/repo hardy mariadb-ourdelta This works for multiple distributions, by adding more sections to the conf/distributions file. ---- For the mysql-client-core-5.1 issue, the solution is to split the mariadb-client-5.1 (and 5.2) package similarly into mariadb-client-core-5.1. The mariadb-client-core-5.1 package then provides: mysql-client-core-5.1. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)