[Maria-developers] Streamlining Buildbot builds
Serg brought this to my attention and asked me to look into it. Basically, over time we keep adding more and more builders to buildbot and we build an ever increasing number of packages. One way to keep turnaround times for builds reasonable is to add more hardware. Which we are doing by adding a third KVM build host. However, building every push on every KVM virtual machine is overkill. For example, if one Ubuntu build succeeds, most other Ubuntu builds will also succeed. We only really need to build "everything" when we are making a release. For other builds we can get away with building on a sub-set of builders. The main benefit of doing this is to speed up the turn-around time for builds. One potential issue is the possibility of not catching an error which, for some reason, only manifests on a certain OS version. There may be other issues. Let me know if you think of any. Assuming we want to move ahead with building most builds on a sub-set of our builders instead of the full set, the easiest thing to do is to configure buildbot to only build on every builder for pushes into the "-release" and top-level (5.3, 5.5, etc...) branches. All other branches would have an abridged set of builders assigned to them. First things first, we need to decide which builders will build every push in every tree (aka "build everything") and which builders will only build releases. Builders on dedicated machines don't need to be changed, only KVM builders. Here are the KVM builders I propose to be in the "build everything" set: "kvm-bintar-precise-amd64", "kvm-bintar-precise-x86", "kvm-deb-debian6-amd64", "kvm-deb-debian6-x86", "kvm-deb-precise-amd64", "kvm-deb-precise-x86", "kvm-rpm-fedora17-amd64", "kvm-rpm-fedora17-x86", "kvm-rpm-centos6-amd64", "kvm-rpm-centos6-x86", The reasoning for choosing the above is to have an x86 and amd64 builder for Ubuntu, CentOS, Fedora, and Debian. And also to always build binary tarballs. Thoughts? Thanks. -- Daniel Bartholomew Google+ - http://gplus.to/dbart Twitter - http://twitter.com/daniel_bart MariaDB: An Enhanced Drop-in Replacement for MySQL Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://gplus.to/mariadb Facebook - http://fb.com/MariaDB.dbms Knowledgebase - http://kb.askmonty.org Monty Program - http://montyprogram.com
Hi, Daniel! On Aug 06, Daniel Bartholomew wrote:
Here are the KVM builders I propose to be in the "build everything" set:
I suppose, you mean builders enabled for all trees...
"kvm-bintar-precise-amd64", "kvm-bintar-precise-x86", "kvm-deb-debian6-amd64", "kvm-deb-debian6-x86", "kvm-deb-precise-amd64", "kvm-deb-precise-x86", "kvm-rpm-fedora17-amd64", "kvm-rpm-fedora17-x86", "kvm-rpm-centos6-amd64", "kvm-rpm-centos6-x86",
The reasoning for choosing the above is to have an x86 and amd64 builder for Ubuntu, CentOS, Fedora, and Debian. And also to always build binary tarballs.
I think it is not as important to build for all linux distributions (well, it is but only for rpm/deb related changes), as it's important to build with different gcc versions. So I'd suggest * one of centos5 or rhel5 * hardy * debian5 * one of debian6, lucid, maverick * natty * oneiric or precise * centos6 * fedora17 probably it's not very important to build on both x86 and amd64 for every platrorm. By the way, when fixing packaging bugs it's important to build on all VMs, so we need one staging tree that builds all. But for now one can use -release trees for that, and later we could create -packaging stage trees when necessary. Regards, Sergei
Daniel Bartholomew <dbart@montyprogram.com> writes:
Serg brought this to my attention and asked me to look into it.
Basically, over time we keep adding more and more builders to buildbot and we build an ever increasing number of packages. One way to keep turnaround times for builds reasonable is to add more hardware. Which we are doing by adding a third KVM build host.
However, building every push on every KVM virtual machine is overkill. For example, if one Ubuntu build succeeds, most other Ubuntu builds will also succeed.
From our (Percona) experience, this isn't necessarily the case, different GCC versions etc do catch different things. Every time somebody suggests building on a subset, we find another issue like this (early).... soo... I wonder if you tempt fate the same way we do :) -- Stewart Smith
participants (3)
-
Daniel Bartholomew
-
Sergei Golubchik
-
Stewart Smith