[Maria-developers] Problem with KVM version in Buildbot?
Hi Daniel, We have a problem with a build in Buildbot: https://internal.askmonty.org/buildbot/builders/kvm-rpm-centos7_0-x86_64/bui... + qemu-img create -b /kvm/vms/vm-centos7_0-x86_64-build.qcow2 -f qcow2 /dev/shm/vm-tmp-2302.qcow2 qemu-img: '' uses a qcow2 feature which is not supported by this qemu version: QCOW version 3 qemu-img: Could not open '/kvm/vms/vm-centos7_0-x86_64-build.qcow2' Failed to clone base image, aborting This failure happens on the build host `terrier'. I assume the problem is that the image was created on a machine with a newer KVM? The host terrier seems to have a rather old one: knielsen@terrier:~> kvm --version QEMU emulator version 0.13.0 (qemu-kvm-devel), Copyright (c) 2003-2008 Fabrice Bellard How do you think we should solve the problem? Should we upgrade terrier? Or should we alternatively make sure to create new images in a backward-compatible version (and re-create affected images) ? Also, another issue related to VM images. Our old images were created with rather small disk size. I noticed that some new images are created much larger, for example: $ qemu-img info /kvm/vms/vm-sid-amd64-build.qcow2 image: /kvm/vms/vm-sid-amd64-build.qcow2 file format: qcow2 virtual size: 40G (42949672960 bytes) disk size: 3.1G cluster_size: 65536 Format specific information: compat: 0.10 That is fine, I think, it is indeed annoying when we run out of space in a build VM. However, the reason I noticed this is because host bb02 ran out of space in /run/shm/ due to a 16 GB copy of this image. So when we use larger VM images, we cannot use the optimisation to have the temporary copies in ramdisk, lest we get disk full errors. If the virtual size is larger to (ram disk size) / (max_builds), then we need to disable using ramdisk for buildbot for that image. I will try to fix the config to not use ramdisk for existing images that have too large virtual size. But it is something to keep in mind for the future. - Kristian.
2014-10-20 12:27 GMT+03:00 Kristian Nielsen <knielsen@knielsen-hq.org>:
$ qemu-img info /kvm/vms/vm-sid-amd64-build.qcow2 image: /kvm/vms/vm-sid-amd64-build.qcow2 file format: qcow2 virtual size: 40G (42949672960 bytes) disk size: 3.1G cluster_size: 65536 Format specific information: compat: 0.10
That is fine, I think, it is indeed annoying when we run out of space in a build VM. However, the reason I noticed this is because host bb02 ran out of space in /run/shm/ due to a 16 GB copy of this image.
There is no need for this image to be so big. The OS is maybe that 3.1 GB and the MariaDB build output is about 3 GB, so something like 9 GB disk size should be well enough by a fair margin.
Otto Kekäläinen writes:
There is no need for this image to be so big. The OS is maybe that 3.1 GB and the MariaDB build output is about 3 GB, so something like 9 GB disk size should be well enough by a fair margin.
Right. On the other hand, there is a lot of work involved in re-doing the VM images in case of too-small image, so I can understand Daniel's reasons for making the images "large enough". This is just the maximum size, the actual size will be no bigger than what is actually used. However, with a large max size, we still need to disable the ramdisk usage. Because in case a lot of space is used, this can cause failures in other, unrelated builds due to disk full. I don't know that the ramdisk optimisation is that important anyway, so it seems reasonable to just disable it here.
+ qemu-img create -b /kvm/vms/vm-centos7_0-x86_64-build.qcow2 -f qcow2 /dev/shm/vm-tmp-2302.qcow2 qemu-img: '' uses a qcow2 feature which is not supported by this qemu version: QCOW version 3 qemu-img: Could not open '/kvm/vms/vm-centos7_0-x86_64-build.qcow2' Failed to clone base image, aborting
For now, I updated the Buildbot config to not use the terrier buildslave for this builder. If any more images are made with the newer file version, the associated builders need to be similarly disabled, using this option: slaves=kvm11_slaves
Also, another issue related to VM images. Our old images were created with rather small disk size. I noticed that some new images are created much larger, for example:
I will try to fix the config to not use ramdisk for existing images that have too large virtual size. But it is something to keep in mind for the future.
So now I added an option ramdisk=True|False to getDebBuilder() and similar functions. And I checked all images, and added ramdisk=False for all those builders that I found used some of the images with large maximum size. Any further image made with large max size should then also use the ramdisk=False option. An alternative method to having big VM max size is to do like getFullTest() does. It adds an extra blank disk image that is not put in ramdisk: args = args + ["--kvm=-hdb", "--kvm=vm-blank-20G.qcow2"] and then mounts and uses it inside the VM. But again, if the ramdisk optimisation is not needed on the newer/better buildbot machines, there might be little need for this extra complexity. - Kristian.
On Mon, Oct 20, 2014 at 7:12 AM, Kristian Nielsen <knielsen@knielsen-hq.org> wrote:
Otto Kekäläinen writes:
There is no need for this image to be so big. The OS is maybe that 3.1 GB and the MariaDB build output is about 3 GB, so something like 9 GB disk size should be well enough by a fair margin.
Right. On the other hand, there is a lot of work involved in re-doing the VM images in case of too-small image, so I can understand Daniel's reasons for making the images "large enough". This is just the maximum size, the actual size will be no bigger than what is actually used.
However, with a large max size, we still need to disable the ramdisk usage. Because in case a lot of space is used, this can cause failures in other, unrelated builds due to disk full.
I don't know that the ramdisk optimisation is that important anyway, so it seems reasonable to just disable it here.
+ qemu-img create -b /kvm/vms/vm-centos7_0-x86_64-build.qcow2 -f qcow2 /dev/shm/vm-tmp-2302.qcow2 qemu-img: '' uses a qcow2 feature which is not supported by this qemu version: QCOW version 3 qemu-img: Could not open '/kvm/vms/vm-centos7_0-x86_64-build.qcow2' Failed to clone base image, aborting
For now, I updated the Buildbot config to not use the terrier buildslave for this builder. If any more images are made with the newer file version, the associated builders need to be similarly disabled, using this option:
slaves=kvm11_slaves
I think it was just the CentOS7 VMs that were created such that they aren't compatible with the older qcow2 format. When I made the opensuse VMs, for example, I made sure they used the old qcow2 format. That said, I think we really need to rebuild terrier, to get it up-to-date, install a larger hard disk, and generally get it more in step with bb01 and bb02, but that's a different discussion.
Also, another issue related to VM images. Our old images were created with rather small disk size. I noticed that some new images are created much larger, for example:
I will try to fix the config to not use ramdisk for existing images that have too large virtual size. But it is something to keep in mind for the future.
So now I added an option ramdisk=True|False to getDebBuilder() and similar functions. And I checked all images, and added ramdisk=False for all those builders that I found used some of the images with large maximum size.
Any further image made with large max size should then also use the ramdisk=False option.
An alternative method to having big VM max size is to do like getFullTest() does. It adds an extra blank disk image that is not put in ramdisk:
args = args + ["--kvm=-hdb", "--kvm=vm-blank-20G.qcow2"]
and then mounts and uses it inside the VM.
But again, if the ramdisk optimisation is not needed on the newer/better buildbot machines, there might be little need for this extra complexity.
Thanks! This solution works for me. -- Daniel Bartholomew, MariaDB Release Manager MariaDB | http://mariadb.com
participants (3)
-
Daniel Bartholomew -
Kristian Nielsen -
Otto Kekäläinen