+++ This bug is a downstream clone. The original bug is: +++ +++ bug 1470435 +++ ====================================================================== Created attachment 1297334 [details] vdsm log from failing hypervisor Version-Release number of selected component (if applicable): 4.1.3 Description of problem: In a new RHV 4.1.3 environment I'm unable to make a template because VDSM reports 'No space left on device". Storage domain is Fibre Channel-based, newly provisoned, and empty except for the OVF_STORE images and a single copy of the rhel-guest-image I uploaded via GUI. Allon mentioned "In the attached log we see qemu-image convert failing with a space error, not sanlock - which is a completely different issue" Additional info at http://post-office.corp.redhat.com/archives/rhev-tech/2017-July/msg00109.html How reproducible: Create a new storage domain using Fibre Channel Upload the rhel-guest-image Create VM Attempt to Make template from VM Actual results: Data is copied for a while (few minutes,) then the task fails and GUI reports "low level Image copy failed" Expected results: Template should be created, space-efficient (e.g. sparse/thin) Additional info: http://post-office.corp.redhat.com/archives/rhev-tech/2017-July/msg00109.html (Originally by John Call)
The rhel-guest-image was downloaded from [1] Search for "KVM Guest Image" I used the 7.3 version with sha256sum of 55a581618199240c5d6ce332653c3e506cb05da863ad2e4a50d5e068e9790d88 [1] - http://access.redhat.com/downloads/content/69/ver=/rhel---7/7.3/x86_64/product-software (Originally by John Call)
John, can you please add the rpm versions of your engine and VDSM? Thanks! (Originally by Allon Mureinik)
The image is compressed, and with block storage the Vdsm did not compute the target size big enough. In file storage, it is not an issue as the file system is sparse. In order to make it work in block storage, you will need to convert to uncompressed before uploading: qemu-img convert -f qcow2 rhel-guest-image-7.3-35.x86_64.qcow2 -O qcow2 -o compat=1.1 uncompressed.qcow2 In qemu version 2.0, in RHEL 7.5, we will be able to get the right size using 'qemu-img mesure' : $ ./qemu-img measure /home/nsoffer/tmp/qcow2-compressed/test.qcow2 required size: 1073741824 fully allocated size: 1073741824 $ ./qemu-img measure -f qcow2 -O qcow2 /home/nsoffer/tmp/qcow2-compressed/test.qcow2 required size: 2490368 fully allocated size: 1074135040 $ ./qemu-img measure -f qcow2 -O qcow2 /home/nsoffer/tmp/qcow2-compressed/test-compressed.qcow2 required size: 2490368 fully allocated size: 1074135040 $ ls -lh /home/nsoffer/tmp/qcow2-compressed/test* -rw-r--r--. 1 nsoffer nsoffer 323K Jul 13 17:38 /home/nsoffer/tmp/qcow2-compressed/test-compressed.qcow2 -rw-rw-r--. 1 nsoffer nsoffer 1.0G Jul 13 17:37 /home/nsoffer/tmp/qcow2-compressed/test.img -rw-r--r--. 1 nsoffer nsoffer 2.4M Jul 13 17:38 /home/nsoffer/tmp/qcow2-compressed/test.qcow2 (Originally by Fred Rolland)
Kevin, Is there a way to know if a qcow image is compressed ? We could not find any indication with qemu-img info. Maybe in the header ? Thanks (Originally by Fred Rolland)
Created attachment 1297717 [details] RPM versions (all rpms) from RHV-M/engine (Originally by John Call)
Created attachment 1297718 [details] RPM versions (all rpms) from HYPERVISOR/vdsm (Originally by John Call)
(In reply to Allon Mureinik from comment #2) > John, can you please add the rpm versions of your engine and VDSM? Thanks! I've attached files that describe all versions. Also, briefly here are the rpm versions for engine and vdsm. $ grep -e "ovirt-engine-[0-9]" -e "vdsm-[0-9]" rhel-h_rpms.txt rhvm_rpms.txt rhel-h_rpms.txt:vdsm-4.19.20-1.el7ev.x86_64 rhvm_rpms.txt:ovirt-engine-4.1.3.5-0.1.el7.noarch (Originally by John Call)
Oops, I cleared needinfo from everybody, not just me. Restoring needinfo for kwolf (Originally by John Call)
(In reply to Fred Rolland from comment #4) > Is there a way to know if a qcow image is compressed ? We could not find any > indication with qemu-img info. Maybe in the header ? There isn't really such a thing as compressed qcow2 image in terms of a property of the whole image. It is just that images can contain uncompressed clusters, compressed clusters or any mix of both. 'qemu-img check' outputs the number of compressed and total clusters, so this should give you an answer. However, note that this is an operation that is a lot more expensive than a simple 'qemu-img info'. (Originally by Kevin Wolf)
'qemu-img check' has information about compressed clusters: $ qemu-img check rhel-guest-image-7.3-35.x86_64.qcow2 No errors were found on the image. 19448/163840 = 11.87% allocated, 96.55% fragmented, 94.38% compressed clusters Image end offset: 562888704 $ qemu-img check img.test No errors were found on the image. 19448/163840 = 11.87% allocated, 0.00% fragmented, 0.00% compressed clusters Image end offset: 1275265024 (Originally by Fred Rolland)
We need the feature described in bug 1433670, which will only be available in RHEL 7.5, pushing out. (Originally by Allon Mureinik)
*** Bug 1491893 has been marked as a duplicate of this bug. *** (Originally by Allon Mureinik)
Fixed in 4.2, we can backport this to 4.1 if needed. (Originally by Nir Soffer)
(In reply to Nir Soffer from comment #18) > Fixed in 4.2, we can backport this to 4.1 if needed. Yes, that'd be great. (Originally by Yaniv Kaul)
We are not blocked on platform bug 1433670, because we have our own measure implementation, basically porting to python of the qcow2 metadata size calculation from qemu. We will drop this code when qemu-img measure will be available. (Originally by Nir Soffer)
Verified, Performed scenario described in comment 0: 1. Uploaded (GUI) a compressed qcow2 disk (rhel guest image) to FC storage domain. $ qemu-img check new.disk No errors were found on the image. 98021/163840 = 59.83% allocated, 96.53% fragmented, 95.04% compressed clusters Image end offset: 2407727104 2. Create a VM using the uploaded disk. 3. Create a template from the VM in step 2 (on the same FC storage domain). Now operation finishes successfully. Another test I've tried is copying the compressed qcow2 disk (that was uploaded) to a different FC storage domain - also works. Builds used: rhevm-4.1.7.4-0.1.el7.noarch / ovirt-engine-4.1.7.4-0.1.el7.noarch vdsm-4.19.35-1.el7ev.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3139