Bug 1470435 - Handle copy of compressed QCOWs image (like rhel guest images) from one to block storage domain.
Handle copy of compressed QCOWs image (like rhel guest images) from one to bl...
Status: VERIFIED
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
4.1.3
Unspecified Unspecified
high Severity medium
: ovirt-4.2.0
: ---
Assigned To: Nir Soffer
Raz Tamir
: ZStream
: 1491893 (view as bug list)
Depends On:
Blocks: 1503219
  Show dependency treegraph
 
Reported: 2017-07-12 20:33 EDT by John Call
Modified: 2017-10-24 10:21 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: When copying disks using qcow2 compressed format, the destination disk size was not calculated correctly, assuming that the disk was not compressed. Consequence: Trying to copy uploaded disk using qcow2 compressed format or a cloning a vim using such disk was failing. Fix: The system estimate the destination disk size based on the qcow2 actual image format. Result: It is possible now copy compressed disks and clone vms using them.
Story Points: ---
Clone Of:
: 1503219 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdsm log from failing hypervisor (932.32 KB, application/x-gzip)
2017-07-12 20:33 EDT, John Call
no flags Details
RPM versions (all rpms) from RHV-M/engine (36.62 KB, text/plain)
2017-07-13 11:10 EDT, John Call
no flags Details
RPM versions (all rpms) from HYPERVISOR/vdsm (22.94 KB, text/plain)
2017-07-13 11:11 EDT, John Call
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3186081 None None None 2017-09-17 04:56 EDT
oVirt gerrit 82084 master MERGED qcow2: Support compressed qcow2 uploads 2017-10-02 11:38 EDT
oVirt gerrit 82892 ovirt-4.1 MERGED qcow2: Support compressed qcow2 uploads 2017-10-17 11:16 EDT

  None (edit)
Description John Call 2017-07-12 20:33:06 EDT
Created attachment 1297334 [details]
vdsm log from failing hypervisor

Version-Release number of selected component (if applicable):
4.1.3

Description of problem:
In a new RHV 4.1.3 environment I'm unable to make a template because VDSM reports 'No space left on device".  Storage domain is Fibre Channel-based, newly provisoned, and empty except for the OVF_STORE images and a single copy of the rhel-guest-image I uploaded via GUI.

Allon mentioned "In the attached log we see qemu-image convert failing with a space error, not sanlock - which is a completely different issue"

Additional info at http://post-office.corp.redhat.com/archives/rhev-tech/2017-July/msg00109.html



How reproducible:
Create a new storage domain using Fibre Channel
Upload the rhel-guest-image
Create VM
Attempt to Make template from VM


Actual results:
Data is copied for a while (few minutes,) then the task fails and GUI reports "low level Image copy failed"

Expected results:
Template should be created, space-efficient (e.g. sparse/thin)

Additional info:
http://post-office.corp.redhat.com/archives/rhev-tech/2017-July/msg00109.html
Comment 1 John Call 2017-07-12 20:37:33 EDT
The rhel-guest-image was downloaded from [1]  Search for "KVM Guest Image"  I used the 7.3 version with sha256sum of 55a581618199240c5d6ce332653c3e506cb05da863ad2e4a50d5e068e9790d88

[1] - http://access.redhat.com/downloads/content/69/ver=/rhel---7/7.3/x86_64/product-software
Comment 2 Allon Mureinik 2017-07-13 07:46:21 EDT
John, can you please add the rpm versions of your engine and VDSM? Thanks!
Comment 3 Fred Rolland 2017-07-13 10:57:01 EDT
The image is compressed, and with block storage the Vdsm did not compute the target size big enough.
In file storage, it is not an issue as the file system is sparse.

In order to make it work in block storage, you will need to convert to uncompressed before uploading:

qemu-img convert -f qcow2 rhel-guest-image-7.3-35.x86_64.qcow2 -O qcow2 -o compat=1.1 uncompressed.qcow2


In qemu version 2.0, in RHEL 7.5, we will be able to get the right size using 'qemu-img mesure' :

$ ./qemu-img measure /home/nsoffer/tmp/qcow2-compressed/test.qcow2 
required size: 1073741824
fully allocated size: 1073741824

$ ./qemu-img measure -f qcow2 -O qcow2 /home/nsoffer/tmp/qcow2-compressed/test.qcow2 
required size: 2490368
fully allocated size: 1074135040

$ ./qemu-img measure -f qcow2 -O qcow2 /home/nsoffer/tmp/qcow2-compressed/test-compressed.qcow2 
required size: 2490368
fully allocated size: 1074135040

$ ls -lh /home/nsoffer/tmp/qcow2-compressed/test*
-rw-r--r--. 1 nsoffer nsoffer 323K Jul 13 17:38 /home/nsoffer/tmp/qcow2-compressed/test-compressed.qcow2
-rw-rw-r--. 1 nsoffer nsoffer 1.0G Jul 13 17:37 /home/nsoffer/tmp/qcow2-compressed/test.img
-rw-r--r--. 1 nsoffer nsoffer 2.4M Jul 13 17:38 /home/nsoffer/tmp/qcow2-compressed/test.qcow2
Comment 4 Fred Rolland 2017-07-13 11:06:50 EDT
Kevin,

Is there a way to know if a qcow image is compressed ?
We could not find any indication with qemu-img info.
Maybe in the header ?

Thanks
Comment 5 John Call 2017-07-13 11:10 EDT
Created attachment 1297717 [details]
RPM versions (all rpms) from RHV-M/engine
Comment 6 John Call 2017-07-13 11:11 EDT
Created attachment 1297718 [details]
RPM versions (all rpms) from HYPERVISOR/vdsm
Comment 7 John Call 2017-07-13 11:14:17 EDT
(In reply to Allon Mureinik from comment #2)
> John, can you please add the rpm versions of your engine and VDSM? Thanks!

I've attached files that describe all versions.  Also, briefly here are the rpm versions for engine and vdsm.

$ grep -e "ovirt-engine-[0-9]" -e "vdsm-[0-9]" rhel-h_rpms.txt rhvm_rpms.txt 
rhel-h_rpms.txt:vdsm-4.19.20-1.el7ev.x86_64
rhvm_rpms.txt:ovirt-engine-4.1.3.5-0.1.el7.noarch
Comment 8 John Call 2017-07-13 11:44:06 EDT
Oops, I cleared needinfo from everybody, not just me.  Restoring needinfo for kwolf@redhat.com
Comment 9 Kevin Wolf 2017-07-13 12:12:18 EDT
(In reply to Fred Rolland from comment #4)
> Is there a way to know if a qcow image is compressed ?  We could not find any
> indication with qemu-img info.  Maybe in the header ?

There isn't really such a thing as compressed qcow2 image in terms of a
property of the whole image. It is just that images can contain uncompressed
clusters, compressed clusters or any mix of both.

'qemu-img check' outputs the number of compressed and total clusters, so this
should give you an answer. However, note that this is an operation that is a
lot more expensive than a simple 'qemu-img info'.
Comment 10 Fred Rolland 2017-07-16 05:32:05 EDT
'qemu-img check' has information about compressed clusters:

$ qemu-img check rhel-guest-image-7.3-35.x86_64.qcow2
No errors were found on the image.
19448/163840 = 11.87% allocated, 96.55% fragmented, 94.38% compressed clusters
Image end offset: 562888704

$ qemu-img check img.test 
No errors were found on the image.
19448/163840 = 11.87% allocated, 0.00% fragmented, 0.00% compressed clusters
Image end offset: 1275265024
Comment 16 Allon Mureinik 2017-09-12 09:44:17 EDT
We need the feature described in bug 1433670, which will only be available in RHEL 7.5, pushing out.
Comment 17 Allon Mureinik 2017-09-17 04:56:43 EDT
*** Bug 1491893 has been marked as a duplicate of this bug. ***
Comment 18 Nir Soffer 2017-10-02 11:40:46 EDT
Fixed in 4.2, we can backport this to 4.1 if needed.
Comment 19 Yaniv Kaul 2017-10-02 11:41:54 EDT
(In reply to Nir Soffer from comment #18)
> Fixed in 4.2, we can backport this to 4.1 if needed.

Yes, that'd be great.
Comment 20 Nir Soffer 2017-10-02 11:45:03 EDT
We are not blocked on platform bug 1433670, because we have our own measure
implementation, basically porting to python of the qcow2 metadata size calculation
from qemu. We will drop this code when qemu-img measure will be available.
Comment 22 Natalie Gavrielov 2017-10-24 10:21:48 EDT
Verified,

Performed scenario similar to the one described in comment 0:
1. Uploaded (GUI) a compressed qcow2 disk (rhel guest image) to ISCSI storage domain.
   $ qemu-img check new.disk
   No errors were found on the image.
   98021/163840 = 59.83% allocated, 96.53% fragmented, 95.04% compressed clusters
   Image end offset: 2407727104
2. Created a VM using the uploaded disk.
3. Created a template from the VM in step 2 (on the same ISCSI storage domain).

Now operation finishes successfully. 

Builds used:
ovirt-engine-4.2.0-0.0.master.20171022103432.gitaf9d8b6.el7.centos.noarch
vdsm-4.20.3-209.git65452bc.el7.centos.x86_64

Note You need to log in before you can comment on or make changes to this bug.