Bug 1433670

Summary: Provide an API that estimates the size of QCOW2 image converted from a raw image
Product: Red Hat Enterprise Linux 7 Reporter: Maor <mlipchuk>
Component: qemu-kvm-rhevAssignee: Stefan Hajnoczi <stefanha>
Status: CLOSED ERRATA QA Contact: Ping Li <pingl>
Severity: high Docs Contact:
Priority: high    
Version: 7.4CC: amureini, chayang, cliao, coli, frolland, jcall, juzhang, knoel, michen, mrezanin, mtessun, ngu, pingl, rjones, stefanha, tnisan, virt-maint, yilzhang, ykaul
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.10.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1475947 1475974 (view as bug list) Environment:
Last Closed: 2018-04-11 00:12:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1358717, 1419240, 1475947, 1475974, 1547960    

Description Maor 2017-03-19 07:51:47 UTC
Description of problem:
I was wondering if that will be possible to provide a new API that estimates the size of qcow2 image converted from a raw image.
We could use this new API to allocate the size more precisely before the convert operation.

The operation that we are trying to do is to convert a raw sparse image from NFS or from block device to qcow2 image on thin provisioned block device

In ovirt thin provisioned block device is a regular lv, and we like
to allocate only the required size for the the qcow file.

Our current (stupid) solution is to allocate the entire LV using the
size of the raw image.


Here is an example flow:

    $ truncate -s 10G test.raw

We don't know what will be the size of the qcow on the block storage,
so we allocate the entire LV:

    $ lvcreate --size 10G vg/lv

Then we convert the file to the new LV:

    $ qemu-img convert -f raw -O qcow2 test.raw /dev/vg/lv

After the copy we can check the actual size:

    $ qemu-img check /dev/vg/lv

And reduce the LV:

    $ lvreduce -L128m vg/lv

But we like to avoid the allocation, and allocate only the needed size
before we convert the image.

We think that the best way to solve this issue is to return this info
from qemu-img, maybe as a flag to qemu-img convert that will
calculate the size of the converted image without doing any writes.


Version-Release number of selected component (if applicable):

Comment 2 Stefan Hajnoczi 2017-07-14 09:01:00 UTC
The following commits are now in qemu.git/master and will be included in the next qemu-kvm-rhev rebase:

32a1681 iotests: add test 178 for qemu-img measure
217a068 qemu-iotests: support per-format golden output files
fd03c2b qemu-img: add measure subcommand
c501c35 qcow2: add bdrv_measure() support
0eb4a8c qcow2: extract image creation option parsing
7c5bcc4 qcow2: make refcount size calculation conservative
95c67e3 qcow2: extract preallocation calculation function
a843a22 raw-format: add bdrv_measure() support
90880ff block: add bdrv_measure() API

Comment 3 Stefan Hajnoczi 2017-09-20 14:54:13 UTC
Adding Yaniv to CC so he gets an email when this BZ moves from POST to MODIFIED.  At that point qemu-kvm-rhev 2.10 can be tested (it will include this fix).

Comment 5 Ping Li 2017-10-10 16:21:04 UTC
Hi Stefan,

According to the test result, file size calculated by "qemu-img measure" is greater than the actual value. Could you help to check whether it is expected?

Packages tested:
kernel-3.10.0-731.el7.x86_64
qemu-kvm-rhev-2.10.0-1.el7

Test steps:
1 Calculate the file size for a new empty image.
1.1 for raw format
# qemu-img measure --size 4G -O raw
required size: 4294967296
fully allocated size: 4294967296

# qemu-img create -f raw test.img 4G
Formatting 'test.img', fmt=raw size=4294967296
[root@localhost test]# qemu-img info test.img 
image: test.img
file format: raw
virtual size: 4.0G (4294967296 bytes)
disk size: 0
# qemu-img check test.img 
qemu-img: This image format does not support checks
# ll test.img 
-rw-r--r-- 1 root root 4294967296 Oct 10 11:13 test.img

1.2 for qcow2 format
# qemu-img measure --size 4G -O qcow2
required size: 917504  ----------> 917504 bytes
fully allocated size: 4295884800  ----------> 4295884800 bytes

# qemu-img create -f qcow2 test.qcow2 4G
Formatting 'test.qcow2', fmt=qcow2 size=4294967296 cluster_size=65536 lazy_refcounts=off refcount_bits=16
# qemu-img info test.qcow2 
image: test.qcow2
file format: qcow2
virtual size: 4.0G (4294967296 bytes)  ----------> 4294967296 bytes
disk size: 196K
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
# qemu-img check test.qcow2 
No errors were found on the image.
Image end offset: 262144----------> 262144 bytes
# ll test.qcow2                     
-rw-r--r-- 1 root root 196672 Oct 10 11:15 test.qcow2

2 Calculate the file size for a converted image.
2.1 Both original and new images are on local filesystem
2.1.1 convert qcow2 to qcow2
# qemu-img measure -f qcow2 -O qcow2 rhel75.qcow2 
required size: 1335033856  ----------> 1335033856 bytes
fully allocated size: 21478375424  ----------> 21478375424 bytes

# qemu-img convert -p -f qcow2 -O qcow2 rhel75.qcow2 new.qcow2
    (100.00/100%)
# qemu-img info new.qcow2 
image: new.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)  ----------> 21474836480 bytes
disk size: 2.0G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
# qemu-img check new.qcow2 
No errors were found on the image.
20085/327680 = 6.13% allocated, 0.00% fragmented, 0.00% compressed clusters
Image end offset: 1317011456  ----------> 1317011456 bytes
# ll new.qcow2 
-rw-r--r-- 1 root root 1317011456 Oct 10 11:39 new.qcow2

2.1.2 convert qcow2 to raw
# qemu-img measure -f qcow2 -O raw rhel75.qcow2 
required size: 21474836480
fully allocated size: 21474836480

# qemu-img convert -p -f qcow2 -O raw rhel75.qcow2 new.img
    (100.00/100%)
# qemu-img info new.img 
image: new.img
file format: raw
virtual size: 20G (21474836480 bytes)
disk size: 1.2G
# ll new.img 
-rw-r--r-- 1 root root 21474836480 Oct 10 11:43 new.img

2.1.3 convert raw to qcow2
# qemu-img measure -f raw -O qcow2 rhel74-64-virtio.raw 
required size: 6923812864  ----------> 6923812864 bytes
fully allocated size: 21478375424  ----------> 21478375424 bytes

# qemu-img convert -p -f raw -O qcow2 rhel74-64-virtio.raw new.qcow2
    (100.00/100%)
# qemu-img info new.qcow2 
image: new.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)  ----------> 21474836480 bytes
disk size: 8.0G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
# qemu-img check new.qcow2 
No errors were found on the image.
68526/327680 = 20.91% allocated, 0.00% fragmented, 0.00% compressed clusters
Image end offset: 4492230656  ----------> 4492230656 bytes
# ll new.qcow2 
-rw-r--r-- 1 root root 4492230656 Oct 10 11:53 new.qcow2

2.1.4 convert raw to raw
# qemu-img measure -f raw -O raw rhel74-64-virtio.raw 
required size: 21474836480
fully allocated size: 21474836480

# qemu-img convert -p -f raw -O raw rhel74-64-virtio.raw new.img 
    (100.00/100%)
# qemu-img info new.img 
image: new.img
file format: raw
virtual size: 20G (21474836480 bytes)
disk size: 4.2G
# ll new.img 
-rw-r--r-- 1 root root 21474836480 Oct 10 11:56 new.img

2.2 New image is on logical volumes
2.2.1 convert raw to qcow2
# qemu-img measure -f raw -O qcow2 test.raw 
required size: 917504  ----------> 917504 bytes
fully allocated size: 4295884800  ----------> 4295884800 bytes

# qemu-img convert -p -f raw -O qcow2 test.raw /dev/vg/lv 
    (100.00/100%)
# qemu-img info /dev/vg/lv 
image: /dev/vg/lv
file format: qcow2
virtual size: 4.0G (4294967296 bytes)  ----------> 4294967296 bytes
disk size: 0
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
# qemu-img check /dev/vg/lv 
No errors were found on the image.
Image end offset: 262144  ----------> 262144 bytes

2.2.2 convert raw to raw
# qemu-img measure -f raw -O raw test.raw 
required size: 4294967296
fully allocated size: 4294967296

# qemu-img convert -p -f raw -O raw test.raw /dev/vg/lv 
    (100.00/100%)
# qemu-img info /dev/vg/lv 
image: /dev/vg/lv
file format: raw
virtual size: 5.0G (5368709120 bytes)  ----------> size of lv is 5g, so it
disk size: 0                                       should be expected.

Comment 6 Stefan Hajnoczi 2017-10-11 16:07:34 UTC
(In reply to pingl from comment #5)
> According to the test result, file size calculated by "qemu-img measure" is
> greater than the actual value. Could you help to check whether it is
> expected?

Yes, it's fine is qemu-img measure reports larger sizes than the actual image.  The qemu-img documentation says: "The values reported are guaranteed to be large enough to fit the image".

Comment 7 Ping Li 2017-10-12 04:47:39 UTC
(In reply to Stefan Hajnoczi from comment #6)
> (In reply to pingl from comment #5)
> > According to the test result, file size calculated by "qemu-img measure" is
> > greater than the actual value. Could you help to check whether it is
> > expected?
> 
> Yes, it's fine is qemu-img measure reports larger sizes than the actual
> image.  The qemu-img documentation says: "The values reported are guaranteed
> to be large enough to fit the image".


I am still confused with the documentation in the qemu-img manpage[1].
For qcow2 format, It should be smaller?

[1]documentation
The "required size" is the file size of the new image.  It may be
smaller than the virtual disk size if the image format supports
compact representation.

Comment 8 Stefan Hajnoczi 2017-10-18 15:37:34 UTC
(In reply to pingl from comment #7)
> (In reply to Stefan Hajnoczi from comment #6)
> > (In reply to pingl from comment #5)
> > > According to the test result, file size calculated by "qemu-img measure" is
> > > greater than the actual value. Could you help to check whether it is
> > > expected?
> > 
> > Yes, it's fine is qemu-img measure reports larger sizes than the actual
> > image.  The qemu-img documentation says: "The values reported are guaranteed
> > to be large enough to fit the image".
> 
> 
> I am still confused with the documentation in the qemu-img manpage[1].
> For qcow2 format, It should be smaller?
> 
> [1]documentation
> The "required size" is the file size of the new image.  It may be
> smaller than the virtual disk size if the image format supports
> compact representation.

The comment about qcow2 refers to the fact that the qcow2 file for a 10 GB virtual disk might only require a host file of size 4 MB, for example, because qcow2 only stores clusters that have been allocated (it has a "compact representation").

That is different from saying that qemu-img measure may report a size that is larger than the actual size of the newly converted file.

Comment 10 Ping Li 2017-10-19 06:06:32 UTC
Closed the bug as verified according to the comment 5 and comment 9

Comment 14 errata-xmlrpc 2018-04-11 00:12:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1104