Bug 1433670
| Summary: | Provide an API that estimates the size of QCOW2 image converted from a raw image | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Maor <mlipchuk> | |
| Component: | qemu-kvm-rhev | Assignee: | Stefan Hajnoczi <stefanha> | |
| Status: | CLOSED ERRATA | QA Contact: | Ping Li <pingl> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 7.4 | CC: | amureini, chayang, cliao, coli, frolland, jcall, juzhang, knoel, michen, mrezanin, mtessun, ngu, pingl, rjones, stefanha, tnisan, virt-maint, yilzhang, ykaul | |
| Target Milestone: | rc | Keywords: | FutureFeature | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | qemu-kvm-rhev-2.10.0-1.el7 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1475947 1475974 (view as bug list) | Environment: | ||
| Last Closed: | 2018-04-11 00:12:33 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1358717, 1419240, 1475947, 1475974, 1547960 | |||
The following commits are now in qemu.git/master and will be included in the next qemu-kvm-rhev rebase: 32a1681 iotests: add test 178 for qemu-img measure 217a068 qemu-iotests: support per-format golden output files fd03c2b qemu-img: add measure subcommand c501c35 qcow2: add bdrv_measure() support 0eb4a8c qcow2: extract image creation option parsing 7c5bcc4 qcow2: make refcount size calculation conservative 95c67e3 qcow2: extract preallocation calculation function a843a22 raw-format: add bdrv_measure() support 90880ff block: add bdrv_measure() API Adding Yaniv to CC so he gets an email when this BZ moves from POST to MODIFIED. At that point qemu-kvm-rhev 2.10 can be tested (it will include this fix). Hi Stefan,
According to the test result, file size calculated by "qemu-img measure" is greater than the actual value. Could you help to check whether it is expected?
Packages tested:
kernel-3.10.0-731.el7.x86_64
qemu-kvm-rhev-2.10.0-1.el7
Test steps:
1 Calculate the file size for a new empty image.
1.1 for raw format
# qemu-img measure --size 4G -O raw
required size: 4294967296
fully allocated size: 4294967296
# qemu-img create -f raw test.img 4G
Formatting 'test.img', fmt=raw size=4294967296
[root@localhost test]# qemu-img info test.img
image: test.img
file format: raw
virtual size: 4.0G (4294967296 bytes)
disk size: 0
# qemu-img check test.img
qemu-img: This image format does not support checks
# ll test.img
-rw-r--r-- 1 root root 4294967296 Oct 10 11:13 test.img
1.2 for qcow2 format
# qemu-img measure --size 4G -O qcow2
required size: 917504 ----------> 917504 bytes
fully allocated size: 4295884800 ----------> 4295884800 bytes
# qemu-img create -f qcow2 test.qcow2 4G
Formatting 'test.qcow2', fmt=qcow2 size=4294967296 cluster_size=65536 lazy_refcounts=off refcount_bits=16
# qemu-img info test.qcow2
image: test.qcow2
file format: qcow2
virtual size: 4.0G (4294967296 bytes) ----------> 4294967296 bytes
disk size: 196K
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
# qemu-img check test.qcow2
No errors were found on the image.
Image end offset: 262144----------> 262144 bytes
# ll test.qcow2
-rw-r--r-- 1 root root 196672 Oct 10 11:15 test.qcow2
2 Calculate the file size for a converted image.
2.1 Both original and new images are on local filesystem
2.1.1 convert qcow2 to qcow2
# qemu-img measure -f qcow2 -O qcow2 rhel75.qcow2
required size: 1335033856 ----------> 1335033856 bytes
fully allocated size: 21478375424 ----------> 21478375424 bytes
# qemu-img convert -p -f qcow2 -O qcow2 rhel75.qcow2 new.qcow2
(100.00/100%)
# qemu-img info new.qcow2
image: new.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes) ----------> 21474836480 bytes
disk size: 2.0G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
# qemu-img check new.qcow2
No errors were found on the image.
20085/327680 = 6.13% allocated, 0.00% fragmented, 0.00% compressed clusters
Image end offset: 1317011456 ----------> 1317011456 bytes
# ll new.qcow2
-rw-r--r-- 1 root root 1317011456 Oct 10 11:39 new.qcow2
2.1.2 convert qcow2 to raw
# qemu-img measure -f qcow2 -O raw rhel75.qcow2
required size: 21474836480
fully allocated size: 21474836480
# qemu-img convert -p -f qcow2 -O raw rhel75.qcow2 new.img
(100.00/100%)
# qemu-img info new.img
image: new.img
file format: raw
virtual size: 20G (21474836480 bytes)
disk size: 1.2G
# ll new.img
-rw-r--r-- 1 root root 21474836480 Oct 10 11:43 new.img
2.1.3 convert raw to qcow2
# qemu-img measure -f raw -O qcow2 rhel74-64-virtio.raw
required size: 6923812864 ----------> 6923812864 bytes
fully allocated size: 21478375424 ----------> 21478375424 bytes
# qemu-img convert -p -f raw -O qcow2 rhel74-64-virtio.raw new.qcow2
(100.00/100%)
# qemu-img info new.qcow2
image: new.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes) ----------> 21474836480 bytes
disk size: 8.0G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
# qemu-img check new.qcow2
No errors were found on the image.
68526/327680 = 20.91% allocated, 0.00% fragmented, 0.00% compressed clusters
Image end offset: 4492230656 ----------> 4492230656 bytes
# ll new.qcow2
-rw-r--r-- 1 root root 4492230656 Oct 10 11:53 new.qcow2
2.1.4 convert raw to raw
# qemu-img measure -f raw -O raw rhel74-64-virtio.raw
required size: 21474836480
fully allocated size: 21474836480
# qemu-img convert -p -f raw -O raw rhel74-64-virtio.raw new.img
(100.00/100%)
# qemu-img info new.img
image: new.img
file format: raw
virtual size: 20G (21474836480 bytes)
disk size: 4.2G
# ll new.img
-rw-r--r-- 1 root root 21474836480 Oct 10 11:56 new.img
2.2 New image is on logical volumes
2.2.1 convert raw to qcow2
# qemu-img measure -f raw -O qcow2 test.raw
required size: 917504 ----------> 917504 bytes
fully allocated size: 4295884800 ----------> 4295884800 bytes
# qemu-img convert -p -f raw -O qcow2 test.raw /dev/vg/lv
(100.00/100%)
# qemu-img info /dev/vg/lv
image: /dev/vg/lv
file format: qcow2
virtual size: 4.0G (4294967296 bytes) ----------> 4294967296 bytes
disk size: 0
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
# qemu-img check /dev/vg/lv
No errors were found on the image.
Image end offset: 262144 ----------> 262144 bytes
2.2.2 convert raw to raw
# qemu-img measure -f raw -O raw test.raw
required size: 4294967296
fully allocated size: 4294967296
# qemu-img convert -p -f raw -O raw test.raw /dev/vg/lv
(100.00/100%)
# qemu-img info /dev/vg/lv
image: /dev/vg/lv
file format: raw
virtual size: 5.0G (5368709120 bytes) ----------> size of lv is 5g, so it
disk size: 0 should be expected.
(In reply to pingl from comment #5) > According to the test result, file size calculated by "qemu-img measure" is > greater than the actual value. Could you help to check whether it is > expected? Yes, it's fine is qemu-img measure reports larger sizes than the actual image. The qemu-img documentation says: "The values reported are guaranteed to be large enough to fit the image". (In reply to Stefan Hajnoczi from comment #6) > (In reply to pingl from comment #5) > > According to the test result, file size calculated by "qemu-img measure" is > > greater than the actual value. Could you help to check whether it is > > expected? > > Yes, it's fine is qemu-img measure reports larger sizes than the actual > image. The qemu-img documentation says: "The values reported are guaranteed > to be large enough to fit the image". I am still confused with the documentation in the qemu-img manpage[1]. For qcow2 format, It should be smaller? [1]documentation The "required size" is the file size of the new image. It may be smaller than the virtual disk size if the image format supports compact representation. (In reply to pingl from comment #7) > (In reply to Stefan Hajnoczi from comment #6) > > (In reply to pingl from comment #5) > > > According to the test result, file size calculated by "qemu-img measure" is > > > greater than the actual value. Could you help to check whether it is > > > expected? > > > > Yes, it's fine is qemu-img measure reports larger sizes than the actual > > image. The qemu-img documentation says: "The values reported are guaranteed > > to be large enough to fit the image". > > > I am still confused with the documentation in the qemu-img manpage[1]. > For qcow2 format, It should be smaller? > > [1]documentation > The "required size" is the file size of the new image. It may be > smaller than the virtual disk size if the image format supports > compact representation. The comment about qcow2 refers to the fact that the qcow2 file for a 10 GB virtual disk might only require a host file of size 4 MB, for example, because qcow2 only stores clusters that have been allocated (it has a "compact representation"). That is different from saying that qemu-img measure may report a size that is larger than the actual size of the newly converted file. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:1104 |
Description of problem: I was wondering if that will be possible to provide a new API that estimates the size of qcow2 image converted from a raw image. We could use this new API to allocate the size more precisely before the convert operation. The operation that we are trying to do is to convert a raw sparse image from NFS or from block device to qcow2 image on thin provisioned block device In ovirt thin provisioned block device is a regular lv, and we like to allocate only the required size for the the qcow file. Our current (stupid) solution is to allocate the entire LV using the size of the raw image. Here is an example flow: $ truncate -s 10G test.raw We don't know what will be the size of the qcow on the block storage, so we allocate the entire LV: $ lvcreate --size 10G vg/lv Then we convert the file to the new LV: $ qemu-img convert -f raw -O qcow2 test.raw /dev/vg/lv After the copy we can check the actual size: $ qemu-img check /dev/vg/lv And reduce the LV: $ lvreduce -L128m vg/lv But we like to avoid the allocation, and allocate only the needed size before we convert the image. We think that the best way to solve this issue is to return this info from qemu-img, maybe as a flag to qemu-img convert that will calculate the size of the converted image without doing any writes. Version-Release number of selected component (if applicable):