Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 970559 - qemu-img truncates files
qemu-img truncates files
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.6
Unspecified Unspecified
unspecified Severity high
: rc
: ---
Assigned To: Kevin Wolf
Virtualization Bugs
:
Depends On:
Blocks: 960931 976625 978140 978143
  Show dependency treegraph
 
Reported: 2013-06-04 07:00 EDT by Arik
Modified: 2013-08-23 13:11 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-02 09:03:35 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Arik 2013-06-04 07:00:43 EDT
Description of problem:
When converting file with size that is not a multiple of 512 (block size), the resulted file is a trimmed version of the original file (and it's size is the closest multiple of 512).

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. create a file with size that is not a multiple of 512
2. invoke qemu-img convert on it
3.

Actual results:
The converted file doesn't have the same content as the original file, the file is truncated

Expected results:
the content of the converted file should be the same as the content of the original file, and its size should be padded to the closest multiple of 512 which is above the current size

Additional info:
output of manual invokation of qemu-img on file with size 855020141:

-bash-4.2$  ls -ls /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com\:_export_images_rnd_ahadas_domain32/d94ec889-63d4-4c13-a79c-3821f042b0dc/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_ahadas_tmp__export__domain/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7 /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccc
490299 -rw-r--r--+ 1 vdsm kvm 855020032 Jun  3  2013 /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccc
490300 -rw-rw----+ 1 vdsm kvm 855020141 Jun  3 06:39 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_ahadas_domain32/d94ec889-63d4-4c13-a79c-3821f042b0dc/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7
490302 -rw-rw----+ 1 vdsm kvm 855020032 Jun  3 06:43 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_ahadas_tmp__export__domain/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7
-bash-4.2$
-bash-4.2$
-bash-4.2$
-bash-4.2$
-bash-4.2$ touch /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccd
-bash-4.2$  ls -ls /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com\:_export_images_rnd_ahadas_domain32/d94ec889-63d4-4c13-a79c-3821f042b0dc/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_ahadas_tmp__export__domain/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7 /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccc /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccd
490299 -rw-r--r--+ 1 vdsm kvm 855020032 Jun  3  2013 /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccc
     1 -rw-r--r--+ 1 vdsm kvm         0 Jun  3  2013 /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccd
490300 -rw-rw----+ 1 vdsm kvm 855020141 Jun  3 06:39 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_ahadas_domain32/d94ec889-63d4-4c13-a79c-3821f042b0dc/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7
490302 -rw-rw----+ 1 vdsm kvm 855020032 Jun  3 06:43 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_ahadas_tmp__export__domain/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7
-bash-4.2$
-bash-4.2$
-bash-4.2$
-bash-4.2$ /usr/bin/qemu-img convert -t none -f raw /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/d94ec889-63d4-4c13-a79c-3821f042b0dc/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7 -O raw /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccd
-bash-4.2$
-bash-4.2$
-bash-4.2$
-bash-4.2$  ls -ls /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com\:_export_images_rnd_ahadas_domain32/d94ec889-63d4-4c13-a79c-3821f042b0dc/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_ahadas_tmp__export__domain/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7 /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccc /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccd
490299 -rw-r--r--+ 1 vdsm kvm 855020032 Jun  3  2013 /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccc
490299 -rw-r--r--+ 1 vdsm kvm 855020032 Jun  3  2013 /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-355368759ccd
490300 -rw-rw----+ 1 vdsm kvm 855020141 Jun  3 06:39 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_ahadas_domain32/d94ec889-63d4-4c13-a79c-3821f042b0dc/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7
490302 -rw-rw----+ 1 vdsm kvm 855020032 Jun  3 06:43 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_ahadas_tmp__export__domain/31ab0d93-d25a-4dfc-9bfd-8381a4c8d2fb/images/f5b1b42b-df23-43e0-be3f-7b1fc7297e68/2365b985-4036-4b04-a3bf-3553687598c7
Comment 4 Kevin Wolf 2013-06-17 10:07:44 EDT
I think rounding up the image size is possible without causing regressions
(though this would have to be checked) and makes some sense.

But I'm not sure if it should matter in practice, because all images should be
aligned to sector size. You can't really use half sectors at the end of a disk.
What's the use case for images with misaligned size?
Comment 5 Ademar Reis 2013-07-26 09:22:42 EDT
(In reply to Kevin Wolf from comment #4)
> I think rounding up the image size is possible without causing regressions
> (though this would have to be checked) and makes some sense.
> 
> But I'm not sure if it should matter in practice, because all images should
> be
> aligned to sector size. You can't really use half sectors at the end of a
> disk.
> What's the use case for images with misaligned size?

^^^ NEEDINFO()
Comment 6 Arik 2013-07-28 11:01:55 EDT
In ovirt/rhev we're using qemu-img to copy images from one storage domain to another (because it preserves the sparseness of the images better than 'cp' command).

We're also treating files that contain memory dump, which are created when suspending the VM or taking a snapshot that includes memory, as images. The images that contain memory as part of snapshots are new in ovirt - they are created as raw images and can be copied between storage domains (in contrast to the other kind of memory images we had so far which cannot be copied to other storage domains). Those memory images can, and most probably will, be with misaligned size.
Comment 7 Kevin Wolf 2013-07-29 05:27:36 EDT
Unfortunately there weren't any replies to my mail on the VDSM mailing list,
but I think at least on IRC we agreed that using qemu-img for anything else
than disk images is abuse and should be fixed in VDSM?
Comment 8 Arik 2013-07-29 07:35:17 EDT
I also think it is an abuse to treat those files as qemu images, but as you noticed this issue is controversial on our side.

Correct me if I'm wrong: from qemu perspective, those images are considered to be invalid images, as you expect all images to have aligned size. You chose to handle this "wrong input" by truncate the images such that they'll have aligned size.

I don't know how difficult is it on your side to change the way you handle this "wrong input" by creating the destination images to have size which is the original size rounded up to the closest aligned size (with zeros).

If you'll be able to do so, it would be helpful for us, because on our side it's not easy - we'll either have to have another step of padding the file right after it is created by libvirt or to have this step right before copying the file to other storage domain. It would work, but if it is simpler on your side, it would be better.
Comment 9 Kevin Wolf 2013-07-29 10:56:31 EDT
(In reply to Arik from comment #8)
> Correct me if I'm wrong: from qemu perspective, those images are considered
> to be invalid images, as you expect all images to have aligned size. You
> chose to handle this "wrong input" by truncate the images such that they'll
> have aligned size.

In fact I think this is not a conscious choice, but just what happened to
result from the obvious code. Perhaps qemu should fail to open raw images that
aren't sector-aligned.

> I don't know how difficult is it on your side to change the way you handle
> this "wrong input" by creating the destination images to have size which is
> the original size rounded up to the closest aligned size (with zeros).

It might be doable, with the danger of making the behaviour inconsistent. At
present, I'm pretty sure that all qemu code consistently rounds down.
Consistency is more important that allowing others to abuse the code.

First and foremost I want you guys to do the Right Thing (tm)...

> If you'll be able to do so, it would be helpful for us, because on our side
> it's not easy - we'll either have to have another step of padding the file
> right after it is created by libvirt or to have this step right before
> copying the file to other storage domain. It would work, but if it is
> simpler on your side, it would be better.

...which this isn't. :-/

You should simply stop using qemu-img for something that is not a disk image.
Making that content just look a bit more like a disk image is _not_ a proper
solution, but just adding to the pile of hacks and potential future problems.
Comment 10 Ademar Reis 2013-07-30 09:52:24 EDT
(In reply to Arik from comment #6)
> In ovirt/rhev we're using qemu-img to copy images from one storage domain to
> another (because it preserves the sparseness of the images better than 'cp'
> command).

If that's the only use-case, wouldn't "cp --sparse=always" work for you? Using qemu-img for this a very hackish thing.

If the cp above doesn't work, maybe you can implement your script/app to copy the images?
Comment 11 Arik 2013-07-30 10:13:48 EDT
iiuc qemu-img worked better than "cp --sparse=always" on some cases, but Ayal is more familiar with this than I do, so I forward those questions for Ayal
Comment 12 Ayal Baron 2013-07-30 10:25:20 EDT
(In reply to Arik from comment #11)
> iiuc qemu-img worked better than "cp --sparse=always" on some cases, but
> Ayal is more familiar with this than I do, so I forward those questions for
> Ayal

There are several reasons why we're using qemu-img and not cp:
1. cp is not guaranteed to work on block devices
2. using qemu-img dedups zeros for qcow images
3. using a single manipulation utility reduces complexity (less code = less bugs)

The use cases we have include copying from file to block device and vice versa (naturally file to file and block to block as well), converting from qcow to raw and vice versa, etc.
Comment 13 Kevin Wolf 2013-07-31 10:00:39 EDT
Nobody is asking you to stop using qemu-img in general. It's a good tool for
handling everything related to disk images and you should use it.

I'm only asking you to stop abusing it, i.e. trying to use it for things that
are not disk images. The advantages that you list are very real for disk
images, but a memory dump isn't a disk, it doesn't need to be on a block device
and it's not in qcow2 format. There's no reason or even justification to use
qemu-img with it.
Comment 14 Ademar Reis 2013-08-02 09:03:35 EDT
So after all the discussions above, it should be clear by now that using qemu-img for non-image files is not expected and definitely not encouraged, so I'm closing this bug.

Note You need to log in before you can comment on or make changes to this bug.