Bug 548723 - qemu-img convert silently corrupts when converting from VMDK 4 to raw
qemu-img convert silently corrupts when converting from VMDK 4 to raw
Status: CLOSED DEFERRED
Product: Fedora
Classification: Fedora
Component: qemu (Show other bugs)
rawhide
All Linux
low Severity medium
: ---
: ---
Assigned To: Fedora Virtualization Maintainers
Fedora Extras Quality Assurance
:
Depends On:
Blocks: 568471 714159 818523
  Show dependency treegraph
 
Reported: 2009-12-18 06:36 EST by Richard W.M. Jones
Modified: 2013-01-09 17:07 EST (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-14 18:30:18 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Richard W.M. Jones 2009-12-18 06:36:07 EST
Description of problem:

qemu-img convert cannot convert a VMDK4 format file to (eg) raw (or
anything else).  It silently produces a large file that mostly
contains zero bytes, and is completely unusable.

Version-Release number of selected component (if applicable):

Tested with qemu-img 0.10.5, 0.11.0, and
git d9a50a366f2178 (2009-12-11).

How reproducible:

Always.

Steps to Reproduce:
1. Export to OVF from VMWare vSphere / ESX 4.0.0.
2. Copy the resultant disk image to a Fedora machine.
3. Check the SHA1 sums (from *.mf file) to make sure image was not
   corrupted during the copy.
4. Run:
     qemu-img convert -O raw TestLinux-disk1.vmdk TestLinux-disk1.raw
5. Try to mount / use the resulting raw file, eg. using guestfish.
  
Actual results:

The raw file contains mostly zeroes, see below.  It contains zeroes
where there should be partition tables, superblocks etc. and so is
totally unusable.

Expected results:

A usable disk image.

Additional info:

Compare the entropy of the VMDK file with the resulting raw disk.
I would expect the entropy to be about the same, but you can see the
raw disk is mostly compressible (zeroes).

  $ ls -l TestLinux-disk1.vmdk
  -rw-rw-r--  1 rjones rjones  887312896 2009-12-18 10:35 TestLinux-disk1.vmdk
  $ gzip -c TestLinux-disk1.vmdk | wc -c
  860846320
  $ gzip -c TestLinux-disk1.raw | wc -c
  8744715

VMWare's OVF metadata says the following about the disk format:

  <References>
    <File ovf:href="TestLinux-disk1.vmdk"
          ovf:id="file1" ovf:size="887312896" />
  </References>
  <DiskSection>
    <Info>Virtual disk information</Info>
    <Disk ovf:capacity="8"
          ovf:capacityAllocationUnits="byte * 2^30"
          ovf:diskId="vmdisk1" ovf:fileRef="file1"
          ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" />
  </DiskSection>

qemu-img 0.10.5 fails in a different way.  It gives the error
message:

  qemu-img: error while reading

qemu-img >= 0.11.0 fail silently, no error message or error code.

I've tried this with several disk images exported from vSphere 4
and they have all failed to convert in the same way.

Test files (at time of writing these are STILL UPLOADING, with ETA
of 2009-12-19).

http://homes.merjis.com/~rich/TestLinux-disk1.vmdk
  SHA1: 2C81BAE89210B075ACC51DA9D025935470149D55
http://homes.merjis.com/~rich/TestLinux.ovf
  SHA1: 30831689B8C6F1B1A1FCBB728769B5F71056A580
Comment 1 Fedora Admin XMLRPC Client 2010-03-09 11:53:51 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 2 Fedora Admin XMLRPC Client 2010-03-09 12:19:45 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 3 Justin M. Forbes 2010-03-11 14:25:48 EST
You don't happen to know if this reproduces with qemu-img > 0.12.x or have a test image I can convert to reproduce handy?
Comment 4 Richard W.M. Jones 2010-03-11 17:04:16 EST
Nothing much has changed in the qemu vmdk block
driver since I looked at it before (I just checked upstream
git), so it's very likely to be still broken.

I have some VMDK images, but I warn you that they
are very large!  If you have somewhere I can upload
them to, I can send some your way ...
Comment 5 Bug Zapper 2010-03-15 09:36:49 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 13 development cycle.
Changing version to '13'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 6 Richard W.M. Jones 2011-03-23 08:40:49 EDT
I just checked upstream git for the driver again,
and apart from code cleanups the code is still the
same as ever.  Therefore moving the bug -> Rawhide.
Comment 7 Richard W.M. Jones 2011-04-18 06:55:16 EDT
Updated links:
http://oirase.annexia.org/tmp/TestLinux-disk1.vmdk                              
  SHA1: 2c81bae89210b075acc51da9d025935470149d55                                
http://oirase.annexia.org/tmp/TestLinux.ovf                                     
  SHA1: 30831689b8c6f1b1a1fcbb728769b5f71056a580
Comment 8 Richard W.M. Jones 2011-04-18 07:02:12 EDT
Latest qemu-img no longer silently converts to zeroes.  Instead
it gives a strange error:

$ qemu-img convert -f vmdk -O raw TestLinux-disk1.vmdk TestLinux-disk1.raw
qemu-img: Could not open 'TestLinux-disk1.vmdk': Operation not permitted
qemu-img: Could not open 'TestLinux-disk1.vmdk'
Comment 9 Daniel Berrange 2011-04-18 07:10:33 EDT
> qemu-img: Could not open 'TestLinux-disk1.vmdk': Operation not permitted

This is probably because qemu-img.c code expects  brdv_open() to return an errno value

    ret = bdrv_open(bs, filename, flags, drv);
    if (ret < 0) {
        error_report("Could not open '%s': %s", filename, strerror(-ret));
        goto fail;
    }

while the vmdk_open function just returns -1 for everything:

   ...
    return 0;
 fail:
    qemu_free(s->l1_backup_table);
    qemu_free(s->l1_table);
    qemu_free(s->l2_cache);
    return -1;
}

and by coincidence, '1 == EPERM'.  There are ~4 codepaths in vmdk_open that could fail, the VMDK magic check and then a couple of reads of metadata
Comment 11 yuping zhang 2012-02-17 04:02:03 EST
The vmdk from "Export as OVF..." doesn't work.
# qemu-img convert -O raw esx4.1-rhel5.7-i386-disk1.vmdk test-vmdk.raw
qemu-img: Could not open 'esx4.1-rhel5.7-i386-disk1.vmdk'
qemu-img: Could not open 'esx4.1-rhel5.7-i386-disk1.vmdk'

I copied a vmdk file from ESX storage directly,and then use qemu-img to convert it to raw,it works.
# qemu-img convert -O raw esx4.1-rhel5.7-i386-flat.vmdk test-vmdk.raw
# ll test-vmdk.raw 
-rw-r--r--. 1 root root 8589934592 Feb 17 16:58 test-vmdk.raw
Comment 12 Richard W.M. Jones 2012-02-17 04:43:15 EST
(In reply to comment #11)
> I copied a vmdk file from ESX storage directly,and then use qemu-img to convert
> it to raw,it works.
> # qemu-img convert -O raw esx4.1-rhel5.7-i386-flat.vmdk test-vmdk.raw
> # ll test-vmdk.raw 
> -rw-r--r--. 1 root root 8589934592 Feb 17 16:58 test-vmdk.raw

*-flat.vmdk files are not VMDK.  They are just raw files
which happen to have a .vmdk extension.  So this doesn't
really prove anything.
Comment 13 Fedora Admin XMLRPC Client 2012-03-15 13:58:28 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 14 Cole Robinson 2012-12-14 18:30:18 EST
This bug has lingered for forever, so I don't think tracking this in Fedora is going to solve much.

Rich, if you can still reproduce this with qemu.git, I'd recommend filing an upstream bug and publishing a reproducer image like you did before.

Note You need to log in before you can comment on or make changes to this bug.