Bug 548723

Summary: qemu-img convert silently corrupts when converting from VMDK 4 to raw
Product: [Fedora] Fedora Reporter: Richard W.M. Jones <rjones>
Component: qemuAssignee: Fedora Virtualization Maintainers <virt-maint>
Status: CLOSED DEFERRED QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: rawhideCC: andrew, berrange, crobinso, djuran, dwmw2, fschwarz, hbrock, itamar, jaswinder, jforbes, mbooth, myllynen, quintela, ricardo.arguello, tburke, virt-maint, yupzhang
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-14 18:30:18 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 568471, 714159, 818523    

Description Richard W.M. Jones 2009-12-18 06:36:07 EST
Description of problem:

qemu-img convert cannot convert a VMDK4 format file to (eg) raw (or
anything else).  It silently produces a large file that mostly
contains zero bytes, and is completely unusable.

Version-Release number of selected component (if applicable):

Tested with qemu-img 0.10.5, 0.11.0, and
git d9a50a366f2178 (2009-12-11).

How reproducible:

Always.

Steps to Reproduce:
1. Export to OVF from VMWare vSphere / ESX 4.0.0.
2. Copy the resultant disk image to a Fedora machine.
3. Check the SHA1 sums (from *.mf file) to make sure image was not
   corrupted during the copy.
4. Run:
     qemu-img convert -O raw TestLinux-disk1.vmdk TestLinux-disk1.raw
5. Try to mount / use the resulting raw file, eg. using guestfish.
  
Actual results:

The raw file contains mostly zeroes, see below.  It contains zeroes
where there should be partition tables, superblocks etc. and so is
totally unusable.

Expected results:

A usable disk image.

Additional info:

Compare the entropy of the VMDK file with the resulting raw disk.
I would expect the entropy to be about the same, but you can see the
raw disk is mostly compressible (zeroes).

  $ ls -l TestLinux-disk1.vmdk
  -rw-rw-r--  1 rjones rjones  887312896 2009-12-18 10:35 TestLinux-disk1.vmdk
  $ gzip -c TestLinux-disk1.vmdk | wc -c
  860846320
  $ gzip -c TestLinux-disk1.raw | wc -c
  8744715

VMWare's OVF metadata says the following about the disk format:

  <References>
    <File ovf:href="TestLinux-disk1.vmdk"
          ovf:id="file1" ovf:size="887312896" />
  </References>
  <DiskSection>
    <Info>Virtual disk information</Info>
    <Disk ovf:capacity="8"
          ovf:capacityAllocationUnits="byte * 2^30"
          ovf:diskId="vmdisk1" ovf:fileRef="file1"
          ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" />
  </DiskSection>

qemu-img 0.10.5 fails in a different way.  It gives the error
message:

  qemu-img: error while reading

qemu-img >= 0.11.0 fail silently, no error message or error code.

I've tried this with several disk images exported from vSphere 4
and they have all failed to convert in the same way.

Test files (at time of writing these are STILL UPLOADING, with ETA
of 2009-12-19).

http://homes.merjis.com/~rich/TestLinux-disk1.vmdk
  SHA1: 2C81BAE89210B075ACC51DA9D025935470149D55
http://homes.merjis.com/~rich/TestLinux.ovf
  SHA1: 30831689B8C6F1B1A1FCBB728769B5F71056A580
Comment 1 Fedora Admin XMLRPC Client 2010-03-09 11:53:51 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 2 Fedora Admin XMLRPC Client 2010-03-09 12:19:45 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 3 Justin M. Forbes 2010-03-11 14:25:48 EST
You don't happen to know if this reproduces with qemu-img > 0.12.x or have a test image I can convert to reproduce handy?
Comment 4 Richard W.M. Jones 2010-03-11 17:04:16 EST
Nothing much has changed in the qemu vmdk block
driver since I looked at it before (I just checked upstream
git), so it's very likely to be still broken.

I have some VMDK images, but I warn you that they
are very large!  If you have somewhere I can upload
them to, I can send some your way ...
Comment 5 Bug Zapper 2010-03-15 09:36:49 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 13 development cycle.
Changing version to '13'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 6 Richard W.M. Jones 2011-03-23 08:40:49 EDT
I just checked upstream git for the driver again,
and apart from code cleanups the code is still the
same as ever.  Therefore moving the bug -> Rawhide.
Comment 7 Richard W.M. Jones 2011-04-18 06:55:16 EDT
Updated links:
http://oirase.annexia.org/tmp/TestLinux-disk1.vmdk                              
  SHA1: 2c81bae89210b075acc51da9d025935470149d55                                
http://oirase.annexia.org/tmp/TestLinux.ovf                                     
  SHA1: 30831689b8c6f1b1a1fcbb728769b5f71056a580
Comment 8 Richard W.M. Jones 2011-04-18 07:02:12 EDT
Latest qemu-img no longer silently converts to zeroes.  Instead
it gives a strange error:

$ qemu-img convert -f vmdk -O raw TestLinux-disk1.vmdk TestLinux-disk1.raw
qemu-img: Could not open 'TestLinux-disk1.vmdk': Operation not permitted
qemu-img: Could not open 'TestLinux-disk1.vmdk'
Comment 9 Daniel Berrange 2011-04-18 07:10:33 EDT
> qemu-img: Could not open 'TestLinux-disk1.vmdk': Operation not permitted

This is probably because qemu-img.c code expects  brdv_open() to return an errno value

    ret = bdrv_open(bs, filename, flags, drv);
    if (ret < 0) {
        error_report("Could not open '%s': %s", filename, strerror(-ret));
        goto fail;
    }

while the vmdk_open function just returns -1 for everything:

   ...
    return 0;
 fail:
    qemu_free(s->l1_backup_table);
    qemu_free(s->l1_table);
    qemu_free(s->l2_cache);
    return -1;
}

and by coincidence, '1 == EPERM'.  There are ~4 codepaths in vmdk_open that could fail, the VMDK magic check and then a couple of reads of metadata
Comment 11 yuping zhang 2012-02-17 04:02:03 EST
The vmdk from "Export as OVF..." doesn't work.
# qemu-img convert -O raw esx4.1-rhel5.7-i386-disk1.vmdk test-vmdk.raw
qemu-img: Could not open 'esx4.1-rhel5.7-i386-disk1.vmdk'
qemu-img: Could not open 'esx4.1-rhel5.7-i386-disk1.vmdk'

I copied a vmdk file from ESX storage directly,and then use qemu-img to convert it to raw,it works.
# qemu-img convert -O raw esx4.1-rhel5.7-i386-flat.vmdk test-vmdk.raw
# ll test-vmdk.raw 
-rw-r--r--. 1 root root 8589934592 Feb 17 16:58 test-vmdk.raw
Comment 12 Richard W.M. Jones 2012-02-17 04:43:15 EST
(In reply to comment #11)
> I copied a vmdk file from ESX storage directly,and then use qemu-img to convert
> it to raw,it works.
> # qemu-img convert -O raw esx4.1-rhel5.7-i386-flat.vmdk test-vmdk.raw
> # ll test-vmdk.raw 
> -rw-r--r--. 1 root root 8589934592 Feb 17 16:58 test-vmdk.raw

*-flat.vmdk files are not VMDK.  They are just raw files
which happen to have a .vmdk extension.  So this doesn't
really prove anything.
Comment 13 Fedora Admin XMLRPC Client 2012-03-15 13:58:28 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 14 Cole Robinson 2012-12-14 18:30:18 EST
This bug has lingered for forever, so I don't think tracking this in Fedora is going to solve much.

Rich, if you can still reproduce this with qemu.git, I'd recommend filing an upstream bug and publishing a reproducer image like you did before.