Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1570407 - virt-v2v -i ova gives checksum error when missing files are mentioned in .mf
Summary: virt-v2v -i ova gives checksum error when missing files are mentioned in .mf
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On: 1551055
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-22 16:07 UTC by Richard W.M. Jones
Modified: 2018-10-30 07:46 UTC (History)
6 users (show)

Fixed In Version: libguestfs-1.38.1-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-30 07:45:35 UTC
Target Upstream Version:


Attachments (Terms of Use)
bz1570407-reproducer.ova (20.00 KB, application/x-tar)
2018-04-22 16:08 UTC, Richard W.M. Jones
no flags Details
RHEL7_3_042218_extra.ovf (12.76 KB, text/plain)
2018-04-24 07:49 UTC, Richard W.M. Jones
no flags Details
RHEL7_3_042218_extra.mf (243 bytes, text/plain)
2018-04-24 07:49 UTC, Richard W.M. Jones
no flags Details
v2v-1.38.1-1.log (3.12 KB, text/plain)
2018-05-10 07:09 UTC, mxie@redhat.com
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:3021 0 None None None 2018-10-30 07:46:21 UTC

Description Richard W.M. Jones 2018-04-22 16:07:15 UTC
Description of problem:

(Bug reported by Nisim Simsolo)

$ virt-v2v -i ova ~/RHEL7_3_042218_extra.ova -o null
[   0.0] Opening the source -i ova /home/rjones/RHEL7_3_042218_extra.ova
tar: RHEL7_3_042218_extra-disk1.vmdk: Not found in archive
tar: Exiting with failure status due to previous errors
virt-v2v: error: checksum of disk RHEL7_3_042218_extra-disk1.vmdk does not 
match manifest /var/tmp/ova.isX3xH/RHEL7_3_042218_extra.mf (actual 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
da39a3ee5e6b4b0d3255bfef95601890afd80709, expected 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
ca3727b9c701ffe2552e4809b3ecf9d85a82873c)

The .ova file is peculiar.  The files it contains are:

$ tar tf ~/RHEL7_3_042218_extra.ova
RHEL7_3_042218_extra.ovf
RHEL7_3_042218_extra.mf
RHEL7_3_042218_extra-disk1.vmdk.000000000

but the .mf file mentions a non-existent file: 

$ cat RHEL7_3_042218_extra.mf
SHA1(RHEL7_3_042218_extra.ovf)= bf626b725f3bd251cc37771dd6ce48c9915f00c3
SHA1(RHEL7_3_042218_extra-disk1.vmdk.000000000)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c
SHA1(RHEL7_3_042218_extra-disk1.vmdk)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c

Version-Release number of selected component (if applicable):

virt-v2v-1.36.10-6.6.rhvpreview.el7ev.x86_64
(also happens with virt-v2v-1.38.0-1.fc28.x86_64)
VMware Version 6.0.0 Build 2594327

How reproducible:

100%

Steps to Reproduce:
1. Use the synthetic OVA file attached to replicate the bug.

Comment 2 Richard W.M. Jones 2018-04-22 16:08:52 UTC
Created attachment 1425375 [details]
bz1570407-reproducer.ova

To reproduce the bug, download the attachment and run:

$ virt-v2v -i ova bz1570407-reproducer.ova -o null
[   0.0] Opening the source -i ova bz1570407-reproducer.ova
tar: RHEL7_3_042218_extra-disk1.vmdk: Not found in archive
tar: Exiting with failure status due to previous errors
virt-v2v: error: checksum of disk RHEL7_3_042218_extra-disk1.vmdk does not 
match manifest /var/tmp/ova.2rqgzX/RHEL7_3_042218_extra.mf (actual 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
da39a3ee5e6b4b0d3255bfef95601890afd80709, expected 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
edb35db6af8384832ac27d63773ee0bb2a15e0bd)

Comment 3 Richard W.M. Jones 2018-04-23 10:08:18 UTC
I posted some preliminary work on this:

https://www.redhat.com/archives/libguestfs/2018-April/msg00163.html

However this is not a full fix.  It will require major work on the
-i ova input mode to deal with these files (which IMHO are basically
broken).

Comment 4 Pino Toscano 2018-04-24 06:58:25 UTC
Is it a real-world OVA? At least from a cursory glance it'd seem generated by VMware, is it correct?

Comment 5 Richard W.M. Jones 2018-04-24 07:48:27 UTC
Yes it is a real world OVA (generated by VMware Version 6.0.0 Build 2594327).

I believe it has something to do with snapshots.

The file list is:

RHEL7_3_042218_extra.ovf
RHEL7_3_042218_extra.mf
RHEL7_3_042218_extra-disk1.vmdk.000000000

I will attach the OVF and MF files.

Comment 6 Richard W.M. Jones 2018-04-24 07:49:17 UTC
Created attachment 1425858 [details]
RHEL7_3_042218_extra.ovf

Comment 7 Richard W.M. Jones 2018-04-24 07:49:39 UTC
Created attachment 1425859 [details]
RHEL7_3_042218_extra.mf

Comment 8 Richard W.M. Jones 2018-04-24 07:52:05 UTC
You can see the original problem from the bug report because the
.mf file references a non-existent file:

SHA1(RHEL7_3_042218_extra-disk1.vmdk)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c

My patch series solve this problem.

However there's a second problem in the .ovf file which also
references a non-existent file:

  <References>
    <File ovf:chunkSize="7516192768" ovf:href="RHEL7_3_042218_extra-disk1.vmdk" ovf:id="file1" ovf:size="721755648"/>
  </References>

and fixing this involves some deeper surgery on v2v/input_ova.ml
(which to be fair we probably need to do at some time).

Comment 10 Richard W.M. Jones 2018-04-25 13:35:57 UTC
This should be a full fix:

https://www.redhat.com/archives/libguestfs/2018-April/msg00167.html

Using this I was able to convert the test image which
Nisim supplied to me privately.

Comment 11 Richard W.M. Jones 2018-04-27 09:14:54 UTC
Upstream in:

e5946165c v2v: -i ova: Handle OVAs containing snapshots (RHBZ#1570407).
80d6d9396 v2v: -i ova: Sanity check hrefs to ensure they stay inside the tarball.
e24771468 v2v: -i ova: Replace subdirectory function with clearer inline code.
075333ede common: mlstdutils: Inline unsafe ‘subdirectory’ function.
184dba598 v2v: -i ova: Factor out code for dealing with OVA files.
ece2c60c5 v2v: parse OVF: Export useful parse_disks function.
d1e95a30f v2v: Ignore miscellaneous tar messages when parsing tar for file locations.
112e3bc31 v2v: -i ova: Ignore non-existent files mentioned in *.mf.
190333b8b mltools: Checksums.verify_checksum{,s} returns an error type instead of throwing exception.

Comment 12 Pino Toscano 2018-05-02 11:22:36 UTC
Fixes backported in 1.38.1.

Comment 14 mxie@redhat.com 2018-05-10 07:08:56 UTC
I can reproduce the bug using attached ova with below builds:
virt-v2v-1.38.0-1.el7.x86_64
libguestfs-1.38.0-1.el7.x86_64

Reproduce steps:
1.# virt-v2v -i ova bz1570407-reproducer.ova -o null
[   0.0] Opening the source -i ova bz1570407-reproducer.ova
virt-v2v: warning: making OVA directory public readable to work around 
libvirt bug https://bugzilla.redhat.com/1045069
tar: RHEL7_3_042218_extra-disk1.vmdk: Not found in archive
tar: Exiting with failure status due to previous errors
virt-v2v: error: checksum of disk RHEL7_3_042218_extra-disk1.vmdk does not 
match manifest /var/tmp/ova.jZTuqW/RHEL7_3_042218_extra.mf (actual 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
da39a3ee5e6b4b0d3255bfef95601890afd80709, expected 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
edb35db6af8384832ac27d63773ee0bb2a15e0bd)

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


By the way, I can't export a ova like bug's attached one even if vmware 6.0 guest has snapshot so far.


Verify the bug with builds:
virt-v2v-1.38.1-1.el7.x86_64
libguestfs-1.38.1-1.el7.x86_64

Steps:
1.# virt-v2v -i ova bz1570407-reproducer.ova -o null
[   0.0] Opening the source -i ova bz1570407-reproducer.ova
virt-v2v: warning: making OVA directory public readable to work around 
libvirt bug https://bugzilla.redhat.com/1045069
virt-v2v: warning: manifest has a checksum for non-existent file 
RHEL7_3_042218_extra-disk1.vmdk (ignored)
[   0.4] Creating an overlay to protect the source from being modified
qemu-img: /var/tmp/v2vovlb05920.qcow2: Failed to read footer: Input/output error
Could not open backing image to determine size.
virt-v2v: error: qemu-img command failed, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


Hi rjones.

   I can see latest virt-v2v has improved the warning for non-existent file in mf,but v2v conversion is failed in my test (according to comment10, you could convert attached ova), could you please help to check?

Comment 15 mxie@redhat.com 2018-05-10 07:09:37 UTC
Created attachment 1434246 [details]
v2v-1.38.1-1.log

Comment 16 Pino Toscano 2018-05-10 09:14:25 UTC
> Verify the bug with builds:
> virt-v2v-1.38.1-1.el7.x86_64
> libguestfs-1.38.1-1.el7.x86_64
> 
> Steps:
> 1.# virt-v2v -i ova bz1570407-reproducer.ova -o null
> [   0.0] Opening the source -i ova bz1570407-reproducer.ova
> virt-v2v: warning: making OVA directory public readable to work around 
> libvirt bug https://bugzilla.redhat.com/1045069
> virt-v2v: warning: manifest has a checksum for non-existent file 
> RHEL7_3_042218_extra-disk1.vmdk (ignored)
> [   0.4] Creating an overlay to protect the source from being modified
> qemu-img: /var/tmp/v2vovlb05920.qcow2: Failed to read footer: Input/output
> error
> Could not open backing image to determine size.
> virt-v2v: error: qemu-img command failed, see earlier errors

Can you please try as non-root user, or to move the .ova file where even non-root users can access it (e.g. /tmp, /var/tmp, etc)?

Comment 17 mxie@redhat.com 2018-05-10 10:21:44 UTC
Hi Pino,
   
    The conversion is still failed after moving ova file to /var/tmp and running command as non-root user

[mxie@hp-dl380eg8-02 home]$ virt-v2v -i ova bz1570407-reproducer.ova -o null
[   0.0] Opening the source -i ova bz1570407-reproducer.ova
virt-v2v: warning: manifest has a checksum for non-existent file 
RHEL7_3_042218_extra-disk1.vmdk (ignored)
[   0.4] Creating an overlay to protect the source from being modified
qemu-img: /var/tmp/v2vovl528738.qcow2: Failed to read footer: Input/output error
Could not open backing image to determine size.
virt-v2v: error: qemu-img command failed, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Comment 18 Richard W.M. Jones 2018-05-16 09:39:12 UTC
(In reply to mxie@redhat.com from comment #17)
> Hi Pino,
>    
>     The conversion is still failed after moving ova file to /var/tmp and
> running command as non-root user
> 
> [mxie@hp-dl380eg8-02 home]$ virt-v2v -i ova bz1570407-reproducer.ova -o null
> [   0.0] Opening the source -i ova bz1570407-reproducer.ova
> virt-v2v: warning: manifest has a checksum for non-existent file 
> RHEL7_3_042218_extra-disk1.vmdk (ignored)

So please note here that bz1570407-reproducer.ova is not a complete,
working OVA.  It's basically something I just hacked up to
reproduce the bug.  If virt-v2v has been patched to fix the bug
then trying to actually convert bz1570407-reproducer.ova will fail.

I will send you a link to the actual OVA which caused the original
bug report (but we cannot redistribute it).

Comment 19 mxie@redhat.com 2018-05-17 10:00:50 UTC
Use the ova file provided by rjones to test the bug again


Reproduce the bug with builds
virt-v2v-1.38.0-1.el7.x86_64 
libguestfs-1.38.0-1.el7.x86_64 

Reproduce steps:
1.Prepare a ova file and there is missing file mentioned in .mf
# tar tf RHEL7_3_042218_extra.ova
RHEL7_3_042218_extra.ovf
RHEL7_3_042218_extra.mf
RHEL7_3_042218_extra-disk1.vmdk.000000000

# cat RHEL7_3_042218_extra.mf
SHA1(RHEL7_3_042218_extra.ovf)= bf626b725f3bd251cc37771dd6ce48c9915f00c3
SHA1(RHEL7_3_042218_extra-disk1.vmdk.000000000)=
ca3727b9c701ffe2552e4809b3ecf9d85a82873c
SHA1(RHEL7_3_042218_extra-disk1.vmdk)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c

2.Use virt-v2v to convert this ova file 
# virt-v2v -i ova RHEL7_3_042218_extra.ova -o null
[   0.0] Opening the source -i ova RHEL7_3_042218_extra.ova
tar: RHEL7_3_042218_extra-disk1.vmdk: Not found in archive
tar: Exiting with failure status due to previous errors
virt-v2v: error: checksum of disk RHEL7_3_042218_extra-disk1.vmdk does not 
match manifest /var/tmp/ova.T82C33/RHEL7_3_042218_extra.mf (actual 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
da39a3ee5e6b4b0d3255bfef95601890afd80709, expected 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
ca3727b9c701ffe2552e4809b3ecf9d85a82873c)

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


Verify the bug with builds:
virt-v2v-1.38.2-1.el7.x86_64
libguestfs-1.38.2-1.el7.x86_64

Steps:
1.Convert the ova file which has missing file mentioned in .mf by virt-v2v
# virt-v2v -i ova RHEL7_3_042218_extra.ova -o null
[   0.0] Opening the source -i ova RHEL7_3_042218_extra.ova
virt-v2v: warning: manifest has a checksum for non-existent file 
RHEL7_3_042218_extra-disk1.vmdk (ignored)
[   4.3] Creating an overlay to protect the source from being modified
[   4.7] Initializing the target -o null
[   4.7] Opening the overlay
[  74.3] Inspecting the overlay
[  82.1] Checking for sufficient free disk space in the guest
[  82.1] Estimating space required on target for each disk
[  82.1] Converting Red Hat Enterprise Linux Client 7.0 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 149.4] Mapping filesystem data to avoid copying unused and blank areas
[ 150.5] Closing the overlay
[ 151.3] Checking if the guest needs BIOS or UEFI to boot
[ 151.3] Assigning disks to buses
[ 151.3] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw)
    (100.00/100%)
[ 171.4] Creating output metadata
[ 171.4] Finishing off

Result:
   virt-v2v warning shows correct during conversion and conversion could be finished successfully

Comment 20 mxie@redhat.com 2018-06-05 03:09:59 UTC
Verify the bug with builds:
virt-v2v-1.38.2-3.el7.x86_64
libguestfs-1.38.2-3.el7.x86_64
libvirt-4.3.0-1.el7.x86_64
qemu-kvm-rhev-2.12.0-3.el7.x86_64


Steps:
1.Prepare a ova file which has missing file mentioned in .mf
1.1 # tar -tf RHEL7_3_042218_extra.ova
RHEL7_3_042218_extra.ovf
RHEL7_3_042218_extra.mf
RHEL7_3_042218_extra-disk1.vmdk.000000000

1.2 # cat RHEL7_3_042218_extra.mf
SHA1(RHEL7_3_042218_extra.ovf)= bf626b725f3bd251cc37771dd6ce48c9915f00c3
SHA1(RHEL7_3_042218_extra-disk1.vmdk.000000000)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c
SHA1(RHEL7_3_042218_extra-disk1.vmdk)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c

2.Convert the ova by virt-v2v
# virt-v2v -i ova RHEL7_3_042218_extra.ova -o rhv -os 10.66.144.40:/home/nfs_export -of qcow2
[   0.0] Opening the source -i ova RHEL7_3_042218_extra.ova
virt-v2v: warning: manifest has a checksum for non-existent file 
RHEL7_3_042218_extra-disk1.vmdk (ignored)
[  10.9] Creating an overlay to protect the source from being modified
[  12.0] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[  12.7] Opening the overlay
[  23.4] Inspecting the overlay
[  36.2] Checking for sufficient free disk space in the guest
[  36.2] Estimating space required on target for each disk
[  36.2] Converting Red Hat Enterprise Linux Client 7.0 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 121.3] Mapping filesystem data to avoid copying unused and blank areas
[ 123.5] Closing the overlay
[ 124.5] Checking if the guest needs BIOS or UEFI to boot
[ 124.5] Assigning disks to buses
[ 124.5] Copying disk 1/1 to /tmp/v2v.jt81DN/ea9cb06f-8bf9-4fc8-a247-478e754d898a/images/ee8c82f5-1d05-4785-9d93-1aa18a665de7/43db4974-aa9f-4b54-b787-dcd0a867efc3 (qcow2)
    (100.00/100%)
[ 174.5] Creating output metadata
[ 174.6] Finishing off

3.Import guest from export domain to data domain, guest could be power on and boot into os normally

Result:
   Warning shows correct during v2v conversion and conversion could be finished successfully, so move the bug from ON_QA to VERIFIED

Comment 22 errata-xmlrpc 2018-10-30 07:45:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3021


Note You need to log in before you can comment on or make changes to this bug.