RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1570407 - virt-v2v -i ova gives checksum error when missing files are mentioned in .mf
Summary: virt-v2v -i ova gives checksum error when missing files are mentioned in .mf
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On: 1551055
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-22 16:07 UTC by Richard W.M. Jones
Modified: 2018-10-30 07:46 UTC (History)
6 users (show)

Fixed In Version: libguestfs-1.38.1-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-30 07:45:35 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
bz1570407-reproducer.ova (20.00 KB, application/x-tar)
2018-04-22 16:08 UTC, Richard W.M. Jones
no flags Details
RHEL7_3_042218_extra.ovf (12.76 KB, text/plain)
2018-04-24 07:49 UTC, Richard W.M. Jones
no flags Details
RHEL7_3_042218_extra.mf (243 bytes, text/plain)
2018-04-24 07:49 UTC, Richard W.M. Jones
no flags Details
v2v-1.38.1-1.log (3.12 KB, text/plain)
2018-05-10 07:09 UTC, mxie@redhat.com
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:3021 0 None None None 2018-10-30 07:46:21 UTC

Description Richard W.M. Jones 2018-04-22 16:07:15 UTC
Description of problem:

(Bug reported by Nisim Simsolo)

$ virt-v2v -i ova ~/RHEL7_3_042218_extra.ova -o null
[   0.0] Opening the source -i ova /home/rjones/RHEL7_3_042218_extra.ova
tar: RHEL7_3_042218_extra-disk1.vmdk: Not found in archive
tar: Exiting with failure status due to previous errors
virt-v2v: error: checksum of disk RHEL7_3_042218_extra-disk1.vmdk does not 
match manifest /var/tmp/ova.isX3xH/RHEL7_3_042218_extra.mf (actual 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
da39a3ee5e6b4b0d3255bfef95601890afd80709, expected 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
ca3727b9c701ffe2552e4809b3ecf9d85a82873c)

The .ova file is peculiar.  The files it contains are:

$ tar tf ~/RHEL7_3_042218_extra.ova
RHEL7_3_042218_extra.ovf
RHEL7_3_042218_extra.mf
RHEL7_3_042218_extra-disk1.vmdk.000000000

but the .mf file mentions a non-existent file: 

$ cat RHEL7_3_042218_extra.mf
SHA1(RHEL7_3_042218_extra.ovf)= bf626b725f3bd251cc37771dd6ce48c9915f00c3
SHA1(RHEL7_3_042218_extra-disk1.vmdk.000000000)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c
SHA1(RHEL7_3_042218_extra-disk1.vmdk)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c

Version-Release number of selected component (if applicable):

virt-v2v-1.36.10-6.6.rhvpreview.el7ev.x86_64
(also happens with virt-v2v-1.38.0-1.fc28.x86_64)
VMware Version 6.0.0 Build 2594327

How reproducible:

100%

Steps to Reproduce:
1. Use the synthetic OVA file attached to replicate the bug.

Comment 2 Richard W.M. Jones 2018-04-22 16:08:52 UTC
Created attachment 1425375 [details]
bz1570407-reproducer.ova

To reproduce the bug, download the attachment and run:

$ virt-v2v -i ova bz1570407-reproducer.ova -o null
[   0.0] Opening the source -i ova bz1570407-reproducer.ova
tar: RHEL7_3_042218_extra-disk1.vmdk: Not found in archive
tar: Exiting with failure status due to previous errors
virt-v2v: error: checksum of disk RHEL7_3_042218_extra-disk1.vmdk does not 
match manifest /var/tmp/ova.2rqgzX/RHEL7_3_042218_extra.mf (actual 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
da39a3ee5e6b4b0d3255bfef95601890afd80709, expected 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
edb35db6af8384832ac27d63773ee0bb2a15e0bd)

Comment 3 Richard W.M. Jones 2018-04-23 10:08:18 UTC
I posted some preliminary work on this:

https://www.redhat.com/archives/libguestfs/2018-April/msg00163.html

However this is not a full fix.  It will require major work on the
-i ova input mode to deal with these files (which IMHO are basically
broken).

Comment 4 Pino Toscano 2018-04-24 06:58:25 UTC
Is it a real-world OVA? At least from a cursory glance it'd seem generated by VMware, is it correct?

Comment 5 Richard W.M. Jones 2018-04-24 07:48:27 UTC
Yes it is a real world OVA (generated by VMware Version 6.0.0 Build 2594327).

I believe it has something to do with snapshots.

The file list is:

RHEL7_3_042218_extra.ovf
RHEL7_3_042218_extra.mf
RHEL7_3_042218_extra-disk1.vmdk.000000000

I will attach the OVF and MF files.

Comment 6 Richard W.M. Jones 2018-04-24 07:49:17 UTC
Created attachment 1425858 [details]
RHEL7_3_042218_extra.ovf

Comment 7 Richard W.M. Jones 2018-04-24 07:49:39 UTC
Created attachment 1425859 [details]
RHEL7_3_042218_extra.mf

Comment 8 Richard W.M. Jones 2018-04-24 07:52:05 UTC
You can see the original problem from the bug report because the
.mf file references a non-existent file:

SHA1(RHEL7_3_042218_extra-disk1.vmdk)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c

My patch series solve this problem.

However there's a second problem in the .ovf file which also
references a non-existent file:

  <References>
    <File ovf:chunkSize="7516192768" ovf:href="RHEL7_3_042218_extra-disk1.vmdk" ovf:id="file1" ovf:size="721755648"/>
  </References>

and fixing this involves some deeper surgery on v2v/input_ova.ml
(which to be fair we probably need to do at some time).

Comment 10 Richard W.M. Jones 2018-04-25 13:35:57 UTC
This should be a full fix:

https://www.redhat.com/archives/libguestfs/2018-April/msg00167.html

Using this I was able to convert the test image which
Nisim supplied to me privately.

Comment 11 Richard W.M. Jones 2018-04-27 09:14:54 UTC
Upstream in:

e5946165c v2v: -i ova: Handle OVAs containing snapshots (RHBZ#1570407).
80d6d9396 v2v: -i ova: Sanity check hrefs to ensure they stay inside the tarball.
e24771468 v2v: -i ova: Replace subdirectory function with clearer inline code.
075333ede common: mlstdutils: Inline unsafe ‘subdirectory’ function.
184dba598 v2v: -i ova: Factor out code for dealing with OVA files.
ece2c60c5 v2v: parse OVF: Export useful parse_disks function.
d1e95a30f v2v: Ignore miscellaneous tar messages when parsing tar for file locations.
112e3bc31 v2v: -i ova: Ignore non-existent files mentioned in *.mf.
190333b8b mltools: Checksums.verify_checksum{,s} returns an error type instead of throwing exception.

Comment 12 Pino Toscano 2018-05-02 11:22:36 UTC
Fixes backported in 1.38.1.

Comment 14 mxie@redhat.com 2018-05-10 07:08:56 UTC
I can reproduce the bug using attached ova with below builds:
virt-v2v-1.38.0-1.el7.x86_64
libguestfs-1.38.0-1.el7.x86_64

Reproduce steps:
1.# virt-v2v -i ova bz1570407-reproducer.ova -o null
[   0.0] Opening the source -i ova bz1570407-reproducer.ova
virt-v2v: warning: making OVA directory public readable to work around 
libvirt bug https://bugzilla.redhat.com/1045069
tar: RHEL7_3_042218_extra-disk1.vmdk: Not found in archive
tar: Exiting with failure status due to previous errors
virt-v2v: error: checksum of disk RHEL7_3_042218_extra-disk1.vmdk does not 
match manifest /var/tmp/ova.jZTuqW/RHEL7_3_042218_extra.mf (actual 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
da39a3ee5e6b4b0d3255bfef95601890afd80709, expected 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
edb35db6af8384832ac27d63773ee0bb2a15e0bd)

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


By the way, I can't export a ova like bug's attached one even if vmware 6.0 guest has snapshot so far.


Verify the bug with builds:
virt-v2v-1.38.1-1.el7.x86_64
libguestfs-1.38.1-1.el7.x86_64

Steps:
1.# virt-v2v -i ova bz1570407-reproducer.ova -o null
[   0.0] Opening the source -i ova bz1570407-reproducer.ova
virt-v2v: warning: making OVA directory public readable to work around 
libvirt bug https://bugzilla.redhat.com/1045069
virt-v2v: warning: manifest has a checksum for non-existent file 
RHEL7_3_042218_extra-disk1.vmdk (ignored)
[   0.4] Creating an overlay to protect the source from being modified
qemu-img: /var/tmp/v2vovlb05920.qcow2: Failed to read footer: Input/output error
Could not open backing image to determine size.
virt-v2v: error: qemu-img command failed, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


Hi rjones.

   I can see latest virt-v2v has improved the warning for non-existent file in mf,but v2v conversion is failed in my test (according to comment10, you could convert attached ova), could you please help to check?

Comment 15 mxie@redhat.com 2018-05-10 07:09:37 UTC
Created attachment 1434246 [details]
v2v-1.38.1-1.log

Comment 16 Pino Toscano 2018-05-10 09:14:25 UTC
> Verify the bug with builds:
> virt-v2v-1.38.1-1.el7.x86_64
> libguestfs-1.38.1-1.el7.x86_64
> 
> Steps:
> 1.# virt-v2v -i ova bz1570407-reproducer.ova -o null
> [   0.0] Opening the source -i ova bz1570407-reproducer.ova
> virt-v2v: warning: making OVA directory public readable to work around 
> libvirt bug https://bugzilla.redhat.com/1045069
> virt-v2v: warning: manifest has a checksum for non-existent file 
> RHEL7_3_042218_extra-disk1.vmdk (ignored)
> [   0.4] Creating an overlay to protect the source from being modified
> qemu-img: /var/tmp/v2vovlb05920.qcow2: Failed to read footer: Input/output
> error
> Could not open backing image to determine size.
> virt-v2v: error: qemu-img command failed, see earlier errors

Can you please try as non-root user, or to move the .ova file where even non-root users can access it (e.g. /tmp, /var/tmp, etc)?

Comment 17 mxie@redhat.com 2018-05-10 10:21:44 UTC
Hi Pino,
   
    The conversion is still failed after moving ova file to /var/tmp and running command as non-root user

[mxie@hp-dl380eg8-02 home]$ virt-v2v -i ova bz1570407-reproducer.ova -o null
[   0.0] Opening the source -i ova bz1570407-reproducer.ova
virt-v2v: warning: manifest has a checksum for non-existent file 
RHEL7_3_042218_extra-disk1.vmdk (ignored)
[   0.4] Creating an overlay to protect the source from being modified
qemu-img: /var/tmp/v2vovl528738.qcow2: Failed to read footer: Input/output error
Could not open backing image to determine size.
virt-v2v: error: qemu-img command failed, see earlier errors

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Comment 18 Richard W.M. Jones 2018-05-16 09:39:12 UTC
(In reply to mxie from comment #17)
> Hi Pino,
>    
>     The conversion is still failed after moving ova file to /var/tmp and
> running command as non-root user
> 
> [mxie@hp-dl380eg8-02 home]$ virt-v2v -i ova bz1570407-reproducer.ova -o null
> [   0.0] Opening the source -i ova bz1570407-reproducer.ova
> virt-v2v: warning: manifest has a checksum for non-existent file 
> RHEL7_3_042218_extra-disk1.vmdk (ignored)

So please note here that bz1570407-reproducer.ova is not a complete,
working OVA.  It's basically something I just hacked up to
reproduce the bug.  If virt-v2v has been patched to fix the bug
then trying to actually convert bz1570407-reproducer.ova will fail.

I will send you a link to the actual OVA which caused the original
bug report (but we cannot redistribute it).

Comment 19 mxie@redhat.com 2018-05-17 10:00:50 UTC
Use the ova file provided by rjones to test the bug again


Reproduce the bug with builds
virt-v2v-1.38.0-1.el7.x86_64 
libguestfs-1.38.0-1.el7.x86_64 

Reproduce steps:
1.Prepare a ova file and there is missing file mentioned in .mf
# tar tf RHEL7_3_042218_extra.ova
RHEL7_3_042218_extra.ovf
RHEL7_3_042218_extra.mf
RHEL7_3_042218_extra-disk1.vmdk.000000000

# cat RHEL7_3_042218_extra.mf
SHA1(RHEL7_3_042218_extra.ovf)= bf626b725f3bd251cc37771dd6ce48c9915f00c3
SHA1(RHEL7_3_042218_extra-disk1.vmdk.000000000)=
ca3727b9c701ffe2552e4809b3ecf9d85a82873c
SHA1(RHEL7_3_042218_extra-disk1.vmdk)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c

2.Use virt-v2v to convert this ova file 
# virt-v2v -i ova RHEL7_3_042218_extra.ova -o null
[   0.0] Opening the source -i ova RHEL7_3_042218_extra.ova
tar: RHEL7_3_042218_extra-disk1.vmdk: Not found in archive
tar: Exiting with failure status due to previous errors
virt-v2v: error: checksum of disk RHEL7_3_042218_extra-disk1.vmdk does not 
match manifest /var/tmp/ova.T82C33/RHEL7_3_042218_extra.mf (actual 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
da39a3ee5e6b4b0d3255bfef95601890afd80709, expected 
SHA1(RHEL7_3_042218_extra-disk1.vmdk) = 
ca3727b9c701ffe2552e4809b3ecf9d85a82873c)

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]


Verify the bug with builds:
virt-v2v-1.38.2-1.el7.x86_64
libguestfs-1.38.2-1.el7.x86_64

Steps:
1.Convert the ova file which has missing file mentioned in .mf by virt-v2v
# virt-v2v -i ova RHEL7_3_042218_extra.ova -o null
[   0.0] Opening the source -i ova RHEL7_3_042218_extra.ova
virt-v2v: warning: manifest has a checksum for non-existent file 
RHEL7_3_042218_extra-disk1.vmdk (ignored)
[   4.3] Creating an overlay to protect the source from being modified
[   4.7] Initializing the target -o null
[   4.7] Opening the overlay
[  74.3] Inspecting the overlay
[  82.1] Checking for sufficient free disk space in the guest
[  82.1] Estimating space required on target for each disk
[  82.1] Converting Red Hat Enterprise Linux Client 7.0 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 149.4] Mapping filesystem data to avoid copying unused and blank areas
[ 150.5] Closing the overlay
[ 151.3] Checking if the guest needs BIOS or UEFI to boot
[ 151.3] Assigning disks to buses
[ 151.3] Copying disk 1/1 to qemu URI json:{ "file.driver": "null-co", "file.size": "1E" } (raw)
    (100.00/100%)
[ 171.4] Creating output metadata
[ 171.4] Finishing off

Result:
   virt-v2v warning shows correct during conversion and conversion could be finished successfully

Comment 20 mxie@redhat.com 2018-06-05 03:09:59 UTC
Verify the bug with builds:
virt-v2v-1.38.2-3.el7.x86_64
libguestfs-1.38.2-3.el7.x86_64
libvirt-4.3.0-1.el7.x86_64
qemu-kvm-rhev-2.12.0-3.el7.x86_64


Steps:
1.Prepare a ova file which has missing file mentioned in .mf
1.1 # tar -tf RHEL7_3_042218_extra.ova
RHEL7_3_042218_extra.ovf
RHEL7_3_042218_extra.mf
RHEL7_3_042218_extra-disk1.vmdk.000000000

1.2 # cat RHEL7_3_042218_extra.mf
SHA1(RHEL7_3_042218_extra.ovf)= bf626b725f3bd251cc37771dd6ce48c9915f00c3
SHA1(RHEL7_3_042218_extra-disk1.vmdk.000000000)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c
SHA1(RHEL7_3_042218_extra-disk1.vmdk)= ca3727b9c701ffe2552e4809b3ecf9d85a82873c

2.Convert the ova by virt-v2v
# virt-v2v -i ova RHEL7_3_042218_extra.ova -o rhv -os 10.66.144.40:/home/nfs_export -of qcow2
[   0.0] Opening the source -i ova RHEL7_3_042218_extra.ova
virt-v2v: warning: manifest has a checksum for non-existent file 
RHEL7_3_042218_extra-disk1.vmdk (ignored)
[  10.9] Creating an overlay to protect the source from being modified
[  12.0] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[  12.7] Opening the overlay
[  23.4] Inspecting the overlay
[  36.2] Checking for sufficient free disk space in the guest
[  36.2] Estimating space required on target for each disk
[  36.2] Converting Red Hat Enterprise Linux Client 7.0 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 121.3] Mapping filesystem data to avoid copying unused and blank areas
[ 123.5] Closing the overlay
[ 124.5] Checking if the guest needs BIOS or UEFI to boot
[ 124.5] Assigning disks to buses
[ 124.5] Copying disk 1/1 to /tmp/v2v.jt81DN/ea9cb06f-8bf9-4fc8-a247-478e754d898a/images/ee8c82f5-1d05-4785-9d93-1aa18a665de7/43db4974-aa9f-4b54-b787-dcd0a867efc3 (qcow2)
    (100.00/100%)
[ 174.5] Creating output metadata
[ 174.6] Finishing off

3.Import guest from export domain to data domain, guest could be power on and boot into os normally

Result:
   Warning shows correct during v2v conversion and conversion could be finished successfully, so move the bug from ON_QA to VERIFIED

Comment 22 errata-xmlrpc 2018-10-30 07:45:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3021


Note You need to log in before you can comment on or make changes to this bug.