Bug 1402301

Summary: Improve OVA import compatibility
Product: Red Hat Enterprise Linux 7 Reporter: tingting zheng <tzheng>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact: Yehuda Zimmerman <yzimmerm>
Priority: high    
Version: 7.4CC: ahadas, akostadi, bugs, fabrice.bacchella, jsuchane, juzhou, michal.skrivanek, mkalinin, mtessun, mxie, mzhan, nsimsolo, pasik, ptoscano, rjones, shavivi, snagar, tgolembi, tzheng, xiaodwan, yzimmerm
Target Milestone: rcKeywords: ZStream
Target Release: 7.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: libguestfs-1.36.1-1.el7 Doc Type: Bug Fix
Doc Text:
*virt-v2v* imports OVAs that do not have the "<ovf:Name>" attribute Previously, the *virt-v2v* utility rejected the import of Open Virtual Appliances (OVAs) without the "<ovf:Name>" attribute. As a consequence, the *virt-v2v* utility did not import OVAs exported by Amazon Web Services (AWS). In this release, if the "<ovf:Name>" attribute is missing, *virt-v2v* uses the base name of the disk image file as the name of the virtual machine. As a result, the *virt-v2v* utility now imports OVAs exported by AWS.
Story Points: ---
Clone Of: 1371843
: 1408124 (view as bug list) Environment:
Last Closed: 2017-08-01 22:11:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1359086, 1371843    
Bug Blocks: 1395265, 1401077, 1401400, 1408124    

Description tingting zheng 2016-12-07 08:37:28 UTC
This bug also need patch from virt-v2v side,so I cloned this bug.

+++ This bug was initially created as a clone of Bug #1371843 +++

For it's AWS Storage Gateway, Amazon provides it's VM gateway as an OVA. But ovirt can't import it using the "import virtual machines wizard", it fails with:

File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 544, in _handle_request
    res = method(**params)
  File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 195, in _dynamicMethod
    result = fn(*methodArgs)
  File "/usr/share/vdsm/API.py", line 1427, in getExternalVmFromOva
    return v2v.get_ova_info(ova_path)
  File "/usr/lib/python2.7/site-packages/vdsm/v2v.py", line 197, in get_ova_info
    _add_general_ovf_info(vm, root, ns)
  File "/usr/lib/python2.7/site-packages/vdsm/v2v.py", line 1060, in _add_general_ovf_info
    raise V2VError('Error parsing ovf information: no ovf:Name')
V2VError: Error parsing ovf information: no ovf:Name

You can test it, the VM is easy to get from:
https://eu-central-1.console.aws.amazon.com/storagegateway/create

--- Additional comment from Arik on 2016-08-31 05:37:21 EDT ---

(In reply to Fabrice Bacchella from comment #0)
The current implementation of import OVAs is targeted for (and therefore tested with) OVAs that represent VMs exported from VMWare. This needs to be enhanced in order to support OVAs from other sources, like Amazon.

--- Additional comment from Fabrice Bacchella on 2016-08-31 05:46:29 EDT ---

OVA and OVF are open standard specified by the DMTF. Ever you support it whatever the source is coming from, or you have a very poor and incomplete implementation that happens to support random instances.

Don't try to nitpick about where it's coming from.

--- Additional comment from Yaniv Kaul on 2016-09-01 03:02:20 EDT ---

(In reply to Fabrice Bacchella from comment #2)
> OVA and OVF are open standard specified by the DMTF. Ever you support it
> whatever the source is coming from, or you have a very poor and incomplete
> implementation that happens to support random instances.

This is the thing about it being an 'open' standard - everyone can write anything there (specifically, in the XML). And in fact, everyone does.
For example, VMware does not support our OVF - and no wonder - they have no idea what virtio is.

Same goes for Amazon.

> 
> Don't try to nitpick about where it's coming from.

Hope the above explains why it's not a nitpick.

--- Additional comment from Fabrice Bacchella on 2016-09-01 03:23:06 EDT ---

I'm not talking about hardware compatibility problem, like a vmdk versus other format, or a strange network card. The problem occurs when parsing the ovf xml.

The VM I tried to import is made specifically to run in vmware. So it's a "vmware compatible" ova. That make it's failure in ovirt even more strange.

--- Additional comment from Yaniv Kaul on 2016-09-04 06:12:16 EDT ---

I have the OVA downloaded. If anyone wants to take a look, ping me.

--- Additional comment from Arik on 2016-09-04 10:20:25 EDT ---

The OVF in the given OVA differs from OVFs that are generated by vSphere in:
1. There is no <Name> tag under <VirtualSystem>
2. The content of <rasd:HostResource> does not have the "ovf:" prefix
By adding these things, I managed to convert the given OVA.

Shahar,
1. this should be added to the code you have added in virt-v2v, right?
2. I guess we should not count on having the name of the VM inside the OVA. IMO the whole import dialog should behave differently for OVA files: when choosing OVA as a source, the user needs to specify the host and the path as today and if they are valid to go directly to the second dialog (and there we already verify that the VM name is not empty).

Fabrice, in the meantime, I suggest to extract the file (it is just a tar file), do the changes mentioned above manually and pack it again. Then you'll be able to import this VM (I also started the VM and got to the login screen).

--- Additional comment from Fabrice Bacchella on 2016-09-05 05:20:13 EDT ---

I made your requested modification to the OVF and it works. Thanks.

--- Additional comment from Shahar Havivi on 2016-09-05 05:39:26 EDT ---

We may need to patch vdsm and virt-v2v as well

--- Additional comment from Yaniv Kaul on 2016-09-05 05:45:49 EDT ---

(In reply to Arik from comment #6)
> The OVF in the given OVA differs from OVFs that are generated by vSphere in:
> 1. There is no <Name> tag under <VirtualSystem>
> 2. The content of <rasd:HostResource> does not have the "ovf:" prefix
> By adding these things, I managed to convert the given OVA.

What does the standard say? (Although if VMware knows how to import these, regardless of standard conformance or not, we should adapt...)

--- Additional comment from Shahar Havivi on 2016-09-05 06:25:25 EDT ---

(In reply to Yaniv Kaul from comment #9)
> (In reply to Arik from comment #6)
> > The OVF in the given OVA differs from OVFs that are generated by vSphere in:
> > 1. There is no <Name> tag under <VirtualSystem>
> > 2. The content of <rasd:HostResource> does not have the "ovf:" prefix
> > By adding these things, I managed to convert the given OVA.
> 
> What does the standard say? (Although if VMware knows how to import these,
> regardless of standard conformance or not, we should adapt...)
I don't think that ovf:name is in the standard,
I found it in the network section:
http://www.dmtf.org/sites/default/files/standards/documents/DSP0243_2.1.0.pdf

In our case VMware does add it in the ova but AWS not...

--- Additional comment from Tomáš Golembiovský on 2016-09-06 07:39:35 EDT ---

I've also looked at the specification referenced by Shahar[1] to make sure whether we're just fixing our mess or somebody else's mess.

(In reply to Arik from comment #6)
> The OVF in the given OVA differs from OVFs that are generated by vSphere in:
> 1. There is no <Name> tag under <VirtualSystem>

The <Name> element is not mentioned anywhere (except examples), but looking at the schema file[2] it says it's optional.

> 2. The content of <rasd:HostResource> does not have the "ovf:" prefix

This is however a different story. Specification clearly states the value is either "ovf:/file/<id>" or "ovf:/disk/<id>". So it has to include the "ovf:" prefix. See Table 3 on page 21.


[1] http://www.dmtf.org/sites/default/files/standards/documents/DSP0243_2.1.0.pdf
[2] http://schemas.dmtf.org/ovf/envelope/2/dsp8023.xsd

--- Additional comment from Shahar Havivi on 2016-09-08 04:41:33 EDT ---

Patch send to virt-v2v:
https://www.redhat.com/archives/libguestfs/2016-September/msg00030.html

--- Additional comment from Richard W.M. Jones on 2016-09-08 05:32:26 EDT ---

(In reply to Arik from comment #6)
> 2. I guess we should not count on having the name of the VM inside the OVA.
> IMO the whole import dialog should behave differently for OVA files: when
> choosing OVA as a source, the user needs to specify the host and the path as
> today and if they are valid to go directly to the second dialog (and there
> we already verify that the VM name is not empty).

You might want an optional name field, which would translate to
passing '-on name' to virt-v2v (or not).  With Shahar's patch,
every Amazon VM will be mapped to the same name "default".

--- Additional comment from Shahar Havivi on 2016-09-08 06:50:40 EDT ---

> You might want an optional name field, which would translate to
> passing '-on name' to virt-v2v (or not).  With Shahar's patch,
> every Amazon VM will be mapped to the same name "default".
it is true only for conversion that will run directly virt-v2v.
vdsm is naming each vm with 'default' if the Name is not exists (current patch), and in oVirt engine you are able to name the VM regardless the ovf:name tag.

--- Additional comment from  on 2016-12-02 12:58:55 EST ---

This bug has been cloned to the                                 z-stream bug #1401077

--- Additional comment from tingting zheng on 2016-12-06 21:46:09 EST ---

Refer to comment 8,the bug needs patch from virt-v2v side,I didn't see any clone for virt-v2v bug,need I clone this bug to virt-v2v?

btw,I saw this bug has been cloned to ovirt 4.0 ZStream,if some patches are needed from virt-v2v,whether there will be a new virt-v2v ZStream build?thanks.

--- Additional comment from Richard W.M. Jones on 2016-12-07 03:16:00 EST ---

Upstream patches:

1ae4252c93c685cb8561b55c1231502b37212b5a
312379c8a61cd4a16aa7c80c1adda3081e601d43 (possibly)
98619f9c26076d034574b8822dfcc6799741ed96

--- Additional comment from Richard W.M. Jones on 2016-12-07 03:20:52 EST ---

(In reply to tingting zheng from comment #16)
> Refer to comment 8,the bug needs patch from virt-v2v side,I didn't see any
> clone for virt-v2v bug,need I clone this bug to virt-v2v?
> 
> btw,I saw this bug has been cloned to ovirt 4.0 ZStream,if some patches are
> needed from virt-v2v,whether there will be a new virt-v2v ZStream
> build?thanks.

Yes I think we will need 7.4 and 7.3 z-stream bugs against
libguestfs, and I don't think those bugs exist now.  Can you
create them?

--- Additional comment from Richard W.M. Jones on 2016-12-07 03:21:56 EST ---

(In reply to Richard W.M. Jones from comment #17)
> Upstream patches:
> 
> 1ae4252c93c685cb8561b55c1231502b37212b5a
> 312379c8a61cd4a16aa7c80c1adda3081e601d43 (possibly)
> 98619f9c26076d034574b8822dfcc6799741ed96

Sorry, those are the libguestfs commits.  Tingting will
create a new bug to track inclusion of those fixes in
RHEL 7.3-z.

Comment 1 Richard W.M. Jones 2016-12-07 10:44:01 UTC
Upstream patches:

1ae4252c93c685cb8561b55c1231502b37212b5a
312379c8a61cd4a16aa7c80c1adda3081e601d43 (possibly)
98619f9c26076d034574b8822dfcc6799741ed96

Comment 6 mxie@redhat.com 2017-04-12 05:08:49 UTC
Verify the bug with below builds:
virt-v2v-1.36.3-2.el7.x86_64
libguestfs-1.36.3-2.el7.x86_64
libvirt-3.2.0-2.el7.x86_64
qemu-kvm-rhev-2.8.0-6.el7.x86_64
libguestfs-winsupport-7.2-2.el7.x86_64
virtio-win-1.9.0-3.el7.noarch

Steps:
Scenario1:
1.Download the AWS-storage-gateway.ova which is mentioned in comment0, uncompress this ova file to get the ovf file
#tar -xvf AWS-storage-gateway.ova

2.Export a linux guest "rhel7.2" to ova file from vmware, uncompress rhel7.2 ova file to get the ovf file

3.Edit rhel7.2.ovf file according to aws-storage-gateway.ovf, the main changes are
3.1 Delete <Name> tag under <VirtualSystem>
3.2 Delete prefix "ovf" at <rasd:HostResource>

4.Using virt-v2v to convert the rhel7.2 ova file which was edited at step3
# virt-v2v -i ova rhel7.2-aws -o rhv -os 10.73.131.93:/home/nfs_export -of qcow2 -n ovirtmgmt -b ovirtmgmt
[   0.0] Opening the source -i ova rhel7.2-aws
virt-v2v: warning: making OVA directory public readable to work around
libvirt bug https://bugzilla.redhat.com/1045069
virt-v2v: warning: could not parse ovf:Name from OVF document
[  15.0] Creating an overlay to protect the source from being modified
[  15.2] Initializing the target -o rhv -os 10.73.131.93:/home/nfs_export
[  16.0] Opening the overlay
[  27.3] Inspecting the overlay
[  39.3] Checking for sufficient free disk space in the guest
[  39.3] Estimating space required on target for each disk
[  39.3] Converting Red Hat Enterprise Linux Server 7.2 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 120.0] Mapping filesystem data to avoid copying unused and blank areas
[ 120.5] Closing the overlay
[ 120.8] Checking if the guest needs BIOS or UEFI to boot
[ 120.8] Assigning disks to buses
[ 120.8] Copying disk 1/1 to /tmp/v2v.fNsSrM/4f699f1d-7f77-4e73-a0d2-a3b73c1a0c29/images/b307cc38-0a6a-4c7d-9ba0-f8048002246d/d4652bf3-53e5-42cc-9136-b6d177694db6 (qcow2)
    (100.00/100%)
[ 606.5] Creating output metadata
[ 613.1] Finishing off

5.After finishing conversion, import the guest to data domain
6.Power on the guest and checkpoint are passed


Scenario2:
1.Export a windows guest "win2008r2" to a ova file from vmware and uncompress rhel7.2 ova file to get the ovf file

2.Edit win2008r2.ovf file according to aws-storage-gateway.ovf, the main changes are
2.1 Delete <Name> tag under <VirtualSystem>
2.2 Delete prefix "ovf" at <rasd:HostResource>

3.Using virt-v2v to convert the win2008r2 ova file which was edited
# virt-v2v -i ova win2008r2-aws -o rhv -os 10.73.131.93:/home/nfs_export -of raw -b ovirtmgmt -n ovirtmgmt
[   0.0] Opening the source -i ova win2008r2-aws
virt-v2v: warning: making OVA directory public readable to work around
libvirt bug https://bugzilla.redhat.com/1045069
virt-v2v: warning: could not parse ovf:Name from OVF document
[  42.4] Creating an overlay to protect the source from being modified
[  42.9] Initializing the target -o rhv -os 10.73.131.93:/home/nfs_export
[  43.1] Opening the overlay
[  48.7] Inspecting the overlay
[  50.3] Checking for sufficient free disk space in the guest
[  50.3] Estimating space required on target for each disk
[  50.3] Converting Windows Server 2008 R2 Standard to run on KVM
virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing.  
Firstboot scripts may conflict with PnP.
virt-v2v: This guest has virtio drivers installed.
[  51.3] Mapping filesystem data to avoid copying unused and blank areas
[  51.7] Closing the overlay
[  52.2] Checking if the guest needs BIOS or UEFI to boot
[  52.2] Assigning disks to buses
[  52.2] Copying disk 1/1 to /tmp/v2v.xRDnsl/4f699f1d-7f77-4e73-a0d2-a3b73c1a0c29/images/92c3725b-f9a5-4ddc-8329-1803c856e838/c061bb90-e87b-4fe2-8d02-1272a1406808 (raw)
    (100.00/100%)
[1197.1] Creating output metadata
[1197.2] Finishing off

4.After finishing conversion, import the guest to data domain

5.Power on the guest and checkpoint are passed


Scenario3:
1.Export a ova file from AWS EC2
#aws ec2 create-instance-export-task --instance-id i-0ed7a871968fb96ae --target-environment vmware --export-to-s3-task DiskImageFormat=VMDK,ContainerFormat=ova,S3Bucket=xx,S3Prefix=rhel6-ec2.ova
EXPORTTASK        export-i-fhdb9u4y        active
EXPORTTOS3TASK        ova        vmdk        xx        rhel6-ec2.ovaexport-i-fhdb9u4y.ova
INSTANCEEXPORTDETAILS        i-0ed7a871968fb96ae        vmware

2.Using virt-v2v convert aws ova to rhv
# virt-v2v -i ova rhel6-ec2.ovaexport-i-fhdb9u4y.ova -o rhv -os 10.73.131.93:/home/nfs_export -of qcow2 -n ovirtmgmt -b ovirtmgmt
[   0.0] Opening the source -i ova rhel6-ec2.ovaexport-i-fhdb9u4y.ova
[  15.6] Creating an overlay to protect the source from being modified
[  15.7] Initializing the target -o rhv -os 10.73.131.93:/home/nfs_export
[  38.3] Opening the overlay
[  42.5] Inspecting the overlay
[  47.7] Checking for sufficient free disk space in the guest
[  47.7] Estimating space required on target for each disk
[  47.7] Converting Red Hat Enterprise Linux Server release 6.2 Beta (Santiago) to run on KVM
virt-v2v: warning: /files/boot/grub/device.map/hd0 references unknown
device "vda".  You may have to fix this entry manually after conversion.
virt-v2v: warning: /files/etc/sysconfig/grub/boot references unknown device
"vda".  You may have to fix this entry manually after conversion.
virt-v2v: This guest has virtio drivers installed.
[  77.8] Mapping filesystem data to avoid copying unused and blank areas
[  77.9] Closing the overlay
[  78.1] Checking if the guest needs BIOS or UEFI to boot
[  78.1] Assigning disks to buses
[  78.1] Copying disk 1/1 to /tmp/v2v.YO8qji/4f699f1d-7f77-4e73-a0d2-a3b73c1a0c29/images/cde3d0f0-2c26-4214-acd4-ff78aaa89d3c/6092a1c4-0796-401a-85e6-11c03ccd5591 (qcow2)
    (100.00/100%)
[ 426.8] Creating output metadata
[ 426.9] Finishing off

3.After finishing conversion, import the guest to data domain

4.Power on the guest and checkpoint are passed




Scenario4:
1.Prepare a ova which was exported from Eaton Corporation and using virt-v2v convert this ova to rhv
# virt-v2v -i ova IPM-1.53.158.VA_OVF10.ova -o rhv -os 10.73.131.93:/home/nfs_export -of qcow2
[   0.0] Opening the source -i ova IPM-1.53.158.VA_OVF10.ova
virt-v2v: warning: making OVA directory public readable to work around 
libvirt bug https://bugzilla.redhat.com/1045069
virt-v2v: warning: could not parse ovf:Name from OVF document
[   2.1] Creating an overlay to protect the source from being modified
[   2.3] Initializing the target -o rhv -os 10.73.131.93:/home/nfs_export
[   2.6] Opening the overlay
[   6.8] Inspecting the overlay
[  11.5] Checking for sufficient free disk space in the guest
[  11.5] Estimating space required on target for each disk
[  11.5] Converting CentOS release 6.8 (Final) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  30.8] Mapping filesystem data to avoid copying unused and blank areas
[  32.4] Closing the overlay
[  33.6] Checking if the guest needs BIOS or UEFI to boot
[  33.6] Assigning disks to buses
[  33.6] Copying disk 1/1 to /tmp/v2v.053f3s/4f699f1d-7f77-4e73-a0d2-a3b73c1a0c29/images/d6cd9438-277b-4795-be87-b0e8bf7be8d6/fe3565d8-b60b-4250-b549-3f5bed06e3f2 (qcow2)
    (100.00/100%)
[ 157.7] Creating output metadata
[ 157.9] Finishing off

2.After finishing conversion, import the guest to data domain

4.Could power on the guest successfully



Hi Pino,
  For scenario1,2,4. there will be virt-v2v warning : could not parse ovf:Name from OVF document, do you think it is necessary to hide this warning during v2v conversion?
  For scenario3, there is no warning about making OVA directory public readable to work around when v2v convert a AWS ova, but this warning will be shown when convert a vmware ova, why? Is it necessary to hide this warning when v2v converting a vmware ova conversion?

Comment 7 Richard W.M. Jones 2017-04-12 07:16:03 UTC
(In reply to mxie from comment #6)
>   For scenario1,2,4. there will be virt-v2v warning : could not parse
> ovf:Name from OVF document, do you think it is necessary to hide this
> warning during v2v conversion?

No, these warnings are not errors, but they describe anomalous
conditions which virt-v2v is working around which the user may
need to be aware of.

>   For scenario3, there is no warning about making OVA directory public
> readable to work around when v2v convert a AWS ova, but this warning will be
> shown when convert a vmware ova, why? Is it necessary to hide this warning
> when v2v converting a vmware ova conversion?

Again, the warnings aren't errors, they are there to warn the user
about something which the program is doing which the user might
need to be aware of.

In this case when the libvirt bug (bug 1045069) is fixed we will
remove the warning anyway.

Comment 8 mxie@redhat.com 2017-04-12 08:29:18 UTC
According to comment6 and comment7, move this bug from ON_QA to VERIFIED

Comment 12 errata-xmlrpc 2017-08-01 22:11:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2023

Comment 13 Aleksandar Kostadinov 2017-09-24 17:29:06 UTC
And how about `pnp_wait.exe`? Where should on install it from?

Comment 14 Richard W.M. Jones 2017-09-24 18:16:39 UTC
At the moment you have to build it yourself.  The source is here:

  https://github.com/rwmjones/rhsrvany

virt-v2v will find the binary if it is called

  /usr/share/virt-tools/pnp_wait.exe

(you can change the path part but not the binary name
by setting $VIRT_TOOLS_DATA_DIR before running virt-v2v).