Bug 1594515 - [v2v] [RFE] Migrating VM with multiple DPG's fail to get assigned with correct NICs on RHV
Summary: [v2v] [RFE] Migrating VM with multiple DPG's fail to get assigned with correc...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: V2V
Version: 5.9.3
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: GA
: 5.10.0
Assignee: Brett Thurber
QA Contact: Shveta
Red Hat CloudForms Documentation
URL:
Whiteboard: v2v
Depends On: 1607274
Blocks: 1601090 1610547
TreeView+ depends on / blocked
 
Reported: 2018-06-23 21:26 UTC by Kedar Kulkarni
Modified: 2019-02-11 14:06 UTC (History)
18 users (show)

Fixed In Version: 5.10.0.5
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1610547 (view as bug list)
Environment:
Last Closed: 2019-02-11 14:06:13 UTC
Category: ---
Cloudforms Team: V2V
Target Upstream Version:


Attachments (Terms of Use)
virt-v2v logs (deleted)
2018-06-23 21:26 UTC, Kedar Kulkarni
no flags Details
virt-v2v logs for migration (6.47 MB, application/octet-stream)
2018-06-23 21:27 UTC, Kedar Kulkarni
no flags Details
virt-v2v wrapper logs (22.84 KB, text/plain)
2018-06-25 15:24 UTC, Kedar Kulkarni
no flags Details
UI mappings (147.40 KB, image/png)
2018-06-29 05:45 UTC, Brett Thurber
no flags Details
RHV mappings post migration (128.16 KB, image/png)
2018-06-29 05:45 UTC, Brett Thurber
no flags Details
Source VM network mappings (190.56 KB, image/png)
2018-06-29 06:43 UTC, Brett Thurber
no flags Details
Database-02 virt-v2v log (7.10 MB, application/octet-stream)
2018-06-29 07:09 UTC, Brett Thurber
no flags Details
post migration mappings for real networks (132.54 KB, image/png)
2018-06-29 21:14 UTC, Brett Thurber
no flags Details
post migration network in RHV (85.24 KB, image/png)
2019-01-21 17:31 UTC, Shveta
no flags Details


Links
System ID Priority Status Summary Last Updated
Github https://github.com/ManageIQ manageiq-content pull 351 None None None 2018-07-05 09:50:57 UTC
Github https://github.com/oVirt ovirt-ansible-v2v-conversion-host pull 6 None None None 2018-07-05 09:50:32 UTC
Red Hat Knowledge Base (Solution) 3637631 None None None 2018-10-03 08:07:07 UTC

Description Kedar Kulkarni 2018-06-23 21:26:37 UTC
Description of problem:
When I tried Migrating VM with multiple NICs, its NICs on RHV side are not correct as I would expect based on Infra Mapping used. 

Version-Release number of selected component (if applicable):
master.20180621230811_8e109d6


How reproducible:
100%

Steps to Reproduce:
1.Create a VM on VMware with 2 different NICs
2.Create infrastructure mapping which maps these NICs to two different networks on RHV
3.Migrate VM to RHV

Actual results:
its NICs on RHV side are not correct as I would expect based on Infra Mapping used. 

Expected results:
VM migration should honor mappings.

Additional info:

Comment 2 Kedar Kulkarni 2018-06-23 21:27:37 UTC
Created attachment 1454050 [details]
virt-v2v logs for migration

adding again since previous attachment failed.

Comment 3 Fabien Dupont 2018-06-25 15:09:16 UTC
Could you also attach the wrapper log, please? It will give us the options passed to virt-v2v.

Comment 4 Kedar Kulkarni 2018-06-25 15:24:18 UTC
Created attachment 1454389 [details]
virt-v2v wrapper logs

Comment 5 Fabien Dupont 2018-06-27 14:30:22 UTC
From the wrapper log file, the generated virt-v2v command looks correct:

/usr/bin/virt-v2v -v -x 'kkulkarn-v2v-fedora' \
  -of raw --bridge ovirtmgmt --root first -i libvirt \
  -ic 'vpx://administrator%40vsphere.local@env-nvc60-vcsa.cfme2.lab.eng.rdu2.redhat.com/Datacenter/Cluster/env-nvc60-h03.cfme2.lab.eng.rdu2.redhat.com?no_verify=1' \
  -it vddk -io vddk-libdir=/opt/vmware-vix-disklib-distrib \
  -io 'vddk-thumbprint=E6:67:CC:A7:5D:16:75:80:DF:41:CA:76:98:E6:2C:F2:62:6B:1F:81' \
  --password-file /tmp/tmp1Cu9Fk.v2v -o rhv-upload \
  -oc 'https://rhvm.v2v.bos.redhat.com/ovirt-engine/api' \
  -os 'V2V-NFS4-Data3' -op /tmp/tmpZWd63z.v2v \
  -oo rhv-cafile=/etc/pki/vdsm/certs/cacert.pem -oo 'rhv-cluster=V2V' \
  -oo rhv-direct -oo 'rhv-verifypeer=false' -oa sparse \
  --bridge 'DPortGroup:VM_Network' --bridge 'VM Network:ovirtmgmt'

As you can see, there are 3 --bridge options: 2 for the VM NICs and a default one in case another network would be involved (low probability). Could you test that command directly from the conversion host ? You will have to create password files for vSphere and RHV, and change the values of --password and -op options to match these files.

Comment 9 Richard W.M. Jones 2018-06-28 18:38:38 UTC
The wrapper generates the following network/bridge-related parameters:

>   --bridge ovirtmgmt \
>   --bridge 'DPortGroup:VM_Network' --bridge 'VM Network:ovirtmgmt'

That basically sets up a mapping table as follows:

    from                  to
-----------------------------------------
    bridge "DPortGroup"   VM_Network
    bridge "VM Network"   ovirtmgmt
    *                     ovirtmgmt

(http://libguestfs.org/virt-v2v.1.html#networks-and-bridges)

In the source guest we have two NICs identified as connected to bridges:

  NICs:
        Bridge "VM Network" mac: 00:50:56:9d:ba:44 [vmxnet3]
        Bridge "eth1" mac: 00:50:56:9d:6c:4a [vmxnet3]

The final OVF XML has:

    <Info>List of networks</Info>
    <!-- mapped from "VM Network" to "ovirtmgmt" -->
    <Network ovf:name='ovirtmgmt'/>
    <!-- mapped from "eth1" to "ovirtmgmt" -->
    <Network ovf:name='ovirtmgmt'/>
       ...
      <Item>
        <rasd:InstanceId>a01fea1d-7098-4fa3-8682-cf12c012f49a</rasd:InstanceId>
        <rasd:Caption>Ethernet adapter on ovirtmgmt</rasd:Caption>
        <rasd:ResourceType>10</rasd:ResourceType>
        <rasd:ResourceSubType>3</rasd:ResourceSubType>
        <Type>interface</Type>
        <rasd:Connection>ovirtmgmt</rasd:Connection>
        <rasd:Name>eth0</rasd:Name>
        <rasd:MACAddress>00:50:56:9d:ba:44</rasd:MACAddress>
      </Item>
      <Item>
        <rasd:InstanceId>4fc57897-558e-428f-84fe-b40c064a61b2</rasd:InstanceId>
        <rasd:Caption>Ethernet adapter on ovirtmgmt</rasd:Caption>
        <rasd:ResourceType>10</rasd:ResourceType>
        <rasd:ResourceSubType>3</rasd:ResourceSubType>
        <Type>interface</Type>
        <rasd:Connection>ovirtmgmt</rasd:Connection>
        <rasd:Name>eth1</rasd:Name>
        <rasd:MACAddress>00:50:56:9d:6c:4a</rasd:MACAddress>
      </Item>

I believe that virt-v2v is doing the mapping correctly (I'm ignoring any
further mapping that oVirt itself might do).  Or at least virt-v2v is
doing the mapping that was requested, but the requested mapping looks
a bit strange to me.  What is "DPortGroup" for instance?

Yes, again, v2v's handling of networks and bridges is crap, but in this case
I don't think virt-v2v is doing anything wrong.

Comment 11 Brett Thurber 2018-06-29 05:45:18 UTC
Created attachment 1455423 [details]
UI mappings

Comment 12 Brett Thurber 2018-06-29 05:45:55 UTC
Created attachment 1455424 [details]
RHV mappings post migration

Comment 13 Michal Skrivanek 2018-06-29 06:38:34 UTC
In addition to commetn #9 - the source VM seem to have these two NICs, at least that's how libvirt understands the configuration:

    <interface type='bridge'>
      <mac address='00:50:56:9d:ba:44'/>
      <source bridge='VM Network'/>
      <model type='vmxnet3'/>
    </interface>
    <interface type='bridge'>
      <mac address='00:50:56:9d:6c:4a'/>
      <source bridge=''/>
      <model type='vmxnet3'/>
    </interface>

so they both indeed map to the ovirtmgmt. How does it look on vmware side? What interface is supposed to be on DPortGroup?

Comment 14 Brett Thurber 2018-06-29 06:42:44 UTC
Michal, Attached is the source VM screen shot in my testing.

Comment 15 Brett Thurber 2018-06-29 06:43:16 UTC
Created attachment 1455442 [details]
Source VM network mappings

Comment 16 Michal Skrivanek 2018-06-29 06:46:18 UTC
libvirt just doesnt' see those. The picture says "disconnected" - could it mean anything?

Comment 17 Brett Thurber 2018-06-29 06:50:14 UTC
(In reply to Michal Skrivanek from comment #16)
> libvirt just doesnt' see those. The picture says "disconnected" - could it
> mean anything?

Just means the VM is powered off.  When the VM is powered on they are connected which was the case during my test migration.

Comment 18 Brett Thurber 2018-06-29 07:09:55 UTC
Created attachment 1455445 [details]
Database-02 virt-v2v log

Comment 19 Michal Skrivanek 2018-06-29 07:37:42 UTC
the vmware side seems to use their vsphere distributed switch for those DPortGroup NICs. Seems libvirt maps that to an empty <source bridge=''/>

Comment 20 Richard W.M. Jones 2018-06-29 07:58:51 UTC
What we need for the ultimate source of truth is the *.vmx file
for the guest.  If you can ssh into the ESXi hypervisor containing
the guest, you'll find the file somewhere under /vmfs

Comment 25 Brett Thurber 2018-06-29 21:14:01 UTC
Further testing reveals that if mapping real networks, they correctly follow the desired mappings in the UI post migration.  The issue is only related to DPG's.  Attached post migration screenshot.

Comment 26 Brett Thurber 2018-06-29 21:14:44 UTC
Created attachment 1455577 [details]
post migration mappings for real networks

Comment 27 Brett Thurber 2018-07-04 06:13:45 UTC
@Michal, what do we need to do to add DPG support for RHV for v2v?

Comment 28 Michal Skrivanek 2018-07-04 06:30:39 UTC
(In reply to Brett Thurber from comment #27)
> @Michal, what do we need to do to add DPG support for RHV for v2v?

There's nothing required in RHV.
There is missing support in libvirt (DPG is not identified as anything, resulting in empty <source bridge=''/>)
There is missing support for the same parsing in virt-v2v

It can either be added to libvirt and virt-v2v, or the mapping needs to be done differently. One suggestion was to do a per-VM mapping of MAC to target network. Per Rich and Fabien that is quite feasible as NIC/MAC information are in CFME and can be sent along with the single conversion. That still needs virt-v2v to add support for mapping MACs instead of bridges

Comment 29 Richard W.M. Jones 2018-07-04 12:07:00 UTC
I posted an implementation of MAC mapping, please review if it satisfies
your needs:

https://www.redhat.com/archives/libguestfs/2018-July/msg00004.html

Note: it does NOT modify MAC addresses inside the guest, which is
a considerably more difficult task (probably impossible if you
consider all variations of proprietary software licensing based
on MAC addresses).

Comment 30 Fabien Dupont 2018-07-05 09:49:56 UTC
I have created a PR to add the support for --mac option in virt-v2v-wrapper: https://github.com/oVirt/ovirt-ansible-v2v-conversion-host/pull/6.

I have also created the related PR in ManageIQ Automate to add the MAC address to the network mappings hash: https://github.com/ManageIQ/manageiq-content/pull/351.

Comment 31 Fabien Dupont 2018-07-18 16:56:50 UTC
https://github.com/ManageIQ/manageiq-content/pull/351 has been merged.

Comment 32 Brett Thurber 2018-07-19 12:38:03 UTC
@KK PR's are merged and should be ready for testing in nightly.

Comment 33 Kedar Kulkarni 2018-07-30 18:53:19 UTC
Depends on 1607274, cannot test. Need libguestfs with correct fix available.

Comment 35 Sudhir Mallamprabhakara 2018-08-01 14:33:12 UTC
Moving this back to Assigned.

Comment 36 Brett Thurber 2018-08-03 22:30:57 UTC
Associated PR for conversion host playbook check for  --mac option:  https://github.com/oVirt/ovirt-ansible-v2v-conversion-host/pull/10

Comment 45 Kedar Kulkarni 2018-10-24 21:46:50 UTC
Tested against VMware 6.0/6.5/6.5 with RHV4.2 build 4.2.7-4 following libguestfs/virt-v2v/vdsm packages installed on conversion host: 
nbdkit-1.2.6-1.1.lp.el7ev.x86_64
nbdkit-devel-1.2.6-1.1.lp.el7ev.x86_64
nbdkit-plugin-python2-1.2.6-1.1.lp.el7ev.x86_64
nbdkit-plugin-python-common-1.2.6-1.1.lp.el7ev.x86_64
nbdkit-plugin-vddk-1.2.6-1.1.lp.el7ev.x86_64
vdsm-4.20.42-1.el7ev.x86_64
vdsm-api-4.20.42-1.el7ev.noarch
vdsm-client-4.20.42-1.el7ev.noarch
vdsm-common-4.20.42-1.el7ev.noarch
vdsm-hook-ethtool-options-4.20.42-1.el7ev.noarch
vdsm-hook-fcoe-4.20.42-1.el7ev.noarch
vdsm-hook-nestedvt-4.20.42-1.el7ev.noarch
vdsm-hook-openstacknet-4.20.42-1.el7ev.noarch
vdsm-hook-vhostmd-4.20.42-1.el7ev.noarch
vdsm-hook-vmfex-dev-4.20.42-1.el7ev.noarch
vdsm-http-4.20.42-1.el7ev.noarch
vdsm-jsonrpc-4.20.42-1.el7ev.noarch
vdsm-network-4.20.42-1.el7ev.x86_64
vdsm-python-4.20.42-1.el7ev.noarch
vdsm-yajsonrpc-4.20.42-1.el7ev.noarch
virt-v2v-1.38.2-12.22.lp.el7ev.x86_64
libguestfs-winsupport-7.2-2.el7.x86_64
libguestfs-1.38.2-12.22.lp.el7ev.x86_64
libguestfs-tools-c-1.38.2-12.22.lp.el7ev.x86_64
python-libguestfs-1.38.2-12.22.lp.el7ev.x86_64




I was successfully able to migrate VMs with DPortGroup on VMware side to RHV, and they were assigned the correct NIC as per Infra Mappings I created. No longer issue with wrong NIC assignment. 
ovirt-imageio-common-1.4.5-0.el7ev.x86_64
ovirt-imageio-daemon-1.4.5-0.el7ev.noarch

Comment 46 Kedar Kulkarni 2018-10-24 21:50:05 UTC
CFME build: 5.9.5.3.20181023135339_256263f

Comment 47 dmetzger 2019-01-07 20:15:00 UTC
Brett,

Comment 48 dmetzger 2019-01-07 20:15:35 UTC
Brett, is this one ready for test?

Comment 50 dmetzger 2019-01-08 13:51:34 UTC
Brett, see comment #44, Satoe asked for you / someone to move to ON_QA when everything is ready for test as there are external dependencies called out.

Comment 51 Shveta 2019-01-21 17:31:17 UTC
Created attachment 1522226 [details]
post migration network in RHV

Comment 52 Shveta 2019-01-21 17:33:18 UTC
Able to migrate VMs with DPortGroup on VMware side to RHV, and correct NICs were assigned as per Infra Mappings created
verified in 5.10.0.32.20190115185124_c957ada


Note You need to log in before you can comment on or make changes to this bug.