Bug 1560917

Summary: vmx parser cannot import more than 4 NICs
Product: Red Hat Enterprise Linux 7 Reporter: Nisim Simsolo <nsimsolo>
Component: libvirtAssignee: Pino Toscano <ptoscano>
Status: CLOSED ERRATA QA Contact: tingting zheng <tzheng>
Severity: medium Docs Contact:
Priority: high    
Version: 7.5CC: bthurber, dyuan, jdenemar, jherrman, jsuchane, lmen, michal.skrivanek, mprivozn, mtessun, mxie, nsimsolo, ptoscano, tzheng, xuzhang
Target Milestone: rcKeywords: Upstream, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-4.3.0-1.el7 Doc Type: Bug Fix
Doc Text:
The VMX parser in libvirt did not parse more than four network interfaces. As a consequence, the esx driver did not expose more than four network interface cards (NICs) for guests running ESXi. With this update, the VMX parser parses all the available NICs in .vmx files. As a result, libvirt reports all the NICs of guests running ESXi.
Story Points: ---
Clone Of:
: 1566524 (view as bug list) Environment:
Last Closed: 2018-10-30 09:53:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 910269, 1566524    
Attachments:
Description Flags
/vdsm/import.log
none
vdsm.log
none
engine.log
none
vmx file none

Description Nisim Simsolo 2018-03-27 09:13:33 UTC
Description of problem:
When importing VMware VM with more than 4 NICs to RHV, only 4 NICs are actually imported.


Version-Release number of selected component (if applicable):
virt-v2v-1.36.10-6.el7.x86_64
libvirt-client-3.9.0-14.el7_5.2.x86_64
qemu-kvm-rhev-2.10.0-21.el7_5.1.x86_64
vdsm-4.20.23-1.el7ev.x86_64
rhvm-4.2.2.5-0.1.el7

How reproducible:
100%

Steps to Reproduce:
1. Create VMware VM with 10 NICs (Automatic, Connect at power on, VMXNET3 and verify NIC is retrieved MAC address).
2. Import VM using RHV environment.
3. Observe imported VM network NICs.

Actual results:
Only 4 NICs are imported.

Expected results:
All NICs should be imported.

Additional info:
import.log, vdsm.log and engine.log attached

Comment 2 Nisim Simsolo 2018-03-27 09:15:54 UTC
Created attachment 1413637 [details]
/vdsm/import.log

Comment 3 Nisim Simsolo 2018-03-27 09:16:20 UTC
Created attachment 1413638 [details]
vdsm.log

Comment 4 Nisim Simsolo 2018-03-27 09:16:44 UTC
Created attachment 1413639 [details]
engine.log

Comment 5 Richard W.M. Jones 2018-03-27 09:22:44 UTC
The libvirt XML shows only 4 NICs in the source:

    <interface type='bridge'>
      <mac address='00:1a:4a:16:01:55'/>
      <source bridge='VM Network'/>
      <model type='vmxnet3'/>
    </interface>
    <interface type='bridge'>
      <mac address='00:1a:4a:16:21:85'/>
      <source bridge='VM Network'/>
      <model type='e1000'/>
    </interface>
    <interface type='bridge'>
      <mac address='00:1a:4a:16:21:82'/>
      <source bridge='VM Network'/>
      <model type='e1000e'/>
    </interface>
    <interface type='bridge'>
      <mac address='00:1a:4a:16:21:69'/>
      <source bridge='VM Network'/>
      <model type='vmxnet3'/>
    </interface>

It could be that:

(1) There are only 4 NICs in the source.

(2) The libvirt vpx:// driver only reports the first 4 NICs for some reason.

(3) The other NICs are of a sort which the libvirt vpx driver cannot
parse for some reason.

Can you go to the source VMware server and grab the guest .vmx file?
If you go to the guest storage the .vmx file should be next to the
.vmdk file.  Usually you can connect to the ESXi hypervisor over https,
add "/folder" (without the trailing /) and go from there.

Comment 6 Nisim Simsolo 2018-03-27 10:32:37 UTC
(In reply to Richard W.M. Jones from comment #5)
> The libvirt XML shows only 4 NICs in the source:
> 
>     <interface type='bridge'>
>       <mac address='00:1a:4a:16:01:55'/>
>       <source bridge='VM Network'/>
>       <model type='vmxnet3'/>
>     </interface>
>     <interface type='bridge'>
>       <mac address='00:1a:4a:16:21:85'/>
>       <source bridge='VM Network'/>
>       <model type='e1000'/>
>     </interface>
>     <interface type='bridge'>
>       <mac address='00:1a:4a:16:21:82'/>
>       <source bridge='VM Network'/>
>       <model type='e1000e'/>
>     </interface>
>     <interface type='bridge'>
>       <mac address='00:1a:4a:16:21:69'/>
>       <source bridge='VM Network'/>
>       <model type='vmxnet3'/>
>     </interface>
> 
> It could be that:
> 
> (1) There are only 4 NICs in the source.
> 
> (2) The libvirt vpx:// driver only reports the first 4 NICs for some reason.
> 
> (3) The other NICs are of a sort which the libvirt vpx driver cannot
> parse for some reason.
> 
> Can you go to the source VMware server and grab the guest .vmx file?
> If you go to the guest storage the .vmx file should be next to the
> .vmdk file.  Usually you can connect to the ESXi hypervisor over https,
> add "/folder" (without the trailing /) and go from there.

In the source VM there are 10 NICs, also when I run it all NICs are listed in "ip a" command. 
8 of the NICs adapter type is VMXNET3, 1 NIC is E1000 and the last one is E1000E.

.vmx file attached

Comment 7 Pino Toscano 2018-03-27 10:45:47 UTC
> .vmx file attached

Not yet? :-?

Comment 8 Nisim Simsolo 2018-03-27 10:46:55 UTC
Created attachment 1413667 [details]
vmx file

Comment 9 Pino Toscano 2018-03-28 11:44:18 UTC
This is actually a bug in libvirt, as its vmx parser (used by the esx driver) hardcodes 4 as maximum number of network interfaces. I just posted a patch series fixing this:
https://www.redhat.com/archives/libvir-list/2018-March/msg01723.html

-----

virt-v2v -i vmx is not affected, and it correctly sees all the network interfaces:
$ virt-v2v -i vmx rh1560917.vmx --print-source 
[   0.0] Opening the source -i vmx rh1560917.vmx
Source guest information (--print-source option):

    source name: RHEL7_10_NICs
hypervisor type: vmware
         memory: 2147483648 (bytes)
       nr vCPUs: 8
     CPU vendor: 
      CPU model: 
   CPU topology: sockets: 4 cores/socket: 2 threads/core: -
   CPU features: 
       firmware: bios
        display: 
          video: vmvga
          sound: 
disks:
        RHEL7_6.vmdk (vmdk) [scsi]
        RHEL7_6_1.vmdk (vmdk) [scsi]
        /vmfs/volumes/5669422e-699d77db-c144-00e0815e303e/block4/block4.vmdk (vmdk) [scsi]
removable media:

NICs:
        Network "VM Network" [vmxnet3]
        Network "VM Network" mac: 00:1a:4a:16:21:85 [e1000]
        Network "VM Network" mac: 00:1a:4a:16:21:82 [e1000e]
        Network "VM Network" mac: 00:1a:4a:16:21:69 [vmxnet3]
        Network "VM Network" mac: 00:1a:4a:16:21:80 [vmxnet3]
        Network "VM Network" mac: 00:1a:4a:16:21:a3 [vmxnet3]
        Network "VM Network" mac: 00:1a:4a:16:21:a8 [vmxnet3]
        Network "VM Network" mac: 00:1a:4a:16:21:a9 [vmxnet3]
        Network "VM Network" mac: 00:1a:4a:16:21:78 [vmxnet3]
        Network "VM Network" mac: 00:1a:4a:16:21:81 [vmxnet3]

Comment 10 Michal Privoznik 2018-04-01 10:49:25 UTC
Now pushed upstream:

commit 5c744a2d445c01fdb9d83a9c3857ba6c0091d834
Author:     Pino Toscano <ptoscano>
AuthorDate: Wed Mar 28 13:40:21 2018 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Sun Apr 1 12:43:42 2018 +0200

    vmx: convert any amount of NICs
    
    Scan the parsed VMX file, and gather the biggest index of the network
    interfaces there: this way, it is possible to parse all the available
    network interfaces, instead of just 4 maximum.
    
    Add the VMX file attached to RHBZ#1560917 as testcase esx-in-the-wild-8.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1560917
    
    Signed-off-by: Pino Toscano <ptoscano>

commit e508e04b108709af0ca2c4d6728440b0edc3cc4b
Author:     Pino Toscano <ptoscano>
AuthorDate: Wed Mar 28 13:40:20 2018 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Sun Apr 1 12:43:42 2018 +0200

    internal: add STRCASEPREFIX
    
    Simple macro to check the prefix of a string in a case-insensitive way.
    
    Signed-off-by: Pino Toscano <ptoscano>

commit 6d394989fda53b53f0a21522bebe44ea01a40de9
Author:     Pino Toscano <ptoscano>
AuthorDate: Wed Mar 28 13:40:19 2018 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Sun Apr 1 12:43:42 2018 +0200

    vmx: allocate space for network interfaces if needed
    
    Dynamically grow the array of network interfaces for each interface
    read, instead of using a single array of size 4.  This way, in the
    future it will be easier to not limit the number of network interfaces
    (which this patch still does not change).
    
    Signed-off-by: Pino Toscano <ptoscano>

commit 5d5430e1fc6f853f79da39fbef08e97bebffc8eb
Author:     Pino Toscano <ptoscano>
AuthorDate: Wed Mar 28 13:40:18 2018 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Sun Apr 1 12:43:42 2018 +0200

    vmx: check for present/enabled devices earlier
    
    When parsing filesystems, network interfaces, serial ports, and
    parallel ports, check earlier whether they are present/enabled, delaying
    the allocation of the objects.
    
    This is mostly a small optimization, with no behaviour change.
    
    Signed-off-by: Pino Toscano <ptoscano>

v4.2.0-6-g5c744a2d44

Comment 12 kuwei@redhat.com 2018-04-08 10:39:25 UTC
I can reproduce the bug with below builds:
libvirt-3.9.0-14.el7_5.2.x86_64
virt-v2v-1.36.10-6.el7_5.1.x86_64

Steps:
Scenario 1:
1.Prepare a guest with 10 NICs on ESXi6.5
2.Using virsh to connect Vcenter:
# virsh -c vpx://root@vcenter_IP/data/ESXi_IP/?no_verify=1
3.Dump the guest xml, but we can only find 4 NICs:
virsh # dumpxml rhel7.4-tmp-test
4.Using virt-v2v convert the guest to RHV4.2, Only 4 NICs were found after conversion.

Scenario 2:
1.Prepare a guest with 10 NICs on ESXi6.5
2.Import the VM using RHV environment
3.Only 4 NICs were found after conversion.

Scenario 3:
1.Prepare a guest with 10 NICs on ESXi6.5
2.Mount the guest metadata folder to v2v conversion server.
3.Using virt-v2v convert the guest to rhv
#virt-v2v -i vmx guest.vmx -o rhv -os IP:/home/nfs_export -of qcow2
4.After conversion, 10 NICs were found on RHV

Scenario 4:
1.Prepare a guest with 10 NICs on ESXi6.5
2.Export the guest to a OVA file.
3.Using virt-v2v convert the guest to rhv
 # virt-v2v -i ova  rhel7.4.ova -o rhv -os IP:/home/nfs_export -of qcow2
4.After conversion, 10 NICs were found on RHV


So, from comment 9, it is a bug in libvirt.
virt-v2v -i vmx and -i ova is not affected, and it correctly sees all the network interfaces

Comment 15 Nisim Simsolo 2018-05-28 12:33:58 UTC
Verified:
rhvm-4.2.4-0.1.el7
libvirt-client-3.9.0-14.el7_5.5.x86_64
sanlock-3.6.0-1.el7.x86_64
qemu-kvm-rhev-2.10.0-21.el7_5.3.x86_64
vdsm-4.20.28-1.el7ev.x86_64
virt-v2v-1.36.10-6.10.rhvpreview.el7ev.x86_64

Verification scenario:
1. Import VMware VM with 10 NICs.
2. Import VMware OVA of VM with 10 NICs.

For both cases:
- Verify VM/OVA imported successfully
- Browse RHV Webadmin -> Compute -> VMs -> click on VM -> Network interfaces tab and verify 10 NICs are added to VM, each one with MAC address from RHV cluster MAC addresses pool
- Run VM and verify VM is running with 10 NICs, and each NIC have IP address from DHCP server.

Comment 17 errata-xmlrpc 2018-10-30 09:53:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113