Bug 1040626

Note that the failing xml follows the example provided here:

http://libvirt.org/formatdomain.html#elementsNICSHostdev

An 82599 supports 64 VFs per PF.  Binary search says that the problem only occurs for 32 or more VFs, same as report in comment 5.

Re-adding tgraf needinfo from comment 3

Just to confirm, I can start the VM with the max 63 VFs configured with the *4 fix that went into libnl 1.1.4

*** Bug 1069548 has been marked as a duplicate of this bug. ***

This bug blocked igb, bnx2x NIC's sr-iov testing. so set TestBlocker flag.

As for this bug, test with the latest build, it can be changed the status to verified now.

package version:
libvirt-1.1.1-26.el7.x86_64
qemu-kvm-rhev-1.5.3-52.el7.x86_64
kernel-3.10.0-105.el7.x86_64
libnl3-3.2.21-5.el7.x86_64

steps:
1. find one host contains 82599 SR-IOV card, and generate the max vfs number on the host. Make sure the vf number is large than 32
# lspci|grep 82599|wc -l
128

2. add the following xml to one shutoff guest.
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:0e:09:61'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x44' slot='0x1f' function='0x4'/>
      </source>
    </interface>

3. the guest can be started up without any error.
# virsh start a
Domain a started

4. check the dumpxml of the guest, make sure the interface is in there.
# virsh dumpxml a|grep hostdev -A5
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:0e:09:61'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x44' slot='0x1f' function='0x4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

(In reply to Ludek Smid from comment #21)
> This request was resolved in Red Hat Enterprise Linux 7.0.
> 
> Contact your manager or support representative in case you have further
> questions about the request.

I am experiencing this same issue while trying to boot a VM. I'm using a Mellanox ConnectX3 configured with 8 VFs on a hypervisor running CentOS 7.  

01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
01:00.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.2 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.3 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.4 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.5 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.6 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:00.7 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]
01:01.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

Here are the relevant package versions:
libnl3-3.2.21-6.el7.x86_64
kernel-3.10.0-123.el7.x86_64
libvirt-1.1.1-29.el7_0.7.x86_64
qemu-kvm-1.5.3-60.el7_0.11.x86_64

The configuration for the PCI interface on the VM:
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:c0:34:2b'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

This configuration fails upon boot with the following error:
error: internal error: missing IFLA_VF_INFO in netlink response

If I define a PCI device in the following manner, the VM boots up fine and I can see the interface:
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev> 

One thing worth mentioning is that the VFs are on top of a infiniband interface. I've been troubleshooting this for a couple of days now without any luck. I've also brought this to the attention of the libvirt mailing list. Any help would be greatly appreciated.

Mellanox cards are a bit different from othe SRIOV cards, and their drivers are (or at least very recently were) under active development to make them more similar to standard SRIOV. The problem you are experiencing may have the same symptoms as this BZ, but it is not the same problem.

(In reply to Laine Stump from comment #24)
> Mellanox cards are a bit different from othe SRIOV cards, and their drivers
> are (or at least very recently were) under active development to make them
> more similar to standard SRIOV. The problem you are experiencing may have
> the same symptoms as this BZ, but it is not the same problem.

Should I open a new ticket or should I attempt to get in touch with Mellanox?

I would recommend direct communication with Mellanox.