Bug 1025397 - libvirt incorrectly assumes that SRIOV virtual functions are listed in order in PF's sysfs directory
libvirt incorrectly assumes that SRIOV virtual functions are listed in order ...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Laine Stump
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-31 11:20 EDT by Laine Stump
Modified: 2014-06-17 20:58 EDT (History)
6 users (show)

See Also:
Fixed In Version: libvirt-1.1.1-12.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-13 09:07:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Laine Stump 2013-10-31 11:20:58 EDT
When libvirt creates the list of an SRIOV Physical Function's (PF) Virtual Functions (VF), it assumes that the order of "virtfn*" links returned by readdir() from the PF's sysfs directory is already in the correct order. Experience has shown that this is not always the case.

This results in 1) incorrect assumptions made by consumers of the output of the virt_functions list of virsh nodedev-dumpxml, and 2) setting MAC address and vlan tag on the wrong VF (since libvirt uses netlink to set mac address and vlan tag, netlink requires the VF#, and the function virPCIGetVirtualFunctionIndex() returns the wrong index due to the improperly ordered VF list).

This showed up on systems with > 10 VFs, but could theoretically happen on systems with any number of VFs, just depending on the ordering of the virtfn* link in the sysfs directory. 

For example, here is the output of nodedev-dumpxml, and an ls listing of the virtfn links, for a PF on a system with 32 VFs/PF (ixgbe driver). Notice that virtfn10 shows up in position 0, while virtfn0 is in position 23.

<device>
  <name>pci_0000_86_00_0</name>
  <path>/sys/devices/pci0000:80/0000:80:01.0/0000:86:00.0</path>
  <parent>pci_0000_80_01_0</parent>
  <driver>
    <name>ixgbe</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>134</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x10fb'>82599EB 10-Gigabit SFI/SFP+ Network Connection</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <capability type='virt_functions'>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x4'/> <<=== viftfn10
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x0'/> <<<=== virtfn0
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x2'/>
    </capability>
    <iommuGroup number='27'>
      <address domain='0x0000' bus='0x86' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x00' function='0x1'/>
    </iommuGroup>
  </capability>
</device>

lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn0 -> ../0000:86:10.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn1 -> ../0000:86:10.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn10 -> ../0000:86:12.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn11 -> ../0000:86:12.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn12 -> ../0000:86:13.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn13 -> ../0000:86:13.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn14 -> ../0000:86:13.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn15 -> ../0000:86:13.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn16 -> ../0000:86:14.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn17 -> ../0000:86:14.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn18 -> ../0000:86:14.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn19 -> ../0000:86:14.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn2 -> ../0000:86:10.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn20 -> ../0000:86:15.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn21 -> ../0000:86:15.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn22 -> ../0000:86:15.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn23 -> ../0000:86:15.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn24 -> ../0000:86:16.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn25 -> ../0000:86:16.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn26 -> ../0000:86:16.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn27 -> ../0000:86:16.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn28 -> ../0000:86:17.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn29 -> ../0000:86:17.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn3 -> ../0000:86:10.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn30 -> ../0000:86:17.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn31 -> ../0000:86:17.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn32 -> ../0000:86:18.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn33 -> ../0000:86:18.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn4 -> ../0000:86:11.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn5 -> ../0000:86:11.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn6 -> ../0000:86:11.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn7 -> ../0000:86:11.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn8 -> ../0000:86:12.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn9 -> ../0000:86:12.2

It truly boggles the mind that this was never before noticed. Apparently we've always been lucky.
Comment 1 Laine Stump 2013-10-31 11:24:50 EDT
One solution to this problem is to modify virPCIGetVirtualFunctions so that, rather than just appending a device address to the virtual function list as a virtfn link is encountered, to append the "virtfn*" link names to a temporary list, then determine the proper size for the virtual function list from that, and place the resolved device addresses into the virtual function list according to the value parses out of the virtfn* link name.

We could also do it by adding a virtualFunction attribute to each device in the list, but that would require changing a lot of plumbing all over the place; sorting the list at creation is more localized and thus less likely to cause regressions elsewhere.
Comment 3 Laine Stump 2013-11-07 06:13:03 EST
A first attempt at a patch was here:

  https://www.redhat.com/archives/libvir-list/2013-November/msg00125.html

but a problem was found in review, resulting in this revised patch:

  https://www.redhat.com/archives/libvir-list/2013-November/msg00231.html
Comment 4 Laine Stump 2013-11-08 07:46:16 EST
It took a third version to get it right:

https://www.redhat.com/archives/libvir-list/2013-November/msg00291.html

This has been pushed upstream:

commit 89e2a6c88c2d26cf3a9a70adc8fdac970a471b64
Author: Laine Stump <laine@laine.org>
Date:   Fri Nov 8 12:39:08 2013 +0200

    util: use size_t instead of unsigned int for num_virtual_functions

commit 88c1fcd5e7b6568fdb1a06e0d9001c37d5e1cb78
Author: Laine Stump <laine@laine.org>
Date:   Tue Nov 5 12:13:05 2013 +0200

    pci: properly handle out-of-order SRIOV virtual functions
    
(both patches are necessary).
Comment 8 Xuesong Zhang 2013-11-12 00:47:35 EST
Tested with the latest libvirt build libvirt-1.1.1-12.el7.x86_64, this bug is verified.

Steps:
1. generate 63 VFs on host with 82599 SR-IOV card.
# lspci|grep 82599
86:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
86:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)

# echo 63 > /sys/bus/pci/devices/0000\:86\:00.0/sriov_numvfs
# ll /sys/bus/pci/devices/0000\:86\:00.0/
......
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn0 -> ../0000:86:10.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn1 -> ../0000:86:10.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn10 -> ../0000:86:12.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn11 -> ../0000:86:12.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn12 -> ../0000:86:13.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn13 -> ../0000:86:13.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn14 -> ../0000:86:13.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn15 -> ../0000:86:13.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn16 -> ../0000:86:14.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn17 -> ../0000:86:14.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn18 -> ../0000:86:14.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn19 -> ../0000:86:14.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn2 -> ../0000:86:10.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn20 -> ../0000:86:15.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn21 -> ../0000:86:15.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn22 -> ../0000:86:15.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn23 -> ../0000:86:15.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn24 -> ../0000:86:16.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn25 -> ../0000:86:16.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn26 -> ../0000:86:16.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn27 -> ../0000:86:16.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn28 -> ../0000:86:17.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn29 -> ../0000:86:17.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn3 -> ../0000:86:10.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn30 -> ../0000:86:17.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn31 -> ../0000:86:17.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn32 -> ../0000:86:18.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn33 -> ../0000:86:18.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn34 -> ../0000:86:18.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn35 -> ../0000:86:18.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn36 -> ../0000:86:19.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn37 -> ../0000:86:19.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn38 -> ../0000:86:19.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn39 -> ../0000:86:19.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn4 -> ../0000:86:11.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn40 -> ../0000:86:1a.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn41 -> ../0000:86:1a.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn42 -> ../0000:86:1a.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn43 -> ../0000:86:1a.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn44 -> ../0000:86:1b.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn45 -> ../0000:86:1b.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn46 -> ../0000:86:1b.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn47 -> ../0000:86:1b.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn48 -> ../0000:86:1c.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn49 -> ../0000:86:1c.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn5 -> ../0000:86:11.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn50 -> ../0000:86:1c.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn51 -> ../0000:86:1c.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn52 -> ../0000:86:1d.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn53 -> ../0000:86:1d.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn54 -> ../0000:86:1d.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn55 -> ../0000:86:1d.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn56 -> ../0000:86:1e.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn57 -> ../0000:86:1e.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn58 -> ../0000:86:1e.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn59 -> ../0000:86:1e.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn6 -> ../0000:86:11.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn60 -> ../0000:86:1f.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn61 -> ../0000:86:1f.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn62 -> ../0000:86:1f.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn7 -> ../0000:86:11.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn8 -> ../0000:86:12.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn9 -> ../0000:86:12.2
......

2. check the nodedev-dumpxml, make sure the order of the vfs are same with the virtfn*
# virsh nodedev-dumpxml pci_0000_86_00_0
<device>
  <name>pci_0000_86_00_0</name>
  <path>/sys/devices/pci0000:80/0000:80:01.0/0000:86:00.0</path>
  <parent>pci_0000_80_01_0</parent>
  <driver>
    <name>ixgbe</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>134</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x10fb'>82599EB 10-Gigabit SFI/SFP+ Network Connection</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <capability type='virt_functions'>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x19' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x19' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x19' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x19' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1a' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1a' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1a' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x1a' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1b' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1b' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1b' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x1b' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1c' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1c' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1c' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x1c' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1d' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1d' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1d' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x1d' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1e' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1e' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1e' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x1e' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1f' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1f' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1f' function='0x4'/>
    </capability>
    <iommuGroup number='27'>
      <address domain='0x0000' bus='0x86' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x00' function='0x1'/>
    </iommuGroup>
  </capability>
</device>
Comment 9 Laine Stump 2013-11-12 06:30:17 EST
I suspect that RHEL6 might suffer from this same problem, but am unable to verify on my own equipment. Before I go to the trouble of cloning this BZ to RHEL6, could you please run the same test on a similar (or the same) machine running RHEL6? (It's also possible that the problem is coincidentally avoided in RHEL6, in which case I don't want to create a huge amount of work by filing an unnecessary BZ)
Comment 10 Xuesong Zhang 2013-11-12 21:44:12 EST
Laine, OK, I'll try one rhel6, will add one new comments while I get the test result.

(In reply to Laine Stump from comment #9)
> I suspect that RHEL6 might suffer from this same problem, but am unable to
> verify on my own equipment. Before I go to the trouble of cloning this BZ to
> RHEL6, could you please run the same test on a similar (or the same) machine
> running RHEL6? (It's also possible that the problem is coincidentally
> avoided in RHEL6, in which case I don't want to create a huge amount of work
> by filing an unnecessary BZ)
Comment 11 Xuesong Zhang 2013-11-12 23:32:48 EST
hi, Laine,

I checked on the rhel6 with latest build libvirt-0.10.2-29.el6.x86_64, rhel6 didn't met this problem.

Steps:
1. Generate 63 VFs on the host with 82599 SR-IOV card.
# lspci|grep 82599
44:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
44:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
[root@hp-dl585g7-01 yum.repos.d]# echo 63 > /sys/bus/pci/devices/0000\:44\:00.0/sriov_numvfs 

# ll /sys/bus/pci/devices/0000\:44\:00.0/
......
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn0 -> ../0000:44:10.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn1 -> ../0000:44:10.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn10 -> ../0000:44:12.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn11 -> ../0000:44:12.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn12 -> ../0000:44:13.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn13 -> ../0000:44:13.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn14 -> ../0000:44:13.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn15 -> ../0000:44:13.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn16 -> ../0000:44:14.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn17 -> ../0000:44:14.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn18 -> ../0000:44:14.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn19 -> ../0000:44:14.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn2 -> ../0000:44:10.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn20 -> ../0000:44:15.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn21 -> ../0000:44:15.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn22 -> ../0000:44:15.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn23 -> ../0000:44:15.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn24 -> ../0000:44:16.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn25 -> ../0000:44:16.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn26 -> ../0000:44:16.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn27 -> ../0000:44:16.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn28 -> ../0000:44:17.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn29 -> ../0000:44:17.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn3 -> ../0000:44:10.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn30 -> ../0000:44:17.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn31 -> ../0000:44:17.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn32 -> ../0000:44:18.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn33 -> ../0000:44:18.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn34 -> ../0000:44:18.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn35 -> ../0000:44:18.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn36 -> ../0000:44:19.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn37 -> ../0000:44:19.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn38 -> ../0000:44:19.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn39 -> ../0000:44:19.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn4 -> ../0000:44:11.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn40 -> ../0000:44:1a.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn41 -> ../0000:44:1a.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn42 -> ../0000:44:1a.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn43 -> ../0000:44:1a.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn44 -> ../0000:44:1b.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn45 -> ../0000:44:1b.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn46 -> ../0000:44:1b.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn47 -> ../0000:44:1b.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn48 -> ../0000:44:1c.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn49 -> ../0000:44:1c.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn5 -> ../0000:44:11.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn50 -> ../0000:44:1c.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn51 -> ../0000:44:1c.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn52 -> ../0000:44:1d.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn53 -> ../0000:44:1d.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn54 -> ../0000:44:1d.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn55 -> ../0000:44:1d.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn56 -> ../0000:44:1e.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn57 -> ../0000:44:1e.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn58 -> ../0000:44:1e.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn59 -> ../0000:44:1e.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn6 -> ../0000:44:11.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn60 -> ../0000:44:1f.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn61 -> ../0000:44:1f.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn62 -> ../0000:44:1f.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn7 -> ../0000:44:11.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn8 -> ../0000:44:12.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn9 -> ../0000:44:12.2
......

2. check the nodedev-dumpxml, make sure the order of the vfs are same with the virtfn*
# virsh nodedev-dumpxml pci_0000_44_00_0
<device>
  <name>pci_0000_44_00_0</name>
  <parent>pci_0000_40_0b_0</parent>
  <driver>
    <name>ixgbe</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>68</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x10fb'>82599ES 10-Gigabit SFI/SFP+ Network Connection</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <capability type='virt_functions'>
      <address domain='0x0000' bus='0x44' slot='0x10' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x10' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x10' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x10' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x11' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x11' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x11' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x11' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x12' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x12' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x12' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x12' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x13' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x13' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x13' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x13' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x14' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x14' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x14' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x14' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x15' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x15' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x15' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x15' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x16' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x16' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x16' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x16' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x17' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x17' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x17' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x17' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x18' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x18' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x18' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x18' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x19' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x19' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x19' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x19' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1a' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1a' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1a' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x1a' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1b' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1b' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1b' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x1b' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1c' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1c' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1c' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x1c' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1d' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1d' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1d' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x1d' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1e' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1e' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1e' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x1e' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1f' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1f' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1f' function='0x4'/>
    </capability>
  </capability>
</device>
Comment 12 Laine Stump 2013-11-13 06:03:40 EST
(In reply to Zhang Xuesong from comment #11)
> I checked on the rhel6 with latest build libvirt-0.10.2-29.el6.x86_64, rhel6
> didn't met this problem.

Thanks! So apparently earlier kernels always place the virtfn* links in the sysfs directory in the proper order. (It would be interesting to know when/how that changed, but at least for now we know that RHEL6 apparently doesn't have the problem).
Comment 13 Ludek Smid 2014-06-13 09:07:30 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.