Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1025397

Summary: libvirt incorrectly assumes that SRIOV virtual functions are listed in order in PF's sysfs directory
Product: Red Hat Enterprise Linux 7 Reporter: Laine Stump <laine>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0CC: acathrow, dallan, dyuan, honzhang, virt-bugs, xuzhang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.1.1-12.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-13 13:07:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Laine Stump 2013-10-31 15:20:58 UTC 
When libvirt creates the list of an SRIOV Physical Function's (PF) Virtual Functions (VF), it assumes that the order of "virtfn*" links returned by readdir() from the PF's sysfs directory is already in the correct order. Experience has shown that this is not always the case.

This results in 1) incorrect assumptions made by consumers of the output of the virt_functions list of virsh nodedev-dumpxml, and 2) setting MAC address and vlan tag on the wrong VF (since libvirt uses netlink to set mac address and vlan tag, netlink requires the VF#, and the function virPCIGetVirtualFunctionIndex() returns the wrong index due to the improperly ordered VF list).

This showed up on systems with > 10 VFs, but could theoretically happen on systems with any number of VFs, just depending on the ordering of the virtfn* link in the sysfs directory. 

For example, here is the output of nodedev-dumpxml, and an ls listing of the virtfn links, for a PF on a system with 32 VFs/PF (ixgbe driver). Notice that virtfn10 shows up in position 0, while virtfn0 is in position 23.

<device>
  <name>pci_0000_86_00_0</name>
  <path>/sys/devices/pci0000:80/0000:80:01.0/0000:86:00.0</path>
  <parent>pci_0000_80_01_0</parent>
  <driver>
    <name>ixgbe</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>134</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x10fb'>82599EB 10-Gigabit SFI/SFP+ Network Connection</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <capability type='virt_functions'>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x4'/> <<=== viftfn10
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x0'/> <<<=== virtfn0
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x2'/>
    </capability>
    <iommuGroup number='27'>
      <address domain='0x0000' bus='0x86' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x00' function='0x1'/>
    </iommuGroup>
  </capability>
</device>

lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn0 -> ../0000:86:10.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn1 -> ../0000:86:10.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn10 -> ../0000:86:12.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn11 -> ../0000:86:12.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn12 -> ../0000:86:13.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn13 -> ../0000:86:13.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn14 -> ../0000:86:13.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn15 -> ../0000:86:13.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn16 -> ../0000:86:14.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn17 -> ../0000:86:14.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn18 -> ../0000:86:14.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn19 -> ../0000:86:14.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn2 -> ../0000:86:10.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn20 -> ../0000:86:15.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn21 -> ../0000:86:15.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn22 -> ../0000:86:15.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn23 -> ../0000:86:15.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn24 -> ../0000:86:16.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn25 -> ../0000:86:16.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn26 -> ../0000:86:16.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn27 -> ../0000:86:16.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn28 -> ../0000:86:17.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn29 -> ../0000:86:17.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn3 -> ../0000:86:10.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn30 -> ../0000:86:17.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn31 -> ../0000:86:17.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn32 -> ../0000:86:18.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn33 -> ../0000:86:18.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn4 -> ../0000:86:11.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn5 -> ../0000:86:11.2
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn6 -> ../0000:86:11.4
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn7 -> ../0000:86:11.6
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn8 -> ../0000:86:12.0
lrwxrwxrwx. 1 root root      0 Oct 30 13:52 virtfn9 -> ../0000:86:12.2

It truly boggles the mind that this was never before noticed. Apparently we've always been lucky.
Comment 1 Laine Stump 2013-10-31 15:24:50 UTC 
One solution to this problem is to modify virPCIGetVirtualFunctions so that, rather than just appending a device address to the virtual function list as a virtfn link is encountered, to append the "virtfn*" link names to a temporary list, then determine the proper size for the virtual function list from that, and place the resolved device addresses into the virtual function list according to the value parses out of the virtfn* link name.

We could also do it by adding a virtualFunction attribute to each device in the list, but that would require changing a lot of plumbing all over the place; sorting the list at creation is more localized and thus less likely to cause regressions elsewhere.
Comment 2 Laine Stump 2013-11-05 10:42:40 UTC 
BTW, this was first reported here:

https://www.redhat.com/archives/libvir-list/2013-October/msg01057.html
https://www.redhat.com/archives/libvir-list/2013-October/msg00799.html
Comment 3 Laine Stump 2013-11-07 11:13:03 UTC 
A first attempt at a patch was here:

  https://www.redhat.com/archives/libvir-list/2013-November/msg00125.html

but a problem was found in review, resulting in this revised patch:

  https://www.redhat.com/archives/libvir-list/2013-November/msg00231.html
Comment 4 Laine Stump 2013-11-08 12:46:16 UTC 
It took a third version to get it right:

https://www.redhat.com/archives/libvir-list/2013-November/msg00291.html

This has been pushed upstream:

commit 89e2a6c88c2d26cf3a9a70adc8fdac970a471b64
Author: Laine Stump <laine>
Date:   Fri Nov 8 12:39:08 2013 +0200

    util: use size_t instead of unsigned int for num_virtual_functions

commit 88c1fcd5e7b6568fdb1a06e0d9001c37d5e1cb78
Author: Laine Stump <laine>
Date:   Tue Nov 5 12:13:05 2013 +0200

    pci: properly handle out-of-order SRIOV virtual functions
    
(both patches are necessary).
Comment 8 Xuesong Zhang 2013-11-12 05:47:35 UTC 
Tested with the latest libvirt build libvirt-1.1.1-12.el7.x86_64, this bug is verified.

Steps:
1. generate 63 VFs on host with 82599 SR-IOV card.
# lspci|grep 82599
86:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
86:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)

# echo 63 > /sys/bus/pci/devices/0000\:86\:00.0/sriov_numvfs
# ll /sys/bus/pci/devices/0000\:86\:00.0/
......
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn0 -> ../0000:86:10.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn1 -> ../0000:86:10.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn10 -> ../0000:86:12.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn11 -> ../0000:86:12.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn12 -> ../0000:86:13.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn13 -> ../0000:86:13.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn14 -> ../0000:86:13.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn15 -> ../0000:86:13.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn16 -> ../0000:86:14.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn17 -> ../0000:86:14.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn18 -> ../0000:86:14.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn19 -> ../0000:86:14.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn2 -> ../0000:86:10.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn20 -> ../0000:86:15.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn21 -> ../0000:86:15.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn22 -> ../0000:86:15.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn23 -> ../0000:86:15.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn24 -> ../0000:86:16.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn25 -> ../0000:86:16.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn26 -> ../0000:86:16.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn27 -> ../0000:86:16.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn28 -> ../0000:86:17.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn29 -> ../0000:86:17.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn3 -> ../0000:86:10.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn30 -> ../0000:86:17.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn31 -> ../0000:86:17.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn32 -> ../0000:86:18.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn33 -> ../0000:86:18.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn34 -> ../0000:86:18.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn35 -> ../0000:86:18.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn36 -> ../0000:86:19.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn37 -> ../0000:86:19.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn38 -> ../0000:86:19.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn39 -> ../0000:86:19.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn4 -> ../0000:86:11.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn40 -> ../0000:86:1a.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn41 -> ../0000:86:1a.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn42 -> ../0000:86:1a.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn43 -> ../0000:86:1a.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn44 -> ../0000:86:1b.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn45 -> ../0000:86:1b.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn46 -> ../0000:86:1b.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn47 -> ../0000:86:1b.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn48 -> ../0000:86:1c.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn49 -> ../0000:86:1c.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn5 -> ../0000:86:11.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn50 -> ../0000:86:1c.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn51 -> ../0000:86:1c.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn52 -> ../0000:86:1d.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn53 -> ../0000:86:1d.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn54 -> ../0000:86:1d.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn55 -> ../0000:86:1d.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn56 -> ../0000:86:1e.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn57 -> ../0000:86:1e.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn58 -> ../0000:86:1e.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn59 -> ../0000:86:1e.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn6 -> ../0000:86:11.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn60 -> ../0000:86:1f.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn61 -> ../0000:86:1f.2
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn62 -> ../0000:86:1f.4
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn7 -> ../0000:86:11.6
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn8 -> ../0000:86:12.0
lrwxrwxrwx. 1 root root      0 Nov 12 00:34 virtfn9 -> ../0000:86:12.2
......

2. check the nodedev-dumpxml, make sure the order of the vfs are same with the virtfn*
# virsh nodedev-dumpxml pci_0000_86_00_0
<device>
  <name>pci_0000_86_00_0</name>
  <path>/sys/devices/pci0000:80/0000:80:01.0/0000:86:00.0</path>
  <parent>pci_0000_80_01_0</parent>
  <driver>
    <name>ixgbe</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>134</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x10fb'>82599EB 10-Gigabit SFI/SFP+ Network Connection</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <capability type='virt_functions'>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x10' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x11' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x12' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x13' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x14' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x15' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x16' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x17' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x18' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x19' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x19' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x19' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x19' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1a' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1a' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1a' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x1a' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1b' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1b' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1b' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x1b' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1c' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1c' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1c' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x1c' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1d' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1d' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1d' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x1d' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1e' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1e' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1e' function='0x4'/>
      <address domain='0x0000' bus='0x86' slot='0x1e' function='0x6'/>
      <address domain='0x0000' bus='0x86' slot='0x1f' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x1f' function='0x2'/>
      <address domain='0x0000' bus='0x86' slot='0x1f' function='0x4'/>
    </capability>
    <iommuGroup number='27'>
      <address domain='0x0000' bus='0x86' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x86' slot='0x00' function='0x1'/>
    </iommuGroup>
  </capability>
</device>
Comment 9 Laine Stump 2013-11-12 11:30:17 UTC 
I suspect that RHEL6 might suffer from this same problem, but am unable to verify on my own equipment. Before I go to the trouble of cloning this BZ to RHEL6, could you please run the same test on a similar (or the same) machine running RHEL6? (It's also possible that the problem is coincidentally avoided in RHEL6, in which case I don't want to create a huge amount of work by filing an unnecessary BZ)
Comment 10 Xuesong Zhang 2013-11-13 02:44:12 UTC 
Laine, OK, I'll try one rhel6, will add one new comments while I get the test result.

(In reply to Laine Stump from comment #9)
> I suspect that RHEL6 might suffer from this same problem, but am unable to
> verify on my own equipment. Before I go to the trouble of cloning this BZ to
> RHEL6, could you please run the same test on a similar (or the same) machine
> running RHEL6? (It's also possible that the problem is coincidentally
> avoided in RHEL6, in which case I don't want to create a huge amount of work
> by filing an unnecessary BZ)
Comment 11 Xuesong Zhang 2013-11-13 04:32:48 UTC 
hi, Laine,

I checked on the rhel6 with latest build libvirt-0.10.2-29.el6.x86_64, rhel6 didn't met this problem.

Steps:
1. Generate 63 VFs on the host with 82599 SR-IOV card.
# lspci|grep 82599
44:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
44:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
[root@hp-dl585g7-01 yum.repos.d]# echo 63 > /sys/bus/pci/devices/0000\:44\:00.0/sriov_numvfs 

# ll /sys/bus/pci/devices/0000\:44\:00.0/
......
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn0 -> ../0000:44:10.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn1 -> ../0000:44:10.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn10 -> ../0000:44:12.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn11 -> ../0000:44:12.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn12 -> ../0000:44:13.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn13 -> ../0000:44:13.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn14 -> ../0000:44:13.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn15 -> ../0000:44:13.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn16 -> ../0000:44:14.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn17 -> ../0000:44:14.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn18 -> ../0000:44:14.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn19 -> ../0000:44:14.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn2 -> ../0000:44:10.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn20 -> ../0000:44:15.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn21 -> ../0000:44:15.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn22 -> ../0000:44:15.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn23 -> ../0000:44:15.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn24 -> ../0000:44:16.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn25 -> ../0000:44:16.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn26 -> ../0000:44:16.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn27 -> ../0000:44:16.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn28 -> ../0000:44:17.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn29 -> ../0000:44:17.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn3 -> ../0000:44:10.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn30 -> ../0000:44:17.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn31 -> ../0000:44:17.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn32 -> ../0000:44:18.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn33 -> ../0000:44:18.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn34 -> ../0000:44:18.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn35 -> ../0000:44:18.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn36 -> ../0000:44:19.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn37 -> ../0000:44:19.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn38 -> ../0000:44:19.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn39 -> ../0000:44:19.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn4 -> ../0000:44:11.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn40 -> ../0000:44:1a.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn41 -> ../0000:44:1a.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn42 -> ../0000:44:1a.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn43 -> ../0000:44:1a.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn44 -> ../0000:44:1b.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn45 -> ../0000:44:1b.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn46 -> ../0000:44:1b.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn47 -> ../0000:44:1b.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn48 -> ../0000:44:1c.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn49 -> ../0000:44:1c.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn5 -> ../0000:44:11.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn50 -> ../0000:44:1c.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn51 -> ../0000:44:1c.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn52 -> ../0000:44:1d.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn53 -> ../0000:44:1d.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn54 -> ../0000:44:1d.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn55 -> ../0000:44:1d.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn56 -> ../0000:44:1e.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn57 -> ../0000:44:1e.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn58 -> ../0000:44:1e.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn59 -> ../0000:44:1e.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn6 -> ../0000:44:11.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn60 -> ../0000:44:1f.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn61 -> ../0000:44:1f.2
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn62 -> ../0000:44:1f.4
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn7 -> ../0000:44:11.6
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn8 -> ../0000:44:12.0
lrwxrwxrwx. 1 root root      0 Nov 13 10:54 virtfn9 -> ../0000:44:12.2
......

2. check the nodedev-dumpxml, make sure the order of the vfs are same with the virtfn*
# virsh nodedev-dumpxml pci_0000_44_00_0
<device>
  <name>pci_0000_44_00_0</name>
  <parent>pci_0000_40_0b_0</parent>
  <driver>
    <name>ixgbe</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>68</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x10fb'>82599ES 10-Gigabit SFI/SFP+ Network Connection</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <capability type='virt_functions'>
      <address domain='0x0000' bus='0x44' slot='0x10' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x10' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x10' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x10' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x11' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x11' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x11' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x11' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x12' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x12' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x12' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x12' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x13' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x13' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x13' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x13' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x14' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x14' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x14' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x14' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x15' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x15' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x15' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x15' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x16' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x16' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x16' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x16' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x17' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x17' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x17' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x17' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x18' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x18' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x18' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x18' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x19' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x19' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x19' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x19' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1a' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1a' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1a' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x1a' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1b' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1b' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1b' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x1b' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1c' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1c' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1c' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x1c' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1d' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1d' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1d' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x1d' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1e' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1e' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1e' function='0x4'/>
      <address domain='0x0000' bus='0x44' slot='0x1e' function='0x6'/>
      <address domain='0x0000' bus='0x44' slot='0x1f' function='0x0'/>
      <address domain='0x0000' bus='0x44' slot='0x1f' function='0x2'/>
      <address domain='0x0000' bus='0x44' slot='0x1f' function='0x4'/>
    </capability>
  </capability>
</device>
Comment 12 Laine Stump 2013-11-13 11:03:40 UTC 
(In reply to Zhang Xuesong from comment #11)
> I checked on the rhel6 with latest build libvirt-0.10.2-29.el6.x86_64, rhel6
> didn't met this problem.

Thanks! So apparently earlier kernels always place the virtfn* links in the sysfs directory in the proper order. (It would be interesting to know when/how that changed, but at least for now we know that RHEL6 apparently doesn't have the problem).
Comment 13 Ludek Smid 2014-06-13 13:07:30 UTC 
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.