Red Hat Bugzilla – Bug 1025397
libvirt incorrectly assumes that SRIOV virtual functions are listed in order in PF's sysfs directory
Last modified: 2014-06-17 20:58:16 EDT
When libvirt creates the list of an SRIOV Physical Function's (PF) Virtual Functions (VF), it assumes that the order of "virtfn*" links returned by readdir() from the PF's sysfs directory is already in the correct order. Experience has shown that this is not always the case. This results in 1) incorrect assumptions made by consumers of the output of the virt_functions list of virsh nodedev-dumpxml, and 2) setting MAC address and vlan tag on the wrong VF (since libvirt uses netlink to set mac address and vlan tag, netlink requires the VF#, and the function virPCIGetVirtualFunctionIndex() returns the wrong index due to the improperly ordered VF list). This showed up on systems with > 10 VFs, but could theoretically happen on systems with any number of VFs, just depending on the ordering of the virtfn* link in the sysfs directory. For example, here is the output of nodedev-dumpxml, and an ls listing of the virtfn links, for a PF on a system with 32 VFs/PF (ixgbe driver). Notice that virtfn10 shows up in position 0, while virtfn0 is in position 23. <device> <name>pci_0000_86_00_0</name> <path>/sys/devices/pci0000:80/0000:80:01.0/0000:86:00.0</path> <parent>pci_0000_80_01_0</parent> <driver> <name>ixgbe</name> </driver> <capability type='pci'> <domain>0</domain> <bus>134</bus> <slot>0</slot> <function>0</function> <product id='0x10fb'>82599EB 10-Gigabit SFI/SFP+ Network Connection</product> <vendor id='0x8086'>Intel Corporation</vendor> <capability type='virt_functions'> <address domain='0x0000' bus='0x86' slot='0x12' function='0x4'/> <<=== viftfn10 <address domain='0x0000' bus='0x86' slot='0x12' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x13' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x13' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x13' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x13' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x14' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x14' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x14' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x14' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x15' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x15' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x15' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x15' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x16' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x16' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x16' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x16' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x17' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x17' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x17' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x17' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x18' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x18' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x10' function='0x0'/> <<<=== virtfn0 <address domain='0x0000' bus='0x86' slot='0x10' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x10' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x10' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x11' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x11' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x11' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x11' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x12' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x12' function='0x2'/> </capability> <iommuGroup number='27'> <address domain='0x0000' bus='0x86' slot='0x00' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x00' function='0x1'/> </iommuGroup> </capability> </device> lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn0 -> ../0000:86:10.0 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn1 -> ../0000:86:10.2 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn10 -> ../0000:86:12.4 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn11 -> ../0000:86:12.6 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn12 -> ../0000:86:13.0 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn13 -> ../0000:86:13.2 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn14 -> ../0000:86:13.4 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn15 -> ../0000:86:13.6 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn16 -> ../0000:86:14.0 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn17 -> ../0000:86:14.2 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn18 -> ../0000:86:14.4 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn19 -> ../0000:86:14.6 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn2 -> ../0000:86:10.4 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn20 -> ../0000:86:15.0 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn21 -> ../0000:86:15.2 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn22 -> ../0000:86:15.4 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn23 -> ../0000:86:15.6 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn24 -> ../0000:86:16.0 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn25 -> ../0000:86:16.2 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn26 -> ../0000:86:16.4 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn27 -> ../0000:86:16.6 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn28 -> ../0000:86:17.0 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn29 -> ../0000:86:17.2 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn3 -> ../0000:86:10.6 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn30 -> ../0000:86:17.4 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn31 -> ../0000:86:17.6 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn32 -> ../0000:86:18.0 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn33 -> ../0000:86:18.2 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn4 -> ../0000:86:11.0 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn5 -> ../0000:86:11.2 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn6 -> ../0000:86:11.4 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn7 -> ../0000:86:11.6 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn8 -> ../0000:86:12.0 lrwxrwxrwx. 1 root root 0 Oct 30 13:52 virtfn9 -> ../0000:86:12.2 It truly boggles the mind that this was never before noticed. Apparently we've always been lucky.
One solution to this problem is to modify virPCIGetVirtualFunctions so that, rather than just appending a device address to the virtual function list as a virtfn link is encountered, to append the "virtfn*" link names to a temporary list, then determine the proper size for the virtual function list from that, and place the resolved device addresses into the virtual function list according to the value parses out of the virtfn* link name. We could also do it by adding a virtualFunction attribute to each device in the list, but that would require changing a lot of plumbing all over the place; sorting the list at creation is more localized and thus less likely to cause regressions elsewhere.
BTW, this was first reported here: https://www.redhat.com/archives/libvir-list/2013-October/msg01057.html https://www.redhat.com/archives/libvir-list/2013-October/msg00799.html
A first attempt at a patch was here: https://www.redhat.com/archives/libvir-list/2013-November/msg00125.html but a problem was found in review, resulting in this revised patch: https://www.redhat.com/archives/libvir-list/2013-November/msg00231.html
It took a third version to get it right: https://www.redhat.com/archives/libvir-list/2013-November/msg00291.html This has been pushed upstream: commit 89e2a6c88c2d26cf3a9a70adc8fdac970a471b64 Author: Laine Stump <laine@laine.org> Date: Fri Nov 8 12:39:08 2013 +0200 util: use size_t instead of unsigned int for num_virtual_functions commit 88c1fcd5e7b6568fdb1a06e0d9001c37d5e1cb78 Author: Laine Stump <laine@laine.org> Date: Tue Nov 5 12:13:05 2013 +0200 pci: properly handle out-of-order SRIOV virtual functions (both patches are necessary).
Tested with the latest libvirt build libvirt-1.1.1-12.el7.x86_64, this bug is verified. Steps: 1. generate 63 VFs on host with 82599 SR-IOV card. # lspci|grep 82599 86:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 86:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) # echo 63 > /sys/bus/pci/devices/0000\:86\:00.0/sriov_numvfs # ll /sys/bus/pci/devices/0000\:86\:00.0/ ...... lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn0 -> ../0000:86:10.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn1 -> ../0000:86:10.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn10 -> ../0000:86:12.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn11 -> ../0000:86:12.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn12 -> ../0000:86:13.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn13 -> ../0000:86:13.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn14 -> ../0000:86:13.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn15 -> ../0000:86:13.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn16 -> ../0000:86:14.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn17 -> ../0000:86:14.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn18 -> ../0000:86:14.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn19 -> ../0000:86:14.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn2 -> ../0000:86:10.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn20 -> ../0000:86:15.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn21 -> ../0000:86:15.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn22 -> ../0000:86:15.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn23 -> ../0000:86:15.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn24 -> ../0000:86:16.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn25 -> ../0000:86:16.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn26 -> ../0000:86:16.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn27 -> ../0000:86:16.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn28 -> ../0000:86:17.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn29 -> ../0000:86:17.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn3 -> ../0000:86:10.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn30 -> ../0000:86:17.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn31 -> ../0000:86:17.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn32 -> ../0000:86:18.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn33 -> ../0000:86:18.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn34 -> ../0000:86:18.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn35 -> ../0000:86:18.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn36 -> ../0000:86:19.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn37 -> ../0000:86:19.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn38 -> ../0000:86:19.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn39 -> ../0000:86:19.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn4 -> ../0000:86:11.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn40 -> ../0000:86:1a.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn41 -> ../0000:86:1a.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn42 -> ../0000:86:1a.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn43 -> ../0000:86:1a.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn44 -> ../0000:86:1b.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn45 -> ../0000:86:1b.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn46 -> ../0000:86:1b.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn47 -> ../0000:86:1b.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn48 -> ../0000:86:1c.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn49 -> ../0000:86:1c.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn5 -> ../0000:86:11.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn50 -> ../0000:86:1c.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn51 -> ../0000:86:1c.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn52 -> ../0000:86:1d.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn53 -> ../0000:86:1d.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn54 -> ../0000:86:1d.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn55 -> ../0000:86:1d.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn56 -> ../0000:86:1e.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn57 -> ../0000:86:1e.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn58 -> ../0000:86:1e.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn59 -> ../0000:86:1e.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn6 -> ../0000:86:11.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn60 -> ../0000:86:1f.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn61 -> ../0000:86:1f.2 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn62 -> ../0000:86:1f.4 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn7 -> ../0000:86:11.6 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn8 -> ../0000:86:12.0 lrwxrwxrwx. 1 root root 0 Nov 12 00:34 virtfn9 -> ../0000:86:12.2 ...... 2. check the nodedev-dumpxml, make sure the order of the vfs are same with the virtfn* # virsh nodedev-dumpxml pci_0000_86_00_0 <device> <name>pci_0000_86_00_0</name> <path>/sys/devices/pci0000:80/0000:80:01.0/0000:86:00.0</path> <parent>pci_0000_80_01_0</parent> <driver> <name>ixgbe</name> </driver> <capability type='pci'> <domain>0</domain> <bus>134</bus> <slot>0</slot> <function>0</function> <product id='0x10fb'>82599EB 10-Gigabit SFI/SFP+ Network Connection</product> <vendor id='0x8086'>Intel Corporation</vendor> <capability type='virt_functions'> <address domain='0x0000' bus='0x86' slot='0x10' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x10' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x10' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x10' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x11' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x11' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x11' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x11' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x12' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x12' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x12' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x12' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x13' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x13' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x13' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x13' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x14' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x14' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x14' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x14' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x15' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x15' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x15' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x15' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x16' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x16' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x16' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x16' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x17' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x17' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x17' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x17' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x18' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x18' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x18' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x18' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x19' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x19' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x19' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x19' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x1a' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x1a' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x1a' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x1a' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x1b' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x1b' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x1b' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x1b' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x1c' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x1c' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x1c' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x1c' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x1d' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x1d' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x1d' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x1d' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x1e' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x1e' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x1e' function='0x4'/> <address domain='0x0000' bus='0x86' slot='0x1e' function='0x6'/> <address domain='0x0000' bus='0x86' slot='0x1f' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x1f' function='0x2'/> <address domain='0x0000' bus='0x86' slot='0x1f' function='0x4'/> </capability> <iommuGroup number='27'> <address domain='0x0000' bus='0x86' slot='0x00' function='0x0'/> <address domain='0x0000' bus='0x86' slot='0x00' function='0x1'/> </iommuGroup> </capability> </device>
I suspect that RHEL6 might suffer from this same problem, but am unable to verify on my own equipment. Before I go to the trouble of cloning this BZ to RHEL6, could you please run the same test on a similar (or the same) machine running RHEL6? (It's also possible that the problem is coincidentally avoided in RHEL6, in which case I don't want to create a huge amount of work by filing an unnecessary BZ)
Laine, OK, I'll try one rhel6, will add one new comments while I get the test result. (In reply to Laine Stump from comment #9) > I suspect that RHEL6 might suffer from this same problem, but am unable to > verify on my own equipment. Before I go to the trouble of cloning this BZ to > RHEL6, could you please run the same test on a similar (or the same) machine > running RHEL6? (It's also possible that the problem is coincidentally > avoided in RHEL6, in which case I don't want to create a huge amount of work > by filing an unnecessary BZ)
hi, Laine, I checked on the rhel6 with latest build libvirt-0.10.2-29.el6.x86_64, rhel6 didn't met this problem. Steps: 1. Generate 63 VFs on the host with 82599 SR-IOV card. # lspci|grep 82599 44:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 44:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) [root@hp-dl585g7-01 yum.repos.d]# echo 63 > /sys/bus/pci/devices/0000\:44\:00.0/sriov_numvfs # ll /sys/bus/pci/devices/0000\:44\:00.0/ ...... lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn0 -> ../0000:44:10.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn1 -> ../0000:44:10.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn10 -> ../0000:44:12.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn11 -> ../0000:44:12.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn12 -> ../0000:44:13.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn13 -> ../0000:44:13.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn14 -> ../0000:44:13.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn15 -> ../0000:44:13.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn16 -> ../0000:44:14.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn17 -> ../0000:44:14.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn18 -> ../0000:44:14.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn19 -> ../0000:44:14.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn2 -> ../0000:44:10.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn20 -> ../0000:44:15.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn21 -> ../0000:44:15.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn22 -> ../0000:44:15.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn23 -> ../0000:44:15.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn24 -> ../0000:44:16.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn25 -> ../0000:44:16.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn26 -> ../0000:44:16.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn27 -> ../0000:44:16.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn28 -> ../0000:44:17.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn29 -> ../0000:44:17.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn3 -> ../0000:44:10.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn30 -> ../0000:44:17.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn31 -> ../0000:44:17.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn32 -> ../0000:44:18.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn33 -> ../0000:44:18.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn34 -> ../0000:44:18.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn35 -> ../0000:44:18.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn36 -> ../0000:44:19.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn37 -> ../0000:44:19.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn38 -> ../0000:44:19.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn39 -> ../0000:44:19.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn4 -> ../0000:44:11.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn40 -> ../0000:44:1a.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn41 -> ../0000:44:1a.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn42 -> ../0000:44:1a.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn43 -> ../0000:44:1a.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn44 -> ../0000:44:1b.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn45 -> ../0000:44:1b.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn46 -> ../0000:44:1b.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn47 -> ../0000:44:1b.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn48 -> ../0000:44:1c.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn49 -> ../0000:44:1c.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn5 -> ../0000:44:11.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn50 -> ../0000:44:1c.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn51 -> ../0000:44:1c.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn52 -> ../0000:44:1d.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn53 -> ../0000:44:1d.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn54 -> ../0000:44:1d.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn55 -> ../0000:44:1d.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn56 -> ../0000:44:1e.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn57 -> ../0000:44:1e.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn58 -> ../0000:44:1e.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn59 -> ../0000:44:1e.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn6 -> ../0000:44:11.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn60 -> ../0000:44:1f.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn61 -> ../0000:44:1f.2 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn62 -> ../0000:44:1f.4 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn7 -> ../0000:44:11.6 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn8 -> ../0000:44:12.0 lrwxrwxrwx. 1 root root 0 Nov 13 10:54 virtfn9 -> ../0000:44:12.2 ...... 2. check the nodedev-dumpxml, make sure the order of the vfs are same with the virtfn* # virsh nodedev-dumpxml pci_0000_44_00_0 <device> <name>pci_0000_44_00_0</name> <parent>pci_0000_40_0b_0</parent> <driver> <name>ixgbe</name> </driver> <capability type='pci'> <domain>0</domain> <bus>68</bus> <slot>0</slot> <function>0</function> <product id='0x10fb'>82599ES 10-Gigabit SFI/SFP+ Network Connection</product> <vendor id='0x8086'>Intel Corporation</vendor> <capability type='virt_functions'> <address domain='0x0000' bus='0x44' slot='0x10' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x10' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x10' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x10' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x11' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x11' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x11' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x11' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x12' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x12' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x12' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x12' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x13' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x13' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x13' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x13' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x14' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x14' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x14' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x14' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x15' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x15' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x15' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x15' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x16' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x16' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x16' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x16' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x17' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x17' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x17' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x17' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x18' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x18' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x18' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x18' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x19' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x19' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x19' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x19' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x1a' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x1a' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x1a' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x1a' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x1b' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x1b' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x1b' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x1b' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x1c' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x1c' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x1c' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x1c' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x1d' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x1d' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x1d' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x1d' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x1e' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x1e' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x1e' function='0x4'/> <address domain='0x0000' bus='0x44' slot='0x1e' function='0x6'/> <address domain='0x0000' bus='0x44' slot='0x1f' function='0x0'/> <address domain='0x0000' bus='0x44' slot='0x1f' function='0x2'/> <address domain='0x0000' bus='0x44' slot='0x1f' function='0x4'/> </capability> </capability> </device>
(In reply to Zhang Xuesong from comment #11) > I checked on the rhel6 with latest build libvirt-0.10.2-29.el6.x86_64, rhel6 > didn't met this problem. Thanks! So apparently earlier kernels always place the virtfn* links in the sysfs directory in the proper order. (It would be interesting to know when/how that changed, but at least for now we know that RHEL6 apparently doesn't have the problem).
This request was resolved in Red Hat Enterprise Linux 7.0. Contact your manager or support representative in case you have further questions about the request.