Description of problem: One of VFs in a pool of SRIOV VFs can not be available when the VFs from different pci bus Version-Release number of selected component (if applicable): libvirt-0.10.2-23.el6.x86_64 How reproducible: 100% Steps to Reproduce: [root@sriov2 images]# lspci|grep 82576 0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 0f:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 0f:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 10:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 10:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 11:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 11:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) [root@sriov2 images]# virsh net-start hostdev-net1 Network hostdev-net1 started [root@sriov2 images]# virsh net-dumpxml hostdev-net1 <network> <name>hostdev-net1</name> <uuid>a57d0acb-3182-28ad-5c47-aae471ca8084</uuid> <forward mode='hostdev' managed='yes'> <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/> <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x1'/> <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/> <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/> </forward> </network> [root@sriov2 images]# virsh destroy r6 Domain r6 destroyed [root@sriov2 images]# virsh start r6 Domain r6 started [root@sriov2 images]# virsh dumpxml r6 <domain type='kvm' id='27'> <name>r6</name> <uuid>60f88110-4c02-0358-46eb-afdce5dc1002</uuid> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='rhel6.5.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/var/lib/libvirt/images/kvm-rhel6.3-i386-raw.img'/> <target dev='hda' bus='ide'/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='ide' index='0'> <alias name='ide0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <serial type='pty'> <source path='/dev/pts/1'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <sound model='ich6'> <alias name='sound0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='cirrus' vram='9216' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c123,c735</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c123,c735</imagelabel> </seclabel> </domain> [root@sriov2 images]# virsh attach-device r6 vfpool.xml Device attached successfully [root@sriov2 images]# virsh attach-device r6 vfpool.xml Device attached successfully [root@sriov2 images]# virsh attach-device r6 vfpool.xml Device attached successfully [root@sriov2 images]# virsh attach-device r6 vfpool.xml error: Failed to attach device from vfpool.xml error: internal error network 'hostdev-net1' requires exclusive access to interfaces, but none are available [root@sriov2 images]# virsh dumpxml r6 |grep interface -A 4 <interface type='network'> <mac address='52:54:00:25:1f:f1'/> <source network='hostdev-net1'/> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:7c:8b:3c'/> <source network='hostdev-net1'/> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:05:6a:3a'/> <source network='hostdev-net1'/> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> Actual results: One of VFs in a pool of SRIOV VFs can not be available when the VFs from different pci bus. Expected results: All VFs in a pool of SRIOV VFs are available. Additional info:
The summary of this bug doesn't make sense based on the test that was performed. There are 4 VFs, two on bus 0x0F, and two on bus 0x11, so if three VFs were successfully assigned, then there is at least one VF from each bus that was successfully assigned. Beyond that, the error message that is being reported is saying that all VFs in the pool are already in use. I'm wondering if perhaps another test had already been run on the host prior to this test, resulting in one of the VFs being marked as unused. Can you please rerun this test on a newly booted host to assure that there wasn't a VF already in use at the time the test was started? Also, before running the test, make sure that hostdev-net1 is marked as autostart, and is started: virsh net-autostart hostdev-net1; virsh net-start hostdev-net1 Finally, both before and after your test, please capture the output of the following commands: ls -ld /sys/bus/pci/devices/*/driver | egrep '0000:(0F|11)' ls -ld /sys/bus/pci/devices/*/physfn (keep in mind that there is an open bug concerning the fact that hostdev and macvtap networks can be used even if they are down (Bug 880483, also reported more recently as Bug 1002427), and a related issue of that is that these networks are marked as "down" each time libvirtd is restarted, even if they were previously started - setting autostart for these nets is a workaround for that problem).
This bug was not selected to be addressed in Red Hat Enterprise Linux 6. We will look at it again within the Red Hat Enterprise Linux 7 product.
It can not be reproduced according to the steps of description after new boot and autostarting hosdev network. But the issues still can be reproduced after restarting network and guest as follows. [root@sriov2 ~]# ls -ld /sys/bus/pci/devices/*/driver | egrep '0000:(0F|11)' lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.0/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.1/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.2/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.3/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.4/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.5/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.6/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.7/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.0/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.1/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.2/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.3/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.4/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.5/driver -> ../../../../../../bus/pci/drivers/igbvf [root@sriov2 ~]# ls -ld /sys/bus/pci/devices/*/physfn lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.0/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.1/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.2/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.3/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.4/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.5/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.6/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.7/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.0/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.1/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.2/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.3/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.4/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.5/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.0/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.1/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.2/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.3/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.4/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.5/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.6/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.7/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.0/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.1/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.2/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.3/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.4/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.5/physfn -> ../0000:10:00.1 [root@sriov2 ~]# virsh net-list Name State Autostart Persistent -------------------------------------------------- default active yes yes hostnet active yes yes [root@sriov2 ~]# virsh net-dumpxml hostnet <network> <name>hostnet</name> <uuid>6b49be3c-bb91-c16d-b475-2929678720f4</uuid> <forward mode='hostdev' managed='yes'> <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/> <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/> </forward> </network> [root@sriov2 images]# virsh attach-device a vfpool.xml Device attached successfully [root@sriov2 images]# virsh attach-device a vfpool.xml Device attached successfully [root@sriov2 images]# virsh net-destroy hostnet Network hostnet destroyed [root@sriov2 images]# virsh net-start hostnet Network hostnet started [root@sriov2 images]# virsh destroy a Domain a destroyed [root@sriov2 images]# virsh start a Domain a started [root@sriov2 images]# virsh attach-device a vfpool.xml error: Failed to attach device from vfpool.xml error: internal error network 'hostnet' requires exclusive access to interfaces, but none are available [root@sriov2 images]# ls -ld /sys/bus/pci/devices/*/driver | egrep '0000:(0F|11)' lrwxrwxrwx. 1 root root 0 Apr 10 00:49 /sys/bus/pci/devices/0000:11:10.0/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.1/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.2/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.3/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.4/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.5/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.6/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.7/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.0/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.1/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.2/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.3/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.4/driver -> ../../../../../../bus/pci/drivers/igbvf lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.5/driver -> ../../../../../../bus/pci/drivers/igbvf [root@sriov2 images]# ls -ld /sys/bus/pci/devices/*/physfn lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.0/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.1/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.2/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.3/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.4/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.5/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.6/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.7/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.0/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.1/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.2/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.3/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.4/physfn -> ../0000:0e:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.5/physfn -> ../0000:0e:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.0/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.1/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.2/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.3/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.4/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.5/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.6/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.7/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.0/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.1/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.2/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.3/physfn -> ../0000:10:00.1 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.4/physfn -> ../0000:10:00.0 lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.5/physfn -> ../0000:10:00.1
The issues in comment 8 can be reproduced in rhel6 and rhel7 . Versions as follows. And also reproduced when the VFs pool includes 4 VFs. rhel6 libvirt-0.10.2-29.el6_5.4.x86_64 kernel-2.6.32-431.el6.x86_64 rhel7 libvirt-1.1.1-29.el7.x86_64 kernel-3.10.0-115.el7.x86_64
Based on the sequence of operations given in the examples, I'm 100% certain that the problem here has nothing to do with some of the VFs being on different PFs (that will make no difference to libvirt). Instead, the problem is that the connection count for a network, and for individual devices in a network pool, is lost when a network is net-destroyed, and not properly restored when the network is re-started. For example, from Comment 8: root@sriov2 images]# virsh attach-device a vfpool.xml Device attached successfully (network connections = 1, 1st dev connections = 1, 2nd dev connections = 0) [root@sriov2 images]# virsh attach-device a vfpool.xml Device attached successfully (network connections = 2, 1st dev connections = 1, 2nd dev connections = 1) [root@sriov2 images]# virsh net-destroy hostnet Network hostnet destroyed [root@sriov2 images]# virsh net-start hostnet Network hostnet started (network connections = 1, 1st dev connections = 0, 2nd dev connections = 0) [root@sriov2 images]# virsh destroy a Domain a destroyed (network connections = -1, 1st dev connections = -1, 2nd dev connections = -1) [root@sriov2 images]# virsh start a Domain a started The next time you try to get a device from the pool, libvirt will look for a device with connections == 0, not find any, and report the failure. Fixing this isn't exactly straightforward, because the necessary info is in the qemu driver's domain status, which the network driver doesn't have direct access to. It will require some sort of system for the network driver to notify all the hypervisor drivers whenever a network is started.
Alternately, we could disallow destroying any network with active connections. This could cause some existing scripts to fail though.
In the meantime, a workaround for this problem is to restart libvirtd after restarting the network in question - the networks' status will start with all connection counts at 0, and as libvirtd reconnects to each domain it will update the connection counts accordingly. The result will be that all connection counts will once again be correct.
I found related issue when using host existing bridge.
There is a more general problem here. When a libvirt network is stopped and restarted, there is no mechanism in libvirt to notify qemu that any associated guest interfaces need to "re-notify" the network that they exist (and/or reconnect their tap devices). Fixing this will involve having libvirt's network driver call the qemu driver, which will call back into the network driver to fix the problem. There isn't time during this release cycle to fix this entire problem, but a series of patches that I pushed upstream this week will at least reconnect all guest tap devices *when libvirtd is restarted*. So, until we come up with a fully automated way to reconnect/recount/re-whatever guest interfaces when a network is stopped and restarted, a workaround for all libvirts with version 3.2.0 and above is to restart libvirtd. Here are the patches that enable reconnection of disconnected tap devices when libvirtd is restarted: commit e75f5bfbe89e0ea4a98fb9b70fc4649fe861a21c Author: Laine Stump <laine> Date: Fri Mar 17 20:57:18 2017 -0400 util: new function virNetDevGetMaster() commit a4541349516b5aff2da6c5309eda86dcbe6f8233 Author: Laine Stump <laine> Date: Sat Mar 18 14:03:20 2017 -0400 util: new function virNetDevTapAttachBridge() commit 85bcc0220fcdc33452421256ef26d6e0080278b7 Author: Laine Stump <laine> Date: Tue Mar 21 11:24:08 2017 -0400 network: reconnect tap devices during networkNotifyActualDevice
Dan is refactoring the network driver, and this problem could be solved as a side effect.
This bug was closed deferred as a result of bug triage. Please reopen if you disagree and provide justification why this bug should get enough priority. Most important would be information about impact on customer or layered product. Please indicate requested target release.