Bug 1002824

Summary: network driver loses track of connections count for networks that are destroyed with active connections
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: hongming <honzhang>
Component: libvirtAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED DEFERRED QA Contact: Luyao Huang <lhuang>
Severity: medium Docs Contact:
Priority: medium    
Version: ---CC: dyuan, mzhan, rbalakri, xuzhang, yalzhang
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-11 13:00:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description hongming 2013-08-30 05:31:41 UTC 
Description of problem:
One of VFs in a pool of SRIOV VFs can not be available when the VFs from different pci bus

Version-Release number of selected component (if applicable):
libvirt-0.10.2-23.el6.x86_64 

How reproducible:
100%

Steps to Reproduce:
[root@sriov2 images]# lspci|grep 82576
0e:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0e:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0f:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0f:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
10:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
10:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
11:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
11:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)


[root@sriov2 images]# virsh net-start hostdev-net1
Network hostdev-net1 started

[root@sriov2 images]# virsh net-dumpxml hostdev-net1
<network>
  <name>hostdev-net1</name>
  <uuid>a57d0acb-3182-28ad-5c47-aae471ca8084</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/>
  </forward>
</network>

[root@sriov2 images]# virsh destroy r6
Domain r6 destroyed

[root@sriov2 images]# virsh start r6
Domain r6 started

[root@sriov2 images]# virsh dumpxml r6
<domain type='kvm' id='27'>
  <name>r6</name>
  <uuid>60f88110-4c02-0358-46eb-afdce5dc1002</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/kvm-rhel6.3-i386-raw.img'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
<label>system_u:system_r:svirt_t:s0:c123,c735</label>
<imagelabel>system_u:object_r:svirt_image_t:s0:c123,c735</imagelabel>
  </seclabel>
</domain>

[root@sriov2 images]# virsh attach-device r6 vfpool.xml
Device attached successfully

[root@sriov2 images]# virsh attach-device r6 vfpool.xml
Device attached successfully

[root@sriov2 images]# virsh attach-device r6 vfpool.xml
Device attached successfully

[root@sriov2 images]# virsh attach-device r6 vfpool.xml
error: Failed to attach device from vfpool.xml
error: internal error network 'hostdev-net1' requires exclusive access to interfaces, but none are available

[root@sriov2 images]# virsh dumpxml r6 |grep interface -A 4
    <interface type='network'>
      <mac address='52:54:00:25:1f:f1'/>
      <source network='hostdev-net1'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:7c:8b:3c'/>
      <source network='hostdev-net1'/>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:05:6a:3a'/>
      <source network='hostdev-net1'/>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>


Actual results:
One of VFs in a  pool of SRIOV VFs  can not be available  when the VFs from different pci bus. 

Expected results:
All VFs in a pool of SRIOV VFs are available. 

Additional info:
Comment 5 Laine Stump 2014-04-04 09:26:29 UTC 
The summary of this bug doesn't make sense based on the test that was performed. There are 4 VFs, two on bus 0x0F, and two on bus 0x11, so if three VFs were successfully assigned, then there is at least one VF from each bus that was successfully assigned.

Beyond that, the error message that is being reported is saying that all VFs in the pool are already in use. I'm wondering if perhaps another test had already been run on the host prior to this test, resulting in one of the VFs being marked as unused.

Can you please rerun this test on a newly booted host to assure that there wasn't a VF already in use at the time the test was started? Also, before running the test, make sure that hostdev-net1 is marked as autostart, and is started:

   virsh net-autostart hostdev-net1; virsh net-start hostdev-net1

Finally, both before and after your test, please capture the output of the following commands:

  ls -ld /sys/bus/pci/devices/*/driver | egrep '0000:(0F|11)'
  ls -ld /sys/bus/pci/devices/*/physfn

(keep in mind that there is an open bug concerning the fact that hostdev and macvtap networks can be used even if they are down (Bug 880483, also reported more recently as Bug 1002427), and a related issue of that is that these networks are marked as "down" each time libvirtd is restarted, even if they were previously started - setting autostart for these nets is a workaround for that problem).
Comment 7 Jiri Denemark 2014-04-04 21:38:07 UTC 
This bug was not selected to be addressed in Red Hat Enterprise Linux 6. We will look at it again within the Red Hat Enterprise Linux 7 product.
Comment 8 hongming 2014-04-10 05:00:16 UTC 
It can not be reproduced according to the steps of description after new boot and autostarting  hosdev network. But the issues still can be reproduced after restarting network and guest as follows.


[root@sriov2 ~]# ls -ld /sys/bus/pci/devices/*/driver | egrep '0000:(0F|11)'
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.0/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.1/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.2/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.3/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.4/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.5/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.6/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.7/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.0/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.1/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.2/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.3/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.4/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.5/driver -> ../../../../../../bus/pci/drivers/igbvf
[root@sriov2 ~]# ls -ld /sys/bus/pci/devices/*/physfn
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.0/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.1/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.2/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.3/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.4/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.5/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.6/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.7/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.0/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.1/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.2/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.3/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.4/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.5/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.0/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.1/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.2/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.3/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.4/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.5/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.6/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.7/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.0/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.1/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.2/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.3/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.4/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.5/physfn -> ../0000:10:00.1



[root@sriov2 ~]# virsh net-list
Name                 State      Autostart     Persistent
--------------------------------------------------
default              active     yes           yes
hostnet              active     yes           yes

[root@sriov2 ~]# virsh net-dumpxml hostnet
<network>
  <name>hostnet</name>
  <uuid>6b49be3c-bb91-c16d-b475-2929678720f4</uuid>
  <forward mode='hostdev' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x0f' slot='0x10' function='0x0'/>
    <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x0'/>
  </forward>
</network>

[root@sriov2 images]# virsh attach-device a vfpool.xml
Device attached successfully

[root@sriov2 images]# virsh attach-device a vfpool.xml
Device attached successfully

[root@sriov2 images]# virsh net-destroy hostnet
Network hostnet destroyed

[root@sriov2 images]# virsh net-start hostnet
Network hostnet started

[root@sriov2 images]# virsh destroy a
Domain a destroyed

[root@sriov2 images]# virsh start a
Domain a started

[root@sriov2 images]# virsh attach-device a vfpool.xml
error: Failed to attach device from vfpool.xml
error: internal error network 'hostnet' requires exclusive access to interfaces, but none are available

[root@sriov2 images]# ls -ld /sys/bus/pci/devices/*/driver | egrep '0000:(0F|11)'
lrwxrwxrwx. 1 root root 0 Apr 10 00:49 /sys/bus/pci/devices/0000:11:10.0/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.1/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.2/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.3/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.4/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.5/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.6/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.7/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.0/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.1/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.2/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.3/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.4/driver -> ../../../../../../bus/pci/drivers/igbvf
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.5/driver -> ../../../../../../bus/pci/drivers/igbvf
[root@sriov2 images]# ls -ld /sys/bus/pci/devices/*/physfn
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.0/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.1/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.2/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.3/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.4/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.5/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.6/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:10.7/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.0/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.1/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.2/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.3/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.4/physfn -> ../0000:0e:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:0f:11.5/physfn -> ../0000:0e:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.0/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.1/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.2/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.3/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.4/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.5/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.6/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:10.7/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.0/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.1/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.2/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.3/physfn -> ../0000:10:00.1
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.4/physfn -> ../0000:10:00.0
lrwxrwxrwx. 1 root root 0 Apr 10 00:46 /sys/bus/pci/devices/0000:11:11.5/physfn -> ../0000:10:00.1
Comment 9 hongming 2014-04-10 05:15:37 UTC 
The issues in comment 8 can be reproduced in rhel6 and rhel7 . Versions as follows. And also reproduced when the VFs pool includes 4 VFs.

rhel6
libvirt-0.10.2-29.el6_5.4.x86_64
kernel-2.6.32-431.el6.x86_64

rhel7
libvirt-1.1.1-29.el7.x86_64
kernel-3.10.0-115.el7.x86_64
Comment 15 Laine Stump 2016-04-18 19:48:26 UTC 
Based on the sequence of operations given in the examples, I'm 100% certain that the problem here has nothing to do with some of the VFs being on different PFs (that will make no difference to libvirt). Instead, the problem is that the connection count for a network, and for individual devices in a network pool, is lost when a network is net-destroyed, and not properly restored when the network is re-started.

For example, from Comment 8:

root@sriov2 images]# virsh attach-device a vfpool.xml
Device attached successfully

(network connections = 1, 1st dev connections = 1, 2nd dev connections = 0)

[root@sriov2 images]# virsh attach-device a vfpool.xml
Device attached successfully

(network connections = 2, 1st dev connections = 1, 2nd dev connections = 1)

[root@sriov2 images]# virsh net-destroy hostnet
Network hostnet destroyed

[root@sriov2 images]# virsh net-start hostnet
Network hostnet started

(network connections = 1, 1st dev connections = 0, 2nd dev connections = 0)


[root@sriov2 images]# virsh destroy a
Domain a destroyed

(network connections = -1, 1st dev connections = -1, 2nd dev connections = -1)

[root@sriov2 images]# virsh start a
Domain a started

The next time you try to get a device from the pool, libvirt will look for a device with connections == 0, not find any, and report the failure.

Fixing this isn't exactly straightforward, because the necessary info is in the qemu driver's domain status, which the network driver doesn't have direct access to. It will require some sort of system for the network driver to notify all the hypervisor drivers whenever a network is started.
Comment 16 Laine Stump 2016-04-18 19:50:14 UTC 
Alternately, we could disallow destroying any network with active connections. This could cause some existing scripts to fail though.
Comment 17 Laine Stump 2016-04-18 20:41:40 UTC 
In the meantime, a workaround for this problem is to restart libvirtd after restarting the network in question - the networks' status will start with all connection counts at 0, and as libvirtd reconnects to each domain it will update the connection counts accordingly. The result will be that all connection counts will once again be correct.
Comment 19 yalzhang@redhat.com 2016-12-30 10:14:50 UTC 
I found related issue when using host existing bridge.
Comment 20 Laine Stump 2017-03-25 01:47:30 UTC 
There is a more general problem here. When a libvirt network is stopped and restarted, there is no mechanism in libvirt to notify qemu that any associated guest interfaces need to "re-notify" the network that they exist (and/or reconnect their tap devices). Fixing this will involve having libvirt's network driver call the qemu driver, which will call back into the network driver to fix the problem. There isn't time during this release cycle to fix this entire problem, but a series of patches that I pushed upstream this week will at least reconnect all guest tap devices *when libvirtd is restarted*.

So, until we come up with a fully automated way to reconnect/recount/re-whatever guest interfaces when a network is stopped and restarted, a workaround for all libvirts with version 3.2.0 and above is to restart libvirtd.

Here are the patches that enable reconnection of disconnected tap devices when libvirtd is restarted:

commit e75f5bfbe89e0ea4a98fb9b70fc4649fe861a21c
Author: Laine Stump <laine>
Date:   Fri Mar 17 20:57:18 2017 -0400

    util: new function virNetDevGetMaster()

commit a4541349516b5aff2da6c5309eda86dcbe6f8233
Author: Laine Stump <laine>
Date:   Sat Mar 18 14:03:20 2017 -0400

    util: new function virNetDevTapAttachBridge()
    
commit 85bcc0220fcdc33452421256ef26d6e0080278b7
Author: Laine Stump <laine>
Date:   Tue Mar 21 11:24:08 2017 -0400

    network: reconnect tap devices during networkNotifyActualDevice
Comment 21 Laine Stump 2019-03-04 02:51:02 UTC 
Dan is refactoring the network driver, and this problem could be solved as a side effect.
Comment 23 Jaroslav Suchanek 2020-02-11 13:00:29 UTC 
This bug was closed deferred as a result of bug triage.

Please reopen if you disagree and provide justification why this bug should
get enough priority. Most important would be information about impact on
customer or layered product. Please indicate requested target release.