Bug 1111455

Created attachment 910636 [details]
gdb  backtrace

Created attachment 910637 [details]
libvirtd debug log

There are 3 levels of problem here:

1) Although it makes sense to support the <pf> element for the macvtap network pools, not just for hostdev, that isn't the case - <pf> is only supported for <forward mode='hostdev'>. This is a good feature to add upstream and in RHEL7, but isn't appropriate for the stage of development RHEL6 is in.

2) We should (but don't) log a proper error message and fail when there is an attempt to use <pf> for a network other than hostdev.

3) In spite of that, if someone still manages to get such a network defined, we *really* need to do a better job of error recovery :-)

jdenemar already provided me with a patch for (3) to test, and I will also make a patch for (2).

Actually I spoke too soon - <pf> *is* supported for macvtap network pools. The problem in this case is that the VFs exist, but for some reason don't have network interface names assigned to them. On my RHEL6 system, if the VFs exist at all, they have a network device name. Did you possibly blacklist the igbvf driver?

(BTW, the network device name of a VF can be found in /sys/bus/pci/$pciaddress/net - that directory should contain a subdirectory which is the net device.)

Hi Laine 

I didn't blacklist the igbvf driver. The VFs have interface names in my host. The pf value of the bug is a PF device name in the host.

for example 
# ll /sys/bus/pci/drivers/igbvf/0000:0f:10.0/net
total 0
drwxr-xr-x. 5 root root 0 Jun 26 16:15 eth6


virsh nodedev-list also can get them.

# virsh nodedev-list --tree
computer
......
 |       |   +- pci_0000_0f_10_0
 |       |   |   |
 |       |   |   +- net_eth6_7a_f4_c5_1e_6c_41
......

Okay, I have been able to reproduce the "no net device name" problem on my system. The reason this happens is that one of the VFs is currently not bound to the net driver (igbvf in our case); most commonly this is because it is bound to the pci-stub driver (or for RHEL7 to the vfio-pci driver), which in turn usually means that this VF is already in use for PCI passthrough device assignment either on the current guest or another guest (it could also just mean that someone has manually detached the VF with "virsh nodedev-detach".

At any rate, when this is the case, the device shouldn't be included in the pool (as it's obviously currently or intended to be used for a different purpose).

A fix has been pushed upstream:

commit cd7759cb96db642aaa556f78f15801609885a650
Author: Laine Stump <laine>
Date:   Tue Aug 5 16:40:52 2014 -0400

    network: make networkCreateInterfacePool more robust
    
    networkCreateInterfacePool was a bit loose in its error cleanup, which
    could result in a network definition with interfaces in the pool that
    were NULL. This would in turn lead to a libvirtd crash when a guest
    tried to attach an interface using the network with that pool.
    
    In particular this would happen when creating a pool to be used for
    macvtap connections. macvtap needs the netdev name of the virtual
    function in order to use it, and each VF only has a netdev name if it
    is currently bound to a network driver. If one of the VFs of a PF
    happened to be bound to the pci-stub or vfio-pci driver (indicating
    it's already in use for PCI passthrough), or no driver at all, it
    would have no name. In this case networkCreateInterfacePool would
    return an error, but would leave the netdef->forward.nifs set to the
    total number of VFs in the PF. The interface attach that triggered
    calling of networkCreateInterfacePool (it uses a "lazy fill" strategy)
    would simply fail, but the very next attempt to attach an interface
    using the same network pool would result in a crash.
    
    This patch refactors networkCreateInterfacePool to bring it more in
    line with current coding practices (label name, use of a switch with
    no default case) as well as providing the following two changes to
    behavior:
    
    1) If a VF with no netdev name is encountered, just log a warning and
    continue; only fail if exactly 0 devices are found to put in the pool.
    
    2) If the function fails, clean up any partial interface pool and set
    netdef->forward.nifs to 0.

Notes for testing: To reliably reproduce the crash *without* the patch, and prove that the patch fixes it, use the following sequence:

1) define a network as in the description of this bz (i.e. uses "<forward mode='passthrough'> <pf dev='eth2'/>...". Start this network.

2) define a guest that defines an <interface type='hostdev' managed='yes'> specifying the PCI address of VF 0 of the above PF as its <source>.

3) after starting this guest, hot-plug an interface using if.xml given in this bug's description.

4) Again attempt to hot-plug the same interface.

Without the patch, step 3 will fail, and step 4 will cause libvirtd to crash. With the patch, both step 3 and 4 will succeed (unless there are less than 3 VFs, in which case step 3 or 4 could fail, but libvirtd would not crash)

I can not reproduce the bug with below version.

Version:
libvirt-0.10.2-44.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.428.el6.x86_64

<1> Check the VF 0 of SRIOV PF isn't bound to a network driver.
[root@sriov2 jiahu]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     rhel6                          shut off

[root@sriov2 jiahu]# virsh net-list --all
Name                 State      Autostart     Persistent
--------------------------------------------------
default              active     yes           yes
passthrough          inactive   no            yes

[root@sriov2 jiahu]# virsh dumpxml rhel6 | grep interface -b5
...
1142:    <interface type='hostdev' managed='yes'>
1187-      <mac address='52:54:00:73:a0:1d'/>
1228-      <source>
1243-        <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/>
1327-      </source>
1343-      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
1425:    </interface>
...

[root@sriov2 jiahu]# ip link show eth5
101: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1b:21:55:b3:bd brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 0e:6e:8a:f9:68:ed
    vf 1 MAC d6:96:02:9b:3d:40
    vf 2 MAC be:7d:5c:78:27:6b
    vf 3 MAC 5e:bf:87:c4:6f:49
    vf 4 MAC ca:f9:79:05:ab:93
    vf 5 MAC 02:b1:28:0b:aa:eb
    vf 6 MAC 02:1c:01:06:d5:75

[root@sriov2 jiahu]# virsh net-dumpxml passthrough
<network>
  <name>passthrough</name>
  <uuid>dcf96e03-5b98-de86-437a-3fcfe7334019</uuid>
  <forward mode='passthrough'>
    <pf dev='eth5'/>
  </forward>
</network>

[root@sriov2 jiahu]# virsh net-start passthrough
Network passthrough started

[root@sriov2 jiahu]# virsh start rhel6
Domain rhel6 started

[root@sriov2 jiahu]# service libvirtd status
libvirtd (pid  30709) is running...

[root@sriov2 jiahu]# cat if.xml 
<interface type='network'>
   <source network='passthrough'/>
   </interface>

[root@sriov2 jiahu]# virsh attach-device rhel6 if.xml
Device attached successfully

Checked the libvirtd.log:
2014-08-20 06:23:42.938+0000: 448: warning : networkCreateInterfacePool:3555 : VF 0 of SRIOV PF eth5 couldn't be added to the interface pool because it isn't bound to a network driver - possibly in use elsewhere

[root@sriov2 jiahu]# virsh attach-device rhel6 if.xml
Device attached successfully

[root@sriov2 jiahu]# virsh attach-device rhel6 if.xml
Device attached successfully

[root@sriov2 jiahu]# virsh attach-device rhel6 if.xml
Device attached successfully

[root@sriov2 jiahu]# virsh attach-device rhel6 if.xml
Device attached successfully

[root@sriov2 jiahu]# virsh attach-device rhel6 if.xml
Device attached successfully

[root@sriov2 jiahu]# virsh attach-device rhel6 if.xml
error: Failed to attach device from if.xml
error: internal error network 'passthrough' requires exclusive access to interfaces, but none are available

[root@sriov2 jiahu]# service libvirtd status
libvirtd (pid  30709) is running...
[root@sriov2 jiahu]# virsh net-dumpxml passthrough
<network connections='5'>
  <name>passthrough</name>
  <uuid>dcf96e03-5b98-de86-437a-3fcfe7334019</uuid>
  <forward mode='passthrough'>
    <pf dev='eth5'/>
    <interface dev='eth12' connections='1'/>
    <interface dev='eth28' connections='1'/>
    <interface dev='eth29' connections='1'/>
    <interface dev='eth30' connections='1'/>
    <interface dev='eth31' connections='1'/>
    <interface dev='eth34' connections='1'/>
  </forward>
</network>

Check any of macvtap interface in host:
[root@sriov2 jiahu]# virsh dumpxml rhel6 | grep interface -b5
...
2954:    <interface type='direct'>
2984-      <mac address='52:54:00:a4:57:6e'/>
3025-      <source dev='eth34' mode='passthrough'/>
3072-      <target dev='macvtap5'/>
3103-      <alias name='net5'/>
3130-      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
3212:    </interface>
...
[root@sriov2 jiahu]# ip ad | grep macvtap5
76: macvtap5@eth34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
[root@sriov2 jiahu]# 

<2> Check no usable VF's present
[root@sriov2 jiahu]# rmmod igb
[root@sriov2 jiahu]# modprobe igb max_vfs=0
[root@sriov2 jiahu]# ip link show eth5
97: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1b:21:55:b3:bd brd ff:ff:ff:ff:ff:ff

[root@sriov2 jiahu]# virsh net-start passthrough
Network passthrough started

[root@sriov2 jiahu]# virsh attach-device rhel6 if.xml
error: Failed to attach device from if.xml
error: internal error No usable Vf's present on SRIOV PF eth5

[root@sriov2 jiahu]# service libvirtd status
libvirtd (pid  30709) is running...

<3> For common scenario, no any Vfs assigned to running domain
[root@sriov2 jiahu]# virsh start rhel6
Domain rhel6 started

[root@sriov2 jiahu]# virsh dumpxml rhel6 | grep interface -b5
[root@sriov2 jiahu]# ip link show eth5
7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1b:21:55:b3:bd brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 1e:06:5a:0f:47:a8
    vf 1 MAC ca:9b:fb:0d:fb:7d
    vf 2 MAC c2:58:73:6b:e2:8b
    vf 3 MAC b2:09:51:b2:e2:c2
    vf 4 MAC 66:57:83:6f:77:a6
    vf 5 MAC e6:cc:7f:4d:71:2a
    vf 6 MAC 5e:36:72:61:dc:7b
[root@sriov2 jiahu]# virsh net-dumpxml passthrough
<network>
  <name>passthrough</name>
  <uuid>dcf96e03-5b98-de86-437a-3fcfe7334019</uuid>
  <forward mode='passthrough'>
    <pf dev='eth5'/>
  </forward>
</network>

[root@sriov2 jiahu]# virsh net-start passthrough
Network passthrough started

[root@sriov2 jiahu]# service libvirtd status
libvirtd (pid  2653) is running...

[root@sriov2 jiahu]# virsh attach-device rhel6 if.xml
Device attached successfully

[root@sriov2 jiahu]# virsh attach-device rhel6 if.xml
Device attached successfully

[root@sriov2 jiahu]# virsh attach-device rhel6 if.xml
Device attached successfully

[root@sriov2 jiahu]# service libvirtd status
libvirtd (pid  2653) is running...

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1374.html