Bug 1111455
Summary: | libvirtd crash when hotplug a nic from macvtap-passthrough network with specified 'pf' | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 6 | Reporter: | hongming <honzhang> | ||||||
Component: | libvirt | Assignee: | Laine Stump <laine> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||
Severity: | high | Docs Contact: | |||||||
Priority: | high | ||||||||
Version: | 6.6 | CC: | dyuan, honzhang, jiahu, jsuchane, mzhan, rbalakri, tlavigne | ||||||
Target Milestone: | rc | ||||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | libvirt-0.10.2-44.el6 | Doc Type: | Bug Fix | ||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2014-10-14 04:22:35 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Description
hongming
2014-06-20 05:41:30 UTC
Created attachment 910636 [details]
gdb backtrace
Created attachment 910637 [details]
libvirtd debug log
There are 3 levels of problem here: 1) Although it makes sense to support the <pf> element for the macvtap network pools, not just for hostdev, that isn't the case - <pf> is only supported for <forward mode='hostdev'>. This is a good feature to add upstream and in RHEL7, but isn't appropriate for the stage of development RHEL6 is in. 2) We should (but don't) log a proper error message and fail when there is an attempt to use <pf> for a network other than hostdev. 3) In spite of that, if someone still manages to get such a network defined, we *really* need to do a better job of error recovery :-) jdenemar already provided me with a patch for (3) to test, and I will also make a patch for (2). Actually I spoke too soon - <pf> *is* supported for macvtap network pools. The problem in this case is that the VFs exist, but for some reason don't have network interface names assigned to them. On my RHEL6 system, if the VFs exist at all, they have a network device name. Did you possibly blacklist the igbvf driver? (BTW, the network device name of a VF can be found in /sys/bus/pci/$pciaddress/net - that directory should contain a subdirectory which is the net device.) Hi Laine I didn't blacklist the igbvf driver. The VFs have interface names in my host. The pf value of the bug is a PF device name in the host. for example # ll /sys/bus/pci/drivers/igbvf/0000:0f:10.0/net total 0 drwxr-xr-x. 5 root root 0 Jun 26 16:15 eth6 virsh nodedev-list also can get them. # virsh nodedev-list --tree computer ...... | | +- pci_0000_0f_10_0 | | | | | | | +- net_eth6_7a_f4_c5_1e_6c_41 ...... Okay, I have been able to reproduce the "no net device name" problem on my system. The reason this happens is that one of the VFs is currently not bound to the net driver (igbvf in our case); most commonly this is because it is bound to the pci-stub driver (or for RHEL7 to the vfio-pci driver), which in turn usually means that this VF is already in use for PCI passthrough device assignment either on the current guest or another guest (it could also just mean that someone has manually detached the VF with "virsh nodedev-detach". At any rate, when this is the case, the device shouldn't be included in the pool (as it's obviously currently or intended to be used for a different purpose). A fix has been pushed upstream: commit cd7759cb96db642aaa556f78f15801609885a650 Author: Laine Stump <laine> Date: Tue Aug 5 16:40:52 2014 -0400 network: make networkCreateInterfacePool more robust networkCreateInterfacePool was a bit loose in its error cleanup, which could result in a network definition with interfaces in the pool that were NULL. This would in turn lead to a libvirtd crash when a guest tried to attach an interface using the network with that pool. In particular this would happen when creating a pool to be used for macvtap connections. macvtap needs the netdev name of the virtual function in order to use it, and each VF only has a netdev name if it is currently bound to a network driver. If one of the VFs of a PF happened to be bound to the pci-stub or vfio-pci driver (indicating it's already in use for PCI passthrough), or no driver at all, it would have no name. In this case networkCreateInterfacePool would return an error, but would leave the netdef->forward.nifs set to the total number of VFs in the PF. The interface attach that triggered calling of networkCreateInterfacePool (it uses a "lazy fill" strategy) would simply fail, but the very next attempt to attach an interface using the same network pool would result in a crash. This patch refactors networkCreateInterfacePool to bring it more in line with current coding practices (label name, use of a switch with no default case) as well as providing the following two changes to behavior: 1) If a VF with no netdev name is encountered, just log a warning and continue; only fail if exactly 0 devices are found to put in the pool. 2) If the function fails, clean up any partial interface pool and set netdef->forward.nifs to 0. Notes for testing: To reliably reproduce the crash *without* the patch, and prove that the patch fixes it, use the following sequence: 1) define a network as in the description of this bz (i.e. uses "<forward mode='passthrough'> <pf dev='eth2'/>...". Start this network. 2) define a guest that defines an <interface type='hostdev' managed='yes'> specifying the PCI address of VF 0 of the above PF as its <source>. 3) after starting this guest, hot-plug an interface using if.xml given in this bug's description. 4) Again attempt to hot-plug the same interface. Without the patch, step 3 will fail, and step 4 will cause libvirtd to crash. With the patch, both step 3 and 4 will succeed (unless there are less than 3 VFs, in which case step 3 or 4 could fail, but libvirtd would not crash) I can not reproduce the bug with below version. Version: libvirt-0.10.2-44.el6.x86_64 qemu-kvm-rhev-0.12.1.2-2.428.el6.x86_64 <1> Check the VF 0 of SRIOV PF isn't bound to a network driver. [root@sriov2 jiahu]# virsh list --all Id Name State ---------------------------------------------------- - rhel6 shut off [root@sriov2 jiahu]# virsh net-list --all Name State Autostart Persistent -------------------------------------------------- default active yes yes passthrough inactive no yes [root@sriov2 jiahu]# virsh dumpxml rhel6 | grep interface -b5 ... 1142: <interface type='hostdev' managed='yes'> 1187- <mac address='52:54:00:73:a0:1d'/> 1228- <source> 1243- <address type='pci' domain='0x0000' bus='0x11' slot='0x10' function='0x1'/> 1327- </source> 1343- <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> 1425: </interface> ... [root@sriov2 jiahu]# ip link show eth5 101: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:55:b3:bd brd ff:ff:ff:ff:ff:ff vf 0 MAC 0e:6e:8a:f9:68:ed vf 1 MAC d6:96:02:9b:3d:40 vf 2 MAC be:7d:5c:78:27:6b vf 3 MAC 5e:bf:87:c4:6f:49 vf 4 MAC ca:f9:79:05:ab:93 vf 5 MAC 02:b1:28:0b:aa:eb vf 6 MAC 02:1c:01:06:d5:75 [root@sriov2 jiahu]# virsh net-dumpxml passthrough <network> <name>passthrough</name> <uuid>dcf96e03-5b98-de86-437a-3fcfe7334019</uuid> <forward mode='passthrough'> <pf dev='eth5'/> </forward> </network> [root@sriov2 jiahu]# virsh net-start passthrough Network passthrough started [root@sriov2 jiahu]# virsh start rhel6 Domain rhel6 started [root@sriov2 jiahu]# service libvirtd status libvirtd (pid 30709) is running... [root@sriov2 jiahu]# cat if.xml <interface type='network'> <source network='passthrough'/> </interface> [root@sriov2 jiahu]# virsh attach-device rhel6 if.xml Device attached successfully Checked the libvirtd.log: 2014-08-20 06:23:42.938+0000: 448: warning : networkCreateInterfacePool:3555 : VF 0 of SRIOV PF eth5 couldn't be added to the interface pool because it isn't bound to a network driver - possibly in use elsewhere [root@sriov2 jiahu]# virsh attach-device rhel6 if.xml Device attached successfully [root@sriov2 jiahu]# virsh attach-device rhel6 if.xml Device attached successfully [root@sriov2 jiahu]# virsh attach-device rhel6 if.xml Device attached successfully [root@sriov2 jiahu]# virsh attach-device rhel6 if.xml Device attached successfully [root@sriov2 jiahu]# virsh attach-device rhel6 if.xml Device attached successfully [root@sriov2 jiahu]# virsh attach-device rhel6 if.xml error: Failed to attach device from if.xml error: internal error network 'passthrough' requires exclusive access to interfaces, but none are available [root@sriov2 jiahu]# service libvirtd status libvirtd (pid 30709) is running... [root@sriov2 jiahu]# virsh net-dumpxml passthrough <network connections='5'> <name>passthrough</name> <uuid>dcf96e03-5b98-de86-437a-3fcfe7334019</uuid> <forward mode='passthrough'> <pf dev='eth5'/> <interface dev='eth12' connections='1'/> <interface dev='eth28' connections='1'/> <interface dev='eth29' connections='1'/> <interface dev='eth30' connections='1'/> <interface dev='eth31' connections='1'/> <interface dev='eth34' connections='1'/> </forward> </network> Check any of macvtap interface in host: [root@sriov2 jiahu]# virsh dumpxml rhel6 | grep interface -b5 ... 2954: <interface type='direct'> 2984- <mac address='52:54:00:a4:57:6e'/> 3025- <source dev='eth34' mode='passthrough'/> 3072- <target dev='macvtap5'/> 3103- <alias name='net5'/> 3130- <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> 3212: </interface> ... [root@sriov2 jiahu]# ip ad | grep macvtap5 76: macvtap5@eth34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 [root@sriov2 jiahu]# <2> Check no usable VF's present [root@sriov2 jiahu]# rmmod igb [root@sriov2 jiahu]# modprobe igb max_vfs=0 [root@sriov2 jiahu]# ip link show eth5 97: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:55:b3:bd brd ff:ff:ff:ff:ff:ff [root@sriov2 jiahu]# virsh net-start passthrough Network passthrough started [root@sriov2 jiahu]# virsh attach-device rhel6 if.xml error: Failed to attach device from if.xml error: internal error No usable Vf's present on SRIOV PF eth5 [root@sriov2 jiahu]# service libvirtd status libvirtd (pid 30709) is running... <3> For common scenario, no any Vfs assigned to running domain [root@sriov2 jiahu]# virsh start rhel6 Domain rhel6 started [root@sriov2 jiahu]# virsh dumpxml rhel6 | grep interface -b5 [root@sriov2 jiahu]# ip link show eth5 7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:55:b3:bd brd ff:ff:ff:ff:ff:ff vf 0 MAC 1e:06:5a:0f:47:a8 vf 1 MAC ca:9b:fb:0d:fb:7d vf 2 MAC c2:58:73:6b:e2:8b vf 3 MAC b2:09:51:b2:e2:c2 vf 4 MAC 66:57:83:6f:77:a6 vf 5 MAC e6:cc:7f:4d:71:2a vf 6 MAC 5e:36:72:61:dc:7b [root@sriov2 jiahu]# virsh net-dumpxml passthrough <network> <name>passthrough</name> <uuid>dcf96e03-5b98-de86-437a-3fcfe7334019</uuid> <forward mode='passthrough'> <pf dev='eth5'/> </forward> </network> [root@sriov2 jiahu]# virsh net-start passthrough Network passthrough started [root@sriov2 jiahu]# service libvirtd status libvirtd (pid 2653) is running... [root@sriov2 jiahu]# virsh attach-device rhel6 if.xml Device attached successfully [root@sriov2 jiahu]# virsh attach-device rhel6 if.xml Device attached successfully [root@sriov2 jiahu]# virsh attach-device rhel6 if.xml Device attached successfully [root@sriov2 jiahu]# service libvirtd status libvirtd (pid 2653) is running... Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-1374.html |