Bug 1365194
Summary: | OVS daemon crashed with segfault when adding SR-IOV dpdk to an OVS-dpdk bridge on a guest | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Jean-Tsung Hsiao <jhsiao> | ||||
Component: | openvswitch | Assignee: | Eelco Chaudron <echaudro> | ||||
Status: | CLOSED NEXTRELEASE | QA Contact: | Network QE <network-qe> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 7.3 | CC: | aloughla, atheurer, atragler, dshaks, echaudro, fleitner, jhsiao, kzhang, rcain | ||||
Target Milestone: | rc | ||||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2017-01-06 08:28:20 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | 1335865 | ||||||
Bug Blocks: | |||||||
Attachments: |
|
Description
Jean-Tsung Hsiao
2016-08-08 15:13:26 UTC
Created attachment 1188821 [details]
Core file of ovs-vswitchd
What exactly does "SR-IOV dpdk to OVS-dpdk" mean? What hardware, what setup, the commands used in this process? (In reply to Panu Matilainen from comment #3) > What exactly does "SR-IOV dpdk to OVS-dpdk" mean? What hardware, what setup, > the commands used in this process? Hi Panu, Below are the steps that I used to create dpdk SR-IOV's. Please let me if you have questions. Thanks! Jean ================================ *** steps to create a pair of dpdk SR-IOV's *** * On the host(bare metal) create a guest with RHEL7.3 kernel 481 * On the host create a pair of vfio's from an ixgbe rmmod ixgbe modprobe ixgbe max_vfs=2 * Change guest's xml file to include the following hostnode sections NOTE: This is from my config; Plese edit accordingly. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x81' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </hostdev> * Start the guest and get into the console; Should see the two SR-IOV's from ixgbe. [root@localhost /]# lspci | grep Ethernet 00:03.0 Ethernet controller: Red Hat, Inc Virtio network device 00:0b.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01) 00:0c.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01) * Configure kernel command line to have huge pages; grub2-mkconfig -o /boot/grub2/grub.cfg * Enable unsafe_noiommu rmmod vfio-pci modprobe vfio enable_unsafe_noiommu_mode=Y modprobe vfio-pci cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode * Run dpdk bind dpdk_nic_bind --status dpdk_nic_bind --bind=vfio-pci 0000:00:0b.0 0000:00:0c.0 dpdk_nic_bind --status Hi Panu, If you need to debug on my test-bed, please let me know. Thanks! Jean So... there are a number of things confusing the situation but I dont think the crash has anything to do with SR-IOV: 1) DPDK 2.2 which is bundled in this versions of OVS does *not* support vfio-noiommu at all. So this is just not going to work with current versions. 2) DPDK is initialized with "--dpdk -c 0xa -n 1 --socket-mem 1024 0 --". What is notably missing is blacklisting any virtio-net devices, in the above case you need to add "-b 00:03.0" to the DPDK options to prevent it from trying to take over the virtio-net device that is in use by the kernel. 2) is where the actual crash likely comes from, DPDK and the kernel are fighting over the same virtio-net device which is bound to end in tears. What exactly happens is timing dependent - I got a bunch of complete hangs + kernel oopses but no OVS segfaults. The behavior wrt virtio-net devices in DPDK < 16.04 is simply terrible, but changing that would also invalidate a whole lot of documentation. Not sure what, if anything, to do about it. It will be fixed once we move to OVS 2.6 + DPDK 16.07 Forgotten from the above is that when the in-use virtio-net device is blacklisted as needed for this DPDK version, the behavior with vfio-noiommu bound SR-IOV devices is sane and as expected: [root@localhost ~]# dpdk_nic_bind -s Network devices using DPDK-compatible driver ============================================ 0000:00:09.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci unused= 0000:00:0a.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:00:03.0 'Virtio network device' if= drv=virtio-pci unused=vfio-pci Other network devices ===================== <none> The device discovery snippet from syslog at OVS startup: Aug 18 07:49:56 localhost ovs-ctl: EAL: PCI device 0000:00:03.0 on NUMA socket - 1 Aug 18 07:49:56 localhost ovs-ctl: EAL: probe driver: 1af4:1000 rte_virtio_pmd Aug 18 07:49:56 localhost ovs-ctl: EAL: Device is blacklisted, not initializing Aug 18 07:49:56 localhost ovs-ctl: EAL: PCI device 0000:00:09.0 on NUMA socket -1 Aug 18 07:49:56 localhost ovs-ctl: EAL: probe driver: 8086:10ed rte_ixgbevf_pmd Aug 18 07:49:56 localhost ovs-ctl: EAL: 0000:00:09.0 not managed by VFIO driver, skipping Aug 18 07:49:56 localhost ovs-ctl: EAL: PCI device 0000:00:0a.0 on NUMA socket -1 Aug 18 07:49:56 localhost ovs-ctl: EAL: probe driver: 8086:10ed rte_ixgbevf_pmd Aug 18 07:49:56 localhost ovs-ctl: EAL: 0000:00:0a.0 not managed by VFIO driver, skipping ...ie DPDK didn't find any devices, so the rest is to be expected: [root@localhost ~]# ovs-vsctl show 80ff4298-4b1e-4b7c-89a5-42286cba37ca Bridge "br0" Port "br0" Interface "br0" type: internal ovs_version: "2.5.0" [root@localhost ~]# ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk ovs-vsctl: Error detected while setting up 'dpdk0'. See ovs-vswitchd log for details. [root@localhost ~]# tail -1 /var/log/openvswitch/ovs-vswitchd.log 2016-08-18T11:50:20.262Z|00020|bridge|WARN|could not open network device dpdk0 (No such device) [root@localhost ~]# This will be fixed by update to DPDK >= 16.04, mark the dep (In reply to Panu Matilainen from comment #8) > This will be fixed by update to DPDK >= 16.04, mark the dep Hi Panu, The bug was identified when guest is using 16.04. Please see the original bug description above. So, did you really mean "This will be fixed by update to DPDK >= 16.04" ? Thanks! Jean The installed DPDK 16.04 in the guest is not being used at all in this scenario, see comment #6. > Version-Release number of selected component (if applicable): > [root@localhost dpdk-bond]# rpm -qa | grep openvswitch > openvswitch-2.5.0-5.git20160628.el7fdb.x86_64 > openvswitch-debuginfo-2.5.0-5.git20160628.el7fdb.x86_64 This OVS build includes a bundled copy of DPDK 2.2, OVS 2.5 does not support 16.04. So its really DPDK 2.2 that's giving you trouble here. Jean, The 2.6.1 + dpdk 16.11 is available for testing. Could you verify if Panu's findings resolves this bug? Thanks! (In reply to Flavio Leitner from comment #12) > Jean, > > The 2.6.1 + dpdk 16.11 is available for testing. > Could you verify if Panu's findings resolves this bug? > > Thanks! Ok, I'll give it a try. Thanks! Jean Yes, I have been working on it, and found the following new bug. Not sure if it will block my progress of testing & debugging this BZ. https://bugzilla.redhat.com/show_bug.cgi?id=1409781 OVS daemon keeps crashing on start: [root@localhost jhsiao]# ls -lrt /core.* -rw-------. 1 root root 4698112 Jan 4 06:40 /core.3058 -rw-------. 1 root root 4698112 Jan 4 06:48 /core.3185 -rw-------. 1 root root 4698112 Jan 4 06:49 /core.3312 -rw-------. 1 root root 4698112 Jan 4 06:59 /core.3471 -rw-------. 1 root root 4698112 Jan 4 07:01 /core.3638 -rw-------. 1 root root 4698112 Jan 4 07:03 /core.798 -rw-------. 1 root root 4698112 Jan 4 07:04 /core.2569 -rw-------. 1 root root 4698112 Jan 4 07:05 /core.2702 -rw-------. 1 root root 4698112 Jan 4 07:08 /core.2835 -rw-------. 1 root root 4698112 Jan 4 07:14 /core.3036 -rw-------. 1 root root 13090816 Jan 4 07:19 /core.3170 -rw-------. 1 root root 13090816 Jan 4 07:20 /core.3307 Below piece is from the daemon log: 2017-01-04T12:20:02.797Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA node 0 2017-01-04T12:20:02.797Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU cores 2017-01-04T12:20:02.797Z|00004|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2017-01-04T12:20:02.797Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2017-01-04T12:20:02.798Z|00006|dpdk|INFO|DPDK Enabled, initializing 2017-01-04T12:20:02.798Z|00007|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2017-01-04T12:20:02.798Z|00008|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0xf --socket-mem 1024,1 2017-01-04T12:20:03.299Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted), core dumped) 2017-01-04T12:20:03.299Z|00003|daemon_unix|EMER|could not detach from foreground session Any advice ? Here is a gdb trace: [root@localhost ~]# gdb /core.3170 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... [New LWP 3170] [New LWP 3171] Reading symbols from /usr/sbin/ovs-vswitchd...Reading symbols from /usr/lib/debug/usr/sbin/ovs-vswitchd.debug...done. done. Missing separate debuginfo for Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/f4/0cf56c998444936e569e905018c61a446e19f3 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfi'. Program terminated with signal 6, Aborted. #0 0x00007f3d5f5c31d7 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.17-157.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-26.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libpcap-1.5.3-8.el7.x86_64 libselinux-2.5-6.el7.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 openssl-libs-1.0.1e-60.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 0x00007f3d5f5c31d7 in raise () from /lib64/libc.so.6 #1 0x00007f3d5f5c48c8 in abort () from /lib64/libc.so.6 #2 0x00007f3d61184d95 in __rte_panic ( funcname=funcname@entry=0x7f3d614846b0 <__func__.9925> "rte_eal_init", format=format@entry=0x7f3d61483e7c "Cannot init memory\n%.0s") at /usr/src/debug/openvswitch-2.6.1/dpdk-16.11/lib/librte_eal/linuxapp/eal/eal_debug.c:86 #3 0x00007f3d6128973a in rte_eal_init (argc=argc@entry=5, argv=<optimized out>) at /usr/src/debug/openvswitch-2.6.1/dpdk-16.11/lib/librte_eal/linuxapp/eal/eal.c:814 #4 0x00007f3d61416516 in dpdk_init__ (ovs_other_config=ovs_other_config@entry=0x7f3d62d497f8) at lib/netdev-dpdk.c:3486 #5 0x00007f3d61416b1c in dpdk_init (ovs_other_config=0x7f3d62d497f8) at lib/netdev-dpdk.c:3541 #6 0x00007f3d613030e2 in bridge_run () at vswitchd/bridge.c:2918 #7 0x00007f3d6118828d in main (argc=10, argv=0x7ffcc105a128) at vswitchd/ovs-vswitchd.c:112 (gdb) The crashes identified in comment #16 were caused by wrong specification of dpdk-socket-mem. It should be like dpdk-socket-mem=1024 instead of dpdk-socket-mem="1024,1" as there is only numa for the guest. With ovs-2.6.1-3 and dpdk-16.11-2 a dpdk interface can now be added to an OVS-dpdk bridge without crashing the OVS daemon. Below is an example of guests using ixgbe SR-IOV dpdk interfaces for balance-tcp bonding. *** guest on netqe9 *** [root@localhost jhsiao]# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:00:0b.0 'Ethernet 10G 2P X520 Adapter' drv=vfio-pci unused=ixgbe 0000:00:0c.0 'Ethernet 10G 2P X520 Adapter' drv=vfio-pci unused=ixgbe [root@localhost jhsiao]# ovs-vsctl show 5696b4f5-114d-4ff1-9ab8-c9b5afceca52 Bridge "ovsbr0" Port dpdkbond Interface "dpdk1" type: dpdk options: {n_rxq="4"} Interface "dpdk0" type: dpdk options: {n_rxq="4"} Port "ovsbr0" Interface "ovsbr0" type: internal Port "int0" Interface "int0" type: internal ovs_version: "2.6.1" [root@localhost jhsiao]# ovs-appctl bond/show ---- dpdkbond ---- bond_mode: balance-tcp bond may use recirculation: yes, Recirc-ID : 2 bond-hash-basis: 0 updelay: 0 ms downdelay: 0 ms next rebalance: 165 ms lacp_status: negotiated active slave mac: a0:36:9f:75:08:a2(dpdk1) slave dpdk0: enabled may_enable: true slave dpdk1: enabled active slave may_enable: true hash 36: 78802 kB load *** guest on netqe10 *** [root@localhost ~]# dpdk-devbind -s Network devices using DPDK-compatible driver ============================================ 0000:00:0b.0 'Ethernet 10G 2P X520 Adapter' drv=vfio-pci unused=ixgbe 0000:00:0c.0 'Ethernet 10G 2P X520 Adapter' drv=vfio-pci unused=ixgbe [root@localhost ~]# ovs-vsctl show 5696b4f5-114d-4ff1-9ab8-c9b5afceca52 Bridge "ovsbr0" Port "int0" Interface "int0" type: internal Port "ovsbr0" Interface "ovsbr0" type: internal Port dpdkbond Interface "dpdk1" type: dpdk options: {n_rxq="4"} Interface "dpdk0" type: dpdk options: {n_rxq="4"} ovs_version: "2.6.1" [root@localhost ~]# ovs-appctl bond/show ---- dpdkbond ---- bond_mode: balance-tcp bond may use recirculation: yes, Recirc-ID : 2 bond-hash-basis: 0 updelay: 0 ms downdelay: 0 ms next rebalance: 9879 ms lacp_status: negotiated active slave mac: a0:36:9f:75:08:90(dpdk0) slave dpdk0: enabled active slave may_enable: true hash 185: 6657 kB load slave dpdk1: enabled may_enable: true Closing BZ as it now works in next release package. |