Created attachment 1188821 [details]
Core file of ovs-vswitchd

What exactly does "SR-IOV dpdk to OVS-dpdk" mean? What hardware, what setup, the commands used in this process?

(In reply to Panu Matilainen from comment #3)
> What exactly does "SR-IOV dpdk to OVS-dpdk" mean? What hardware, what setup,
> the commands used in this process?

Hi Panu,

Below are the steps that I used to create dpdk SR-IOV's.

Please let me if you have questions.

Thanks!

Jean
================================

*** steps to create a pair of dpdk SR-IOV's ***

* On the host(bare metal) create a guest with RHEL7.3 kernel 481

* On the host create a pair of vfio's from an ixgbe

rmmod ixgbe
modprobe ixgbe max_vfs=2

* Change guest's xml file to include the following hostnode sections

NOTE: This is from my config; Plese edit accordingly.

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </hostdev>

* Start the guest and get into the console; Should see the two SR-IOV's from ixgbe.

[root@localhost /]# lspci | grep Ethernet
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
00:0b.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
00:0c.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

* Configure kernel command line to have huge pages;  grub2-mkconfig -o /boot/grub2/grub.cfg

* Enable unsafe_noiommu

rmmod vfio-pci
modprobe vfio enable_unsafe_noiommu_mode=Y
modprobe vfio-pci
cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

* Run dpdk bind
dpdk_nic_bind --status
dpdk_nic_bind --bind=vfio-pci 0000:00:0b.0 0000:00:0c.0
dpdk_nic_bind --status

Hi Panu,

If you need to debug on my test-bed, please let me know.

Thanks!

Jean

So... there are a number of things confusing the situation but I dont think the crash has anything to do with SR-IOV:

1) DPDK 2.2 which is bundled in this versions of OVS does *not* support vfio-noiommu at all. So this is just not going to work with current versions.
2) DPDK is initialized with "--dpdk -c 0xa -n 1 --socket-mem 1024 0 --". What is notably missing is blacklisting any virtio-net devices, in the above case you need to add "-b 00:03.0" to the DPDK options to prevent it from trying to take over the virtio-net device that is in use by the kernel.

2) is where the actual crash likely comes from, DPDK and the kernel are fighting over the same virtio-net device which is bound to end in tears. What exactly happens is timing dependent - I got a bunch of complete hangs + kernel oopses but no OVS segfaults.

The behavior wrt virtio-net devices in DPDK < 16.04 is simply terrible, but changing that would also invalidate a whole lot of documentation. Not sure what, if anything, to do about it. It will be fixed once we move to OVS 2.6 + DPDK 16.07

Forgotten from the above is that when the in-use virtio-net device is blacklisted as needed for this DPDK version, the behavior with vfio-noiommu bound SR-IOV devices is sane and as expected:

[root@localhost ~]# dpdk_nic_bind -s 
Network devices using DPDK-compatible driver
============================================
0000:00:09.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci unused=
0000:00:0a.0 '82599 Ethernet Controller Virtual Function' drv=vfio-pci unused=

Network devices using kernel driver
===================================
0000:00:03.0 'Virtio network device' if= drv=virtio-pci unused=vfio-pci 

Other network devices
=====================
<none>

The device discovery snippet from syslog at OVS startup:
Aug 18 07:49:56 localhost ovs-ctl: EAL: PCI device 0000:00:03.0 on NUMA socket -
1
Aug 18 07:49:56 localhost ovs-ctl: EAL:   probe driver: 1af4:1000 rte_virtio_pmd
Aug 18 07:49:56 localhost ovs-ctl: EAL:   Device is blacklisted, not initializing
Aug 18 07:49:56 localhost ovs-ctl: EAL: PCI device 0000:00:09.0 on NUMA socket -1
Aug 18 07:49:56 localhost ovs-ctl: EAL:   probe driver: 8086:10ed rte_ixgbevf_pmd
Aug 18 07:49:56 localhost ovs-ctl: EAL:   0000:00:09.0 not managed by VFIO driver, skipping
Aug 18 07:49:56 localhost ovs-ctl: EAL: PCI device 0000:00:0a.0 on NUMA socket -1
Aug 18 07:49:56 localhost ovs-ctl: EAL:   probe driver: 8086:10ed rte_ixgbevf_pmd
Aug 18 07:49:56 localhost ovs-ctl: EAL:   0000:00:0a.0 not managed by VFIO driver, skipping

...ie DPDK didn't find any devices, so the rest is to be expected:

[root@localhost ~]# ovs-vsctl show
80ff4298-4b1e-4b7c-89a5-42286cba37ca
    Bridge "br0"
        Port "br0"
            Interface "br0"
                type: internal
    ovs_version: "2.5.0"
[root@localhost ~]# ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
ovs-vsctl: Error detected while setting up 'dpdk0'.  See ovs-vswitchd log for details.
[root@localhost ~]# tail -1 /var/log/openvswitch/ovs-vswitchd.log 
2016-08-18T11:50:20.262Z|00020|bridge|WARN|could not open network device dpdk0 (No such device)
[root@localhost ~]#

This will be fixed by update to DPDK >= 16.04, mark the dep

(In reply to Panu Matilainen from comment #8)
> This will be fixed by update to DPDK >= 16.04, mark the dep

Hi Panu,

The bug was identified when guest is using 16.04. Please see the original bug description above.

So, did you really mean "This will be fixed by update to DPDK >= 16.04" ?

Thanks!

Jean

The installed DPDK 16.04 in the guest is not being used at all in this scenario, see comment #6.

> Version-Release number of selected component (if applicable):
> [root@localhost dpdk-bond]# rpm -qa | grep openvswitch
> openvswitch-2.5.0-5.git20160628.el7fdb.x86_64
> openvswitch-debuginfo-2.5.0-5.git20160628.el7fdb.x86_64

This OVS build includes a bundled copy of DPDK 2.2, OVS 2.5 does not support 16.04. So its really DPDK 2.2 that's giving you trouble here.

Jean,

The 2.6.1 + dpdk 16.11 is available for testing.
Could you verify if Panu's findings resolves this bug?

Thanks!

(In reply to Flavio Leitner from comment #12)
> Jean,
> 
> The 2.6.1 + dpdk 16.11 is available for testing.
> Could you verify if Panu's findings resolves this bug?
> 
> Thanks!

Ok, I'll give it a try.

Thanks!

Jean

Yes, I have been working on it, and found the following new bug. Not sure if it will block my progress of testing & debugging this BZ.

https://bugzilla.redhat.com/show_bug.cgi?id=1409781

OVS daemon keeps crashing on start:
[root@localhost jhsiao]# ls -lrt /core.*
-rw-------. 1 root root  4698112 Jan  4 06:40 /core.3058
-rw-------. 1 root root  4698112 Jan  4 06:48 /core.3185
-rw-------. 1 root root  4698112 Jan  4 06:49 /core.3312
-rw-------. 1 root root  4698112 Jan  4 06:59 /core.3471
-rw-------. 1 root root  4698112 Jan  4 07:01 /core.3638
-rw-------. 1 root root  4698112 Jan  4 07:03 /core.798
-rw-------. 1 root root  4698112 Jan  4 07:04 /core.2569
-rw-------. 1 root root  4698112 Jan  4 07:05 /core.2702
-rw-------. 1 root root  4698112 Jan  4 07:08 /core.2835
-rw-------. 1 root root  4698112 Jan  4 07:14 /core.3036
-rw-------. 1 root root 13090816 Jan  4 07:19 /core.3170
-rw-------. 1 root root 13090816 Jan  4 07:20 /core.3307

Below piece is from the daemon log:

2017-01-04T12:20:02.797Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA node 0
2017-01-04T12:20:02.797Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU cores
2017-01-04T12:20:02.797Z|00004|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2017-01-04T12:20:02.797Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2017-01-04T12:20:02.798Z|00006|dpdk|INFO|DPDK Enabled, initializing
2017-01-04T12:20:02.798Z|00007|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2017-01-04T12:20:02.798Z|00008|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0xf --socket-mem 1024,1
2017-01-04T12:20:03.299Z|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted), core dumped)
2017-01-04T12:20:03.299Z|00003|daemon_unix|EMER|could not detach from foreground session

Any advice ?

Here is a gdb trace:

[root@localhost ~]# gdb /core.3170
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 3170]
[New LWP 3171]
Reading symbols from /usr/sbin/ovs-vswitchd...Reading symbols from /usr/lib/debug/usr/sbin/ovs-vswitchd.debug...done.
done.
Missing separate debuginfo for 
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/f4/0cf56c998444936e569e905018c61a446e19f3
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfi'.
Program terminated with signal 6, Aborted.
#0  0x00007f3d5f5c31d7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-157.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-26.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libpcap-1.5.3-8.el7.x86_64 libselinux-2.5-6.el7.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 openssl-libs-1.0.1e-60.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64 zlib-1.2.7-17.el7.x86_64

(gdb) bt
#0  0x00007f3d5f5c31d7 in raise () from /lib64/libc.so.6
#1  0x00007f3d5f5c48c8 in abort () from /lib64/libc.so.6
#2  0x00007f3d61184d95 in __rte_panic (
    funcname=funcname@entry=0x7f3d614846b0 <__func__.9925> "rte_eal_init", 
    format=format@entry=0x7f3d61483e7c "Cannot init memory\n%.0s")
    at /usr/src/debug/openvswitch-2.6.1/dpdk-16.11/lib/librte_eal/linuxapp/eal/eal_debug.c:86
#3  0x00007f3d6128973a in rte_eal_init (argc=argc@entry=5, argv=<optimized out>)
    at /usr/src/debug/openvswitch-2.6.1/dpdk-16.11/lib/librte_eal/linuxapp/eal/eal.c:814
#4  0x00007f3d61416516 in dpdk_init__ (ovs_other_config=ovs_other_config@entry=0x7f3d62d497f8)
    at lib/netdev-dpdk.c:3486
#5  0x00007f3d61416b1c in dpdk_init (ovs_other_config=0x7f3d62d497f8)
    at lib/netdev-dpdk.c:3541
#6  0x00007f3d613030e2 in bridge_run () at vswitchd/bridge.c:2918
#7  0x00007f3d6118828d in main (argc=10, argv=0x7ffcc105a128) at vswitchd/ovs-vswitchd.c:112
(gdb)

The crashes identified in comment #16 were caused by wrong specification of dpdk-socket-mem. It should be like dpdk-socket-mem=1024 instead of dpdk-socket-mem="1024,1" as there is only numa for the guest.

With ovs-2.6.1-3 and dpdk-16.11-2 a dpdk interface can now be added to an OVS-dpdk bridge without crashing the OVS daemon.

Below is an example of guests using ixgbe SR-IOV dpdk interfaces for balance-tcp bonding.

*** guest on netqe9 ***

[root@localhost jhsiao]# dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
0000:00:0b.0 'Ethernet 10G 2P X520 Adapter' drv=vfio-pci unused=ixgbe
0000:00:0c.0 'Ethernet 10G 2P X520 Adapter' drv=vfio-pci unused=ixgbe

[root@localhost jhsiao]# ovs-vsctl show
5696b4f5-114d-4ff1-9ab8-c9b5afceca52
    Bridge "ovsbr0"
        Port dpdkbond
            Interface "dpdk1"
                type: dpdk
                options: {n_rxq="4"}
            Interface "dpdk0"
                type: dpdk
                options: {n_rxq="4"}
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "int0"
            Interface "int0"
                type: internal
    ovs_version: "2.6.1"
[root@localhost jhsiao]# ovs-appctl bond/show
---- dpdkbond ----
bond_mode: balance-tcp
bond may use recirculation: yes, Recirc-ID : 2
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 165 ms
lacp_status: negotiated
active slave mac: a0:36:9f:75:08:a2(dpdk1)

slave dpdk0: enabled
	may_enable: true

slave dpdk1: enabled
	active slave
	may_enable: true
	hash 36: 78802 kB load

*** guest on netqe10 ***

[root@localhost ~]# dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
0000:00:0b.0 'Ethernet 10G 2P X520 Adapter' drv=vfio-pci unused=ixgbe
0000:00:0c.0 'Ethernet 10G 2P X520 Adapter' drv=vfio-pci unused=ixgbe

[root@localhost ~]# ovs-vsctl show
5696b4f5-114d-4ff1-9ab8-c9b5afceca52
    Bridge "ovsbr0"
        Port "int0"
            Interface "int0"
                type: internal
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port dpdkbond
            Interface "dpdk1"
                type: dpdk
                options: {n_rxq="4"}
            Interface "dpdk0"
                type: dpdk
                options: {n_rxq="4"}
    ovs_version: "2.6.1"
[root@localhost ~]# ovs-appctl bond/show
---- dpdkbond ----
bond_mode: balance-tcp
bond may use recirculation: yes, Recirc-ID : 2
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 9879 ms
lacp_status: negotiated
active slave mac: a0:36:9f:75:08:90(dpdk0)

slave dpdk0: enabled
	active slave
	may_enable: true
	hash 185: 6657 kB load

slave dpdk1: enabled
	may_enable: true

Closing BZ as it now works in next release package.