Hide Forgot
Description of problem: The Broadcom 10Gb NICs are fairly common but currently the support for them is disabled. This bugzilla is a request to evaluate whether it is possible or not to enable support for bnx2x cards in openvswitch-dpdk. Version-Release number of selected component (if applicable): openvswitch-dpdk-2.5
dpdk_nic_bind of bnx2x was successful, but unable to add its dpdk to a ovs-dpdk bridge. Network devices using DPDK-compatible driver ============================================ 0000:05:00.0 'NetXtreme II BCM57810 10 Gigabit Ethernet' drv=vfio-pci unused= 029d107c-e529-48cf-bca5-36b70b8e3eb8 Bridge "ovsbr0" Port "ovsbr0" Interface "ovsbr0" type: internal Port "int0" Interface "int0" type: internal Port "dpdk0" Interface "dpdk0" type: dpdk error: "could not open network device dpdk0 (No such device)" ovs_version: "2.5.0" [root@netqe5 dpdk-multique-scripts]# cat ovs_config_dpdk0_int0.sh ovs-vsctl --if-exists del-br ovsbr0 ovs-vsctl set Open_vSwitch . other_config={} ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=1 ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x200000 ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev ovs-vsctl add-port ovsbr0 dpdk0 \ -- set interface dpdk0 type=dpdk ofport_request=10 ovs-vsctl add-port ovsbr0 int0 -- set interface int0 type=internal ip addr add 192.168.13.105/24 dev int0 ip link set int0 up ovs-vsctl show ovs-ofctl dump-ports ovsbr0 ip r 2016-04-26T18:00:55.287Z|00024|bridge|WARN|could not open network device dpdk0 (No such device) 2016-04-26T18:00:55.298Z|00025|bridge|WARN|could not open network device dpdk0 (No such device) 2016-04-26T18:01:37.339Z|00036|bridge|WARN|could not open network device dpdk0 (No such device) 2016-04-26T18:01:37.343Z|00037|bridge|WARN|could not open network device dpdk0 (No such device) 2016-04-26T18:01:37.348Z|00039|bridge|WARN|could not open network device dpdk0 (No such device) 2016-04-26T18:01:37.350Z|00040|bridge|WARN|could not open network device dpdk0 (No such device)
(In reply to Jean-Tsung Hsiao from comment #1) > dpdk_nic_bind of bnx2x was successful, but unable to add its dpdk to a > ovs-dpdk bridge. > > Network devices using DPDK-compatible driver > ============================================ > 0000:05:00.0 'NetXtreme II BCM57810 10 Gigabit Ethernet' drv=vfio-pci unused= > > 029d107c-e529-48cf-bca5-36b70b8e3eb8 > Bridge "ovsbr0" > Port "ovsbr0" > Interface "ovsbr0" > type: internal > Port "int0" > Interface "int0" > type: internal > Port "dpdk0" > Interface "dpdk0" > type: dpdk > error: "could not open network device dpdk0 (No such device)" > ovs_version: "2.5.0" > > [root@netqe5 dpdk-multique-scripts]# cat ovs_config_dpdk0_int0.sh > ovs-vsctl --if-exists del-br ovsbr0 > ovs-vsctl set Open_vSwitch . other_config={} > ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=1 > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x200000 > ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev > ovs-vsctl add-port ovsbr0 dpdk0 \ > -- set interface dpdk0 type=dpdk ofport_request=10 > > ovs-vsctl add-port ovsbr0 int0 -- set interface int0 type=internal > ip addr add 192.168.13.105/24 dev int0 > ip link set int0 up > ovs-vsctl show > ovs-ofctl dump-ports ovsbr0 > ip r > > 2016-04-26T18:00:55.287Z|00024|bridge|WARN|could not open network device > dpdk0 (No such device) > 2016-04-26T18:00:55.298Z|00025|bridge|WARN|could not open network device > dpdk0 (No such device) > 2016-04-26T18:01:37.339Z|00036|bridge|WARN|could not open network device > dpdk0 (No such device) > 2016-04-26T18:01:37.343Z|00037|bridge|WARN|could not open network device > dpdk0 (No such device) > 2016-04-26T18:01:37.348Z|00039|bridge|WARN|could not open network device > dpdk0 (No such device) > 2016-04-26T18:01:37.350Z|00040|bridge|WARN|could not open network device > dpdk0 (No such device) *** some test-bed info *** [root@netqe5 dpdk-multique-scripts]# rpm -qa | grep dpdk dpdk-tools-2.2.0-3.el7.x86_64 kernel-kernel-networking-dpdk-only-1.0-4.noarch dpdk-2.2.0-3.el7.x86_64 openvswitch-dpdk-2.5.0-3.el7.x86_64 [root@netqe5 dpdk-multique-scripts]# uname -a Linux netqe5.knqe.lab.eng.bos.redhat.com 3.10.0-327.13.1.el7.x86_64 #1 SMP Mon Feb 29 13:22:02 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
(In reply to Jean-Tsung Hsiao from comment #1) > dpdk_nic_bind of bnx2x was successful, but unable to add its dpdk to a > ovs-dpdk bridge. FYI, ability to bind means very little really, basically *any* PCI device can be bound to vfio-pci or uio_pci_generic.
devel_ack, it needs a tiny patch to OVS autoconfig foo but easy enough.
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `ovs-vswitchd --dpdk -l 1,3 -n 4 --socket-mem 4096 1 -- unix:/var/run/openvswitc'. Program terminated with signal 11, Segmentation fault. #0 0x00007f39aeaea01d in bnx2x_dma_alloc (sc=sc@entry=0x7f317ffc9540, size=size@entry=56, dma=dma@entry=0x7f317ffcee48, msg=msg@entry=0x7f39aed37700 "def_sb", align=align@entry=64) at /usr/src/debug/openvswitch-2.5.0/dpdk-2.2.0/drivers/net/bnx2x/bnx2x.c:179 179 z = rte_memzone_reserve_aligned(mz_name, (uint64_t) (size), Missing separate debuginfos, use: debuginfo-install glibc-2.17-133.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-16.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libgcc-4.8.5-8.el7.x86_64 libselinux-2.5-3.el7.x86_64 openssl-libs-1.0.1e-57.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 0x00007f39aeaea01d in bnx2x_dma_alloc (sc=sc@entry=0x7f317ffc9540, size=size@entry=56, dma=dma@entry=0x7f317ffcee48, msg=msg@entry=0x7f39aed37700 "def_sb", align=align@entry=64) at /usr/src/debug/openvswitch-2.5.0/dpdk-2.2.0/drivers/net/bnx2x/bnx2x.c:179 #1 0x00007f39aeaf200e in bnx2x_alloc_hsi_mem (sc=sc@entry=0x7f317ffc9540) at /usr/src/debug/openvswitch-2.5.0/dpdk-2.2.0/drivers/net/bnx2x/bnx2x.c:8862 #2 0x00007f39aeb9bf0e in bnx2x_dev_configure (dev=0x7f39af085cc0 <rte_eth_devices>) at /usr/src/debug/openvswitch-2.5.0/dpdk-2.2.0/drivers/net/bnx2x/bnx2x_ethdev.c:118 #3 0x00007f39aeb0c82f in rte_eth_dev_configure (port_id=<optimized out>, nb_rx_q=nb_rx_q@entry=1, nb_tx_q=nb_tx_q@entry=1, dev_conf=dev_conf@entry=0x7f39aed7ace0 <port_conf>) at /usr/src/debug/openvswitch-2.5.0/dpdk-2.2.0/lib/librte_ether/rte_ethdev.c:995 #4 0x00007f39aed0468c in dpdk_eth_dev_queue_setup (n_txq=1, n_rxq=1, dev=0x7f317ff30540) at lib/netdev-dpdk.c:449 #5 dpdk_eth_dev_init (dev=0x7f317ff30540) at lib/netdev-dpdk.c:515 #6 0x00007f39aed04ae5 in netdev_dpdk_init (netdev_=netdev_@entry=0x7f317ff30540, port_no=0, type=type@entry=DPDK_DEV_ETH) at lib/netdev-dpdk.c:628 #7 0x00007f39aed05954 in netdev_dpdk_construct (netdev=0x7f317ff30540) at lib/netdev-dpdk.c:740 #8 0x00007f39aec7ba0d in netdev_open (name=<optimized out>, type=0x7f39b02aa580 "dpdk", netdevp=netdevp@entry=0x7fffc1fa1ab0) at lib/netdev.c:382 #9 0x00007f39aec03b3d in iface_do_create (port_cfg=<optimized out>, errp=0x7fffc1fa1aa0, netdevp=<synthetic pointer>, ofp_portp=0x7fffc1fa1a9c, iface_cfg=0x7f39b02b07a0, br=0x7f39b02a9f50) at vswitchd/bridge.c:1762 #10 iface_create (port_cfg=0x7f39b02afc80, iface_cfg=0x7f39b02b07a0, br=0x7f39b02a9f50) at vswitchd/bridge.c:1816 #11 bridge_add_ports__ (br=br@entry=0x7f39b02a9f50, wanted_ports=wanted_ports@entry=0x7f39b02aa030, with_requested_port=with_requested_port@entry=true) at vswitchd/bridge.c:892 #12 0x00007f39aec06705 in bridge_add_ports (wanted_ports=0x7f39b02aa030, br=0x7f39b02a9f50) at vswitchd/bridge.c:903 #13 bridge_reconfigure (ovs_cfg=ovs_cfg@entry=0x7f39b02af550) at vswitchd/bridge.c:646 #14 0x00007f39aec09e80 in bridge_run () at vswitchd/bridge.c:2975 #15 0x00007f39aeadb0c5 in main (argc=11, argv=0x7fffc1fa2008) at vswitchd/ovs-vswitchd.c:120
Jun 28 15:42:06 netqe5.knqe.lab.eng.bos.redhat.com kernel: traps: ovs-vswitchd[4085] general protection ip:7f39aeaea01d sp:7fffc1fa16f0 error:0 in ovs-vswitchd[7f39aeab0000+354000] Jun 28 15:42:06 netqe5.knqe.lab.eng.bos.redhat.com ovs-vswitchd[4084]: ovs|00003|daemon_unix(monitor)|ERR|1 crashes: pid 4085 died, killed (Segmentation fault), core dumped, restarting Jun 28 15:42:06 netqe5.knqe.lab.eng.bos.redhat.com kernel: device ovs-netdev entered promiscuous mode # ls -la /core.4085 -rw-------. 1 root root 135065600 Jun 28 15:42 /core.4085
Created attachment 1173546 [details] ovs-vswitchd crash core file ovs-vswitchd crashed on adding dpdk(bnx2x) to a OVS-dpdk bridge.
Reset the status to ASSIGNED.
After QA/code review, it has been decided that bnx2x isn't ready to be supported. The driver is disabled on FD-Beta 2.5.0-11.git20160727. Requesting exception flag to fix on 7.3 branch as well.
Disabled again (so fixed) in openvswitch-dpdk-2.5.0-5.git20160628.el7
Got Errors when adding (bnx2x) dpdk0 and dpdk1 to a dpdk OVS: . ovs_config_dpdk0_dpdk1.sh ovs-vsctl: Error detected while setting up 'dpdk0'. See ovs-vswitchd log for details. ovs-vsctl: Error detected while setting up 'dpdk1'. See ovs-vswitchd log for details. The daemon got "Segmentation fault". Please see the following: 2016-10-05T20:00:03.185Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation 2016-10-05T20:00:03.185Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3 2016-10-05T20:00:03.185Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids 2016-10-05T20:00:03.185Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_state 2016-10-05T20:00:03.185Z|00014|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_zone 2016-10-05T20:00:03.185Z|00015|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_mark 2016-10-05T20:00:03.185Z|00016|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_label 2016-10-05T20:00:03.207Z|00017|bridge|INFO|bridge ovsbr0: added interface ovsbr0 on port 65534 2016-10-05T20:00:03.207Z|00018|bridge|INFO|bridge ovsbr0: using datapath ID 000006e204ed6b43 2016-10-05T20:00:03.207Z|00019|connmgr|INFO|ovsbr0: added service controller "punix:/var/run/openvswitch/ovsbr0.mgmt" 2016-10-05T20:00:03.445Z|00003|daemon_unix(monitor)|ERR|1 crashes: pid 7272 died, killed (Segmentation fault), core dumped, restarting 2016-10-05T20:00:03.457Z|00004|ovs_numa|INFO|Discovered 12 CPU cores on NUMA node 0 2016-10-05T20:00:03.457Z|00005|ovs_numa|INFO|Discovered 12 CPU cores on NUMA node 1 2016-10-05T20:00:03.457Z|00006|ovs_numa|INFO|Discovered 2 NUMA nodes and 24 CPU cores 2016-10-05T20:00:03.457Z|00007|memory|INFO|54736 kB peak resident set size after 86.1 seconds 2016-10-05T20:00:03.457Z|00008|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2016-10-05T20:00:03.457Z|00009|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2016-10-05T20:00:03.466Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation 2016-10-05T20:00:03.466Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3 2016-10-05T20:00:03.466Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids 2016-10-05T20:00:03.466Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_state 2016-10-05T20:00:03.466Z|00014|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_zone 2016-10-05T20:00:03.466Z|00015|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_mark 2016-10-05T20:00:03.466Z|00016|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_label 2016-10-05T20:00:03.482Z|00017|bridge|WARN|could not open network device dpdk0 (Cannot allocate memory) 2016-10-05T20:00:03.488Z|00018|bridge|INFO|bridge ovsbr0: added interface ovsbr0 on port 65534 2016-10-05T20:00:03.488Z|00019|bridge|INFO|bridge ovsbr0: using datapath ID 000006e204ed6b43 [root@netqe5 dpdk-multique-scripts]# rpm -qa | grep openvswitch openvswitch-2.5.0-5.git20160628.el7fdb.x86_64 [root@netqe5 dpdk-multique-scripts]# uname -a Linux netqe5.knqe.lab.eng.bos.redhat.com 3.10.0-510.el7.x86_64 #1 SMP Wed Sep 21 14:44:14 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux [root@netqe5 dpdk-multique-scripts]# set status back to ASSIGNED.
Fixed In Version: openvswitch-dpdk-2.5.0-5.git20160628.el7 vs [root@netqe5 dpdk-multique-scripts]# rpm -qa | grep openvswitch openvswitch-2.5.0-5.git20160628.el7fdb.x86_64 This bug is about openvswitch-dpdk, version + release are meaningless across different packages. Please test the correct package. Back to QA.
The fix was verified --- bnx2x dpdk interface is disabled. See comment #17. Verification steps: * Installed openvswitch-dpdk-2.5.0-5.git20160628.el7. * bind bnx2x ports to get dpdk0 and dpdk1. * start openvswitch * Check errors error: "could not open network device dpdk0 (No such device)" error: "could not open network device dpdk1 (No such device)" * So, the errors confirmed that bnx2x dpdk interface is disabled.