Description of problem: [OVS-DPDK][bnxt] Ping failed over OVS-dpdk/bnxt using openvswitch-2.9.7-1.el7fdn under 7.9 One of the failed automation jobs: https://beaker.engineering.redhat.com/jobs/4690877 The test bed is netqe10/bnxt <-> netqe29/i40e. Need to login netqe10, and ran "systemctl restart openvswitch" to make ping work. Same issue on another test bed: netqe10/bnxt <-> netqe30/nfp Version-Release number of selected component (if applicable): openvswitch-2.9.7-1.el7fdn Rhel-7.9 How reproducible: Reproducible Steps to Reproduce: 1. Use one of the test beds mentioned above. 2. 3. Actual results: Expected results: Additional info:
I also run pvp ovs dpdk performance case failed on bnxt_en card. beaker job: https://beaker.engineering.redhat.com/jobs/4694206 https://beaker.engineering.redhat.com/jobs/4692789
This same issue happened again under 2.9.9. It can be reproduced easily using the following loop: [root@netqe10 home]# for i in {1..5}; do echo Test $i; date; systemctl stop openvswitch; systemctl start openvswitch; sh ovs-dpdk.sh; done Test 1 Fri Apr 16 15:21:12 EDT 2021 default via 10.19.15.254 dev em3 proto dhcp metric 100 10.19.15.0/24 dev em3 proto kernel scope link src 10.19.15.47 metric 100 192.168.9.0/24 dev ovsbr0 proto kernel scope link src 192.168.9.106 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 link_state : down PING 192.168.9.105 (192.168.9.105) 56(84) bytes of data. From 192.168.9.106 icmp_seq=1 Destination Host Unreachable From 192.168.9.106 icmp_seq=2 Destination Host Unreachable From 192.168.9.106 icmp_seq=3 Destination Host Unreachable From 192.168.9.106 icmp_seq=4 Destination Host Unreachable 64 bytes from 192.168.9.105: icmp_seq=5 ttl=64 time=0.526 ms 64 bytes from 192.168.9.105: icmp_seq=6 ttl=64 time=0.071 ms 64 bytes from 192.168.9.105: icmp_seq=7 ttl=64 time=0.134 ms 64 bytes from 192.168.9.105: icmp_seq=8 ttl=64 time=0.089 ms 64 bytes from 192.168.9.105: icmp_seq=9 ttl=64 time=0.131 ms 64 bytes from 192.168.9.105: icmp_seq=10 ttl=64 time=0.129 ms --- 192.168.9.105 ping statistics --- 10 packets transmitted, 6 received, +4 errors, 40% packet loss, time 9000ms rtt min/avg/max/mdev = 0.071/0.180/0.526/0.156 ms, pipe 4 Test 2 Fri Apr 16 15:21:32 EDT 2021 default via 10.19.15.254 dev em3 proto dhcp metric 100 10.19.15.0/24 dev em3 proto kernel scope link src 10.19.15.47 metric 100 192.168.9.0/24 dev ovsbr0 proto kernel scope link src 192.168.9.106 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 link_state : up PING 192.168.9.105 (192.168.9.105) 56(84) bytes of data. From 192.168.9.106 icmp_seq=1 Destination Host Unreachable From 192.168.9.106 icmp_seq=2 Destination Host Unreachable From 192.168.9.106 icmp_seq=3 Destination Host Unreachable From 192.168.9.106 icmp_seq=4 Destination Host Unreachable From 192.168.9.106 icmp_seq=5 Destination Host Unreachable From 192.168.9.106 icmp_seq=6 Destination Host Unreachable From 192.168.9.106 icmp_seq=7 Destination Host Unreachable From 192.168.9.106 icmp_seq=8 Destination Host Unreachable 64 bytes from 192.168.9.105: icmp_seq=9 ttl=64 time=0.554 ms 64 bytes from 192.168.9.105: icmp_seq=10 ttl=64 time=0.089 ms --- 192.168.9.105 ping statistics --- 10 packets transmitted, 2 received, +8 errors, 80% packet loss, time 9001ms rtt min/avg/max/mdev = 0.089/0.321/0.554/0.233 ms, pipe 4 Test 3 Fri Apr 16 15:21:51 EDT 2021 default via 10.19.15.254 dev em3 proto dhcp metric 100 10.19.15.0/24 dev em3 proto kernel scope link src 10.19.15.47 metric 100 192.168.9.0/24 dev ovsbr0 proto kernel scope link src 192.168.9.106 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 link_state : up PING 192.168.9.105 (192.168.9.105) 56(84) bytes of data. 64 bytes from 192.168.9.105: icmp_seq=1 ttl=64 time=1001 ms 64 bytes from 192.168.9.105: icmp_seq=2 ttl=64 time=2.33 ms 64 bytes from 192.168.9.105: icmp_seq=3 ttl=64 time=0.154 ms 64 bytes from 192.168.9.105: icmp_seq=4 ttl=64 time=0.070 ms 64 bytes from 192.168.9.105: icmp_seq=5 ttl=64 time=0.160 ms 64 bytes from 192.168.9.105: icmp_seq=6 ttl=64 time=0.083 ms 64 bytes from 192.168.9.105: icmp_seq=7 ttl=64 time=0.082 ms 64 bytes from 192.168.9.105: icmp_seq=8 ttl=64 time=0.083 ms 64 bytes from 192.168.9.105: icmp_seq=9 ttl=64 time=0.068 ms 64 bytes from 192.168.9.105: icmp_seq=10 ttl=64 time=0.083 ms --- 192.168.9.105 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9000ms rtt min/avg/max/mdev = 0.068/100.508/1001.968/300.487 ms, pipe 2 Test 4 Fri Apr 16 15:22:11 EDT 2021 default via 10.19.15.254 dev em3 proto dhcp metric 100 10.19.15.0/24 dev em3 proto kernel scope link src 10.19.15.47 metric 100 192.168.9.0/24 dev ovsbr0 proto kernel scope link src 192.168.9.106 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 link_state : up PING 192.168.9.105 (192.168.9.105) 56(84) bytes of data. 64 bytes from 192.168.9.105: icmp_seq=1 ttl=64 time=1001 ms 64 bytes from 192.168.9.105: icmp_seq=2 ttl=64 time=2.32 ms 64 bytes from 192.168.9.105: icmp_seq=3 ttl=64 time=0.150 ms 64 bytes from 192.168.9.105: icmp_seq=4 ttl=64 time=0.087 ms 64 bytes from 192.168.9.105: icmp_seq=5 ttl=64 time=0.096 ms 64 bytes from 192.168.9.105: icmp_seq=6 ttl=64 time=0.082 ms 64 bytes from 192.168.9.105: icmp_seq=7 ttl=64 time=0.082 ms 64 bytes from 192.168.9.105: icmp_seq=8 ttl=64 time=0.095 ms 64 bytes from 192.168.9.105: icmp_seq=9 ttl=64 time=0.085 ms 64 bytes from 192.168.9.105: icmp_seq=10 ttl=64 time=0.083 ms --- 192.168.9.105 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9000ms rtt min/avg/max/mdev = 0.082/100.481/1001.736/300.419 ms, pipe 2 Test 5 Fri Apr 16 15:22:31 EDT 2021 default via 10.19.15.254 dev em3 proto dhcp metric 100 10.19.15.0/24 dev em3 proto kernel scope link src 10.19.15.47 metric 100 192.168.9.0/24 dev ovsbr0 proto kernel scope link src 192.168.9.106 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 link_state : up PING 192.168.9.105 (192.168.9.105) 56(84) bytes of data. 64 bytes from 192.168.9.105: icmp_seq=1 ttl=64 time=1002 ms 64 bytes from 192.168.9.105: icmp_seq=2 ttl=64 time=3.51 ms 64 bytes from 192.168.9.105: icmp_seq=3 ttl=64 time=0.149 ms 64 bytes from 192.168.9.105: icmp_seq=4 ttl=64 time=0.137 ms 64 bytes from 192.168.9.105: icmp_seq=5 ttl=64 time=0.142 ms 64 bytes from 192.168.9.105: icmp_seq=6 ttl=64 time=0.065 ms 64 bytes from 192.168.9.105: icmp_seq=7 ttl=64 time=0.080 ms 64 bytes from 192.168.9.105: icmp_seq=8 ttl=64 time=0.094 ms 64 bytes from 192.168.9.105: icmp_seq=9 ttl=64 time=0.085 ms 64 bytes from 192.168.9.105: icmp_seq=10 ttl=64 time=0.084 ms --- 192.168.9.105 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9000ms rtt min/avg/max/mdev = 0.065/100.714/1002.787/300.692 ms, pipe 2 *** ovs-dpdk.sh *** [root@netqe10 home]# cat ovs-dpdk.sh ovs-vsctl set Open_vSwitch . other_config={} ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x002002 ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="4096,4096" ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0xa00a00 ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true ovs-vsctl --if-exists del-br ovsbr0 ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev ovs-vsctl add-port ovsbr0 dpdk-10 -- set interface dpdk-10 type=dpdk ofport_request=10 \ options:dpdk-devargs=0000:83:00.0 options:n_rxq=4 ip a a 192.168.9.106/24 dev ovsbr0 ip lin set ovsbr0 up ip r ovs-vsctl list interface dpdk-10 | grep link_state ping 192.168.9.105 -c 10 [root@netqe10 home]#
Please let me know if we need to have Broadcom take a look at this bug.